`services`

This section defines the services that are required, based on the service types provided by the application. The services item is a JSON array of objects, each defining a service. Each service object has the following fields:

Key Description

Key	Description
`serviceName`	The name of the service. This must be unique across all services defined for the application.
`serviceType`	The service type. This must be one of the service types provided by the application.
`description`	A free format description of the service.
`config`	A JSON object that provides both; framework defined and application defined additional configuration for the service. The format of this object is described below.

serviceName

The name of the service.
This must be unique across all services defined for the application.

serviceType

The service type.
This must be one of the service types provided by the application.

description

A free format description of the service.

config

A JSON object that provides both; framework defined and application defined additional configuration for the service.
The format of this object is described below.

Ensure that different services are not configured to publish to the same Diffusion topic unless you are aware of and accept the behaviour of updates when this is the case.
With such configuration, if an application is using streaming update mode, there will be two concurrent updaters of the same topic, which causes conflict during publication.
Similarly, if the application is using simple update mode, it may work, but the updates would be interleaved, and ordering of updates would be indeterminate.

Service `config`

Key Description Default

Key	Description	Default
`sharedConfigName`	The name of a sharedConfig (configured as a separate item in `sharedConfigs`). Refer to the sharedConfigs section.	No sharedConfig is used by the service.
`framework`	A JSON object defining any additional framework specific configuration for the service, and dependent upon the service mode (as defined by the service type). See below for details.	All framework options will be defaulted.
`application`	A JSON object defining any additional configuration required by the service, as defined by the service type of the application. `The application documentation must provide details of what can be supplied by users for this configuration for specific service type.` `The user supplied values for this configuration will be passed to the application as a map of parameters when the service is added.`	No parameters will be passed to the application.
`state`	State of a service. This can be `ACTIVE` OR `PAUSED`. If `PAUSED` is used, the service will be added but not started and will be in paused state. This config for a service will be updated by the framework, if a service is paused/resumed.	`ACTIVE`

sharedConfigName

The name of a sharedConfig (configured as a separate item in sharedConfigs).
Refer to the sharedConfigs section.

No sharedConfig is used by the service.

framework

A JSON object defining any additional framework specific configuration for the service, and dependent upon the service mode (as defined by the service type).
See below for details.

All framework options will be defaulted.

application

A JSON object defining any additional configuration required by the service, as defined by the service type of the application.

The application documentation must provide details of what can be supplied by users for this configuration for specific service type.

The user supplied values for this configuration will be passed to the application as a map of parameters when the service is added.

No parameters will be passed to the application.

state

State of a service. This can be ACTIVE OR PAUSED.
If PAUSED is used, the service will be added but not started and will be in paused state.
This config for a service will be updated by the framework, if a service is paused/resumed.

ACTIVE

Framework configuration for Source Services

The following framework specific configuration items can be provided for both; streaming and polling source services:

Key Description Default

Key	Description	Default
`publicationRetries`	The number of times an update will be retried, in the event of transient errors before failing.	`5`
`retryIntervalMs`	The time interval in milliseconds between publication retries.	`5000`
`topicProperties`	The topic properties which will be used together with the source service properties specified by application, to create topics for the service.	See below for details
`payloadConverters`	The array of payloadConverter details. One or more converters can be set in a chain, which will be executed in the order of converters in the array. See below for detailed configuration parameters. If it is not specified, it is inferred from `topicType` in `topicProperties`, or payload converters defined for the service type will be used. See payload converter overview for details.	N/A

publicationRetries

The number of times an update will be retried, in the event of transient errors before failing.

5

retryIntervalMs

The time interval in milliseconds between publication retries.

5000

topicProperties

The topic properties which will be used together with the source service properties specified by application, to create topics for the service.

See below for details

payloadConverters

The array of payloadConverter details. One or more converters can be set in a chain, which will be executed in the order of converters in the array. See below for detailed configuration parameters.

If it is not specified, it is inferred from topicType in topicProperties, or payload converters defined for the service type will be used. See payload converter overview for details.

N/A

When configuring payloadConverters, ensure that the value published by the service type is compatible with the type of value expected by the first converter in the list to convert. This information can be found in the documentation of service types supported by the application.

topicProperties

The following topic properties configuration items can be provided for both; streaming and polling source services:

Key Description Default

Key	Description	Default
`topicType`	The Diffusion topic type of the topic where data will be published for this service. See below for accepted values. If it is not specified, it is inferred from `payloadConverters`, or the TopicType defined for the service type will be used. See payload converter overview for details.	N/A
`persistencePolicy`	Specifies the persistence policy that determines the lifespan of all topics created by the service. `SERVICE` indicates that topics created by the service will be removed from the Diffusion server whenever the service is paused or removed. Topics created by the service will also be removed when the framework connection with Diffusion is lost (causing a pause) or when the application closes. `SESSION` indicates that topics created by the service will be removed from the server when the framework connection to Diffusion is lost or when the application closes. `SERVER` indicates that topics created by the service persist even when the framework connection to Diffusion is lost and when the application closes, as long as the Diffusion server is running. `FULL` indicates that topics created by the service will persist indefinitely (assuming that persistence is enabled at the Diffusion server) and can only be removed by some other mechanism (for example using the `Publisher.remove` method or using the Diffusion console).	`SESSION`
`timeSeries`	Specifies whether topics created by the service should be Diffusion `time series` topics. See below for details	false
`timeSeriesRetainedRange`	Specifies time series retain range, if `timeSeries` is true. See below for details	null
`timeSeriesSubscriptionRange`	Specifies time series subscription range, if `timeSeries` is true. See below for details	null
`publishValuesOnly`	Specifies whether the Diffusion topic should publish only full values. See below for details.	false
`dontRetainValue`	Specifies whether or not a topic should not retain its last value. See below for details.	false

topicType

The Diffusion topic type of the topic where data will be published for this service. See below for accepted values.

If it is not specified, it is inferred from payloadConverters, or the TopicType defined for the service type will be used. See payload converter overview for details.

N/A

persistencePolicy

Specifies the persistence policy that determines the lifespan of all topics created by the service.
SERVICE indicates that topics created by the service will be removed from the Diffusion server whenever the service is paused or removed.
Topics created by the service will also be removed when the framework connection with Diffusion is lost (causing a pause) or when the application closes.
SESSION indicates that topics created by the service will be removed from the server when the framework connection to Diffusion is lost or when the application closes.
SERVER indicates that topics created by the service persist even when the framework connection to Diffusion is lost and when the application closes, as long as the Diffusion server is running.
FULL indicates that topics created by the service will persist indefinitely (assuming that persistence is enabled at the Diffusion server) and can only be removed by some other mechanism (for example using the Publisher.remove method or using the Diffusion console).

SESSION

timeSeries

Specifies whether topics created by the service should be Diffusion time series topics.
See below for details

false

timeSeriesRetainedRange

Specifies time series retain range, if timeSeries is true.
See below for details

null

timeSeriesSubscriptionRange

Specifies time series subscription range, if timeSeries is true.
See below for details

null

publishValuesOnly

Specifies whether the Diffusion topic should publish only full values.
See below for details.

false

dontRetainValue

Specifies whether or not a topic should not retain its last value.
See below for details.

false

If a service creates persistent topics, and that service is subsequently changed to have a less persistent option, for example, SESSION to SERVICE or SERVER to SESSION, then any topics it has created earlier will need to be removed manually.

timeSeries

The timeSeries parameter specifies whether topics created by the service should be Diffusion time series topics.

A time series is a sequence of events. Each event contains a value and has server-assigned metadata comprised of a sequence number, timestamp, and author.

A time series topic allows sessions to access a time series that is maintained by the server. A time series topic has an associated event data type which is defined by the specified topic type and determines the type of value associated with each event.

If setting to true then the timeSeriesRetainedRange and timeSeriesSubscriptionRange parameters may also optionally be specified. If setting to false these parameters are ignored.

Time series topics are an advanced feature of Diffusion. They should be used with extreme care as each retained event will be retained in memory at the server. They are not designed for very large event values and the user should configure the timeSeriesRetainedRange property with the memory impact of the topic upon the server in mind. Consider that the topic will consume at least the average event value size multipled by the number in the retained range. If persisted this memory overhead wil be doubled and if the server is operating in a clustered mode there is further overhead still. See the Diffusion documentation for more information about time series topics.

timeSeriesRetainedRange

The timeSeriesRetainedRange parameter configures the range of historic events retained by a time series topic.

When a new event is added to the time series, older events that fall outside the range are discarded.

If the parameter is not specified or null value is used, a time series topic will retain the ten most recent events.

The parameter value is a time series range expression string composed of one or more constraint clauses. Constraints are combined to provide a range of events from the end of the time series.

limit constraint: A limit constraint specifies the maximum number of events from the end of the time series.
last clause: A last constraint specifies the maximum duration of events from the end of the time series. The duration is expressed as an integer followed by one of the following time units. MS - milliseconds; S - seconds; H - hours.

If a range expression contains multiple constraints, the constraint that selects the smallest range is used.

Retained Range expressions
Parameter value	Meaning
`limit 5`	The five most recent events
`last 10s`	All events that are no more than ten seconds older than the latest event
`last 10s limit 5`	The five most recent events that are no more than ten seconds older than the latest event

Range expressions are not case sensitive: limit 5 last 10s is equivalent to LIMIT 5 LAST 10S.

timeSeriesSubscriptionRange

The timeSeriesSubscriptionRange parameter configures a time series topic to send a range of historic events from the end of the time series to new subscribers.

Configuring a subscription range is a convenient way to provide new subscribers with an appropriate subset of the latest events.

If the parameter is not specified or null value is used, default subscription range is used. The default subscription range depends on whether the topic is configured to publish delta streams. If delta streams are enabled, new subscribers are sent the latest event if one exists. If delta streams are disabled, new subscribers are sent no events. Delta streams are enabled by default and can be disabled by setting publishValuesOnly to true.

A larger subscription range can be configured using the subscriptionRange parameter. Regardless of the parameter setting, if delta streams are enabled, new subscribers will be sent at least the latest event if one exists.

If the range of events is insufficient, the subscribing session can use a range query to retrieve older events.

The value for this property is a time series range expression, following the format used for timeSeriesRetainedRange.

publishValuesOnly

The publishValuesOnly parameter specifies whether a topic should publish only values.

By default, a topic that supports delta streams will publish the difference between two values (a delta) when doing so is more efficient than publishing the complete new value.

Setting publishValuesOnly to true disables this behaviour so that deltas are never published. Doing so is usually not recommended because it will result in more data being transmitted, less efficient use of network resources, and increased transmission latency. On the other hand, calculating deltas can require significant CPU from the server or, if update streams are used, from the updating client. The CPU cost will be higher if there are many differences between successive values, in which case delta streams confer fewer benefits. If successive values are unrelated to each other, consider setting to true. Also consider setting to true if the network capacity is high and the bandwidth savings of deltas are not required.

dontRetainValue

The dontRetainValue parameter specifies whether a topic should not retain its last value.

By default, a topic will retain its latest value. The latest value will be sent to new subscribers. Setting this property to true disables this behaviour. New subscribers will not be sent an initial value. No value will be returned for fetch operations that select the topic. This is useful for data streams where the values are only transiently valid.

Setting to true also disables delta streams, regardless of the publishValuesOnly value. If subsequent values are likely to be related, delta streams usually provide performance benefits (see publishValuesOnly above). Consider leaving as false to benefit from delta streams, even if there is no other requirement to retain the last value.

Bearing in mind the performance trade-offs of disabling delta streams, there are two reasons to consider setting to true. First, it stops the server and each subscribed client from keeping a copy of the value, reducing their memory requirements. Second, when a topic has a high update rate and is replicated across a cluster, it can significantly improve throughput because the values need not be persisted to the cluster.

topicType

The topicType parameter is an enum and encompasses all Diffusion topic types. They include:

topicType values
TopicType
JSON
STRING
INT64
DOUBLE
BINARY

payloadConverters

payloadConverters contains an array of payload converter details. One or more converters can be specified as required.

Parameters for each payload converter detail
Parameter Name	Type	Description	Mandatory
name	String	Name of the payload converter. This information will be documented in the application user guide. See here for framework-issued converter names.	yes
parameters	Map of String to primitive Object	Parameters required by the payload converter. These are specific to the payload converter used and will be documented in the application user guide for the respective converter. See here for configuration details of framework-issued converters.	no

Payload converter and TopicType should match each other. See here for details.

Example 1. Sample framework configuration containing payloadConverters config

        "framework": {
          "payloadConverters": [
            {
              "name": "$Object_to_String"
            },
            {
              "name": "StringPrefixConverter",
              "parameters": {
                "prefix": "testPrefix"
              }
            }
          ],
          "topicProperties": {
          "topicType": "STRING"
          }
        }

framework Configuration for Polling Source Services

In addition to the values for all source services, the following framework specific configuration items can be provided for polling source services:

Key Description Default

Key	Description	Default
`pollIntervalMs`	The time interval in milliseconds between a polling source handler returning from a previous poll, and the next poll.	`30000` (30 seconds)
`pollTimeoutMs`	The time given for a polling source handler to return from a poll request, before it is deemed to have timed out.	`300000` (5 minutes)

pollIntervalMs

The time interval in milliseconds between a polling source handler returning from a previous poll, and the next poll.

30000
(30 seconds)

pollTimeoutMs

The time given for a polling source handler to return from a poll request, before it is deemed to have timed out.

300000
(5 minutes)

framework Configuration for Sink Services

Key Description Default

Key	Description	Default
`diffusionTopicSelector`	Specifies the selection of topics to subscribe to, in the form of a Diffusion topic selector. Refer to the the Diffusion manual for detailed information on the topic selector. Whether it needs to be supplied depends on the supported sink service type. Refer to the application’s user guide to check if passing `diffusionTopicSelector` is not required.	N\A
`updateTimeoutMs`	The time period (in milliseconds) to wait for an update to be successfully processed by a sink service handler, before sending another update from a Diffusion topic. If the sink fails to indicate that it has processed the update within this time, the service will be paused.	`5000` (5 seconds)
`payloadConverters`	The list of payloadConverter details. One or more converters can be set in a chain, which will be executed in the order of converters in the array. See above for detailed configuration parameters. If it is not specified, it is inferred from the payload converters defined for the service type or a default converter will be used. See payload converter overview for details.	N/A

diffusionTopicSelector

Specifies the selection of topics to subscribe to, in the form of a Diffusion topic selector.

Refer to the the Diffusion manual for detailed information on the topic selector.

Whether it needs to be supplied depends on the supported sink service type. Refer to the application’s user guide to check if passing diffusionTopicSelector is not required.

N\A

updateTimeoutMs

The time period (in milliseconds) to wait for an update to be successfully processed by a sink service handler, before sending another update from a Diffusion topic.
If the sink fails to indicate that it has processed the update within this time, the service will be paused.

5000
(5 seconds)

payloadConverters

The list of payloadConverter details. One or more converters can be set in a chain, which will be executed in the order of converters in the array. See above for detailed configuration parameters.

If it is not specified, it is inferred from the payload converters defined for the service type or a default converter will be used. See payload converter overview for details.

N/A

When configuring payloadConverters, ensure that the value produced by the final converter in the list is compatible with the value expected by the sink service type. This information can be found in the documentation of service types supported by the application.

framework Configuration for Hybrid Services

The application users can supply both; streaming source specific framework configuration and sink specific framework configuration for a hybrid service.
See source framework configuration and sink framework configuration for more details. A difference in the framework configuration between the sink service and the hybrid service is that it is not allowed to specify payloadConverters in the sink-specific configuration for the hybrid service. The payloadConverters should be defined in the source configuration block if required.

Example: Hybrid framework configuration

"framework": {
    "sink": {
        "diffusionTopicSelector": "?rate//"
    },
    "source": {
        "payloadConverters": [
          {
            "name": "$Object_to_JSON"
          }
        ],
        "topicProperties": {
            "timeSeries": true,
            "timeSeriesRetainedRange": "limit 20",
            "persistencePolicy": "SERVICE"
        }
    }
}

Example: `services` section

"services": [
    {
        "serviceName": "simpleSource",
        "serviceType": "SIMPLE_POLLING_SOURCE",
        "description": "A simple source"
    },
    {
        "serviceName": "simpleSink",
        "serviceType": "SIMPLE_SINK",
        "description": "A simple sink",
        "config": {
            "framework": {
                "diffusionTopicSelector": "?data//"
            }
        }
    },
    {
        "serviceName": "publishPollingSource",
        "serviceType": "PUBLISH_POLLING_SOURCE",
        "description": "Publishes score as a Json data",
        "config": {
            "framework": {
                "pollIntervalMs": 500,
                "pollTimeoutMs": 4000,
                "topicProperties": {
                    "timeSeries": true,
                    "persistencePolicy": "FULL"
                }
            },
            "application": {
                "topic": "polling/publish/game",
                "batchSize": 123
            }
        }
    },
    {
        "serviceName": "simplePatchPollingSource",
        "serviceType": "PATCH_POLLING_SOURCE",
        "description": "Sends patch updates",
        "config": {
            "framework": {
                "pollIntervalMs": 2000,
                "pollTimeoutMs": 4000,
                "topicProperties": {
                    "persistencePolicy": "FULL"
                }
            },
            "application": {
                "jsonTopic": "polling/patch/counter"
            }
        }
    },
    {
        "serviceName": "aStreamingSource",
        "serviceType": "A_STREAMING_SOURCE",
        "description": "Publishes from streaming source defined in `ConnectionConfig`",
        "config": {
            "sharedConfigName": "connectionConfig",
            "application": {
                "topic": "streaming/game",
                "batchSize": 123
            }
        }
    },
    {
        "serviceName": "aSink",
        "serviceType": "SINK_SERVICE",
        "description": "Subscribes to diffusion topic",
        "config": {
            "framework": {
                "diffusionTopicSelector": "?streaming//"
            }
        }
    },
    {
        "serviceName": "hybridService",
        "serviceType": "HYBRID_SERVICE",
        "description": "Subscribes to diffusion topic and publishes to another Diffusion topic",
        "config": {
            "framework": {
                "sink": {
                    "diffusionTopicSelector": "?rate//"
                },
                "source": {
                    "topicProperties": {
                        "timeSeries": true,
                        "timeSeriesRetainedRange": "limit 20",
                        "persistencePolicy": "SERVICE"
                    }
                }
            },
            "application": {
                "buffer": 10,
                "publicationPath": "newRate"
            }
        }
    }
]

services

Service config

Framework configuration for Source Services

topicProperties

timeSeries

timeSeriesRetainedRange

timeSeriesSubscriptionRange

publishValuesOnly

dontRetainValue

topicType

payloadConverters

framework Configuration for Polling Source Services

framework Configuration for Sink Services

framework Configuration for Hybrid Services

Example: Hybrid framework configuration

Example: services section

`services`

Service `config`

Example: `services` section