Diffusion 6.1 Release Notes

6.1.5 (15 Jan 2019)

Changes made to existing features at release 6.1.5

.NET Client: Improved memory profile during connection loss (19950)

Previously a connection loss under high load could cause a significant increase in memory usage due to the exceptions thrown by asynchronous methods. The behavior has now been changed so that a GC has an easier time cleaning up memory during a connection loss so that the increase in memory usage won't be quite as severe.

Web Server/HTTP: Extending the maximum HTTP header size (19924)

The maximum size of an HTTP request header has been increased from 4K to 16K, preventing exceptions stating "HTTP header is too long at line ..."

Bugs cleared at release 6.1.5

.NET Client: Connection establishment can be stuck with high CPU (19833)

A WebSocket connection or reconnection could get stuck in an infinite loop during the handshake phase. This has now been fixed.

.NET Client: Automatic reconnection fails to free resources (19841)

Automatic reconnection of a client session did not close attempt threads properly, leaving them open and stuck in a waiting state. This has now been fixed.

.NET Client: Closing a session during reconnection does not prevent further reconnection attempts (19896)

Closing a session during reconnection now correctly prevents any further reconnection attempts.

.NET Client: The request of a cancelled task is still being sent to the server (19970)

A asynchronous request to the Diffusion server (such as AddTopicAsync for example) that got invoked with a canceled CancellationToken would send the request to the server before completing the returned Task in a canceled state. This behavior has been changed now. Any asynchronous request that gets invoked with a canceled CancellationToken will immediately complete in a canceled state and NOT send the request to the Diffusion server.

Demos: Drawing board demo allows drawing in multiple rooms at the same time (19808)

After switching rooms in the drawing board demo, drawn strokes would be added to all previously visited rooms. This is now fixed.

Server: Diffusion script contains bad settings=profile for JFR configuration (19738)

The default settings=profile configuration for the Java Flight Recorder is no longer recommended, since it truncates the JFRs and generates data for only a few minutes prior to shutdown. The configuration has been removed.

Web Server/HTTP: The web server does not cache large resources correctly (19942)

In previous releases the web server did not cache large resources correctly (files larger than 1 MiB). This affected the Diffusion console, which would always be downloaded from the server rather than being served from browser cache. The bug has been fixed in this release.

6.1.4 (22 Nov 2018)

Changes made to existing features at release 6.1.4

Android Client: Android library is no longer signed (19607)

The diffusion-android.jar is no longer signed. Additionally, the javax.annotation classes have been removed from the jar. These changes reduce the size of the library and make it easier to package.

Java & Android Client: Proxy negotiation now reuses a single socket for subsequent requests (19397)

Previously, a new socket was used for each step of a proxy authentication handshake. This has been changed so that the same socket is used for subsequent requests, allowing for the implementation of multi-step authentication handshakes such as NTLM2.

Bugs cleared at release 6.1.4

.NET Client: ISession.Attributes always returns null (19712)

The ISession.Attributes property would always return null. This has now been fixed to return a ISessionAttributes instance.

Android Client: Correct packaging of stream support to work on API 24+ (19567)

The Diffusion Android client library includes the backports of JDK 8 classes for compatibility with versions of Android earlier than API 24. This release fixes a bug in the packaging of the backported classes which caused an incompatibility with Android versions API 24 and above.

Console: Line and area graphs not generated from numeric topics (18880)

The console was unable to draw line or area graphs of data from numeric topics. This issue has been resolved.

Fan-out: Fan-out secondary server disconnected : UnsupportedOperationException: Key 'TIDY_ON_UNSUBSCRIBE not supported at primary (19657)

An UnsupportedOperationException could be reported in a primary server and cause clients to be disconnected. This could occur when a fan-out secondary server connected to a primary server that had topics with the TIDY_ON_UNSUBSCRIBE property set. It could also happen when using the getTopicDetails operation against such topics. This problem has now been resolved.

Installation: Command-line installer silently uses existing license file (18793)

If the command-line installer detected a Diffusion license file (licence.lic) in the current working directory, it silently used that instead of any pre-packaged license. In this situation, the installer now prompts you to select which license to use.

Installation: Diffusion installers do not verify selected .jar file (18821)

The Diffusion GUI and command-line installers would accept a .jar file that was not a Diffusion .jar, resulting in a misleading "Install Exception" error message. The installers now check to see if the .jar file selected is valid, using attributes in the manifest, and display a more informative error message.

Java & Android Client: DeltaType.readDelta() fails if offset != 0 (19405)

A bug introduced in 6.1.0 caused DeltaType.readDelta() to fail to parse data if the offset was non-zero. This is now fixed.

JavaScript Client: `details` property in `subscribe` event lacks a type and a description (19369)

Filled a gap in the documentation of the Subscription class, subscribe event details.

JavaScript Client: CONFLATION topic property missing (19562)

The CONFLATION topic property was missing from the JavaScript client API. It has now been added.

JavaScript Client: RequestContext field 'sessionId' is written with capital 'S' and maps wrongly (19666)

Fixed TypeScript definition of RequestContext.sessionId

JavaScript Client: RangeQuery.fromSequence is not a function (19668)

Using RangeQuery.fromSequence resulted in a runtime error: "TypeError: session.timeseries.rangeQuery(...).fromSequence is not a function". This has been resolved by fixing the TypeScript definition to change the interface function to "from".

Server: Incorrect handling of WebSocket fragmented frames leading to "Unrecognised message" errors (19513)

The server handled fragmented WebSocket frames incorrectly. This affected requests sent from JavaScript clients to the server, leading to "Unrecognised message" exceptions. This could cause the Diffusion monitoring console to become unreliable when the total number of topics approached 1000. The issue is now fixed.

Server: Correct handling of MESSAGE_QUEUE_LIMIT_REACHED and MESSAGE_LOSS fatal conditions (19546)

Sessions encountering a full message queue or message loss on reconnection did not report these conditions correctly to session properties listeners. Further, rather than being closed on these fatal conditions, the sessions were incorrectly kept alive pending reconnection. The reconnection would always fail. This has been fixed. Sessions fail cleanly, and session properties listeners will receive MESSAGE_QUEUE_LIMIT_REACHED or MESSAGE_LOSS as appropriate.

6.1.3 (26 Sep 2018)

Bugs cleared at release 6.1.3

Java Client: gettingstarted/PublishingClient.java example uses incorrect class (19322)

The PublishingClient.java now uses the Long.class instead of Integer.class for the topic type of INT64.

JavaScript Client: JavaScript API docs missing RecordContent.addRecord() function description (18979)

The missing description has now been added to the Javascript API documentation.

Server: Conflation processing fails to terminate (19364)

Due to a bug introduced in 6.1.0, internal data structures could be corrupted. Consequently, conflation processing could loop for ever (the server would become unresponsive and the multiplexer threads would consume CPU). Other errors such as NullPointerExceptions may also be observed in the server log. The bug has been resolved in this release.

6.1.2 (5 Sep 2018)

Changes made to existing features at release 6.1.2

.NET Client: Microsoft.CSharp dependency no longer needed (19267)

A dependency on Microsoft.CSharp is no longer needed when using the .NET client library. It can be safely removed from your dependencies if you don't use any of its features directly in your project.

Bugs cleared at release 6.1.2

.NET Client: Establishing a secure connection with an invalid server certificate does not fail as expected (16018)

SSL certificate validation was not taking place correctly. This behavior has now been corrected. When trying to connect with an invalid server certificate, the connection now fails. When testing or using a self-signed certificate, you may wish to override the default behavior by supplying your own validation code. For example: ISessionFactory.SslContext((sender, cert, chain, errors) => {\* your validation here *\});

.NET Client: Escaped backslash at the end of a JSON string causes InvalidDataException (19247)

An escaped backslash at the end of a JSON string ( "\\" threw an InvalidDataException. This behavior has now been fixed.

C Client: diffusion_value_dup does not duplicate the value correctly (19199)

Applications would crash when duplicating a DIFFUSION_VALUE_T with function diffusion_value_dup, if the underlying value was of either binary, int64, double or recordV2 datatype. This is now fixed. The fix also addresses an issue where the datatype of a duplicated value is not set.

Configuration: Logs.xml <file-pattern> property ignores %u value (19169)

Diffusion does not support saving logs in multiple files at the same time. The Java logging framework %u option is intended to resolve filename conflicts by appending a unique number to each filename. Since there is only ever one Diffusion log file, it will always append 0. File-patterns using the %u argument still work and will append 0 to the filename, but since the option is useless, it will no longer be documented.

JavaScript Client: Incorrect diffusion.version and diffusion.build constants on node.js (19163)

The diffusion.version and diffusion.build constants were incorrect in the npm packaged version of the JavaScript client. This bug has been corrected.

JavaScript Client: PUSH-000473 messages showing when the console is closed (19174)

When a session properties listener is closed, session property events can still be deliver for a short while. The JavaScript client used to handle this as an error logging "failed: 0: No such conversation" on the server. The JavaScript client no longer handles this as an error.

JavaScript Client: datatypes.string().readValue silently reads non-string CBOR (19176)

DataType classes Int64, String and Double would read values that were not CBOR strings without returning an error. This is fixed, and they now throw an error if deserializing CBOR of an incorrect type, as documented.

Replication: Session replication memory leak leading to reduced performance (19214)

The Hazelcast datagrid used for session replication was incorrectly retaining details of sessions even after they were closed. This caused Diffusion clusters with session replication enabled to leak memory, leading to reduced performance due to garbage collector pauses unless all members of the cluster were rebooted at once. The issue is now fixed.

Server: PUSH-000760 topic removal message logged at INFO level (18982)

Previously, the PUSH-000760 message indicating a topic was removed using the automatic topic removal feature was logged at INFO level. This could generate a large volume of messages in the server log, but the message does not indicate a problem that requires action. The message is now logged at DEBUG level.

6.1.1 (18 Jul 2018)

Bugs cleared at release 6.1.1

.NET Client: HTTPProxy connection failure (18924)

A bug prevented the .NET client establishing a session using an HTTP proxy. This issue has now been resolved.

.NET Client: Session timeout during connection (19329)

Fragmented handshake response packets have not been handled correctly during the connection of a newly created session. Sessions would simply time out during connection. This behavior has now been corrected.

Apple Client: Interfaces set to be removed not marked as deprecated (18919)

The following interfaces are set to be removed in a future version, and should have been deprecated in 6.0. recordContent property on PTDiffusionContent, recordContentWithSchema:error: method on PTDiffusionContent, PTDiffusionTopicAddFailReason, PTDiffusionTopicAddFailReasonErrorKey

C Client: WebSocket session open cannot be called concurrently (17785)

WebSocket sessions could not establish a connection concurrently. This has now been addressed.

C Client: Incorrect encoding of the WS URI (17847)

Addresses an issue where a valid Websocket URI could be incorrectly encoded.

C Client: New value streams fail to invoke on_subscription and on_value if added after a matching topic has been subscribed to (18816)

If a value stream was added after the session had subscribed to a matching topic, it failed to invoke the on_subscription and on_value callbacks. This has now been addressed. Note that on_value is only invoked if a value is available.

C Client: Memory leaks (18834)

Several minor memory leaks in the client API are now fixed.

C Client: Failure to notify STREAM_CHANGED on addition of a value stream which selects the topic (18840)

Addresses an issue where the on_unsubscription callback was not fired for fallback value streams upon the addition of a stream which selects the topic. The callback is now fired correctly, with an unsubscription reason of STREAM_CHANGED.

C Client: Incompatible datatype value causes failure with no error callback (18853)

The API silently failed when a value stream's datatype and topic datatype were incompatible and could not be converted. The stream's on_error callback is now called with an INVALID_DATA error code.

C Client: Missing documentation for value stream and add topic callback struct (18860)

Documentation for the value stream and add topic callback struct was missing. This has now been corrected.

C Client: Fallback streams not notified of topic unsubscription (18939)

Fallback streams did not have their on_unsubscription callback fired when a topic had been unsubscribed. This is now fixed.

Console: Console dialog boxes not graying out some background items (18792)

Modal dialog boxes in the Diffusion web console were graying out some, but not all elements. This issue has been resolved.

Demos: Drawing board demo applying autocapitalise and autocorrect to principal input field on mobile (19051)

When entering a principal to use for the drawing board demo on mobile devices, autocorrect capitalised the input. Autocapitalise and autocorrect is now disabled for principal input field.

Environment: Environment validation script reports wrong memory and kernel version values (18857)

The environment validation script reported wrong values for physical memory and kernel version checks. The correct values are now reported, and error messages have been improved to make the recommended values clearer.

Environment: Environment validation script displays only one value for multi-value checks (18888)

Only a single value was displayed in [expected]/[result] for checks which test multiple values. This has now been fixed.

Environment: Environment validation script wrongly records all single value checks as failed (18933)

For single value checks, the script always recorded status as [AMBER]/[RED] in the validation.properties file, even if the check passed. This is now fixed. Passed checks are registered as [GREEN].

JavaScript Client: JavaScript clients using exclusive updaters sometimes fail to calculate deltas (18847)

Previously, JavaScript clients using exclusive updaters to update topics could sometimes fail to calculate a delta between two values. The defect has been corrected.

JavaScript Client: Error for missing message handler is incorrect (18922)

When a JavaScript client received messaging for which it had not established a stream, the server log recorded a misleading ErrorReasonException: "First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object". The misleading log message has been replaced with a more informative message stating that there is no registered message handler for the path.

JavaScript Client: Errors decoding CBOR in Time Series Topic updates (19015)

Compressed topic values and updates could be processed out of order by the JavaScript client. This was especially visible when subscribing to stateful topics. This issue is now resolved. However, only the Publisher API is able to transmit compressed messages in Diffusion v6.1.

Replication: Compaction failure in backup partition log (18899)

An issue compacting the replicated topic log could cause the server to shut down. The compaction failure was caused by the compaction of a log with a delta updated followed by a topic mapping. The mapping was causing the last value to be lost before attempting to apply the delta. In most cases compaction happens either beginning from topic mappings or without them present. The issue is now fixed by preserving the last value when the topic mapping is applied.

Replication: Partitions left unclaimed during startup in a server cluster due to race condition (18900)

It was possible for some partitions to remain unclaimed when starting a Diffusion server in a cluster. This was caused by a race condition setting up a listener for partition migrations, and would cause some operations to fail with the CLUSTER_REPARTITION error reason. This problem has now been resolved.

Topics: Split-path and full-path pattern topic selectors with escaped characters could fail (18913)

A bug introduced in 6.1.0 caused incorrect evaluation of some topic selectors using split-path or full-path patterns containing the escape character '\'. Sessions using such selectors could fail to be subscribed to matching topics. The bug has been fixed in this release.

Topics: Time series topics memory leak with persistence and replication (19057)

When using time series topics with persistence or replication a memory leak could occur. This has now been fixed.

6.1.0 (26 Apr 2018)

Features introduced at release 6.1.0

.NET Client: Support for Time Series topics (16848)

The Diffusion .NET Client Library now supports time series topics. This feature can be accessed from a session via `ISession.TimeSeries`.

.NET Client: IClientControl.SetSessionPropertiesAsync (16901)

Added IClientControl.SetSessionPropertiesAsync methods. This provides a control client (with sufficient privilege) with the ability to modify other clients' session properties. For more information please see IClientControl.SetSessionProperties API documentation.

.NET Client: Request/Response messaging (16984)

Introduces a "request-response" addition to the Messaging API. Requests in the form of typed messages can be dispatched to a specific session, to a number of sessions matching a specific filter, or to a message path (to which a server registered handler on that path will receive the request). This functionality is in line with the standard Messaging API.

.NET Client: Topic Notifications (17434)

Introduces the Topic Notification feature. Users may select topic paths, for which notifications of the presence, addition or removal of matching topics will be received. This allows users to build a view of the topic model held within the server.

.NET Client: .NET Standard 2.0 Compatibility (18195)

The Diffusion .NET Client Library is now fully compatible with the .NET Standard 2.0. Officially supported platforms are Windows (via .NET Framework, .NET Core), MacOS (.NET Core), and various Linux distributions (via .NET Core). In order to make this change clear we removed the old `PushTechnology.ClientInterface` library and NuGet package and replaced it with `Diffusion.Client`. This requires re-compilation of your project.

Apple Client: Subscription Control (13171)

Adds PTDiffusionSubscriptionControlFeature, allowing clients using this library to subscribe or unsubscribe other sessions to topics, as well as providing them with a mechanism for handling requests to subscribe to routing topics.

Apple Client: Topic Notification Feature (16763)

Adds support for the Topic Notification feature. Users may select topic paths, for which notifications of the presence, addition or removal of matching topics will be received. This allows users to build a view of the topic model held within the server.

Apple Client: Messaging Control - Send Request to Filter (17074)

Adds the capability to send requests to more than one session using a filter. Adds sendRequest:toFilter:path:responseStream:completionHandler: to PTDiffusionMessagingControlFeature.

C Client: Creating Topics from Specifications (16181)

Topics can now be added with a new function named: add_topic_with_specification This function requires a session (SESSION_T *), topic name (char *) and a topic specification (TOPIC_SPECIFICATION_T *)

C Client: Support for String, Int64, Double and RecordV2 topics (16396)

The C client API now supports String, Int64, Double and RecordV2 topic types. These topic types can now be added, subscribed to and updated within the API.

C Client: Value streams (18148)

Value streams are now available in the C client API. These can be used to receive notifications on subscription, topic updates and unsubscription to topics of a particular datatype. Currently supported datatypes are: Binary, JSON, Recordv2, String, Int64 and Double. Relevant documentation regarding this is located in streams.h and value-stream.h

Java & Android Client: Session locks (18216)

A new API for per-session locks has been added to the Java and Android clients. See com.pushtechnology.diffusion.client.session.Session.lock(). The locking facility will be extended to other clients in a future release.

Monitoring: Log message statistics (18129)

New statistics have been added which count messages logged to the server log, partitioned by the message ID. The statistics are available through JMX and the Prometheus integration. The statistics provide counts of significant events that are useful for monitoring and diagnostic purposes.

Replication: Split-brain mitigation - Quorums (12202)

Support for quorums has been introduced. It is possible to configure a minimum quorum size. A Diffusion server will not finish starting up until it can join a cluster of quorum size. A Diffusion server will shutdown automatically if the cluster it is connected to falls below quorum size.

Server: Automatic Conflation (15958)

A new conflation mechanism has been introduced for certain topic types. The topic types it supports are: JSON, binary, string, int64, double, record V2, time series. Diffusion understands how to conflate messages for these topics without the matchers and mergers required by the existing conflation mechanism. Conflation can be configured on a per topic basis using policies specified with the CONFLATION topic property. The values for this property are: 1) "off" - disables conflation for the topic 2) "conflate" - automatically conflates topic updates when back pressure is detected by the server 3) "unsubscribe" - automatically unsubscribes the topic when back pressure is detected by the server 4) "always" - automatically conflates topic updates as they are queued for the session. When not specified the "conflate" policy is used by default. The queue-definition element in the Server.xml file has been changed so that conflation is enabled when not specified. In Server.xml conflation is explicitly enabled for the "DefaultQueue" queue definition. The existing conflation mechanism is still present but has been deprecated. It only applies to the deprecated topic types: stateless, single value and record; or slave and routing topics that reference them.

Server: Improved outbound queue management (17525)

There are new ways of placing limits on the outbound queues of the server. It is now possible to configure the maximum number of bytes that can be queued per session. It is also possible to set the maximum number of bytes that can be queued for all sessions using the same connector. This allows the outbound queues to be limited in terms of how much memory can be dedicated to them. No byte limits are applied if not configured. It is possible to configure both the per session maximum queued bytes and maximum queued messages limits. These limits are configured using the XML configuration system. The per session limit is configured in the queue-definition element in the Server.xml file. The per connector limit is configured in the connector element of the Connectors.xml file. The Server.xml included with Diffusion 6.1 sets the per connector limit to 1 GiB. Since the outbound queue now has two ways of configuring a per session limit, the maximum queued messages limit has been made optional. If not configured no limit will be applied.

Topics: Topic Ownership (OWNER Topic Property) (16957)

A new OWNER topic property has been introduced that allows the creator of a topic to extend READ_TOPIC, MODIFY_TOPIC and UPDATE_TOPIC permissions to a specific principal, in addition to the permissions granted by the authorisation rules in the security store.

Topics: PERSISTENT Topic Property (18070)

A new topic property has been introduced which can be used to prevent certain topics being persisted when topic persistence is in use at the server. By default all persistable topics are persisted when persistence is enabled at the server, but if the property PERSISTENT is set to 'false' when the topic is created then the topic will not be persisted.

Topics: Automatic Topic Removals (REMOVAL Topic Property) (18071)

There is now a new REMOVAL topic property which can be used to indicate that a topic is to be automatically removed according to some criteria. This greatly simplifies the management of the lifetime of a topic. The criteria are specified in terms on an expression which can include conditions that can specify removal after an absolute time, when the topic has less than a number of subscribers, when the topic has not been updated for some time or when there are no sessions with certain session property settings. See the documentation for the full specification and details of how to use.

Changes made to existing features at release 6.1.0

.NET Client: IServiceRequest deprecated (14577)

PushTechnology.ClientInterface.Client.Features.Control.Topics.IServiceRequest has been deprecated and will be removed in a future release.

.NET Client: ISession.GetXXXFeature() methods removed (17124)

The previously deprecated ISession.GetXXXFeature() methods have been removed. Use the equivalent ISession.XXX property instead. For instance: Instead of ISession.GetClientControlFeature() use ISession.ClientControl.

.NET Client: SessionId class removed (18309)

The SessionId class deprecated in Diffusion 6.0 has been removed. ISessionId is now used everywhere inside the codebase.

.NET Client: ISessionFactory.SslContext(RemoteCertificateValidationCallback, LocalCertificateSelectionCallback) Deprecated (18517)

ISessionFactory.SslContext(RemoteCertificateValidationCallback, LocalCertificateSelectionCallback) has been deprecated. Use the ISessionFactory.SslContext(RemoteCertificateValidationCallback) method instead.

.NET Client: IFetchRequest, IFetchResponder and IFetchResponseRegistration Deprecated (18529)

The IFetchRequest, IFetchResponder and IFetchResponseRegistration unused interfaces have been deprecated.

.NET Client: IStatelessSubscriptionRequest and IStatelessSubscriptionRequestHandler removed (18530)

The previously deprecated IStatelessSubscriptionRequest and IStatelessSubscriptionRequestHandler have now been removed.

.NET Client: ISubscriptionRequest removed (18531)

The previously deprecated ISubscriptionRequest interface has been removed.

.NET Client: IClientSession removed (18532)

The previously deprecated IClientSession interface has been removed.

.NET Client: IClientControlFeatureHandler removed (18533)

The previously deprecated IClientControlFeatureHandler interface has been removed.

.NET Client: IQueueListener removed (18534)

The previously deprecated IQueueListener interface has been removed.

.NET Client: IStateListener removed (18535)

The previously deprecated IStateListener interface has been removed.

.NET Client: IAuthControlFeatureHandler removed (18536)

The previously deprecated IAuthControlFeatureHandler interface has been removed.

.NET Client: IClientEndpoint Deprecated (18537)

IClientEndpoint has been deprecated. It will be removed in a future release.

.NET Client: ITopicManagementHandler & ITopicManagementFactory removed (18538)

The previously deprecated ITopicManagementHandler and ITopicManagementFactory interfaces have now been removed.

.NET Client: IClientInfo removed (18539)

The previously deprecated IClientInfo interface has now been removed.

.NET Client: DefaultStreamCallback and StreamDefault removed (18540)

The previously deprecated DefaultStreamCallback and StreamDefault classes have now been removed. Use the DefaultStream implementation instead.

.NET Client: IPingDetails.RoundTripTime property removed (18541)

The previously deprecated IPingDetails.RoundTripTime property has now been removed. Use the IPingDetails.RoundTripTimeSpan property instead.

Android Client: SessionFactory.localSocketAddress() no longer fails if bind is unsupported by runtime Android API level (18065)

SessionFactory.localSocketAddress() only works with Android API level 24 ("Nougat") or higher. Previously, if an application used this session factory setting and was run against an earlier API level, an exception would be thrown when attempting to open a connection. From this release, an informational message will be logged instead.

Apple Client: getPathPrefix() equivalent interface added to topic selector (14210)

Adds a pathPrefix property to the PTDiffusionTopicSelector interface.

Apple Client: Default delegate implementations (18018)

Adds default implementations for some protocols, to be used as a base for prototyping as well as when logged output for some methods is adequate. Default implementations of the following protocols have been added: - Session response stream delegate. - Stream delegate. - Subscriber stream delegate. - Topic notification listener. - Topic tree registration delegate.

Apple Client: AddTopicResult on completion of call to add topic by specification (18319)

Adds PTDiffusionAddTopicResult available by calling new add topic methods in the topic control feature.

Apple Client: New reason for topic addition failure when licensed topic limit reached (18322)

Adds exceededLicenseLimit to PTDiffusionAddTopicFailReason.

Apple Client: String escape function (18579)

Adds the PTDiffusionEscape function, escaping special characters in a string that is to be used within a topic property or a session filter.

C Client: Cascade URL for reconnection removed (17855)

Cascade URLs for reconnection strategies have been removed from the C client API. Applications will no longer be able to supply a list of server URLs to connect to in the scenario that a C client session loses its connection a server. Reconnection is now targeted only at the server to which the connection was lost.

C Client: New LIST_T functionality (18149)

Two new functions have been added to the LIST_T API: list_size - which will return the number of elements currently stored in a given list. list_of_lists_free - which will free a list which has list elements.

Client: Added 'UNKNOWN' to some API enums to allow for future extension (17686)

To improve interoperability between this version of the client and new behavior added in later versions of the server, new "unknown" values have been added to several enumerations. There is now an UNKNOWN_TOPIC_TYPE topic type, UNKNOWN_GLOBAL_PERMISSION and UNKNOWN_TOPIC_PERMISSION permission types, and an UNKNOWN_UNSUBSCRIBE_REASON unsubscription reason.

Client: removeTopicsWithSession Deprecated (18072)

The removeTopicsWithSession function on the TopicControl feature has been deprecated. The new REMOVAL topic property should be used in preference.

Client: TopicEventListener (Subscriber Notifications) Deprecated (18073)

The TopicEventListener (subscriber notifications) methods in the TopicControl feature have been deprecated as they only worked for local sessions. The use case for these methods was typically removal of unused topics which is now better addressed using the REMOVAL topic property (which is cluster aware). These methods will be removed in a future release.

Client: Content deprecations (18454)

The 'Content' interface is being phased out. However it is still used in some interfaces, such as Fetch and Messaging when receiving data. In all other cases it's use should be avoided as it will be removed at a future release. At this release some remaining interfaces that used Content (such as builders and readers) have been deprecated.

Configuration: Default programmatic server configuration defaults (8654)

At previous releases, if a server was created in a programmatic environment (without XML configuration files) then it was necessary to set up certain configuration items in order to establish a working server that could accept normal web clients. All configuration items now have sensible defaults (see config javadoc for details) enabling a working server to be started without any specific configuration.

Configuration: The <server-statistics/> elements in Statistics.xml and the ServerStatisticsConfig interface have been removed (18033)

The <server-statistics/> elements in the Statistics.xml and the ServerStatisticsConfig interface have been removed. These configuration settings are not used by the product and were deprecated in 6.0.

Configuration: The Replication customConfigurator element is deprecated (18107)

In 6.0 a customConfigurator element was added to the Replication.xml configuration. This was intended to enable Kubernetes support but was never documented. A boolean element 'kubernetes-enabled' was also added which allows simple configuration of Kubernetes. The customConfigurator element is deprecated from this release, together with the corresponding methods in ReplicatonConfig. The element will be removed in a future version.

Configuration: Diffusion.bat now follows the updated format implemented in diffusion.sh (18136)

diffusion.bat now allows for enabling/disabling a specific configuration by uncommenting/commenting the line with it. Default configuration is suitable for local development. Changing it should be justified on a case by case basis, and should be tested with an expected workload before being applied to the production environment.

Console: Drag (and drop) support (18475)

Console users can drag both the topic name and the topic value from the Topics tab to any application that handles text/plain content.

Demos: Update samples to use the latest version of dar-maven-plugin (18407)

The samples have been updated to use version 1.3 of the dar-maven-plugin. This version of the plugin fixes a known issue (https://github.com/pushtechnology/mvndar/issues/3) when running on Windows.

Fan-out: Fan-out connections no longer appear in session property queries and session property listener notifications (18725)

Previously fan-out connections would appear in session property queries and would also be notified to session property listeners as if they were client connections. This is no longer the case.

Java & Android Client: New escape function (18458)

In various parts of the client API DSL (Domain Specific Language) is used. Examples are session property filters, security scripts and topic properties. When quoted values are used within these then it is possible to escape certain characters (single quotes, double quotes and backslash) by preceding with a backslash. A new method called Diffusion.escape has been provided to simplify this.

JavaScript Client: Session.subscribe returns a Result instead of a Stream (15898)

New method Session.select is preferred over deprecated Session.subscribe. Session.select returns a Result object to indicate whether a client was successfully subscribed. New methods Session.addStream and Session.addFallbackStream are preferred over deprecated Session.stream. In previous release, Session.stream returned a stream which emits raw buffer objects. Both #addStream and #addFallbackStream will return a value stream and a fallback value stream of a provided data type respectively.

JavaScript Client: ClientControl: setSessionProperties (16884)

Added Session.clients#setSessionProperties and Session.clients#setSessionPropertiesByFilter. This provides a control client with sufficient privilege with the ability to modify other clients' session properties. For more information please see Session.clients#setSessionProperties and Session.clients#setSessionPropertiesByFilter API documentation.

JavaScript Client: Zlib module separated from client bundle (18085)

Zlib code for message decompression has been externalised out to 'zlib-bundle.js'. Clients will log out a warning at startup if the 'zlib-bundle.js' is not included. The client's initial connection request will set the per message compression capability depends whether the 'zlib-bundle.js' is include. This will indicate to the server whether messages should be compressed before they are sent to the client. Previous JavaScript client libraries bundled the Zlib library code in the core client code increased the client bundle size substantially even after minified. The 'zlib-bundle.js' can be included in the browser for clients that want to make use of the client compression capability.

JavaScript Client: New escape function (18580)

In various parts of the client API DSL (Domain Specific Language) is used. Examples are session property filters, security scripts and topic properties. When quoted values are used within these then it is possible to escape certain characters (single quotes, double quotes and backslash) by preceding with a backslash. A new method called Diffusion.escape has been provided to simplify this.

JavaScript Client: TypeScript definitions file consolidated into common source (18739)

TypeScript definitions have been consolidated into a single definition. In the past different TypeScript definitions have been provided for the web and node JavaScript clients. The definitions included in the "diffusion" module published to NPM incorrectly exported the definitions so that the correct usage of the JavaScript library would generate compilation errors. This has been fixed to make TypeScript usable with the node client without having the include the web definitions on top of those packaged in the module. Additionally the definitions also export a "diffusion" namespace that supports the use of the definitions without a module loader. This is suitable for implementing browser clients where the Diffusion JavaScript library is included with a script tag. The TypeScript definitions no longer declare a module. This means when using a module loader with the definitions file you may need to change how you identify the module to import. The path syntax should be used to import the module.

Logging: Use of standard jar manifest version fields to improve log4j stack traces (17419)

When Diffusion uses log4j2 as a logging framework, it will now provide information about the jar version and a changelist number in the stack trace. This will make identification of the Diffusion version possible from logs files.

Monitoring: Client statistics summaries are no longer written to a separate log file (7991)

If client statistics are enabled in Statistics.xml, a summary of the connected sessions is regularly logged. In previous releases, these reports were logged to a separate log file, specified by the log-file configuration property. From this release, the reports are logged to the server log. The log-file configuration property now has no purpose, and has been deprecated together with the corresponding configuration API property. If separate log files are required or the reports are not desired, use a third-party SLF4J logging back-end such as Log4j 2, and configure it appropriately to partition or filter the server log.

Monitoring: The server no longer maintains rates for statistic metrics (11792)

Previously, the server maintained exponentially-weighted moving averages for some metric values, calculated over 1 minute, 5 minute and 15 minute intervals. These were available through the JMX statistics API and (if specially configured) the Diffusion console. The derived rates were rarely used and have been removed to reduce the CPU and memory overhead of statistics.

Monitoring: Some rarely used statistics have been removed (18093)

To improve performance, statistics that were rarely used have been removed. These statistics were only available through JMX and include the connection count per connection type; the number of publishers started and loaded; the publisher name; the per-publisher subscription count; the global count of subscriptions; the per-session connection type; and the per-topic counts of subscriptions added and removed.

Publisher API: DataMessage.nextObject and putObject deprecated. (13747)

The methods nextObject and putObject on the DataMessage interface have been deprecated as the same effect can easily be achieved by wrapping the input or output stream from the message in an ObjectInputStream or ObjectOutputStream respectively.

Publisher API: New method to access server-scoped statistics (18011)

A getServerStatistics() method has been added to the publisher API. This provides access to server-scoped message count and byte count statistics for inbound and outbound messages.

Publisher API: Access to publisher-scoped statistics is deprecated (18012)

To simplify the statistics model and reduce the cost of reporting, per-publisher statistics are being retired in favour of server-scoped statistics. The following methods are deprecated: Publisher.getStatistics(), Publisher.startStatistics(), Publisher.stopStatistics(), Publisher.isStatisticsEnabled(). These methods will be removed in a future release. Configuration of per-publisher instance statistics (topic-statistics/instance-monitoring in the Statistics.xml file) is also deprecated. This configuration and the per-publisher statistics it enables will be removed in a future release. Publisher implementations can access the server statistics through the new getServerStatistics() method.

Publisher API: TopicClient.send(TopicMessage) and Client.send(TopicMessage, MessagePriority) are deprecated (18087)

The TopicClient.send(TopicMessage) and Client.send(TopicMessage, MessagePriority) methods have been deprecated. These methods allow messages to be sent to individual sessions, inserting these messages into a topic stream. These are a hangover from the classic API, completely removed in Diffusion 6.0, which allowed an application direct access to these messages. With the unified API (the only client API from Diffusion 6.0), these methods only work reliably when used in conjunction with stateless topics and require the client application to expect and know how to decode the messages. Alternatives include publishing an update to a session topic to which only the target session is subscribed, or moving the publisher logic to a control client and using the MessagingControl client API feature.

Publisher API: The unused statistics reporter type FILE has been removed (18098)

The FILE value has been removed from the ReporterConfig.Type enumeration. The FILE value was never used. Previously attempting to configure a statistics reporter of type "FILE" would fail with an unhandled exception. From this release, the server will fail to start with a clear configuration error.

Publisher API: Deprecated use of MessagePriority (18369)

Message priority is only useful when used with stateless topics (those without data) for sending messages to clients. Even when used in this way, the messages are delivered to clients via the topic update stream and not via the messaging feature and there is the possibility that a high priority message could reach a client before the subscription notification, leading to errors. Message priority should never be used with topics that have topic data. The MessagePriority capability has been deprecated and will be removed in a future release.

Publisher API: TopicLoader Deprecated (18631)

Topic loaders can only be used with STATELESS topics which were deprecated at release 6.0. They have therefore been deprecated and will be removed in a future release.

Publisher API: Topic.setTidyOnUnsubscribe deprecated (18659)

The Topic.setTidyOnUnsubscribe method has been deprecated. This is only useful for setting this option after a topic has been created. In future only setting the value on creation of the topic with the corresponding property will be supported.

Bugs cleared at release 6.1.0

.NET Client: Incorrect encoding of the WS URI (17849)

Corrects encoding of host and query in WebSocket connection HTTP requests to conform with RFC 3986, allowing characters that need percent encoding to be used.

.NET Client: The ClientControl.setThrottled method incorrectly documents that it enables conflation (18614)

The IClientControl.setThrottled method incorrectly documented that it enabled conflation. It does not enable conflation when setting the throttler. The documentation has been fixed.

Apple Client: Incorrect encoding of the WS URI (17846)

Corrects encoding of host and query in WebSocket connection HTTP requests to conform with RFC 3986, allowing characters that need percent encoding to be used.

C Client: update_value does not notify callbacks of update success/failure (18542)

Addresses an issue where on_update_success or on_update_failure is not called after a successful or failed update using the update_value function.

Client: Session properties filter parsing does not allow for escaped characters (18457)

Session properties filter DSL parsing did not allow for the escaping of special characters \ " and ' - this has now been resolved.

Console: CPU chart y axis and labels incorrect (18140)

The Diffusion Console previously presented CPU Utilisation and Load Average on a single line chart. These metrics have now been separated in to two discrete line charts with the appropriate scales.

Console: Slave topics rendered incorrectly (18371)

Slave topics and routing topics were displayed incorrectly upon subscription. This issue has been resolved.

Console: Double topics incorrectly displayed as empty string (18439)

Double topics, with value zero incorrectly rendered as having no value. This issue has been resolved.

Demos: SportsbookPublisher: Deadlock on shutdown (18118)

In some conditions when a sportsbook demo was deployed to a server, dead-lock could occur when shutting down the server during its initialisation.

Installation: Java version warning only given when running as a GUI (16283)

Installer warned about wrong Java version only when running in GUI mode. When installer was running in headless (command line) mode it processed the installation even if an unsupported Java version was installed. Headless installer now communicates that unsupported version of Java is used and terminates installation.

Java & Android Client: The ClientControl.setThrottled method incorrectly documents that it enables conflation (14471)

The ClientControl.setThrottled method incorrectly documents that it enables conflation. It does not enable conflation when setting the throttler. The documentation has been fixed.

Java Client: Published diffusion-client and diffusion-jms-client JARs declare simple SLF4J binding as dependency (18101)

Th diffusion-client and diffusion-jms-client JARs published to our Maven repository declare the simple SLF4J binding as dependency. This may have prevented the use of your chosen SLF4J binding. The dependency has been replaced with one for the SLF4J API. This will allow you to select the logging framework to use. If you do not include a binding as a dependency no logging will be generated.

JavaScript Client: Incorrect encoding of session tokens preventing reconnection (17893)

JavaScript clients using the HTTP polling transport incorrectly encoded the session token when making reconnection requests. This sometimes prevented reconnection from succeeding. The HTTP polling transport has been updated to correctly encode session tokens.

JavaScript Client: TypeScript definitions for topic notifications feature incorrect (18151)

The TypeScript definition for topic notifications specifies an incorrect method name for adding topic notification listeners. This prevents writing TypeScript code using the topic notifications feature that satisfies the definitions and calls the correct method. The definitions have been updated to use the correct name.

JavaScript Client: Missing topic types in web friendly TypeScript definitions (18185)

Not all of the topic types are included in the TypeScript definitions packaged with the Diffusion server. The definitions have been added.

JavaScript Client: TypedSubscription and ValueStream TypeScript definitions declare methods that are not implemented (18308)

The TypeScript definitions for ValueStream and TypedSubscription were not correct. They extended Subscription, including methods like asType that are not implemented for the ValueStream. The definitions have been changed to extend Stream.

Logging: String representations of byte amounts are rounded down (18333)

When reporting byte counts in the logging, values were always converted to the most appropriate unit and then rounded down. This has been changed to convert to the most appropriate unit and then round to the nearest integer value.

Logging: Incorrect server log message when update source registration fails (18711)

When an update source registration failed the server logged an incorrect message where the session id and topic path were the wrong way around. This has now been fixed.

Monitoring: Subscription messages are not counted towards publisher statistics (18436)

Subscription notifications were not counted towards publisher statistics. The statistics under reported the number of messages and bytes sent. Subscription notifications are now included in the statistics.

Replication: Partition can be left unclaimed if it is migrated before recovery (18009)

It is possible for all members of the cluster to believe that a partition is claimed by another member of the cluster if it is migrated during partition recovery. This could prevent topic operations with the CLUSTER_REPARTITION error reason for some topics. This has been addressed by attempting to claim partitions associated with closed peer connections.

Replication: ConcurrentModificationException deregistering distributed resources (18212)

Access to the registry of certain distributed resources such as update sources was not thread safe. This could cause access of stale data or throw a ConcurrentModificationException when interleaving access and modification. The registry has been made thread safe.

Replication: IllegalStateException error message does not match the cause of the problem (18304)

An error message when there is a problem with the state transitions does not accurately describe the cause of the problem. The message has been updated to be more relevant and contain more contextual information.

Replication: Partition fails to be claimed by new owner (18698)

When a Hazelcast partition begins to migrate to a different owner an entry processor is used to update the owner and increment a generation counter. If the entry processor runs before migration completes it would fail to update the owner or increment the generation. This prevented other servers recognising the ownership had changed. This would cause all operations applied to that partition to fail with the CLUSTER_REPARTITION error reason. This has been fixed by retrying the entry processor until it succeeds or the generation changes (which indicates another server claimed the partition).

Replication: When the cluster master shuts down a new master may fail to be selected (18704)

When the cluster master shuts down, all other servers attempt to claim ownership. It's possible that they will all fail to claim ownership as it's still held by the shutting down server. This can prevent certain clean up operations being applied to the cluster, leaking memory and other resources. This has been fixed so that the server being shutdown does not hold onto ownership and another server will be able to claim ownership.

Security: Slave Topic Permissions Inadequate (14087)

Previously a session could create a slave topic that referred to a master topic for which the session did not have READ_TOPIC permission. A session that could create a slave topic could abuse this to bypass READ_TOPIC permission checks. It is now required that a session creating a slave topic has READ_TOPIC permission to the master topic path.

Server: Conflation prevented when using indexed message queue (18138)

The indexed message queue implementation does not correctly maintain the indices when grown. The indexed message queue is used only when requested by a system property and no consequences would usually be noticed. A corrupted index would prevent the indexed lookup of messages from succeeding. This would prevent the conflation of messages.

Server: Closing replicated update source should release resources (18437)

Closing a replicated update source should free all resources associated with it across all servers in the cluster. It was observed that when the update source was in the standby state some resources would not be freed across the cluster. This could interfere with the registration of new update sources. This was because of the failure to broadcast the state change. This has been fixed so that state transitions are consistently broadcast to the cluster.

Server: ArrayIndexOutOfBoundsException queue overflows dispatching topic notifications (18586)

When queuing topic notifications it is possible for the server to overflow the outbound queue for a session. It was sometimes possible to see an ArrayIndexOutOfBoundsException in the server log when this happens. This happened when the server attempted to queue a notification for a session that has already overflowed the queue. The server will no longer attempt queue these notifications.

Server: Unsubscribe could corrupt data structures and cause NullPointerException when evaluating selections (18765)

Under rare circumstances, a bug could result in corrupt internal data structures following an unsubscribe operation. This prevented correct evaluation of new topic selectors. A NullPointerException error similar to the following would appear in the server log. 2018-04-24 16:46:36.619|ERROR|multiplexer1|PUSH-000229|Error while handling a multiplexer event.|com.pushtechnology.diffusion.multiplexer.impl.AbstractMultiplexer java.lang.NullPointerException at com.pushtechnology.diffusion.multiplexer.server.ServerMultiplexerStateImpl.lambda$evaluateSelections$0(ServerMultiplexerStateImpl.java:293) at com.pushtechnology.diffusion.collections.UnsafeBitSetAccess.wordForEach(UnsafeBitSetAccess.java:128) at com.pushtechnology.diffusion.collections.UnsafeBitSetAccess.internalSparseForEach(UnsafeBitSetAccess.java:110) at com.pushtechnology.diffusion.collections.UnsafeBitSetAccess.forEach(UnsafeBitSetAccess.java:63) at com.pushtechnology.diffusion.multiplexer.server.ServerMultiplexerStateImpl.evaluateSelections(ServerMultiplexerStateImpl.java:293) ... The bug has been fixed in this release.

Topics: Topic Notification Listener feature provides ambiguous descendant notifications in certain cases. (18156)

The Topic Notification Listener feature will now emit additional descendant selection notifications to remove ambiguity from situations where a change in selections or the topic tree would leave a client with incomplete knowledge of their current selection state. This improvement applies to all Client SDKs.

Known Issues

Client: Removing topics with session does not work with persistence or replication (18607)

The removeTopicsWithSession capability of the TopicControl feature does not work correctly if persistence is turned on. This is because the topics would be persisted before the removal is executed, and if the server stops without executing the removal, then topics would have effectively 'leaked' into persistent store. The removal capability does not work across a cluster and so should not be used when replication is in use. These methods have now been deprecated and the new REMOVAL topic property should be used in preference.

Client: TopicEventListener subscription notifications do not work across a cluster (18619)

The TopicEventListener subscriber notifications do not work across a cluster and therefore should not be used with replication. The main use case for this feature is detecting when topics are no longer in use, so that they can be removed. This is now better done using the new REMOVAL topic property.

Client: New user defined properties are not reported to Session Properties Listeners (19541)

The documentation for session properties listeners says that when there is a session event, the 'previous values' map will contain an entry with a null value for any new property. This does not occur - there will be no entries in the previous values map for new properties. This will be resolved in a future release.

Console: Logging into console with non-admin credentials provokes errors (18688)

Logging into the web-based monitoring console with credentials for a principal which lacks the required admin permissions outputs errors on the JavaScript console, and display empty tabs instead of a useful UI.

Console: Console shows fractional users connected. (18768)

The concurrent clients and client connections graphs on the console can display fractional users.

Console: The button to the console sometimes does not appear (18769)

The button to the console does not appear on the server dashboard under some circumstances.

Java & Android Client: Record Topics do not notify current value to TopicStreams (18767)

There is a known issue with the legacy record topic type. If the Topics feature is used to add a new TopicStream which covers record topics that are already subscribed to, the stream will be notified of subscription to the topic but will not be given the latest value for the topic and will be unable to process subsequent deltas. For this reason, when using record topics, it is important to add the stream that will process the topics is added before the topics are subscribed to. This does not apply to the new recordV2 topic type. If possible, you should use the recordV2 topic type, not the legacy record topic type, which is deprecated.

Publisher API: Rapid adds and deletes of topics can cause TopicTreeListener notifications to arrive out of order (18766)

Under some circumstances, when the same topic is being added and removed very rapidly, it is possible for the added and removed notifications to be delivered to the TopicTreeListener out of order. For example, you may get several adds before a remove. Even the addition and removal of different topics very rapidly may not notify in the exact order that the requests were made, although this is only likely to cause an issue where processing of branch overlaps. There is no immediate plan to resolve this issue as the cases where it would have an impact are thought to be extremely rare.

Publisher API: TopicTreeListener notifications incorrect for topic detach/attach operations (18770)

Since release 5.9 it is possible to have unbound topic tree nodes created by the standard API (i.e a node that has no topic). So when a hierarchic topic A/B/C is created then nodes A and A/B will have no associated topic. But for backwards compatibility when such a topic is created by a publisher then A and A/B will be stateless topics. The Publisher API will see such unbound nodes as if they had a stateless topic, even though there is no usable topic at the node. For consistency when the standard API adds a topic to an unbound node the TopicTreeListener is notified of a remove (the notional stateless topic) followed by an add (of the newly attached topic) and when a topic is removed from a node (as opposed to the topic branch being removed) then the TopicTreeListener is notified of a remove (the previous topic) followed by an add (of a notional stateless topic). The problem with this is that in both cases the remove notification suggests that the whole branch has been removed when in fact it has not. It would be more correct if the remove was notified on the TopicDeletionListener but the fact that these are different interfaces makes this impractical. This may be resolved in a future release by rationalizing the notifications into a single notification interface.

Replication: Routing topic handlers cannot reliably create replicated topics on demand (18771)

In a deployment without topic replication, a routing topic handler can handle a routing request by creating a topic on demand and responding with its path. Such a strategy is unreliable with topic replication. If a routing topic handler creates a replicated topic, the topic may not be created on the server managing the routing subscription before the routing subscription is evaluated, leaving the session unsubscribed.

Server: Console button not displayed (15872)

Under some circumstances, the button linking to the Diffusion monitoring console was not displayed when viewing the server homepage in a web browser.