Diffusion 6.5 Release Notes

6.5.12 (24 February 2022)

Fixes in 6.5.12

Topics

28319: IllegalArgumentException in TopicTreeNodeImpl

A concurrency bug in previous releases could corrupt topics in the topic tree. One side-effect is that a subsequent attempt to add a topic could fail with an IllegalArgumentException. The bug has been fixed in this release.

6.5.11 (11 February 2022)

Fixes in 6.5.11

Server

28131: New subscription inadvertently removed existing subscriptions

Due to a bug in previous releases, the topic selectors maintained by the server for a session could be corrupted by subscription and unsubscription operations. Specifically, the problem could be triggered if a session subscribed to a topic selector with a descendant pattern qualifier ("/", or "//"), for example "?a//", then later redundantly subscribed to a topic selector that is a strict sub-selector of the first one, for example "a/b". The bug could cause another topic selector to be removed in ways that were hard to predict.

The bug has been fixed in this release.

6.5.10 (16 December 2021)

Fixes in 6.5.10

Security

28012: Upgrade log4j2 to address CVE-2021-45046 security vulnerability

The log4j2 logging library used by Diffusion has been upgraded to version 2.16.0. This addresses a critical security bug [CVS-2021-45046] in log4j2. See https://logging.apache.org/log4j/2.x/security.html for details.

6.5.9 (13 December 2021)

Fixes in 6.5.9

Console

27921: Console does not allow connection timeout to be specified

The Diffusion management console did not allow a connection timeout to be specified at login. This option has been added.

JavaScript Client

27928: Connection timeout not configurable

The JavaScript client was missing an option to specify the connection timeout on establishing a session. This has been rectified.

27951: Authenticator throws 'Cannot read properties of null'

When closing an authenticator, it would throw a "REGISTERED_HANDLER_EXCEPTION TypeError: Cannot read properties of null". This has been fixed.

Security

27980: Upgrade log4j2 to address CVE-2021-44228 security vulnerability

The log4j2 logging library used by Diffusion has been upgraded to version 2.15.0. This addresses a critical security bug [CVS-2021-44228] in log4j2. See https://logging.apache.org/log4j/2.x/security.html for details.

6.5.8 (16 November 2021)

Fixes in 6.5.8

.NET Client

26966: Disconnection due to "Http fragmentation and extension not supported"

An issue was identified with the .Net client's handling of partial reads. This has now been addressed.

C Client

27410: Segmentation fault in add_time_series_stream

Fixed a segmentation fault that occurred when calling add_time_series_stream with a NULL session.

27556: hash_num_new is using minimum slots instead of maximum slots provided as parameter

hash_num_new now correctly uses the maximum number of slots when creating a hash map.

Java Client

27838: Memory leak in Java client on multiple reconnections

Repeated reconnections from the Java client could cause a memory leak of session related objects in the client VM. This has now been resolved.

JavaScript Client

27137: In docs of topic specification, some entries in the topic properties table are incorrect

The documentation of topic properties in TopicSpecification contained incorrect entries and has been fixed.

27143: int64 docs reference overflowing number

The documentation of Int64 referenced a number that would cause an overflow. It has now been replaced with a number that fits into a signed 64-bit integer.

27699: Sessions can reconnect even if explicitly closed by another session

An issue has been resolved where the server allowed clients to reconnect during the reconnection timeout, even if they had been explicitly closed by another session. This would only occur if session replication was enabled.

Replication

27325: "replicated-topics-restored" start condition does not work

Connectors can be configured not to accept connections until a set of conditions is satisfied.

Due to a coding error in previous releases, the "replicated-topics-restored" condition was never triggered. This has been corrected in this release. The condition is satisfied after a server has joined the cluster and received all of the topic data from existing members of the cluster. The server will log a PUSH-000834 INFO message when this happens.

27335: Correct cluster recovery of time-series topics

In previous releases, due to a coding error, re-distributing time series topic data when servers join and leave a cluster did not scale to large numbers of time series events. This could cause protracted instability whenever the cluster topology changed. The problem has been fixed in this release.

Server

27839: Possible leak of sessions (and session metrics) that time out during connection

In certain situations, a client session failing to connect due to a timeout could lead to a memory leak where the server side client object remains. This would also affect metrics as the failed session would remain in the 'open' and 'connected' counts.

This problem has now been resolved.

6.5.7 (21 July 2021)

Fixes in 6.5.7

.NET Client

27123: Credentials type is incorrect when authenticating

The .NET client was giving the wrong credentials type for custom credentials when authenticating. This has now been resolved.

JavaScript Client

26941: JSON.jsonDiff(original) does not take arbitrary objects - as documented

Previously, JSON.jsonDiff(original) did not take arbitrary plain objects as documented. This has been fixed.

Security

26952: TOPIC_OWNER topic property not respected when a session changes its principal

When a session re-authenticates using the "change principal" API, the server re-evaluates its subscriptions. In previous releases, the re-evaluation failed to take account of the TOPIC_OWNER topic property. This bug has been fixed in this release. Now a session's subscriptions can change if it re-authenticates using a different principal so that it gains or loses ownership of a topic.

Topics

27014: Time Series range query returns incorrect values

In some situations, a time series topic range query could return incorrect values. This could be seen in the Console or in range queries issued by clients. This has been resolved in this release.

6.5.6 (16 June 2021)

Fixes in 6.5.6

Apple Client

26251: Inconsistent HTTP Header format

An issue has been fixed regarding the HTTP header format in the Apple client, which prevented certain proxy tools from being able to correctly validate.

26571: Apple client cannot connect securely to Diffusion server via proxy

When the Apple client connected via secure WebSockets to a Diffusion server via a proxy, the connection was incorrectly downgraded to non-secure. This has now been fixed.

Java & Android Client

26360: Header parsing during proxy authentication is not case-insensitive

Checks on headers such as Keep-Alive and Connection during proxy authentication were wrongly case-sensitive. This has been fixed.

JavaScript Client

26301: JSON Delta fails with indefinite length arrays and objects

JSON deltas failed when the JSON was encoded using CBOR with indefinite length arrays and objects. JSON deltas now work with definite and indefinite length encoded CBOR.

26550: Deregistration callbacks return Promises but this is not reflected in the TypeScript API or documentation

Deregistration callbacks in MissingTopicHandler and SessionPropertiesListener return a Promise that resolves when the listener has been deregistered. This is now reflected in the documentation and in the TypeScript API.

26553: DataType.writeValue can accept null but this is not reflected in the TypeScript API

writeValue for primitive and JSON datatypes can accept null as a value. This is now reflected in the TypeScript API.

Replication

26409: Cluster routing could fail to find a handler hosted by a peer

When a server belongs in a cluster, if it has no local handler for various types of event (such as messages, missing topic notifications, and authentication requests) but a peer server does, the event will be forwarded to the peer for processing.
Due to a bug in previous releases, some requests would not be forwarded even though a peer with a suitable handler existed. The bug has been fixed in this release.

Server

26232: NullPointerException thrown by ServerMultiplexerStateImpl#enableReplication

A NullPointerException which was logged when recovering a time series topic from the replication log after restoring it from file persistence has been fixed.

System Monitoring/Statistics

26586: JMX service binding to all interfaces, despite configuration

Diffusion 6.5.0 added support for JMX over TLS, and made this the default configuration.
Due to a bug, when configured for TLS, the JMX service always bound to all local network interfaces regardless of whether a specific interface was specified by the host parameter in Management.xml. The bug has been fixed in this release, and the host parameter is now respected.

Topic Views

26250: Missing or incorrect values in reference topics when topic view evaluations are chained

In some cases where a topic view generated reference topics that were then evaluated as source topics for another topic view, other reference topics generated by the original view were not generated as expected (either not created or with invalid values).
This problem has now been resolved.

26455: A throttled topic view can produce an invalid delta stream

Due to a bug in previous releases, it was possible for a topic view with a throttle clause to produce a corrupt delta stream. The bug has been fixed in this release.

26591: Range query fails on time series reference topic created from a remote topic view

If a time series reference topic was created by a remote topic view, range queries on the reference topic would fail and falsely indicate that the topic was not a time series topic. This problem has now been resolved.

Topics

26191: IllegalArgumentException when adding or removing topics

Diffusion 6.5.5 included a fix to avoid an IllegalArgumentException when adding or removing topics (reference 26083). The fix was incorrect and IllegalArgumentExceptions could still occur. The problem has been fully remedied in this release.

6.5.5 (2 March 2021)

Improvements in 6.5.5

C Client

24868: APR Library update

Following the GLIBC update to v2.32, APR and APR-Utils libraries have been updated to 1.70 and 1.6.1 respectively.

Server

24801: Optimise subscription re-evaluation when a session's roles are changed

When a session's roles are changed using change principal or change roles API functions, it may gain or lose READ_TOPIC permission to topic paths. The server re-evaluates the topic selections for these paths, and the session will be subscribed to new topics for which it gains READ_TOPIC permission, and unsubscribed from topics for which it no longer has READ_TOPIC permission. This re-evaluation has been optimized in this release.

Fixes in 6.5.5

Configuration

26025: Configured inbound thread pool sizes were ignored

Since Diffusion 6.5.0, the configured size of inbound thread pools has been ignored. Instead, a default value (5 threads) was used. This bug has been rectified in this release so the configured size will be used once again.

Console

24789: Console users can see add/modify metric collector buttons without required permissions

The Diffusion management console displayed buttons and forms for adding and modifying metric collectors, even if the user only had viewing permissions. Attempting to use these buttons would cause an error. These buttons are now only displayed for users with the appropriate permissions.

26095: Logging out on Security/Auth tab with changes breaks console

The Diffusion management console could display an unrecoverable error if a user attempted to log out while they had unsaved settings on the Authentication or Security tabs. Logging out will now automatically discard unsaved changes.

JavaScript Client

24779: Time series value stream event datatype compatibility problem

Value streams are compatible with parent types. For example, a ValueStream<JSON> can deserialize values from a String topic type.

ValueStream<Event<JSON>> did not propagate the values for time series topics with the String event type. This has now been fixed.

24881: RecordModel.get should allow obtaining a record by key only

The implementation of RecordModel.get could take a single key argument but the documentation and the Typescript API did not reflect this. This has now been corrected.

24884: messes.setRequestStream and timeseries.append should take topic types and plain JS type constructors for datatypes

The implementation of setRequestStream allows topic types and plain JS type constructors for datatypes. This is now correctly reflected in the public TypeScript API .

24888: First call of UpdateStream.set returns a TopicCreationResult

The first call to UpdateStream.set returns a TopicCreationResult. This was not reflected in the API. This has now been rectified to the TypeScript definitions and added to the documentation.

24946: diffusion-worker.js is missing from package

The diffusion-worker.js bundle was not included in the distribution package. It has now been added.

26002: Cannot pass true to responder.respond()

The first call to UpdateStream.set returns a TopicCreationResult. This was not reflected in the API. This has now been rectified to the TypeScript definitions and added to the documentation.

26006: Req/Resp message handlers calling responder.reject throws unexpected exception and prevents further handling

Calling responder.reject in request/response messaging caused an unexpected exception and prevented further handling. This has been fixed.

26084: QueryResult.merge throws exception: "Error: TypeError: latestByOriginalSequence[k].sequence.toNumber is not a function

Time Series QueryResult.merge() threw an exception when the two query results had events in common. Now, merge works as expected.

Server

24831: Server deadlock during start up

In previous releases, the server could potentially fail to start, logging a PUSH-000773 message indicating the server was making slow progress executing DefaultPicoContainer.addOrderedComponentAdapter. This problem has been resolved in this release.

24871: Sessions can reconnect even if explicitly closed by another session

The server allowed client sessions to reconnect during the reconnection timeout, even if they had been explicitly closed by another session. This would only occur if session replication was enabled.

26001: StackOverflowError when calculating differences between values

In previous releases, when the rate and complexity of difference calculations caused the adaptive delta streaming algorithm to reduce the difference quality, it was possible for difference calculations to fail abnormally with a StackOverflowException. The bug has been fixed in this release.

26080: Time series topic stored byte metric not reset on sequence gap

Time series topic metrics were not reset when a sequence gap was detected. This affected topic metrics for these topics. A sequence gap may occur when a clustered topic is updated while offline with file persistence activated.

26083: IllegalArgumentException when adding a topic

A concurrency bug introduced in Diffusion 6.5.0 could leave stale topics in the topic tree. One side-effect is that a subsequent attempt to add a topic could fail with an IllegalArgumentException. The bug has been fixed in this release.

System Monitoring/Statistics

26078: diagnostics.js does not support Java 11

The diagnostics script has been updated so that it can be run under both versions 8 and 11 of Java.

Topic Views

26000: Delayed topic views do not work with Time Series topics

There was a problem with topic views that use the 'delay' option if they selected a time series source topic. No resulting reference topic was ever published and an attempt to fetch the reference topic cause a ClassCastException.
This problem has now been resolved.

26099: Failure to process topic view if an expand clause has a path pointer that does not match the expanded structure

The topic view expand clause has an optional a path pointer that can be used to extract an alternative part of the expanded value to use in the derived path in place of the natural key.

In previous releases, processing of topic views with expand clauses could fail in arbitrary ways if the path pointer did not match the expanded value. The bug has been fixed in this release.

6.5.4 (16 December 2020)

Improvements in 6.5.4

System Monitoring/Statistics

24567: Issue log warning the first time a deprecated service is called

The server now logs a warning the first time a deprecated service is used, indicating a client is using a deprecated API.

Fixes in 6.5.4

Apple Client

24660: HTTP Headers content checked with case-sensitive values

Addresses an issue where the Apple client fails to establish a session due case-sensitive mismatches on HTTP headers. This has now been resolved.

C Client

24490: Selectors with descendant qualifiers not evaluated

Selector strings containing descendant qualifiers were not evaluated in the C client. This has now been resolved.

Configuration

24273: Obfuscated values in Env.xml are not parsed correctly

Obfuscated values entered in Env.xml were not being parsed correctly, meaning that property obfuscation could only be used in other configuration files.

This has now been resolved.

Installation

23853: Service not correctly installed from RPM

RPM was copying diffusion.service to /etc/system. This is now copied to %{_unitdir} which customers can override to /etc/systemd/system if needed.

JavaScript Client

24511: Buffer implementation not supported outside of NodeJS

Previously, there was no access to the Buffer API when running the client in a browser context. The Buffer API is now available through diffusion.buffer.

Topics

24302: Automatic topic removal policy using "when this session closes" - topic can be removed while session still exists

If a topic had the 'when this session closes' REMOVAL clause and the creating session then failed over to another server in the cluster then the topic would be removed. This is incorrect as the topic should remain as long as the original session does.
A similar problem could also occur if the creating session has its session properties changed in any way.
These problems have now been resolved.

24306: Potential failure to process time series updates

In previous releases, a rapid sequence of updates to time series topics could fail to be processed correctly, leading to data corruption. This bug has been fixed in this release.

6.5.3 (29 October 2020)

Fixes in 6.5.3

Client

24070: Session properties were not delivered to request handlers registered using ALL_FIXED_PROPERTIES and ALL_USER_PROPERTIES

An issue has been resolved where using Session.ALL_FIXED_PROPERTIES and Session.ALL_USER_PROPERTIES was not supported for selecting the properties delivered in the context delivered to a request handler.

Console

23818: Can't type negative values in console topic editor

The topic value editor in the Diffusion management console did not accept negative values for int64 and double topics. This has been fixed.

23978: Console wrongly allows creating time series topics with DONT_RETAIN_VALUE enabled

The Diffusion management console allowed users to configure new time series topics with the DONT_RETAIN_VALUE topic property, which is not allowed for time series topics and was ignored. This property is no longer listed for time series topics.

24096: Clear button produces error for time series topics

The Diffusion management console wrongly presented the "clear" button for time series topics. Using it would cause an error. The button has been removed from time series topics.

Java & Android Client

23957: Flow control can stall update streams under high load

In previous releases, a bug could cause a Java client using update streams to never recover from flow control, resulting in updates stalling. The bug has been fixed in this release.

JavaScript Client

24065: Error when client attempts to establish an insecure connection from a secure origin

Error "Failed to connect TypeError: Cannot read property 'close' of undefined" could occur when attempting to establish an insecure connection from a secure (HTTPS) origin. This problem has now been resolved.

24084: Cannot use numbers/int64/float in JSON constraints

Previously, int64 topic data was serialized inconsistently for 32-bit integers and wrongly for negative 64-bit integers. This caused partial JSON constraints to fail for integer values. It also prevented negative 64-bit integers from being passed between clients. Integers are now serialized correctly and in line with other clients. Partial JSON constraints can be used with the Int64 datatype.

Security

23746: Updating security configuration incorrectly requires the VIEW_SESSION permission

From Diffusion 6.5.0, security control updates that change path permissions required the VIEW_SESSION permission in addition to the MODIFY_SECURITY permission. From this release, the previous behavior has been restored – MODIFY_SECURITY alone is sufficient for all security control updates.

24068: Failures to process locked security resources

The API allows security roles and system authentication principals to be locked by a particular principal. Due to bugs in previous releases, processing of locked resources could fail:
A server could not boot against a Security.store or SystemAuthentication.store configuration file containing one or more lines that modify a locked resource. The work around was to remove the offending lines.
In a cluster, a modification to a locked resource was not properly replicated. Instead, an error message would be logged by all servers receiving the replicated update.
Both bugs have been fixed in this release. Locked security roles and system authentication principals now function as designed.

Topic Views

24054: Topic view publishes new value to old topic before removing it if scalar mapping is used in a simple (non-derived) view mapping

A topic view specification that contained a scalar path mapping but no other JSON type value mappings could result in a new value on the source topic being published to the reference topic before the removal of the reference topic due to a change to the indicated scalar value.

This problem has now been resolved.

6.5.2 (24 September 2020)

Fixes in 6.5.2

Topic Views

23619: Multiple sessions creating the same topic view causes topic view re-evaluation

Creating an identical topic view with the same name and specification by the same principal (using the same roles) would unnecessarily cause the view to be replaced, leading to the removal and recreation of all reference topics derived from it.
This has now been resolved.

23659: Deadlock due to TopicViewEvaluationImpl.reevaluateViewForAdds conflicting with update

A deadlock could occur in the server when a re-evaluation of a topic view conflicted with a concurrent update, leading to clients being unable to connect. This caused updates to stall. This has now been resolved.

6.5.1 (13 August 2020)

Fixes in 6.5.1

.NET Client

23072: .NET reconnection fails

In release 6.5.0, the .NET Client was unable to reconnect to the Diffusion server if connection was lost. This has now been resolved.

23113: Incorrect annotation for methods added to IMessaging


23362: Failure to receive messages larger than the configured buffer size

The .NET client was unable to receive messages larger than the configured buffer size. This has now been resolved.

Console

23107: Console unable to show subscriptions to time series topics

The Diffusion management console's subscriptions tab did not display the value for subscribed time series topics. Time series topic values are now displayed in the same manner as for other topic types.

23191: Management console subscriptions tab scrolls to top upon topic value update

When viewing topic subscriptions in the Diffusion management console, the page could sometimes scroll unpredictably when a topic value was updated. This has been fixed.

23205: Console JSON parser does not accept integers in non-canonical string forms

The Diffusion management console incorrectly refused to accept certain non-canonical numeric values in user-supplied JSON while editing topic values. This has been addressed. If you are providing integer data in JSON, to avoid loss of precision, do not enter values with a decimal point. For example, enter "12" and not "12.000".

23206: Incorrect session metrics in management console

In the Diffusion management console, all metric collector breakdown categories incorrectly displayed the same information as the first listed category. All breakdown categories now display the correct information.

23235: Unexpected session sharing between Diffusion console using different principals

If you attempted to log in to the Diffusion management console in two tabs concurrently, it could appear that you had logged in using two different principals, when in fact only one shared session had been established. You will now be warned if you log in while a shared session exists.

23242: Remote servers tab: status checks attempted without required permission

The Diffusion management console could present loading messages and warnings when viewing the network tab as a user without the CONTROL_SERVER permissions. These misleading messages have been removed.

23295: Adding a topic fails to update topic browser in certain cases

The Diffusion management console would fail to refresh the topic tree after adding a new topic in some circumstances. The topic tree should now always refresh.

Federation

23068: Remote server connections require unique server names to be configured

In the previous version, remote server connections prevented loopback connections to the same server by checking that the server name was not the same. However, this meant every server had to have a unique name, which caused problems with some configurations. Loopback is now prevented with a unique server ID, rather than the server name, removing the need for unique server names.

JavaScript Client

23154: addStream with AnyDataType does not receive time series events

A stream created using session.addStream with datatypes.any() previously did not receive time series events. This has been fixed so that time series events now correctly trigger the value handler.

23348: remoteTopics.js example - undefined variable 'datatypes'

The removeTopics.js example for the JavaScript client has the undefined variable 'datatypes'. This has been fixed.

Replication

23267: Cluster operations incorrectly return SessionException

When using server clustering, a SessionException was returned in CompletableFuture results instead of a subclass (for example, NoSuchTopicException). This issue has been fixed.

Server

23076: Subscription operations occasionally delayed

Due to a bug in previous releases, the result of a subscription operation would occasionally be delayed. Further, if the server was under heavy load, the bug could cause the failure of a multiplexer thread, requiring the server to be restarted. The bug has been fixed in this release.

23356: PUSH-000798|Unexpected multiplexer recursion - TopicRemovalDownstreamChecksImpl

A "PUSH-000798 Unexpected multiplexer recursion" error could be logged to the server when automatic topic removal was in use. The recursion occurred in TopicRemovalDownstreamChecksImpl.

Though this was logged as an error, it is not known to have caused problems with the actual processing of topic removals.

This problem has now been resolved.

6.5.0 (13 July 2020)

New features in 6.5.0

Console

21585: Redesigned management console

The Diffusion management console has been completely refreshed, with changes including a complete graphical overhaul, support for shared sessions (stay logged in over multiple tabs), redesigned pages for configuring security and authentication settings, the ability to configure remote server settings, an updated topic view editor with assistants for new topic view features, plus many small bug fixes, usability tweaks and performance improvements.

JavaScript Client

20395: Shared sessions between web browser tabs

The JavaScript client now supports sharing a single session across multiple tabs and pages in the same web browser. This avoids each tab having to separately subscribe to the same topic data, improving consistency and responsiveness, and removes the need to re-authenticate when opening a link in a new tab.

Passing the name of a shared session and the URL of the diffusion-shared.js file to diffusion.connect() will create a session that runs in a SharedWorker context. A call to diffusion.connectShared() will connect through an existing shared session. Shared sessions support the same API as a regular session except for the deprecated Session.fetch() operation.

The management console makes use of this feature. If the browser is already hosting a console session, additional console pages can be opened in different browser tabs without logging in again.

Security

10668: Subscriptions are updated when authorisation rules are changed

In previous releases, changes to the security store that alter READ_TOPIC permission assignments only affected an existing session if the roles assigned to the session were subsequently changed. A session's assigned roles change if it reauthenticates using Security.changePrincipal(), or a control session updates the session using ClientControl.changeRoles().

From this release, changes to the security store are immediately applied to all sessions, resulting in the appropriate subscriptions and unsubscriptions. Each session's topic selections will be re-evaluated against topics for which the session has gained READ_TOPIC permission, and sessions will be unsubscribed from topics for which they no longer have READ_TOPIC permission.

21790: System authenticator accepts client-proposed properties from a trusted list

In previous releases, the system authenticator would ignore session properties proposed by a connecting client. From this release, you can specify that certain client-proposed properties properties should be trusted in the system authentication configuration. For each trusted property, a validation rule is configured, specified as a list of allowed property values or a regular expression that the supplied value must match.

System Monitoring/Statistics

15656: Service metrics

Metrics for Diffusion's internal service layer are now published to a JMX MBean and Prometheus. These metrics are unsupported and subject to change.

Topic Views

20299: Remote topic views

You can now specify a topic view that takes its source topics from a remote server (in a different cluster). This is enabled using the new 'from' clause in a topic view specification which allows a named server to be specified for the location of the source topics. The named server and its connection details are configured using the new 'Remote Servers' feature.

Remote topic views effectively supersede fan-out. Where fan-out only allows a one-to-one mapping of topics from primary to secondary servers, remote topic views include all of the path mapping and topic transformation capabilities of topic views. Also, where fan-out requires file-based configuration at every secondary server, remote topic views can be configured using the client API and are replicated across a cluster.

20399: Delayed topic views

Topic view expressions now support a new, optional delay clause. When the source topics of a delayed topic view change, there is a fixed delay before the change is reflected in its reference topics. Topic additions, updates, and removals are all delayed. Delays can range from one second to many days​.

Improvements in 6.5.0

.NET Client

22182: The ISecurityControl and ISystemAuthenticationControl script builders are now mutable

The security control and system authentication control features both provide script builders to help with constructing scripts.

In previous releases, script builders were immutable: each builder operation returned a fresh instance, leaving the original instance unchanged.

From this release, script builders are no longer immutable. Each builder operation mutates this script builder and returns it. This avoids copying, so is more memory efficient, and much faster when using a script builder to create a large script.

22367: ClientControl.close() and setConflated() now accept a session filter

Two new functions have been added to IClientControl.cs, both taking a session filter parameter

CloseAsync - close all sessions matching a session filter
SetConflatedAsync - enable/disable conflation for all sessions matching a session filter

Adapters

22123: JMS adapter now uses Request-Response messaging

The JMS adapter now supports request-response messaging in the place of one way messaging which has been removed in this release.

Apple Client

20436: FetchResult.size() and Fetch.isEmpty() methods

FetchResult now provides FetchResult.size and FetchResult.isEmpty methods for direct access to the number of fetched results.

22455: Debug symbols are now included

Debug symbol files (.dSYM) are now included with the Apple client SDKs.

C Client

22366: Clients matching a session filter can be closed and have conflation enabled or disabled

Two new functions have been added to client-control.h:

diffusion_client_close_with_filter - close all client sessions which match a session filter
diffusion_client_set_conflated_with_filter - enable/disable conflation for all client sessions which match a session filter

Client

21446: Request-response messaging is now cluster aware

Previously requests could only be sent to handlers and sessions connected to the server the calling session was connected to. Now, if the receiver is connected to a different server in the cluster, the request will be forwarded to the remote session and similarly the response will be routed back to the caller.

21499: Improved API reporting of cluster routing issues

Replicated configuration and replicated topic data is stored in partitions. Each server in a cluster owns a share of the partitions. Partitions can be migrated to different servers in the cluster when servers are started, stopped, or fail. During partition migration, cluster operations may fail. Cluster operations can also fail due to a communication error between servers.

In previous releases, if an API call failed due to a cluster error, a CLUSTER_REPARTIONING error was always returned. From this release, the error reporting has been improved. CLUSTER_REPARTITION (represented by various constants or exceptions, depending on the SDK) is now only used when there has been a definite change in partition ownership. In cases where a cluster operation failed but partition ownership might not have changed, a new CLUSTER_ROUTING error is returned. Both errors are transient and occur while the cluster is recovering from failure. It is reasonable to code client applications to retry operations that fail with either error.

21747: Topic permission has been renamed to path permission

The server's security model allows roles to be associated with permissions which are either global or apply to a path. In previous releases, the path-scoped permissions were called "topic permissions". This was a source of confusion since some of the permissions apply to message paths or to session lock names, both of which are hierarchical but have nothing to do with topics in the topic tree.

In this release, "topic permission" has been renamed "path permission" throughout the API. Additional methods have been added that reflect this renaming. The old methods are deprecated and will be removed in a future release.

21982: Improvements to setSessionProperties(String filter...)

In this release, two improvements have been made to the ClientControl.setSessionProperties() method that accepts a session filter. The method now returns a result that provides the number of matched sessions. The method will now complete with an InvalidFilterException containing further details if the filter is invalid.

22105: Clients now respect the DONT_RETAIN_VALUE property

From this release, clients no longer cache the last value for topics with the DONT_RETAIN_VALUE property set to "true". This reduces the memory footprint on the client for subscriptions to such topics, but when a new stream is registered it will no longer be provided with the last value.

22271: ClientControl operations are now cluster aware

Client control operations have now been enhanced to work across a cluster. The affected operations are:

changeRoles
close
setConflated
setSessionProperties
getSessionProperties.

This change affects both versions that identify sessions using a session ID as well as a session filter. If the session identified by a session ID is not found on the server that the calling session is connected to, the operation will be routed to the server in the cluster that hosts the target session. Commands that identify sessions using a session filter will be executed on all servers.

22363: All operations operating on a client filter now require the VIEW_SESSION permission

A security constraint has been added so that any operation performed with a client filter will require the calling session to have VIEW_SESSION permission in addition to any others previously required.

22448: setSessionProperties now allows some fixed property values to be changed

The setSessionProperties API now allows the following fixed session property values to be changed:

$Country
$Language
$Latitude
$Longitude

Note that $Country will be normalized to upper case, $Language will be normalized to lower case and $Latitude and $Longitude values which are invalid will be set to "NaN" (not a number).

22598: Fetch can optionally include unpublished delayed reference topics

Topic views that use the new "delay by" clause initially create reference topics in an unpublished state. The topics are published once the delay time has expired. A topic in the unpublished state prevents a lower priority topic view from creating a reference topic with the same path. A "withUnpublishedDelayedTopics" fetch request option has been added to the fetch request API to allow sessions to browse unpublished topics.

Console

20775: The console allows a replacement license file to be uploaded

The licence can now be replaced via the management console.

21965: The console in now deployed in the web server rather than as a packaged DAR

Previously the Diffusion console was provided using the legacy DAR packaging system. The console is now deployed directly to the embedded web server if selected during installation. To remove the console from an existing installation, the http/console and http/classic-console folders can be deleted safely. When upgrading from a previous version of Diffusion, it is best to create a fresh install then migrate the configuration: otherwise the console DAR will be ignored in the deploy directory.

Demos

20554: Demos are no longer packaged using the DAR format

During installation, Diffusion demos can be optionally deployed. This previously had the effect of adding demos to the deploy folder using the DAR format. From this release, publishers have been removed as well as support for the DAR archive format. Instead, deploying demos is performed by placing them in the default webserver's file service at html/demos. The publishing client for the sportsbook demo is no longer started automatically, but is available in examples/demos and can be started by running the command 'java -jar sportsbook-demo-publisher-jar-with-dependencies.jar'.

22520: Source code for Sportsbook demo publishing client

The source code for the publishing client of the Sportsbook demo is now available under `examples/demos/sportsbook` if demos and examples are selected during installation.

To build, run `mvn clean install` at the root of the project.

Installation

22502: Separate server start scripts for Java 11

There is now a separate shell script to start Diffusion with Java 11. If you are using Java 11, use diffusion_java11.sh or diffusion_java11.bat to start the server.

Java & Android Client

19672: ClientControl.close() and setConflated() now accept a session filter

New versions of ClientControl.close and ClientControl.setConflated have been added to the ClientControl feature which accept an a session filter as an argument to allow batch operations. These complement the existing methods which allowed the same operations on a single session.

22181: The SecurityControl and SystemAuthenticationControl script builders are now mutable

The security control and system authentication control features both provide script builders to help with constructing scripts.

In previous releases, script builders were immutable: each builder operation returned a fresh instance, leaving the original instance unchanged.

From this release, script builders are no longer immutable. Each builder operation mutates this script builder and returns it. This avoids copying, so is more memory efficient, and much faster when using a script builder to create a large script.

22398: CompletableFuture version of getSessionProperties

A method for accessing session properties which returns a CompletableFuture is now available. This accepts a sessionId and collection of required properties and returns a CompletableFuture containing a map of the requested properties for the client identified by the sessionId.

22816: New ServerInitializingException returned from SessionFactory.open

A new subclass of SessionEstablishmentException has been added called ServerInitializingException.

This exception is thrown by SessionFactory.open() if a connection failed because the server was starting and was not yet ready to accept connections.

The new exception allows a client to determine whether it makes sense to try to connect again.

JavaScript Client

20262: Use data type compatibility to support polymorphic streams

The JavaScript client now supports an AnyDatatype for creating polymorphic ValueStreams. This datatype allows subscribing to multiple topics, independent of their value type.

22098: Session.sessionID now a SessionId, not a string

A new property Session.sessionId of type SessionId has been added. Session.sessionID of type string has been deprecated

22183: The security and authentication script builders are now mutable

The security control and system authentication control features both provide script builders to help with constructing scripts. In previous releases, script builders were immutable: each builder operation returned a fresh instance, leaving the original instance unchanged. From this release, script builders are no longer immutable. Each builder operation mutates this script builder and returns it. This avoids copying, so is more memory efficient, and much faster when using a script builder to create a large script.

22368: ClientControl.close and setConflated now accept a session filter

ClientControl.close and ClientControl.setConflated have been extended in the ClientControl feature to allow a session filter as an argument to allow batch operations. These complement the existing behavior which allows the same operations on a single session.

22532: ClientControl.setConflated()

The JavaScript client now supports ClientControl.setConflated() to toggle queue conflation for specific sessions at runtime.

22590: Use of AnyDataType in fetchRequest.withValues() supported

AnyDataType can also be used together with FetchRequest.withValues().

Logging

9089: Log messages include session context

From this release, the session ID and security principal are included as additional fields in log messages where they are available.

The log fields are available in the Log4j thread context map under the 'session' and 'principal' keys. This can be used for context-sensitive logging. For example, to only log debug messages for a particular security principal. This can be achieved without restarting the server by setting a new log configuration using JMX.

If you are upgrading an existing configuration, review etc/log4j2.xml from a fresh installation to see the required changes. Also, see etc/log4j.xml for a commented example of context-sensitive-logging.

22399: Changes to connection type names

The internal names and descriptions of some of the client connection types have been changed for consistency. This will only affect messages logged to server logs. The changes are as follows:

JAVA_POLLING_CLIENT is now HTTP_JAVA_CLIENT.
HTTP_BROWSER_CLIENT is now HTTP_JAVASCRIPT_CLIENT.
WEBSOCKET_BROWSER_CLIENT is now WEBSOCKET_JAVASCRIPT_CLIENT.
POLLING_FANOUT_CLIENT is now HTTP_FANOUT_CLIENT.

Replication

14891: Connectors can be configured not to start until replicated configuration, or replicated topics, have been restored

A "cluster-configuration-recovered" start condition has been added. If a connector is configured to depend on this condition, it will not accept connections until the server has joined the cluster and completed restoration of shared configuration.

A "replicated-topics-recovered" start condition has been added. If a connector is configured to depend on this condition, it will not accept connections until the server has joined the cluster and completed restoration of replicated topics.

If a connector has no configured start conditions, it will reject attempts to create new sessions until the cluster-wide configuration has been restored.

Security

21693: Improved path permission inheritance

This release significantly changes the evaluation of path permissions for different roles. In previous releases, path permission rules from different roles were merged before evaluation, so creating a permission rule for a particular path would mask rules for parent paths from other roles. From this release, path permissions are evaluated independently for each role, making it much easier to design role rule sets that can be composed together usefully.

A new security language statement, "isolate path" has been added that disables inheritance of path permissions assigned to parent paths, allowing a branch of the path hierarchy to be configured independently.

If a Diffusion 6.5 server is started against a security store file from Diffusion 6.4 or earlier, the file will automatically be translated to an equivalent configuration under the new rules. This is achieved as follows: for each path in a path permission assignment for a role, add a separate statement to isolate the path. This produces a strictly equivalent model. In practice it is typical that many of these path isolation statements can be removed without affecting an application's security policy, resulting in a simpler configuration.

22045: The security and system authentication files can be read-only

The server will no longer fail to start if the Security.store and/or SystemAuthentication.store files are read-only. Instead, changes made to the stores will succeed, but will be lost on server restart.

22638: The server supports TLS 1.3 if the Java runtime does

In previous releases, the highest Transport Layer Security protocol version supported was TLS 1.2. From this release, if the Java runtime supports it, then so will Diffusion. TLS 1.3 is available in most Java 11 JDKs, but less well supported by Java 8 JDKs – a notable exception being Azul Zulu with the -XX:UseOpenJSSE option.

In previous releases, the TLS versions used by the Diffusion server and Diffusion Java client could be configured using the diffusion.tls.protocols property. From this release, the diffusion.tls.protocols property has been removed. To enable or disable particular protocols, use the interfaces provided by the JDK; see https://www.java.com/en/configure_crypto.html.

In previous releases, the cipher suites used by the Diffusion server and Diffusion Java client were set to a small subset of the available ciphers and could be explicitly overridden using the https.cipherSuites property. From this release, the ciphers are now inherited directly from the JDK, regardless of the https.cipherSuites property. To enable or disable particular ciphers, use the interfaces provided by the JDK; see https://www.java.com/en/configure_crypto.html. Diffusion previously used a subset of ciphers to compensate for Java distributions which had insecure cipher suites enabled. Java distributions are now much better: insecure ciphers are disabled in their default configurations, and adjusted as necessary in JDK patch releases.

Server

21603: Active adjustment of delta quality based on CPU utilisation

Diffusion uses delta streams to improve the efficiency of broadcasting topic updates. The CPU cost of calculating a delta (the difference between a pair of successive values) depends on how different the values are, and is highly dependent on the application workload. If many differences are found, the difference calculation used by Diffusion reacts by reducing delta quality (increasing the size of the delta) to reduce CPU. From this release, additional feedback has been added to actively adjust delta quality based on the amount of CPU time spent by each server thread on difference calculations. This improves the server's ability to adapt to different workloads.

22088: The HTTP polling transport now uses transport compression more efficiently

In this release, several optimizations have been made to the server to improve the efficiency of the HTTP polling transport when the client supports 'deflate' compression.

The ClientServiceConfig compressionThreshold setting is no longer used. The setting has been deprecated and will be removed in a future release.

System Monitoring/Statistics

9215: JMX now supports TLS connections

The JMX connector can now be configured to support TLS connections by setting a file and password in the key-store section of the Management.xml configuration file. If unset, TLS connections will be disabled.

21538: Log metrics now include Hazelcast messages

From this release, the log metrics recorded by the server included messages logged by Hazelcast.

Topic Views

21453: Topic views can no longer be created with an empty name

Topic Views can no longer be created with an empty name.

Topics

22032: Automatic topic removal rules can exclude fan-out sessions

In Diffusion 6.4, the "subscriptions less than" and "no session has" clauses of automatic topic removal policies were extended to consider sessions hosted by servers connected to the local cluster via fanout connections.

In this release, a "local" keyword has been added to automatic topic removal policy expressions which allows these clauses to be constrained to consider only sessions that belong to the local cluster.

For example, the expression "local subscriptions < 10" will evaluate to true if there are fewer than 10 sessions subscribed to the topic in the local cluster, but "subscriptions < 10" will evaluate to true if there are fewer than 10 sessions subscribed to the topic in the local cluster or hosted by servers connected via fanout connections.

Similarly, "no local session has ..." restricts evaluation to sessions belonging to the local cluster only.

22058: An automatic topic removal policy can now contain multiple session clauses

In previous releases, an automatic topic removal policy specified by the REMOVAL property in a topic specification could contain at most one "no session has" or "when this session closes" clause. From this release, multiple such clauses are allowed.

Web Server/HTTP

21964: The server log includes HTTP URLs through which the landing page can be accessed

When opening a network port, the server now logs the URLs the file service is available on. This includes both IPv4 and IPv6, but excludes loopback addresses. Additionally, link-local addresses are excluded because they are not available in browsers.

22052: Changes to the HTTPResponse class

Several minor changes have been made to the HTTPResponse class. With the removal of the Publisher API, HTTPResponse class is only of use to implementations of the HTTPServiceHandler interface.

The following changes have been made.
1. A new setBody() overload has been added that accepts a byte[].
2. The setContentRange() method has been removed; this was of no practical use to applications.
3. The deprecated setServer() method has been removed.
4. The generateResponse(boolean headersOnly) overload has been removed; applications should setHeadResponse() to generate a response without a body.

Deprecations in 6.5.0

Client

22262: The MessagingControl feature has been deprecated

The MessagingControl feature has been deprecated and all types and methods have been moved to the Messaging feature. It extends the Messaging feature for compatibility with legacy code.

Java & Android Client

21978: The Stream methods of the sendRequestToFilter FilteredRequestCallback are deprecated

In previous releases, FilteredRequestCallback.onError() was called if a sendRequestToFilter operation failed. In addition, the failure was also notified via exceptional completion of the sendRequestToFilter result. FilteredRequestCallback.onClose() was never called.

From this release, FilteredRequestCallback.onError() is no longer called. Applications should rely on the completable future returned by sendRequestToFilter to detect errors. Both onError() and onClose() methods have been deprecated. In a future release, these methods will be removed from FilteredRequestCallback and it will be changed not to extend Stream.

JavaScript Client

22379: TopicType stateful and functional properties are deprecated

All TopicType properties (id, stateful, functional) have been deprecated. TopicType will be replaced by a TypeScript string enum in the future.

Removals in 6.5.0

Adapters

22150: Push notification bridge support of one way messaging has been removed

Support for one-way messaging has been removed from the push notification bridge. It still supports request-response messaging.

C Client

21944: The remove_topics method has been removed

The deprecated remove_topics function has been removed in this release. Use the topic_removal method instead.

Client

18759: One-way messaging has been removed

The legacy one-way messaging feature has been removed. Use request-response messaging in its place.

20516: TopicControl.removeTopicsWithSession() has been removed

The deprecated TopicControl.removeTopicsWithSession methods has been removed. Instead, use the topic REMOVAL property to remove a topic if a session closes.

21891: Per-session throttling has been removed

Per-session throttling has been removed. Topic views provide a throttle clause which allows for throttled updates of a topic.

Publisher API

21868: Publisher API removed

The Publisher API has been removed in this release. Diffusion publishers are no longer supported and should be replaced with clients. In addition, all aspects of the server configuration deprecated at release 6.2 or before have also now been removed.

Some functions related to server configuration and running Diffusion embedded within a Java application were previously documented alongside the Publisher API and are now referred to as the Server API. For details see https://docs.pushtechnology.com/docs/6.5.0/java-classic/

Fixes in 6.5.0

Apple Client

22699: "(notifier) is nil" NSInvalidArgumentException replaced with NSError

The NSInvalidArgumentException: (notifier) is nil has been replaced with a NSError.

PTDiffusionTopicsFeature methods have been updated to reflect this change.

Diffusion will set the error when it fails to dispatch the notifications.

Console

21592: Load average graph flatlines by default

The Load Average graph shown by default in the Diffusion management console was misconfigured and constantly showed 0 load. The default configuration has been fixed. Users may need to restore the default layout in order for the fix to be applied.

21644: TypeError from console logs tab

In the Diffusion management console, skipping backward or forwards while following the server log could sometimes fail to work and cause errors in the browser console. This has been fixed.

22126: While adding role, adding then removing path permission closes the modal

In the Diffusion management console, removing a topic permission while adding or editing a role would result in the role dialog closing. This has been resolved.

22129: Roles only referenced by principals not listed in roles table (classic console)

The Diffusion management console would not allow configuration or selection of roles which were only referenced by principals in the system authentication configuration. These roles are now displayed with a note and are configurable through the console.

22775: Unknown licence fields cause dashboard failure in classic console

The Diffusion management console could fail to display some status information when running under a licence with MAC address constraints. This is no longer the case.

JavaScript Client

22842: Connection options interpreted incorrectly when credentials field is set to an empty string

Previously, when an empty string was provided for connection credentials it was wrongly interpreted as "no credentials provided". Now an empty string is sent to the server, allowing named principals to connect with an empty credentials string.

Replication

22607: Insufficient permission for <none> to restore topic view

An error of the form "Insufficient permission for <none> to restore topic view" could be logged if repartitioning occurred at the same time as a topic view was being created or removed. This could result in a topic view being lost from the cluster. This problem has now been resolved.

Known Issues

Console

23049: Console shows fractional users connected

The concurrent clients and client connections graphs on the console can display fractional users.