Diffusion Cloud 6.12 Release Notes

DIF-1475: Asynchronous Method for Open in ISessionFactory

OpenAsync in ISessionFactory has been added. It is the asynchronous method for Open which returns a Task.

DIF-1477: Asynchronous Method for SetQueueEventHandler in IClientControl

SetQueueEventHandlerAsync has been added. It is the asynchronous method for SetQueueEventHandler using IQueueEventStream and which returns a Task.

DIF-3665: Ability to Set a Custom Connection URL path

The Dotnet SDK client can now specify a custom URL path to connect to Diffusion through use of the new parameterless ISessionFactory Open and OpenAsync API in conjunction with the new ISessionFactory attributes: ServerHost, ServerPort, SecureTransport and RequestPath.

DIF-3808: .Net SDK compatible with .netstandard2.0

The Dotnet SDK Client is now built with the .Net Standard 2.0 target and can therefore be used by applications targeting both .Net Framework 4.6.1+ and .Net Core 2.0+.

DIF-2754: Session Lock Query API

In this release, there is now a new API available in the ClientControl feature, which enables control clients to query session locks. There is a getSessionLocks method, which returns details of all current session locks, as well as a getSessionLock method to query a specific named session lock.

DIF-3945: Removal of Various Deprecated Items

The following previously deprecated items have now been removed:-

Java:
------
All callback methods and associated interfaces deprecated in 6.7 have been removed.
TopicControl.newSpecification - deprecated at 6.7
ExclusiveUpdaterConflictException - deprecated at 6.6
The createRemoteServer methods in the RemoteServers feature that took individual parameters - deprecated at 6.7
The SERVER_CLOSING value of ClientControl.CloseReason has been removed - deprecated at 6.6

.NET:
------
All callback methods and associated interfaces deprecated in 6.7 have been removed.
ITopicControl.NewSpecification - deprecated at 6.7
The CreateRemoteServerAsync methods in the RemoteServers feature that took individual parameters - deprecated at 6.7

JavaScript:
------------
UpdateFailReason.INVALID_UPDATER - deprecated at 6.6
UpdateFailReason. EXCLUSIVE_UPDATER_CONFLICT - deprecated at 6.6

TopicType.id - deprecated at 6.5
TopicType.stateful - deprecated at 6.5
TopicType.functional - deprecated at 6.5
All internal fields of TopicType have been removed. The TopicType class has been converted to a TypeScript enum.

Apple:
-------
PTDiffusionSessionError - deprecated since 6.7

PTDiffusion
PTDiffusionErrorDomain - deprecated since 6.7
PTDiffusionErrorCode - deprecated since 6.7

PTDiffusionCloseReason
serverClosing - deprecated since 6.6

PTDiffusionMessagingFeature
PTDiffusionMessagingFeatureErrorCode - deprecated since 6.7
PTDiffusionMessagingFeatureErrorDomain - deprecated since 6.7

PTDiffusionRemoteServersFeature
createRemoteServer:withURL:principal:credentials:completionHandler: - deprecated since 6.7
createRemoteServer:withURL:principal:credentials:connnectionOptions:completionHandler: - deprecated since 6.7

PTDiffusionSessionError - deprecated since 6.7

PTDiffusionSessionResponseStreamDelegate
diffusionStream:didFailWithError - deprecated since 6.6
diffusionDidCloseStream: - deprecated since 6.6

PTDiffusionSubscriptionControlFeature
PTDiffusionSubscriptionControlFeatureErrorCore - deprecated since 6.7

PTDiffusionTopicControlFeature
PTDiffusionTopicControlFeatureErrorCore - deprecated since 6.7
removeDiscreteWithTopicSelectorExpression:completionHandler: - deprecated since 6.6

PTDiffusionTopicsFeature
addFallbackStream: - deprecated since 6.5
addStream:withSelector: - deprecated since 6.5
addStream:withSelectorExpression: - deprecated since 6.5

PTDiffusionUpdateStream
PTDiffusionUpdateStreamErrorCore - deprecated since 6.7

PTDiffusionVersion
major - deprecated since 6.5
minor - deprecated since 6.5
patch - deprecated since 6.5

C:
--
metrics
diffusion_session_metric_collector_builder_create_collector - deprecated since 6.8
diffusion_topic_metric_collector_builder_create_collector - deprecated since 6.8

topic-control
ADD_TOPIC_FAILURE_REASON_USER_CODE_ERROR - deprecated since 6.2

topic-update
diffusion_topic_update_create_update_stream - deprecated since 6.9
diffusion_topic_update_create_update_stream_with_constraint - deprecated since 6.9
diffusion_topic_update_create_update_stream_adding_topic - deprecated since 6.9
diffusion_topic_update_create_update_stream_adding_topic_with_constraint - deprecated since 6.9

content_types
content_free - deprecated since 6.1

session_types
SESSION_LISTENER_T.on_handler_error - deprecated since 6.9

selector
selector_match - deprecated since 6.7

DIF-4065: VIEW_SESSION Permissions Requirements Relaxed

Previously, several operations that required MODIFY_SESSION permission also required VIEW_SESSION permission. This has now been relaxed so that for operations that change a specified session's state or the state of a filtered selection of sessions, only MODIFY_SESSION permission is required.
In addition, VIEW_SESSION permission was also required for some messaging operations, but now SEND_TO_SESSION is sufficient.

The affected features are:-
ClientControl
SubscriptionControl
Messaging

DIF-4119: New ClientControl.CloseReason.SESSION_TAKEN_OVER

There is now a new value for the ClientControl.CloseReason enum called SESSION_TAKEN_OVER. This is only relevant to MQTT sessions and indicates that the session has been closed to make way for a new one.

DIF-4201: Methods and Interfaces Deprecated

Deprecated in the Dotnet SDK client:
--------------------------------------
PushTechnology.ClientInterface.Client.Features.Control.Clients.IClientControl.SetQueueEventHandler(IQueueEventHandler)
PushTechnology.ClientInterface.Client.Features.Control.Clients.IQueueEventHandler.OnLowerThresholdCrossed(ISessionId,IMessageQueuePolicy)
PushTechnology.ClientInterface.Client.Features.Control.Clients.IQueueEventHandler.OnUpperThresholdCrossed(ISessionId,IMessageQueuePolicy)
PushTechnology.ClientInterface.Client.Features.Control.Clients.IQueueEventHandler
PushTechnology.ClientInterface.Client.Features.Control.Clients.ISessionPropertiesListener.OnSessionClose(ISessionId,IDictionary<string,string>,CloseReason)
PushTechnology.ClientInterface.Client.Features.Control.Clients.ISessionPropertiesListener.OnSessionEvent(ISessionId,SessionPropertiesListenerEventType?,IDictionary<string,string>,IDictionary<string,string>)
PushTechnology.ClientInterface.Client.Features.Control.Clients.ISessionPropertiesListener.OnSessionOpen(ISessionId,IDictionary<string,string>)
PushTechnology.ClientInterface.Client.Features.Control.Clients.ISessionPropertiesListener
PushTechnology.ClientInterface.Client.Session.ISessionFactory.Open(string,ISessionOpenCallback)
PushTechnology.ClientInterface.Client.Session.ISessionFactory.Open<TContext>(string,TContext,ISessionOpenContextCallback<TContext>)
PushTechnology.ClientInterface.Client.Session.ISessionOpenCallback.OnOpened(ISession)
PushTechnology.ClientInterface.Client.Session.ISessionOpenCallback
PushTechnology.ClientInterface.Client.Session.ISessionOpenContextCallback<in TContext>.OnOpened(TContext,ISession)

Deprecated in the Apple client:
--------------------------------
tidyOnSubscribePropertyKey is now deprecated in favour of tidyOnUnsubscribePropertyKey due to an error in the property name.

Deprecated in the JavaScript Client
-------------------------------------
The `Result` type was an alias of the native `Promise` type. It has been deprecated in favour of `Promise`.

DIF-1309: Session locks - ability to view per session locks on Console

The Diffusion management console now provides a facility for viewing a summary of the session locks currently held in the cluster. (VIEW_SERVER permission is required for this feature to be displayed)

DIF-3434: Allow partial update of gateway configurations

Following the release of Gateway Framework 2.3.0, gateway clients can accept partial configuration changes. The management console will no longer prevent the user from submitting new gateway client configurations which do not provide all schema-required fields.

DIF-4206: Management Console Refinements

Various small improvements have been made to the Diffusion management console, including improvements for setting time series range properties.

DIF-635: Open and Connected Session Counts on the Console Session Browser

The Diffusion management console now displays the numbers of open and connected sessions on the Open Sessions page. Disconnected sessions in the table now have an icon next to their name indicating their status.

DIF-302: Server and Client have dependency on Java 17

From this release, both the Diffusion server and the Java Client have a dependency on Java 17. Older clients running on earlier Java versions will still be able to access the new server, but to benefit from all the new features, the 6.12 Java Client will be required.

DIF-4362: Packaged Script Changes

The shell scripts packaged with the Diffusion installer have been updated for Java 17.

DIF-4123: Client jar Size Reduced

The size of the diffusion-client jar file has been reduced.

DIF-2752: Support for Safari Browser

The JavaScript Client and Console now support the Safari Browser.

DIF-3251: Messaging API consistency with other SDKs, added documentation and type hints

This change adds accurate type hints to the RequestHandler and the callback passed into its constructor.

It is unlikely to break customer code as it simply add documentation and accurate type hints, but is marked as a Functional Change due to the following corner cases where existing code could break:

1. If there is runtime type checking attempting to validate something type annotated, but insufficiently precisely or incorrectly type annotated, against the callback type expected by the RequestHandler constructor.
2. Static type checking such as MyPy/PyRight etc could complain in the same circumstances.

DIF-3628: Fix Python Datatype Type Hints and Hierarchy

This change introduces accurate type annotations and docs for JSON subtypes.

This should not break existing code in general, but there are a couple of corner cases meaning this is being released as a Functional Change:

# There is the corner-case of legacy end-user code using runtime type introspection and this breaking that.
# It may also break static type checking for end-user code that's incorrectly or insufficiently precisely typed.

The solution in both cases would be to correct the end user code to be more precise. Failing that one can suppress type warnings in the end user code although this is undesirable.

Neither untyped code (including code with items marked as `typing.Any` that refer to JSON subtypes), nor code that is already sufficiently precisely typed should be affected.

DIF-3795: Type Annotate Handler API Clarifications

Due to the previously relaxed nature of type annotations on ValueStreamHandler and its callbacks, a callback could be passed that would never be triggered, and type checking would not pick this up.

This change provides corrections and precise signatures and type annotations for these callbacks, which means that the signature and name of the callbacks are much clearer.

Note that GitHub examples were previously passing a ‘value’ handler rather than an 'update’ handler; they have been fixed to conform to these new type annotations.

The semantics and behaviour of the code and any end-user code should be unchanged, except in a few unlikely corner cases.

1. Runtime type validation is added by the end user who checks whether something is a valid Callback.
2. The end user's workflow is dependent on type checking, e.g. MyPy, etc, and callbacks passed into ValueStreamHandler are explicitly annotated with an incompatible type annotation.

DIF-2440: SELECT_TOPIC Permissions Changes

Change to Topic Selector Permissions

Starting with this release, the SELECT_TOPIC path permission is required for all paths that a topic selector may match, not just the path prefix.

Previous Behaviour

In earlier versions, the SELECT_TOPIC permission was only required for the prefix of a topic selector. A session could use a selector to access an entire branch of the topic tree as long as it had permission at the top-level path. As a result, it was not possible to selectively remove access from a sub-branch — access to the parent path implicitly granted access to everything below it.

For example, if a session had SELECT_TOPIC permission for a/b, but not a/b/c, it could still use a selector like ?a/b// to fetch a/b/c/d.

New Behaviour

The permission model has been made more restrictive. To fetch or subscribe using a topic selector, a session must now have SELECT_TOPIC permission for each individual path that the selector may match. This allows SELECT_TOPIC to be granted to a branch of the topic tree and explicitly revoked for specific sub-branches.

The model is even more restrictive for full path pattern topic selectors: SELECT_TOPIC permission is required for all paths at and below the path prefix. This is necessary to prevent circumvention using advanced regular expressions.

Impact

This change enables more granular and secure control over topic access when using selectors, improving both flexibility and protection in permission management.

DIF-2747: Session Authentication - Expiry

This release introduces a new session property called "$ExpiryTime" which can be set by an authenticator and if set will cause the session to be automatically closed by the server when the specified time is reached. The expiry time is specified in milliseconds since the epoch. If it is not set, the session will not expire.

The client session can reauthenticate before it expires using the new 'reauthenticate' method provided in the Security feature. This replaces the changePrincipal method (deprecated at this release) as it can be used to change the principal, or reauthenticate using the same principal. It also allows session properties to be proposed. The session properties may be the same as when supplied when the session was opened, or can be different.

The client session itself now has a new "getSessionProperties" method, which allows it to get the values of its own fixed session properties, and thus determine when it is due to expire.

The AuthenticationControl feature also has a new "revokeAuthentication" method, which allows a session with sufficient privilege to revoke the authentication of a client session, causing its immediate closure.

DIF-2632: Improvements to Client Queue Flushing Performance

If a peer is slow to accept network data and the backlog of messages on the sender has reached the configured maximum queue size, the sender will prioritise the sending of network data to the peer. If the peer still does not accept the data are a short period, the connection will be failed. From this release, this processing has been improved to reduce the number of network writes and increase the grace period.

DIF-2782: Topic selector optimisations

The processing of subscriptions and subscriptions to topic selectors has been optimized to reduce CPU compared to previous releases.

DIF-3500: SubscriptionControl Feature now Cluster Aware

In previous releases, the SubscriptionControl feature methods to subscribe and unsubscribe sessions would only work for sessions connected to the same server as the control client using the feature. This meant that when Diffusion was running in a cluster a control client would need to connect separately to all servers in the cluster in order to manage all sessions. At this release, all methods in the SubscriptionControl feature apply to all sessions across a cluster.

DIF-4063: DSL Parsing Error Improvements

Previously, the error messages returned or logged from the parsing of various Diffusion DSL specifications (topic views, topic removal, session filters etc.) were very difficult to understand as they rendered internal token keys to indicate what the parser expected. These messages have now been improved so that what was expected is described more clearly.

DIF-11: Metric Alerts

At this release, there is a new Metrics Notifications feature which allows metrics alerts to be notified to topics when certain criteria are satisfied. For example, an alert could be set up which will write to a topic when the memory utilisation exceeds a specified threshold.

Alerts can be configured using the Java or .NET SDKs (within the Metrics feature) or within the Diffusion Management Console.

See the user manual and API documentation for further details.

DIF-1635: Topic View 'insert's now Allowed at any Point in the Specification

In previous releases topic view 'insert' clauses could only occur at the end of the topic view specification. This restriction has now been lifted so that 'insert's can be interspersed with other topic view transformations.

DIF-1922: Regular Expression Matching for String Comparison in Topic Views

There is now a new "matches" or "=~" operator available in topic view "process" conditionals. This allows for string comparisons using regular expressions.

DIF-323: New Extended Functionality in the 'set' Operation of the Topic Views 'process' Clause

At this release, the 'set' operation in the Topic Views process clause has been enhanced in several ways in order to satisfy additional use cases.

1) It is now possible to empty structures in the output.

Previously, when there was a large amount of input data and the view only required a small number of items to be output, it was very difficult to remove those items that were not required. It is now possible to clear output structures (objects or arrays, even the root) and then copy only the required items (optionally renaming them) to the output data.
An output structure can be cleared by using some new special values in the 'set to value' variant of the 'set' operation. These are 'empty_object' and 'empty array' (shorthand $O and $A respectively). So, to clear the whole data structure, you could use set(, $O) or to clear a specific data structure, you could use set(/pointer, $O).
You can then follow this (in the same process) by copying selected items into the structure as described in 2 and 3 below.
This can also be used to create empty parent structures, which can be populated in any way you require.

2) It is now possible to copy items to the same location in the output.

The previous 'set to scalar' variant only catered for copying a scalar value to one at a different pointer.
The new 'set to pointer' capability (see 3 below) allows the item (scalar or structure) to be copied with the same pointer, which is useful when a structure has previously been cleared as described in 1 above. Rather than specifying the same pointer twice, if you specify a single pointer only, it will copy to the same pointer location (e.g. set(/pointer)).

3) It is now possible to copy data structures, as well as scalar values.

Previously, the 'set to scalar' variant of the set operation only supported setting a scalar item in the output from a scalar item in the input.
The 'set to scalar' variant has now been extended/renamed to be the 'set to pointer' variant, so the item being set can be an object or an array, as well as a scalar value.
This means that whole structures can now be moved from the input to the output. This is mainly useful if a higher-level structure has been cleared (see 1 above), but also allows for copying to entirely different structures.

For full details, see the Topic Views page in the user guide.

DIF-3438: Topic Selection Scopes

At this release there is a new feature, known as 'Topic Selection Scopes'.

Previously, if two different components of a client application sharing a single Diffusion connection subscribed to a topic and one of them later unsubscribed, the topic would be unsubscribed for the session and thus for all components. This is often not what is required as the other component may still want the topic.

Now, it is possible to specify a named scope when subscribing, such that a later unsubscription specifying that scope has no effect on other subscriptions that named a different scope. This allows subscriptions to be localised to the components that need them.

There are now new subscription and unsubscription methods that take a 'scope' parameter to support this. Applications using the old methods will continue to work as before, as their subscriptions will all be assigned to a 'default' scope. An application that uses scopes should specify a named scope to all subscribe and unsubscribe calls.

NOTE: This feature is not available in the Python SDK.

For more information, see the API documentation and the User Guide.

DIF-226: Incorrect handling of regular expression quotes in topic selector expressions

Split-path pattern and full-path pattern topic selectors allow the use of regular expression quotes (\Q, \E). In previous releases, the quotes were handled incorrectly: any backslash (\) character between the quotes was interpreted as the end of the literal section.

This bug has been fixed in this release. Now everything between \Q and \E is correctly treated as literal characters.

DIF-4368: Corrupt Data when Using UpdateStreams

The Dotnet SDK client did not store messages sent to the client during the reconnection phase. Therefore on successful reconnection these messages were lost. This bug has been fixed in this release.

DIF-2538: API_ERROR_DIFFUSION_VALUE_INCORRECT_DATATYPE when reading double value

When using read_diffusion_double_value to read a DIFFUSION_VALUE_T containing a double value, it would return API_ERROR_DIFFUSION_VALUE_INCORRECT_DATATYPE. This issue has now been fixed.

DIF-3850: Memory leaks in C SDk

Several internal memory leaks in the C SDK have been detected and fixed.

DIF-4301: Transport Loop Not Breaking on Invalid Host

When providing an invalid hostname to the C SDK the transport layer could hang indefinitely when closing. This issue has now been resolved.

DIF-4166: Management Console Issues

Refinements have been made to the Diffusion management console in the following areas:
- In some circumstances an error could occur when attempting to save a new topic view specification without providing a name. Validation is now refined.
- An error would result when specifying an empty JSON pointer within a topic view specification 'process' clause (i.e. in order to set the whole value of a JSON topic). This is now supported.
- When creating a new topic subscription using the topic browser, a confirmation message was not always displayed. Now corrected.

DIF-4244: Console Topic View Property Override Editor Exhibits Strange Behaviour

When using the graphical topic view editor, changes could fail to apply correctly when attempting to remove property overrides. This now works correctly.

DIF-2727: Spurious Reverse Remote Server log messages - PUSH-000882 and PUSH-000888

When running in a cluster the message PUSH-000882 (indicating that a server has become the master for primary initiator remote server connections) was logged even if no such remote servers are configured.

In addition, PUSH-000888 (indicating that the server is no longer the master for primary initiator remote server connections) even if the server was not the master or if it has no primary initiators configured.

This has been resolved so that these messages are now only logged in situations where they actually apply.

DIF-2731: Potential Deadlock when using Remote Server Reverse Connections

A thread deadlock could occur between RemoteServerManagerImpl and DefaultPicoContainer when closing a server that has remote server secondary acceptors defined and a primary server tries to connect to it at the same time. This has now been resolved.

DIF-2831: Possible thread deadlock during server startup or closedown when remote servers are in use

In rare circumstances, a thread deadlock could occur when starting or closing a server when remote servers are in use. A thread dump will show a deadlock between the class DefaultPicoContainer and one or more other remote server classes. This has now been resolved.

DIF-998: Problems with Propagation of Missing Topic Notifications over Remote Server Connections

Missing topic notifications were being propagated from a secondary server to a primary server even if the subscription selector used did match topics on the secondary. A notification should only be sent to the primary server if the selector matches no topics on the secondary server. In addition, when a notification was received on the primary server it would only be propagated onwards if the selector matched no topics on the primary server, where it should be propagated regardless of any matches on the primary server. These problems have been resolved in this release.

DIF-3985: Details of the Passive Gateway Clients are not Available After Session Re-creation

After a server restart, the details of the passive Gateway clients are lost. Gateway clients must re-validate so the server can restore details of the passive clients. This re-validation is now required for the Gateway clients to maintain the correct state in the server. The fix also addresses an issue where missing passive client details caused false validation errors.

To resolve this issue for adapters, they must be updated to use the latest version of the Gateway Framework, which includes the 6.12 version of the Java client SDK, and should be used in conjunction with the 6.12 version of the Diffusion server.

DIF-4223: Passive Gateway Client Validation Fails when JSON Configuration is Logically Equivalent but Structurally Different from the Active Client

When a passive Gateway client submits a validation request with a hash of its identifier that includes a JSON configuration containing logically similar but structurally different content compared to that of the active client, the validation would fail. This issue is fixed in this version of the server.

To resolve this issue for adapters, they must be updated to use the 2.4 version of the Gateway Framework, which includes the corresponding fix on the framework side.

DIF-2740: Deadlock when using RecoverableUpdateStream and set fails

In previous versions a deadlock could occur when using recoverable update streams. This has been resolved at this release.

DIF-4081: Misleading SessionDisconnectedException Reported

Previously, when a session failed over to another server, handlers would be discarded, reporting the misleading internal SessionDisconnectedException. This has now been changed to report a ClusterRoutingException, indicating that an operation has been cancelled because the session has connected to a different server.

DIF-4346: Java Session Does Not Wait for the Server to Acknowledge Session Following Failover and Reconnection

The Session.close() operation waits for a short grace period (default 1 second) for the server acknowledgement before shutting down the connection, improving the likelihood that the closure notification has been delivered to the server and the server has correctly recorded the reason for the session closure. Due to a bug in previous releases, the grace period was not applied if the session was closed following fail over and reconnection. This has been fixed in this release.

DIF-3135: The roles parameter in SystemAuthenticationScriptBuilder.allowAnonymousConnections should be optional

While the `roles` parameter in `SystemAuthenticationScriptBuilder.allowAnonymousConnections(roles)` is specified as optional, calling the function without parameters resulted in an exception. This has been fixed, and the `roles` parameter can now be left undefined, defaulting to the empty array, as specified in the API documentation.

DIF-3702: ErrorReportList is stringified incorrectly

The toString() function of ErrorReportList was not implemented. This resulted in a messages containing '[Object]'. The function has now been implemented to return a meaningful message and errors based on ErrorReportList contain more meaningful information.

DIF-4082: Javascript docs are not built properly

The documentation for the JS client API was not generated properly. This has been fixed.

DIF-4517: Passing SessionId Object into Functions that take SessionId or String Fails in the Modular Client

Functions such as ClientControl.close may take a SessionId or a string as parameter. When using the modular JS client, these functions sent the wrong data to the server when a SessionId was passed. This has been fixed and SessionId instances can be passed successfully.

DIF-2864: Usage Provider connected to Usage Collector, but not sending usage snapshots

Under certain circumstances, a Usage Provider could stop sending data to a Usage Collector even though it was connected. In addition, errors could be seen in the server log when closing a Usage Provider. These problems have been resolved in this release.

DIF-2875: Usage Collector start-up configuration

A configuration bug in the Usage Monitor service has been patched at this release.

DIF-3978: Metrics within a Usage Snapshot can be Affected by Diffusion Metric Collectors, Causing the Wrong Value to be Captured

Previously, metrics within a usage snapshot could be affected by Diffusion metric collectors, causing the wrong value to be captured. For example, a mismatch between the actual number of diffusion_sessions_open and the number recorded in the usage snapshot.
Usage monitor metrics are now correctly filtered in/out based upon the presence of labels.

DIF-4647: Usage Providers Fail to Send Snapshots After a Collector Restart

Under certain circumstances, a Diffusion Server acting as a Usage Provider could fail to send any more snapshots after reconnecting to a Usage Collector Server. This problem has now been resolved.

DIF-2922: Topic removal operations may complete before removals have been persisted to file

Due to a bug in previous releases, if a server was configured for file persistence, topic removal operations could complete before the being recorded in the file store. If the server failed and was restarted, the topics could be incorrectly restored. The bug has been fixed in this release.

DIF-3473: The self-signed sample certificate has expired

The sample self-signed certificate packaged in previous releases expired on 16th January 2025. In this release, the certificate has been refreshed with a new certificate that expires on 18th January 2035. The installer places the certificate in etc/sample.keystore. The certificate is not secure and should be replaced in production installations with a customer provided certificate.

DIF-2550: 'no running event loop' due to API breakage in aiohttp

Fixes the 'no running event loop' issue due to breaking behavioural changes backported into aiohttp 3.10 (see https://github.com/aio-libs/aiohttp/issues/8555).

DIF-3217: Non Messaging APIs are not Concurrency-safe

This fix migrates internal serialisation to use concurrency-safe methods.

DIF-3258: Messaging API is not Concurrency-safe

The Messaging APIs were previously not entirely safe for concurrent calls. This is resolved in this release.

DIF-3380: Pydantic 1.10.20 breaks ForwardRefs on Python 3.12.4 (ForwardRef._evaluate() got an unexpected keyword argument 'type_params')

For Pydantic V1 users, there was a regression in Pydantic 1.10.20 as described here: https://github.com/pydantic/pydantic/issues/11230

This change blacklists this version with the expectation that the regression will be fixed in an imminent Pydantic 1.x release.

DIF-3604: Structlog Configuration is Overriden by Client, and Exceptions are not Logged Correctly

This fix prevents Diffusion from overriding structlog settings. It also logs exceptions correctly according to the structlog API.

DIF-3697: Messaging Callback Code Does Not Correctly Decode SessionID Components as an Unsigned Int64

The `session_id` passed into messaging callbacks was sometimes incorrectly decoded. This is now resolved. The undocumented 'conversation_id' parameter has also been removed. If you previously had a 'conversation_id' non-optional parameter before, you will need to remove it or make it optional - it will not be set in future.

DIF-3844: ValueStreamHandler being Triggered when Selector Should Not Match

ValueStreamHandlers were being triggered regardless of their associated topic selector when no topic-specific selector was available - effectively, all datatype-compatible ValueStreamHandlers were being treated as fallback handlers. This has now been resolved.

Please check that any ValueStreamHandlers are associated with the intended topic selectors. This brings the behaviour into line with the other clients.

DIF-4005: Client Does Not Return Error to Server on Messaging Callback Failure/Rejection

Messaging callback errors were being swallowed and logged to the console rather than sent to the server, meaning that the server was waiting indefinitely for a response.

This change catches any errors raised from Messaging callbacks and returns them to the server, where they will be passed in turn to the initiator of the request.

DIF-2887: Possible topic loss if cluster topology changes while a server is recovering file persisted topics

Due to a bug in previous releases, topic data could be lost if the cluster topology changed while a server was recovering file persisted topics. The bug has been fixed in this release.

DIF-2918: In a topic-replicated cluster, some topic removals may fail to be persisted to file

Due to a bug in previous releases, in a cluster configured for topic replication and file persistence, some topic removals were not recorded in the file store leading to the topics being incorrectly restored if the cluster was fully restarted. The bug has been fixed in this release.

DIF-3034: Warnings Logged when Hazelcast is Shut Down

Unnecessary HazelcastInstanceNotActiveException messages can be logged when a server is stopped if they are not individually handled by Diffusion code. In this release, further instances have been suppressed.

DIF-3155: Inconsistent Topic State Across the Cluster Following Replication Failure

Due to a bug in previous releases, failure of a topic replication operation could leave a server in an inconsistent state. The problem has been fixed in this release.

DIF-3235: Replicated time series topic memory leak

In previous releases, expired events were not properly removed from replicated time series topics, causing a server memory leak. The problem has been fixed in this release.

DIF-3327: Failure to Replicate Initial Value when a File-recovered Replicated Topic is Reused

Due to a bug in previous releases, the value of a replicated topic recovered from file could be incorrectly replicated when a cluster is first started. The bug has been fixed in this release.

DIF-3350: Assertion failure when recovering a replicated time series topic from file

Due to a bug in previous releases, a server could with an assertion error when recovering a replicated time series topic from file. The bug has been fixed in this release.

DIF-3404: StackOverflowError during cluster repartitioning

In previous releases, failure to create a replicated configuration item such as a topic view due to the cluster topology changing could cause a stack overflow error to be logged. From this release, the operation will fail cleanly with a "cluster repartitioning" error.

DIF-3419: File restoration unnecessarily logs about failure to replace a replicated topic

In previous releases, PUSH-000679 log messages could be unnecessarily logged during server start up if a file-recovered topic was already replicated. The problem has been fixed in this release.

DIF-4067: Incorrect Replication of Recovered Time Series Topics

Due to a bug in previous releases, if a server joined or left a cluster, time series events could fail to be properly replicated. The problem has been fixed in this release.

DIF-4103: AccessDeniedException Preventing Claim of Hazelcast Partition

In extremely rare recovery situations on Diffusion Cloud, an AccessDeniedException could occur and cause Hazelcast partition 0 not to be recovered. This would be accompanied by a message of the form 'The metric collector "x" is reserved for system use and cannot be modified.' This could lead to possible data loss from topics in the partition. This problem has been resolved in this release.

DIF-4032: TopicNotifications Feature Notifies Topics that the Caller does not have READ_TOPIC Permission for

In previous releases, the Topic Notifications feature notified the listener of all topics that satisfy the specified selector, even though the documentation states that the listener will only receive notifications for topics for which it has READ_TOPIC permissions. This has now been resolved, and the listener will only receive notifications for topics for which it has READ_TOPIC permissions.

DIF-4497: Authenticators are Passed the $Environment Property in the Proposed Properties Map

The $Environment session property introduced in 6.11.0 was being passed in both the sessionProperties and proposedProperties maps which are passed to an authenticator. Its presence in the proposedProperties map is misleading as it cannot be changed by the authenticator. It is now no longer passed in proposedProperties.

DIF-2832: Spurious PUSH-000773 multiplexer slow progress warnings

Spurious PUSH-000773 Multiplexer slow progress warnings could be logged with mostly idle servers, from this release this is fixed

DIF-2895: Deadlock in Server when using Throttled Topics

A thread deadlock between classes ThrottlerImpl and DerivedReferenceTopicImpl could occur when using throttled topics. This has now been resolved.

DIF-2907: Poor topic recovery performance in a cluster

In previous releases, replication of a large number of low priority topics could take an unnecessarily long time and cause multiplexer slow progress warnings to be logged. The problem has been fixed in this release.

DIF-3016: Memory leak due to never-ending retry of replicated topic removal

In previous releases, if a cluster failure caused removal of replicated topics to be aborted, the operation would be repeatedly rescheduled by the coordinating server. If the cluster failure remained unresolved, the Java heap memory of the coordinating server would eventually be exhausted. In this release, the rescheduling code has been re-written to have a constant memory footprint and to eventually abort the removal operation if the cluster does not recover.

DIF-3056: Deadlock in DiffusionMetricRegistryImpl during Server Shutdown

In very rare circumstances a thread deadlock could occur between the classes MeasuredEntityClassMetricsCollator and DiffusionMetricRegistryImpl. This problem has now been resolved.

DIF-3630: SubscriptionControl methods do not validate SELECT_TOPIC permissions correctly

The SubscriptionControl feature documents that the caller must have SELECT_TOPIC permissions to the path prefix of the selector being used to subscribe other sessions. However, in previous releases, this permission check was only being applied when subscribing using a filter and not when subscribing a single session. This has been resolved in this release.
In addition, the same SELECT_TOPIC permission check should have been applied when unsubscribing sessions. From release 6.12, the same checks will be applied when unsubscribing sessions.

DIF-4487: Zero Length Value Dumps are Produced, and Dumps Directory is Never Tidied

When an invalid CBOR value is passed as an update to a topic, the value is dumped to a file in the persistence/dumps directory. However, this occurred even if the value was empty, which is pointless. Additionally, the number of these dump files can grow indefinitely, potentially leading to file resource failures. At this release, if an empty value is passed, a warning is logged, but no dump file is produced. In addition, there is now a tidying mechanism that ensures that when there are more than 50 of these dump files, any that are more than a week old will be removed.

DIF-4531: A Message Larger than the Configured Queue Byte Limits Causes the Server to Spin Forever

Due to a bug in previous releases, if the server is configured to limit the number of bytes that can be queued for a session or a connector, sending a message larger than that limit could cause the server to spin for ever. The bug has been fixed in this release.

DIF-4558: SessionEventStream Receives All Sessions as ACTIVE During the Initial Burst, Even if Some Were Disconnected

Previously, the session event listener could report sessions that were currently disconnected as being active during the initial notification of current sessions. This has now been resolved.

DIF-3499: Unable to Set the Whole Value of a Topic Using the Process Clause

In previous releases, it was not possible to specify an empty JSON pointer (as permitted by the JSON Pointer Specification RFC6901) within the Topic Views 'process' clause (would fail to parse the specification). This meant that it was not possible to set the whole output value to a specified value. This has now been resolved so that leaving the pointer parameter as empty indicates the whole JSON document - for example set(,"a string").

DIF-3534: Deadlock in Diffusion Server when using Delayed Topic Views

In previous releases, there was a possibility of a thread deadlock between UniversalTopicImpl and DerivedReferenceTopicImpl when using delayed topic views. This has been resolved in this release.

DIF-3762: Potential Deadlock Between UniversalTopicImpl and ScalarDependenciesImpl

In rare situations, a thread deadlock could occur in the Diffusion server between objects of type UniversalTopicImpl and ScalarDependenciesImpl. This has now been resolved.

DIF-3855: Topic Views Using Scalar Path Indexes are not Re-Evaluated after being Replaced

There was a problem whereby if a topic view was replaced by a new view which differed only in the parameters of a scalar directive, when either the old or new view used the second parameter of the directive, the new view would not be evaluated. This has been resolved in this release.

DIF-2640: Redundant value updates can be published to subscribers during recovery

In some circumstances, during cluster recovery, the same value of a topic could be published twice. This has now been resolved.

DIF-2767: Subscriptions are unnecessarily re-evaluated on reconnection

Previously, on reconnection, a session's subscriptions were always being re-evaluated even if the session's properties had not changed. This led to an unnecessary CPU load on the server. This has now been resolved - subscriptions are only re-evaluated if the session's properties change during reconnection.

DIF-2781: Topic set selectors are incorrectly combined with existing selectors

In previous releases, set selectors were incorrectly merged with existing selectors, potentially incorrectly subscribing sessions to topics. This could also have a serious performance impact when using set selectors. The bug has been fixed in this release.

DIF-2866: NullPointerException in multiplexer thread

Due to a bug introduced in 6.10, adding topics that are replicated across a cluster was incorrectly coordinated with adding or modifying topic views. This could cause corruption of the servers internal state, and lead to NullPointerExceptions. The bug has been fixed in this release.

DIF-2994: ClassCastException when performing a Time Series Range Query on a non existent Topic

If an attempt was made to perform a time series range query on a topic that does not exist but where there were topics below it in the tree would cause a ClassCastException to be reported in the server log and no response returned to the client. For example, requesting a range query on a non-existent topic called 'Test' where a topic called 'Test/Topic' exists would result in this failure. This has been resolved at this release.

DIF-3359: Automatic Topic Removal with an "or" operator in the expression does not respect one of the clauses if another is "subscriptions less than" AND there are subscriptions to the topic

In previous releases, if an automatic topic removal clause specified 'subscriptions less than' with an 'or' operator where the or'd expression is satisfied AND the topic in question had subscribers the topic would not be removed. This was because the 'subscriptions less than' expression was wrongly taking priority over the satisfied 'or' expression. This problem has now been resolved.

DIF-3585: Some unsubscriptions are incorrectly discarded

Due to a bug in previous releases, some unsubscription requests were ignored. This could occur for an unsubscription request with a unique selector that has the same path prefix as a previous subscription. The bug has been fixed in this release.

DIF-2527: Hazelcast Warning messages on Windows and Mac

Hazelcast Warning

Hazelcast provides clustering support within Diffusion; with this release of Diffusion, Hazelcast - 5.3.x is used. This newer version of Hazelcast adds options for per-socket keep-alive configuration. When using particular operating systems (typically Windows/MacOS) and mostly older JVM combinations, log messages are output by Hazelcast warning:

* "Ignoring TCP_KEEPCOUNT. It seems your JDK does not support jdk.net.ExtendedSocketOptions on this OS. Try upgrading to the latest JDK or check with your JDK vendor.Alternatively, on Linux, configure tcp_keepalive_probes in the kernel (affecting default keep-alive configuration for all sockets): For more info see https://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/. If this isn't dealt with, idle connections could be closed prematurely."

This type of message is then repeated for: "TCP_KEEPIDLE" and "TCP_KEEPINTERVAL".

Hazelcast have documented the warning and suitable configuration, please refer to: https://docs.hazelcast.com/hazelcast/5.3/clusters/network-configuration#configuring-tcp-keep-alive

DIF-166: Reference Topics Retained by 'preserve topics' are not Persisted Across Server Instances or Clusters

A new 'preserve topics' clause was introduced to topic views in release 6.6.

This clause means that reference topics created by a view (that have a path dependent upon the source topic value) are retained until the source topic is removed or the topic view is removed.

Though this is true in the context of a single server instance, it is not the case if the server is restarted as all such topics created during the previous server instance will be lost. It is also not the case if a new server enters a cluster as the new server will only have reference topics generated from the point in time where it joined the cluster and will not reflect reference topics previously created within other cluster peers.

This issue occurs because reference topics are not persisted, either to file or across the cluster.

DIF-167: Restrictions on Mapping Single Value Topics to Time Series Reference Topics

The ability to specify a target type in a topic view was introduced in release 6.7.0.

When using this feature to map a single value topic to a time series topic there are the following restrictions.

1) The retained events in the target topic are not replicated across the cluster (as reference topics are not replicated). This means that a new server joining the cluster will not have the same number of retained events as other cluster members. For this reason mapping single value to time series topics should not be used in a clustered environment.

2) Retained events in the target topic are not persisted therefore when a server is restarted the target time series will initially start with a single event and will only grow as the source topic is updated.

The ability to map single-value topics to time series topics is deprecated at release 6.11 and an error message will be logged if used. If used in a single server a warning message will be logged.

DIF-2482: The Memory Overhead of Remote Topics is not Accounted for in Metrics and Fetched Topic Sizes

Remote Topic Views maintain local caches of the values of the topics from the primary servers, however, this memory overhead is not accounted for in the metrics relating to topic sizes and memory. This means that when using remote topic views, there is an 'invisible' memory overhead which amounts to the size of each topic selected by the remote topic views from the primary server.

Furthermore, from release 6.11 it is also possible to 'fetch' the size of topics. For reference topics that are mapped from remote topics, a similar problem to the above occurs. A reference topic that is directly mapped from a remote topic will report its size as zero because under normal circumstances the size of such a reference topic is accounted for in the source topic. Similarly, a reference topic that is indirectly mapped from a remote topic will report its value size but will not consider the additional size of the source topic. The reference topic cannot simply add the size of the source topic as many reference topics can be mapped from the same remote topic and that would result in over accounting for the sizes.

Both of the above issues will be resolved in a future release.