What is the Diffusion journal?

The Diffusion journal allows certain 'actions' to be written out to a log file. In order to use the journal, some configuration steps are required - these are detailed in the sections below. Once the journal has been enabled and configured, it will write to a log file in JSON format. The data in the log file can be viewed or ingested into another system as required.

For example, you could log details every time a client session connects, or every time a session creates a topic.

The actions that will be logged are determined by the settings in the journal configuration file. An example of this file is provided in the ./etc directory of the Diffusion installation, with all actions disabled. The journal service is enabled within the Server.xml configuration file in the ./etc directory.

Journal format

The journal writes details of the actions to a file in JSON format. Below is an example of a log entry:

{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"path":"topic-123", "type":"JSON", "properties":{}}},"action":"TOPIC_ADD","timestamp":1644853378421}

Log entry contents

A journal log entry has the following parts: session, action, success, context, and timestamp.

session

Details of the connected session performing the action.

Example:

{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"}

action

This is the type of action being performed.

Example:

TOPIC_ADD

success

Indicates if the action was performed successfully true or failed false.

Example:

true

context

As its name implies, this is the 'context' of the action being performed. The values contained within the context are specific to the action.

Example:

{"path":"topic-123", "type":"JSON", "properties":{}}

timestamp

The time that the action was performed in milliseconds since the epoch.

Example:

1644853378421

How to configure the journal

This section details the steps required to enable the journal and configure it so that it logs to a file. Out of the box, the journal is disabled - this can be seen in the below example log message:

2022-02-14 14:28:49.741|INFO|main|PUSH-000866|||Journal not enabled.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider

Configuring the journal requires the following steps:

• Enabling and configuring the journal feature.
• Configuring which actions to journal.
• Configuring the logger. Internally, the journal uses Log4J2 which allows control over the log file rotation and housekeeping of log files.

Enable and configure the journal

The journal is enabled and configured within the journal block of the Server.xml file in the ./etc directory. It is possible to configure the location and/or name of the journal configuration file. By default the file will be read from a file called journaling-status.properties in the ./etc directory. It is also possible to configure which session properties of the session calling the action are to be logged. By default no session properties are logged. To simply enable the journal with the default settings the Server.xml entry can be as follows:

<journal enabled="true"/>

<journal enabled="true">
	<config-directory>journal</config-directory>
	<config-file>journal.properties</config-file>
	<session-properties>$fixed</session-properties>
</journal>
            

Once the journal has been enabled and Diffusion started, the start-up log message will indicate that the journal has been enabled - however, if it hasn't been configured with any actions to journal - as can be seen, the configured actions is zero:

2022-02-14 14:31:43.814|INFO|main|PUSH-000866|||Journal is enabled.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider
2022-02-14 14:31:43.814|INFO|main|PUSH-000871|||Journal enabled with formatter: 'json', actions configured to be journaled: 0.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider

Configure actions to journal

Actions to be journaled must be specified in the journal configuration file (defaulted to journaling-status.properties in the ./etc directory), the format of the file is a standard Java properties file containing key and value pairs. The example file below demonstrates a selection of actions being enabled for journaling.

TOPIC_ADD=true
TOPIC_REMOVAL=true
CREATE_TOPIC_VIEW=true
REMOVE_TOPIC_VIEW=true
ADD_AND_SET_TOPIC=true
CREATE_UPDATE_STREAM=true
CREATE_UPDATE_STREAM_AND_SET=true
CLIENT_CONNECTION_ACTION=true

The issued properties file has the full list of actions that can be specified. Any actions not in the file will not be journaled.

Once the properties file has been configured the journal will output a message similar to the below indicating that there are actions which have been enabled for journaling:

2022-02-14 14:34:14.678|INFO|main|PUSH-000866|||Journal is enabled.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider
2022-02-14 14:34:14.678|INFO|main|PUSH-000871|||Journal enabled with formatter: 'json', actions configured to be journaled: 8.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider

Configure logger

Internally the journal feature uses Log4J2 (referred to as Log4J in this document). Log4J is very powerful and can be configured in a variety of ways, such as the desired log message format, the log rotation policy and also the log housekeeping approach, such as zipping log files into an archive directory.

The log4j2.xml file in the Diffusion ./etc directory contains three commented out sections labelled "Journal JSON Configuration: part n of 3". Simply uncommenting these sections will enable the JSON logging for the Journal service.

Refer to the Log4J website Log4J website for a full list of all the configuration options.

Once the Log4J configuration has been specified and Diffusion started up, a new log file will be present in the ./logs directory:

JsonJournal.log

Looking at the contents of the new log file, log entries similar to the following can be seen:

{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"$Country":"","$ClientIP":"0:0:0:0:0:0:0:1","$SessionId":"6200c4761fa5223c-0000000000000000","$Transport":"WEBSOCKET","$Principal":"admin","$ServerName":"server-abc","$StartTime":"1644853363538","$Language":"","$Latitude":"NaN","$Connector":"Client Connector","$ClientType":"JAVASCRIPT_BROWSER","$Longitude":"NaN","$Roles":"\"ADMINISTRATOR\""},"action":"CLIENT_CONNECTION_ACTION","timestamp":1644853363571}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"path":"topic-123","type":"JSON","properties":{}},"action":"TOPIC_ADD","timestamp":1644853378421}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"name":"topic-view-123","specification":"map >topic-123 to  topic-view-123 "},"action":"CREATE_TOPIC_VIEW","timestamp":1644853418257}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"selector":">topic-123"},"action":"TOPIC_REMOVAL","timestamp":1644853432156}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVA_GATEWAY_CLIENT","id":"6200c4761fa5223c-0000000000000001"},"success":true,"context":{"$Country":"","$GatewayType":"$rest-adapter-runner","$ClientIP":"127.0.0.1","$SessionId":"6200c4761fa5223c-0000000000000001","$Transport":"WEBSOCKET","$Principal":"admin","$ServerName":"server-abc","$StartTime":"1644853463897","$Language":"","$Latitude":"NaN","$Connector":"Client Connector","$ClientType":"JAVA","$GatewayId":"gateway-application-1","$Longitude":"NaN","$Roles":"\"ADMINISTRATOR\""},"action":"CLIENT_CONNECTION_ACTION","timestamp":1644853463903}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVA_GATEWAY_CLIENT","id":"6200c4761fa5223c-0000000000000001"},"success":true,"context":{"path":"openweather/london","specification":{"properties":{"PERSISTENT":"false","REMOVAL":"when this session closes"},"type":"JSON"},"constraint":{}},"action":"ADD_AND_SET_TOPIC","timestamp":1644853465527}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVA_GATEWAY_CLIENT","id":"6200c4761fa5223c-0000000000000001"},"success":true,"context":{"path":"reqres/users","type":"JSON","properties":{"PERSISTENT":"false","REMOVAL":"when this session closes"},"constraint":{}},"action":"ADD_AND_SET_TOPIC","timestamp":1644853466562}

Troubleshooting

• The journal is reporting it is not enabled.

  • Ensure you have enabled it in the Server.xml file.

• The journal is reporting that no actions have been configured.

  • Ensure the journaling-status.properties is in the ./etc directory of the Diffusion installation (or if another directory or file has been configured, that the file is present in the directory.

• No Journal file is being produced.

  • Ensure the 3 required sections in the Log4J2.xml file are present and uncommented.