What is the Diffusion journal?

The Diffusion journal allows certain 'actions' to be written out to a log file. In order to use the journal, some configuration steps are required - these are detailed in the sections below. Once the journal has been enabled and configured, it will write to a log file using a specified format. The data in the log file can be viewed or ingested into another system as required.

The following actions can be journaled:

Formatters

The journal writes details of the actions to a file, there are currently two file formats: simple and json. Below is an example of a log entry in the different formats:

Simple formatter

2022-02-14 15:01:54.183|{session={principal=admin, protocol=24, connection_type=WEBSOCKET_JAVASCRIPT_CLIENT, id=6200c47612e6a8a1-0000000000000000}, success=true, context={path=topic-123, specification=TopicSpecification [Type=STRING, Properties={}]}, action=TOPIC_ADD, timestamp=1644850914183}

SON formatter

{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"path":"topic-123","specification":{"properties":{},"type":"JSON"}},"action":"TOPIC_ADD","timestamp":1644853378421}

Log entry contents

The contents of a log entry are the same regardless of the output format being used. A journal log entry has the following parts: action, success, context, session and timestamp.

action

This is the type of action being performed.

Example:

TOPIC_ADD

success

Indicates if the action was performed successfully true or failed false.

Example:

true

context

As its name implies, this is the 'context' of the action being performed. The values contained within the context are specific to the action.

Example:

{"path":"topic-123","specification":{"properties":{},"type":"JSON"}}

session

Details of the connected session performing the action.

Example:

{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"}

timestamp

The time that the action was performed in milliseconds since the epoch.

Example:

1644853378421

How to configure the journal

This section details the steps required to enable the journal and configure it so that it logs to a file. The steps to configure either the simple formatter or the json formatter are very similar, with just a few minor differences depending upon the chosen format. Out of the box, the journal is disabled - this can be seen in the below example log message:

2022-02-14 14:28:49.741|INFO|main|PUSH-000866|||Journal not enabled.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider

Configuring the journal requires the following steps:

• Enabling the journal feature.
• Configuring which actions to journal.
• Optionally specifying a formatter to use, the default is simple.
• Optionally specifying the location of the configuration properties file containing the actions to be journaled, the default directory is ./etc
• Configuring the logger. Internally, the journal uses Log4J2 which allows a lot of control over the log file rotation and house keeping of log files.

Enable the journal

The journal is enabled using a system property, this will need to be added to the start script or the start-up process that launches Diffusion:

-Ddiffusion.journal.enabled=true

Once the system property has been added and Diffusion started, the start-up log message will indicate that the journal has been enabled - however, it hasn't been configured with any actions to journal - as can be seen, the configured actions is zero:

2022-02-14 14:31:43.814|INFO|main|PUSH-000866|||Journal is enabled.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider
2022-02-14 14:31:43.814|INFO|main|PUSH-000871|||Journal enabled with formatter: 'simple', actions configured to be journaled: 0.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider

Configure actions to journal

Actions to be journaled must be specified in the file journaling-status.properties, the format of the file is a standard Java properties file containing key and value pairs. The example file below demonstrates all of the actions being enabled for journaling.

TOPIC_ADD=true
TOPIC_REMOVAL=true
CREATE_TOPIC_VIEW=true
REMOVE_TOPIC_VIEW=true
ADD_AND_SET_TOPIC=true
CREATE_UPDATE_STREAM=true
CREATE_UPDATE_STREAM_AND_SET=true
CLIENT_CONNECTION_ACTION=true

By default, Diffusion checks for the journaling-status.properties to be in the ./etc directory of the Diffusion installation, however the directory can be overridden.

Optionally specifying location of configuration properties

If a different directory location is required, a system property can be used to specify the alternative location of where the properties should be loaded from, as per below:

-Ddiffusion.journal.config.dir=/a/b/c

The system property will need to be added to the Diffusion start script or start-up process. Once the properties file has been configured (and the optional directory specified if required) the journal will output a message similar to the below indicating that there are actions which have been enabled for journaling:

2022-02-14 14:34:14.678|INFO|main|PUSH-000866|||Journal is enabled.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider
2022-02-14 14:34:14.678|INFO|main|PUSH-000871|||Journal enabled with formatter: 'simple', actions configured to be journaled: 8.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider

Configure formatter

This step is optional, by default the journal will use the simple log output format. If the json log format is required, it can be configured using a system property which will need to be added to the start script or start-up process:

-Ddiffusion.journal.formatter=json

When Diffusion is now started up, it can be seen that the JSON formatter has been configured successfully:

2022-02-14 15:33:13.905|INFO|main|PUSH-000866|||Journal is enabled.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider
2022-02-14 15:33:13.905|INFO|main|PUSH-000871|||Journal enabled with formatter: 'json', actions configured to be journaled: 8.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider

Configure logger

Internally the journal feature uses Log4J2 (referred to as Log4J in this document). Log4J is very powerful and can be configured in a variety of ways, such as the desired log message format, the log rotation policy and also the log house keeping approach, such as zipping log files into an archive directory.

The Log4J configuration differs slightly depending upon the formatter being used. This section covers what configuration is required for the simple formatter and the json formatter, each requires the appropriate pattern layout, appender and logger to be configured.

Refer to the Log4J website for a full list of all the configuration options: https://logging.apache.org/log4j/2.x/

Configuring logger for simple formatter

This section describes the configuration elements required to use the simple formatter

Log4J Property:

The use of a property is optional. In this example a property has been defined for the pattern layout which will be referenced by the appender.

<Properties>

  <!-- Any other properties defined -->

  <Property name="journalPattern">%date{DEFAULT_PERIOD}|%replace{%msg}{\|}{¦}%xEx</Property>

</Properties>

Log4J Appender:

The below is an example of configuring an appender. The appender is using the property to configure the required layout of the log messages.

<Appenders>

  <!-- Any other appenders defined -->

  <RollingRandomAccessFile name="journalFile" immediateFlush="true" fileName="${log.dir}/Journal.log"  
  filePattern="${log.dir}/$${date:yyyy-MM}/Journal-%d{MM-dd-yyyy}-%i.log.gz">  
    
  <PatternLayout pattern="${journalPattern}"/>  
    
  <Policies>  
    <OnStartupTriggeringPolicy/>   
    <TimeBasedTriggeringPolicy/>   
    <SizeBasedTriggeringPolicy size="250 MB"/>   
  </Policies>  

  <DefaultRolloverStrategy max="20"/>  
  </RollingRandomAccessFile> 

</Appenders>

Log4J Logger:

A logger configuration is required so that log messages are output. The class name must be as per below and specify the ActionJournalSimpleImpl:

<Loggers>

  <!-- Any other loggers defined -->

  <asynclogger name="com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalSimpleImpl" level="trace"  
  includeLocation="false" additivity="false">  
    <AppenderRef ref="journalFile"/>
  </asynclogger>
  
</Loggers>

Note: the reference to the appender.

Once the Log4J configuration has been specified and Diffusion started up, a new log file will be present in the ./logs directory:

Journal.log

Looking at the contents of the new log file, log entries can be seen using the simple formatter:

2022-02-14 15:00:48.594|{session={principal=admin, protocol=24, connection_type=WEBSOCKET_JAVASCRIPT_CLIENT, id=6200c47612e6a8a1-0000000000000000}, success=true, context={$Country=, $ClientIP=0:0:0:0:0:0:0:1, $SessionId=6200c47612e6a8a1-0000000000000000, $Transport=WEBSOCKET, $Principal=admin, $ServerName=server-abc, $StartTime=1644850848562, $Language=, $Latitude=NaN, $Connector=Client Connector, $ClientType=JAVASCRIPT_BROWSER, $Longitude=NaN, $Roles="ADMINISTRATOR"}, action=CLIENT_CONNECTION_ACTION, timestamp=1644850848593}
2022-02-14 15:01:54.183|{session={principal=admin, protocol=24, connection_type=WEBSOCKET_JAVASCRIPT_CLIENT, id=6200c47612e6a8a1-0000000000000000}, success=true, context={path=topic-123, specification=TopicSpecification [Type=STRING, Properties={}]}, action=TOPIC_ADD, timestamp=1644850914183}
2022-02-14 15:02:52.872|{session={principal=admin, protocol=24, connection_type=WEBSOCKET_JAVASCRIPT_CLIENT, id=6200c47612e6a8a1-0000000000000000}, success=true, context={name=topic-view-123, specification=map >topic-123 to  topic-view-123  }, action=CREATE_TOPIC_VIEW, timestamp=1644850972872}
2022-02-14 15:03:19.256|{session={principal=admin, protocol=24, connection_type=WEBSOCKET_JAVASCRIPT_CLIENT, id=6200c47612e6a8a1-0000000000000000}, success=true, context={actionId=139, value=topic-view-123}, action=REMOVE_TOPIC_VIEW, timestamp=1644850999256}
2022-02-14 15:03:28.441|{session={principal=admin, protocol=24, connection_type=WEBSOCKET_JAVASCRIPT_CLIENT, id=6200c47612e6a8a1-0000000000000000}, success=true, context={selector=>topic-123}, action=TOPIC_REMOVAL, timestamp=1644851008441}
2022-02-14 15:28:24.574|{session={principal=admin, protocol=24, connection_type=WEBSOCKET_JAVA_GATEWAY_CLIENT, id=6200c4765d0cbf4d-0000000000000000}, success=true, context={$Country=, $GatewayType=$rest-adapter-runner, $ClientIP=127.0.0.1, $SessionId=6200c4765d0cbf4d-0000000000000000, $Transport=WEBSOCKET, $Principal=admin, $ServerName=server-abc, $StartTime=1644852504544, $Language=, $Latitude=NaN, $Connector=Client Connector, $ClientType=JAVA, $GatewayId=gateway-application-1, $Longitude=NaN, $Roles="ADMINISTRATOR"}, action=CLIENT_CONNECTION_ACTION, timestamp=1644852504573}
2022-02-14 15:28:26.069|{session={principal=admin, protocol=24, connection_type=WEBSOCKET_JAVA_GATEWAY_CLIENT, id=6200c4765d0cbf4d-0000000000000000}, success=true, context={path=openweather/london, specification=TopicSpecification [Type=JSON, Properties={PERSISTENT=false, REMOVAL=when this session closes}], constraint=Unconstrained}, action=ADD_AND_SET_TOPIC, timestamp=1644852506069}
2022-02-14 15:28:27.126|{session={principal=admin, protocol=24, connection_type=WEBSOCKET_JAVA_GATEWAY_CLIENT, id=6200c4765d0cbf4d-0000000000000000}, success=true, context={path=reqres/users, specification=TopicSpecification [Type=JSON, Properties={PERSISTENT=false, REMOVAL=when this session closes}], constraint=Unconstrained}, action=ADD_AND_SET_TOPIC, timestamp=1644852507126}

Configuring logger for JSON formatter

This section describes the configuration elements required to use the JSON formatter, firstly ensure that the json formatter has been specified and it being used:

2022-02-14 15:33:13.905|INFO|main|PUSH-000866|||Journal is enabled.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider
2022-02-14 15:33:13.905|INFO|main|PUSH-000871|||Journal enabled with formatter: 'json', actions configured to be journaled: 8.|com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerProvider

Log4J Property:

The use of a property is optional as the pattern layout can be specified directly in the appender element.

<Properties>

  <!-- Any other properties defined -->

  <Property name="jsonJournalPattern">%replace{%msg}{\|}{¦}%xEx</Property>

</Properties>

Log4J Appender:

The below is an example of configuring an appender. The appender is using the property to configure the required layout of the log messages.

<Appenders>
  
  <!-- Any other appenders defined -->

  <RollingRandomAccessFile name="jsonJournalFile" immediateFlush="true" fileName="${log.dir}/JsonJournal.log"  
  filePattern="${log.dir}/$${date:yyyy-MM}/JsonJournal-%d{MM-dd-yyyy}-%i.log.gz"> 
    
  <PatternLayout pattern="${jsonJournalPattern}"/>
    
  <Policies>  
    <OnStartupTriggeringPolicy/>   
    <TimeBasedTriggeringPolicy/>   
    <SizeBasedTriggeringPolicy size="250 MB"/>   
  </Policies>  
    
    <DefaultRolloverStrategy max="20"/> 
    </RollingRandomAccessFile>
</Appenders>

Log4J Logger:

A logger configuration is required so that log messages are output. The class name must be as per below and specify the ActionJournalerJsonImpl:

<Loggers>
  <asynclogger name="com.pushtechnology.diffusion.journal.journaler.impl.ActionJournalerJsonImpl" level="trace"  
  includeLocation="false" additivity="false">  
    <AppenderRef ref="jsonJournalFile"/> 
  </asynclogger>
</Loggers>

Note: the reference to the JSON appender.

Once the Log4J configuration has been specified and Diffusion started up, a new log file will be present in the ./logs directory:

JsonJournal.log

Looking at the contents of the new log file, log entries can be seen using the json formatter:

{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"$Country":"","$ClientIP":"0:0:0:0:0:0:0:1","$SessionId":"6200c4761fa5223c-0000000000000000","$Transport":"WEBSOCKET","$Principal":"admin","$ServerName":"server-abc","$StartTime":"1644853363538","$Language":"","$Latitude":"NaN","$Connector":"Client Connector","$ClientType":"JAVASCRIPT_BROWSER","$Longitude":"NaN","$Roles":"\"ADMINISTRATOR\""},"action":"CLIENT_CONNECTION_ACTION","timestamp":1644853363571}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"path":"topic-123","specification":{"properties":{},"type":"JSON"}},"action":"TOPIC_ADD","timestamp":1644853378421}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"name":"topic-view-123","specification":"map >topic-123 to  topic-view-123 "},"action":"CREATE_TOPIC_VIEW","timestamp":1644853418257}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVASCRIPT_CLIENT","id":"6200c4761fa5223c-0000000000000000"},"success":true,"context":{"selector":">topic-123"},"action":"TOPIC_REMOVAL","timestamp":1644853432156}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVA_GATEWAY_CLIENT","id":"6200c4761fa5223c-0000000000000001"},"success":true,"context":{"$Country":"","$GatewayType":"$rest-adapter-runner","$ClientIP":"127.0.0.1","$SessionId":"6200c4761fa5223c-0000000000000001","$Transport":"WEBSOCKET","$Principal":"admin","$ServerName":"server-abc","$StartTime":"1644853463897","$Language":"","$Latitude":"NaN","$Connector":"Client Connector","$ClientType":"JAVA","$GatewayId":"gateway-application-1","$Longitude":"NaN","$Roles":"\"ADMINISTRATOR\""},"action":"CLIENT_CONNECTION_ACTION","timestamp":1644853463903}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVA_GATEWAY_CLIENT","id":"6200c4761fa5223c-0000000000000001"},"success":true,"context":{"path":"openweather/london","specification":{"properties":{"PERSISTENT":"false","REMOVAL":"when this session closes"},"type":"JSON"},"constraint":{}},"action":"ADD_AND_SET_TOPIC","timestamp":1644853465527}
{"session":{"principal":"admin","protocol":24,"connection_type":"WEBSOCKET_JAVA_GATEWAY_CLIENT","id":"6200c4761fa5223c-0000000000000001"},"success":true,"context":{"path":"reqres/users","specification":{"properties":{"PERSISTENT":"false","REMOVAL":"when this session closes"},"type":"JSON"},"constraint":{}},"action":"ADD_AND_SET_TOPIC","timestamp":1644853466562}

Troubleshooting

• The journal is reporting it is not enabled.

  • Ensure you have the system property in the start script or start-up process.

• The journal is reporting that no actions have been configured.

  • Ensure the journaling-status.properties is in the ./etc directory of the Diffusion installation or if the optional directory override has been used that the file is present in the alternative directory.

• No log file has been created.

  • Check the Log4J configuration is aligned with the formatter that has been chosen, e.g. if the json formatter has been selected then ensure the correct steps to configure Log4J for the ActionJournalerJsonImpl have been followed.