The CDC adapter for Diffusion™ enables sharing data
between Diffusion and a compatible database.
The CDC (Change Data Capture) adapter can be used to connect a Diffusion server (or server cluster) to a database server.
It uses Debezium (1.6.1.Final), under the hood to connect with databases.
The adapter can fetch the latest data, with its schema, from multiple databases and publish to JSON Diffusion topics.
Any data change events captured by the Debezium connector are processed and published to Diffusion topics according to the configuration.
Since the adapter uses Debezium as a solution to capture the data change events, all configuration options supported by Debezium are allowed.
Please refer to the Debezium documentation at https://debezium.io/documentation/ to understand the supported options.
The CDC adapter comprises the following files all located in the
directory of your Diffusion
- This JAR file contains the
Java™ application that links the Diffusion server and a database.
- This file is used to configure the CDC adapter.
- This JSON schema file can be used to validate the configuration.json configuration file.
- Log file.
The CDC adapter requires Java. The version requirements are the same as for the Diffusion server.
The CDC adapter is tested with MySQL and PostgreSQL. Connection to other databases may work but is not supported.
A service represents a specific function that this adapter supports.
This can be of a specific type which is identified by a unique name, specific to adapters.
Each service has a type, name, description and configuration.
The overall configuration of the adapter is defined by each service configuration.
The services can be added/updated/removed via the Diffusion console, after the adapter is started.
If the adapter is started with a complete configuration file, these configurations will be used to create services, during adapter startup,
which will be visible in the console.
The CDC adapter supports two types of service which are required to set up required interactions between databases and Diffusion.
Database Connector Service
The Database Connector Service is used to define Debezium connector configuration for a specific database, in the adapter.
Once this service is defined, it can be referred to in the configuration of the Diffusion publisher service.
This means that any common Debezium connector configuration can be defined as a Database connector service, and reused in multiple Diffusion publisher services.
Adding this service will not start any process, but will update the overall configuration of the adapter to contain
the Debezium connector configuration detail.
This service does not support any operations.
Diffusion Publisher Service
A Diffusion publisher service is used to start the Debezium connector engine, and begin streaming changed data values from the database to Diffusion topics.
There can be multiple instances of this service added to the adapter, to consume data from tables in the database, using different configuration.
Using the CDC adapter
Before you run the adapter, you can configure it by editing the
configuration.json to define the adapter behavior. You can also pass a bootstrap configuration
as a system property, or you can combine system properties and the configuration file. For more
information, see Configuring the CDC adapter.
Configuration can also be updated during runtime via the Diffusion console.
To run the adapter, run the .jar file. See Running the CDC adapter.
Monitoring the CDC adapter
The adapter can be visualised and monitored in the Diffusion management console, using the