CDC Adapter User Guide

The CDC (Change Data Capture) adapter can be used to replicate data from databases into a Diffusion server (or server cluster). This adapter uses the Debezium(version 2.5.4.Final) engine to connect to databases. Debezium uses a log-based change data capture mechanism where it reads transaction logs from the database to identify the row-level change events. These will be processed by the adapter to publish to a specific Diffusion topic.

Each row-level data change event (insert/update/delete) captured by Debezium will be processed and published to a JSON Diffusion topic. More details about mapping these events to Diffusion topics can be found in the Database reader service topic.
Every event consists of details about data changes as well as its schema. The schema of the data (or a table) can be optionally published to a separate Diffusion topic, specific to that table of the database. This is configurable and is disabled by default.

Debezium provides connectors for many databases. The Diffusion CDC adapter supports fully-tested MySQL and PostgreSQL connectors out-of-the-box. Connection to other databases can be done by adding the corresponding Debezium connector .jar in the classpath when running the adapter. Here is the comprehensive list of supported connectors in Debezium 2.5.4.Final.

With the usage of Debezium at its core, the adapter supports setting any configuration options supported by Debezium. Users can use these Debezium configurations according to their requirement in the adapter. The configuration parameters in Debezium provide an option to exclude/include a list of databases/tables/columns to track. So, users will be able to configure a single adapter to track different tables in a database with different sets of configurations. Similarly, any restrictions and requirements for using Debezium also apply to this adapter.

Refer to the Debezium documentation to understand the supported configuration for different databases.

The adapter uses the Diffusion Gateway Framework, hence supports all features supported by the framework. See Gateway Application section in the Framework’s used guide to understand the concepts of the framework and to understand how to configure a Gateway application.

The adapter can be viewed and managed from the Diffusion management console of the connected server or on any other server in the same cluster. Once started, the adapter is visible on the Diffusion console, in the 'Network' section, as type 'CDC Adapter' with the id specified in the configuration.

This document should be used in conjunction with the Gateway Framework user guide. Together, these resources provide comprehensive instructions on how to set up and utilise the CDC Adapter with Diffusion.