​​​​​​​Note – Apache Kafka is supported as a Target since version 6, by Repstance Advanced Edition only (see Supported Database Requirements).

This page describes the messages that the Kafka Apply Process publishes: the topics, the key, the value and the events of DDL. The messages are JSON. The value is an envelope with the fields before, after, source, op and ts_ms. How to set up the replication is described in Replication to Apache Kafka.

Topics, Keys and Headers

• Topic – A topic holds the messages of one table. Its name is {prefix}.{schema}.{table}, made of the name after the mapping rules, see kafka.topicprefix and kafka.topictemplate in Kafka Apply Parameters. The changes of the tables are in the topic of the table, the events of DDL are in one topic, ddl or {prefix}.ddl.
• Key – The key of a message is a JSON object with the columns of the primary key, or of the unique index, of the table with their new values: {"id":1}. All the changes of a row are in one partition, in the order of the source. A table without a primary key or unique index has messages without a key. A mapping rule with the attribute PK can name the key columns.
• Key with the table – With kafka.keyincludetable=1 the key has the source table in the first field: {"__table":"dbo.employee","id":1}. It is for tables that share a topic, for example by a mapping rule.
• Header – Every message has the header lsn, the LSN of the transaction of the source, as text.

The Value of a Change

The pair (source.lsn, source.seq) identifies a change. The Apply Process publishes a message at least once: after a failure or a restart it can publish again a message that it published before, and the repeat has the same pair, so a consumer can drop it. A consumer that writes the rows by their keys gets the same result with a repeat. With batchmode=2 the numbering of a batch that is published again can differ from the first one, so such messages are applied by the key.

An insert:

An update has both images:

A delete has the row in before, and a tombstone follows it:

Tombstones

With kafka.tombstones=1, the default, a delete is followed by a tombstone, a message with the same key and no value. A topic with the setting cleanup.policy=compact forgets the row after it. A change of the key columns of a row is an update with the new key, followed by a tombstone with the old key:

A table without a key has no tombstones: its messages have no key, and an update and a delete carry the whole row in before. A topic with cleanup.policy=compact refuses messages without a key, so validate process reports a table without a key that goes to a compacted topic as an error (see Compacted Topics in Replication to Apache Kafka).

Initial Load

The rows of the Initial Load are published to the topic of the table before the changes, with op r and source.snapshot "true". The key is the key of the row:

Values

An insert with a NULL and a text that needs escapes:

Updates with Less Than the Whole Row

A message has the columns that the trail has. When the Source Database logs only the changed columns and the key, for example Oracle without supplemental logging of all columns, the images of an update have only those columns. A column that is not in the message was not in the trail, it is not NULL.

Large Values

A key and a value together are at most kafka.maxmessagebytes bytes. When a row is larger, the longest strings of the row, the text and the binary values, are cut to fit, in the images before and after alike, and the message lists the columns that were cut:

In a test a row with a text of 3,000,000 characters and a binary value of 2,000,000 bytes became one message of 979,990 bytes, with both columns cut to about half of a million characters, and a value of 400 KB stayed whole.

Events of DDL

A change of a table, a create, an alter, a drop and a truncate, is an event in the topic of the DDL events. The topic has one partition, so the events are in the order of the trail. A change of an index that does not change the key columns is not an event. The key is the name of the table: {"table":"dbo.employees"}.

• op – create, alter, drop or truncate
• before, after – The columns of the table before and after the change, in the order of their ids. null when there is no table: before of a create, after of a drop and a truncate. Every column is an object with the fields id, name, type, key, nullable and identity. The id stays when a column is renamed. The type is the name of the type in SQL Server, with its length, for every source: nvarchar(40) for a MySQL varchar with the character set utf8mb4. key is true for a column of the primary key or of the unique index. A change of the key columns is an alter
• oldTable – The old name of the table, when the change renamed it
• ddl – The statement, or the statements, of the source that made the change, a list of strings, as the source ran them. The names in before and after are those after the mapping rules, the statement is not changed. The field is not there when the statements together are longer than 64 KB (65536 bytes); the Apply Process says so in its log. The columns are in the event anyway
• source, ts_ms – As in the messages of the rows

An alter that adds a column:

The messages that were published before the change do not have the new column. A consumer reads the events of the DDL topic to learn which columns the messages after the change have.

The load option of an Initial Load (see Initial Load) cannot clean up a topic: a topic has no delete and no truncate. The option is published as events of DDL, before the first row of the table, so that a consumer is told that the table starts again. The events have no field ddl, because no statement of the source made them:

• D (delete) and T (truncate) – one event, truncate
• C (create) – one event, create, with the columns of the table
• R (recreate) – two events, drop and then create
• A (preserve) and M (merge) – no event

Apache, Apache Kafka, Kafka and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.