​​​​​​​Note – Apache Kafka is supported as a Target since version 6, by Repstance Advanced Edition only (see Supported Database Requirements).

The following parameters are valid for the Kafka Apply Process. How the changes become messages is described in Replication to Apache Kafka and Kafka Message Format.

Process Parameters

• process – The only possible value is apply
• id – The Apply Process id
• dbtype – Type of RDBMS, the only possible value is kafka
• capturename – The name of the Capture Process which is providing the data to be used

Kafka Connectivity

• server – The bootstrap brokers of the cluster: host[:port],host[:port]. A name without a port gets the value of port
• port – The port of the brokers whose name in server has none. The default is 9092
• kafka.tls – 1 connects to the brokers with TLS and checks their certificates. The default is 0
• kafka.cafile – PEM file with the certificate authorities that signed the certificates of the brokers. The authorities of the host are used when it is empty
• kafka.certfile, kafka.keyfile – PEM files with the client certificate and its private key, for a cluster that asks for a client certificate. A key with a passphrase is not supported
• kafka.sasl – The login to the brokers: none (default), plain, scram-sha-256 or scram-sha-512. PLAIN sends the password as it is: use it with kafka.tls=1
• user, password – The user name and the password of the login. Required with kafka.sasl

The same settings, and how they are checked, are described in Kafka as Target. The files are read when the Apply Process starts, so a renewed certificate is used after the next start.

Topics

• kafka.topicprefix – The first part of the topic names, the logical name of the source: {prefix}.{schema}.{table}. Letters, digits and the symbols . _ -. Without a prefix the topics are named {schema}.{table}
• kafka.topictemplate – Template of the topic name. The tokens %PREFIX%, %SCHEMA% and %TABLE% are replaced, empty parts are left out. The default is %PREFIX%.%SCHEMA%.%TABLE%. For example, cdc-%SCHEMA%-%TABLE% gives the topic cdc-hr-staff for the table hr.staff. The schema and the table are those after the mapping rules. A symbol that Kafka does not accept in a topic name is replaced by _, the case is kept, and a name longer than 249 characters is cut
• kafka.createtopics – Specifies if the Apply Process creates the topics that do not exist. The possible values are:
  • 0 – the topics must exist before the process runs. validate process lists the missing ones (default value)
  • 1 – create the topics
• kafka.partitions – The number of partitions of a topic that the process creates. A table without a primary key or unique index and the topic of the DDL events always get one partition. The default is 1
• kafka.replicationfactor – The number of replicas of a topic that the process creates. The cluster refuses more than it has brokers. The default is 1
• kafka.topicconfig – Settings of a topic that the process creates: name=value,name=value, for example cleanup.policy=compact (never on the topic of the DDL events). A value has no comma
• kafka.ddltopic – The topic that receives the events of DDL, a change of a table. It has one partition. The default is ddl, with a prefix {prefix}.ddl

Messages

• kafka.tombstones – Specifies if a delete, and a change of the key, are followed by a tombstone: a message with the old key and no value, which lets a compacted topic forget the row. The possible values are:
  • 0 – no tombstones
  • 1 – write the tombstones (default value)
• kafka.keyincludetable – 1 puts the source table into the message key, so that tables which share a topic keep their keys apart. The default is 0
• kafka.maxmessagebytes – The largest size of the key plus the value of a message, in bytes. The longer strings of a row are cut to fit, and the message lists the columns that were cut in the field truncated. The default is 980000, a little below the 1 MiB that the brokers accept by default. A larger value needs the settings message.max.bytes of the brokers and max.message.bytes of the topics to be raised
• batchmode – Specifies how the changes of a batch of transactions are published. The possible values are:
  • 0 – a message for every change (default value)
  • 1 – the same as 0
  • 2 – the changes of a row within a batch are netted into one message: a row that was changed three times is one update, an insert and a delete of the same row give nothing, an insert and an update give one insert, an update and a delete give one delete. The order of the transactions, and of the changes of different tables, is not kept

Transactions and Initial Load

• maxtxbytes – Size, in bytes, of a transaction above which the Apply Process reads the transaction from the trail files and applies it in parts. For Kafka a transaction that is applied in parts is one Kafka transaction. The value 0 disables splitting of the transactions. The default value is 125829120 (120 MiB)
• kafka.transactiontimeout – The number of seconds that a Kafka transaction of such a transaction may stay open; the broker aborts it after that. The brokers limit it with transaction.max.timeout.ms (15 minutes unless the administrator of the cluster has changed it). The default is 600
• ldtxsize – The number of rows of the Initial Load between two saves of its progress. A load that is started again goes on after the last save. The default is 500000

Filters and Local Transformations

• map – The set of the parameters to determine local objects’ transformation (see Objects Mapping and Possible Transforms for the details)

Note – The Kafka Apply Process uses the rules to rename and to exclude schemas, tables and columns, and the attribute PK to give a table a key. The names after the rules are the names of the topics and of the fields. A column that a rule adds, and a data transformation that is produced by a SQL function call, are not part of the messages.

• filter – The set of the parameters to determine objects’ filtering (see Objects Filtering for the details)

Note – A filter of the Kafka Apply Process skips whole operations of a table: INSERT, UPDATE or DELETE. The skipped operation is no message, and a delete or an update that changes the key has no tombstone either. A filter with a condition is refused: The filter [filter=1] has a condition, which a Kafka target does not evaluate: the rows would not be skipped.

Performance Tuning Parameters

• chunkrows – The maximum number of messages that the Apply Process sends to the brokers in one call. The default value is 5000
• chunkbytes – The maximum size of the keys and values that are sent to the brokers in one call, in bytes. The default value is 10485760 (10 MiB)
• readbytes – Approximate number of bytes of the trail data which the Apply Process reads in one cycle. The transaction that crosses the limit is still read completely. The default value is 5242880 (5 MiB)

Other Parameters

• autostart – Specifies if the process is started automatically when it is created and when the Repstance service starts. The possible values are 0 (default) and 1
• debuglevel – The level of detail that is written to the log of the Apply Process, 0-16. The default value is 0
• cleanuptrail – The number of days that the applied trail files are kept. The value 0 deletes them when they are applied, the value -1 never deletes them. The default value is 0
• restartattempts – The number of automatic restarts of a failed Apply Process; the server checks every 5 minutes. The value 0 means never. The default value is 5
• notification – The ids of the e-mail channels that are told when a failed Apply Process cannot be restarted any more, for example 1,2

Note – The parameters of the SQL targets that have no meaning for Kafka are refused, for example ddlprocessing (a change of a table is always an event in the topic of the DDL events), dbname, bulkthreshold and skiploopback.

Examples

Create an Apply Process that publishes the changes of the Capture Process demo to a Kafka cluster. The topics are named prod.{schema}.{table}, and the process creates them with three partitions:

Show the parameters:

Check the process. A new process starts with the first transaction that is available in the trail:

Publish only the deletes of the table dbo.t and send its messages to the topic of another name:

A filter with a condition is refused:

A parameter of the SQL targets is refused and the reply lists the parameters of Kafka:

A value that is not accepted:

Apache, Apache Kafka, Kafka and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.