Try now
Try now
Use Cases
Replication and data migration Zero downtime migration Logical standby Zero downtime upgrade Distributed clusters Real-time reporting systems and Business Intelligence
Support
Current Feature Set Documentation Professional Services Roadmap
Company
About Us Contact Us Become a Partner News Updates
Try now
Try now
Home / Repstance User Guide / DATABASE CONFIGURATION COMMANDS / Prepare Target Database / Kafka as Target

​​​​​​​Note – Apache Kafka is supported as a Target since version 6, by Repstance Advanced Edition only (see Supported Database Requirements).

With Apache Kafka as the Target the Apply Process publishes the changes of the Source Database as messages to the topics of a Kafka cluster. There is no database and there are no Repstance objects, so prepare and remove with dbtype=kafka do nothing: they are answered with the status In Process. Only validate works. It asks the brokers for their metadata, so it tells whether a broker answers with the connection settings and the login that are given. It does not look at the topics of an Apply Process; Validate Process does. The permissions that the Kafka user needs are described in Database Privileges.

Parameters

ParameterDescription
databasetarget. Required.
dbtypekafka. Required.
serverThe bootstrap brokers of the cluster: host[:port],host[:port]. A name without a port gets the value of port. The command is valid when one of the brokers answers. Required.
portThe port of the brokers whose name in server has none. The default is 9092.
kafka.tls1 connects to the brokers with TLS. The certificate of every broker is checked against the certificate authorities (see kafka.cafile) and against the name of the broker. The default is 0.
kafka.cafileA PEM file on the Repstance server with the certificate authorities that signed the certificates of the brokers. The authorities of the host are used when it is empty. It is for kafka.tls=1.
kafka.certfile, kafka.keyfilePEM files on the Repstance server with the client certificate and its private key, for a cluster that asks for a client certificate. Both are needed, and both are for kafka.tls=1. A key with a passphrase is not supported.
kafka.saslThe login to the brokers: none, plain, scram-sha-256 or scram-sha-512. The default is none. PLAIN sends the password as it is: use it with kafka.tls=1. A valid command with plain and without TLS carries the warning kafka.sasl=plain without kafka.tls=1 sends the password to the brokers as it is.
userThe user name of the login. Required with kafka.sasl.
passwordThe password of the user. Required with kafka.sasl.

The topics and the other Kafka parameters belong to the Apply Process, see Kafka Apply Parameters.

Examples

Check that a broker answers:

repcli validate database=target dbtype=kafka server=127.0.0.1:9092
[Valid]: Kafka is valid

The brokers can be listed. The command is valid when one of them answers, here the second:

repcli validate database=target dbtype=kafka server=127.0.0.1:9093,127.0.0.1:9092
[Valid]: Kafka is valid

A name without a port gets the value of port:

repcli validate database=target dbtype=kafka server=127.0.0.1 port=9092
[Valid]: Kafka is valid

No broker answers:

repcli validate database=target dbtype=kafka server=127.0.0.1:9093
[Invalid]: no broker of 127.0.0.1:9093 answers: unable to dial: dial tcp 127.0.0.1:9093: connect: connection refused

A cluster with TLS, a client certificate and a login with SCRAM:

repcli validate database=target dbtype=kafka server=kafka1.example.com:9093 kafka.tls=1 kafka.cafile=/opt/repstance/kafka/ca.pem kafka.certfile=/opt/repstance/kafka/client.pem kafka.keyfile=/opt/repstance/kafka/client.key kafka.sasl=scram-sha-512 user=repstance password=Passw0rd123
[Valid]: Kafka is valid

When a login or a certificate is refused, the reply names the security settings that were used, without the password, for example no broker of kafka1.example.com:9093 answers (kafka.tls=1, kafka.sasl=scram-sha-512 as repstance), because the brokers often answer a refusal by closing the connection. A login without a user, and a certificate without TLS, are refused before any connection is made:

repcli validate database=target dbtype=kafka server=127.0.0.1:9092 kafka.sasl=plain
[Failed]: kafka.sasl=plain needs user.
  user                      User name of the SASL login to the brokers. [required with kafka.sasl]repcli validate database=target dbtype=kafka server=127.0.0.1:9092 kafka.certfile=/opt/repstance/kafka/client.pem
[Failed]: kafka.certfile is for kafka.tls=1.
  kafka.certfile     File with the client certificate, for a cluster that asks for one; needs kafka.keyfile and kafka.tls=1. [path of a PEM file]

prepare and remove do nothing:

repcli prepare database=target dbtype=kafka server=127.0.0.1:9092
[In Process]: Command [PREPARE] has been sent. Use [validate] command to validate the state

Web UI

The form “Database Settings” of an Apply Process for Kafka has the fields “Brokers”, “Port”, “Use TLS”, “Authentication” (None, PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512), “Username”, “Password”, “CA File”, “Client Certificate” and “Client Key”. The user name and the password are asked when an authentication is selected, and the three files when “Use TLS” is ticked. “Test connection” is validate. The form has no buttons to prepare the database or to clean it up. The topic settings of the Apply Process are on the tab “Process Parameters” (see Kafka Apply Parameters).

Apache, Apache Kafka, Kafka and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

 
On this page
Parameters
Examples
Web UI