Capture and Source Databases

What to do when a Capture Process cannot start, stops, or captures nothing, and when the Source Database is not ready. The status of a failed Capture Process shows the error in brackets after the state (see Status of any Capture or Apply Processes). The same text is in the process log and in Incident Management of the Web UI (see Logs and Information for Support). A Capture Process runs the checks of validate database every time it starts, so every message of that command can also appear as RUN (Failed).

The Capture Process cannot start

Unable to connect to the Source Database

The message is the error of the database. For SQL Server it is unable to open tcp connection with host 'db-src.example.com:1433': dial tcp 10.0.0.5:1433: i/o timeout or login error: Login failed for user. For Oracle it starts with username="repuser" sid="db-src.example.com:1521/service" and ends with an ORA- error (for example ORA-12541, no listener, or ORA-01017, invalid user name or password). Incident Management shows it as Unable to connect to database.

The host name, the port or the service name is not right, something between the two servers blocks the connection (a firewall, or the security group of the instance), or the user or the password is not right. Check from the Repstance server that the host name resolves and that the port can be reached, then correct the parameters with alter and run the process again.

DPI-1047: Cannot locate a 64-bit Oracle Client library

The server does not find the Oracle Instant Client. Install the Oracle Instant Client (64-bit) on the Repstance server, make sure that its directory is known to the dynamic linker (the directory in a file of /etc/ld.so.conf.d, then ldconfig), and restart the Repstance service. The Instant Client 23ai connects to Oracle Database 19c and later only, so use the 19c client for an older database.

The database is not ready: validate database says what is missing

The checks of validate database (see Validate Database) decide whether a Capture Process can start. The usual messages:

The process needs an instance with at least 32 GB of RAM

On AWS and Azure the Data Comparison and the parameter asyncredo=1 need an instance with at least 32 GB of RAM. The messages are Data comparison needs an instance with at least 32 GB of RAM (this instance has 15.5 GB) and, for the Capture Process that runs without the asynchronous mode, Async redo mode (asyncredo=1) needs an instance with at least 32 GB of RAM. See Requirements and Limits and Oracle Capture Parameters.

The Capture Process stops

A redo log or an archived log is not available

The Capture Process fails and its status shows RUN (Failed) with the reason. The text depends on the mode. With asyncredo=1 it is Sync Error: [the redo log file does not exist: ... The database lists it but it is not on the disk (purged or deleted), so the capture cannot continue from this SCN: ...]. Without it, Missed logfile for sequence [1234] (directlogmode=0, LogMiner) or Missed logfile for [thread#, sequence] [1,1234] (directlogmode=1 or 2) means that the Source Database lists the log without a file name. When the database lists the log with its name and the file is gone, the status shows the error of the Source Database about opening the file, for example ORA-22288: file or LOB operation FILEOPEN failed.

A log that the Capture Process has not read yet was deleted (by RMAN, by a retention setting, on Amazon RDS by the retention of the archived logs), or the directories ARCHIVELOG_DIR and ONLINELOG_DIR do not point at the files, or the user may not read them. The server runs the failed process again every 5 minutes, at most restartattempts times, and each run fails in the same way.

What to do: restore the log from the backup (for example with RMAN) to the place where the database lists it, and run the process again. Nothing is lost then. When the log cannot be restored, the changes in it cannot be captured: restart the Capture Process from the current position of the Source Database, as described below, and repair the Target Database afterwards. To prevent it, keep the archived logs until the Capture Process has read them.

Restart the Capture Process from the current position

Use it when the redo log or the archived log that the Capture Process needs cannot be restored. The Capture Process then continues with the changes that are made from the moment it is run. The changes in the missing logs are not captured, and the Target Database does not get them.

1. Write down when the failure began. Incident Management of the Web UI shows the time of the incident (see Monitoring Replication). The changes made after that time are the ones that will be missing in the Target Database.
2. Stop the Capture Process, so that the server does not run the failed process again while you work.
3. Reset it to the current position with lsn=now. The command returns at once, and the status shows RESET (Completed) when it is done (see Reset Process).
4. Run the Capture Process. It reads the current SCN of the Source Database at this moment and starts there.
5. Look at the status: RUN (In Process) and a line Last Captured DB Change that moves show that it works.

The position is taken when the process is run, not when the command is given, so the changes made between the reset and the run are not captured either.

A transaction that is open in the Source Database when the Capture Process starts is read from its beginning, so that it is captured completely. When the beginning of such a transaction is in the log that is missing, the Capture Process fails again with the same kind of message. Wait until the transaction has ended and repeat the steps. Or reset the Capture Process to an SCN after the missing logs, a decimal number that you take from the Source Database (the column CURRENT_SCN of V$DATABASE), for example lsn=2468013579. A transaction that is open at that SCN is captured only from the SCN on, so what it changed before is lost.

The Apply Processes need no action: they continue with the trail files that the Capture Process writes after the restart. Nothing marks the gap in the trail files and no message reports it, so the Target Database misses the changes of the gap. Compare the tables and repair them (see DATA COMPARISON). The repair inserts the rows that are missing and updates the rows that differ, but it does not delete the rows that exist only in the Target Database, so the rows that were deleted in the gap stay: load such tables again with the Initial Load (see Initial Load).

A DDL statement that was run in the Source Database in the gap is not replicated. Compare the structure of the replicated tables and make the same change in the Target Database; otherwise the Apply Process stops with the error of a missing column (see Apply and Target Databases).

Not enough space left on device

The Capture Process stops with Not enough space left on device [/opt/repstance/trail] when less than 250 MB are free in the file system that holds the trail directory. Free the space, and see that the Apply Processes run, because they delete the trail files that they have applied (the parameter cleanuptrail, see Housekeeping). The server runs a failed process again at most restartattempts times; run it yourself when they are used up.

invalid argument, permission denied: the trail cannot be written

A message such as [invalid argument] or [open captureckpt.1: permission denied] with Write CDC data error, Write Metadata error or Write checkpoint error means that the trail files or the checkpoint file cannot be written. invalid argument often hides the real error. Check that the directory /opt/repstance/trail exists, that the user repstance may write in it and in /opt/repstance (files that were restored by root belong to root), and that the disk is not full.

Capture checkpoint is corrupted or doesn't exist

The line Capture checkpoint [captureckpt.1] is corrupted or doesn't exist in the log is normal when a Capture Process is run for the first time. When it appears for a Capture Process that has run before, the checkpoint file captureckpt.{id} in /opt/repstance is lost. A Capture Process of a SQL Server Source Database rebuilds it from the trail files. A Capture Process of an Oracle Source Database starts like a new one, at the current SCN of the Source Database, so the changes made in the meantime are not captured: restore the file from the backup (see Backup Repstance Files), or reset the process to the position that you need and repair the data.

Nothing is captured

The process runs, but nothing arrives in the Target Database

The status RUN (In Process), Waiting for Transactions with zero counters means that the Capture Process finds no matching change. Check in this order:

• The masks dmlinclude and ddlinclude have the form schema.table, an entry without the dot is ignored, and % matches any text. Without ddlinclude no DDL is captured. For an Oracle Source Database the masks are changed to upper case.
• alter process=capture id=1 dmlinclude=SCHEMA.% preview=1 lists the tables that the masks select, and saves nothing; an empty list means that nothing matches (see Capture Objects Specification).
• validate process=capture id=1 warns when no table is configured (No Tables are Configured for Replication), when the rules leave a table without columns, and, for Oracle, when a column type is not supported.
• Set debuglevel to 1 for a short time: the log then says whether the Capture Process mined nothing, or found changes and left them out (see Logs and Information for Support).

SQL Server: No data is provided by CDC jobs

Incident Management shows No data is provided by CDC jobs with the errors that the CDC jobs of the database have reported since the last start. The CDC capture job does not run or fails. Start SQL Server Agent and the job, and read the error that the incident gives.

Initial Load failed for table

The message Initial Load failed for table [x]. Error: ... names the table and gives the error of the Source Database. The log and Incident Management (Initial Load error) have it. Remove the cause and load the table again (see Initial Load).