Redshift Apply Parameters

The following parameters are valid for the PostgreSQL Apply Process:

Process Parameters

• process – The only possible value is apply
• id – The Apply Process id
• dbtype – Type of RDBMS, the only possible value is redshift
• capturename – The name of the Capture Process which is providing the data to be used

Database Connectivity

• server – Host name or IP address of the database server
• port – Database port
• dbname – Database name
• user – Database User name
• password – Database User password

S3 Connectivity

• s3.accessKeyID – S3 Access Key ID
• s3.bucketName – S3 bucket name
• s3.secretAccessKey –S3 Secret Access Key (See Manage access keys for IAM users for the details on Access Key ID and Secret Access Key)
• s3.region – AWS region
• s3.IAMRole – IAM Role with a trust policy that allows the Redshift service to access S3 

Transaction Processing Parameters

• ddlcreate – Used to define the Apply Process behaviour when creating a table, if a table with the same name is found. The table is specified in schema _name.table_name format. In order to specify the list of the tables, each table must be separated by comma. The % symbol may be used in order to match any number of characters. It has the following options:
  • skip– do not run DDL create command if the table already exists
  • recreate – to drop the existing table and recreate it
• ddldrop – Used to define the Apply Process behaviour when dropping a table, if a table with the same name is not found. The table is specified in schema _name.table_name format. In order to specify the list of the tables, each table must be separated by comma. The % symbol may be used in order to match any number of characters. It has the following option:
  • skip – do not run DDL drop command if the table doesn’t exist
• ddlprocessing – Determines if DDL must be processed based on the user statement or generated based on the dictionary changes. The possible values are:
  • native – to process DDL based on the user statement (default value)
  • dictionary – to generate DDL based on the dictionary changes
  • map – to generate DDL based on the dictionary and the objects’ transformation rules

Note – For Homogeneous type of replication it is recommended to use "native" method of DDL processing. The "dictionary" value is commonly used in the case of Heterogeneous replication. “map” is used when the DDL statement must be reformatted following objects' transformation rules 

• ldtxsize – Determines the number of records to be inserted in one transaction at the Initial Load stage. If the value is 0 the table is loaded in a single transaction. The default value is 500000
• batchmode – This parameter is to specify transaction processing method.  The possible values are:
  • 0 – apply all the data changes within transactions as they were occurred in Source (default value)
  • 1 – combine and processes set of transactions in a single transaction
  • 2 – combine set of the transactions into a single transaction and analyse data within it to produce the only one net change for each row that has been changed

Note – setting “batchmode” parameter to “2” requires all the foreign keys to be disabled, otherwise the Apply Process may failed with the “foreign key violation” error 

• BulkThreshold –  Default Value: 1000. Specifies the minimum number of captured data changes (CDC records) required to trigger bulk loading mode. When the volume of pending changes is below this threshold, transactions are processed sequentially or in standard batch mode to minimize latency. When the number of pending changes meets or exceeds this threshold, the pipeline dynamically switches to high-throughput bulk processing to optimize database performance and maximize ingestion speed.

Note – Setting this value too low may cause unnecessary transactional overhead for minor updates. Setting it too high may prevent the system from ever utilizing bulk mode, resulting in slower processing times during high-volume periods.

• skipapply – Used to define the behaviour of the Apply Process when inserting data, which was originally produced by Apply Processes in the Source Database as part of the “loop-back” control function, in that, this Apply Process will insert, or ignore the data. The Apply Processes to be skipped are specified by ids separated by comma. In order to specify all Processes the “all” value to be used. In order to avoid skipping any Apply Processes the “none” value to be used. The default value is “none”.

Filters and Local Transformations 

• map – The set of the parameters to determine local objects’ transformation (see  Objects Mapping and Possible Transforms for the details)
• filter – The set of the parameters to determine objects’ filtering (see Objects Filtering for the details)

Other Parameters

• autostart – Determines if the Apply Process must be run automatically. Possible values are:
  • 0 – do not run the Apply Process automatically (default value)
  • 1 – to run the Apply Process automatically
• cleanuptrail – Determines number of days the Apply Process must keep trail files. Possible values are:
  • -1 – Never cleanup
  • 0 – Cleanup once the trail files is processed (default value)
  • 1 .. 999 – Number of days the Apply Process keeps the files
• restartattempts –Number of attempts to restart the failed process
• debuglevel – The level of debugging. Possible values are 0-15

Web UI