AWS S3 Apply Parameters

The following parameters are valid for the AWS S3 Apply Process:

Process Parameters

• process – The only possible value is apply
• id – The Apply Process id
• dbtype – Type of RDBMS, the only possible value is s3
• capturename – The name of the Capture Process which is providing the data to be used

AWS S3 Connectivity

• s3.accesskeyid – S3 Access Key ID
• s3.secretaccesskey – S3 Secret Access Key (See Manage access keys for IAM users for the details on Access Key ID and Secret Access Key)
• s3.region – AWS region. Default is us-east-1
• s3.bucketname – S3 bucket name

AWS S3 File Parameters

• s3.filetype – Output file format. The possible values are:
  • csv.gz – Comma-separated variable (.csv) file compressed using gzip (default value)
  • csv – Comma-separated variable (.csv) file
  • parquet.gz – Parquet (.parquet) file compressed using gzip
  • parquet – Parquet (.parquet) file
• s3.filetemplate – Template of the folder and file name of the files with the changes written to the S3 bucket, without the extension (the extension is added according to the “s3.filetype” parameter). The “/” symbol separates the folders. The following tokens can be used in the template:
  • %SCHEMA% – schema name
  • %TABLE% – table name
  • %DATE% – date (yyyymmdd) of the commit of the last transaction in the file
  • %TIME% – time of the commit of the last transaction in the file: hours, minutes, seconds and milliseconds (hhmmssmmm)
  • %SDATE%, %STIME% – current date and time of the Repstance host (the time as %TIME%)
  • %YYYY%, %MM%, %DD% – current year, month and day of the Repstance host (the month and the day without a leading zero)
  • %SCN% – SCN of the last transaction in the file
  • %LSN% – LSN of the last transaction in the file

Note – The names of the files must be unique, otherwise a file overwrites the file with the same name in the bucket. Add %SCN% or %LSN% to the template to guarantee it.

• s3.loadfiletemplate – Template of the folder and file name of the files created by the Initial Load, without the extension. The same tokens as for the “s3.filetemplate” parameter can be used. The default value is %SCHEMA%/%TABLE%/LOAD00000001.

Note – With the default value a repeated Initial Load of a table overwrites the file of the previous one.

• s3.csvincludeheader – This parameter is valid for “csv” and “csv.gz” formats only. It determines if the column header names must be included into the output file. The possible values are:
  • 0 – do not include header
  • 1 – include header (default value)
• s3.includebeforeupdate – Specifies output format for the “update” operation. The possible values are:
  • 0 – to output “after-update” record only
  • 1 – to output “before-update“ and “after-update” records (default value)
• s3.loadincludeop  – Specifies if the column contains “type of the operation” must be included into the output file. The possible values are:
  • 0 – do not include the column
  • 1 – include the column (default value). The column name is specified by the "s3.operationcolumnname" parameter. The column has the following values:
    • “I” - insert operation
    • “D” - delete operation
    • “U” - “after-update” value for the update operation. Valid if the “s3.includebeforeupdate” parameter is set to 0
    • “BU” - “before-update” value for the update operation. Valid if the “s3.includebeforeupdate” parameter is set to 1
    • “AU” - “after-update” value for the update operation. Valid if the “s3.includebeforeupdate” parameter is set to 1
• s3.operationcolumnname – Name of the column enabled by the "s3.loadincludeop" parameter. The default value is __$OPERATION
• s3.csvnullsymbol – Specify output string for the null values (NULL is default).  This parameter is valid for “csv” and “csv.gz” formats only.

Filters and Local Transformations 

• map – The set of the parameters to determine local objects’ transformation (see  Objects Mapping and Possible Transforms for the details)

Note – The S3 Apply Process doesn't support data transformation that are produced by a SQL function call. 

• filter – The set of the parameters to determine objects’ filtering (see Objects Filtering for the details)

Note - The S3 Apply Process doesn't support SQL statement as a part of filtering options

Performance Tuning Parameters

• readbytes – Approximate number of bytes of the trail data which the Apply Process reads in one cycle. The transaction that crosses the limit is still read completely. The default value is 5242880 (5 MiB).
• maxtxbytes – Size, in bytes, of a transaction above which the Apply Process reads the transaction from the trail files and applies it in parts, to keep the memory usage under control. The value 0 disables splitting of the transactions. The default value is 125829120 (120 MiB).

Other Parameters

• skiploopback – Used to define the behavior of the Apply Process when inserting data, which was originally produced by Apply Processes in the Source Database as part of the “loop-back” control function, in that, this Apply Process will insert, or ignore the data. The Apply Processes to be skipped are specified by ids separated by comma. In order to specify all Processes the “all” value to be used. In order to avoid skipping any Apply Processes the “none” value to be used. The default value is “none”.
• autostart – Determines if the Apply Process must be run automatically. Possible values are:
  • 0 – do not run the Apply Process automatically (default value)
  • 1 – to run the Apply Process automatically
• cleanuptrail – Determines number of days the Apply Process must keep trail files. The incremental data comparison reads the changes from these files and fails if the Apply Process has already deleted them, so use -1 or a number of days that is longer than the time between two comparisons (see Start/Stop Task and Tasks Control). Possible values are:
  • -1 – Never cleanup
  • 0 – Cleanup once the trail files is processed (default value)
  • 1 .. 999 – Number of days the Apply Process keeps the files
• restartattempts – Number of attempts to restart the failed Apply Process automatically. The server checks the processes every 5 minutes. The value 0 disables the automatic restart. When the attempts are used up, the notification channels (see the “notification” parameter) are informed. The default value is 5.

Note – The number of attempts is restored to the full value as soon as the process makes progress after a failure.

• notification – The ids of the notification channels, separated by comma (see Monitoring Replication for the details on the channels), which are informed by e-mail when the process has failed and can not be restarted automatically any more (see the “restartattempts” parameter). By default nobody is notified.
• pollingdelay – Number of seconds the Apply Process waits before it looks at the trail files again when there is nothing to apply. If the value is 0 the Apply Process waits 1 second. The default value is 0.
• debuglevel – The level of debugging. Possible values are 0-15
• preview – Determines if the configuration of the “prepare” or “alter” command must be saved. The possible values are:
  • 0 – save the configuration (default value)
  • 1 – do not apply the configuration to the Apply Process but provide the list of the tables to be applied along with the transformation rules. The trail files of the Capture Process must exist