AWS S3 Apply Parameters

The following parameters are valid for the AWS S3 Apply Process:

Process Parameters

• process – The only possible value is apply
• id – The Apply Process id
• dbtype – Type of RDBMS, the only possible value is s3
• capturename – The name of the Capture Process which is providing the data to be used

AWS S3 Connectivity

• s3.accesskeyid – S3 Access Key ID
• s3.secretaccesskey – S3 Secret Access Key (See Manage access keys for IAM users for the details on Access Key ID and Secret Access Key)
• s3.region – AWS region. Default is us-east-1
• s3.bucketname – S3 bucket name

AWS S3 File Parameters

• s3.filetype – Output file format. The possible values are:
  • csv.gz – Comma-separated variable (.csv) file compressed using gzip (default value)
  • csv – Comma-separated variable (.csv) file
  • parquet.gz – Parquet (.parquet) file compressed using gzip 
  • parquet – Parquet (.parquet) file 
• s3.csvincludeheader – This parameter is valid for “csv” and “csv.gz” formats only. It determines if the column header names must be included into the output file. The possible values are:
  • 0 – do not include header
  • 1 – include header (default value)
• s3.includebeforeupdate – Specifies output format for the “update” operation. The possible values are:
  • 0 – to output “after-update” record only
  • 1 – to output “before-update“ and “after-update” records (default value)
• s3.loadincludeop  – Specifies if the column contains “type of the operation” must be included into the output file. The possible values are:
  • 0 – do not include the column
  • 1 – include the column (default value). The column name is specified by the "s3.operationcolumnname" parameter. The column has the following values:
    • “I” - insert operation
    • “D” - delete operation
    • “U” - “after-update” value for the update operation. Valid is the “s3.loadincludeop” parameter is set to 0 
    • “BU” - “before-update” value for the update operation. Valid is the “s3.loadincludeop” parameter is set to 1
    • “AF” - “after-update” value for the update operation. Valid is the “s3.loadincludeop” parameter is set to 1 
• s3.operationcolumnname – Name of the column enabled by the "s3.loadincludeop" parameter. The defaul value is __$OPERATION
• s3.csvnullsymbol – Specify output string for the null values (NULL is default).  This parameter is valid for “csv” and “csv.gz” formats only. 

Filters and Local Transformations 

• map – The set of the parameters to determine local objects’ transformation (see  Objects Mapping and Possible Transforms for the details)

Note – The S3 Apply Process doesn't support data transformation that are produced by a SQL function call. 

• filter – The set of the parameters to determine objects’ filtering (see Objects Filtering for the details)

Note - The S3 Apply Process doesn't support SQL statement as a part of filtering options

Other Parameters

• autostart – Determines if the Apply Process must be run automatically. Possible values are:
  • 0 – do not run the Apply Process automatically (default value)
  • 1 – to run the Apply Process automatically
• debuglevel – The level of debugging. Possible values are 0-15