S3

Destination S3 #

The extracted replicant-cli will be referred to as the $REPLICANT_HOME directory in the proceeding steps.

I. Setup Connection Configuration #

  1. From $REPLICANT_HOME navigate to the sample S3 connection configuration file

    vi conf/conn/s3.yaml
    
  2. Make the necessary changes as follows:

    type: S3
    
    access-key: "REPLICANT" #Replace REPLICANT with the access key of AWS user created from IAM management
    secret-key: "Replicant#123" #Replace Replicant#123 with the secret Key of the AWS User (note: make sure the specified user has  AmazonS3FullAccess)
    
    bucket: "bucket_name" #Replace bucket_name with the name of your S3 bucket
    root: "root_dir" #Replace root_dir with the name of the directory inside the s3 bucket where the CSV files will be stored
    
    stage:
      type: SHARED_FS
      root-dir: /home/user/stage #Enter the path of the directory where CSV files will be staged before uploading to S3
    
    #file-format: CSV #Possible file formats are CSV or JSON
    max-connections: 50 #Maximum number of connections Replicant can open in the target s3
    

Setup Applier Configuration #

  1. From $REPLICANT_HOME navigate to the sample S3 applier configuration file

    vi conf/dst/s3.yaml        
    
  2. If necessary, make the necessary changes as follows:

    snapshot:
      #threads: 16
      #max-file-size: 33_554_432 #32MB
      #delimiter: `,` #CSV files created will have provided delimiter
      #quote: `”`
      #escape:   `\`
      #include-header: false #enable or disable toggle column names as header in CSV file
    
    #realtime:
      #threads: 16
    

III. S3 File Format #

When data is being loaded into S3, the data is first converted to either a CSV or JSON file. Follow the link below to understand the data format for the CSV and JSON converted files.

S3 File Format

Note: It is highly recommended that you read the S3 file format page when using S3 as the target system.