AutoComplete DMS Lab

Introduction

Labs in the Data Engineering workshop are to be completed in sequence. This lab is designed to automate the Data Lake hydration with AWS Database Migration Service (AWS DMS), so we can fast forward to the following Glue lab.

If you prefer to get hands-on with AWS DMS service, please skip this lab and proceed to Prelab2 and DMS Main Lab.

Pre-requisite

PreLab Setup 2 is not required. Otherwise, duplicate S3 buckets and IAM roles will appear in your lab environment. If that happens, please use the resources with a name prefix “auto-dmslab-”.

  • Prelab Setup 1 - is completed and the source RDS database is fully populated.

  • RDS Database Server Name - If you are in an AWS hosted event, please ask your instructor. Otherwise, check the Outputs tab on your CloudFormation Console, note down the RDS Server value named DMSInstanceEndpoint.

  • dms-cloudwatch-logs-role & dms-vpc-role - Check if the Identity and Access Managment (IAM) roles exist in your workshop AWS account. Go to the IAM console, copy & paste the names in the search box respectively.

Note whether these role are present or not. In this example screenshot the cloudwatch role is absent.

AutoComplete DMS

  1. Click the “Deploy to AWS” icon and open the link in a new web browser tab. It will load the CloudFormation dashboard to start the DMS automation process:
Launch Template Region
N.Virginia (us-east-1)

It completes the following tasks on your behalf:

  • Set up the workshop enviroment based on PreLab Setup 2
  • Create a DMS subnet group within the VPC
  • Create a DMS replication instance
  • Create a source endpoint for RDS source database
  • Create a target endpoint for full data load
  • Create a target endpoint for CDC
  • Create a task to perform the initial full data migration
  • Create a task to support the ongoing replication of data changes(CDC)
  1. Proceed through the wizard to launch the stack. At the Specify stack details page:
  • Stack Name:
    • auto-dmslab
  • Parameters:
    • DMSCWRoleCreated: - If you have this role in your account, keep to yes. If you do NOT have the role then choose no.
    • DMSVPCRoleCreated: - no if the role doesn’t exist. Otherwise, change to yes
    • ServerName: - Enter the RDS Database Server Name. It likes this: dmslabinstance.xxxx.rds.amazonaws.com
  1. In Review page, review all the details, scroll down and check the box to acknowledge the policy and then click on Create Stack.

  2. The stack launch may take 5-6 minutes. Wait until your stack status advances to “CREATE_COMPLETE”.

  3. At this point, the source data has been fully loaded from RDS database to your S3 bucket via DMS. Go to AWS DMS console, you should see two Database migration tasks are 100% completed. If not, please wait until they are finished, then proceed to the Glue lab