00 Setup

Choose macOS or Windows in the page header. Windows commands run inside Ubuntu on WSL 2, not PowerShell, unless the page explicitly says PowerShell.

Outcome

In 30 to 45 minutes you have an AWS account you can provision RDS in, a ClickHouse Cloud organization with an authenticated CLI, and a verified toolchain: Terraform, psql and pg_dump 17 or newer, Docker, Python 3.11 or newer, the AWS CLI, and clickhousectl. Nothing is provisioned yet. Module 01 runs terraform apply; this module ends with terraform validate passing and both AWS permission probes succeeding, so the apply cannot fail halfway through on a missing permission.

Step 1 — Accounts and region

You need two accounts:

• an AWS account in which you may create a DB instance, a DB parameter group, a DB subnet group and a security group. That is a broader permission set than a read-only or developer-sandbox role usually carries, which is what Step 6 checks.

• a ClickHouse Cloud organization at clickhouse.cloud, and the ability to create an API key in it whose role may create services. The free trial covers the ClickHouse service, the Managed Postgres instance and the ClickPipes you create in modules 03 and 04.

  The API key is the ClickHouse-side equivalent of the AWS permission caveat above, and it fails the same way: late. Module 03 creates the Managed Postgres instance with clickhousectl and module 07 deletes it, so a key that can authenticate but not create services passes Step 5's first check and strands you mid-migration. Step 5 verifies both. If your organization is centrally administered and you cannot mint a key, ask for one before the session rather than during it.

Put everything in one region. Use us-east-2 unless your trainer names another: ClickHouse Managed Postgres is in beta and its region list is narrower than RDS's, and a cross-region pair adds wide-area latency to every replication-lag number you measure in module 03. The RDS instance, the Managed Postgres instance and the ClickHouse service all go in the region you pick here.

Step 2 — Install the toolchain

Install Homebrew first if you do not have it, then:

brew tap hashicorp/tap
brew install hashicorp/tap/terraform
brew install libpq awscli python@3.12

Install Docker Desktop for Mac and start it. Docker only runs Grafana in this workshop; nothing else depends on it.

libpq is keg-only, so Homebrew does not link psql and pg_dump onto your PATH. Export it for this shell and for future ones:

export PATH="$(brew --prefix libpq)/bin:$PATH"
echo "export PATH=\"$(brew --prefix libpq)/bin:\$PATH\"" >> ~/.zprofile

brew list --formula | grep '^postgresql@' || echo "no versioned postgresql formula installed"
brew unlink postgresql@14 2>/dev/null || true

Finally, on both platforms, install clickhousectl. Module 03 creates the ClickHouse Managed Postgres instance with it rather than clicking through a beta console, and module 07 deletes it the same way:

curl https://clickhouse.com/cli | sh
export PATH="$HOME/.local/bin:$PATH"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zprofile

Step 3 — Clone the workshop repository

cd ~
git clone https://github.com/ClickHouse/ClickHouse_Demos.git
cd ClickHouse_Demos
cd "$(git rev-parse --show-toplevel)/workshops/postgres_migration"
pwd

Every later module starts from this directory, and reaches it the same way: cd "$(git rev-parse --show-toplevel)/workshops/postgres_migration".

Step 4 — Verify the toolchain

Run all of these in the shell you will use for the rest of the workshop. On Windows that is the Ubuntu terminal.

terraform version
psql --version
pg_dump --version
docker version
docker compose version
python3 --version
aws --version
clickhousectl --version

What each line must show:

pg_dump --version is the one to look at twice. It is the only check here whose failure shows up hours later, in the middle of module 04's cutover window, and its recovery is in Troubleshooting.

Step 5 — Authenticate clickhousectl

Module 03 creates the target instance with this CLI and module 07 deletes it, so it has to be authenticated before you get there rather than in the middle of a cutover.

Create an API key in the ClickHouse Cloud console under your organization's API keys, with a role that may create services, then log in. The interactive form keeps the secret out of your shell history:

clickhousectl cloud auth login --interactive

Verify the credentials and that they can actually see your organization — the second command is the one that fails if the key's role is too narrow:

clickhousectl cloud auth status
clickhousectl cloud org list

clickhousectl writes project credentials to .clickhouse/ in the current directory, so run Cloud commands from the workshop directory and never commit that folder. The repository's .gitignore already excludes it; check before you push if you have added your own.

Step 6 — Prove your AWS permissions

Configure a profile, then probe the two permissions that a read-only or narrowly scoped role most often lacks. This workshop needs to create a DB parameter group and edit a security group on top of creating the instance itself, and an apply that gets halfway there leaves you paying for orphaned resources with no working replication.

aws configure
aws configure get region
aws sts get-caller-identity
aws rds describe-db-parameter-groups --max-items 1 > /dev/null && echo "rds: ok"
aws ec2 describe-security-groups --max-items 1 > /dev/null && echo "ec2: ok"

Answer Default region name with the region from Step 1. The aws CLI does not read terraform.tfvars, so it needs its own copy.

Expected: us-east-2 from aws configure get region, your account and role from get-caller-identity, then rds: ok and ec2: ok.

Stop here if either probe prints AccessDenied or UnauthorizedOperation. Get the permissions before module 01 rather than debugging a partial apply; the escalation path and the exact actions to ask for are in Troubleshooting.

Step 7 — Fill in the Terraform variables

Every variable in terraform/variables.tf has a default except one. subscriber_cidrs has no default on purpose: it is the allowlist for port 5432, and a default would either be wrong or be 0.0.0.0/0.

cd "$(git rev-parse --show-toplevel)/workshops/postgres_migration/terraform"
printf 'region           = "us-east-2"\nsubscriber_cidrs = ["%s/32"]\n' \
  "$(curl -fsS https://checkip.amazonaws.com)" > terraform.tfvars
cat terraform.tfvars
terraform init
terraform validate

Expected: Terraform has been successfully initialized and Success! The configuration is valid. Do not run terraform apply yet.

terraform.tfvars is participant-specific and stays local. Never commit it, and never commit a terraform.tfstate file: the state contains the generated admin password.

Step 8 — What the Terraform encodes, and why

Read terraform/main.tf before you apply it in module 01. Three things in it are the RDS prerequisites for logical replication, and they are required together:

1. rds.logical_replication = 1 in a custom DB parameter group, plus an instance reboot. It is a static parameter, so the parameter group carries apply_method = "pending-reboot" and the value does nothing until the instance restarts. That reboot is the workshop's first piece of planned downtime, and module 01 verifies it with SHOW wal_level returning logical.
2. backup_retention_period >= 1. RDS only gives an instance long-term WAL retention when automated backups are on. The Terraform sets 1, the cheapest value that satisfies it.
3. The rds_replication role, granted to the replication user. The master user has rds_superuser, which is what lets it make the grant, but not rds_replication itself. sql/01_schema.sql performs the grant.

Skipping the first produces an honest, immediate failure: wal_level is replica and nothing can stream from a publication. Skipping the second or third is worse. The pipe is created, everything reports success, and rows never arrive. That is why module 01 checks all three before it starts the seed, and why the recovery for each is in Troubleshooting.

What this costs, and how to stop it

You provision into your own AWS account, so the bill is yours.

PROVISIONAL: about USD 3 in AWS charges for a six-hour run in us-east-2, replaced by a measured figure after the end-to-end run.

The breakdown, every figure PROVISIONAL:

Your ClickHouse Cloud service, the Managed Postgres instance and the ClickPipe are billed by ClickHouse, not AWS, and are not in that total.

The number that matters more than the total is the one that keeps running: an RDS instance left up costs roughly USD 4 a day until it is destroyed, and a forgotten Managed Postgres instance and ClickHouse service add to that. The last step of module 07 is the teardown sequence, listed with the rest of the modules on the learner track index: terraform destroy, then delete the ClickPipe, the Managed Postgres instance and the ClickHouse service. Run it even if you run out of time for the module it sits in.

Done when

• terraform version, psql --version, pg_dump --version, docker version, python3 --version, aws --version and clickhousectl --version all meet the table in Step 4;
• pg_dump --version reports 17 or newer;
• clickhousectl cloud auth status shows saved credentials and clickhousectl cloud org list names your organization — the second is what fails if the API key's role is too narrow;
• aws sts get-caller-identity names your account, and both permission probes print ok;
• terraform validate reports the configuration is valid, with terraform.tfvars holding your region and your own address in subscriber_cidrs; and
• on Windows, wsl --list --verbose shows Ubuntu at VERSION 2 and pwd in the checkout begins with /home/.

Next: module 01 applies this Terraform, reboots the instance for wal_level, seeds the shop database, and takes the baseline measurement you will compare everything against.