01 Source environment — instructor notes

Budget 45 to 60 minutes of attention, and treat the module's wall clock as a separate quantity from that. The seed runs unattended for a measured 29 minutes and the terraform apply for PROVISIONAL: 8 to 12 minutes before it, so the module occupies more than an hour of the room's day while asking for under an hour of it. Module 02 is scheduled to absorb the difference; that only works if the seed starts early.

Pre-empt the row-count question. Once the writer is running, count(*) on order_items exceeds 48,000,000 and keeps climbing, and a room that has just been told the seed produces 48,000,000 will ask whether that is a bug. It is not: the seed's rows are fixed at exactly 12,000,000 x 4 and the writer's are added on top at roughly 500 a second. Send them to "What the numbers should be" in the learner Step 4 rather than answering it fifteen times; it carries the order_id <= 12000000 split query that separates the two. Saying this once, before anyone asks, costs a minute.

Expect at least one participant to report the seed as hung. It is not: each table is loaded by a single INSERT ... SELECT, so count(*) on order_items returns zero until the statement commits and then returns 48 million. Point them at pg_total_relation_size('order_items') growing towards 5,241 MB, and at /tmp/seed.log ending with ANALYZE as the only completion signal. A room that watches row counts concludes the seed has failed at about the twenty-minute mark.

The seed is the critical path, so invert the page

The learner page opens with two conceptual sections — the two-replication-tools table and "the road not taken" — and then provisions. In a room, run it the other way round. Get every participant to terraform apply, reboot, verify wal_level, load sql/01_schema.sql and start sql/02_seed.sql, and only then teach. The concepts cost nothing to deliver while a seed runs; the seed costs the room its afternoon if it starts after the talk track.

The dependency order is fixed and it is short:

1. terraform apply — PROVISIONAL: 8 to 12 minutes, unattended.
2. aws rds reboot-db-instance plus aws rds wait db-instance-available, then SHOW wal_level. Nothing later works if this returns replica, so gate on it in the room.
3. sql/01_schema.sql. Four tables, two roles, the guarded GRANT rds_replication.
4. nohup psql -f sql/02_seed.sql detached, with /tmp/seed.log to tail.

That is the whole critical path, and everything after step 4 is either concurrent with the seed or waiting on it. Say the target out loud: every participant has a seed running inside the first 25 minutes of the module. A participant who is still fighting terraform apply at that point goes to the fallback AWS account rather than debugging in the room.

One ordering trap inside those four steps, and it is the one that quietly ruins module 06: the baseline cannot be taken until the seed has finished. It has its own section below.

No writer runs before the baseline, which is deliberate and worth saying to the room if anyone has seen an older version of this module. The baseline is two readings of the same writer — alone, then under dashboard load — and they are only comparable if both are taken on a settled instance. A writer started during the seed competes with it for the same disk and reads far below its rate, so using that as the "no dashboard load" figure would credit the dashboard with throughput the seed took. Measured on db.m6g.large during the seed: about 11 TPS against a target of 200.

What to demo, what to let them do

• Demo the plan, not the apply. Put your own terraform apply plan on screen for two minutes and read the parameter group out loud: rds.logical_replication at apply_method = "pending-reboot", backup_retention_period = 1, max_replication_slots = 10. Those are the three RDS prerequisites and the retry headroom, visible in the plan before anyone has a failure to attribute to them. Then let everyone apply their own.
• Demo nothing about the seed. It is one detached command; watching it is the wrong use of a room.
• Demo the writer's output when you reach the baseline, once, on your screen: the ten-second JSON line, and specifically that --rate is a target rather than a guarantee. That framing is what makes the second reading legible minutes later, when tps has halved and nobody has changed anything.
• Let them run ./scripts/preflight.sh themselves. It is the mechanical form of the claim the whole workshop rests on, and a participant who runs it here believes it in module 06.

Talk track, while the seed runs

• The two legs do two different jobs, and both are ClickPipes. What differs is the destination type: a Managed Postgres destination for the migration, a ClickHouse destination for analytics. Expect someone to say ClickPipes only lands data in ClickHouse — that was true until the Postgres destination shipped, so it is a well-informed wrong answer rather than a careless one, and it is worth saying so. The distinction that still matters is what each leg is for: the first moves the write path and needs a cutover, the second copies for reading and needs no downtime.
• If someone asks why not hand-roll logical replication, the answer is "you can". CREATE PUBLICATION and CREATE SUBSCRIPTION remain a documented path into Managed Postgres, with no beta dependency and every step scriptable. The workshop takes the pipe because that is what a customer reaches for now, then reads the slot the pipe creates so nobody leaves thinking the primitive went away.
• The road not taken is a real answer, and saying so buys you credibility. Point ClickPipes at RDS, leave the application where it is, route the dashboard to ClickHouse: two modules, no cutover. For a team whose only problem is a slow dashboard, that is the right design. Say it before a participant says it, then say what it costs — the write path stays on the slower engine and nobody learns a cutover.
• The fact table is sized above the buffer cache on purpose. Ask why the instance is a db.m6g.large with 200 GB of default-IOPS gp3 rather than something bigger. The answer is that a dashboard query against a fact table that fits in shared_buffers measures the buffer cache, not the disk, and every number in module 06 would then be a comparison between two cache-hit rates. The measured figures, if someone asks for them: shared_buffers 1,884 MB against an order_items of 5,241 MB, with all four tables at 6,707 MB. Note that this is sized against shared_buffers, not against the instance's 8 GB of RAM — the dataset does not exceed RAM, and claiming it does invites a participant to check.

Questions worth asking, in this order:

1. "Who has a dashboard in production reading the same database as their checkout path?" Most rooms answer yes, and the rest have one and do not know.
2. "If that dashboard got slower, who would notice, and how?" The useful answer is that somebody watching the dashboard notices. Nobody watches writer p95.
3. "Would a read replica fix this?" Park it explicitly until module 06 — the answer needs the measured first row of that module's table, and answering it now costs you the punchline.

The baseline, and the mistake that survives to module 06

The most expensive error in this module produces no error message. A participant who runs bench/run.py while the seed is still going gets a clean exit 0, a plausible-looking before.json, and a measurement of a quarter-loaded fact table that mostly fits in memory. In module 06 that becomes an improvement figure nobody can explain, two hours after it can be cheaply re-taken.

Gate it in the room, on two conditions, both checkable:

• /tmp/seed.log ends with its ANALYZE. That is the only completion signal; do not let anyone gate on a row count, because order_items reads zero until its single statement commits and then reads 48 million.
• Nothing else is touching the instance when the harness starts. By this point in the module nothing should be: the dashboard is not brought up until after the baseline, precisely so there is no start-stop cycle around a measurement. run.py spawns its own app/writer.py given --writer-dsn and opens its own eight dashboard sessions, so anything else running offers load the printed config block does not record. That is a wrong benchmark, not a noisy one. Have them confirm with pgrep -f 'writer.py --dsn' and docker ps rather than by memory — the writer from Measurement 1 is the one people forget, and an orphan from a previous day is the one nobody suspects.
• Expect someone to ask why the harness needs Grafana stopped when the measurement is called "under dashboard load". Because the load is the harness's own sessions running the same panel SQL, not Grafana's. It is a fair question and answering it once, out loud, saves a room's worth of confusion — the two things are the same queries from different clients, and only one of them is counted.
• Both readings are written down before they move on. Measurement 1's TPS exists nowhere else: the harness discards its own warm-up samples, so a participant who skipped it has no "no dashboard load" number and no comparison to make.

Then have them read three things off the run before they believe it: the exit status, the WARNINGS block, and the config block. Exit 1 means a query never succeeded or the writer died mid-run; exit 2 means it never started. Comparable query set | NO and Writer held the load for the whole run | NO each mean the numbers on screen are not a measurement under load.

Tell them not to pass --dashboard-concurrency or --duration-seconds. The defaults are pinned at 8 and 300 so that every participant's two tables are the same experiment, and a room where three people tuned the harness has no comparable results to discuss.

Common failures

• terraform apply fails partway on IAM. AccessDenied on the parameter group or the security-group rule, after the instance already exists. Run terraform destroy before retrying so a half-built stack stops charging, and move the participant to the fallback account. Recovery path: AWS refuses the terraform apply.
• SHOW wal_level still returns replica. The reboot was skipped, or the parameter group still reports pending-reboot. Reboot and re-check; do not let the participant continue. Recovery.
• The seed was started before the schema. /tmp/seed.log fills with relation "customers" does not exist. Kill it, run sql/01_schema.sql, restart the seed. Cheap here, and the log is the only place it is visible because the command was detached.
• The writer exits immediately. writer: customers or products is empty means the seed has not populated those tables yet, which should not happen now the writer runs only after the seed; writer: first failure: on stderr is almost always the DSN or a missing grant. Have them read stderr rather than watching failed climb.
• tail shows an interval line where the final summary should be. Two writers are sharing /tmp/writer.log. Because each holds its own write offset, the file ends up with a hole of NUL bytes and the visible tail belongs to the other process, so the log actively misleads rather than merely confusing. pgrep -f 'writer.py --dsn' finds the extra one; rm /tmp/writer.log before restarting clears the corruption.
• Docker is not running, or WSL has eaten the host. Grafana will not start. The two recoveries are Docker inside Ubuntu and capping the WSL VM. Note that wsl --shutdown kills the detached psql running the seed: if the seed had not finished, it restarts from module 01, which is the most expensive recovery in this module.
• ./scripts/preflight.sh fails with bash\r. A CRLF checkout on Windows. Fix Git's policy and renormalize; recovery.
• Panels are empty and the participant thinks the dashboard is broken. While the seed runs the 30 and 90 day windows are genuinely incomplete. Expected, and worth pre-announcing so nobody debugs it.

End checkpoint

Do not start module 03 until, for every participant:

• SHOW wal_level is logical, BackupRetentionPeriod is at least 1, and the parameter group reads in-sync;
• the seed has finished — /tmp/seed.log ends with ANALYZE — and order_items holds tens of millions of rows;
• both baseline readings are recorded, and the standing writer from Step 7 is committing at roughly its target rate with failed at zero and exactly one process running;
• all eight panels render and ./scripts/preflight.sh passes; and
• results/before.json exists from a run that exited 0, taken after the seed finished and with no second writer running, and the participant has its config block saved beside it.

Module 02 is the legitimate place to be while any of this is still finishing. Module 03 is not: it needs a live write path and a trustworthy baseline, and a participant who starts it early spends the cutover window discovering that they have neither.