Day 7 — Shipping It: Infra, CI/CD, Testing, Operations + Capstone

Version: 1.0 | written against code 2026-09-16 (medyzen-backend at e0b914b)

Days 1–6 covered what the code does. Today covers how it gets to production, how it is watched once it is there, and how it is tested before it leaves your laptop. At the end you build a small read endpoint from migration to test.

Every path below was opened when this lesson was written. Terraform and workflow files change often, so if a line number is off, search for the resource or step name.

---

Goals

By the end of today you can:

1. Draw the AWS layout from deploy/aws/*.tf: which subnet the API tasks run in, where RDS runs, what the ALB does, and how secrets get into the container.
2. Walk through .github/workflows/deploy-aws.yml step by step and say what fails a deploy and what does not.
3. Say why a migration that is safe on your laptop can still take production down, and when a migration belongs in ops/pending-migrations/ instead.
4. Explain testlab: the engines, what -preflight checks, what the nightly snapshot is, and how the deploy gate uses preflight.
5. Find a production problem using logs, the CloudWatch alarms, Sentry and Prometheus metrics, and know what each one can not see.
6. Pick the right test style (unit with fakes, sqlmock, testdb integration, contracttest golden files) and run it locally.
7. Add a new authenticated read endpoint to an existing module, following the house pattern rather than the out-of-date skills/ examples.

---

Reading order

#	File	What to look for
1	medyzen-backend/deploy/aws/vpc.tf	Public vs private subnets, the private route table with no default route, the three security groups
2	medyzen-backend/deploy/aws/ecs.tf	Task definition (ARM64, app_env + secrets), service in public subnets, circuit breaker
3	medyzen-backend/deploy/aws/alb.tf	HTTP→HTTPS redirect, ACM cert, /health target group check
4	medyzen-backend/deploy/aws/rds.tf	Postgres 17, private subnets, parameter group, the generated DATABASE_URL secret
5	medyzen-backend/deploy/aws/secrets.tf + iam.tf	managed_secret_names feeds both container secrets and the IAM policy
6	medyzen-backend/deploy/aws/cicd.tf	GitHub OIDC trust limited to refs/heads/main, deploy role permissions
7	medyzen-backend/deploy/aws/monitoring.tf	SNS topic, ALB/RDS alarms, the payment_anomaly log filter (read its pattern closely)
8	medyzen-backend/deploy/aws/runner.tf, dbtools.tf, ecr.tf, variables.tf, prelaunch.tfvars	Self-hosted runner, psql one-off task, image retention, default sizes vs prelaunch sizes
9	medyzen-backend/Dockerfile	Two-stage build, migrations copied into the image, non-root user
10	medyzen-backend/.github/workflows/deploy-aws.yml	guard → test (calls ci.yml) → deploy: build, migrate, preflight gate, roll, smoke
11	medyzen-backend/.github/workflows/ci.yml	build job and testlab job; the comments about shell: bash
12	medyzen-backend/.github/workflows/testlab-preflight.yml, testlab-snapshot.yml	Scheduled production preflight and nightly snapshot
13	medyzen-backend/scripts/testlab-preflight-gate.sh, scripts/check-migration-index-safety.sh, ops/pending-migrations/index.md	Deploy gate logic; index lock checks; migrations held back from auto-run
14	medyzen-backend/cmd/server/main.go (flags ~L89–130, health ~L552, admin mux ~L631–698, migrations ~L871–915), cmd/server/preflight.go, cmd/server/health.go	Command-line modes, loopback admin server, cached health probe
15	medyzen-backend/internal/testlab/index.md, internal/testlab/engine/preflight/{preflight,flags,data,storage,vendor}.go, cmd/testlab/main.go	What testlab is and what preflight checks
16	medyzen-backend/internal/pkg/logger/logger.go, internal/pkg/observability/{sentry,scrub}.go, internal/middleware/metrics.go	JSON logs, Sentry with an allow-list and PHI scrubbing, the one HTTP histogram
17	medyzen-backend/internal/database/testdb/testdb.go, internal/pkg/contracttest/contracttest.go, internal/order/repositories/order_search_test.go, internal/leads/**	The four test styles; leads is the capstone module
18	medyzen-backend/Makefile, skills/*.md	Make targets; the skills docs (and where they no longer match the code)

---

1. Infrastructure on AWS (deploy/aws)

1.1 The shape

Everything is in one Terraform root in ap-south-1. State lives in an S3 backend with a lockfile (deploy/aws/versions.tf:15-21). Every resource is tagged ManagedBy = terraform through default_tags.

Where things actually run (this was marked "not verified" in BACKEND_ARCHITECTURE.md; it is now verified):

• ECS tasks run in the public subnets with a public IP: ecs.tf:73-77 (subnets = aws_subnet.public[*].id, assign_public_ip = true). The deploy workflow's one-off tasks do the same (deploy-aws.yml, assignPublicIp=ENABLED).
• RDS runs in the private subnets: rds.tf:1-6 builds the DB subnet group from aws_subnet.private[*], and rds.tf:40 sets publicly_accessible = false.
• The private route table (vpc.tf:51-55) has no routes at all, so there is no NAT gateway. Why? A NAT gateway costs more per month than this whole prelaunch setup. Tasks reach the internet (vendors, Redis, R2) through their own public IP. They are still protected: the ecs security group only accepts port 8080 from the ALB's security group (vpc.tf:92-112).
• RDS only accepts port 5432 from the ecs security group (vpc.tf:114-127). This means any task started with the ecs security group can reach the database, including dbtools and the migration tasks.

1.2 ALB and TLS (alb.tf)

• An ACM certificate for var.domain_name (default api.medyzen.in) with DNS validation (alb.tf:1-10). The HTTPS listener waits for validation, which can take up to 60 minutes (alb.tf:64-73).
• Port 80 always returns a 301 to 443 (alb.tf:48-62). The 443 listener uses the ELBSecurityPolicy-TLS13-1-2-2021-06 TLS policy.
• drop_invalid_header_fields = true, idle_timeout = 65 (alb.tf:18-19).
• Target group: type ip (Fargate needs this), health check GET /health expecting 200 every 15s, 2 successes to be healthy and 3 failures to be unhealthy, 30s deregistration delay (alb.tf:25-46).

1.3 ECS: task definition and service (ecs.tf)

• One cluster, medyzen-backend-prod, with Container Insights on.
• Task family medyzen-backend, ARM64 on Fargate (ecs.tf:24-27). That is why the Dockerfile cross-compiles and the runner is a Graviton t4g.
• Image: <ecr>:${var.image_tag}, and image_tag defaults to "latest" (variables.tf:86-89). The deploy does not register a new task definition. It pushes a new :latest image and forces a new deployment. Remember this for Trap 4.
• Environment = every entry in var.app_env, plus CORS_ORIGINS and R2_ACCESS_KEY_ID (ecs.tf:42-48). Secrets = local.container_secrets (ecs.tf:50).
• Service: deployment_minimum_healthy_percent = 100, maximum_percent = 200. New tasks start next to the old ones, and old tasks stop only after the new ones pass health checks. The deployment circuit breaker has rollback = true (ecs.tf:85-92), so a new revision that never gets healthy rolls back automatically.
• stopTimeout = 30 gives the Go server 30 seconds to drain after SIGTERM.

1.4 RDS (rds.tf)

• Postgres 17, gp3 storage, encrypted, autoscaling up to 100 GB, 7-day backups, deletion_protection = true, and a final snapshot on destroy.
• Parameter group medyzen-backend-pg17 sets one parameter: log_min_duration_statement = 1000. Any query slower than 1 second goes to the Postgres log, which is exported to CloudWatch (enabled_cloudwatch_logs_exports = ["postgresql"]).
• The master password comes from random_password and is written into Secrets Manager as medyzen/${environment}/DATABASE_URL with sslmode=require (rds.tf:62-76). You never type it anywhere.

1.5 Secrets and IAM (secrets.tf, iam.tf)

• managed_secret_names is a plain list: JWT_SECRET, the encryption keys, the Razorpay keys, REDIS_URL, OTP_PEPPER, and others. Each name is looked up as a data source medyzen/${environment}/<NAME> (secrets.tf:92-95). Terraform does not create these secrets. They must already exist.
• The same list builds the container secrets block and the IAM GetSecretValue resource list (secrets.tf:97-116, iam.tf:22-33). The comment at secrets.tf:78-87 says why this matters: if a secret is left out of the list, the container never gets it, and a secret the role cannot read fails task start with ResourceInitializationError ("this has bitten this project five times").
• Two roles: task_execution pulls the image, writes logs and reads secrets. task is the application's own identity and has no policies attached here.
• The Sentry DSN is a plain app_env value, not a secret (variables.tf, comment above SENTRY_DSN). A DSN can only send events in, and keeping it out of the list also keeps it out of the IAM policy.
• Comments at the top of secrets.tf show that break-glass values (EMERGENCY_OTP, reviewer bypasses) are left out of the list on purpose. Leaving a name out is how you switch a credential off.

1.6 GitHub OIDC deploy role (cicd.tf)

• GitHub Actions gets short-lived AWS credentials through OIDC. No access keys are stored. The trust policy requires aud = sts.amazonaws.com and sub like repo:${var.github_repo}:ref:refs/heads/main (cicd.tf:27-48). Only workflows running on main can assume the deploy role. A PR branch cannot.
• Permissions (cicd.tf:55-158): push to the one ECR repository, register task definitions, update the service, RunTask, PassRole on the two task roles only, S3 read/write under the testlab console bucket's runs/* prefix, a CloudFront invalidation, read-only access to the app log group (so the preflight script can read the PREFLIGHT_JSON line), and sns:Publish on the alerts topic (a fallback for testlab alerts).

1.7 Monitoring (monitoring.tf)

Alarm	Metric	Fires when
unhealthy-hosts	ALB UnHealthyHostCount	≥1 for 2×60s
no-healthy-hosts	ALB HealthyHostCount	<1 for 60s; missing data counts as breaching
target-5xx	HTTPCode_Target_5XX_Count	>10 per 5 min, 2 periods
target-p99-latency	TargetResponseTime p99	>2s, 3×5min
rds-cpu / rds-free-storage / rds-connections	RDS	>80% / <2 GiB / >150
app-error-logs	custom Medyzen/Backend AppErrorLogs	>5 per 5 min

All alarms publish to the SNS topic medyzen-backend-alerts, which has an email subscription (monitoring.tf:6-14).

Read the payment_anomaly filter closely (monitoring.tf:122-133). The name suggests it matches payments, but the pattern is { $.level = "error" }. It counts every error-level JSON log line from the app, not only payment ones. The alarm on it is really "the app logged more than 5 errors in 5 minutes". This only works because the logger writes JSON with a level field (section 5.1). If you change the log format, this alarm silently stops matching.

1.8 Self-hosted runner, dbtools, ECR, sizes

• Runner (runner.tf): an AL2023 ARM64 EC2 instance in public subnet 0 with a public IP and a security group with no ingress. It is managed through SSM (AmazonSSMManagedInstanceCore). Why self-hosted? The org is on the GitHub Free plan with a $0 spending limit, and deploys were 94% of the minutes used (runner.tf:1-10). Jobs on your own runner use no GitHub minutes. It is also the only runner the whole organisation has (runner.tf:13-19), so if it is down, every repo's CI stops. user_data installs docker, gh, aws, buildx, gcc (needed for -race), and a pre-job hook that removes leftover _postgres service containers so port 5432 does not collide between jobs. Registration is manual through scripts/register-runner.sh, because a registration token expires within an hour.
• dbtools (dbtools.tf): a Fargate task definition using the postgres:17-alpine image, with entryPoint = ["/bin/sh","-c"] and DATABASE_URL injected from the secret. You override command to run a psql one-liner. Because the entrypoint is already sh -c, do not put sh -c in the override too. A doubled sh -c exits 0 with no output. ops/pending-migrations/index.md names this task as a way to run held-back migrations.
• ECR (ecr.tf): tags are mutable (the deploy reuses :latest), images are scanned on push, and only the last 15 images are kept.
• Sizes: defaults vs prelaunch.tfvars. variables.tf defaults are the "launched" sizes: db.t4g.medium, Performance Insights on, task_cpu 512 / memory 1024, desired_count 2, 30-day logs. prelaunch.tfvars shrinks them to db.t4g.micro, PI off, 256/512, 1 task, 7-day logs, plus db_apply_immediately = true. Prod is currently the downsized version. If you run terraform apply without -var-file=prelaunch.tfvars, prod scales back up and the bill goes up. Changing the RDS instance class also causes a DB restart. This is one reason exercises never run apply.

---

2. The container (Dockerfile)

• Stage 1 is golang:1.26-alpine pinned to --platform=$BUILDPLATFORM, so the compiler runs natively and cross-compiles with GOARCH=${TARGETARCH:-arm64}, CGO_ENABLED=0, -trimpath -ldflags="-s -w". The comment explains that emulating ARM through QEMU took about 25 minutes. BuildKit cache mounts keep the module cache and build cache between builds.
• Stage 2 is alpine:3.21 with ca-certificates and tzdata, and a non-root user medyzen.
• It copies internal/database/migrations into the image (Dockerfile:32). runMigrations opens file://internal/database/migrations relative to the working directory (cmd/server/main.go:887), so the same image can run ./server -migrate up as a one-off task. The migrations always match the binary's code.
• ENTRYPOINT ["./server"]. A one-off task only overrides command (for example ["-migrate","up"]), and those become flags.

The server binary has several modes (cmd/server/main.go:89-130). Each one runs a job and exits instead of serving:

Flag	Does
-migrate up / -migrate down -steps N	golang-migrate up, or down N steps (N must be >0)
-migrate-force N	Force schema_migrations to version N and clear dirty (main.go:871-884)
-preflight	Run the preflight engine, print the report and one PREFLIGHT_JSON line, exit 1 on any failure
-seed-admin / -revoke-admin	Grant or revoke super_admin by email
-reconcile-payments YYYY-MM-DD	Reconcile one IST day against Razorpay
-prune-endpoint-catalog	Sync, then deactivate api_endpoints rows no route serves
-encrypt-patient-pii [-commit]	Backfill PII ciphertext (dry run unless -commit)

---

3. CI/CD

3.1 The big picture

3.2 deploy-aws.yml step by step

Triggers are push to main and workflow_dispatch. There is one concurrency group, deploy-aws-prod, with cancel-in-progress: false, because stopping an ECS rollout halfway is worse than a wasted run.

1. guard (on the self-hosted runner, 5 min timeout).
   • Uses gh api to read the current main SHA. If it is not this run's SHA, the output is stale=true and nothing else runs. Ten quick pushes in a row lead to one deploy, not ten.
   • Works out code_changed: a force-push, a first push or a manual dispatch counts as changed. Otherwise it compares with the previous push and ignores files matching (_test\.go$|^docs/|\.md$|^\.github/). A push that only changes tests or docs still runs CI but does not roll out. Note: a change to .github/ alone does not deploy. A change to deploy/aws/*.tf does count as code, so it rebuilds and rolls the image, but it still does not apply Terraform (Trap 4).
2. test: uses: ./.github/workflows/ci.yml. Production gets exactly the checks a PR gets. The comment says the old hand-copied test job had drifted from the real one.
3. deploy: needs guard and test, and runs only if not stale and code_changed == 'true'.
   • configure-aws-credentials assumes medyzen-backend-github-deploy over OIDC.
   • docker/build-push-action with platforms: linux/arm64, provenance: false, tags :${sha} and :latest. The comments say why there is no QEMU step (Docker Hub outage pages broke builds) and no GitHub Actions cache (the cache service's outage page failed a build after the push).
   • Run database migrations: find the public subnets by tag medyzen-backend-public-* and the medyzen-backend-ecs security group, aws ecs run-task with the command overridden to ["-migrate","up"], wait tasks-stopped, then read containers[0].exitCode. If it is not 0, fail. The service has not been touched yet, so old tasks keep serving. But the database has already changed. Expand/contract rules (section 3.4) exist for exactly this moment.
   • Preflight the production environment: ./scripts/testlab-preflight-gate.sh | tee -a "$GITHUB_STEP_SUMMARY" with shell: bash (section 4.3).
   • Deploy service: update-service --force-new-deployment then wait services-stable. The new tasks pull :latest.
   • Smoke test: find the ALB's DNS name and try curl -k -H 'Host: api.medyzen.in' https://$ALB/health up to 10 times, 10 seconds apart. Anything other than 200 fails the job. The job does not roll back. ECS's circuit breaker is the rollback mechanism.

3.3 ci.yml

Triggers: pull_request, push to staging, and workflow_call (from the deploy). PR runs cancel older runs of the same PR.

build job (self-hosted, with Postgres 17 and Redis 7 service containers on random host ports):

Step	Why it exists
go build ./...	Compiles
Migration index-safety self-test + check-migration-index-safety.sh <base>	Fails on a new/changed migration that does a non-CONCURRENTLY index build, or a CONCURRENTLY build in a file with more than one statement (the file would be wrapped in a transaction and fail in prod, not CI). On a push with no resolvable previous commit (force-push) it refuses to guess and fails.
go vet ./... and go vet -tags integration ./...	Integration files are behind a build tag, so vet them separately
staticcheck with GOTMPDIR set to the workflow expression runner.temp	See Trap 5
govulncheck	Known-vulnerable dependencies
go test -tags integration -race -count=1 -coverpkg=./... with TEST_DATABASE_URL/REDIS_TEST_URL pointing at the services	The real test gate
Coverage by package → step summary	Reporting
upload-artifact with continue-on-error: true	Trap 3
PR only: diff-cover with --fail-under 70	Changed lines must be ≥70% covered
make swagger then git diff --exit-code internal/docs/	Committed OpenAPI docs must match the annotations

testlab job (self-hosted, with a throwaway Postgres):

1. Sets fake-but-valid config through env: a 32+ char JWT secret, rzp_test_… keys, hex encryption keys, RBAC_WHITELIST_MODE=enforce, METRICS_ENABLED=true, ADMIN_PORT=6060.
2. go run ./cmd/server -migrate up, then builds and starts bin/server in the background, polling /health for up to 60s.
3. ./bin/server -preflight with PREFLIGHT_ENV=ci.
4. Builds bin/testlab and runs the engines, each piped into tee -a $GITHUB_STEP_SUMMARY with shell: bash:
   • authz -mutating: every role × every endpoint, compared with role_permissions.
   • security -mutating -concurrency 4: auth stack checks.
   • contract: response shapes.
   • tenant: seeds two warehouses and checks one cannot read the other's rows.
   • load -metrics http://127.0.0.1:6060/metrics -vus 15 -duration 20s -error-rate 0.01.
5. Consumer graph: checks out the 7 frontend repos with read-only deploy keys, then testlab consumer -strict. It only gates when every consumer was actually read (scripts/testlab-consumer-armed.sh). Otherwise it only reports, because a pass that looked at nothing proves nothing.
6. On a PR from this repo: serve this run's results, fetch main's nightly snapshot from S3 (runs/latest/*.json), testlab diff, and post or update a single PR comment marked <!-- testlab-diff -->.
7. Upload findings (continue-on-error: true).

-mutating is only safe here because the database is a container that gets thrown away (see the comment in ci.yml above the authz step).

3.4 Migration safety

The four facts that matter:

1. Migrations run automatically on every production deploy, before the rollout, from the new image. Nobody reviews them in between.
2. Old code runs against the new schema for a while: from the end of the migration task until services-stable, and permanently if the circuit breaker rolls back. So a migration has to be additive: add a nullable column, a new table, or an index built CONCURRENTLY. Renames, drops and NOT NULL without a default must wait for a later deploy, after no running code reads the old shape.
3. golang-migrate wraps a multi-statement file in a transaction. CREATE INDEX CONCURRENTLY cannot run in a transaction, so it must be the only statement in its file. See internal/database/migrations/295_add_orders_order_no_trgm_index.up.sql: one line, CREATE INDEX CONCURRENTLY IF NOT EXISTS .... Its .down.sql is also a single DROP INDEX CONCURRENTLY IF EXISTS. CI enforces this (scripts/check-migration-index-safety.sh:1-28).
4. A migration that fails partway leaves schema_migrations.dirty = true, and every later -migrate up refuses to run until someone fixes it by hand (Trap 8).

ops/pending-migrations/ holds SQL that is deliberately kept out of the auto-migrate directory. Read the README. The reasons show up again and again:

• CREATE UNIQUE INDEX CONCURRENTLY must be run with psql in autocommit mode, not through the migration CLI (221, 234, 290).
• A person must look at the data first. For example, 290 must stop if duplicate phone blind indexes exist, because those are two real patient accounts and must not be merged just to get an index built.
• A data fix needs a value only a person can supply (201 has a 1970 sentinel that raises until someone replaces it with migration 103's real apply time).

You run them through the medyzen-backend-dbtools one-off task or psql. When one is applied, the README records it (234 is marked APPLIED 2026-08-30).

Also: migration numbers must go up. golang-migrate only tracks the highest version it has applied. If a higher-numbered migration reaches prod first, a lower-numbered one merged later is silently never applied. Push them together or lowest first. Current highest: 295.

---

4. Testlab and preflight

4.1 What testlab is

Testlab is a set of test engines that live inside the backend repo (internal/testlab, about 11k lines) and a CLI, cmd/testlab. They live here because they need internal/ packages that no other module is allowed to import: the route Collector, token minting, the public-route list, and the whitelist exempt list (internal/testlab/index.md, "Why this lives in the backend repo"). The console UI is a separate repo (medyzen-testlab) that reads JSON from testlab serve, which only binds to loopback.

Subcommands (cmd/testlab/main.go:58-80): authz, catalog, tenant, load, contract, security, preflight, cost, consumer, serve, schema, diff.

Key idea in the authz engine: an oracle works out what the answer should be from api_endpoints/role_permissions/role_assignments (public, exempt, allow, deny). A probe with a real minted token records what actually happens. Only a 401 or 403 counts as a denial. A 400 or 404 means the request got past the gate.

4.2 Preflight: checking the environment, not the code

internal/testlab/engine/preflight/preflight.go:1-15 says it directly: the other engines assume the environment is right and test the code. Preflight assumes the code is right and tests the environment, because that is where the real outages came from: empty warehouse_service_areas, a blank SMS template, an R2 token scoped to the wrong buckets, a mail vendor out of credit.

Each check has an id, a severity, the symptom a user would see, an owner and a fix. The verdicts are pass, fail, warn, skip. skip is not a pass. A check that cannot run here says why.

Group	File	Examples
flags.*	flags.go	flags.jwt_secret not the public default, flags.dev_master_otp unset, flags.emergency_otp not armed, flags.rbac.whitelist_mode enforced, flags.cookie_secure, flags.swagger_disabled, flags.pprof_disabled, flags.sentry_dsn, flags.payment.key_mode (live key in prod, test key elsewhere), flags.careers_retention_sweep
data.*	data.go	data.migrations.clean (not dirty), data.migrations.current (DB version ≥ highest file in the build), data.super_admin, data.warehouse_service_areas, data.products
storage.*	storage.go	A real write to each R2 bucket (assets, kyc, restricted_docs)
vendor.*	vendor.go	2Factor balance, ZeptoMail auth/credit, Redis ping, Razorpay accepts the live key

The environment comes from PREFLIGHT_ENV, or is guessed: SENTRY_ENVIRONMENT=production → prod, GITHUB_ACTIONS=true → ci, otherwise local (preflight.go InferEnv). Rows marked prodOnly are skipped outside prod.

How it runs (cmd/server/preflight.go:27-71): the server binary runs the engine, because only a task from the real task definition holds the database URL, R2 credentials and vendor keys. It prints a human-readable report, then exactly one line PREFLIGHT_JSON {...}, and exits 0/1/2. The JSON must stay on one line, because the publishing script pulls it out of CloudWatch with a per-line sed (a test, TestTheMarkerLineIsOneLine, keeps it that way).

4.3 Where preflight runs

• Deploy gate (scripts/testlab-preflight-gate.sh): it blocks on a regression, not on a failure. A check that passed in the baseline and now fails, or a new check that fails, blocks the deploy. A check that was already failing sends an alert but lets the deploy through (a gate that blocked until every old failure was fixed would also block the fixes). The baseline is gate-baseline.json, which only moves forward after a run that passed the gate, so a retry cannot turn a regression into "pre-existing". Exit 2 (no output at all) also blocks.
• Scheduled (testlab-preflight.yml): cron 15 2,14 * * * (twice a day, UTC). Runs scripts/testlab-preflight-prod.sh, then checks that the published console snapshot runs/latest/meta.json is no more than 36 hours old, and alerts via scripts/testlab-notify.sh (Slack webhook, with SNS as the fallback) if the run itself failed.
• CI (ci.yml testlab job): ./bin/server -preflight with PREFLIGHT_ENV=ci.
• Local: make testlab-preflight ENV=local.

4.4 The nightly snapshot (testlab-snapshot.yml)

Cron 30 20 * * * UTC. It sets up a fresh Postgres service container, migrates it, starts the API with fake config (SMS_PROVIDER=none, push off), checks out the consumer repos, and runs scripts/testlab-snapshot.sh. That script runs authz/contract/load/security/tenant against BASE (default http://localhost:8080), publishes to S3 runs/… plus runs/index.json, invalidates CloudFront, and sends an alert when counts regress. This snapshot is the "main" baseline that PR comments compare against. It never touches the production database.

---

5. Observability

5.1 Logs (internal/pkg/logger/logger.go)

• logrus with a JSONFormatter writing to stdout. The awslogs driver ships it to /ecs/medyzen-backend (30-day retention by default, 7 days with the prelaunch sizes).
• logger.WithContext(ctx) adds request_id when the RequestID middleware has set one. Use it in request paths so one request's lines can be found together.
• Level comes from LOG_LEVEL (info in prod). An Error-level line counts toward the app-error-logs alarm (§1.7), so do not log expected client mistakes at error level.

5.2 Sentry (internal/pkg/observability/sentry.go, scrub.go)

• Init does nothing if SENTRY_DSN is empty (sentry.go:45-48). Tracing is off, SendDefaultPII: false.
• PHI is removed before sending, and fields are kept only if allow-listed. beforeSend (sentry.go:99-144):
  • drops event.Request and ServerName completely;
  • runs Redact over the message and exception values;
  • keeps only User.ID;
  • keeps only the tag keys in allowedTagKeys, and only the medyzen context keys in allowedExtraKeys (request ids, and payment identifiers like order_no and razorpay_payment_id that do not identify a patient). Other contexts are limited to runtime/os/device/trace, and those are sanitised.
• Redact (scrub.go:57-67) replaces JWTs, Bearer tokens, emails, Indian mobile numbers and any run of 6+ digits with [redacted]. IsSensitiveKey matches key names containing phone, otp, token, address, patient, prescription, name, and many more.
• observability.Middleware clones a hub per request and tags method and a sanitised endpoint (UUIDs, numbers and opaque path segments are replaced).
• Use observability.CaptureError(ctx, err, extra). Only allow-listed extra keys survive, so if you want a new field to show up it must be added to allowedExtraKeys, and a reviewer should confirm it is not PHI.

5.3 Metrics and the admin port

• /metrics, /swagger/ and /debug/pprof/ are never on the public mux. They run on a separate server bound to 127.0.0.1:${ADMIN_PORT} (cmd/server/main.go:631-698). The comment says why: pprof profiling is an easy DoS, and a heap dump would contain decrypted patient data and the JWT secret. Each is switched on separately with METRICS_ENABLED, SWAGGER_ENABLED, PPROF_ENABLED. config.ValidateAdminPort refuses to start if the ports are misconfigured.
• The public ALB → task path cannot reach loopback, so in prod metrics can only be read from inside the task. testlab's load engine reads them in CI at 127.0.0.1:6060.
• App metrics: one histogram, medyzen_http_request_duration_seconds{method,route,status} (internal/middleware/metrics.go). It uses r.Pattern as the route label so the number of label values stays small. Metrics must be inside Audit in the middleware chain, otherwise every route shows as unmatched (main.go comment just above the handler := line, around L662-674). There are also DB pool and Redis pool collectors (main.go ~L625-629).

5.4 Health (cmd/server/health.go)

• GET /health pings Postgres and Redis (if configured). It returns 200 {"status":"ok"} or 503 "service unavailable", and does not say which dependency failed in the response. That detail is only in the log.
• Results are cached for 2 seconds using double-checked locking, with a fixed 2-second probe timeout. Why: /health is public and not rate-limited. A flood of health requests could fill up the same DB pool that OTP verification uses, and a pool timeout on the OTP path counts as a failed login. The ALB polls this endpoint every 15 seconds.

---

6. Testing patterns

Style	Where to see it	Needs	Use for
Unit with fakes	internal/leads/services/lead_test.go (fakeLeadRepo embeds repositories.LeadRepository, stubLeadAccess)	nothing	Service rules, error mapping, authorization checks
sqlmock	internal/order/repositories/order_search_test.go (GORM over sqlmock.New(), regex-matched SQL + args)	nothing	Checking the exact SQL, e.g. that an OR does not break the hospital scope
testdb integration	internal/leads/repositories/lead_integration_test.go with //go:build integration, testdb.New(t)	TEST_DATABASE_URL	Real Postgres behaviour: constraints, JSON columns, migrations
contracttest golden	internal/leads/dtos/contract_test.go + testdata/*.json	nothing	Freezing the JSON shape of DTOs that frontends read
Route registration	internal/leads/routes/routes_registration_test.go	nothing	ServeMux panics on conflicting patterns; catch that before boot

testdb (internal/database/testdb/testdb.go):

• It never calls config.Load(), because that reads the real .env and could point tests at a real database. Its only input is TEST_DATABASE_URL, and if that is unset, tests skip.
• setup runs every migration once per test binary (sync.Once), then New(t) gives each test its own transaction that is rolled back in t.Cleanup.
• Shared(t) returns the real connection pool, for race tests that need two committed transactions. You must clean up anything you write through it.

contracttest: contracttest.RunAll(t, []contracttest.Case{contracttest.For[dtos.X]("name")}) fills every field with a known value, serialises it, and compares with testdata/name.json. If it fails, a wire format changed. Regenerate with go test ./internal/<mod>/dtos -update only after checking every consumer of that payload. CI removes internal/pkg/contracttest/ from coverage.

Running tests locally:

make test                                   # go test -race -count=1 ./...  (integration tests skip)
createdb medyzen_test                        # a throwaway DB, never your dev DB
make test-integration TEST_DATABASE_URL="postgres://postgres:postgres@localhost:5432/medyzen_test?sslmode=disable"
make coverage                                # same profile CI reports
make lint                                    # vet + staticcheck
make swagger-check                           # regenerate OpenAPI and fail on diff
go test ./internal/leads/...                 # one module

Other useful Makefile targets: make run, make migrate-up / migrate-down, make docker-up (the compose API listens on 8080, Postgres on host port 5434), make testlab, make testlab-authz BASE=…, make testlab-preflight ENV=local, make prune-endpoint-catalog.

---

Traps

1. Pushing to main deploys production, and migrations run automatically.deploy-aws.yml:3-6 triggers on push: branches: [main]. The deploy job runs -migrate up against the production database before the rollout, with no human approval step in the workflow. The OIDC trust (cicd.tf:45) is exactly what lets main do this. Work on a branch, open a PR against staging or main, and only merge once you mean "this goes live". The team's promote flow is staging → merge into main.

2. cmd | tee in GitHub Actions passes even when cmd fails, unless the step has shell: bash. The default run: shell is bash -e without pipefail, so engine | tee -a $GITHUB_STEP_SUMMARY returns tee's exit status, which is always 0. The comment in ci.yml above the authz step records that every testlab engine step used to pass no matter what it found. Naming shell: bash gives you -eo pipefail. Every | tee step in ci.yml and the preflight gate step in deploy-aws.yml has it. When you add a step, check it. a | grep -q x is fine as a gate, because the command whose status matters is last in the pipe.

3. actions/upload-artifact can fail a green build. Artifact storage is an org-wide quota. When it is full, the upload step fails, the ci job fails, and the deploy that needs it is skipped, even though every test passed. Every upload step in ci.yml (coverage, diff-coverage, testlab-findings) has continue-on-error: true. Keep it that way for any new diagnostic upload. A diagnostic upload must never be able to block a deploy.

4. The deploy does not apply Terraform, so a new app_env variable or secret may never reach the task.deploy-aws.yml (and scripts/deploy-manual.sh / deploy-daemonless.sh) build the image, migrate and roll the service with --force-new-deployment. They reuse the existing task definition. A value you add to variables.tf app_env or secrets.tf managed_secret_names gets committed, reviewed and "deployed", but it is still missing from the running container until someone registers a new task definition revision. The deploy is green and /health returns 200, but the feature is effectively off. This happened with CAREERS_RETENTION_SWEEP_ENABLED. The reliable check is aws ecs describe-task-definition on the service's current revision (read-only). A better fix is to add a preflight flags.* check so a missing variable fails the gate (flags.careers_retention_sweep exists for this reason). Do not fix it with a plain terraform apply, because that also removes the prelaunch downsizing (§1.8).

5. The runner's /tmp is a small tmpfs, so set GOTMPDIR. On the self-hosted runner, /tmp is a tmpfs of about 1 GB. Large Go builds and staticcheck fail with no space left on device while df / shows plenty of free space. ci.yml's Static analysis step sets GOTMPDIR to the workflow expression runner.temp (on the root disk). According to the team's ops notes, GOTMPDIR has since also been set for the whole host in the runner's .env. That host setting is not in runner.tf and was not verified from code. Keep the per-step setting in workflows anyway, in case the runner is rebuilt from user_data. Also remember that the root disk filling up shows the same error and is a different problem.

6. The first integration run on a fresh TEST_DATABASE_URL can flake.go test ./... builds one test binary per package and runs them in parallel. Each binary has its own sync.Once and runs m.Up(). The comment in testdb.go says golang-migrate's advisory lock makes them wait for each other. Even so, the team has seen a cold first run fail about 10 unrelated tests (assets/hospital), then pass with -p 1, and pass in parallel once the schema was up to date. The exact cause of the race is not verified from code. In practice: if a brand-new test DB gives you unrelated failures, migrate it first (go test -tags integration -p 1 ./internal/database/...) or just re-run before you go looking for a bug in your change.

7. A stale process on :8080 produces false testlab violations. The tell is 401.scripts/testlab-snapshot.sh defaults BASE=http://localhost:8080 and only polls /health. If some other server already holds 8080, yours exits with bind: address already in use (easy to miss in a background process), /health still succeeds against the other process, and every engine tests it. Tokens signed with your JWT_SECRET get rejected, and you get hundreds of allow → deny "violations". The tell: the status is 401 across every role, including super_admin (403 would be a real RBAC decision; 401 means the token was rejected), and tenant probed 0 pairs. Before running, check lsof -nP -iTCP:8080 -sTCP:LISTEN. Do not kill a process you did not start. Use PORT=8090 / BASE=http://localhost:8090 instead. A snapshot run like this can publish to the console and send a real SNS alert, so do not run the publishing scripts from a laptop.

8. A dirty schema_migrations blocks every deploy until someone runs -migrate-force. If a migration fails partway (a DO $$ … RAISE assertion, a lock timeout), golang-migrate records the version with dirty = true. Every later -migrate up refuses to run, the deploy's migration step exits non-zero, and nothing deploys, including the fix. Preflight's data.migrations.clean reports it as Critical. The fix: look at what the failed migration actually changed, repair it by hand (through dbtools/psql), then run a one-off task with -migrate-force <N> (cmd/server/main.go:871-884). N is the last version that is fully applied. This only clears the flag. It does not run or undo any SQL. Choosing the wrong N either skips a migration or re-runs one. Prevent it by making data migrations idempotent and by moving anything that needs a person to look first into ops/pending-migrations/.

9. The payment_anomaly filter is not about payments. Its pattern is { $.level = "error" } (monitoring.tf:125). It counts every error log. See §1.7.

10. skills/*.md is out of date. All three skills files show gin (c *gin.Context, router.Group), middleware.RequireAuth() / RequireRole(...), and YYYYMMDD_ migration names. The code uses stdlib net/http ServeMux patterns registered through router.Registrar, middleware.Auth(jwtSecret, blacklist) plus a database-backed RBAC whitelist, and sequential NNN_ migration numbers. Use the skills files as a checklist of steps, and copy the actual code from a real module. The capstone does this.

11. Swagger annotations cover almost nothing. Only internal/dashboard/handlers/* have // @Router annotations. CI's docs check only fails when an annotation changed and internal/docs/ was not regenerated. It does not check that new routes have annotations.

---

Exercises

All local. Never terraform apply, never push to main, never run the testlab-*-prod/snapshot/publish scripts, never point anything at production.

1. Map the network (read-only). From vpc.tf, ecs.tf, rds.tf and runner.tf, list every security group and exactly what it accepts. Then answer: can the runner EC2 instance connect to RDS on 5432? (Look at the rds security group's ingress.) If not, how does a deploy migrate the DB?
2. Plan without applying. Only if you have been given read-only AWS credentials, run terraform -chdir=deploy/aws init and then terraform -chdir=deploy/aws plan -var-file=prelaunch.tfvars -lock=false. Note any drift. Now think through (do not run) what a plan without the var-file would change. If you have no credentials, work it out from variables.tf vs prelaunch.tfvars.
3. Reproduce the tee trap. Write a scratch workflow snippet on paper, or in a local shell run bash -e -c 'false | tee /dev/null; echo still here' and then bash -eo pipefail -c 'false | tee /dev/null; echo still here'. Explain the difference in one sentence.
4. Break the index-safety gate. On a local branch, add internal/database/migrations/296_scratch.up.sql containing CREATE INDEX idx_scratch ON leads(email); and a matching down file. Commit locally and run ./scripts/check-migration-index-safety.sh HEAD~1. Read the error. Change it to CONCURRENTLY and add a second statement to the same file, and run it again. Delete the branch afterwards.
5. Run preflight locally. Start Postgres (make docker-up or a local instance), blank the secrets in your .env, run go run ./cmd/server -migrate up, then go build -o bin/server ./cmd/server && PREFLIGHT_ENV=local ./bin/server -preflight. Count pass/fail/warn/skip. Choose two skips and explain why each could not be checked locally.
6. Health cache. Read cmd/server/health_test.go. Start the server, stop Postgres, and time how long before /health returns 503. Relate it to healthCacheWindow.
7. Scrubber. Add a temporary local test (do not commit) that calls observability.Redact("call +91 9876543210 re order 1234567 token eyJabc.def.ghi"). Predict the output first. Why is 1234567 redacted, and what does that mean for putting order numbers in error messages?
8. Dirty migration drill (throwaway DB only). In a scratch DB, create 296_fail.up.sql with CREATE TABLE scratch_a(id int); SELECT 1/0;. Run -migrate up, then look at SELECT version, dirty FROM schema_migrations. Try -migrate up again. Check whether scratch_a exists (the implicit transaction may have rolled it back; drop it if present), delete the file, and running -migrate-force 295. Confirm dirty = false. Drop the scratch DB.

---

Capstone: add GET /leads/api/v1/leads/stats

Goal: a super_admin-only read endpoint that returns lead counts per status, optionally filtered by source_page. Response: {"success":…,"data":{"new":N,"contacted":N,"converted":N,"total":N}} (use the real envelope response.Success produces).

Work on a local branch (git switch -c capstone/leads-stats). Do not push it.

Follow the steps from skills/add-api-endpoint.md, skills/add-auth-middleware.md and skills/database-migration.md, but use the real code patterns shown below (Trap 10).

Step 1 — Migration (the RBAC catalog row)

The leads table and idx_leads_status already exist (038_create_leads.up.sql:18), so no schema change is needed. What is needed is authorization. RBAC is default-deny, and the endpoint needs a row in api_endpoints.

• Boot already upserts every collected route into api_endpoints (EndpointSyncService.Sync, internal/human/services/endpoint_sync.go, called in main.go ~L601). The house convention still registers new admin routes in a migration, so the catalog and any grants ship together and are reviewable. See 258_grant_careers_rbac.up.sql.
• super_admin is let through by PermissionStore.Decide without an explicit grant (see the comment in 258), so insert no role_permissions row. Do it the way 258 does: add a DO $$ block that asserts the endpoint row exists and that 0 grants exist.
• Name it 296_register_leads_stats_endpoint.up.sql (the next number, not YYYYMMDD_). Include ON CONFLICT (endpoint_key) DO NOTHING so it can run more than once safely. The .down.sql deletes that endpoint_key.
• Do not create an index without CONCURRENTLY, and do not put a CONCURRENTLY index in a multi-statement file (§3.4).
• Run make migrate-up against your local DB and check the row.

Step 2 — Model

No new model. Reuse internal/leads/models.Lead and the status constants StatusNew/StatusContacted/StatusConverted.

Step 3 — DTO

In internal/leads/dtos/lead.go, add a LeadStatsView struct with json tags new, contacted, converted, total. Add contracttest.For[dtos.LeadStatsView]("lead_stats_view") to internal/leads/dtos/contract_test.go, then run go test ./internal/leads/dtos -update once to create testdata/lead_stats_view.json. Read that JSON file. It is the contract the super-admin panel will depend on.

Step 4 — Repository

In internal/leads/repositories/lead.go:

• Add CountByStatus(ctx context.Context, sourcePage string) (map[string]int64, error) to the LeadRepository interface.
• Implement it on leadRepo the same way List does: database.DBFromContext(ctx, r.db).WithContext(ctx).Model(&models.Lead{}), an optional Where("source_page = ?", …), then Select("status, count(*) AS n").Group("status").Scan(&rows) into a small struct slice.
• Remember the ops note about raw scans: always use ? parameters and never string-concatenate sourcePage.

Step 5 — Service

In internal/leads/services/lead.go, add Stats(ctx, actor common.Actor, sourcePage string) (dtos.LeadStatsView, error):

• First line: if err := s.requireLiveSuperAdmin(ctx, actor); err != nil { return …, err }. The service checks authorization again even though RBAC already did, the same as List does.
• Call the repo, fill all three statuses (a status with no leads must show 0, not be missing), and compute total.
• Wrap repository errors with the existing wrap(ctx, err).

Step 6 — Handler

In internal/leads/handlers/lead.go, add Stats, based on List:

• actor, ok := request.Actor(r); if !ok, response.ServiceError(w, r, svcerrors.ErrNotAuthenticated).
• Read source_page from r.URL.Query().
• On error response.ServiceError(w, r, err), otherwise response.Success(w, view).
• Take the identity only from the request context, never from the body or query (skills/add-auth-middleware.md checklist, and it is still true).
• Add a swagger block in the format of internal/dashboard/handlers/warehouse_summary.go:20-30 (@Summary, @Tags leads, @Security BearerAuth, @Param source_page query string false …, @Success 200 {object} …LeadStatsView, @Router /leads/api/v1/leads/stats [get]).

Step 7 — Route + auth

In internal/leads/routes/lead.go, add:

mux.Handle("GET /leads/api/v1/leads/stats", authMw(http.HandlerFunc(h.Stats)))

• It must be wrapped in authMw (middleware.Auth). That middleware validates the JWT and then runs checkWhitelist (internal/middleware/auth.go:118), which looks up r.Pattern in the permission store.
• Do not add it to publicPatterns.
• /leads/api/v1/leads/stats does not conflict with anything, because the only wildcard route is /leads/{id}/status with a different shape. routes_registration_test.go will catch it if you are wrong.

Step 8 — Tests

1. Unit (services/lead_test.go): with fakeLeadRepo (add a countByStatus field and method), test that (a) a non-super-admin actor gets ErrForbidden and the repo is never called; (b) missing statuses come back as 0; (c) total is the sum; (d) a repo error is wrapped.
2. Integration (repositories/lead_integration_test.go, already //go:build integration): db := testdb.New(t), seed with the existing seedLead helper across two source_pages, and check the counts with and without the filter. Other tests may have left rows in the shared DB, so compare against a count taken before seeding rather than absolute numbers, or use a unique source_page for your test.
3. Contract: done in Step 3.
4. Run:

go build ./... && go vet ./... && go vet -tags integration ./...
go test ./internal/leads/...
make test-integration TEST_DATABASE_URL="postgres://postgres:postgres@localhost:5432/medyzen_test?sslmode=disable"
./scripts/check-migration-index-safety.sh main
make swagger && git diff --stat internal/docs/

Step 9 — Prove the gate locally with testlab

lsof -nP -iTCP:8090 -sTCP:LISTEN          # must be empty (Trap 7)
go build -o bin/server ./cmd/server && go build -o bin/testlab ./cmd/testlab
SERVER_PORT=8090 RBAC_WHITELIST_MODE=enforce ./bin/server &   # local .env with blanked secrets
./bin/testlab authz -base http://localhost:8090 -json /tmp/authz.json

Find the GET /leads/api/v1/leads/stats cells. Every role except super_admin should be deny with a 403. super_admin should be allow. If every role shows 401, you are testing the wrong server. Stop the server you started (by PID, not pkill).

Step 10 — Self-review against the CI list

Tick each one: build, index-safety, vet (both tags), staticcheck, tests with -race, changed-line coverage ≥70%, swagger in sync, contract golden reviewed, no PHI in logs or Sentry extras, no new env var (if you had added one: Trap 4 plus a preflight flag). Then delete the branch or keep it local. Do not push.

---

Self-check

1. In which subnets do the API Fargate tasks run, and in which does RDS run? How do the tasks reach external vendors without a NAT gateway?
2. Which Terraform local builds both the container secrets block and the IAM policy for reading secrets, and what happens at task start if a secret is missing from it?
3. Why can a PR branch not assume medyzen-backend-github-deploy?
4. List the steps of the deploy job in deploy-aws.yml in order.
5. A push changes only deploy/aws/variables.tf to add a new app_env entry. Does the image roll out? Does the running task get the variable?
6. Why must CREATE INDEX CONCURRENTLY be the only statement in its migration file?
7. What does the preflight deploy gate block on, and what does it deliberately let through?
8. What does the payment_anomaly metric filter actually count?
9. Why are /metrics and /debug/pprof/ bound to 127.0.0.1 on a separate server?
10. In testlab output, every role including super_admin shows allow → deny with status 401. What most likely happened?
11. The deploy fails at "Run database migrations" and schema_migrations.dirty = true. What do you do, and what does -migrate-force N not do?
12. Why does testdb refuse to use config.Load(), and what does a test get from testdb.New(t)?

---

Answers

1. Public subnets with assign_public_ip = true (ecs.tf:73-77). RDS is in the private subnets via the DB subnet group, with publicly_accessible = false (rds.tf:1-6,40). The tasks use their own public IP and the internet gateway. The private route table has no routes, and the ecs security group only accepts 8080 from the ALB.
2. local.managed_secret_names (secrets.tf:25-89) feeds both container_secrets and secret_arns. A name not in the list is never injected and is not readable by the execution role. A secret the role cannot read fails task start with ResourceInitializationError.
3. The OIDC trust policy's sub condition is repo:<repo>:ref:refs/heads/main (cicd.tf:42-46).
4. Checkout → OIDC credentials → ECR login → buildx → build and push arm64 :sha + :latest → one-off -migrate up task (exit code must be 0) → preflight gate → update-service --force-new-deployment + wait stable → smoke test /health via the ALB.
5. The image does roll out (.tf files are not in the ignore list). The variable does not reach the task, because the deploy never runs Terraform or registers a new task definition. Check with describe-task-definition, and add a preflight flag check.
6. golang-migrate sends a file as one query string. Postgres wraps a multi-statement string in an implicit transaction, and CONCURRENTLY cannot run inside a transaction. It would fail in prod mid-deploy, and CI's check-migration-index-safety.sh catches it.
7. It blocks on regressions: a check that passed in gate-baseline.json and now fails, a new check that fails, or no output at all (exit 2). Checks that were already failing send an alert but do not block, so fixes can still ship.
8. Every JSON log line from the app with level = "error", not only payment errors (monitoring.tf:125).
9. pprof profiling is an easy DoS and a heap dump contains decrypted patient data and the JWT secret. Loopback-only means that even if someone turns the flags on, nothing outside the container can reach them (main.go:631-637).
10. You probed some other process. Most likely a stale server on the port, so your server failed to bind and your JWTs were rejected (401 means the token was rejected, not an RBAC decision). Check lsof and use another port.
11. Look at what the failed migration actually applied, repair or undo it by hand on the DB, then run a one-off -migrate-force <last fully-applied version>, then redeploy. -migrate-force only sets the version and clears dirty. It runs no SQL and undoes nothing.
12. config.Load() reads the real .env and could point tests at a real database. testdb reads only TEST_DATABASE_URL and skips if it is unset. New(t) returns a *gorm.DB bound to a transaction on the migrated schema, rolled back when the test ends.