Day 2 — Request Path & Security

Version: 1.0 | Status: written against code 2026-09-16 (medyzen-backend at e0b914b)

Yesterday you watched cmd/server/main.go boot the process. Today you follow one HTTP request from the socket to a handler and back. On the way you learn how a patient, a staff member and a rider get a session, how that session is checked on every call, how it is refreshed and revoked, who is allowed to call what, and what gets written down about it.

Most of the security in this backend is not in one place. It is spread across boot checks, a stack of global middleware, per-route middleware, and service code. The aim today is to hold all of it in your head as one picture.

---

Goals

By the end of today you can:

1. Name the global middleware in the order they wrap, and say why each sits where it does.
2. Trace a patient phone-OTP login and a staff cookie refresh end to end, including which Redis keys and database rows they touch.
3. Explain the three token artefacts (access JWT, refresh token, blacklist entry) and when each is created, checked and killed.
4. Tell the difference between the three authorization layers: entity type checks, the role_permissions whitelist, and in-service capability checks.
5. List every login bypass in the code (dev master OTP, emergency OTP, store-reviewer accounts) and the guard on each.
6. Say what refuses to boot, and why, for JWT secret, trusted proxy secret, CORS, Redis and the OTP pepper.
7. Explain how the audit trail is written without any handler remembering to write it.

---

Reading order

Open these in order. Paths are relative to medyzen-backend/.

#	File	What to look for
1	cmd/server/main.go lines 138–171	Boot checks: JWT secret, trusted proxy secret, CORS, OTP pepper, emergency OTP, Redis.
2	cmd/server/main.go lines 660–674	The one line that builds the global middleware chain, and the comment above it.
3	internal/config/config.go lines 208–260	ValidateJWTSecret, ValidateTrustedProxySecret, ValidateCORSOrigins.
4	internal/middleware/security.go, cors.go, request_id.go, recovery.go, timeout.go, logging.go, metrics.go	Small files. Note which ones write a response and which only decorate.
5	internal/middleware/audit.go + internal/pkg/audit/recorder.go	Denylist of unaudited paths; a pointer in the context that deeper layers fill in.
6	internal/middleware/auth.go	Bearer vs cookie, the X-Mz-Client CSRF gate, trying every cookie candidate, blacklist check, then two gates.
7	internal/middleware/whitelist.go	PermissionStore.Decide, exempt patterns, ValidateWhitelistMode.
8	internal/middleware/hospital_pages.go	The extra gate for hospital attender accounts.
9	internal/middleware/rbac.go	RequireEntityType, RequireHospitalScope, ForbidKioskHeaderForEntityType.
10	internal/middleware/rate_limit.go, keyed_rate_limit.go, redis_limiter.go	IP limiter, keyed limiter, the Redis token-bucket Lua script and its memory fallback.
11	internal/human/common/auth.go, refresh.go, roles.go, role_priority.go, permissions.go, actor.go	JWT claims, refresh-token generation/hash, role lists, capability map.
12	internal/human/common/otp.go, otp_pepper.go, redis.go	OTP store: Redis Lua scripts vs in-memory maps, lockout, HMAC fingerprint.
13	internal/human/common/token_blacklist.go	Per-token logout and per-subject deactivation, fail-closed on Redis error.
14	internal/human/common/emergency_otp.go, delivery_otp.go	Break-glass code with a 24-char floor; the delivery handover OTP hash.
15	internal/human/services/master_otp_dev.go, master_otp_prod.go, master_otp_build_test.go	How DEV_MASTER_OTP is compiled out of release builds.
16	internal/human/services/phone_auth.go	Patient phone login, reviewer account, sign-up on first verify.
17	internal/human/services/staff_phone_auth.go, employee.go (VerifyOTP ~line 216)	Staff login, uniform answers for unknown accounts.
18	internal/human/services/token.go, token_role.go	IssuePair, Rotate, RotateForEntity, RotateForRole, reuse detection, family max age.
19	internal/human/handlers/auth.go, session.go, phone_auth.go + internal/human/routes/*.go	Refresh and logout handlers, cookie writing, which routes are rate-limited vs authenticated.
20	internal/pkg/authcookie/authcookie.go	Cookie names per client, paths, why there can be several cookies with one name.
21	internal/pkg/acting/acting.go, internal/pkg/ctxkeys/ctxkeys.go	Kiosk "acting for a patient" resolution; the request-id context key.
22	internal/pkg/crypto/aead.go, internal/pkg/pii/pii.go, internal/pkg/maskpii/maskpii.go, internal/pkg/turnstile/turnstile.go	AES-GCM, blind indexes, masking for logs, Cloudflare Turnstile.
23	internal/audit/ (module, routes, services)	The read side of the audit log: super-admin only, bounded queries.
24	internal/database/db.go lines 27–53	ParameterizedQueries and the global RecorderParamsFilter override.

---

Explanation

1. Before any request: the boot checks

A lot of the security posture is decided before the server listens. cmd/server/main.go calls logger.Log.Fatal (the process exits) when config is unsafe. The idea: a misconfigured security control should stop the deploy, not quietly run in a weaker mode.

Check	Where	Refuses to boot when	Why
JWT secret	config.ValidateJWTSecret (config.go:213)	empty, equal to DefaultJWTSecret ("change-me-in-production"), or shorter than 32 bytes	Anyone who knows the key can mint an access token for any account. The default is public in the repo.
Trusted proxy secret	config.ValidateTrustedProxySecret (config.go:239)	non-empty and shorter than 32 chars	Empty is allowed on purpose: it turns the proxy-auth path fully off. A short one would be a weak control that looks armed.
CORS	config.ValidateCORSOrigins (config.go:253)	any entry is *	middleware.CORS implements * by echoing back the caller's Origin, which is the same as allowing every site.
OTP pepper	main.go:154–157	REDIS_URL set and OTP_PEPPER empty	Codes go to Redis as an HMAC of the code. With no pepper the HMAC key is empty and a six-digit code is trivially recoverable.
Redis reachable	main.go:158–161	REDIS_URL set but connect fails	Better to fail than to run with per-instance OTP and logout state.
Whitelist mode	middleware.ValidateWhitelistMode (whitelist.go:377, called at main.go:604)	mode is off or log without RBAC_ALLOW_WEAK_WHITELIST=true, or any unknown string	The whitelist is the only role-level gate. A typo would silently allow everything.
Emergency OTP	common.SetEmergencyOTP (emergency_otp.go:19)	does not fail boot; shorter than 24 chars logs an error and stays disabled	See Traps.
SMS provider	smsProviderSelected / newSMSProvider (main.go ~1165–1190)	unknown SMS_PROVIDER, or twofactor without key/template	A provider that cannot deliver a code breaks every phone login.

When REDIS_URL is unset the server still boots. It logs:

REDIS_URL is not set: OTP store and token blacklist are per-instance memory; do not run more than one instance

That is the dev/CI path. Keep it in mind for the Traps section.

Note the RATE_LIMIT_BACKEND setting (default memory, config.go:370). Even when Redis is connected, the rate limiters only use it if this is redis (main.go:165–167). The OTP store and blacklist use Redis whenever REDIS_URL is set.

2. The global middleware chain

Everything the public server does is wrapped in one expression at cmd/server/main.go:674:

handler := mw.SecurityHeaders(mw.CORS(cfg.CORSOrigins)(mw.RequestID(observability.Middleware(mw.Recovery(mw.Timeout(cfg.RequestTimeout)(mw.Logging(mw.Audit(auditWriter)(mw.Metrics(mux)))))))))

Read it from the outside in. A request passes through them in this order, and the response comes back in reverse:

What each one does and why it sits there:

Layer	File	What it does	Why this position
SecurityHeaders	middleware/security.go	Sets nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer, a CSP of default-src 'none', and HSTS for two years.	Outermost, so even a CORS preflight or a panic response carries the headers. This is a JSON API, so default-src 'none' costs nothing.
CORS	middleware/cors.go	Echoes Origin only if it is in the allowlist; sets allowed methods and headers (Authorization, X-Kiosk-Id, X-Mz-Client, ...); sets Allow-Credentials: true when not wildcard. Answers OPTIONS with 204 and stops.	Preflights must be answered before auth, since browsers send no credentials on preflight. Side effect: preflights are never logged or audited.
RequestID	middleware/request_id.go	Takes X-Request-ID or makes a UUID, stores it under ctxkeys.RequestID, echoes it in the response.	Everything inside can log the same id.
observability.Middleware	internal/pkg/observability	Sentry integration. Not covered today (Day 7).	Inside RequestID so events carry the id.
Recovery	middleware/recovery.go	recover(), sends the panic to Sentry, logs stack, writes a 500.	Outside Audit, so a panicking request still gets an audit row (Audit writes in a defer).
Timeout	middleware/timeout.go	Only puts a deadline on the request context (REQUEST_TIMEOUT_SECONDS, default 30). It does not write a 503.	DB calls using ctx stop when the deadline passes; the handler decides what to answer.
Logging	middleware/logging.go	One request log line with method, path, status, duration. Its statusWriter also implements Flush so the SSE feed works.	Not deferred, so a request that panics produces the panic recovered line but no request line.
Audit	middleware/audit.go	Parks a mutable *audit.Entry in the context, writes one row in a defer after the handler returns.	Outside the mux so it sees the final status; inside RequestID so the row has the id.
Metrics	middleware/metrics.go	Prometheus histogram labelled by r.Pattern (the route template).	Must be directly on the mux. The comment at main.go:664–673 explains: Audit replaces the request with r.WithContext(...), so anything outside it reads a request the mux never set Pattern on, and every route showed as unmatched.

/metrics, /swagger/ and /debug/pprof/ are not on this mux at all. They live on a separate admin server bound to 127.0.0.1 (main.go:630–657, default ADMIN_PORT 6060), because pprof can dump memory that holds decrypted patient data and the JWT secret.

3. Per-route middleware: which routes are public

Global middleware runs for every route. Authentication does not. Each module's routes package wraps its own handlers. In internal/human/routes/routes.go:

authMw := middleware.Auth(jwtSecret, blacklist)
rl := authLimiter.Limit

and then, for example (routes/phone_auth.go, routes/auth.go, routes/employee.go):

Route	Wrapped with	Meaning
POST /human/api/v1/phone-auth/otp/send	rl	Public, IP rate-limited.
POST /human/api/v1/phone-auth/otp/verify	rl	Public, IP rate-limited.
POST /human/api/v1/phone-auth/refresh	rl	Public; the refresh token in the body is the credential.
POST /human/api/v1/auth/refresh	rl	Public; the refresh cookie is the credential. Used by the web panels.
POST /human/api/v1/auth/logout	rl	Public; possession of the refresh cookie is enough. Rate-limited because it writes one DB row per cookie sent.
POST /human/api/v1/employees/phone-otp/send / verify	rl	Public staff login.
POST /human/api/v1/delivery-partners/refresh	rl	Rider refresh.
GET /human/api/v1/users/me	authMw	Needs a valid access token.
POST /human/api/v1/employees/register	authMw(rl(...))	Authenticated and rate-limited.

The authLimiter is middleware.NewRateLimiter(ctx, AUTH_RATE_LIMIT, AUTH_RATE_BURST, TRUST_PROXY) (human/module.go:112), defaults 10/s with burst 20 (config.go:356–357).

4. The Auth middleware in detail

internal/middleware/auth.go is the most important 140 lines of the day. Step by step:

1. Find candidate tokens. If there is an Authorization header it must be Bearer <token> or the answer is 401 invalid authorization format. With no header, it reads all access cookies for this client via authcookie.ReadAllAccess(r) (line 37).
2. No candidates → 401 missing authorization header.
3. CSRF gate for cookies (line 46). If the token came from a cookie and the method is not GET/HEAD/OPTIONS, the request must carry an X-Mz-Client header, else 403 missing client header. A cross-site form post cannot set a custom header, so cookies alone cannot change state.
4. Try each candidate (lines 61–85). For each: common.ValidateToken (HMAC only, checks expiry), parse sub as UUID, then blacklist.CheckRequest(ctx, token, subject, iat). The first candidate that passes wins. Why loop: a browser can hold several cookies with the same name (see authcookie.ReadAllFor comment) and the stale one is sent first.
5. No winner → 401 with one of three messages: account is deactivated, token has been revoked, or invalid or expired token. Deactivation is a 401, not a 403, because the shared api-client only tries refresh-then-logout on 401 (comment lines 88–93).
6. Put claims in the context under ClaimsKey and the raw token under RawTokenKey. Handlers read them with middleware.GetClaims(ctx).
7. Tell the audit row who is acting: audit.SetActor(ctx, subject, claims.EntityType, claims.Role) (line 115). This is why every authenticated request is attributed without handler code.
8. Whitelist gate: checkWhitelist (line 118).
9. Attender page gate: checkHospitalPages (line 121).
10. Call the handler.

The JWT itself (internal/human/common/auth.go):

type Claims struct {
	jwt.RegisteredClaims
	Email      string `json:"email"`
	EntityType string `json:"entity_type"`
	Role       string `json:"role"`
}

• Signed HS256 with JWT_SECRET. ValidateToken rejects any non-HMAC alg (line 44), which blocks the classic "alg: none" / RS-vs-HS confusion.
• sub is the document id (a UUID), never the internal integer id.
• entity_type is one of employee, hospital_operator, user (roles.go:3–7). A rider is an employee with the delivery_partner role.
• role is a single string. An account can hold many roles; the token carries the highest one by RolePriority (next section).
• Lifetime is JWT_EXPIRY_HOURS, default 1 (config.go:344).

claims.Actor() turns the claims into a common.Actor{DocID, EntityType, Role} (actor.go), which is what services take as "who is calling".

5. Roles, role priority and three kinds of authorization

Roles. internal/human/common/roles.go lists which roles are valid for each entity type. IsValidAuthRoleFor("hospital_operator", "super_admin") is false. That map stops a token that claims an impossible combination from getting capabilities (HasCapabilityAs).

Role priority. role_priority.go orders roles from super_admin down to app_user. HighestPriorityRole(names) picks the first match. tokenRole (services/token_role.go) loads the subject's role names from role_assignments and calls it. On a lookup error it mints a token with an empty role (and logs a warning) instead of failing the login.

The order is not alphabetical and not "most powerful". interviewer is deliberately second from bottom: a warehouse manager who also interviews must still get warehouse_manager in the token, or the warehouse panel sends them to a home route they do not have. The comment calls these insertion points locked; role_priority_test.go pins them.

Three layers of authorization. Do not confuse them.

Layer	Where	Granularity	Source of truth
Entity type / scope	middleware/rbac.go: RequireEntityType, RequireHospitalScope, ForbidKioskHeaderForEntityType; also appOnly wrappers in human/routes	"Only patients", "only operators of hospital {id}"	Token claims + HospitalScoper.EnforceHospital
Endpoint whitelist	middleware/whitelist.go via Auth	Per route pattern, per role	DB tables read by PermissionStore (role_assignments, role permissions), cached
Capabilities	common/permissions.go: HasCapability, HasCapabilityAs	Actions inside a handler: payout:approve, hr:document:restricted, ...	Hard-coded map in Go

The whitelist gate (whitelist.go:333 Decide): load the subject's role ids from role_assignments (30-second per-subject cache, line 27), then for each role whose entity type matches the token's entity type: allow if the role is super-admin, or if the role's permission set contains the endpoint key. The endpoint key is r.Pattern, e.g. GET /human/api/v1/users/me. So adding a route means someone must grant it to roles, or every non-super-admin gets 403 insufficient permissions.

Details worth knowing:

• The decision uses roles from the database, not the role claim. A role removed from an account stops working within the 30-second cache, even with a still-valid token.
• A small set of self-service routes is exempt (whitelistExemptPatterns, line 361): me, logout, and similar.
• Invalidation across instances goes over a Redis pub/sub channel rbac:invalidate (line 31) when Redis is set.
• On a DB error Decide denies (line 336).
• In log mode it logs the denial and lets the request through. That is why boot refuses log unless you opt in.
• The route catalogue is synced into the DB at boot (EndpointSyncService.Sync, main.go:601). Pruning stale rows is a separate -prune-endpoint-catalog run.

The attender page gate (hospital_pages.go:138) only applies when entity_type == hospital_operator and role == attender. For mapped routes (hospitalPageRoutes, line 19) the operator must have at least one of the listed page keys assigned, else 403. On lookup failure it denies. Note this one does use the role claim.

Capabilities are the finest layer. HasCapability(role, cap) returns true for super_admin always. hr_manager gets most HR caps but deliberately not hr:document:restricted, payroll:approve or payroll:pay, so "the person computing a payroll is never the person releasing the money" (permissions.go:40–47).

Kiosk acting. internal/pkg/acting/acting.go decides whose data a request is about when a hospital operator at a kiosk acts for a patient (X-Patient-Id header plus a kiosk id). No header means "myself". A patient token with a kiosk id is refused. Any other entity with a kiosk id is refused. With a header, the binder must confirm the operator is at that kiosk and the patient is bound to it. Day 3 uses this.

6. The OTP store

internal/human/common/otp.go. One process-wide singleton (NewOTPStore uses sync.Once). Every auth surface shares it, so keys are scoped:

Key prefix	Used by	File
login:+91...	patient phone login	services/phone_auth.go:113
chg:<docId>:+91...	patient phone change	phone_auth.go:117
empp:+91...	employee phone login	staff_phone_auth.go:15
hop:+91...	hospital operator phone login	staff_phone_auth.go:16
usr:, emp:, ho:, dpe: + email	email OTP per surface	services/otp_keys.go

Without scopes, a code sent to an email for the patient app would work on the staff login for the same email (otp_keys.go:5–7).

Two back-ends.

With Redis (REDIS_URL set):

• Generate stores HMAC-SHA256(OTP_PEPPER, key || 0x00 || code) at otp:code:<key> with TTL OTP_TTL_SECONDS (default 600). The raw code never reaches Redis.
• VerifyWithStatus runs otpVerifyScript (Lua, atomic): if otp:lock:<key> has a TTL, return locked; if the stored fingerprint matches, delete code and fail counter, return OK; else INCR otp:fail:<key>; at 5 failures set otp:lock:<key> for 15 minutes.
• CheckWithStatus is the same but does not delete the code. ConsumeExclusive deletes it and returns true only to the caller whose DEL removed something. The hospital operator flow uses check → do lookups → consume exclusive, so a failed kiosk lookup does not burn the code, and two parallel verifies with the same code cannot both win (staff_phone_auth.go:174–227).
• Any Redis error on verify means "wrong code" and increments medyzen_otp_redis_failure_total{op=...}. A patient with the right code sees a failure. The health check drains the task if Redis is down (main.go:528–540).

Without Redis:

• Codes are kept as plain strings in s.entries (line 210). The pepper is not used. That is acceptable only because this memory never leaves the process.
• Lockout counts live in s.attempts. A cleanup goroutine clears expired entries each minute.
• Every instance has its own maps. See Traps.

Codes are six digits from crypto/rand (generateCode, line 448). Generate refuses to issue if the random source fails.

Send-side limits. Separate from lockout. human/module.go:72 builds otpSendLimiter, a keyed limiter at OTP_SEND_PER_HOUR (default 12) per key otp:send:<e164> or otp:send:<scoped email>. SMS sends also go through a daily budget (SMS_DAILY_SEND_CAP).

Delivery OTP (common/delivery_otp.go) is a different thing: the code a rider asks the patient for at handover. It is stored as HMAC(DELIVERY_OTP_PEPPER, salt || code) with a per-order random salt and at most 5 attempts. Boot refuses DELIVERY_OTP_ENFORCE=true with an empty pepper (main.go:336–338). Day 3 covers the order side.

7. Login bypasses: three, each with a guard

Bypass	Config	Accepts	For which logins	Guard
Dev master OTP	DEV_MASTER_OTP	one code for any account	email OTP verify for users, employees, hospital operators (user.go:147, employee.go:218, hospital_operator.go:230)	Only compiled when built with -tags dev. master_otp_prod.go (//go:build !dev) always returns false. master_otp_build_test.go fails CI if Dockerfile, Makefile, ci.yml or deploy-aws.yml mention a dev tag.
Emergency OTP	EMERGENCY_OTP	one code for any staff account with that account's real role	employee email + phone, hospital operator email + phone, delivery partner (staff_phone_auth.go:77,172, employee.go:219, delivery_partner_auth.go:114,226). Not patients.	Must be ≥ 24 chars (MinEmergencyOTPLen) or stays disabled. Compared in constant time. Every use logs a Warn. Boot logs a Warn while active.
Store reviewer	REVIEWER_PHONE_E164 + REVIEWER_OTP; DELIVERY_REVIEWER_*	one fixed code for one phone number	patient phone login; rider phone login	Both values required; code ≥ 24 chars (minReviewerCodeLen, phone_auth.go:34); failures still count toward lockout (VerifyStaticWithStatus). Patient and rider pairs are separate so one cannot sign into the other app.

The emergency bypass skips the OTP store entirely (if !emergencyMatch { ...verify... }), so it is not subject to lockout. That is one reason for the length floor.

8. Sequence: patient phone OTP login

This is the patient app path. Routes in human/routes/phone_auth.go, service in human/services/phone_auth.go, handler in human/handlers/phone_auth.go.

Points to notice:

• A patient token carries an empty role (IssuePair(..., EntityUser, "", meta), phone_auth.go:199). Patient routes are gated by entity type and the whitelist, not by the role claim.
• First successful verify creates the account. There is no separate sign-up.
• An old unverified row with clinical history cannot be claimed by whoever now owns the number: ErrPhoneClaimNeedsSupport (phone_auth.go:225–227).
• The mobile app uses the JSON body tokens. The cookies are set too, which matters for web callers.

9. Refresh tokens and rotation

common/refresh.go: a refresh token is 32 random bytes, base64url. The database stores only sha256(raw) hex. A DB dump does not give usable refresh tokens.

services/token.go issues and rotates. Each row has a family id (one login = one family) and family_issued_at. Rules on rotate (Rotate, line 191):

Situation	Result
Hash not found	401 invalid token
Row already revoked	Reuse detected: revoke the whole family (RevokedReasonReuse), 401. A stolen token used after the real client rotated kills both.
Expired (REFRESH_TOKEN_DAYS, default 30)	401
Family older than REFRESH_TOKEN_FAMILY_MAX_DAYS (default 90)	revoke family, 401. Forces a real login at least every 90 days.
Subject inactive (for operators: also hospital not active)	revoke all of subject's tokens, ErrAccountDeactivated
Revoke-by-hash affects 0 rows (lost a race)	treat as reuse, revoke family, 401
OK	revoke old row (rotated), issue a new pair in the same family, role re-read from DB

Three flavours:

Method	Called by	Extra rule	Role in new token
Rotate	POST /human/api/v1/auth/refresh (web panels)	none	tokenRole → highest priority role from DB (users: empty)
RotateForEntity	POST /human/api/v1/phone-auth/refresh (patient app)	row's entity type must be user	empty
RotateForRole	POST /human/api/v1/delivery-partners/refresh (rider app)	must be employee and still hold delivery_partner	always delivery_partner

10. Sequence: panel refresh with cookies

The web panels use @medyzen-health/api-client from medyzen-shared. It sends X-Mz-Client (the hospital panel sends '2', medyzen-hospital/src/api/client.ts:16) and withCredentials: true.

If rotate returns 401 or 403, the handler clears both cookies for that client (handlers/auth.go:49–52), and the api-client treats 401/403 from refresh as "session dead" and signs out.

11. Cookies (internal/pkg/authcookie)

Cookie	Path	Content
mz_access or mz_access_<client>	/	access JWT
mz_refresh or mz_refresh_<client>	/human/api/v1/auth	raw refresh token

• <client> comes from X-Mz-Client (or ?mz_client= for EventSource, which cannot set headers). It must match ^[A-Za-z0-9_-]{1,16}$. Separate names let two panels on the same domain keep separate sessions.
• All cookies are HttpOnly. Secure is forced when SameSite=None. Defaults: COOKIE_SECURE=true, COOKIE_SAMESITE=lax (config.go:349–350).
• The refresh cookie's narrow path means the browser only sends it to refresh/logout, not to every API call.
• LegacyDomain exists because narrowing a cookie's domain does not delete what browsers already hold; every write also expires the old wide cookie.
• Reads return all values under a name, stale one first. That is why both Auth and Refresh loop over candidates.

12. Logout, blacklist and deactivation

common/token_blacklist.go has two kinds of revocation, both checked in one Redis pipeline by CheckRequest:

Kind	Redis key	Set by	Rejects
Token logout	jwt:blacklist:<sha256(token)>, TTL = remaining token life	Add on logout	that exact access token
Subject revoked	subject:revoked:<docId> = RFC3339 timestamp, TTL = access token TTL	MarkSubjectRevoked on deactivation / role change	any token for that subject with iat ≤ timestamp

Key behaviours:

• Fail closed. A Redis error in CheckRequest returns (true, true), i.e. reject (line 243–246). "Rejecting a still-valid session is recoverable, honouring a revoked one is not."
• A token with no iat is treated as revoked once any subject marker exists (subjectRevokedAsOf).
• The subject marker is a timestamp, not a flag, so reactivating an account does not resurrect old tokens; a fresh login has a later iat.
• If Redis write fails in MarkSubjectRevoked, the revocation is kept in local memory on this instance and the error is returned so the admin does not see a false success.

Logout (handlers/auth.go:66 for the cookie endpoint): blacklist every valid access token presented, revoke every refresh cookie sent (checking it belongs to the caller when a valid access token identifies them; otherwise revoke by possession), clear cookies. The per-entity logout routes (/users/logout, /employees/logout, ...) sit behind authMw and do the same through the service.

NewTokenBlacklist is a singleton, so every module that calls it (audit module, delivery KYC, human) shares one instance.

13. Staff phone login and enumeration

services/staff_phone_auth.go. The concern: a login form that says "no such number" tells an attacker who works here.

• SendOTPByPhone returns nil (HTTP 200 "OTP sent") when there is no active employee with that number, and when more than one shares it. It sleeps otpEnumerationDelay (250 ms, employee.go:169) so timing matches a real send.
• VerifyOTPByPhone answers an unknown number exactly like a wrong code, with the same delay.
• The lookup is GetActiveByPhoneUnique(ctx, e164). Staff phone numbers are normalized to E.164 on every write (staff_phone_normalize.go). A number stored as ten bare digits cannot match.
• The "unknown number" log line is at Debug. With the default LOG_LEVEL=info (config.go:354) you will not see it. The ambiguous case logs at Error.
• The same pattern applies to email verify for employees (employee.go:228–240) and delivery partners.

14. Rate limiting

Two limiter types, one Redis script.

RateLimiter (rate_limit.go) keys on IP.

• connectingIP: with TRUST_PROXY=true, the last X-Forwarded-For hop (the one the AWS ALB appended, which a client cannot forge). Otherwise the socket address.
• NewProxyAwareRateLimiter (careers) can key on X-Medyzen-Client-IP from the landing page's Next.js proxy, but only when the caller proves it is the proxy: its connecting IP is in CAREERS_TRUSTED_PROXY_CIDRS, or it sends X-Medyzen-Proxy-Auth equal to TRUSTED_PROXY_SECRET. An empty secret is checked before subtle.ConstantTimeCompare, because that function treats two empty slices as equal (lines 245–250).

KeyedRateLimiter (keyed_rate_limit.go) keys on anything a KeyFunc returns (phone, email, token hash). An empty key goes to a shared __unkeyed__ bucket rather than being allowed.

Back-end (redis_limiter.go): a Lua token bucket stored in a Redis hash (rl:ip:<ip>, rl:key:<key> or a custom prefix). Used only after mw.SetRedis ran, i.e. REDIS_URL set and RATE_LIMIT_BACKEND=redis. If the Redis call fails it logs a warning and falls back to the in-process golang.org/x/time/rate limiter. Rate limiting fails open to per-instance, unlike the OTP store and blacklist which fail closed. A limiter degraded to per-instance is still a limiter.

15. Audit trail

Two halves.

Write side (internal/middleware/audit.go + internal/pkg/audit/recorder.go):

• Every request gets a row unless its path starts with /health, /metrics, /debug/pprof/ or /swagger/. This is a denylist on purpose: a new route is audited by default.
• Audit puts *audit.Entry in the context. Deeper code fills it: Auth calls SetActor; handlers call audit.SetSubject(ctx, type, id) (first one wins) and audit.SetAction(ctx, "prescription.download").
• Why a pointer: a context value added deeper cannot be seen by an outer middleware. A shared pointer can.
• Writer.Record never blocks. It pushes to a buffered channel (default 4096); batches of 128 or every 2 s go to InsertBatch. A full buffer increments dropped and logs at Error. Close drains on shutdown.
• Row fields (internal/audit/models/audit_log.go, table audit_log): time, request id, actor id/type/role, subject type/id, method, route pattern, path, status, action, action ref, source IP, user agent, duration.

Read side (internal/audit): one route, GET /audit/api/v1/audit. AuditService.Query requires entity_type=employee and role=super_admin in the token and re-checks super-admin against live role assignments. It refuses unbounded queries (needs subject, actor or from-date) and defaults to the last 90 days. Reading the audit log is itself audited.

16. PII helpers

Package	What it gives you	Use it when
internal/pkg/crypto	AEAD (AES-256-GCM, random nonce, base64 output; refuses empty plaintext), BlindIndex (HMAC over a normalized value), Equal	Low-level. You rarely call it directly.
internal/pkg/pii	Cipher built from PATIENT_ENCRYPTION_KEY + PATIENT_BLIND_INDEX_KEY; Encrypt/Decrypt, PhoneIndex (last 10 digits), PhoneLast4Index, EmailIndex, NamePrefixIndex (first 3 letters); process-wide Register/Active for GORM hooks; Encrypt backfill	Anything that stores or looks up patient identity. Lookups go through blind indexes, never plaintext WHERE phone = ?.
internal/pkg/maskpii	Tail, TailPtr, ContainsMask, Email	Showing or logging an account number, Aadhaar or email. ContainsMask refuses a PATCH that echoes a masked value back.
internal/pkg/turnstile	Client.Verify against Cloudflare siteverify	Public forms (careers). Verified server-side, because a POST straight to the ALB skips the Next.js proxy. Any error means reject (fail closed).

Boot refuses to start if only one of the two patient keys is set, or neither (main.go:245–255), because the models no longer map the plaintext columns. The patient keys are separate from the KYC keys used for employee Aadhaar/bank data (pii.go:18–22). Day 6 covers the employee side.

Two more log-safety details:

• Phone numbers in logs go through phone.Mask, emails through maskpii.Email. Look at any log line in staff_phone_auth.go.
• GORM is configured with ParameterizedQueries: true so slow-query logs show $1, not the value (database/db.go:52). See Traps for the .Raw().Scan() hole and its fix.

---

Traps

Each of these has been verified against the code on 2026-09-16.

Trap 1 — DEV_MASTER_OTP must never reach production

DEV_MASTER_OTP authenticates any user, employee or hospital operator by email OTP. It only works in a binary built with -tags dev (services/master_otp_dev.go). The production file (master_otp_prod.go, //go:build !dev) ignores it.

The danger is not the env var alone, it is the pair: a dev-tagged build plus the env var. The dev tag has come close to production before. internal/order/payment_gateway_dev.go (the mock payment gateway) is also behind dev, so "I just want the mock gateway in CI" drags in the master OTP. Both .github/workflows/ci.yml (~line 322) and testlab-snapshot.yml (~line 79) carry comments about avoiding exactly this; master_otp_build_test.go fails if a release file mentions a dev tag.

Rule: never set DEV_MASTER_OTP in any deployed environment, and never add -tags dev to Dockerfile, Makefile release targets or deploy workflows.

Trap 2 — Emergency OTP is env-driven, long, and still dangerous

It used to be a six-digit constant in the repo. Now it is EMERGENCY_OTP, and SetEmergencyOTP keeps it disabled if shorter than 24 characters. Note: a short value does not fail boot; it logs an Error and the bypass is off. If staff say "the emergency code doesn't work", check the length first.

While active, one code signs into any staff, operator or rider account with that account's real role. An admin email plus this code is an admin session, and it skips lockout. The startup Warn line (EMERGENCY_OTP bypass ACTIVE) and per-use Warn lines are how you notice. Target state is empty.

Trap 3 — Staff phone login: unknown number gets a 200 on purpose

POST /human/api/v1/employees/phone-otp/send returns 200 "OTP sent" for a number that belongs to nobody, or to two active employees. This is anti-enumeration, not a bug. When someone reports "I got 200 but no SMS":

1. Is the stored number E.164 (+91...)? Staff lookups match on the normalized form only.
2. Is it shared by two active employees? That logs at Error: more than one active employee shares this number.
3. The "no active employee" line is Debug and invisible at the default LOG_LEVEL=info.

Do not "fix" the endpoint to return 404.

Trap 4 — The panel role is overwritten on every refresh

AuthHandler.Refresh returns role = HighestPriorityRole of the account's current DB roles (token.go:439, role_priority.go). The panels' onRefreshed callback stores it: if (role) localStorage.setItem('role', role) (medyzen-hospital/src/api/client.ts:19–21; super-admin has the same hook in warehouseClient.ts).

So if a panel logs someone in with a role it chose (or a role that is not their highest-priority one), the first refresh, about an hour later, silently replaces it. Routing guards keyed on that role then send the user elsewhere or log them out. A panel's expected role must be exactly what refresh would return for that account. This is also why RolePriority insertion points are locked: re-ordering changes what every refresh returns.

The rider app avoids this by using RotateForRole, which always returns delivery_partner.

Trap 5 — Without REDIS_URL, OTP and logout state are per-instance memory

With REDIS_URL unset:

• A code generated on instance A does not exist on instance B. Behind a load balancer, verify fails about half the time.
• Lockout counts are per instance, so an attacker gets 5 guesses per instance per 15 minutes.
• A logout or deactivation on A does not reject the token on B until it expires.
• OTP codes are held in plain text in memory (pepper unused).

The server only logs a Warn and keeps running. That is fine for a single local process. In any environment with more than one task it is a security hole. Production sets REDIS_URL (the comment at main.go:528 says "production always"). Also remember the separate RATE_LIMIT_BACKEND=redis switch: Redis connected does not mean rate limits are shared.

Trap 6 — GORM .Raw().Scan() bypasses ParameterizedQueries

ParameterizedQueries: true in the GORM logger config keeps bound values (patient phone, names) out of slow-query logs. But db.Raw(...).Scan(dest) does not use that logger: GORM swaps in a package-level logger.Recorder whose default RecorderParamsFilter ignores the setting and inlines values. The codebase has 100+ such calls, some on patient tables (for example hasClinicalHistory in phone_auth.go:278).

The fix lives in internal/database/db.go:27–41: an init() that replaces gormlogger.RecorderParamsFilter globally so it returns the SQL without values. Do not delete that init, do not move DB setup to a package that skips it, and do not assume ParameterizedQueries alone is enough when you read a GORM upgrade's release notes. If a slow-query log ever shows a literal phone number, this is the first place to look.

Trap 7 — Smaller ones worth knowing

• Deactivation is 401, not 403. Changing it breaks the api-client's sign-out path (auth.go:88–93).
• Cookie POSTs need X-Mz-Client. A new web client without it gets 403 missing client header on every write, while GETs work. It looks like an RBAC bug; it is the CSRF gate.
• New route → 403 for everyone but super-admin until it is granted in role permissions (whitelist enforce). Check r.Pattern spelling, including the method prefix.
• The audit row's source_ip uses the first X-Forwarded-For hop (recorder.go:317–323), which the client controls, while the rate limiter and session.go use the last hop. Do not use audit_log.source_ip as proof of where a request came from.
• Timeout does not abort the handler. It only cancels the context; a handler that ignores ctx runs to completion.
• CORS preflights are neither logged nor audited, because CORS answers them before RequestID.
• docker-compose.yml sets JWT_SECRET: change-me-in-production for the api service, which ValidateJWTSecret rejects. Override it locally. The compose file also has no Redis service.

---

Exercises

All exercises run on your laptop. Never point any of this at production. config.Load() reads .env; before starting, make sure your .env has no real secrets (blank RAZORPAY_*, ZEPTO_API_KEY, TWOFACTOR_*, R2 keys, DATABASE_URL pointing at local Postgres only).

Setup used below:

cd medyzen-backend
docker compose up -d db
docker run -d --name mz-redis -p 6379:6379 redis:7-alpine

Generate throwaway values with openssl rand -hex 32 (for 64-hex keys) and openssl rand -base64 32 (for secrets). Do not reuse anything from a deployed environment.

Exercise 1 — Watch the boot checks refuse

Run go run ./cmd/server four times, changing one variable each time, and write down the fatal message:

1. JWT_SECRET=change-me-in-production
2. JWT_SECRET set to 20 characters
3. CORS_ORIGINS=*
4. REDIS_URL=redis://localhost:6379 with OTP_PEPPER= empty

Then set a valid 32+ char secret, a local pepper, and EMERGENCY_OTP=short. Confirm the server starts and logs the "refusing to enable a bypass shorter than 24" error. Question for yourself: why is this one a log line and not a fatal?

Exercise 2 — Draw the middleware chain from headers

With the server running (valid local config, REDIS_URL set):

curl -si http://localhost:8080/health
curl -si -X OPTIONS http://localhost:8080/human/api/v1/users/me -H 'Origin: http://localhost:5173'

(Adjust the port to your SERVER_PORT.) Identify which middleware set each response header. Send X-Request-ID: my-test-id and confirm it is echoed. Check your server log: which of the two requests produced a request log line, and why did the other not?

Exercise 3 — Patient login without SMS, via the reviewer path

Set locally:

SMS_PROVIDER=none
REVIEWER_PHONE_E164=+919999900001
REVIEWER_OTP=<a 24+ character throwaway value>

Then:

curl -s -X POST localhost:8080/human/api/v1/phone-auth/otp/send -H 'Content-Type: application/json' -d '{"phone":"9999900001"}'
curl -s -X POST localhost:8080/human/api/v1/phone-auth/otp/verify -H 'Content-Type: application/json' -d '{"phone":"9999900001","otp":"<your value>"}'

1. Decode the token (paste the middle segment into base64 -d). Confirm entity_type is user and role is empty.
2. In Postgres, find the new refresh_tokens row. Confirm token_hash is not the raw refresh_token you received. Compute printf '%s' <raw> | shasum -a 256 and match it.
3. Call GET /human/api/v1/users/me with Authorization: Bearer <token>.
4. Send the wrong OTP five times. Inspect Redis with redis-cli keys 'otp:*' and find the otp:lock: key and its TTL.

Exercise 4 — Refresh rotation and reuse detection

Using the refresh token from Exercise 3 (use a fresh login if locked out):

1. POST /human/api/v1/phone-auth/refresh with {"refresh_token":"R1"}. You get R2.
2. Call refresh again with R1. Expect 401.
3. Now call refresh with R2. Explain why it also fails, and find the rows in refresh_tokens showing the revoked reason for each.

Exercise 5 — Logout, blacklist, and the per-instance trap

1. Log in, call /users/me (200), then POST /human/api/v1/users/logout with the bearer token. Call /users/me again: expect 401 token has been revoked.
2. Find the jwt:blacklist:* key in Redis and check its TTL against the token's exp.
3. Stop the server, unset REDIS_URL, and start two instances on different ports (set SERVER_PORT; also set ADMIN_PORT differently if the admin server is enabled). Log in on port A, log out on port A, then call /users/me on port B with the same token. Write down the result and connect it to Trap 5.

Exercise 6 — Staff enumeration behaviour

With SMS_PROVIDER=none, call POST /human/api/v1/employees/phone-otp/send for a number that is not in your local employees table. Note the status and time taken (curl -w '%{time_total}'). Then set LOG_LEVEL=debug, repeat, and find the log line. Compare with what happens for a number that does exist (insert a local test employee with an E.164 phone, and then again with a bare 10-digit phone).

Exercise 7 — Read the whitelist decision

1. With a local patient token, call a staff-only route such as GET /human/api/v1/employees. Note the status and body message.
2. Find the rbac: whitelist denied log line and read its endpoint, entity_type and roles fields.
3. In the local DB, find how GET /human/api/v1/users/me is treated for this token and explain it using whitelistExemptPatterns.

Exercise 8 — Audit row for your own requests

After the exercises above, query your local audit_log table for the last 20 rows. For each, say which middleware or handler filled actor_id, route, status and (if present) subject_id. Find a row with an empty actor_id and explain why it is empty.

---

Self-check

Answer without looking, then check below.

1. List the global middleware from outermost to innermost.
2. Why is Metrics directly on the mux rather than outside Audit?
3. A web panel's PATCH request with a valid access cookie gets 403 missing client header. What is missing and what attack does the check stop?
4. What three things does Auth check about a token before calling the whitelist, and what does it do on a Redis error?
5. Where does the whitelist get the caller's roles: the JWT role claim or the database? Which gate does use the claim?
6. In Redis mode, what exactly is stored at otp:code:<key>, and why does boot refuse REDIS_URL without OTP_PEPPER?
7. A refresh token that was already rotated is presented again. What happens, and why is that the right response?
8. Name the three login bypasses and the guard on each.
9. Why does SendOTPByPhone for staff return 200 for an unknown number, and how do you debug a real user who "gets 200 but no SMS"?
10. A hospital panel user's role changes from what the panel expected an hour after login. What caused it?
11. What breaks, and how, if two instances run without REDIS_URL?
12. Why is ParameterizedQueries: true not enough on its own to keep patient data out of GORM logs?

---

Answers

1. SecurityHeaders → CORS → RequestID → observability.Middleware → Recovery → Timeout → Logging → Audit → Metrics → mux. Then per-route: rate limiter and/or Auth (which runs the whitelist and attender page gates), optional entity/scope middleware, handler. (cmd/server/main.go:674)

2. The mux sets r.Pattern on the request it receives. Audit replaces the request with r.WithContext(...) before the mux sees it, so middleware outside Audit reads a request without Pattern and records every route as unmatched. (main.go:664–673)

3. The X-Mz-Client header. When the token comes from a cookie and the method is not GET/HEAD/OPTIONS, Auth requires it. A cross-site form or link can make the browser send cookies but cannot add a custom header, so this blocks CSRF. (middleware/auth.go:46)

4. Signature and expiry with HMAC only (ValidateToken); that sub parses as a UUID; and CheckRequest: whether this exact token was blacklisted and whether the subject was revoked at or after the token's iat. On a Redis error CheckRequest returns revoked for both, so the request gets 401 (fail closed).

5. The database: PermissionStore.subjectRoles reads role_assignments (30 s cache), and Decide matches role entity type and permissions against r.Pattern. The attender page gate (checkHospitalPages) uses the role claim to decide whether it applies.

6. HMAC-SHA256(OTP_PEPPER, key || 0x00 || code) as hex, with TTL OTP_TTL_SECONDS. With an empty pepper the HMAC key is empty, and a six-digit code could be recovered from Redis by trying all million values, which the log message calls "plaintext".

7. Rotate sees RevokedAt != nil, logs "refresh token reuse detected", revokes the whole family and returns 401. Either the real client or an attacker is holding an old token; the server cannot tell which, so it ends every session from that login. The real user logs in again; the attacker loses access.

8. DEV_MASTER_OTP: only compiled in with -tags dev, and a test fails if release files use that tag. EMERGENCY_OTP: at least 24 characters or disabled, constant-time compare, logged on boot and on every use, staff surfaces only. Store reviewer (REVIEWER_*, DELIVERY_REVIEWER_*): one phone number, code at least 24 characters, both values needed, failures still count toward lockout.

9. Anti-enumeration: a different answer would tell anyone which phone numbers belong to staff. It also sleeps 250 ms to match timing. To debug: check the stored phone is E.164, check for duplicates among active employees (that logs at Error), and raise LOG_LEVEL to debug locally to see the "no active employee" line.

10. The first refresh. /human/api/v1/auth/refresh returns HighestPriorityRole of the account's DB roles, and the panel's onRefreshed writes it to localStorage.role. If the panel's expected role is not what refresh returns, it is replaced. Access tokens last 1 hour by default, so that is when it shows up.

11. OTP codes and lockout counters live in each process's memory, so a code sent via one instance fails verify on the other and lockout allows 5 tries per instance. Logout and deactivation blacklist entries exist only on the instance that handled them, so a revoked token still works on the other until it expires. The server only warns at boot.

12. db.Raw(...).Scan(...) runs through GORM's package-level logger.Recorder, whose default RecorderParamsFilter ignores ParameterizedQueries and inlines bound values. internal/database/db.go overrides gormlogger.RecorderParamsFilter in init() to drop the values globally.