Incident guides
Auth/session
Section titled “Auth/session”Check cookie domain, BETTER_AUTH_URL, BETTER_AUTH_URL_API, CORS_ORIGINS, trusted proxy settings, and browser Origin behavior. For mobile, confirm the app points at the same API host used by Better Auth.
BullMQ jobs
Section titled “BullMQ jobs”First checks: VALKEY_URL, TLS settings, QUEUE_SUFFIX (must match the tier — -dev / -stg / empty in PRD), and whether the worker process is running at all.
The queue-admin CLI
Section titled “The queue-admin CLI”Bull Board exists on DEV/STG only — PRD has no dashboard (a Cloud Run worker pool has no URL). Queue inspection and failed-job recovery in every tier go through apps/worker/scripts/queue-admin.ts:
- DEV/STG: a terminal in the worker container —
bun run queue:admin <cmd>; or locally,cd apps/worker && bun --env-file=../../.env.local run queue:admin <cmd>. - PRD: a one-off Cloud Run job using the worker image with a command override (Memorystore is VPC-private, so the job must run inside the VPC). The exact
gcloud run jobs executeinvocation lands with the provisioning work.
Safety model: every mutating command is a dry run by default — re-run with --execute to apply. The script prints an env banner (Valkey host, QUEUE_SUFFIX, resolved queue names) before acting; read it before executing against a shared Valkey. Lead payloads contain PII (name/email/phone/message) — the list commands (list-failed, list-dlq) print them only with --json; inspect always prints the full job, payload included.
Start with queue:admin counts for waiting/active/delayed/failed totals and repeatable-scheduler health across all six queues, then list-failed <queue> / inspect <queue> <jobId> for detail.
Per-queue recovery
Section titled “Per-queue recovery”| Queue | Failure record | Recovery |
|---|---|---|
leads (lead-ingestion) | failed_lead_ingestions DLQ row + admin Knock alert + admin UI (/dashboard/admin/failed-leads) | list-dlq --status PENDING → retry-dlq --id <row> --execute. Bad payload: fix it in SQL (jsonb_set) first — the script skips rows the worker’s type guard would reject, leaving them PENDING. The script is the authoritative replay path: it clears the stale lead-<dedupeKey> job that otherwise makes a re-enqueue silently no-op (the admin-UI replay has that limitation). Purge semantics: the daily 03:00 dlq-purge deletes terminal-status rows immediately and PENDING rows after DLQ_RETENTION_DAYS (default 90) — the script therefore enqueues, verifies, and only then marks REPLAYED. |
alerts (price-drop-notify, status-change-notify) | Valkey retained-failed set only (no DLQ table) — genuinely lossy once removeOnFail: 500 evicts | list-failed alerts → retry-failed alerts --all --execute, promptly. saved-search-scan self-heals on its next tick. |
search-indexing | Valkey retained-failed set only | retry-failed search-indexing for individual jobs; for wholesale drift use the existing backfills — bun run backfill:properties / backfill:search. |
billing | Repeatable scans only (trial-scan, stripe-reconcile) | Self-healing — each run re-derives state. Verify the schedulers exist via counts billing. |
ai (generate-description) | ai_jobs row (durable), but no automatic redispatch — the 30s reconciler only covers ROOM_STAGING | retry-failed ai for retained failures; redispatch-descriptions for lost dispatches / stuck rows (rebuilds the payload from the ai_jobs row; never writes ai_jobs.status). |
ai-staging (stage-room) | Fully protected: Postgres dispatch outbox + 30s reconciler | Read-only inspection only — the script refuses to mutate this queue. The manual lever is the ai_jobs row in Postgres; see AI room staging. |
Rehearsal (run quarterly, and after changing queue code)
Section titled “Rehearsal (run quarterly, and after changing queue code)”On DEV, with the worker running (cd apps/worker && bun --env-file=../../.env.local run dev); all queue:admin commands via bun --env-file=../../.env.local run queue:admin …:
queue:admin counts— banner shows the-devsuffix and six queues.queue:admin inject-test-failure --execute— enqueues a lead with an invalidpropertyId; the FK violation exhausts 3 attempts in ~30s. (The command refuses to run whenQUEUE_SUFFIXis empty, i.e. production-style names.)queue:admin list-dlq --status PENDING— the row appears; admins receive thelead-ingestion-failedKnock alert.queue:admin retry-dlq --id <rowId>(dry run) — the printed plan includes removing the stale failed job.- Fix the payload to a real property:
UPDATE failed_lead_ingestions SET payload = jsonb_set(payload, '{propertyId}', to_jsonb('<realPropertyId>'::text)) WHERE id = '<rowId>';then re-run with--execute— the row flips to REPLAYED and the script reports the created inquiry. - Loss-safety check: inject a second failure and replay it without fixing the payload — a fresh PENDING row appears after it fails again, proving a REPLAYED-then-purged row loses nothing.
discard-dlqit. - Clean up:
DELETE FROM inquiries WHERE "dedupeKey" = '<dedupeKey>';(lead_events cascade).
Knock delivery
Section titled “Knock delivery”Check workflow payload shape, recipient id, channel ids, signing key, device registration, and generated workflow docs for trigger sites.
Images/imgproxy
Section titled “Images/imgproxy”Check S3 credentials, bucket endpoint, upload route ownership checks, imgproxy key/salt, URL TTL, and CSP/image host allowlists.
Search drift
Section titled “Search drift”Check whether the source of truth changed in Postgres, the worker indexed the update, and Valkey/Meilisearch keys match the expected index.
Database reset/migration
Section titled “Database reset/migration”Use the repo reset script documented in Conventions → Monorepo. Avoid ad hoc force-reset commands because pre/post SQL ordering matters.