Skip to content
Real Estate Core Dev Docs

AI room staging rollout

This runbook is the release gate for room-staging v2. The implementation uses Replicate’s official openai/gpt-image-2 wrapper, but calls the asynchronous predictions API directly so prediction IDs can be persisted, polled, resumed, and canceled. There is no v1 provider fallback.

The GPT Image 2 provider contract is prompt plus input_images; roomType and canonicalStyle remain audited application inputs and are rendered into the immutable prompt. TV_ROOM is explicitly prompted as a TV/media room—it is not silently converted to a provider living_room enum.

Keep these official references open during rollout because provider schemas, pricing, and limits can change independently of this repository:

How to use this runbook

Section titled “How to use this runbook”

Treat every numbered section as a release gate. Complete the sections in order and do not continue past a failed pass condition. Commands are labeled by execution location:

  • Trusted workstation means a private terminal with curl, jq, openssl, psql, and optionally redis-cli/Docker. Shell history, screen recording, and support-session sharing should be disabled while handling secrets.
  • Deployment dashboard means the environment-variable and deploy/restart controls for the named service.
  • Worker terminal means a shell inside the deployed worker image, normally rooted at /app.
  • Database terminal means psql or the provider SQL console connected to the same writable PostgreSQL database used by apps/api and the worker.
  • Bull Board means the authenticated queue dashboard exposed by the worker. Never expose it without BULL_BOARD_USER and BULL_BOARD_PASSWORD.

Before starting, fill out and keep this worksheet with the release ticket. It prevents commands from being run against the wrong environment:

ItemValue to record
Environmentdevelopment, staging, or production
Release/image tagimmutable Git SHA or container tag
Public apps/api originfor example https://api.example.com
PostgreSQL host/databasehostname and database name only—never the password
Valkey host/databasehostname and database number only
QUEUE_SUFFIXexact value, including an empty value
Backup/snapshot IDprovider snapshot ID or encrypted dump path
Internal smoke listing IDunpublished/disposable listing
Release operatorperson running this runbook
Rollback ownerperson authorized to disable the feature
Start timeUTC timestamp

Global stop conditions

Section titled “Global stop conditions”

Stop the rollout, keep AI_ROOM_STAGING_ENABLED=false, and investigate if any of these occur:

  • the backup cannot be restored or its integrity cannot be checked;
  • migrations fail, are partially applied, or the second migration run is not a no-op;
  • an invariant query reports staged covers, invalid review state, duplicate variants, or duplicate nonterminal jobs;
  • the webhook probe returns anything other than 401 for an unsigned request;
  • a logical job creates more than two provider-attempt rows or a retry changes a still-active prediction ID;
  • a pending variant appears on an anonymous/public surface;
  • cancellation finalizes an image, or rejection/deletion removes a database row without a deletion-outbox record;
  • a staged image can become the listing cover;
  • provider spend, output quality, or error rate is not understood.

The kill switch is deliberately fail-closed: only the exact string true enables new paid prediction creation. Disabling it does not stop cancellation, polling of an already-persisted prediction, or storage-deletion cleanup. If deletion cleanup must also be paused for forensic review, stop/scale the worker process itself rather than relying on the staging flag.

Release invariants

Section titled “Release invariants”
  • PostgreSQL ai_jobs is the durable dispatch outbox; BullMQ is repairable delivery state.
  • A logical job can reserve only provider attempts 1 and 2.
  • A persisted prediction ID is polled after restarts and BullMQ retries; a retry must not buy the same attempt again.
  • Every new result is private PENDING_REVIEW; only APPROVED variants reach public detail/search results.
  • A staged image can never be the cover, including after approval.
  • Rejection and original-image deletion write durable storage-deletion outbox entries.
  • AI_ROOM_STAGING_ENABLED=false stops creation of new paid predictions. It does not erase jobs or prevent recovery/cancellation of an already-persisted attempt.

1. Prepare Replicate manually

Section titled “1. Prepare Replicate manually”

Purpose: establish the provider account, credentials, callback authentication, and a deliberate spend owner before any process can create a paid prediction.

  1. On a trusted workstation, sign in to the Replicate organization that will own production predictions. Record the organization/account name and the teammate responsible for billing. Use an organization-owned account rather than a developer’s personal account so token revocation and billing survive staff changes.

  2. Open the official openai/gpt-image-2 model page while signed in and run or inspect the playground. Confirm all of the following:

    • the account can access the model;
    • billing or prepaid credit is active;
    • the account’s current rate limits can support the initial 6 predictions/minute cap;
    • the provider’s data-processing/retention terms are acceptable for listing photos;
    • the current price is understood by the release and finance owners.

    Do not copy an old interior-design price into the application. The implementation intentionally displays no estimate when the AI_STAGING_COST_MICROUSD_* values are unset.

  3. Create a dedicated API token for the worker. Name it so the environment and purpose are obvious, such as real-estate-core-prod-ai-staging-worker. Save it directly in the deployment secret store as REPLICATE_API_TOKEN on the worker only. The API and clients never need this token; limiting its placement reduces the number of systems that can buy predictions or cancel them.

  4. Generate an independent pseudonymization secret on the trusted workstation:

    Terminal window
    openssl rand -hex 32

    Save the 64-character result as AI_STAGING_USER_HASH_SECRET on the worker. It HMAC-hashes the internal user ID before the value is sent as Replicate’s user_id, allowing abuse/cost correlation without revealing the database ID. Do not reuse BETTER_AUTH_SECRET, and do not rotate it during an incident unless you accept losing continuity in provider-side user attribution.

  5. Fetch Replicate’s default webhook signing secret from the trusted workstation. The command prints a secret, so do not run it in a recorded/shared terminal or paste its output into a ticket:

    Terminal window
    set +o history 2>/dev/null || true
    read -s REPLICATE_API_TOKEN
    export REPLICATE_API_TOKEN
    WEBHOOK_RESPONSE="$(curl --fail --silent --show-error \
      --header "Authorization: Bearer $REPLICATE_API_TOKEN" \
      https://api.replicate.com/v1/webhooks/default/secret)"
    printf '%s' "$WEBHOOK_RESPONSE" | jq -e '.key | startswith("whsec_")' >/dev/null
    printf '%s' "$WEBHOOK_RESPONSE" | jq -r '.key'
    unset WEBHOOK_RESPONSE
    unset REPLICATE_API_TOKEN
    set -o history 2>/dev/null || true

    The jq -e command must exit zero and the printed value must begin with whsec_. A 401/403 means the token or organization is wrong. An empty/non-whsec_ value is a stop condition.

  6. Save the returned value as REPLICATE_WEBHOOK_SIGNING_SECRET on apps/api only. This secret authenticates Replicate to the inbound webhook receiver; it is not the API token and should not be placed on the worker or clients. The receiver verifies the raw request body, timestamp, signature, and five-minute replay window before touching a job.

  7. Choose the public HTTPS apps/api origin for REPLICATE_WEBHOOK_BASE_URL, for example https://api-dev.real-estate-core.com. Put this value on the worker because the worker includes the callback URL when it creates a prediction. Use only the origin:

    Correct:   https://api-dev.real-estate-core.com
    Incorrect: https://api-dev.real-estate-core.com/
    Incorrect: https://api-dev.real-estate-core.com/api/webhooks/replicate/staging
    Incorrect: http://internal-api:3000

    The worker appends /api/webhooks/replicate/staging/{providerAttemptId}. The origin must be reachable by Replicate over public HTTPS without VPN, Basic Auth, an interactive login, or an IP allowlist that excludes Replicate. Replicate does not follow webhook redirects, so the final callback route itself must answer directly rather than redirecting between hosts, schemes, or trailing-slash variants.

  8. Record the token creation date, token owner, and webhook-secret retrieval date in the secret manager metadata. Define a rotation owner, but do not rotate either credential during an active prediction unless the existing credential is compromised.

Pass gate: the model is accessible and funded; all three values are stored on the correct services; the webhook origin is public HTTPS; no secret appears in Git, client configuration, logs, or the release ticket.

2. Place environment values on the correct services

Section titled “2. Place environment values on the correct services”

Purpose: keep every producer and consumer on one canonical provider configuration while guaranteeing that deployment alone cannot spend money.

In the deployment dashboard, set the kill switch to false everywhere before deploying. Environment-variable changes normally require a redeploy/restart; saving a value in the dashboard does not prove a running process has received it.

ServiceRequired staging values
svelte-web runtimeAI_ROOM_STAGING_ENABLED=false, AI_STAGING_LEGACY_ALIAS_SUNSET_AT, AI_STAGING_QUALITY, AI_STAGING_ASPECT_RATIO, AI_STAGING_OUTPUT_FORMAT, AI_STAGING_OUTPUT_COMPRESSION, plus the existing DATABASE_URL, VALKEY_URL, and QUEUE_SUFFIX
apps/apiThe same producer values as svelte-web, plus REPLICATE_WEBHOOK_SIGNING_SECRET; apps/api does not need REPLICATE_API_TOKEN
workerAI_ROOM_STAGING_ENABLED=false, REPLICATE_API_TOKEN, REPLICATE_WEBHOOK_BASE_URL, AI_STAGING_USER_HASH_SECRET, AI_STAGING_CONCURRENCY, AI_STAGING_RATE_LIMIT_PER_MINUTE; keep producer quality/format values during the one-release legacy bridge

Recommended initial non-secret values:

Terminal window
AI_ROOM_STAGING_ENABLED=false
AI_STAGING_LEGACY_ALIAS_SUNSET_AT=2026-10-19T00:00:00.000Z
AI_STAGING_QUALITY=medium
AI_STAGING_ASPECT_RATIO=1:1
AI_STAGING_OUTPUT_FORMAT=jpeg
AI_STAGING_OUTPUT_COMPRESSION=92
AI_STAGING_CONCURRENCY=4
AI_STAGING_RATE_LIMIT_PER_MINUTE=6
AI_DESCRIPTION_CONCURRENCY=2

The values have the following operational meaning:

VariableWhy it exists and initial guidance
AI_ROOM_STAGING_ENABLEDPaid-create kill switch. Only exact true enables creation; start with false.
AI_STAGING_LEGACY_ALIAS_SUNSET_ATLast instant old style names are accepted. 2026-10-19 is 90 days from the v2 rollout baseline; move it only through an intentional compatibility decision.
AI_STAGING_QUALITYPersisted provider input. Start at medium; changing it changes cost/output and requires API/web/worker deployment together.
AI_STAGING_ASPECT_RATIODefaults to 1:1; supported values are 1:1, 3:2, and 2:3. The downloaded output is validated against the selected ratio.
AI_STAGING_OUTPUT_FORMATjpeg is a pragmatic photo default. png is larger; webp requires confirming all downstream download/share clients.
AI_STAGING_OUTPUT_COMPRESSIONJPEG/WebP quality percentage. 92 balances detail and storage; allowed range is 0–100.
AI_STAGING_CONCURRENCYMaximum simultaneous staging jobs in one worker process. Total effective concurrency is this value multiplied by worker replicas.
AI_STAGING_RATE_LIMIT_PER_MINUTEBullMQ’s queue-wide start limiter across workers sharing this queue/Valkey namespace. Start at or below the verified Replicate account limit.
AI_DESCRIPTION_CONCURRENCYSeparate description-worker concurrency. Keeping it independent demonstrates staging cannot head-of-line block descriptions.

Use the same QUEUE_SUFFIX on svelte-web, apps/api, and worker. Confirm the target Valkey URL, logical database number, TLS mode, and suffix are identical; otherwise producers write to one queue while the worker listens to another. An intentionally empty suffix is valid—record it as <empty> in the worksheet instead of leaving the worksheet ambiguous.

Producer values (AI_STAGING_QUALITY, aspect ratio, format, and compression) must match on apps/api, svelte-web, and the worker during the one-release legacy bridge. New v2 jobs persist these values in PostgreSQL, but a legacy queue message is materialized by the worker from its environment. A staggered configuration change could therefore produce different results depending on which client version submitted the request.

Leave every AI_STAGING_COST_MICROUSD_* variable unset until the current provider price has been independently verified. When pricing is verified, store integer micro-US-dollars, set AI_STAGING_PRICING_VERSION to a dated source identifier, and review whenever Replicate pricing changes. These fields are estimates/telemetry only; they do not authorize spend.

The shared contract rejects provider values outside the current official GPT Image 2 schema. Supported aspect ratios are 1:1, 3:2, and 2:3. Re-check the live model schema before changing this list; deploy the producer and worker together when it changes.

After each service is deployed—but before enabling—open its terminal and verify presence without printing secret contents. Run only the block appropriate to that service:

apps/api
for name in DATABASE_URL VALKEY_URL REPLICATE_WEBHOOK_SIGNING_SECRET; do
  if [ -n "$(printenv "$name")" ]; then echo "$name=set"; else echo "$name=MISSING"; fi
done
printf 'AI_ROOM_STAGING_ENABLED=%s\n' "$AI_ROOM_STAGING_ENABLED"
printf 'QUEUE_SUFFIX=%s\n' "${QUEUE_SUFFIX:-<empty>}"
Terminal window
# worker
for name in DATABASE_URL VALKEY_URL REPLICATE_API_TOKEN REPLICATE_WEBHOOK_BASE_URL AI_STAGING_USER_HASH_SECRET AWS_ENDPOINT_URL AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_S3_BUCKET_NAME; do
  if [ -n "$(printenv "$name")" ]; then echo "$name=set"; else echo "$name=MISSING"; fi
done
printf 'AI_ROOM_STAGING_ENABLED=%s\n' "$AI_ROOM_STAGING_ENABLED"
printf 'QUEUE_SUFFIX=%s\n' "${QUEUE_SUFFIX:-<empty>}"
Terminal window
# svelte-web runtime
for name in DATABASE_URL VALKEY_URL; do
  if [ -n "$(printenv "$name")" ]; then echo "$name=set"; else echo "$name=MISSING"; fi
done
printf 'AI_ROOM_STAGING_ENABLED=%s\n' "$AI_ROOM_STAGING_ENABLED"
printf 'QUEUE_SUFFIX=%s\n' "${QUEUE_SUFFIX:-<empty>}"

Every required line must say set, every kill-switch line must say false, and the suffix values must match the worksheet. Do not use env, printenv without a variable name, or deployment screenshots; those expose secrets.

Pass gate: all variables are on the intended services, all processes still report the kill switch as false, and producer/queue configuration matches exactly.

3. Back up and inspect before migrating

Section titled “3. Back up and inspect before migrating”

Purpose: create a tested recovery point and a written baseline before the backfill normalizes historical styles, approves legacy variants, clears staged cover flags, removes duplicate/orphan rows, and schedules their objects for deletion. Migration 0009 intentionally changes data; a schema-only rollback is not an acceptable substitute for a backup.

  1. Announce a maintenance window and name the release and rollback owners. Disable the staging action at the API/web layer first, then wait for active legacy staging jobs to finish or cancel them through the supported UI/API. Description generation can continue, but no one should manually retry or delete BullMQ jobs during this window.

  2. Take a provider-managed PostgreSQL snapshot of the writer database. Record its immutable snapshot ID, creation time, database name, retention period, and the provider’s restore procedure in the release ticket. A snapshot is useful only if the operator has permission to restore it.

  3. Verify the snapshot using the provider’s integrity/status check. Prefer restoring it to an isolated temporary database and connecting with psql; this proves more than a dashboard’s Completed badge. Never connect application or worker processes to the restored clone.

  4. If provider snapshots are unavailable, create an encrypted/private custom-format logical backup from a trusted workstation:

    Terminal window
    umask 077
    BACKUP_FILE="ai-room-staging-pre-v2-$(date -u +%Y%m%dT%H%M%SZ).dump"
    PGDATABASE="$DATABASE_URL" pg_dump --format=custom --no-owner --no-acl \
      --file "$BACKUP_FILE"
    pg_restore --list "$BACKUP_FILE" >/dev/null
    printf 'Verified backup: %s\n' "$BACKUP_FILE"

    pg_dump and pg_restore --list must both exit zero, and the file must be stored in an encrypted location with restricted access. The dump contains customer and authentication data. Do not upload it to Git, a ticket, chat, or an unencrypted shared drive.

  5. From a database terminal, confirm the connection target and record a UTC timestamp before collecting baseline counts:

    SELECT current_database() AS database,
           current_user AS database_user,
           inet_server_addr() AS server_address,
           CURRENT_TIMESTAMP AT TIME ZONE 'UTC' AS captured_at_utc;

    Compare the database/host with the worksheet. Stop if they do not match.

  6. Record pre-migration job, image, orphan, cover, and raw-style counts. Save query output with the release ticket; it is the basis for explaining backfill changes later:

    SELECT "status", count(*)
    FROM "ai_jobs"
    WHERE "type" = 'ROOM_STAGING'
    GROUP BY "status"
    ORDER BY "status";
    

    SELECT count(*) AS staged_images,
           count(*) FILTER (WHERE "isMain") AS staged_covers,
           count(*) FILTER (WHERE "styledFromId" IS NULL) AS staged_orphans
    FROM "property_images"
    WHERE "isStyled" = true;
    

    SELECT COALESCE("style", '<null>') AS submitted_style, count(*)
    FROM "property_images"
    WHERE "isStyled" = true
    GROUP BY COALESCE("style", '<null>')
    ORDER BY submitted_style;
    

    SELECT "styledFromId", COALESCE("style", '') AS submitted_style, count(*)
    FROM "property_images"
    WHERE "isStyled" = true
    GROUP BY "styledFromId", COALESCE("style", '')
    HAVING count(*) > 1
    ORDER BY count(*) DESC;

    staged_covers and staged_orphans may be nonzero in legacy data; the migration repairs them. Unexpectedly large counts, unknown style values, or many duplicates should be reviewed before continuing because the deterministic newest row will win.

  7. In Bull Board, open ai{QUEUE_SUFFIX} and record waiting, active, delayed, completed, and failed counts. Open a few legacy stage-room jobs and record their IDs without copying image URLs or user data. Wait for active staging jobs to reach a terminal state. Do not use Retry all, Clean, Drain, or Obliterate.

  8. If the platform permits it, scale all worker replicas to zero immediately before the migration. This gives the operator time to inspect migrated deletion-outbox entries before the 30-second cleanup loop processes them. If the platform cannot run a one-off migration with workers stopped, the verified backup and baseline are mandatory: once the new worker sees the new tables, object cleanup can begin even though AI_ROOM_STAGING_ENABLED=false.

Pass gate: the correct database is documented, a restorable backup exists, baseline query and queue outputs are attached, no legacy staging job is active, and the team understands which historical rows the backfill may remove.

Section titled “4. Rehearse migrations on an empty database (recommended)”

Purpose: catch migration ordering, SQL syntax, extension, and idempotency failures before the release task touches shared data.

From the repository root on a trusted workstation, first remove a container left by an earlier rehearsal, if present:

Terminal window
docker rm --force staging-v2-postgres 2>/dev/null || true

Then apply the full migration chain to an isolated PostGIS database and prove a second run is a no-op:

Terminal window
docker run --name staging-v2-postgres \
  --detach \
  --publish 55439:5432 \
  --env POSTGRES_PASSWORD=staging_test \
  --env POSTGRES_DB=app \
  postgis/postgis:16-3.5


until pg_isready -h 127.0.0.1 -p 55439 -U postgres; do sleep 1; done


DATABASE_URL=postgresql://postgres:[email protected]:55439/app \
  bun run packages/database/scripts/migrate.ts
DATABASE_URL=postgresql://postgres:[email protected]:55439/app \
  bun run packages/database/scripts/migrate.ts


docker rm --force staging-v2-postgres

The first migration command must exit zero and list the newly applied migrations. The second must exit zero and print Migrations up to date. Leave the container running for investigation if either command fails; capture the first SQL error and stop the rollout instead of editing the production migration table.

An empty database proves the chain is internally valid, but it cannot exercise legacy duplicates or malformed rows. For the strongest rehearsal, restore the section 3 backup into a network-isolated temporary PostgreSQL instance, ensure no API/worker can reach it, run the migration twice, and run every query in section 6 against that clone. Delete the clone according to the organization’s data-retention policy when the verification is complete.

Never substitute a shared development/staging/production URL into the rehearsal command. Before pressing Enter, read the literal host and port back to another release participant; the example must use local port 55439.

Pass gate: both empty-database runs pass, and a restored-data rehearsal either passes or is explicitly recorded as unavailable with the backup verification used instead.

5. Deploy schema and backward-compatible worker first

Section titled “5. Deploy schema and backward-compatible worker first”

Purpose: make the database and consumer understand both old and new messages before any new API/client can produce v2 work. This is the only safe direction for a rolling deployment.

  1. Reconfirm AI_ROOM_STAGING_ENABLED=false in the worker’s running environment, not only in the dashboard. Confirm the immutable worker image tag matches the worksheet.

  2. Prefer a one-off/release task using the new worker image while normal worker replicas remain stopped. It must receive the production DATABASE_URL but must not start the worker entrypoint. In that task, run:

    Terminal window
    cd /app/packages/database
    test -n "$DATABASE_URL"
    pwd
    bun run scripts/migrate.ts

    pwd must print /app/packages/database. test -n verifies presence without printing the connection string. Run the task exactly once at a time; two operators must not race migrations.

  3. If Dokploy cannot create a one-off task, deploy one new worker replica with the feature disabled, open its terminal, and run the same commands immediately. The worker may briefly log a recoverable missing-table error before migration. Remember that the disabled feature flag does not pause deletion cleanup after the schema appears.

  4. Require successful migration entries for all three names:

    0008_ai_room_staging_v2_schema
    0009_ai_room_staging_v2_backfill
    0010_ai_room_staging_v2_constraints

    Migration 0008 creates states/tables/columns, 0009 normalizes legacy data and writes cleanup/audit rows, and 0010 installs cascading relationships, uniqueness, staged-cover checks, and deletion/cancellation triggers. A failure between them is a stop condition—do not hand-edit kysely_migration or rerun only a copied SQL fragment.

  5. Run bun run scripts/migrate.ts a second time from the same task. It must print Migrations up to date. This proves the migration runner sees a complete transaction history.

  6. Run every database check in section 6 before scaling normal worker replicas up when a one-off task was used. If the worker is already running, perform the checks immediately; deletion rows may already be DELETED, which is acceptable, but FAILED requires investigation.

  7. Start exactly one worker replica with AI_ROOM_STAGING_ENABLED=false. Watch its service startup for at least 60 seconds. Require no crash loop, successful PostgreSQL/Valkey connections, and no repeated schema/table errors. If a durable pending job is reconciled, require its job-level worker_started event and confirm no provider prediction is created while disabled. With an empty queue, the absence of worker_started is expected because that event is emitted when a staging job begins processing, not when the process boots.

  8. In Bull Board, confirm both ai{QUEUE_SUFFIX} and ai-staging{QUEUE_SUFFIX} exist and that their names use the worksheet suffix. Verify description jobs remain on ai; new staging jobs must use ai-staging. Failed or stalled legacy entries should be investigated individually, not bulk-retried.

  9. Increase to the intended worker replica count only after the single-replica observation passes. Effective concurrency is per-replica concurrency multiplied by replicas, while BullMQ’s limiter remains queue-wide for workers sharing the queue namespace. Confirm both values are acceptable to the provider account.

  10. Rebuild every active property search document so approved staged-card images carry their disclosure marker. From the worker container:

    Terminal window
    cd /app/apps/worker
    bun run search:schema -- --dry-run
    bun run search:schema -- --enqueue

    The dry run must complete without writing and report a plausible number of active property documents. Save that count. The enqueue command creates stable backfill jobs; it may be safely rerun, but do not run multiple copies simply because the queue is moving slowly.

    In Bull Board, wait until search-indexing{QUEUE_SUFFIX} has no active, waiting, or delayed backfill jobs and no failed backfill jobs. Compare completed/enqueued counts with the dry-run count, then run:

    Terminal window
    bun run search:schema -- --verify

    --verify must report success. Do not run it while jobs are still moving: it verifies completed search-document contents, not queue submission. Existing search hashes remain readable during this additive rollout, but a card cannot disclose an approved staged slide until its hash includes isStyled.

Pass gate: migrations apply once and are a no-op the second time; section 6 passes; one disabled worker remains healthy; both queues have the correct suffix; descriptions continue; and search backfill verification succeeds before API/client deployment.

6. Verify the migrated database before enabling

Section titled “6. Verify the migrated database before enabling”

Purpose: prove that the backfill produced a constrained, internally consistent state before the system is allowed to buy predictions.

Run the following from a database terminal and save the complete results. These checks are read-only:

SELECT "name", "timestamp"
FROM "kysely_migration"
WHERE "name" LIKE '0008%'
   OR "name" LIKE '0009%'
   OR "name" LIKE '0010%'
ORDER BY "name";


SELECT to_regclass('public.ai_room_staging_jobs') AS staging_jobs,
       to_regclass('public.ai_provider_attempts') AS provider_attempts,
       to_regclass('public.ai_provider_webhook_receipts') AS webhook_receipts,
       to_regclass('public.property_image_review_events') AS review_events,
       to_regclass('public.storage_deletion_outbox') AS deletion_outbox;


SELECT indexname
FROM pg_indexes
WHERE indexname IN (
  'ai_jobs_active_staging_dedupe_uidx',
  'property_images_live_staged_variant_uidx',
  'property_images_live_legacy_staged_variant_uidx',
  'property_images_aiJobId_uidx'
)
ORDER BY indexname;


SELECT conname
FROM pg_constraint
WHERE conname IN (
  'property_images_staged_cannot_be_main_check',
  'property_images_review_status_check',
  'ai_provider_attempts_attemptNumber_check'
)
ORDER BY conname;


SELECT count(*) AS invalid_staged_covers
FROM "property_images"
WHERE "isStyled" = true AND "isMain" = true;


SELECT count(*) AS invalid_review_state
FROM "property_images"
WHERE ("isStyled" = true AND "reviewStatus" IS NULL)
   OR ("isStyled" = false AND "reviewStatus" IS NOT NULL);


SELECT "styledFromId", "canonicalStyle", count(*)
FROM "property_images"
WHERE "isStyled" = true AND "canonicalStyle" IS NOT NULL
GROUP BY "styledFromId", "canonicalStyle"
HAVING count(*) > 1;


SELECT "styledFromId", COALESCE("style", '') AS legacy_style, count(*)
FROM "property_images"
WHERE "isStyled" = true AND "canonicalStyle" IS NULL
GROUP BY "styledFromId", COALESCE("style", '')
HAVING count(*) > 1;


SELECT "dedupeKey", count(*)
FROM "ai_jobs"
WHERE "type" = 'ROOM_STAGING'
  AND "status" IN ('PENDING', 'PROCESSING', 'RETRYING', 'CANCEL_REQUESTED')
GROUP BY "dedupeKey"
HAVING count(*) > 1;


SELECT "aiJobId", count(*) AS provider_attempts
FROM "ai_provider_attempts"
GROUP BY "aiJobId"
HAVING count(*) > 2;

Interpret the results precisely:

QueryRequired result
Migration historyExactly one row each for 0008, 0009, and 0010
to_regclassFive non-null table names
Index listFour rows, one for every requested index
Constraint listThree rows, one for every requested constraint
Invalid staged coversCount 0
Invalid review stateCount 0
Canonical duplicate variantsZero rows
Display-only legacy duplicate variantsZero rows
Active dedupe collisionsZero rows
Jobs exceeding provider-attempt capZero rows

Next, review cleanup work created by the backfill:

SELECT "reason", "status", count(*)
FROM "storage_deletion_outbox"
GROUP BY "reason", "status"
ORDER BY "reason", "status";


SELECT "id", "objectKey", "reason", "status", "attempts",
       "lastError", "createdAt", "processedAt"
FROM "storage_deletion_outbox"
WHERE "reason" LIKE 'MIGRATED_%'
ORDER BY "createdAt", "id";


SELECT "id", "objectKey", "reason", "attempts", "lastError"
FROM "storage_deletion_outbox"
WHERE "status" = 'FAILED'
ORDER BY "createdAt";

MIGRATED_ORPHAN, MIGRATED_DUPLICATE, MIGRATED_DUPLICATE_LEGACY, and MIGRATED_DUPLICATE_JOB_RESULT are expected only when the baseline contained corresponding legacy cleanup. For each reason, reconcile the count with the section 3 baseline or the restored-clone rehearsal. PENDING, PROCESSING, or DELETED are valid depending on whether the worker has started; any FAILED row must have an understood retryable storage error before rollout continues.

Do not manually delete S3 objects or outbox rows. The outbox is the durable proof that database deletion and storage deletion are coordinated; removing it defeats recovery and auditability.

Pass gate: all fixed row counts match the table, every invariant query is empty/zero, migration cleanup is reconciled, and no unexplained failed deletion is present.

7. Deploy APIs, web, then native clients

Section titled “7. Deploy APIs, web, then native clients”

Purpose: publish readers and thin adapters only after the database/worker understand the v2 contract, while the kill switch still prevents paid work.

  1. Deploy apps/api using the same immutable release tag as the worker. Keep AI_ROOM_STAGING_ENABLED=false. From its runtime terminal, repeat the non-secret configuration check in section 2 and confirm its health/readiness endpoint passes before testing staging.

  2. Confirm the webhook route is publicly reachable and rejects an unsigned request. On a trusted workstation, set the actual public origin from the worksheet and send a deliberately unsigned payload:

    Terminal window
    API_ORIGIN=https://api-dev.real-estate-core.com
    curl --include --request POST \
      "$API_ORIGIN/api/webhooks/replicate/staging/not-a-real-attempt" \
      --header 'content-type: application/json' \
      --data '{"id":"probe","status":"starting"}'

    Expected result: HTTP 401 with an application response. Interpret any other response before continuing:

    • 404 means the route or reverse-proxy path is missing;
    • 502/503 means the API is unhealthy or unreachable from the public ingress;
    • 500 usually means the signing secret/runtime configuration is missing;
    • 200/204 is a security failure because an unsigned webhook was accepted;
    • an HTML login/error page means Replicate will not reach the application handler correctly.

    A 401 does not prove a real signed Replicate callback succeeds; the paid canary in section 8 verifies that end-to-end path.

  3. Record the current database time, then deploy svelte-web with identical producer settings and the feature disabled:

    SELECT CURRENT_TIMESTAMP AS disabled_submission_test_started_at;

    As an authorized agent, open the staging control and attempt one submission. The UI must show the localized unavailable state, must not spin indefinitely, and must not imply a job was purchased. Then prove no room-staging job was inserted after the recorded timestamp:

    SELECT "id", "status", "createdAt"
    FROM "ai_jobs"
    WHERE "type" = 'ROOM_STAGING'
      AND "createdAt" >= '<DISABLED_SUBMISSION_TEST_STARTED_AT>'
    ORDER BY "createdAt";

    The result must contain no row caused by the test. Account for any known legacy traffic explicitly rather than assuming every row is the test.

  4. Deploy the shared API/client contract and agent-facing web/iOS/Android builds. Verify each agent client can decode ENQUEUED, EXISTING, and style_exists, plus RETRYING, CANCEL_REQUESTED, and CANCELED, without crashing or relabeling a terminal state.

  5. Deploy customer web/iOS/Android builds that render the exact localized AI · Virtually staged disclosure on staged thumbnails, galleries, and lightboxes. Test both the default locale and at least one translated locale. The disclosure is UI metadata; it must not be burned into or watermarked onto the image object.

  6. Confirm version/support policy before release. If mobile rollout is phased, the server must remain compatible with every supported installed version. Do not remove the legacy style aliases or legacy ai consumer merely because the new store build has been submitted; wait until old producers are outside the supported floor and the queue is empty.

  7. Exercise ordinary, non-staging image upload/edit/listing reads with the feature still disabled. This catches accidental contract regressions without spending provider credits.

Pass gate: API/web/native clients are healthy with staging disabled, unsigned webhooks return exactly 401, disabled submissions create no job, ordinary image flows still work, and all supported public clients contain the disclosure UI before any pending image can be approved.

8. Enable new predictions

Section titled “8. Enable new predictions”

Purpose: enable consumption before production so every newly committed durable job has a compatible worker ready to reserve and persist its provider attempt.

  1. Start a live release call with the operator, rollback owner, and someone able to inspect PostgreSQL, Bull Board, S3, logs/Sentry, and the public listing surface. Record the UTC enablement time.
  2. Reconfirm apps/api can reach PostgreSQL and Valkey. Reconfirm the worker can reach PostgreSQL, Valkey, S3, and Replicate and that its QUEUE_SUFFIX matches. Repeat the unsigned webhook probe and require 401.
  3. Set AI_ROOM_STAGING_ENABLED=true on the worker first, redeploy/restart it, and verify its running environment prints exactly true. Watch for a healthy worker_started event and no rapid provider/configuration errors. At this point the worker can recover durable work, but disabled producers should not accept a new submission.
  4. Set the flag to true on apps/api and svelte-web, then redeploy/restart both. Verify the running value on each service; do not infer a restart from a successful dashboard save.
  5. Keep Bull Board, structured worker logs, Sentry, and a database terminal open. Submit one low-risk job against the unpublished internal listing. Submit only once; a slow UI response is not a reason to click repeatedly because the durable dedupe path is part of what is being tested.
  6. Record the response’s original image ID, canonical style, AI job ID, disposition, and submission time. For a fresh image/style the disposition must be ENQUEUED. If it says EXISTING or style_exists, inspect existing state and choose a clean style instead of deleting records manually.
  7. Confirm the job appears in ai-staging{QUEUE_SUFFIX} with the AI job ID as its BullMQ ID, then moves through PENDING and PROCESSING to SUCCESS. A temporary RETRYING state is not a pass unless its error is understood and it still respects the two-attempt cap.
  8. Use the section 9 SQL query to verify exactly one provider-attempt row, a non-null stable prediction ID, provider status succeeded, persisted dimensions/bytes/format, and one result image in PENDING_REVIEW. Confirm isMain=false.
  9. Open the listing in a signed-out browser or incognito session and make a direct public API request. The pending image must be absent. Do not approve this canary until the privacy check passes.

If any step fails, immediately disable API/web producers first, then the worker flag, while leaving the worker process running for persisted prediction recovery/cancellation and outbox cleanup. Follow section 12.

Pass gate: one canary completes with one paid attempt and one private pending result; no public surface exposes it; queue/log/database identifiers agree; and there are no unexplained errors or duplicate charges.

9. Paid GPT Image 2 smoke matrix

Section titled “9. Paid GPT Image 2 smoke matrix”

Purpose: validate prompt behavior, persistence/resume semantics, storage metadata, review privacy, approval publication, client disclosure, and every canonical room type against the real paid provider.

This is a manual paid release test. Seven successful logical jobs create at least seven predictions. Because each logical job may buy at most two provider attempts after a retryable terminal failure, authorize a worst-case budget of 14 predictions at the provider’s current price before starting. Stop if spend or retry rate exceeds the approved amount.

Prepare one internal unpublished/disposable listing with seven original photos already owned by the application’s S3 bucket—one representative image per canonical room type. Each original should be a normal supported image under the worker’s 20 MiB provider-download ceiling, free of people/sensitive documents, and classified through the normal agent UI. Do not use customer data unless provider-processing/retention terms and organizational policy permit it.

Before generating, record for every source: property ID, original image ID/object key, room classification, pixel dimensions, aspect ratio, file size, checksum if available, intended style, and tester. Confirm every source is an original (isStyled=false), belongs to the property, and is not a dangling external URL.

Run one generation for each row. Spread the six styles across the seven rows; repeating one style for TV_ROOM is acceptable.

CaseCanonical roomWhat to inspect visually
1BEDROOMbed/furniture scale; walls, openings, flooring, crop unchanged
2LIVING_ROOMseating/contact shadows; no new built-ins or architecture
3DINING_ROOMtable/chair scale; hardwired lighting unchanged
4KITCHENonly movable decor/stools; cabinets, counters, appliances, plumbing unchanged
5BATHROOMonly towels/mats/decor; tile, mirror, vanity, fixtures unchanged
6OFFICEfreestanding desk/chair; no invented built-in shelves
7TV_ROOMTV/media intent is explicit; no wall alterations or silently generic living-room result

For each job:

  1. Submit from an authorized agent surface. Record the original image ID, submitted room classification, canonical style, AI job ID, disposition, and UTC time. A clean first submission must return ENQUEUED; save the per-assignment response rather than only a screenshot of a toast.

  2. Query the original image and verify its submitted room classification was persisted. The database value is now generation source of truth; changing only a client-side label after submission is not sufficient.

  3. In Bull Board, verify ai-staging{QUEUE_SUFFIX} uses the AI job ID as the BullMQ job ID. Record queue wait time and ensure staging activity does not delay a concurrently submitted description job.

  4. Run the SQL below repeatedly. A provider-attempt row should be reserved before purchase and its providerPredictionId should appear immediately after asynchronous creation. That ID must remain stable through polling and ordinary BullMQ retries.

  5. For at least one row, restart one worker replica only after providerPredictionId is non-null and before provider completion. Record the ID before and after. The worker must resume the same prediction; attempt 2 is valid only when attempt 1 contains a retryable terminal provider/transport failure. A restart alone must never buy attempt 2.

  6. Confirm successful output metadata includes dimensions, format, download bytes, and checksum. Compare output aspect ratio with the persisted provider input/source expectation, and open the stored image through the normal signed URL path. Reject truncated, non-image, unexpectedly huge, or architecturally altered results.

  7. Confirm exactly one staged row exists, has reviewStatus=PENDING_REVIEW, isMain=false, canonical provider inputs/prediction ID, and is visible only in authorized agent editing surfaces. Verify the localized disclosure appears there.

  8. Before approval, inspect anonymous web, a direct public API response, customer iOS, and customer Android. Refresh data and bypass local image/listing caches. The pending variant must be absent from all four surfaces.

  9. Record the staged object’s byte count/checksum, approve it through the review action, then refetch each public surface. The variant must appear with AI · Virtually staged on thumbnails, gallery, and lightbox. Download the object again and compare byte count/checksum: approval changes database visibility only and must not watermark or rewrite the generated pixels.

  10. Confirm the active listing’s Valkey search hash preserves the staged marker (replace the ID with the internal property ID):

    Terminal window
    redis-cli -u "$VALKEY_URL" --raw \
      HGET "psearch:property:PROPERTY_ID" images | jq

    The approved variant must contain "isStyled": true; originals must contain false. If the field is absent or wrong, stop the rollout and repeat the search-document rebuild in section 5 before exposing search results.

  11. Record a visual decision for the row: pass, reject/regenerate, or provider-quality incident. Review walls, windows, doors, fixed cabinetry, plumbing, electrical fixtures, perspective, crop, and room identity—not only whether the furniture looks attractive.

Use this query during the matrix:

SELECT
  j."id" AS ai_job_id,
  j."status",
  j."phase",
  j."errorCode",
  s."canonicalRoomType",
  s."canonicalStyle",
  s."providerModel",
  s."promptVersion",
  s."resultImageId",
  original."roomType" AS original_room_type,
  a."attemptNumber",
  a."providerPredictionId",
  a."status" AS attempt_status,
  a."providerStatus",
  a."downloadBytes",
  a."outputWidth",
  a."outputHeight",
  a."outputFormat",
  a."costMicrousd",
  a."costKind",
  result."reviewStatus",
  result."isMain",
  result."providerInputs",
  result."providerPredictionId" AS result_prediction_id,
  result."outputBytes" AS stored_output_bytes,
  result."outputWidth" AS stored_output_width,
  result."outputHeight" AS stored_output_height,
  result."outputFormat" AS stored_output_format,
  result."outputChecksum"
FROM "ai_jobs" j
JOIN "ai_room_staging_jobs" s ON s."aiJobId" = j."id"
LEFT JOIN "property_images" original ON original."id" = s."originalImageId"
LEFT JOIN "ai_provider_attempts" a ON a."aiJobId" = j."id"
LEFT JOIN "property_images" result ON result."id" = s."resultImageId"
WHERE j."id" = '<AI_JOB_ID>'
ORDER BY a."attemptNumber";

After all seven rows, confirm seven successful logical jobs, seven live variants, no job with more than two attempts, and no unexplained storage-deletion failure. Attach the completed matrix and provider spend to the release ticket. Keep the listing unpublished until lifecycle/race tests in section 10 finish.

Pass gate: every room category—including the distinct TV_ROOM prompt—passes persistence, restart, metadata, privacy, approval, disclosure, search-index, and visual-architecture checks within the authorized spend.

10. Lifecycle and race smoke tests

Section titled “10. Lifecycle and race smoke tests”

Purpose: prove the hardening behavior under concurrency and cancellation timing, not only the happy-path image quality.

Complete every scenario before release using disposable originals on the unpublished smoke listing. Keep logs, Bull Board, and the section 9 SQL query open. Record the AI job, attempt, prediction, image, and outbox IDs produced by each scenario. Do not simulate results by directly editing application tables.

Configuration changes in this section affect the whole environment. Announce them, change one worker replica at a time, restart it, and restore the documented value immediately after the scenario. Each accidental job may spend up to two predictions, so keep the test set bounded.

Generate → approve → public

Section titled “Generate → approve → public”
  1. Generate a new style.
  2. Wait for SUCCESS and PENDING_REVIEW; record the pending image ID. Query the review-event table and confirm no APPROVED event exists yet.
  3. While pending, open the listing anonymously on web and customer mobile and call the public listing API. Refresh server data rather than relying on a previously cached response. The variant must be absent everywhere.
  4. Approve as the assigned/owning agent. An unrelated agent must receive an authorization/not-found response and must not learn the image’s existence.
  5. Confirm reviewStatus=APPROVED, reviewedByUserId and reviewedAt are populated, and exactly one APPROVED review event exists.
  6. Wait for the property search-index job to complete, then reopen/refetch public surfaces. The variant and exact disclosure label must appear; the original image remains available.

Cancel while queued

Section titled “Cancel while queued”
  1. Temporarily set the worker’s AI_STAGING_RATE_LIMIT_PER_MINUTE=1, restart it, and confirm the running value. Do not raise concurrency to manufacture this test.
  2. Submit one controlled job to consume the available start, then a second clean image/style and wait until the second job is visibly waiting in ai-staging. Record the second AI job ID; it is the cancellation subject.
  3. Cancel that waiting job from the authorized agent UI/API before it becomes active. Repeat cancellation once to verify the operation is idempotent rather than returning a server error.
  4. Confirm the database status is CANCELED, cancelRequestedAt/canceledAt are set, the waiting BullMQ job is removed, and zero ai_provider_attempts rows exist for that logical job. A prediction ID or charge is a failed test.
  5. Restore AI_STAGING_RATE_LIMIT_PER_MINUTE=6 (or the worksheet’s approved value), restart the worker, and verify the running value before continuing.

Cancel while running

Section titled “Cancel while running”
  1. Start a job and wait until providerPredictionId is non-null.
  2. Record the prediction ID and cancel from the authorized agent action. The immediate response/database status must be CANCEL_REQUESTED unless the worker has already finalized it.
  3. Repeat the same cancellation request. It must remain safe/idempotent and must not create another provider attempt.
  4. Confirm the worker calls provider cancellation and the logical job becomes CANCELED. The attempt should end CANCELED or DISCARDED depending on the completion race; the prediction ID must remain the original ID.
  5. Confirm no staged row is finalized for the job. If Replicate wins the race and output reached storage, require an output_discarded event and a storage_deletion_outbox row for the uploaded object; wait for DELETED.

Reject → cleanup → regenerate

Section titled “Reject → cleanup → regenerate”
  1. Generate a pending result and reject it.
  2. Confirm a REJECTED review event preserves the image ID snapshot, AI job ID, actor, timestamp, and optional reason even though the image row is gone.
  3. Confirm the image row is removed in the same transaction and an outbox row exists for its exact object key with reason STAGED_IMAGE_DELETED. Do not delete the S3 object manually.
  4. Wait for worker cleanup and confirm outbox status becomes DELETED, processedAt is populated, and the object is no longer retrievable through a fresh signed URL.
  5. Generate the same canonical style again. It must return ENQUEUED, not EXISTING or style_exists, and eventually create one new pending variant with a different job/image/object ID.

Concurrent duplicate requests

Section titled “Concurrent duplicate requests”
  1. Sign in as two authorized sessions and prepare the same original, room classification, and canonical style in both. Do not use different legacy aliases for this test.
  2. Release both submissions within the same second. Preserve the full per-assignment responses and request timestamps.
  3. One response may say ENQUEUED; the other must say EXISTING and return the same logical job ID. A completed/pending variant may instead make both responses style_exists; in that case reject it and repeat with a clean style.
  4. Confirm one nonterminal ai_jobs row for the dedupe key, one BullMQ job ID, at most two attempts for that logical job, and exactly one resulting image. Search logs for uniqueness errors; handled conflict resolution is acceptable, an HTTP 500 is not.

Original deletion cascade

Section titled “Original deletion cascade”
  1. Generate at least two staged derivatives for a disposable original.
  2. Approve one derivative and leave the other pending. Optionally start a third generation so cancellation-on-delete is exercised.
  3. Record all database image IDs/object keys, then delete the original through the normal API/UI. Never issue direct SQL for this scenario.
  4. Confirm every derivative row cascades. A queued job becomes CANCELED; an active/retrying job becomes CANCEL_REQUESTED and then CANCELED after provider handling.
  5. Confirm the original and every derivative object key have deletion-outbox records (ORIGINAL_IMAGE_DELETED or STAGED_IMAGE_DELETED) and eventually become DELETED.
  6. Confirm the listing/search document no longer references any deleted image and that the remaining property has a valid non-staged cover. If no original remains, the listing must have no cover rather than promote a staged variant.

Staged cover prohibition

Section titled “Staged cover prohibition”

After approving a variant, attempt the normal cover endpoint with an authenticated test session. Keep session credentials out of shell history; the placeholder below must be replaced only in a private terminal:

Terminal window
curl --include --request PATCH \
  https://api-dev.real-estate-core.com/api/property-images/<STAGED_IMAGE_ID> \
  --header 'content-type: application/json' \
  --header 'cookie: <REDACTED_TEST_SESSION_COOKIE>' \
  --data '{"isMain":true}'

Expect HTTP 400. Then independently verify the database constraint in a transaction that is rolled back:

BEGIN;
UPDATE "property_images"
SET "isMain" = true
WHERE "id" = '<STAGED_IMAGE_ID>';
ROLLBACK;

The update statement must fail with property_images_staged_cannot_be_main_check; issue ROLLBACK again if the SQL client leaves the transaction aborted. If the update succeeds, stop the release.

Then send DELETE /api/property-images/<STAGED_IMAGE_ID> with the same authorized session and expect HTTP 400. Reject the variant through ai.rejectStagedVariant; do not use the generic photo endpoint, because rejection must write review audit metadata before the deletion trigger schedules storage cleanup.

Pass gate: all seven lifecycle scenarios have recorded IDs/evidence, cancellation and rejection leave no live result, duplicate requests converge, cascades delete every derivative safely, and cover prohibition succeeds at both service and database boundaries.

11. Observability gates

Section titled “11. Observability gates”

Purpose: confirm operators can detect stuck, duplicated, expensive, or cleanup-failed work without waiting for a customer report.

During and after smoke tests, require the following worker structured events for the relevant job IDs:

  • worker_started with Bull attempt and queue wait;
  • prediction_created with attempt and prediction identifiers;
  • provider_terminal with provider status;
  • output_validated with byte count, format, and dimensions;
  • completed with queue/provider timings;
  • canceled for cancellation tests;
  • output_discarded for a completion/cancellation race;
  • retry_scheduled or terminal_error with structured error code/category when deliberately exercised.

Also require API events ai_room_staging.cancel_requested, ai_room_staging.variant_approved, and ai_room_staging.variant_rejected. Logs may contain internal IDs and prediction IDs, but must not contain provider tokens, signing secrets, signed source/output URLs, cookies, or raw image bytes.

Operational gates:

  • No job PENDING over five minutes without a Sentry alert.
  • No stale PROCESSING/CANCEL_REQUESTED heartbeat over two minutes or RETRYING heartbeat over five minutes without a Sentry alert.
  • No logical job with more than two ai_provider_attempts rows.
  • No unbounded growth in storage_deletion_outbox FAILED rows.
  • Description generation latency remains normal while staging is busy, proving queue isolation.

Run these read-only queries after the canary, after the full matrix, and again 15 minutes later:

SELECT "status", count(*)
FROM "ai_jobs"
WHERE "type" = 'ROOM_STAGING'
  AND "createdAt" > CURRENT_TIMESTAMP - INTERVAL '24 hours'
GROUP BY "status"
ORDER BY "status";


SELECT "status", count(*), max("attempts") AS max_attempts
FROM "storage_deletion_outbox"
GROUP BY "status"
ORDER BY "status";


SELECT "id", "status", "phase", "createdAt", "heartbeatAt", "errorCode"
FROM "ai_jobs"
WHERE "type" = 'ROOM_STAGING'
  AND "status" = 'PENDING'
  AND "createdAt" < CURRENT_TIMESTAMP - INTERVAL '5 minutes'
ORDER BY "createdAt";


SELECT "id", "status", "phase", "heartbeatAt", "errorCode"
FROM "ai_jobs"
WHERE "type" = 'ROOM_STAGING'
  AND (
    ("status" IN ('PROCESSING', 'CANCEL_REQUESTED')
      AND "heartbeatAt" < CURRENT_TIMESTAMP - INTERVAL '2 minutes')
    OR
    ("status" = 'RETRYING'
      AND "heartbeatAt" < CURRENT_TIMESTAMP - INTERVAL '5 minutes')
  )
ORDER BY "heartbeatAt";


SELECT "aiJobId", count(*) AS attempt_count
FROM "ai_provider_attempts"
GROUP BY "aiJobId"
HAVING count(*) > 2;


SELECT "id", "objectKey", "reason", "attempts", "lastError", "createdAt"
FROM "storage_deletion_outbox"
WHERE "status" = 'FAILED'
ORDER BY "createdAt";

The four exception queries must return zero rows after expected in-flight work settles. Trigger one non-production alert path or inspect a captured test event to prove the Sentry environment, release tag, job ID, and error code are searchable. Compare description queue wait/provider latency to the pre-release baseline while staging is saturated; a shared-limiter slowdown is a release failure.

Pass gate: all expected events are searchable by job ID, Sentry routing is verified, exception queries are empty, deletion failures are not accumulating, and description latency remains within its established baseline.

12. Rollback / incident response

Section titled “12. Rollback / incident response”

Purpose: stop new purchases quickly while preserving enough state to finish, cancel, clean up, and understand work already in progress.

  1. Declare the incident and record the UTC time, symptom, first affected job/prediction ID, and incident commander. Preserve logs before changing replica counts.
  2. Set AI_ROOM_STAGING_ENABLED=false on apps/api and svelte-web first, then restart/redeploy them. Verify an attempted submission shows unavailable and creates no new ai_jobs row after the disable timestamp. Disabling producers first closes the intake before consumer behavior changes.
  3. Set the worker flag to false and restart/redeploy it. New jobs without a paid prediction remain durable PENDING with phase PROVIDER_DISABLED; they can resume after recovery. A reserved-but-uncreated attempt also remains unpaid and resumable.
  4. Leave the worker process running unless storage deletion itself is suspected. The flag does not stop polling/canceling a persisted prediction or deletion-outbox cleanup. If deletion must be frozen for forensics, scale the worker to zero and accept that descriptions, heartbeats, cancellation, and cleanup will also pause.
  5. Decide what to do with already-paid active predictions. If output must not finalize, cancel each through the authorized API/UI and verify CANCEL_REQUESTED → CANCELED. Provider cancellation may not refund an already-started prediction. Do not delete the BullMQ entry or null the prediction ID.
  6. Inspect Bull Board, Sentry, ai_jobs, ai_room_staging_jobs, ai_provider_attempts, webhook receipts, and deletion outbox. Export only the minimum IDs/status/metrics needed for the incident; redact signed URLs and credentials.
  7. Do not revert migrations, restore the backup over a live database, or delete job/attempt/outbox rows as a feature rollback. The additive schema and durable records are required by the already-deployed readers and recovery paths. A data restore is a separate disaster-recovery decision requiring downtime and reconciliation of post-snapshot writes.
  8. Fix and deploy a compatible release with the flag still false. Repeat sections 5–7 as applicable, then run one canary. Re-enable the worker first and API/web second; pending durable jobs reconcile automatically.
  9. Never route to the old interior-design model or silently call another provider. A provider outage remains an explicit unavailable state until GPT Image 2 is safe again.

Pass gate for a disabled incident state: all producer runtimes report false, disabled submissions add no jobs, no new prediction ID appears after the worker disable time, existing paid attempts are either intentionally polling or canceled, and cleanup state is preserved.

13. Compatibility cleanup after 90 days

Section titled “13. Compatibility cleanup after 90 days”

Purpose: remove temporary compatibility code only when telemetry and the supported-client floor prove it is unused; the calendar date alone is not sufficient.

On or after the exact UTC instant in AI_STAGING_LEGACY_ALIAS_SUNSET_AT:

  1. Confirm product/release owners still want to end alias support. If a supported native version can submit aliases, extend the sunset explicitly and document the new date rather than breaking that client silently.

  2. Confirm current web/iOS/Android code and observed request telemetry submit only the six canonical styles. Query the last 30 days of persisted submissions:

    SELECT "submittedStyle", count(*)
    FROM "ai_room_staging_jobs"
    WHERE "createdAt" >= CURRENT_TIMESTAMP - INTERVAL '30 days'
      AND "legacyStyle" IS NOT NULL
    GROUP BY "submittedStyle"
    ORDER BY "submittedStyle";

    Require zero rows. Also search structured/API logs for validation or alias-normalization events; database evidence alone misses requests rejected before insertion.

  3. Check the application-store adoption dashboard and server-supported minimum versions. The supported client floor—not the latest available version—must be above the canonical-style release.

  4. In Bull Board, confirm the legacy ai{QUEUE_SUFFIX} queue has no waiting, active, delayed, or failed stage-room job. Completed historical entries can remain until normal retention removes them.

  5. Remove alias acceptance from the shared contract/service and legacy queue bridge in a normal reviewed release. Keep the kill switch false during rollout and repeat contract tests, disabled-submission verification, and one canonical canary.

  6. Keep historical style, submittedStyle, legacyStyle, and legacyMetadata values readable for display/audit. Do not rewrite them merely to make telemetry appear clean.

  7. Do not add no-furniture to the generation catalog. It remains display-only historical data and must never be sent as a GPT Image 2 staging request.

Pass gate: no alias submission exists in 30 days of database/log evidence, all supported clients are canonical, the legacy queue is empty, product approves removal, and a post-removal canonical canary passes.

Final release checklist

Section titled “Final release checklist”
  • Provider token, billing/access, public webhook origin, webhook signing secret, and user-hash secret configured.
  • Kill switch false during schema/worker/API/web rollout.
  • Database backup recorded; migrations 0008–0010 applied twice/idempotently verified.
  • Schema/invariant SQL passes; migration cleanup outbox reviewed.
  • Dedicated ai-staging queue visible; legacy ai compatibility retained.
  • Active property search documents rebuilt and verified; staged search images retain isStyled: true.
  • Webhook unsigned probe returns 401.
  • Seven paid room smoke jobs pass, including explicit TV_ROOM behavior and a worker restart.
  • Approval/public privacy, queued/running cancellation, rejection/regeneration, duplicate race, original deletion, and cover rejection pass.
  • Public web/iOS/Android show only approved variants with AI · Virtually staged.
  • Sentry/log/queue/outbox dashboards are quiet and description jobs are not blocked.
  • Rollback owner knows the kill-switch procedure; no v1 fallback exists.
  • Release ticket contains worksheet values, baseline/migration SQL, smoke matrix, lifecycle evidence, spend, UTC enablement time, and operator/rollback sign-off.

Record the final decision and time:

Sign-offNameUTC timeDecision/notes
Release operatorGO / NO-GO
Database verifier
Client/privacy verifier
Rollback owner