All documentation

Disaster Recovery Plan

The formal version of the same real content already surfaced in-app at /admin/backups (Disaster Recovery Checklist / Restore Procedure sections) — written here as a standalone plan an operator can follow during an actual incident without being logged into the app (which may itself be part of what's down). See docs/BACKUP_STRATEGY.md for what's backed up and how; this document is about what to *do* when something has actually failed.

Scope & assumptions

This project runs on Supabase Cloud (managed Postgres + Auth), not self-hosted infrastructure — recovery of the database itself is Supabase's responsibility per their own SLA/backup tier; this plan covers what's specific to *this application*: what this app's own deployment must additionally protect, and the steps to bring the whole system back to a working state after Supabase-side recovery (or any other failure) completes.

Recovery targets (stated honestly, not aspirationally)

  • RPO (Recovery Point Objective) — bounded by Supabase's backup

frequency/retention for your project's specific plan tier (verify in Supabase Dashboard → Database → Backups; this varies by plan and is not a number this codebase can state on Supabase's behalf). Point-in- time recovery (PITR), where available on your plan, gives a much tighter RPO than daily snapshots — confirm which your project has *before* an incident, not during one.

  • RTO (Recovery Time Objective) — no formal SLA is defined for this

application; realistically bounded by (a) how long a Supabase-side restore takes (outside this app's control) plus (b) redeploying application code/containers (minutes, given the existing Docker images — see DEPLOYMENT.md §1) plus (c) the manual verification steps below. Treat this as "as fast as possible, correctness first," not a promised number, until this has been rehearsed at least once against a real or staging environment.

What can fail, and where recovery actually happens

FailureRecovery path
Database data loss/corruptionSupabase Dashboard → Database → Backups (PITR or snapshot restore) — see Procedure A below
Application code bug/bad deployGit revert + redeploy — see Procedure B below
CREDENTIAL_ENCRYPTION_KEY lostNot recoverable from any backup — every stored broker credential becomes permanently undecryptable; see docs/BACKUP_STRATEGY.md for why this must be backed up separately, outside the database entirely
Copy Engine worker process crashrestart: unless-stopped (or your platform's equivalent) restarts it automatically; reconnection and session state re-hydrate on its own boot sequence — see DEPLOYMENT.md §10
Supabase Cloud outage (not data loss)Not this app's incident to resolve — check status.supabase.com; this app has no failover to a second database provider
Stripe outage/webhook delivery gapStripe redelivers webhooks on request (Stripe Dashboard → Developers → Webhooks → this endpoint) once service resumes
Broker-side outage (a specific platform down)Each connector's own reconnect/backoff handles transient issues automatically; a sustained broker-side outage is outside this app's control — see docs/BROKER_INTEGRATION_REPORT.md's per-broker reconnect behavior

Procedure A: Database restore

1. Do not act before confirming the incident and the desired restore point. A restore is destructive to every write made after that point — confirm via Incident Management (/admin/incidents) what actually happened and when, first. 2. In the Supabase Dashboard: Database → Backups → restore to a specific point in time (PITR) or from a daily snapshot, depending on plan tier. 3. Supabase performs the restore into the same project (or a new one, depending on the method chosen) — follow the Dashboard's own confirmation flow; this is entirely Supabase-managed, not something this app's code can automate or accelerate. 4. Re-verify before declaring success:

  • supabase db query --linked against a few key tables (profiles,

subscriptions, trades) to confirm data looks sane.

  • Re-run supabase db push --linked if any migrations were applied

after the restore point — a restore can leave the schema behind what the deployed code expects.

  • Run the full verification sequence (tsc --noEmit, eslint,

vitest run, next build) against the restored environment. 5. Rotate SUPABASE_SERVICE_ROLE_KEY and CREDENTIAL_ENCRYPTION_KEY only if the incident involved credential exposure — a routine restore does not require this, and rotating CREDENTIAL_ENCRYPTION_KEY unnecessarily makes every currently-valid stored broker credential unrecoverable (see docs/BACKUP_STRATEGY.md). 6. Verify Stripe webhook delivery resumed (Dashboard → Developers → Webhooks → delivery log) — request redelivery for anything missed during the incident window. 7. Communicate proactively to affected users if broker_connections state was rolled back — they may need to reconnect broker accounts, and discovering that silently is a worse experience than being told. 8. Post an incident note (Incident Management) documenting the restore point chosen and the resulting data-loss window (trades/copy activity between the restore point and the incident are genuinely unrecoverable from the database), then resolve the incident.

Procedure B: Application code rollback

git log --oneline -10          # find the last known-good commit
git checkout <commit>          # or: git revert <bad-commit>
docker build -t ai-trade-copier:rollback .
docker build --target worker -t ai-trade-copier-worker:rollback .
# redeploy both images — see DEPLOYMENT.md §1/§2

On Vercel: Dashboard → Deployments → "Promote to Production" on a previous deployment — instant, no rebuild needed (web app tier only; the Copy Engine worker, if deployed separately per DEPLOYMENT.md §10, needs its own rollback since it doesn't run on Vercel's request model).

Database migrations in this project are additive-by-design (new columns/tables, never destructive drops in a single migration — see every migration file's own begin;...commit; pattern) specifically so a code rollback doesn't require a matching schema rollback in the common case — confirm this holds for the specific migrations involved before assuming it, not by default.

Roles & communication (fill in for your organization)

This plan doesn't assume a specific on-call structure — a real production incident needs: who has Supabase Dashboard access, who has CREDENTIAL_ENCRYPTION_KEY/other secrets backed up and can retrieve them, who can redeploy application code, and how affected users are notified. Define these explicitly for your team before the first real incident, not during one; this document deliberately doesn't invent placeholder names/contacts that would be actively misleading.

Rehearsal

This plan has not been executed end-to-end against a real or staging environment as part of this pass — it's built from real, working procedures (the restore steps mirror what /admin/backups already documents, and the rollback steps use the same Docker images DEPLOYMENT.md already describes), but a written plan and a rehearsed one are different levels of confidence. Running a real (non-production!) restore drill is a recommended pre-launch or early-post-launch action — see the Beta Launch Checklist.