Alert System
Two dispatch paths, five shared low-level channel senders — how system-health alerts (existing, admin-only) and workflow alerts (new, per-user) relate.
Two paths, one set of senders
triggerAlert() (existing) | AlertRuleDispatcher (new) | |
|---|---|---|
| File | src/lib/alerts/dispatch.ts | src/lib/alerts/user-dispatch.ts |
| Who it's for | Platform operators (admins) | The individual user who owns the workflow |
| What triggers it | 10 fixed system-health AlertTypes (broker disconnected, copy stopped, queue backlog, failed order, Stripe failure, high latency, health endpoint failure, test alert, automation notification, copy worker unresponsive) | Any workflow event matched against a user's own alert_rules |
| Destination | One platform-wide destination per channel, from env vars (SLACK_ALERT_WEBHOOK_URL, DISCORD_ALERT_WEBHOOK_URL, ALERT_EMAIL_TO, ALERT_TELEGRAM_CHAT_ID, ALERT_SMS_TO) | Each rule's own destination (the user's own Slack webhook, own email, own Telegram chat, own phone number, ...) |
| Persisted to | alert_events (feeds Incident Management) | Nothing separate — in-app notifications land in the existing notifications table |
Both call the exact same 5 functions — sendSlackAlert(), sendDiscordAlert(), sendEmailAlert(), sendTelegramAlert(), sendSmsAlert() (src/lib/alerts/channels/*.ts) — each with an optional destination override parameter rather than duplicated:
sendSlackAlert(alert, fetchFn?, webhookUrlOverride?)
sendDiscordAlert(alert, fetchFn?, webhookUrlOverride?)
sendEmailAlert(alert, fetchFn?, toOverride?)
sendTelegramAlert(alert, fetchFn?, chatIdOverride?)
sendSmsAlert(alert, fetchFn?, toOverride?)Every pre-existing call site (the Stripe webhook, connector reconnect paths, /api/admin/alerts/check) omits the new parameter and is completely unaffected — verified by tests/alerts-channels.test.ts passing unchanged. AlertRuleDispatcher is the only caller that ever supplies it.
Channels
| Channel | Status |
|---|---|
in_app | Real — a row in the existing notifications table, same "backend-written, user-read" shape used throughout this app. Default channel for every new rule. |
email | Real — sendEmailAlert() via Resend, needs RESEND_API_KEY configured platform-wide (the rule supplies only the destination address). |
slack | Real — a plain webhook POST to the user's own slackWebhookUrl, no platform-wide config needed for this channel to work per-user. |
discord | Real — same shape as Slack. |
telegram | Real — sendTelegramAlert() via the Telegram Bot API's sendMessage, needs TELEGRAM_BOT_TOKEN configured platform-wide (one bot for the whole app; the rule supplies only the destination telegramChatId, obtained by messaging the bot once and reading its getUpdates). |
sms | Real — sendSmsAlert() via Twilio's Messages REST API (plain fetch, no SDK), needs TWILIO_ACCOUNT_SID/TWILIO_AUTH_TOKEN/TWILIO_FROM_NUMBER configured platform-wide (one purchased Twilio number for the whole app; the rule supplies only the destination smsPhoneNumber, E.164 format). |
push | Architecture-ready, not implemented — PushNotificationPort (src/lib/alerts/push/push-port.ts) is a real, documented interface with an honestly unconfigured UnconfiguredPushProvider behind it (same pattern as MarketDataProvider/RithmicTransport). No push vendor (FCM/APNs/web-push) was named for this milestone, and this app has no device-registration flow to obtain a token from in the first place. AlertRuleDispatcher catches the resulting PushNotConfiguredError and reports the channel as not delivered rather than crashing. |
Alert rules
alert_rules (per user, RLS-owned): a name, an event_types list (matching trigger types — empty means "any event"), an optional workflow_id scope (a rule can watch one specific workflow or every workflow), a channels list, a per-channel destination (email address / webhook URLs / Telegram chat ID / phone number), and a min_severity. ruleMatchesEvent() (src/lib/alerts/user-dispatch.ts, pure, unit tested) is the exact matching logic: event-type match OR empty list, workflow scope match OR unscoped, and the event's severity meets or exceeds the rule's minimum.
Managed on /dashboard/workflows/alerts (create/toggle/delete).
CopyWorker liveness (copy_worker_unresponsive)
GET /api/admin/alerts/check previously had no real way to tell whether the persistent CopyWorker process (docs/LIVE_COPY_ENGINE.md) was even running: its only CopyWorker-adjacent signal was checkCopyEngine()'s in-memory metrics.snapshot(), which reflects whichever process reads it — for that scheduled route, the web app itself, never the separate CopyWorker process where live copying actually happens.
src/lib/observability/copy-worker-heartbeat.ts closes this with a durable, DB-backed heartbeat: CopyWorker upserts a single row (copy_worker_heartbeat, service-role-only, RLS-enabled with no authenticated policies — the same pattern as stripe_webhook_events) on start, every sync tick (10s default), and on a graceful stop, with a status, a last_heartbeat_at timestamp, and a small non-sensitive metrics snapshot (queue depth, copy-latency p95, retry/failure counts — never account ids, symbols, or credentials). A heartbeat older than 60 seconds (six missed ticks) is treated as stale — the worker likely crashed — and fires a critical copy_worker_unresponsive alert (auto-opening an incident, same as any other critical alert).
The pre-existing queue_backlog/high_latency checks now read their numbers from this same durable heartbeat row instead of the route's own local metrics.snapshot(), for the same reason — that local registry never reflected CopyWorker's real activity in the first place. Both are skipped while the worker is already flagged unresponsive (a frozen last-known snapshot from a dead process isn't a meaningful backlog/ latency signal).
The decision logic (shouldAlertCopyWorkerUnresponsive, shouldAlertQueueBacklog, shouldAlertHighLatency, shouldAlertHealthFailure) lives in src/lib/alerts/threshold-checks.ts as small, pure, directly-tested functions — the route handler itself stays a thin wrapper, this codebase's usual convention.
copy_stopped (Emergency Stop / risk-limit triggered)
copy_stopped existed in the AlertType union from the start but had no real trigger anywhere in the codebase until now. CopySessionManager.emergencyStopAll() (src/lib/copy-engine/ copy-session-manager.ts) fires a warning-severity copy_stopped alert whenever it actually stops one or more relationships. Every "stop everything" path converges on this one method — the dashboard's manual Emergency Stop All button (src/app/dashboard/copier/actions.ts) and the AI Risk Manager's risk-limit-triggered lock_trading action (src/lib/ai/risk-manager/automation-service.ts, fired on a daily-loss/ max-drawdown/consecutive-losses/overtrading/revenge-trading breach) — so this one wiring point covers both without duplicating it. Calling emergencyStopAll() again with nothing left to stop (everything already emergency_stopped, or no active relationships for that user) is a safe no-op and does not re-alert.
What's NOT built
- No admin-facing per-user alert override or moderation UI beyond what
/admin/automation/failures already surfaces indirectly.
- No de-duplication/rate-limiting specific to the Alert Center — a
workflow that keeps re-matching (subject to its own cooldown, see docs/EXECUTION_SAFETY.md) will keep notifying every time it does.
- No in-app flow to look up a Telegram chat ID or verify a phone
number — both are pasted in manually, same as a Slack/Discord webhook URL today.