Health monitoring

The health monitor is Forven's background watchdog. It runs inside the backend on a 30-second heartbeat and folds the state of the moving parts — the scheduler, the scanner, the daemon, the brain workers, the bots, the data collector, and the AI providers — into a colour-coded picture: green, amber, or red. It also checks whether the market data your strategies use is fresh enough, by the same freshness rule the Data screens and the gauntlet's data gate use, so a stalled feed surfaces before it quietly corrupts a backtest or a paper decision.

This page is for operators who want to read those states correctly and know when to act.

Where you see it

• The Forge (/lab) — the System health chip on the Engine card reads Healthy, Degraded or Critical, with the number of failed checks. Open it for every component with its state and message, followed by the data checks. Health notes, lower on the same card, repeats the first few problems.
• The dashboard — a Critical banner at the top lists any critical health alerts, and the System card at the bottom reads Healthy or Degraded: … and links to Diagnostics.
• Data → Health — the data-freshness view: a verdict, tier cards, and a Needs attention list of late series with Download now or Refresh now.
• The notification inbox — every health alert that crosses the routing threshold lands there, and in Discord if you route it (see how alerts route).
• Diagnostics — a separate, point-in-time snapshot of configuration checks (database, providers, scheduler, and so on). Use it when something looks wrong right now.

Over the API: GET /api/health/status for components and data checks, and GET /api/health/alerts for recent alerts.

What the monitor watches

Components. Each long-running loop reports in:

On top of those, periodic data checks (every five minutes) test the freshness of the candles live and paper strategies trade on, pipeline consistency, and the SQLite database itself.

States. A scheduled loop turns amber when it is overdue by twice its expected interval, and red at five times. Data freshness turns amber when a series that a live, paper or gauntlet strategy reads is in breach, and red after three consecutive collection failures on a stream.

Data freshness

Forven judges every stored series by one freshness SLA. How late a series may be depends on who uses it — its tier — and its timeframe:

allowed lag = the larger of (missed bars + 1) × the timeframe, and the tier's floor

Lag is measured from the opening time of the last stored bar. A series within its allowance is fresh; past it, late; past three allowances, in breach. A series nobody has downloaded is missing, and one deliberately not collected (delisted or renamed) is frozen.

For example, a 1h series a paper strategy trades is allowed max(3 × 1 h, 45 min) = 3 hours of lag, and is in breach after 9.

The tiers, the floors and the breach multiplier are in Settings → Data → Freshness SLAs. The Data Collector (freshness SLA) job keeps series within their allowance, most overdue first, within a per-venue request budget (Settings → Data → Collection).

How the monitor uses it:

• The Data freshness component only counts breaches in the live, paper and pipeline tiers. A stale research-universe or idle series never turns it amber.
• The candle data check is critical when a live series is in breach or missing, and a warning when a live series is late or a paper series is late, in breach or missing.
• The sidebar's Data badge counts only breaches or missing data behind live or paper strategies.

Reading the states

Read top-down:

1. Components first. An amber or red component is usually the root cause — a hung scheduler starves everything downstream.
2. Then the data. Amber data freshness with green components usually means a data-source problem (rate limit, upstream outage), not an app fault. Data → Health shows which series are late and why.
3. Cross-check the scheduler. Stale data is often a stalled or skipped collection job. The health monitor and Agents → Schedules tell two halves of the same story.

Health is about whether the machine is running honestly, not whether a strategy is working. A perfectly green system can still be running strategies that should be retired.

How alerts route

The monitor turns state changes into notifications through the same path every other alert uses, so health events follow your notification routing:

• Red fires a critical alert immediately, with a 10-minute cooldown so a flapping component can't spam you.
• Amber fires a warning once the component has stayed amber on two consecutive checks.
• Back to green sends an informational recovery notice.

Deduplication is scoped by severity: an earlier warning or recovery notice for the same component can never suppress a critical one.

Checks and alerts run in every system mode. The automatic recovery the monitor attempts for a red component runs only in Semi and Auto — in Manual it observes and alerts, and leaves the fix to you.

Turning a health alert into a fix

A notification can be handed to an agent as a repair task. Substitute the notification id:

# Create a repair task from a notification (operator action)
curl.exe -X POST http://127.0.0.1:8003/api/notifications/NOTIFICATION_ID/repair-task `
  -H "Content-Type: application/json" `
  -d '{\"agent_id\": \"full-stack-engineer\"}'

That queues an agent run carrying the alert; the agent investigates and writes its findings back to the run. You can also acknowledge an alert (POST /api/notifications/NOTIFICATION_ID/acknowledge) or re-send it through the current routing (POST /api/notifications/NOTIFICATION_ID/resend).

Steps: check system health

1. Open The Forge and look at the System health chip on the Engine card. Healthy with no issue count means you're done.
2. Otherwise, open the chip. Note which components are amber or red, and read their messages.
3. For a scheduler problem, open Agents → Schedules and look for a failing job or one running far longer than usual — see troubleshooting.
4. For stale data, open Data → Health: refresh the late series, and check that its data source is reachable.
5. For an audit trail, open the notification inbox in Diagnostics: every health alert that crossed the routing threshold is logged there with its event type, severity, source, and summary.

What you'll see: the chip refreshes as the monitor ticks (about every 30 seconds). Critical alerts also appear in the dashboard banner and, if routed, in Discord.

History and retention

Heartbeats are short-lived by design: the heartbeat rows in the activity log are pruned after about 2 days. Notifications are kept about 60 days and the rest of the activity log about 90 days. If you are reconstructing an incident from last week, the raw heartbeats will already be gone — take a backup if you need a longer record. The windows are adjustable; see database & maintenance.

Caveats

• Manual mode freezes collection. The scheduler skips every job in Manual, the data collector included, so stored series go late until you switch to Semi or Auto or refresh them yourself.
• Startup catch-up can look alarming. After a restart, overdue scheduler jobs run once immediately rather than replaying every missed tick. A burst of activity right after startup is the catch-up, not a fault.
• A hung job isn't always recoverable on sight. The scheduler won't force-release a job's lock while its worker thread is still alive, so a persistently red scheduler often means a slow external call (a model or exchange request) is still running. See troubleshooting.
• Health says nothing about trade quality. It tracks liveness and freshness only.

Forven is a research tool. A green health state means the system is running and its data is fresh — nothing more. It is not a measure of strategy quality, results are not predictive of future performance, and nothing here is financial advice.

Related

• Diagnostics — the point-in-time health checks and cost roll-up
• Managing market data — the Data screens, including Health
• Notifications — how health alerts are routed
• Troubleshooting & recovery — stale locks, hung jobs, and recovery