Troubleshooting & recovery

When something in the lab stalls, the cause is almost always one of a handful of known faults: the system mode is holding the pipeline back, the agents have no model they're allowed to call, a strategy is waiting on a gate, a scheduler lock never released, a safety control tripped, or an MCP client can't reach the backend. This page is the operator's recovery runbook: the symptom, the cause, and what clears it.

Forven is local-first, so nearly every fault is recoverable from your own machine. Start with Diagnostics and the System health chip on The Forge's Engine card (see health monitoring) to see what's red, then jump to the matching section.

Before you start

Three quick checks resolve a surprising share of "it's broken" reports. All three are visible in the control strip at the top of every page.

1. What system mode are you in? A fresh install starts in Manual, where the scheduler skips every job and nothing autonomous runs. Semi runs the pipeline on what you submit, but agents write no new ideas. Only Auto runs everything. See autonomy modes.
2. Is generation paused? Generation can be frozen on its own over the API, to drain a backlog. That's a throttle, not a fault.
3. Is trading halted? The control strip reads Trading Allowed or Trading Halted. A halt — the system pause, startup exchange recovery, the kill-switch, or the daily-loss halt — blocks new live entries only. It doesn't stop the scheduler, research, or paper trading.

Fresh-install warnings

A new install with no keys shows a few red and amber warnings. None of them stops you from downloading data, backtesting, running the gauntlet or paper trading.

Trading halted: startup exchange recovery active

What you see. The control strip, the Dashboard and the Risk page say Trading halted: startup exchange recovery active — … HyperLiquid private key is required …. Reset Halt doesn't clear it.

What it means. Before it allows new live entries, Forven reconciles its records with the exchange. Without Hyperliquid credentials that check can't run, so live entries stay blocked. It concerns the exchange connection only: backtests, the gauntlet, the paper scanner and manual paper orders all work anyway — the paper desk notes that account halts don't apply to paper.

What to do.

• Paper only? Nothing. Leave it.
• To clear it, add Hyperliquid testnet credentials in Settings → HyperLiquid and save. Forven drops the stale recovery flag straight away, and the next reconcile checks the exchange.

Reset Halt clears the system pause, the kill-switch and the daily-loss halt. It has no effect on startup exchange recovery.

"Action required: No connected & selected model is configured…"

What you see. A red banner that can't be dismissed, and often an amber one: 6 agents use provider openai with no credentials….

What it means. Forven only spends on a model whose provider you connected in the app and that you selected for something. With no provider connected, every agent run stops at that check, so nothing is charged and nothing is researched. Runs that hit it are re-queued and retry once a model is available.

What to do. Skip it if you aren't using the agents yet. Otherwise:

1. Open Agents → Setup → Providers (the banner's Open Health button goes there) and click Connect on a provider. Paste its API key, or sign in for OpenAI or MiniMax.
2. Open Agents → Setup → Agent models and give every agent a model — or use Apply to all to give them all one — then Save. If agents still point at the old provider, Move them to a connected provider on the Providers page does it in one go.

The amber banner can be dismissed with ✕. Some backend error messages still name the old Agents → Models and Routing & Fallbacks tabs; both now live under Agents → Setup. See models & providers.

The strategy-creation loop is quiet

In Auto, the Strategy Developer writes ideas and builds strategies from them within a daily budget, and the pipeline tests them. If nothing is moving:

1. Mode. Manual freezes everything; Semi builds only the ideas you submit with The Forge → + Submit idea. Set the mode to Auto in the control strip.
2. Generation paused? Resume it with POST /api/system/generation/resume (an API control; see operations).
3. Agents idle? See the next section.
4. Budget. The throughput preset (Settings → System → Throughput) sets the daily strategy-creation budget — 4 tasks a day on Trickle, 40 on Balanced. On Trickle, the loop is deliberately slow. Separately, once agents have registered the Daily agent registration cap (120 by default, same section) in a UTC day, no new creation tasks start until 00:00 UTC.
5. Heartbeats. The health monitor flags a stale scheduler, daemon, or brain-worker loop as amber or red.

Agents aren't doing anything

Agents only spend on models you have both connected and selected — Forven never falls back to a provider you didn't choose. So a fresh install with no provider has agents that sit idle by design.

1. Open Diagnostics. A warning on the auth_providers check means no provider has a valid token.
2. Connect a provider and give each agent a model, as in the Action required entry under fresh-install warnings.
3. Check Agents → Setup → Providers for a provider marked as failing or rate-limited. A provider that keeps failing trips its own cool-down; one whose sign-in expired stays under Connected with Sign in again.
4. If you are on free OpenRouter models, switch the throughput preset to Conserve so the loop stops tripping rate limits.
5. Look at Agents → Runs: the Blocked and Failed buckets say why each run stopped, and Resume re-queues it.

A strategy is stuck

A strategy that isn't moving is almost always waiting on something specific — a gate, missing evidence, a work-in-progress cap, an approval, or data. Don't guess: open the pipeline board, which shows each strategy's stage and the exact reason it is where it is. The same explanation is available per strategy from GET /api/lifecycle/strategies/{id}/explain.

• Waiting for the held-back test → each candidate gets one walk-forward on the held-back recent data — by default when it applies for live, or before paper under Strict paper (or with Held-back test runs at → Gauntlet to paper). The tests per strategy family are budgeted; a family that has spent its budget waits for the next quarter's cutoff. See promotion gates.
• Waiting on approval → gauntlet → paper is automatic by default. Paper → live waits for you to type GO LIVE in Approvals, unless you turned on Allow automatic go-live (off by default).
• Evidence is stale → an engine or data change marked its earlier verdicts stale; it re-validates automatically. See stale evidence.
• Data unavailable → the market or an enrichment stream it needs has no data locally. See managing market data.

Why isn't my strategy in paper (or live)?

By default the door into paper only stops strategies that are broken or clearly losing; the selection-bias checks are recorded there and enforced at paper → live. So a strategy that passed the gauntlet but isn't trading, or a paper strategy that isn't going live, is usually in one of these states. The pipeline board and the gate report say which.

• Waiting for a paper slot (paper_queue_full). It passed the door, but the paper pool (40 strategies by default), its market (3) or its family (10) is full. It stays in the gauntlet stage, in a queue ranked by its recorded evidence, with a message such as Paper pool full: 40/40 strategies in paper — waiting for a paper slot (queue position 3 of 12, ranked by recorded evidence). That's capacity, not a rejection, and it never counts toward the auto-archive. Slots free up as the hourly pool pass retires losing paper strategies. See the queue with python -m forven.agent paper-pool (or GET /api/lifecycle/paper-pool); raise Settings → Lab → Paper pool → Paper pool size if your machine has room.
• Collecting forward evidence (insufficient_paper_evidence, then live_evidence_pending). The paper → live gate waits for 20 closed paper trades and 30 days in paper (the defaults); then the sequential test keeps saying continue until the evidence is decisive either way. If it decides the forward trades contradict the backtest (live_evidence_fail), the strategy is retired from paper on the next hourly pool pass, unless you switched Retire when forward contradicts backtest off.
• Breadth test pending (breadth_pending). Going live needs a finished, current breadth test — the unchanged rule on 15 coins. A missing one is started from the gate, and the strategy keeps paper trading meanwhile. A home_only verdict (breadth_home_only, an edge on its home coin only) keeps it from going live; it still paper-trades.
• Multi-coin live review pending (multi_coin_live_review). Its breadth test read general, so it paper-trades several coins as one record. Going live with several coins is held until the live order path is reviewed for them; it keeps paper trading.
• Retired from paper (paper_retired:<why>). The hourly forven-paper-pool-tick job archived it: losing (15+ closed forward trades with a negative average and a t-stat below −1), drawdown (a 15% forward drawdown), idle (60 days without a trade) or sequential_fail (its forward trades contradict its backtest). The limits are in Settings → Lab → Paper pool, where Retire losing paper strategies automatically switches retirement off.
• Archived by a check the door now only records. Strategies archived recently for a deflated Sharpe, walk-forward, jitter or held-back result can be listed with python -m forven.agent readmit-candidates --days 30. python -m forven.agent requeue <ids> sends them back to the quick screen for a fresh gauntlet run under today's settings.

See configuration for every limit.

A new strategy sits in Quick screen

Common on a fresh install. In Manual mode the scheduler skips every job, including the ones that screen new strategies, so a strategy you saved or sent to The Forge waits in Quick screen.

• Test it yourself. Open its strategy page and use Run backtest and Run robustness suite. Actions you start yourself run in every mode.
• Or let the pipeline do it. Switch to Semi (or Auto) in the control strip. The gauntlet workflow advancer then screens the strategy and carries it through the gauntlet.

Scheduler stuck, stale locks & zombie threads

The scheduler runs dozens of built-in jobs (strategy creation, the scanner, data collection, maintenance, and more). A job that's mid-run holds a lock in the scheduler_jobs table via its running_since timestamp. If the job hangs or its process is killed, the lock can survive and the job looks stuck.

Symptoms

• A job's next run is in the past, but it never fires.
• Agents → Schedules shows a job running long past its usual duration, or under the Failing filter.
• A last_error is set and the job won't retry.

In Manual mode every job is skipped on purpose and its next run simply moves forward. That isn't a stuck lock.

How recovery works

Forven recovers stale locks automatically. On each tick, and at startup, the scheduler checks every held lock:

1. If the job still has a live background task, or a live worker thread that outlived a timed-out run, the lock is kept — deliberately, so the same job can never run twice at once.
2. Otherwise, a lock older than the job kind's stale threshold, or older than the hard cap of 3,900 seconds, is released.
3. The next tick runs the job again.

A worker thread that eventually exits releases its own lock.

What to do

Most stale locks clear within a tick. If a job stays stuck, restart Forven — startup clears any orphaned lock before the first tick. Then watch Agents → Schedules: the job's next run should advance.

If a lock still won't clear, the cause is usually a genuinely live worker blocked on a slow external call — a cold backtest cache, a slow model completion, or an exchange request that hasn't returned. That is the recovery system working as designed: it won't pull a lock out from under a thread that's still running. Find the hung call rather than forcing the lock.

Kill-switch tripped & trading halted

The kill-switch is the drawdown circuit breaker for real capital. When drawdown from the high-water mark reaches max_drawdown_pct (default 30% once settings are saved), it closes every open live position at market and halts new live entries until you reset it. Paper is never affected. For the full picture, see the risk page.

Symptoms

• The control strip reads Trading Halted and Kill Switch Active; the Risk page shows the kill-switch banner with its trigger time.
• GET /api/risk reports kill_switch_active = true.
• New live entries are refused.

Recovery

1. Open Risk (/risk) and read the trigger time and the current account state.
2. Confirm the positions actually closed. A close that failed after its retries leaves that position open, and it needs manual review.
3. Reset only once you understand why drawdown breached: Reset Halt in the control strip (or Reset Trading Halt on Risk) clears the pause, the kill-switch, and the daily-loss halt together (POST /api/system/trading/reset); POST /api/kill-switch/reset resets the kill-switch alone.

The daily-loss halt is a related automatic halt: when same-day loss passes max_daily_loss_pct (default 2%), new live entries are blocked until the next UTC day. It clears itself at UTC midnight — there's nothing to reset.

Reconciliation: positions and records disagree

If a live position exists on the exchange with no matching trade record, or a record is stuck pending, the issue is usually reconciliation, not corruption.

• Phantom positions — filled on the exchange, missing locally — are detected at startup and periodically after. Recovery creates a fresh trade record that is not linked to a strategy and may need a look. See live trading.
• Pending opens with no exchange order ID free their risk slot after 180 seconds; the exchange check closes a genuinely unfilled trade.
• To take one position off the board by hand, force-close it (POST /api/trades/{id}/force-close). A 502 means the close may have failed at the exchange — check again before assuming it's flat.

If a strategy is misbehaving at the lifecycle level rather than the position level, demote it and let it re-validate through the gauntlet rather than editing state by hand.

Restoring from a backup

Forven snapshots its database every day and before every schema migration, into FORVEN_HOME/backups/database. To roll back, stop Forven, set the current forven.db (and its -wal and -shm files) aside, copy the snapshot in as forven.db, and start again. The full procedure is on database & maintenance.

Two cautions:

• Copying by hand? Stop Forven first. The database runs in WAL mode, so a copy of forven.db alone, taken while it runs, can miss the latest writes.
• The encryption key lives elsewhere — %LOCALAPPDATA%\Forven\.forven_key on Windows, outside FORVEN_HOME. Back it up alongside your data, or the encrypted credentials in auth.json can't be decrypted after a restore. See encryption & secrets.

An update was refused

The in-app updater only fast-forwards. It refuses — with the reason — when your checkout has uncommitted changes, local commits, or is on another branch. See updating for how to resolve each.

MCP connection issues

The Forven MCP server is a separate process your MCP client (such as Claude Desktop or Claude Code) spawns, and it makes HTTP calls to your running backend. Most failures are configuration. Match the symptom.

The client can't import forven

What you see. The server never starts, and the client's MCP log shows ModuleNotFoundError: No module named 'forven'.

Why. The snippet on Integrations → AI Clients uses a bare python, and your client may find a different Python than the one Forven runs in.

Fix. Point command at the virtualenv Forven created, by absolute path — C:\Users\you\Forven\.venv\Scripts\python.exe on Windows, or /home/you/Forven/.venv/bin/python on macOS and Linux. Forven is installed into that environment (pip install -e ., which start_all.ps1 runs for you on Windows), so its Python can import forven from any folder. Adding an absolute PYTHONPATH pointing at the repository root, as below, covers a checkout where that install didn't happen.

"Tool call timed out"

Long backtests on a cold cache can exceed the default HTTP timeout. Raise it:

{
  "mcpServers": {
    "forven": {
      "command": "C:\\Users\\you\\Forven\\.venv\\Scripts\\python.exe",
      "args": ["-m", "forven.mcp_server"],
      "env": {
        "PYTHONPATH": "C:\\Users\\you\\Forven",
        "FORVEN_API_URL": "http://127.0.0.1:8003",
        "FORVEN_MCP_TIMEOUT": "180"
      }
    }
  }
}

FORVEN_MCP_TIMEOUT is in seconds (default 60).

"Connection refused"

The MCP server is just an HTTP client — if the backend is down or on another port, every call fails. Confirm the backend is up and FORVEN_API_URL matches it:

# Is the backend answering on the expected port?
Invoke-WebRequest http://127.0.0.1:8003/api/health -UseBasicParsing

If it isn't, start Forven the usual way — the Windows launcher or start_all.

"401" errors

If the backend requires keys (FORVEN_API_KEY, FORVEN_OPERATOR_KEY), the MCP config must carry them in its env block. A keyless localhost backend doesn't need them — remove the placeholder key entries from the copied snippet.

Tools don't appear

Clients read MCP config only at startup. After editing it, restart the client — tools don't hot-reload.

When all else fails

A clean restart resolves transient lock and worker faults more often than any single command:

1. Make sure no live position is mid-close (on a paper install, nothing to check).
2. Stop Forven — close it from the launcher, or stop start_all.
3. Start it again. Startup re-runs lock recovery, reconciliation, and the health checks.
4. Open Diagnostics and the System health chip, and confirm everything is green before setting the mode back to Auto.

If a fault survives a restart, it's a real state issue. Capture the component state, the job's last_error, and the relevant log before changing anything by hand.