The gauntlet

The gauntlet is the robustness battery a strategy must survive after the cheap quick screen and before it may trade, even in simulation. It is one stage of the pipeline: quick_screen → gauntlet → paper → live_graduated. The public site sometimes calls the whole funnel "the gauntlet"; in the app, gauntlet is the specific stage that proves an edge is real rather than curve-fitted.

A strategy enters the gauntlet once the quick screen has judged it on a fair sample and found it isn't broken, leaky or clearly losing. The gauntlet then runs five validation tests, cheapest first so weak strategies fail fast. By default the door into paper admits a strategy that isn't broken or clearly losing and records the selection-bias verdicts; the paper → live gate enforces them. Passing earns a place in paper trading (the public site's "candidate"), not real capital.

Forven is a research tool. The gauntlet measures how a strategy behaved on historical and simulated data; it doesn't predict the future, and nothing here is financial advice. Survival is evidence of discipline, not a promise of profit.

What the gauntlet runs

Before the robustness tests proper, the gauntlet prepares the strategy:

1. Quick-screen gate — re-checks the quick screen at entry. Under the default fair-sample rule, it widens the window (doubling from the 730-day default up to 1,460 days, data permitting) until it holds the trade floor before judging.
2. Timeframe sweep — backtests across 15m, 1h, 4h and 1d and keeps the best. A strategy that declares a timeframe (_timeframe in its params, or its idea's timeframe) is swept and judged on that timeframe only.
3. Optimization — a parameter search over the strategy's space, including execution settings such as stop-loss, take-profit and leverage.
4. Apply optimized defaults — writes the best parameters into the strategy, but only if they pass an out-of-sample acceptance check.
5. Confirmation backtest — re-runs the strategy with those parameters.

Then the five robustness tests run:

With the Default stance preset, the required tests are walk_forward and param_jitter (gauntlet.required_tests), and the minimum robustness score is 30. Cost stress is required under Strict, and the paper→live gate always re-checks it. An empty list enforces all five. Walk-forward is mandatory: a config that drops it is repaired on load, because without it there is no honest out-of-sample signal.

These criteria decide each test's verdict. Under the default research door a FAIL doesn't stop the strategy at the door: it is recorded and enforced at paper → live. The door itself only enforces floors from this battery: no loss under 2× costs (a positive cost-stressed Sharpe) and a Monte Carlo 95th-percentile drawdown of at most 40%. Strict paper makes a failed required test block paper again.

Each test must persist a verdict (pass, fail or blocked) before the door into paper is judged. A test that errors or times out produces a non-result, never a merit verdict: it is re-run, not counted as a failure. A test with nothing to measure (for example jitter on a strategy with no tunable parameters) records NOT_APPLICABLE.

Walk-forward analysis

Walk-forward is the spine of the battery. In the gauntlet, the optimizer's selection window is the in-sample prefix, and the whole part of the window the optimizer never saw is tiled into out-of-sample folds. Every unseen bar counts toward a fold. (A standalone walk-forward from the strategy page still splits the data into folds with a train ratio: 5 folds, 70% in-sample by default.)

• Folds with fewer than 5 OOS trades are excluded from both sides of the pass rate, so a thin-trading strategy can still pass on the trades it does make.
• Walk-forward applies realistic costs: 4.5 bps fees and 2.0 bps slippage per trade by default.
• The window is sized from the strategy's own trade rate, and also reserves enough unseen days for the Monte Carlo test to see a fair number of trades.
• Thin folds are not a merit failure. When a required walk-forward ends with some, but fewer than 2, judgeable folds, the result is insufficient evidence (wfa_window_insufficient) and is re-runnable. Under the research door a thin walk-forward is recorded and passed on to the door instead; the fold count is enforced at paper → live. Folds that can't be judged never count toward a promotion.

The overall walk-forward verdict can be FAIL for a non-fold reason (a weak average in-sample Sharpe, or IS→OOS degradation over the limit) while the fold pass rate still clears the 33% floor. The gauntlet then lets the strategy proceed, and the fold floor is re-checked where it is enforced: at paper → live by default, at the door under Strict paper. These fold-rescued runs carry an amber PASS* badge, so a rescue is never mistaken for an outright pass.

Parameter jitter

Jitter nudges each tuned parameter by up to ±10% and reruns the backtest, up to 30 times. The number of reruns is sized to fit the time the step has left, and the reference (unperturbed) rerun goes first. It needs a baseline of at least 10 trades to measure anything. If a sweep falls short of its plan and fewer than 10 reruns finish, the row is an errored non-result and the gauntlet retries the step. If most reruns survive, the parameters describe a real basin of edge rather than a lucky coordinate. The Default pass floor is 50% (Strict: 60%).

Cost stress

Cost stress replays the baseline backtest's own window at 2× fees and slippage and measures how much the Sharpe decays. Only the costs differ, so the verdict is about costs, not about whether the strategy suits the most recent years. The edge must survive: degradation ≤ 60%, and for the strict live gate a stressed Sharpe ≥ 0.3. The door into paper only asks that the strategy doesn't lose money at 2× costs.

Monte Carlo and regime split

Monte Carlo resamples the trade sequence to estimate tail risk. At least 65% of resamples must stay profitable, and the 95th-percentile drawdown must stay at or under 40% (an editable safety floor caps any relaxation at 50%). It bootstraps the trades the baseline already produced, so it measures sensitivity to trade order and the drawdown tail. It does not test whether the edge is real: a curve-fitted strategy with a lucky trade set sails through. Catching that is the job of walk-forward, regime split, the Deflated Sharpe gate and the held-back test.

Regime split partitions trades by market regime at entry (trending up, trending down, range-bound, high volatility) and requires the strategy to be profitable in at least half of them, so it isn't just a bull-market artifact.

The checks after the battery

More checks decide what the battery's results mean. By default only the door's floors stand between a strategy and paper; the selection-bias checks bite at paper → live.

The held-back test is the last check before live by default (before paper under Strict paper). The most recent quarter, 3–6 months of data, is sealed away from all research, including the gauntlet's own backtests. Each candidate gets one walk-forward on that held-back period, run automatically once every other check at its gate passes, and never re-run for the same parameters. See promotion gates for the pass rules, settings and reason codes.

Breadth test

The breadth test runs the strategy's frozen code and parameters, unchanged, on 15 coins over sealed research data, one coin at a time. A real edge in a generic mechanism (trend, mean reversion, order flow) should show up on most liquid coins; one that appears only on its home coin is more likely fitted to it. The reading is one of:

• General — at least 5 other coins traded, at least 60% of them made money, and their median Sharpe is above 0.
• Home coin only — the home coin made money but at most 40% of the others did.
• Mixed — neither clearly general nor home-only.
• Too few coins — fewer than 5 other coins produced trades.

It also shows a sign test, labelled optimistic because coins move together. Run it from the strategy page: Robustness → Breadth → Run breadth test, or from a terminal with python -m forven.agent breadth <strategy_id>.

The verdict matters in three places:

• The paper queue. It adds to a waiting strategy's rank: General moves it up, Home coin only moves it down. See the paper pool.
• Going live. By default (Live gate: require the breadth test) a strategy needs a finished run on its current rule with any verdict but Home coin only. A missing or out-of-date run is started from the gate, and the strategy keeps paper trading meanwhile.
• Multi-coin paper. A paper strategy that reads General also paper-trades the coins where its rule made money, as one multi-coin paper strategy.

The paper pool's hourly job starts one breadth test per pass, paper strategies first.

The robustness score and the paper gate

The gauntlet produces a robustness score from 0 to 100 and a verdict per test. Under the default research door, the door into paper requires:

• at least 20 confirmation trades
• total return ≥ 0% and maximum drawdown ≤ 30%
• OOS profit factor ≥ 1.0 and an OOS Sharpe above 0
• no loss under 2× costs, and a Monte Carlo 95th-percentile drawdown ≤ 40%
• no implausible metrics
• a saved verdict for every required test

The robustness score (≥ 30), the walk-forward fold rate, parameter jitter, regime split, the Deflated Sharpe (≥ 0.90) and the held-back test are recorded at the door and enforced at paper → live. Strict paper enforces them all at the door, with an OOS profit factor of at least 1.05, as before the gate plan.

These thresholds are deliberately reachable so the funnel keeps moving while still filtering clear losers; the much stricter paper→live gate guards capital. Relaxed presets and custom values are clamped by the safety floors. Promotion into paper is automatic by default; when the paper pool is full, the strategy waits in a queue ranked by its recorded evidence.

Running the gauntlet

Everything happens on the strategy's page, which you open from The Forge. On a fresh install the system is in Manual mode, so nothing runs on its own: start the work yourself. In Semi or Auto mode the pipeline starts the gauntlet for strategies in quick_screen automatically.

For a guided version with screenshots, follow the Run the gauntlet walkthrough.

The page calls these endpoints under the hood:

# Where is this strategy in the gauntlet?
curl.exe http://127.0.0.1:8003/api/gauntlet/strategies/$StrategyId/status

# What's left before promotion, and why is it blocked?
curl.exe http://127.0.0.1:8003/api/lifecycle/strategies/$StrategyId/readiness
curl.exe http://127.0.0.1:8003/api/lifecycle/strategies/$StrategyId/explain

# Retry one blocked step, or resume / cancel a whole workflow
curl.exe -X POST http://127.0.0.1:8003/api/gauntlet/steps/$StepId/retry
curl.exe -X POST http://127.0.0.1:8003/api/gauntlet/workflows/$WorkflowId/resume
curl.exe -X POST http://127.0.0.1:8003/api/gauntlet/workflows/$WorkflowId/cancel

When steps block or fail

The gauntlet doesn't retry forever, and it doesn't loop:

• Merit failure. The strategy genuinely failed a gate. There is no auto-retry: the workflow ends and the strategy is archived to the graveyard. You can bring it back with Recover from graveyard. Under the research door, a FAIL from walk-forward, jitter, regime split, Monte Carlo or cost stress isn't one: the step passes, the FAIL is recorded for the live gate, and the workflow goes on to the door.
• Nothing measured. A screen with no trades, code that can't run, or transient blocks that exhausted their retries are not merit verdicts: the strategy is archived as untestable (untestable:<code>), which skips the failure post-mortem.
• Transient block (blocked_data, blocked_runtime). A data or runtime hiccup. Retries with backoff (2/4/8/16/30+ minutes) up to eight attempts.
• Slot contention (gate_contention). The market slot is held by an incumbent; the step waits until the slot frees or a dethrone resolves.
• Evidence absence (stale engine or data artifacts, too few judgeable walk-forward folds, a pending held-back test, an uncomputable DSR). Re-queued automatically. None of these counts toward the five-strike auto-archive.
• Paper pool full (paper_queue_full). The strategy passed the door and waits in the gauntlet stage for a paper slot, ranked by its recorded evidence. Capacity, not merit: it never counts toward the auto-archive.
• Spent held-back budget (holdout_budget_exhausted). Under Strict paper, the strategy waits in the gauntlet and is re-checked every 6 hours until the next quarterly roll. By default the held-back test runs at paper → live, so the wait happens in paper instead.
• Three demotions. A strategy sent from the gauntlet back to quick_screen a third time is archived as untestable:max_demotions.

A step stuck in running past gauntlet.async_result_max_age_minutes (default 60) is recovered automatically. Gauntlet work runs on a bounded pool of workers (set by the throughput preset), keeps a durable history, and preserves failed evidence on recovery rather than inventing a pass.

Tuning the gauntlet

Most users should pick a stance preset in Settings → Lab and adjust Robustness Gauntlet (Step 2) if needed. Behind the UI, the thresholds live under the gauntlet.*, walk_forward.* and robustness_thresholds.* key groups:

{
  "gauntlet": {
    "required_tests": ["walk_forward", "param_jitter"],
    "min_robustness_score": 30,
    "max_drawdown_pct": 0.30,
    "mc_max_dd_p95": 0.40,
    "wfa_baseline_hurdle_paper": "observe",
    "wfa_baseline_hurdle_live": "enforce"
  },
  "walk_forward": {
    "n_folds": 5,
    "in_sample_pct": 0.70,
    "fee_bps": 4.5,
    "slippage_bps": 2.0
  },
  "robustness_thresholds": {
    "monte_carlo_percentile_min": 0.65,
    "param_jitter_pass_rate_min": 0.50,
    "cost_stress_max_degradation_pct": 60.0,
    "regime_split_profitable_min": 0.50,
    "wfa_fold_pass_rate_min": 0.33,
    "deflated_sharpe_gate_enabled": true,
    "min_deflated_sharpe": 0.90
  }
}

Drawdown and return values are stored as fractions (0.30 = 30%) and shown as percentages. Any value you relax is still clamped by the safety floors. The door into paper has its own settings (Settings → Lab → Door into paper, the paper_door.* keys); see promotion gates. See the configuration reference for precedence and the full key list.

Honest caveats

• Thin data surfaces warnings. Walk-forward flags short OOS windows and sparse folds. Read them; they usually point at a data gap, not a crash.
• Engine and data upgrades re-queue validation. Verdicts are stamped with the engine version and data semantics they were scored on; when either changes, old verdicts go stale and the gauntlet re-runs. Expect a burst of re-validation after updating Forven.
• Implausible metrics are rejected, not celebrated. A Sharpe ≥ 5 or profit factor ≥ 8 is treated as a look-ahead leak and blocked at both the quick screen and the door into paper.
• testing_mode bypasses the quick-screen and gauntlet gates for rapid iteration, but never the paper or live gates.
• Passing earns paper, not capital. The path to real money runs through the strict paper→live gate in promotion gates.

The gauntlet is built to make killing fragile strategies unavoidable: a verdict it records at the door is enforced before any real capital.

Related

• The pipeline — the lifecycle the gauntlet sits inside
• Promotion gates — the door into paper, the paper pool, the held-back test and the paper→live gate in detail
• Metrics — every metric the gauntlet reads, and why to trust OOS
• The Forge — where you open a strategy and follow its gauntlet