Metrics

Every backtest Forven runs produces a block of metrics describing how a strategy behaved on historical data. This page explains each metric, the flags that mark a metric untrustworthy, and the one distinction that matters most: in-sample versus out-of-sample.

The short version: a strategy is split into a training window and an unseen window. Metrics computed on the unseen window are the ones the promotion gates lean on, and the ones you should read too.

Forven is a research tool. The numbers on these pages describe past behaviour on historical data. They are illustrative, they are not predictive, and nothing here is financial advice.

In-sample vs out-of-sample

Forven splits every backtest's data 70/30 at a fixed index:

• In-sample (IS) — the first 70% of the data. Signals are generated and tuned here. IS Sharpe is optimistic and does not predict live behaviour.
• Out-of-sample (OOS) — the last 30%, unseen during signal generation. OOS metrics are the ground truth for fitness gates and any deployment decision.

Both windows end at the held-back test's cutoff: by default the most recent 3–6 months (one quarter back) are sealed away from every research backtest and used once, as a final check before live (before paper under Strict paper).

The lab renders IS and OOS side by side. A strategy that looks excellent IS and poor OOS is overfit. A strategy that holds up across both is worth promoting to paper (the public site calls this a "candidate").

A backtest result returns three blocks: in_sample, out_of_sample, and robustness. When you read a result, start with out_of_sample.

The core metrics

compute_metrics() derives 30+ values from each trade ledger. The ones you will read most often:

These are computed identically for the IS and OOS blocks, so you can compare the same strategy across the train/test split directly.

How Sharpe is computed

Sharpe and Sortino are computed from the per-bar returns of the mark-to-market equity curve — every bar counts, including bars where the strategy was flat — and annualized by bars per year. That is a calendar measure: a strategy that trades rarely is not flattered by averaging only its few trades. The older per-trade figure is still reported as trade_sharpe / trade_sortino for comparison. (The paper→live gate reads forward trades differently again: its sequential test uses the per-trade Sharpe of the forward paper trades, and the Fixed floors rule a per-trade t-statistic — see promotion gates.)

Deflated Sharpe

Strategies are optimized before they are validated, and the best of many tries always looks better than it is. The deflated Sharpe ratio estimates the probability that a Sharpe is genuine given how many variants were tried. Forven always computes it and shows it in The Forge's DSR / Rob column and on the strategy page. The Deflated Sharpe gate is on by default and rejects a DSR below 0.90 at paper → live, where it is computed on the backtest trades plus the strategy's forward paper trades. At gauntlet → paper the default research door only records it, and it ranks the paper queue; Strict paper rejects there instead (see promotion gates for the settings).

How the trials are counted:

• Optimizer trials. Every optimization run on the strategy counts, not just the latest. An optimization that searched only one combination searched nothing, so the trial count falls back to the default of 50. Declare a parameter_space() to have the real search measured. A sandboxed strategy (registered by an agent or through the AI Drop Zone) that declares none gets a small automatic grid instead: ×0.8, ×1.0 and ×1.2 of up to three of its numeric defaults, at most 27 combinations, and the DSR is charged the combinations actually tried. Automatic search grid (Settings → Lab → Robustness Gauntlet (Step 2), on by default) switches it off.
• Failed siblings. The trial count is multiplied by the sibling strategies from the same family and asset that failed in the last 90 days, counting only siblings from the same search origin (AI Drop Zone, in-app agents, or another source). A failed sibling counts as a trial only if it completed a backtest that reached the quick-screen trade floor and has no untestable or lookahead verdict.
• Which trades. By default the DSR is computed over every trade of the gauntlet's confirmation backtest, in-sample and out-of-sample, both after the optimizer's selection window.

A DSR that can't be computed (for example the per-trade returns were never stored) is pending evidence, never grounds for archiving: the gate waits with dsr_unavailable until a re-run backtest makes it computable.

In the Strategy Creator, the deflated Sharpe in the vitals charges for every result you have seen this session; the top bar shows how many.

The strategy page's Deflated Sharpe card still calls the value "advisory" and marks 0.95 as the conventional bar. The gate itself is on, at 0.90, and by default bites at paper → live.

Breakdowns

Two slices help you see where performance came from:

• by_side — long vs short. A strategy that only makes money on one side is really half a strategy.
• by_regime — performance split across the four market regimes (trend_up, trend_down, range_bound, high_vol). This shows whether an edge is regime-specific or broad-based.

Robustness and walk-forward

A single 70/30 split is one test. The gauntlet goes further with walk-forward analysis (WFA) — N-fold cross-validation that re-runs the IS/OOS split across multiple windows.

Two robustness figures matter:

• Robustness score — defined as 1.0 - max(IS→OOS Sharpe degradation, 0). A high score means performance held up from training to test; a low score means the edge decayed out-of-sample.
• Degradation % — 1 - (avg_oos_sharpe / avg_is_sharpe) aggregated across folds. A standalone walk-forward run reports PASS when degradation is below 50% and there are at least 5 OOS trades. Before live, the walk-forward needs at least 2 judgeable folds with at least 33% of them OOS-positive, and the stricter degradation limit (at most 35%). By default the fold checks are recorded at the door into paper and enforced at the paper→live gate; Strict paper enforces them before paper. See the gauntlet.

Defaults for a standalone walk-forward: 5 folds, 70% IS / 30% OOS per fold (walkforward_folds, walkforward_train_ratio). WFA needs at least 420 bars of data. Inside the gauntlet it works differently: the optimizer's selection window is the in-sample prefix and the whole unseen part of the window is tiled into OOS folds, sized from the strategy's own trade rate so each fold has enough trades to judge. See the gauntlet. Adequacy warnings are logged when an OOS window is shorter than 30 days or a fold has too few bars — the numbers still compute, but treat them as weak.

Reliability flags — read these first

Short windows produce metrics that look spectacular and mean nothing. Forven attaches boolean flags so the gates and the UI can suppress numbers that aren't yet trustworthy.

Two reasons these exist:

1. Sharpe annualization inflates wildly on short samples. Under 20 trades, the figure is flagged unreliable.
2. Annualized return is capped at a 3-month window for display. A 25-day run can report a return in the thousands of percent — that is an artifact, not a result. The flag suppresses the display; it does not change the underlying fitness math.

When a flag is false, the number is hidden rather than shown with a false air of confidence. That is deliberate: transparency over confidence.

Funding-cost provenance

Forven applies perpetual-futures funding to backtest trades by default (backtest_include_funding), using Binance funding prints normalized per hour by each print's own interval and accrued inside the execution kernel exactly once. Because a funding-blind backtest can look better than reality, two flags record what actually happened:

• funding_applied — at least one trade carried a funding cost.
• funding_complete — every trade had complete funding history.

If funding_complete is false, the promotion gates may reject the run rather than promote a strategy validated in a window with missing funding data. On a fresh install the first backtest auto-backfills funding history from the exchange, so this self-heals over time.

How fees, slippage, and leverage enter the numbers

Metrics already include trading costs. The defaults applied to each round-trip:

Fees and slippage are charged on entry and on exit. A strategy's metrics therefore reflect a return after costs — there is no separate "gross vs net" toggle to forget. In risk-based sizing, leverage does not change the loss at the stop: the position shrinks as leverage rises.

Metric semantics are versioned

Every verdict is stamped with the backtest engine version and a fingerprint of the data it was scored on. When the way a metric is computed changes — the move to per-bar Sharpe was one such change — older verdicts are marked stale and re-validated automatically, rather than silently compared against new ones. See promotion gates.

How the gates read metrics

The promotion gates compute a single fitness score (0–100) from five weighted factors, all read from the OOS metrics:

fitness = 30% Sharpe (capped at 3.0)
        + 20% win rate
        + 20% profit factor (capped at 5.0)
        + 15% max drawdown (penalty: 0 at 10%+ drawdown)
        + 15% trade-count bonus (min 20 trades)

The thresholds that move a strategy along the pipeline:

• >= 60 — eligible for paper (the public "candidate" stage)
• >= 70 — eligible for deploy / live
• < 40 — eligible for retirement

A metrics integrity check (check_metrics_integrity()) runs before any promotion. Impossible ratios, NaN, and other anomalies quarantine the result and block promotion — a strategy cannot be promoted on numbers that don't add up.

Reading a result via the API

You can pull the raw metrics directly. Submit a run and read the result:

# Submit a backtest
curl.exe -X POST http://127.0.0.1:8003/api/backtesting/run `
  -H "x-api-key: $env:FORVEN_API_KEY" `
  -H "content-type: application/json" `
  -d '{ "strategy_type": "rsi_momentum", "asset": "BTC", "timeframe": "1h" }'

# Retrieve the full ledger + metrics
curl.exe http://127.0.0.1:8003/api/results/<result_id> `
  -H "x-api-key: $env:FORVEN_API_KEY"

The response carries the in_sample, out_of_sample, and robustness blocks described above. In the UI, the same data renders in a backtest's full report — chart, trades table, and metrics panel, with regime shadings on the chart — and in All Backtests (/lab/backtests).

Caveats

• Trust OOS, not IS. In-sample metrics are tuning artifacts. The quick screen judges the whole window to get a fair sample, but every later gate leans on out-of-sample evidence, and so should you.
• Honor the flags. A great Sharpe under 20 trades or a four-digit annualized return on a three-week window is noise, not edge.
• Infinite profit factor is real. A strategy with zero losses reports profit_factor = inf plus profit_factor_is_infinite. It usually means too few trades, not a perfect strategy.
• Costs are baked in. Fees, slippage, and funding are already in the metrics; don't double-count them.
• Past is not prologue. Every number here describes historical behaviour on historical data. It is illustrative, not predictive, and nothing here is financial advice.

Related

• The gauntlet — the full robustness battery these metrics feed
• Backtesting a strategy — where these metrics are produced
• Promotion gates — how fitness and reliability flags gate the pipeline
• Market regimes — the regimes behind the by_regime breakdown