Data sources

This is the source-by-source reference for Forven's market-data layer. The data manager page covers the /data screens and the day-to-day workflow; this page documents what each source provides, how symbols are formatted, the research universe, the enrichment streams that ride alongside candles, how data provenance protects verdicts, and the API.

It is written for developers and operators who need exact source names, symbol conventions, settings, and endpoints.

The sources at a glance

Data → Get data → A market downloads candles from one of these places, under Where from:

None of these needs a key. Each venue row in Get data says whether that venue lists the market, and a download from a venue series never feeds a backtest unless it is the market's research series.

Binance and the research series

Crypto history is stored perp-canonical — with the perpetual-futures semantics Forven would trade — and every stored series carries a venue stamp. Paper trading reads Binance prices live (market_data_source, default binance), so paper and backtest judge a strategy on one venue's prices.

Binance Vision

Binance Vision is the bulk downloader for data.binance.vision. It probes for a symbol's true start date, tracks what it has covered, resumes interrupted runs, and parses the older header-less archives. It also supplies derivatives metrics — open interest, long/short ratio, and taker volume. On a series page or in the Catalog, Extend history uses it to reach further back.

CSV import

Upload a file on Data → Import file, map its columns, check the preview, then import it as a New series or Add bars to it (patch). Where the file and stored bars differ, you choose Keep the stored values or Use the file's values. Pick one canonical source per symbol and timeframe, and stick to it.

The research universe

The symbols you trade and the symbols you research are different sets. The research universe is a liquidity-ranked set of Binance USD-M perpetuals that Forven keeps deep history for, so ideas can be tested on more markets and more history than you trade:

• A symbol registry tracks every perp listed on the venue, with its listing and delisting dates.
• A timeframe ladder sets the depth: 1h, 4h and 1d for the whole universe, 15m and 5m for the top tier, and 1-minute bars for the top few.
• Delisted symbols keep their history, so research is not biased toward survivors.

Settings → Data → Research universe holds the plan:

Downloads never start on their own. To fill the plan, open Data → Get data → A research universe and click Bring the universe up to plan (or Resume the seed). It runs as one job from Binance Vision archives and resumes where it stopped. Over the API: GET /api/data/universe, POST /api/data/universe/config, POST /api/data/universe/seed.

Symbol formats

The same instrument is spelled differently by each source. Forven normalizes at the import and export boundaries, and the symbol registry treats spelling variants of one market as the same market — SOLUSDT, SOL/USDT, and SOL-USDT all resolve to one pair.

Asset class is detected from the symbol's shape, so you do not declare it.

Strategies, ideas and backtest requests share one symbol check. A dataset id where a symbol belongs (SUI/USDT-1h-ccxt, dataset-3-ETH/USDT-4h) is repaired to the pair, and something that isn't a market, such as 1/USDT, is refused with a reason that says why. 1INCH and 1000PEPE are real coins and pass.

Enrichment streams

Candles are the spine. Alongside them, Forven collects derivative and context streams and merges them onto bars for a backtest. In Get data, the Perp add-ons checkboxes choose which ones to fetch with a market (funding and open interest are ticked by default):

Enrichment happens on demand during a backtest: each bar is matched to the nearest prior value of each stream (a merge-asof join), so no future information leaks backward. Two further rules keep it honest:

• Bucket-aggregate streams are shifted to bucket close. Taker flow, liquidations, and basis summarize a forward window that is only known at its close, so their timestamps move to the close before merging.
• Missing stays missing. If a stream is unavailable for a symbol or window, its columns are still on the frame but empty (NaN) — never zero-filled into something a strategy might trade on. A strategy that reads one simply doesn't fire on those bars, instead of crashing. Each stream also has a maximum age based on its own cadence, so a stale print is not carried forward forever.

Funding and open interest for backtests and paper come from Binance, not HyperLiquid, so the enrichment matches the series the strategy is validated on.

Keeping data fresh

Once a series is stored, the Data Collector (freshness SLA) job keeps it current: every 120 seconds it refreshes the most overdue series first, within a 90-second budget per run and 120 requests a minute per venue. How late a series may be depends on who reads it — live, paper, gauntlet, research universe, or nothing — as described in data freshness. A series that finds no newer data three refreshes in a row (a delisted or renamed symbol) is frozen. The settings are under Settings → Data → Freshness SLAs and Collection.

Going the other way, when a run needs older history than a stored series holds but a finer timeframe of the same symbol goes back further, the missing older bars are rebuilt from that finer series (complete buckets only) before anything is downloaded. Native bars win wherever both exist.

Data provenance

A verdict is only as good as the data it was scored on. Every persisted backtest is stamped with a compact fingerprint of its data's semantics: each enrichment stream's print cadence, value scale, and whether it existed. Appending new bars never changes the fingerprint; a cadence flip, a unit rescale, or a stream appearing or disappearing does.

When the fingerprint of today's data no longer matches the one a verdict was scored on, the verdict is treated as stale: the gauntlet reads it as missing evidence and re-runs through the normal flow. See promotion gates. This is what stops a transient data artifact — a funding file rewritten mid-day, say — from quietly "validating" a strategy.

The market calendar

Crypto trades around the clock, but equities and forex do not. Forven carries a market calendar — NYSE hours and holidays, forex session windows, always-on crypto — so a stock backtest does not treat an overnight gap as a missing bar.

How data lands on disk

Every persisted bar is a closed bar. Writes are atomic: Forven writes a temporary file, fsyncs it, then renames it into place (retrying the rename on Windows share-violation races). A crash leaves a stray .tmp that a background orphan scan cleans up. Before any bar enters the lake it passes an OHLC sanity check — high ≥ low, open and close within the bar's range, positive prices, non-negative volume — and a corrupt Parquet file is detected and skipped rather than crashing a read. Dataset paths are confined to the data directory.

Configuration

Where data is stored

The Data Engine

Settings → Data → Data Engine controls how data is read. Data Engine enabled is on by default: candles and enrichment are read through DuckDB over the Parquet lake. Turning it off falls back to a legacy read path that is kept only as a fallback and will be removed. The same section holds Point-in-time mode (latest, or an as-of pin for reproducible backtests) and the Source reconciliation gate, which holds a strategy out of paper and live when its backtest price source diverges from the live trade venue by more than Max source divergence (default 2%).

Remote engine (optional)

For a shared-server setup, Settings → System → Remote engine offloads backtests to a remote Forven engine process.

API surface

The data layer is served on the local backend (127.0.0.1:8003). The endpoints behind the Data screens:

The older endpoints — /api/datasets, /api/data/ingestion/submit and /api/data/ingestion/runs, /api/upload/csv — still answer, for scripts written against them.

See the API reference for the full router catalog and authentication.

Caveats

• Venue series don't feed backtests. Only the research series does, unless Binance doesn't list the market.
• Hyperliquid history is short. Only the latest 5,000 candles per timeframe are available there.
• Source metadata is last-write-wins. Pick one canonical source per symbol.
• Point-in-time is OHLCV-only. Enrichment streams cannot be reconstructed as-of a past date.
• Deep history is large. A big research universe with minute bars takes hours to download and gigabytes of disk. Get data and Set up data show an estimate before you start.

Forven is a research tool. Clean data improves the honesty of a backtest, but no dataset makes a result predictive of future performance, and nothing here is financial advice.

Related

• Managing market data — the /data screens, from download to delete.
• Health monitoring — freshness tiers and data alerts.
• Promotion gates — how stale data provenance re-queues validation.
• Environment variables — every data and storage variable in detail.