Database & maintenance

Forven keeps everything on your machine, in one place. The cost of that local-first design is that the store grows as the research loop works: backtests, activity rows, notifications, and heartbeats accumulate. Maintenance keeps that store fast and bounded without losing the data you care about — and mostly runs by itself.

This page is for operators. It covers the automatic jobs, the retention windows you can tune, database snapshots and how to restore one, reclaiming disk space, factory reset, and the one-time legacy home migration for returning users.

Where your data lives

All state lives under FORVEN_HOME, which defaults to ~/.forven. The pieces maintenance touches:

• forven.db — the source of truth (SQLite). Strategies, trades, agent tasks, scheduler jobs, notifications, the activity log, and the key-value settings store all live here.
• forven.db-wal and forven.db-shm — the write-ahead log and its index. SQLite runs in WAL mode for concurrency, so the live database is three files, not one.
• forven_lab.db — the lab database.
• workspace/ — markdown workspace files for the agents.
• backups/database/ — the automatic snapshots (below).

Market data lives in a separate Parquet data lake, with its own trash and housekeeping — see market-data storage below.

The automatic jobs

Three scheduler jobs keep the store in shape:

The two daily jobs are scheduled in the America/Halifax time zone out of the box; change a job's time or zone from the scheduler if you want them to run in your own quiet hours. Pruning runs in small batches (about 500 rows per transaction), so it never holds the write lock long enough to stall trading or scanning.

Without pruning, a few tables grow without bound — on busy installs the backtest results table has reached hundreds of megabytes and the activity log hundreds of thousands of rows. Your numbers depend on how hard you run the loop, but an untrimmed store gets slow.

Retention windows

Set a window to 0 to stop pruning that table. The funding-history floor exists because the data collectors re-download two years of funding history: a shorter window would just delete rows the next collection fetches again.

If you are investigating an event from more than 60 days ago, its notifications may already be pruned, and heartbeats older than two days are almost certainly gone. Keep your own copies if you need a longer audit trail.

Tuning them

Settings → System → Database maintenance shows four of the windows — Backtest trash retention, Activity log retention, Scanner results retention and Gate rejections retention — and the Daily VACUUM switch. All the windows are stored with the pipeline settings, so you can set any of them over the API:

# Read the current pipeline settings (includes the retention_* windows)
curl.exe http://127.0.0.1:8003/api/settings/pipeline

# Change a window (send x-operator-key if FORVEN_OPERATOR_KEY is set)
curl.exe -X PUT http://127.0.0.1:8003/api/settings/pipeline `
  -H "Content-Type: application/json" `
  -d '{ \"retention_activity_log_days\": 120 }'

The next maintenance run uses the new windows. Longer windows keep more history at the cost of disk and query speed.

Backups

Snapshots are written to FORVEN_HOME/backups/database as forven-<reason>-<timestamp>.db — a daily one from the backup job and a pre-migration one before any schema migration runs. Each snapshot is a consistent single-file copy, safe to take while Forven is running.

Retention is per reason: the last 7 daily snapshots and the last 3 pre-migration ones (3 for anything else). A size ceiling bounds the whole folder — the larger of 2 GiB or six times the database size — and a snapshot is refused outright if writing it would leave the disk dangerously full. Both are adjustable with environment variables.

Steps: restore a snapshot

1. Stop Forven — close it from the launcher, or stop start_all.
2. Keep the current database aside: rename forven.db to something like forven.db.broken, and move forven.db-wal and forven.db-shm out of the folder with it.
3. Copy the snapshot you want from backups/database to FORVEN_HOME and rename it forven.db.
4. Start Forven. Any migrations the snapshot predates run again on startup.

What you'll see: the app comes back in the state of the snapshot — strategies, trades, and settings as they were at that time. Anything that happened after it is gone, which is why step 2 keeps the old file.

A copy of forven.db alone, taken by hand while Forven is running, is not a complete backup — the latest writes may still be in the WAL. Use the snapshots, or stop Forven before copying.

Reclaiming disk with VACUUM

Pruning frees space inside the database file, which SQLite reuses, but it doesn't shrink the file on disk. VACUUM rewrites the file compactly. It takes an exclusive lock and can take a while on a large database, blocking every other writer until it finishes — which is why it is off by default.

• Automatic: turn on Daily VACUUM in Settings → System → Database maintenance, and the daily maintenance job runs it after a prune.
• By hand: stop Forven first, then run it with the app closed:

python -c "import sqlite3; c = sqlite3.connect(r'C:\Users\you\.forven\forven.db'); c.execute('VACUUM'); c.close()"

What you'll see: forven.db shrinks to roughly the size of its live data.

Stopping Forven stops paper trading until you start it again, and does not close any open positions. On a paper install that is harmless. If you have wired up unsupported live trading, exit positions deliberately before any disruptive maintenance.

Market-data storage

The candle lake and the enrichment streams have their own housekeeping, on Data → Storage:

Reclaiming moves files to the trash first; nothing is deleted until the trash is emptied or the retention runs out. Deleting a series from the Catalog shows which strategies use it and asks for a typed confirmation, and the series goes to the same trash.

The defaults live in Settings → Data → Storage:

Factory reset

When you want to clear data rather than prune it, Settings → Danger Zone → Factory reset wipes selected categories. You tick the categories to keep, click Wipe & factory reset…, then type FACTORY RESET and click Wipe everything:

Factory reset is destructive and operator-gated (POST /api/system/factory-reset; GET /api/system/factory-reset/categories lists the catalog). Treat it as a last resort, and make sure you have a recent snapshot first.

Legacy home migration

Earlier builds stored state under ~/.juddex or ~/.judex. On first startup Forven detects those directories and merges them into ~/.forven, renaming the database files (juddex.db → forven.db). Returning users keep their history with no manual steps.

• Workspace writes are mirrored back to the legacy workspace/ folder for compatibility. Where the two copies of a file have diverged, the longer one wins — check a backup if a workspace file looks wrong after migration.
• The encryption key lives outside FORVEN_HOME (%LOCALAPPDATA%\Forven\.forven_key on Windows), while the encrypted auth.json lives inside it. Back them up together, or encrypted credentials can stop decrypting. See encryption & secrets.

Caveats

• The live database is three files. Use the snapshots, or stop Forven before copying forven.db by hand.
• Notifications and heartbeats are short-lived. Their 60-day and 2-day windows mean old audit rows disappear.
• VACUUM is exclusive. Run it only while the app is quiet, or stopped.
• Maintenance never touches positions. Pruning and snapshots operate only on local records.
• The encryption key doesn't live with your data. Back it up with FORVEN_HOME.

Forven is a research tool. Nothing here is financial advice, results are not predictive, and maintenance routines operate only on local records — they do not affect any exchange position.

Related

• Updating & backups — updating Forven, and the snapshot taken before migrations
• Configuration reference — settings precedence and where the retention keys live
• Health monitoring — component states and data-stream freshness
• Managing market data — the Data screens, from download to delete
• Environment variables — backup and log sizing overrides