Data Storage

All data is stored as JSONL files under velocirepo/data/. This entire directory is meant to be committed to git — it’s your permanent metric history. Data is organized into three categories:

Metrics

Sources like PyPI, CRAN, Homebrew, Plausible, OpenVSX, GitHub Traffic, YouTube, and LinkedIn store one JSON object per metric per day:

{"source":"pypi","metric":"daily_downloads","project_id":"plotnine","target":"plotnine","date":"2026-06-15","value":1523}
{"source":"openvsx","metric":"total_downloads","project_id":"quarto","target":"quarto/quarto","date":"2026-06-15","value":1250000}

Metric names are prefixed with daily_ (for deltas — values that reset each day) or total_ (for snapshots — cumulative totals at a point in time). Homebrew metrics use their own naming (downloads_30d, downloads_365d, etc.).

total_* series are stored only on the days their value changes. Each project’s _watermark.json records the last day each target was checked, and the metrics_filled view fills in the days in between.

Fields:

Field Description
source Source name (pypi, cran, homebrew, plausible, openvsx, github-traffic, youtube, linkedin)
metric Metric name (e.g., daily_downloads, total_views, daily_pageviews)
project_id Project ID from your config
target Specific package, repo, site, or extension being tracked
date Date of the measurement (YYYY-MM-DD)
value Integer value
extra Optional key-value metadata (e.g., {"video_id": "..."} for YouTube)

Events

Event sources store individual events rather than pre-computed counts, giving you full historical detail including who performed each action and when. Currently, GitHub is the only event source.

{"source":"github","type":"fork","project_id":"quarto","target":"quarto-dev/quarto-cli","datetime":"2026-06-15T14:23:01Z","user":"alice"}
{"source":"github","type":"issue_close","project_id":"quarto","target":"quarto-dev/quarto-cli","datetime":"2026-06-15T09:11:44Z","ref":1234,"user":"bob"}

Fields:

Field Description
source Source name (e.g., github)
type Event type (e.g., fork, issue_open, issue_close, pr_open, pr_merge, comment)
project_id Project ID from your config
target Specific repo or resource being tracked (e.g., owner/repo)
datetime Full timestamp of the event (ISO 8601, UTC)
ref Issue or pull request number, for events about one
user Who performed the action, if known
extra Optional key-value metadata

These events are automatically aggregated into daily counts in the metrics DuckDB view (as daily_forks, daily_issues_opened, etc.) so you can query them alongside other sources.

Stars aren’t events. GitHub only lists the stargazers of repositories your token can administer, so velocirepo reads GitHub’s star history instead, which gives the number of stars per day for any repository. It’s stored as the daily_stars metric. Its days end in US Pacific time, and each fetch re-reads the last 7 days, since stars that are removed again lower the count of their day. The total_stars metric holds the star count GitHub reports on each fetch. Star events stored by earlier versions stay in the events view, but aren’t counted in daily_stars.

The star history covers a repository’s whole life. The first fetch of a repository without daily_stars reads all of it, so data from earlier versions gets complete star counts without re-fetching any events.

Content

Sources that implement the ContentProvider interface also write entity data to data/content/<source>/<project-id>/<name>.jsonl. These files use upsert-by-id (new entries are merged with existing ones) rather than date-partitioned concatenation.

YouTube (videos), LinkedIn (posts), RSS (feed items) and GitHub (issues, pull requests and repositories) are content sources. Entries are identified by their target and id. YouTube, for example, writes video metadata to velocirepo/data/content/youtube/<project-id>/videos.jsonl:

{"source":"youtube","target":"@Fireship","id":"ML3q7Ok4hJg","title":"God-Tier Developer Roadmap","published_at":"2024-03-15T16:00:00Z","duration":423,"tags":["programming","roadmap"]}

LinkedIn writes post metadata to velocirepo/data/content/linkedin/<project-id>/posts.jsonl. Content is exposed as the content DuckDB view; filter by source to query source-specific entities.

Concatenation

Daily files are concatenated into monthly files, and monthly files into yearly files, to keep the file count manageable for long-running histories. A month is concatenated on or after the 8th of the following month ([settings] concatenate_on_day), once every target and series of the project in the current config has been fetched past the month’s end. A target that’s further behind than its source’s freshness limit doesn’t hold the others up: velocirepo check reports it as data.freshness. A year is concatenated with its December, and yearly files are kept.

Waiting a week keeps corrections small: most problems surface in the first days, and while the month is still in daily files, a correction only touches those. A missed concatenation day is caught up by the next run. In the nightly workflow, concatenation is a commit of its own, checked to move records between files without changing them.

Data for a day whose month or year is already concatenated, from a backfill, a repair or a new target, is written into that period file. velocirepo never creates daily files next to one; validate-data reports such an overlap as something made outside velocirepo.

A write replaces, for every target and day it covers, the stored records of the metrics or event types it contains; other metrics of the same target and day stay. So a re-fetch corrects earlier values, and a repair can also remove an event that’s gone.

Migrating data

When a new version of velocirepo changes the on-disk data format, it tracks this with a schema version number in velocirepo/data/.schema-version. Commands like fetch, query, and export will refuse to run against stale data:

Error: data schema is at version 0, but version 1 is required; run `velocirepo migrate` to update

To migrate your data to the latest schema:

velocirepo migrate

Use --force to re-run all migrations from scratch:

velocirepo migrate --force

This is safe to run repeatedly — all migrations are idempotent. To merge data from an older schema into a current data directory (e.g., data from another repository), first migrate it in a data directory of its own, then copy the migrated files in. The migrations that move files only look at the old layout, and refuse to merge into directories that already exist.

Repository layout

You can either keep metrics in the same repository as your code, or create a dedicated metrics repository. A separate repo is useful when you want to track multiple projects in one place or keep metric history out of your main codebase.

Next to the data, velocirepo/ holds:

  • data/, the data directory, which queries, views and build-db read;
  • checks.toml, the findings you’ve accepted;
  • gate-state.json, the findings the nightly gate keeps open;
  • held-back/<source>/<project>/, data the gate holds back, in the same format as the data and invisible to queries.

Raw API responses archived for replays aren’t committed: they’re stored as release assets (see Keeping the data right).

Next steps