Keeping the data right

Third-party APIs change, break and return less than they should, and a bug or a hand-made commit can remove history that can’t be fetched again. velocirepo’s nightly workflow is built so that the worst outcome of any of that is new data that waits, not data that’s lost.

The nightly gate

velocirepo install-ci generates three workflows:

Workflow	Runs	Does
`velocirepo.yml`	nightly	the gate: fetch, check, commit, push
`velocirepo-heartbeat.yml`	three hours later	reports a nightly run that didn’t start
`velocirepo-digest.yml`	weekly	posts a digest of the data

They use the workflow’s own token for everything in the data repository. GITHUB_TOKEN, passed as github-token, stays the token velocirepo fetches with. No repository setting has to change.

The gate, velocirepo gate, does this every night:

Checks the commits made since its last run, such as a re-fetch run by hand and pushed. It remembers the last commit it checked in refs/velocirepo/checked.
Fetches, through yesterday in every source’s time zone.
Checks the night’s changes with velocirepo check.
Commits the data, as “Fetch metrics on 2026-10-21”, with how far each source is stored in the commit message.
Concatenates on or after the 8th of the month, in a commit of its own.
Pushes, and reports in a tracking issue.

The rule

A night only adds data. It never removes a stored record and never changes a stored value; content files, which are updated in place, are the exception.

The gate reports by default: it commits everything, and its job summary says what it would have done. With

[settings]
gate = "enforce"

it enforces the rule:

Removed records are restored, and reported as diff.records-removed.
Changed values keep their stored value, and are reported as diff.values-changed.
New records of a source and project with an error about them are held back, until the error is fixed or accepted. The other sources and projects land.

Held-back data

Held-back data is kept in velocirepo/held-back/<source>/<project>/, outside the data, so queries and views don’t see it, while the project’s data stays as it was. Its watermarks are held back with it, so nothing is skipped: the next night fetches on from the held-back data, and checks it again together with the new night. Once it passes, all of it lands.

Run the nightly workflow by hand to deal with held-back data yourself:

release: github/positron lands it regardless of its findings.
discard: github/positron deletes it, so it’s fetched again where the API still has it.

Findings

Every check reports findings with a severity and a stable ID:

Check	Reports
`fetch.failed`	a job failed; it stored nothing, and the next night fetches the same days again
`fetch.deprecated`	an API endpoint announced its removal
`diff.records-removed`, `diff.values-changed`, `diff.content-removed`	the data lost records, values or content entries compared with the previous commit
`diff.repaired`	the changes of a repair
`data.integrity`	duplicates, overlapping period files or temporary files; an error for duplicates added tonight
`data.freshness`	a target that’s further behind than its source allows
`data.volume`	a target that usually has data and suddenly has none
`reconcile.totals`	stored totals that differ from the API’s own totals
`reconcile.unavailable`	an API’s own totals couldn’t be read
`concatenate.records-moved`	concatenation changed the records instead of only moving them

Only errors about the night’s new records hold data back. Warnings never do, and neither do findings about data that was stored already: they’re reported until they’re resolved.

Run the same checks locally, against HEAD or between two commits:

velocirepo check
velocirepo check --base 03f2f5e66^ --head 03f2f5e66

Accepting a finding

A known false positive is accepted in velocirepo/checks.toml, next to the data, so accepting it is a reviewed change:

[[accept]]
id = "952675f07b21"
reason = "Two bot comments in the same second"
expires = "2027-01-01"

The tracking issue

The gate keeps one issue, labelled velocirepo-health, with the findings that are open. Its body is rewritten every night; it’s commented on only when findings change, and closed when none are left. [settings] owners are assigned to it, or mentioned for teams.

Reconciliation

Missing or duplicated history looks like normal data to every other check, so check compares stored totals with the totals the APIs compute themselves:

Every night, GitHub’s all-time numbers of issues, pull requests, merged pull requests, forks and stars, 40 repositories per GraphQL query.
Each repository once a week, its last complete month: GitHub’s search counts of issues and pull requests opened, closed and merged, cranlogs’ monthly downloads, and Plausible’s monthly pageviews and visits.

It uses at most 10% of each shared API quota, and the search API at most 1,500 times a night, at 25 searches a minute. What doesn’t fit is reported as quota.low and checked on a later night.

When GitHub refuses a request because a rate limit is reached, during a fetch or reconciliation, velocirepo waits until the limit resets (at most 65 minutes, as GitHub’s response says) and tries again, so a large first fetch of many repositories with long histories completes instead of starting over the next night.

The archive

Some data can’t be fetched again: snapshots such as star and reaction counts, GitHub traffic after 14 days, Homebrew, Open VSX and YouTube totals. The gate archives the raw responses of those requests, with the tokens left out, as assets of a monthly raw-YYYY-MM pre-release of the data repository. They’re never committed, so the repository doesn’t grow. [settings] archive = false turns this off.

A bug in how responses were processed is repaired by replaying them:

velocirepo download-archive 20261021T093105Z-12345-1 --archive raw/
velocirepo replay 20261021T093105Z-12345-1 --archive raw/

replay runs the original requests against the archived responses, never the network, with the options the run had.

Repairs

Run the nightly workflow by hand with repair: plausible/dplyr 2026-06-22..2026-10-16 to repair a range:

What the API still has is re-fetched: events, the star history, CRAN, Plausible, and traffic and PyPI while they’re within the APIs’ 14 and about 180 days.
What it doesn’t is replayed from the archived runs of the range.
Snapshots are never re-fetched for a past range: that would store today’s values on past dates.

The repair’s changes are reported as diff.repaired, and committed as “Repair plausible/dplyr 2026-06-22..2026-10-16” with a Velocirepo-Repair: trailer. A commit you make by hand with that trailer is treated the same way.

When the nightly run doesn’t run

A workflow that doesn’t run can’t report it. The heartbeat workflow checks three hours after the nightly run that it started, and comments on the tracking issue, or opens it, when it didn’t. GitHub disables the schedules of repositories without activity for 60 days.

For a heartbeat outside GitHub, set a Healthchecks ping URL as the HEALTHCHECKS_URL secret: the gate pings it when it starts and ends, and Healthchecks alerts on a missing ping.

The weekly digest

Every week, a sparkline per source and project over the last 8 weeks, with the week’s findings, is posted on an issue labelled velocirepo-digest. It’s for the problems no check was written for: three months of zero events stand out in the first week.