Exporting Data

Export to Parquet, CSV or DuckDB

Export the data as plain tables for use in other tools:

velocirepo export ./out/

This writes one Parquet file per table:

  out/projects.parquet (13 KB)
  out/targets.parquet (12 KB)
  out/metrics.parquet (9.3 MB)
  out/metrics_filled.parquet (35.0 MB)
  out/indicators.parquet (20.0 MB)
  out/events.parquet (6.8 MB)
  out/content.parquet (87.7 MB)
  out/meta.parquet (324 B)

Use --format csv for CSV files, or --format duckdb for one standalone DuckDB database, out/velocirepo.duckdb. Use --source or --project to filter:

velocirepo export ./out/ --format csv
velocirepo export ./out/ --format duckdb
velocirepo export ./out/ --source github
velocirepo export ./out/ --project quarto

Add --data-dict to also write data-dict.yaml, a data dictionary of the database; it implies --format duckdb. See Data Dictionary.

The tables

Every format has the same tables and columns:

Table One row per
projects project in velocirepo.toml
targets project, source and target in velocirepo.toml; canonical is false for a copy of a target another project keeps
metrics project, source, target, metric and date (and page, video, post or issue)
metrics_filled the same, with every total_* series carried forward to every day
indicators daily metric, date and indicator
events event
content issue, pull request, repository, video, post or feed item
meta export: as_of, the latest day of the metrics, and built_at

The tables are plain data:

  • The source-specific fields that sources declare are columns of their own: page, video_id, post_id and ref in the metrics tables, author and state in content. A column is there when the exported rows have it. All the source-specific fields are also in extra.
  • extra is JSON text, not a JSON-typed column, so DuckDB reads the export without its json extension; tags is a list of strings.
  • Project fields that the config leaves out are NULL.

Using the data with other tools

velocirepo provides three ways to access your metrics data from external tools:

  1. Live DuckDB file: velocirepo/data/velocirepo.duckdb, with views over the JSONL files and the projects and targets tables. It’s rebuilt after every fetch and project command, and with velocirepo build-db. Every query reads the JSONL files again, and the views hold the absolute path of the data directory, so it only works on the machine that built it.
  2. Standalone DuckDB file: velocirepo export ./out/ --format duckdb writes the tables as plain data. Copy it anywhere.
  3. Parquet export: velocirepo export ./out/ writes Parquet (or CSV) files that any data tool can read.

Tool compatibility

Tool / Package DuckDB file Parquet
DuckDB CLI Yes Yes
Python duckdb Yes Yes
Polars (Python/Rust) No Yes
pandas No Yes
Marimo Yes (via duckdb) Yes
R duckdb / DBI Yes Yes
R arrow No Yes
Observable / Evidence Yes Yes
Excel / Google Sheets No Yes (via CSV)

Examples

DuckDB CLI:

duckdb out/velocirepo.duckdb "SELECT project, target, arg_max(value, date) AS stars FROM metrics WHERE metric = 'total_stars' GROUP BY ALL ORDER BY stars DESC"

Python (duckdb):

import duckdb

con = duckdb.connect("out/velocirepo.duckdb", read_only=True)
df = con.sql("SELECT * FROM metrics WHERE source = 'pypi' ORDER BY date DESC LIMIT 10").df()

Polars (from Parquet export):

import polars as pl

metrics = pl.read_parquet("out/metrics.parquet")
stars = (
    metrics.filter(pl.col("metric") == "total_stars")
    .sort("date")
    .group_by("project", "target")
    .agg(pl.col("value").last())
)

Next steps