Create a data source

A data source is the set of tables available to a commons() agent.

Usage

data_source(..., tables = NULL, exclude = NULL, dictionary = NULL)

Arguments

...

A single DBI connection, a single pins board, or named data frames to register as tables. When passing data frames, each name becomes a table name the agent can query.

tables

Which tables to expose, used when a connection or a board is supplied.

For a connection, a character vector of table names, schema-qualified strings like "schema.table", or DBI::Id objects. Defaults to every table returned by DBI::dbListTables(). Strings containing dots are interpreted as schema-qualified names; use DBI::Id(table = "a.b") for literal table names containing dots. For Snowflake and Databricks connections, a DBI::Id ending in catalog or schema selects every table and view in that namespace. Leaving tables unset selects the current schema. A Databricks hive_metastore selection must include a schema. Snowflake selections import semantic views, and Databricks selections import metric views, as native trusted metrics and dimensions. Namespace selections retain lightweight model metadata, then read a model's definition when the agent describes or uses it. Explicitly selected models are read and validated when the data source is created. Snowflake semantic variables and Databricks metric-view parameters are passed as typed JSON arguments to call_metrics. Databricks wildcard members require concrete column metadata from the warehouse. Native semantic models are available through search_pool, describe_table, and call_metrics, but are not registered as physical tables. Snowflake verified queries are exposed separately as exact trusted calculations through search_pool and call_calculation; their SQL is executed as stored rather than parsed to infer dependencies. An exact physical-table selection also imports associated models when every physical dependency is selected. Public relationships, facts, filters, and instructions become table-scoped first-touch and retrieval context; private members remain hidden.

For a board, a named character vector of pins to read: the names become table names, and the values are pin names passed to pins::pin_read().

exclude

For Snowflake and Databricks namespace selections, optional unqualified object-name globs to omit, such as "TMP_*".

dictionary

An optional path to a data dictionary describing the source's tables and columns, in the data-dict.yaml format. See the Data dictionaries section.

Value

A commons_data_source object.

Details

data_source() accepts data in several forms, picked by the class of what you pass:

• A DBI connection is queried as-is. Nothing is copied; the agent queries the database directly.

• Named data frames are loaded into an in-process DuckDB database. Use this when the data isn't already in a database.

• A pins board, e.g. pins::board_connect(), is read into the same in-process database: each pin in tables becomes a table. Pin names are validated against the board at construction (a single listing call), but each pin is downloaded only when its table is first used—by the describe_table tool, a SQL query that references it, or a measure that takes the source's connection. commons_server() starts a background process right after startup that downloads the remaining pins into the local pins cache, so a first use typically only reads an already-downloaded file. A table reflects the pin's value at first use and is not refreshed for the lifetime of the data source; if a pin can't be read (e.g. a network failure), the error surfaces at that first use and the read is retried on the next one.

The resulting object gives the agent a DBI connection plus a table registry.

Data dictionaries

A data dictionary describes a data source's tables and columns: what each table's rows represent, what its columns mean, allowed values and units, how tables join, and definitions of domain terms. Its content reaches the agent three ways:

• The dataset-level description and details, along with the glossary, are included in the system prompt. These fields are the place for rules that span tables and for guidance on which tables answer which kinds of questions.

• The first time a conversation touches a table—via the describe_table tool or a SQL query—the table's full dictionary entry rides along with the tool result: its prose, documented columns, relationships, and definitions of glossary terms it references. describe_table merges documented columns with the table's live schema.

• For Snowflake and Databricks sources, a fully qualified dictionary table name matches the same selected relation. A relative name is accepted when it matches only one selected relation. Authored prose takes precedence, while warehouse column types remain authoritative.

• When the agent also has a context_layer(), the dictionary's prose is indexed for the search_context tool.

A table's entry can also declare definitions: named expressions in the data-dict expression language. Commons validates their inferred types and references, compiles them for the source's SQL backend, and lets the model apply them as {{name}} tokens in run_sql or through call_metrics(). Definitions are delivered through all three channels above.

Trust

The run_sql tool runs only read-only SELECT queries; statements that would modify data or schema (INSERT, UPDATE, DROP, and similar) are rejected before reaching the database. For the in-process DuckDB built from data frames, commons additionally disables extension loading and filesystem access. These are safeguards, not a sandbox: when you supply your own connection, still open it in read-only mode where the backend supports it. Snowflake and Databricks sources snapshot the principal, active role, and namespace at creation, then reject catalog and governed execution after those values change. Authored and native semantic material is exposed only after a zero-row query succeeds for the current principal.

Examples

src <- data_source(
  sales = data.frame(id = 1:2, revenue = c(100, 200))
)