----------------------------------------------------------------------
This is the API documentation for the commons library.
----------------------------------------------------------------------


## Main functions

The main agent constructor, plus the building blocks for a commons agent.


Commons(client: 'Chat', data_sources: 'DataSource | Mapping[str, DataSource]', semantic_layer: 'SemanticLayer | None' = None, context_layer: 'ContextLayer | None' = None, *, instructions: 'str | None' = None) -> 'None'

A trustworthy agent that answers questions about its data.

Given a `chatlas.Chat` for the provider and model, the data sources it
can query, and optionally a semantic layer of trusted calculations and a
context layer of prose, a `Commons` agent will allow for agent interactions
with answers classified by how they were produced.

A `Commons` agent inherits directly from `chatlas.Chat` and relies on the
chatlas infrastructure to set up the LLM provider and model. `Commons`
initializes its own chat state and system prompt to ensure provenance
and citation tracking. Passing a custom system prompt in the `Commons`
constructor is ignored with a warning; use `instructions` to add to
commons' prompt instead. For best results, enable thinking where the
provider and model support it.

`chat()` and `stream_async()` are the currently supported ways to
interact with a `Commons` agent. The other entry points chatlas offers
(`chat_async()`, `stream()`, `chat_structured()`, etc.)
are disabled and raise `NotImplementedError`s because they are not (yet)
tied in to the commons framework. The rest of chatlas's surface works as
it does on any chat.

`data_sources` is a `DataSource`, or a mapping of name to `DataSource`;
a measure can take a named source's connection as an argument named
after it. `instructions` is extra text placed under an
`## Additional instructions` heading at the end of commons' built-in
system prompt, as a string or the path to a text or Markdown file.

Construction raises a TypeError if `client` is not a `chatlas.Chat`, if
an entry of `data_sources` is not a `DataSource`, or if a layer is not
the layer its argument claims; a ValueError if `data_sources` names no
source or a measure asks for an injection no named source can fill; and
a FileNotFoundError if `instructions` names a file that does not exist.

data_source(*args: 'Any', tables: 'Any' = None, exclude: 'Any' = None, dictionary: 'Any' = None, **frames: 'Any') -> 'DataSource'

Create a data source from an engine, a pins board, or named frames.

A thin dispatcher over the constructors, which are the documented way in.

`tables`, `exclude`, and `dictionary` are keyword-only options, so all
three names are reserved in every form: a frame passed under one of them
is rejected with a TypeError naming it, never silently consumed.
`tables` selects tables of the engine and board forms, `exclude` drops
relations from a warehouse catalog listing by glob, and `dictionary`
attaches a data dictionary to any form. To use one as a frame name, call
`DataSource.from_frames()` directly.

A dictionary's governed definitions are compiled for the source's
dialect during construction, so construction raises if the dialect has
no emitter (DuckDB, Snowflake, and Databricks have one), if a definition
sits on a table the source does not expose, or if a metric mixes row and
aggregate grain. On a warehouse the authored column spellings are bound
to the names the catalog reported before anything is lowered, so it
raises there only if a table declaring definitions matched no exposed
relation, or if a definition names an authored column the selected
relation does not have.

semantic_layer(*items: 'Any') -> 'SemanticLayer'

Collect measures into a semantic layer.

Each item is a measure, a list (or tuple) of measures, a module, or a
path to a Python file or a directory of them. Directory searches are
not recursive.

A sibling file is imported by plain absolute import; its directory is
on sys.path only while the file loads. Even a single requested file
puts its whole directory on sys.path, so every .py file beside it --
and every subdirectory, which is importable as a package -- is checked:
a name that collides with the standard library, an installed package,
or a module already imported from elsewhere is a construction error. A
sibling imported this way stays in sys.modules under its bare name for
the rest of the process, so two directories that each define a
same-named helper cannot both be loaded in one process.

Collecting the same measure twice -- the same file passed alongside
its own directory, say -- is not an error; two different measures
sharing one name is. The R implementation is stricter here: it rejects
a repeated name even when it is the same measure twice.

measure(*, description: 'str | None' = None, name: 'str | None' = None, title: 'str | None' = None, provenance: 'Sequence[str]' = ()) -> 'Callable[[Callable[..., Any]], Callable[..., Any]]'

Mark a function as a measure.

The decorated function is returned unchanged, so measures and the helpers
they call stay ordinary callables. Model-supplied arguments are expected
to be scalars, enums, or arrays of those; richer shapes are not rejected,
but the schema block renders them only approximately.

``provenance`` records links back to wherever the measure's definition
came from. The R implementation attaches provenance through a roxygen
tag instead, and only to measures sourced from a file.

context_layer(files: 'Iterable[str | os.PathLike[str]]' = ()) -> 'ContextLayer'

Create a context layer from text or Markdown files.

``files`` must be a collection of paths; a bare string or path raises
``TypeError``. Files are read eagerly and decoded as UTF-8, so a missing
path (``FileNotFoundError``), a directory (``IsADirectoryError``), or a
file in another encoding (``UnicodeDecodeError``) fails here rather than
mid-conversation.


## Commons Methods

Methods for the Commons class


__repr__(self) -> 'str'

__deepcopy__(self, memo: 'dict[int, Any]') -> 'NoReturn'

chat(self, *args: 'Content | str', echo: 'EchoOptions' = 'output', stream: 'bool' = True, kwargs: 'SubmitInputArgsT | None' = None) -> 'ChatResponse'

Ask a question and wait for the whole answer.

A reminder queued with `queue_restore_reminder()` rides this turn,
and a turn that fails leaves it queued for the next one.

stream_async(self, *args: 'Content | str', content: "Literal['text', 'all']" = 'text', echo: 'EchoOptions' = 'none', data_model: 'type[BaseModel] | None' = None, kwargs: 'SubmitInputArgsT | None' = None, controller: 'StreamController | None' = None) -> 'AsyncGenerator[Any, None]'

Ask a question and stream the answer as it arrives.

The signature is identical to chatlas's, so a chat UI can drive this agent
directly and needs the attachment content, the mode, and the controller its
stop button cancels through.

The Commons agent does not accept `data_model`. If you pass it, this
method raises NotImplementedError. In chatlas, using `data_model` means
the chunks are JSON that the caller parses as one document. Commons adds
provenance markers and citations to the stream that are not compatible with
`data_model`, so it is explicitly forbidden.

chat_async(self, *args: 'Any', **kwargs: 'Any') -> 'NoReturn'

stream(self, *args: 'Any', **kwargs: 'Any') -> 'NoReturn'

chat_structured(self, *args: 'Any', **kwargs: 'Any') -> 'NoReturn'

chat_structured_async(self, *args: 'Any', **kwargs: 'Any') -> 'NoReturn'

extract_data(self, *args: 'Any', **kwargs: 'Any') -> 'NoReturn'

extract_data_async(self, *args: 'Any', **kwargs: 'Any') -> 'NoReturn'

to_solver(self, *args: 'Any', **kwargs: 'Any') -> 'NoReturn'

citation_corpus(self) -> 'list[CorpusEntry]'

The trusted text this agent's citations are verified against.

prewarm(self) -> 'None'

Build the caches the first question would otherwise pay for.

Failures propagate: a direct call is typically warming caches ahead
of a deployment, so a cold cache should fail the deploy.

add_turn(self, turn: 'Turn') -> 'None'

Add a turn, restarting the citation request if a person spoke.

A user turn of nothing but tool results is the same question still
running, and an assistant turn is nobody asking anything.

set_turns(self, turns: 'Sequence[Turn]') -> 'None'

Replace the conversation, dropping any reminder queued for it.

queue_restore_reminder(self) -> 'None'

Tell the next turn that the session behind its history is gone.


## Supporting types

The objects the constructors return, and the values they carry.


DataSource(backend: 'Backend', tables: 'list[str]', table_ids: 'dict[str, TableId]' = <factory>, pending: '_PendingPins | None' = None, dictionary: 'DataDictionary | None' = None, relations: 'dict[str, Relation] | None' = None, manifest: 'Manifest | None' = None, session: 'SessionSnapshot | None' = None, definition_bindings: 'dict[str, Any] | None' = None) -> None

Tables an agent can query, and the dictionary that describes them.

SemanticLayer(measures: 'Mapping[str, Measure]', source_text: 'Mapping[str, str]') -> None

The trusted calculations an agent can run.

``source_text`` holds the source of the measures and the module-level
helpers they call, keyed by Python function name. Only text is kept:
the agent's worker session reads measure definitions but never receives
a callable. Two functions that share a Python name share one entry, and
the first definition collected wins, so a measure whose function shares
its name with an earlier one is shown that earlier function's source
instead. The R implementation sources a path's files into one shared
environment, so there the last definition of a same-named helper wins
instead, and is what every measure calling it actually runs.

The layout of this object is internal and may change without notice.

ContextLayer(docs: 'Iterable[str]' = ()) -> 'None'

Text that helps an agent interpret its data source.

Construct one with :func:`context_layer`. Internals are private and may
change without notice.

Measure(name: 'str', title: 'str', description: 'str', func: 'Callable[..., Any]', params: 'type[BaseModel]', injected: 'tuple[str, ...]' = (), provenance: 'tuple[str, ...]' = ()) -> None

A trusted calculation the agent can run.

``params`` describes only the arguments the model supplies; ``injected``
names the arguments commons supplies, which the model never sees.

Annotated(*args, **kwargs)

Runtime representation of an annotated type.

At its core 'Annotated[t, dec1, dec2, ...]' is an alias for the type 't'
with extra annotations. The alias behaves like a normal typing alias.
Instantiating is the same as instantiating the underlying type; binding
it to types is also the same.

The metadata itself is stored in a '__metadata__' attribute as a tuple.

Tag(*values)

How an answer was produced.

A and B are set by tools on their results. C is only ever derived: it is
what a B becomes when its citation does not verify.

StrEnum rather than a plain `str, Enum` mixin, which formats as "Tag.A".
The tag is written to the commons.provenance.tag span attribute, where R
writes the bare letter and a mismatch would corrupt traces silently.

list_tables(source: 'DataSource') -> 'list[str]'

The table names an agent can query on `source`.


## DataSource Methods

Methods for the DataSource class


from_frames(**frames: 'Any') -> 'DataSource'

Load named data frames into a locked-down in-process DuckDB.

from_engine(engine: 'sqlalchemy.Engine', tables: 'Any' = None, exclude: 'list[str] | None' = None, dictionary: 'DataDictionary | None' = None) -> 'DataSource'

Query a caller's database directly. Nothing is copied.

A Snowflake or Databricks engine imports its catalog: the selection is
resolved against the warehouse, access to it is verified for the
current principal, and what it reports is folded into `dictionary`.

With `tables` unset on any other backend, its own listing is taken as
given: it reports what exists, so there is nothing to check and no
round trip worth paying for.

`exclude` takes unqualified object-name globs to drop from a
warehouse catalog listing, such as `"TMP_*"`. Only a warehouse has a
listing to drop from, so any other engine refuses it.

On a warehouse `dictionary` is taken here because the catalog listing
is folded into it during construction; on any other engine it is
simply attached. Its governed definitions are lowered once the
dialect and the final table set are known, at the end of
construction.

from_board(board: 'Any', tables: 'Any', dictionary: 'DataDictionary | None' = None) -> 'DataSource'

Expose a pins board's pins as tables, each read on first use.

`dictionary` is taken here so that the argument survives the
dispatcher; a board has no catalog listing to fold into it. Its
governed definitions are lowered at the end of construction, once
the dialect and the final table set are known.

query(self, sql: 'str') -> 'list[dict[str, Any]]'

Run one read-only statement, rejecting anything else first.

On a warehouse source the connection identity is checked before the
statement is read: access to these tables was decided for one
principal, role, and namespace, so a query raises rather than runs
once any of those has moved.

ensure_loaded(self) -> 'None'

Read every pin this source has not read yet.

A board source loads a pin when a query names it, and that recovery
lives on `query()`. A measure is handed the connection itself and
never goes through `query()`, so nothing there would trigger the
read and the measure would fail on a relation that does not exist
yet. `source_ensure_all()` in `pkg-r/R/data-source.R` is the same
step for the same reason. A source with nothing pending, which is
every source that is not board-backed, does nothing.

dialect(self) -> 'str'

A hint for the system prompt, not a contract.


## Shiny UI and server

Put a commons agent inside of an interactive chat app. Needs the `shiny` extra.


app(client: 'Commons', *, toolbar: 'bool' = True, **kwargs: 'Any') -> 'shiny.App'

Build a complete app around a commons agent.

Every session the app serves shares the one agent passed here via ``client``:
a second visitor joins the first one's conversation, and two questions
answered at once interleave the agent's citation and provenance state, so
neither answer can be trusted. To give each session its own agent, assemble
the page and the server yourself with `commons.ui.theme()` and
`commons.ui.server()`, building the agent inside the server function.

Parameters
----------
client
    A commons agent.
toolbar
    Whether to show the development toolbar, a dark-mode switch.
**kwargs
    Passed to `shiny.App()`.

server(id: 'str', client: 'Commons', **kwargs: 'Any') -> 'shinychat.Chat'

Wire a commons agent to the chat element `id` on the page.

Pair this with a page built on `commons.ui.theme()`, so the chat assets
the answers reference are served. In a deployed app, build the agent
inside the server function and pass it here, so each session gets its
own agent state.

A `client` that is not a commons agent raises `TypeError`: a plain
chatlas chat has none of the citation or provenance handling the chat
surface renders.

Parameters
----------
id
    The id of the chat element on the page.
client
    A commons agent.
**kwargs
    Passed to `shinychat.Chat()`.

theme(preset: 'str | None' = 'shiny', **variables: 'str | float | bool | None') -> 'Theme'

Build the theme a commons chat page uses.

Layers the commons chat variables over `shinychat.page_chat_theme()` and
attaches the dependency serving the chat script, stylesheet and icons.

Parameters
----------
preset
    A Shiny or Bootswatch preset name.
**variables
    Sass-variable overrides, in either `snake_case` or `kebab-case`.
    These win over the commons defaults.

commons_chat_dependency() -> 'HTMLDependency'

The dependency serving the chat script, stylesheet and icons.

asset_base_url() -> 'str'

The URL the assets are served under on a page.


----------------------------------------------------------------------
This is the User Guide documentation for the package.
----------------------------------------------------------------------

### Introduction to commons

This page explains the structure of a commons project and how to start building a commons agent in Python.

```python
import commons
```

## Design philosophy

AI agents for data analysis range from cautious and narrowly correct to wildly untrustworthy. `commons` increases the chance of a correct answer by giving the agent access to trusted code that you already have, while the agent keeps enough freedom to answer new, realistic questions. `commons` also derives a provenance outcome from the path the agent took, so a user can decide how much to trust each answer.

If you are a data analyst, data scientist, or other data practitioner, you probably have a collection of trusted code. This is the code you depend on for your analyses, apps, and reports. The core idea of `commons` is that an agent gives more correct answers when it can run this code and read its documentation.

The high-trust path occurs when a question matches one of these **trusted calculations**. For example, a `commons` agent can analyze biodiversity data, and a user asks:

> How many total animals were observed at Oak Bluff?

The agent searches for a trusted calculation that answers the question. If it finds one, it runs that calculation and reports the result with the green check-shield provenance marker for the `Verified answer` outcome.

> At Oak Bluff, 59 individual animals were observed across 5 species, based on 28 hours of survey effort. Note this reflects observed individuals during surveys, not necessarily a full census of every animal present at the site. <img class="commons-documentation-marker" src="assets/figs/trusted-icon.svg" alt="Verified answer">

Although the agent had to decide *which* trusted calculation to run. It did not have to decide *what code to write*, reducing degrees of freedom and allowing the agent to use code that you already vetted.

However, we also expect users to ask questions that stray from the “happy path.” For those, the agent searches for additional context and writes custom code (currently SQL, but soon custom Python code as well). Answers from this path either include a blue quote-mark citation marker if a citation was found in the added context, or display the yellow exclamation provenance marker for an `Untrusted` outcome.

## Trust flow

A `commons` agent uses trusted calculations when it can. When the user asks a question, the agent first searches the semantic layer for a trusted calculation. If it finds one, it calls that calculation, and the answer shows the green check-shield provenance marker for the `Verified answer` outcome.

If a relevant trusted calculation is not found, the agent proceeds down the lower-trust path. It searches through the context for additional information, then uses that information to write custom SQL (or soon Python) code to answer the user’s question. These answers either include blue quote-mark citation markers that open details about verified sources or display the yellow exclamation provenance marker for an `Untrusted` outcome.

The lower-trust path has two possible provenance outcomes. When the agent writes custom SQL or Python, it can also include supporting text quoted from a trusted source. If `commons` verifies that the quoted text appears in that source, the provenance outcome is `Cited` and the answer displays blue quote-mark citation markers that open the source details. If no citation verifies, the provenance outcome is `Untrusted` and the answer displays the yellow exclamation provenance marker

The table below lists how each provenance outcome can occur:

| How the answer is produced | Provenance outcome |
|---|---|
| A trusted Python [measure](#semantic-layer) | `Verified answer` <img class="commons-documentation-marker" src="assets/figs/trusted-icon.svg" alt=""> |
| A [data dictionary metric](#definitions), possibly grouped or filtered with definitions | `Verified answer` <img class="commons-documentation-marker" src="assets/figs/trusted-icon.svg" alt=""> |
| Custom SQL, including SQL that uses [data dictionary definitions](#definitions) | `Cited` <img class="commons-documentation-marker" src="assets/figs/citation-mark.svg" alt=""> or `Untrusted` <img class="commons-documentation-marker" src="assets/figs/warning-icon.svg" alt=""> |
| No data tool used (for example, the agent already had the information, or no accessible information answered the question) | No provenance outcome |

The agent does not decide the provenance outcome. `commons` derives it from the tools the agent called and the citations it made.

## Information layers

`commons` has two primary layers of information: the **semantic layer** and the **context layer**. The semantic layer holds trusted calculations, ideally lifted from reliable code that you already use. The context layer holds background information. The agent uses this information to decide what custom SQL to write and how to interpret results. Data dictionaries span the two layers.

| Layer | Sources | Role |
|---|---|---|
| Semantic layer | Measures in `.py` files and [`definitions`](#definitions) in `data-dict.yaml` | Provides trusted calculations. |
| Context layer | Markdown files and descriptive fields in [`data-dict.yaml`](#data-dictionaries) | Informs custom SQL and guides interpretation. |

### Semantic layer

The most direct way to add a trusted calculation is a **measure**. A measure is a Python function decorated with `@commons.measure`. The decorator needs a description, from its `description` argument or from the docstring of the function. When measures exist, the agent searches for a measure that matches the question. If it finds one, it calls that measure, with arguments if the measure takes any.

A measure can take two kinds of parameter in its signature. A parameter declared as `Annotated[T, Field(description=...)]` gets its value from the model, which reads the description to decide what value to pass. A parameter declared as `commons.Injected[T]` gets its value from `commons`. The model never sees this parameter: `commons` passes the connection of the data source that has the same name as the parameter. A parameter with no description and no `Injected` annotation will result in an error.

As an example, in this measure, `commons` passes `biodiversity` and the model passes `site`:

```python
@commons.measure(description="Species observed at one site.")
def species_at_site(
    biodiversity: commons.Injected[Any],  # commons passes the connection
    site: Annotated[str, Field(description="Site name.")],  # the model passes a value
) -> Any:
    ...
```

When a user asks "Which species were observed at Oak Bluff?", the model reads the description of `site` and passes `"Oak Bluff"`. `commons` passes the connection of the data source named `biodiversity`, and the measure runs with both.

`data-dict.yaml` files also add to the semantic layer through [`definitions`](#definitions).

### Context layer

The context layer holds unstructured text from Markdown files and the descriptive fields of `data-dict.yaml`. commons indexes this text and retrieves the parts that match a question. Facts that every conversation needs belong in `instructions`, not in the context layer.

### Examples

Here are short examples of a data dictionary, a measure file, and a context document for the biodiversity example:

#### Data dictionary

`dictionaries/biodiversity.yaml`

```yaml
tables:
  - name: observations
    description: Species observations by nature preserve.
    columns:
      - name: count
        description: Individuals observed during surveys.
```

#### Measure file

`measures/biodiversity.py`

```python
from typing import Annotated, Any

from pydantic import Field

import commons


@commons.measure(description="Species richness by site.")
def biodiversity_by_site(
    biodiversity: commons.Injected[Any],
    site: Annotated[str, Field(description="Site name.")],
) -> Any:
    return biodiversity.execute(
        "SELECT COUNT(DISTINCT species) AS species_richness "
        "FROM observations WHERE obs_site = ?",
        [site],
    ).fetchdf()
```

The `biodiversity` parameter receives the connection of the data source named `biodiversity`. The model supplies `site`.

#### Context document

`context/biodiversity.md`

```markdown
# Interpreting survey results

Observed individuals reflect organisms recorded during surveys. They are not a complete population census of a nature preserve.
```

## Data sources

One of the primary decisions you’ll need to make when building a `commons` agent is which data sources to grant the agent access to. Each data source combines the underlying data with the tables to expose to the agent. It can also include a data dictionary describing those tables and trusted calculations on them.

Data sources are created with `commons.data_source()`. The data can be named `pandas` or `polars` data frames, a [pins](https://rstudio.github.io/pins-python/) board, or a SQLAlchemy `Engine`. `commons` loads data frames and pins into an in-process DuckDB database. `commons` queries a database engine directly without copying data into a local database.

For example, the following code creates a data source from two data frames. Each name becomes a table that the agent can query. `dictionary` is an optional path to a `data-dict.yaml` file.

```python
biodiversity = commons.data_source(
    observations=observations,
    site_area=site_area,
    dictionary="dictionaries/biodiversity.yaml",
)
```

For an engine or a pins board, the `tables` argument selects the tables to expose. An engine takes a list of table names:

```python
import sqlalchemy

engine = sqlalchemy.create_engine("duckdb:///surveys.duckdb")
surveys = commons.data_source(engine, tables=["observations", "sites"])
```

With a pins board, `tables` maps each table name the agent sees to the pin that supplies the data:

```python
surveys = commons.data_source(
    board,
    tables={"observations": "survey-observations", "sites": "survey-sites"},
)
```

A Snowflake or Databricks engine also imports its catalog, so commons can resolve the selection against the warehouse and check access. `exclude` drops objects from the catalog listing by glob:

```python
warehouse = commons.data_source(engine, exclude=["TMP_*"])
```

If the catalog is too large for the system prompt, the agent gets a `search_catalog` tool instead of a table listing. The system prompt reports only the number of selected catalog objects and tells the agent to call `search_catalog` before `describe_table`. The agent then searches the catalog for objects that match the question and describes only those before it writes SQL.

### Data dictionaries

A data dictionary provides structured documentation for one data source. Use it to state what each table represents, the meaning and type of each column, relationships between tables, and glossary terms. It also holds trusted `definitions`. `commons` reads the [`data-dict.yaml` specification](https://data-dict.tidyverse.org/).

A `commons` agent uses a data dictionary in three ways:

- Dataset-level descriptions give broad context that is always available. Glossary terms go in the system prompt as space allows.
- When the agent first uses a documented table in a conversation, it receives the description, columns, relationships, and glossary terms of that table.
- Descriptive fields, such as `description` and `details`, are part of the context layer.

#### Definitions

**Definitions** are named, governed expressions attached to tables in `data-dict.yaml`. They let an agent reuse trusted metrics, filters, and derived values, so they add to the semantic layer.

Each definition is an expression in the [data-dict expression language](https://data-dict.tidyverse.org/expressions.html), not in the SQL dialect of your database:

```yaml
tables:
  - name: observations
    columns:
      - name: count
        type: number
    definitions:
      - name: total_individuals
        label: Total individuals observed
        description: Sum of the individuals recorded in surveys.
        expr: SUM(count)
```

There are three kinds of definition. A definition can take part in a trusted metric calculation, or the agent can use it in custom SQL:[^definition-sql]

| Kind | Example | Use in a trusted metric calculation |
|---|---|---|
| Metric | `SUM(n)` | Computes the metric |
| Filter | `status = 'active'` | Restricts rows or provides a grouping dimension |
| Derived value | `price * quantity` | Provides a grouping dimension |

`commons` infers the kind from the expression. An aggregate or constant expression is a metric. A row-level Boolean expression is a filter. Every other row-level expression is a derived value.

[^definition-sql]: In custom SQL, the agent writes a definition as a `{{name}}` token, and `commons` expands it to the SQL compiled for the data source. This is still custom SQL, so the provenance outcome is `Cited` or `Untrusted`, not `Verified answer`.

See the [DevRel Agent's `data-dict.yaml`](https://github.com/posit-dev/devrel-agent/blob/main/dictionaries/devrel.data-dict.yaml) for examples of definitions.

When a data source is constructed, `commons` validates each definition and compiles it to the source’s SQL dialect.

## Project directory organization

A `commons` agent is easiest to maintain when each piece has its own file:

```text
.
|-- app.py
|-- agent.py
|-- pyproject.toml
|-- AGENTS.md # or the file your coding agent reads (for example, CLAUDE.md)
|-- instructions.md
|-- dictionaries/
|   `-- biodiversity.yaml
|-- measures/
|   `-- biodiversity.py
`-- context/
    `-- context.md
```

## Constructing the agent

Use `commons.Commons()` to construct an agent. Pass it a `chatlas.Chat` and one or more data sources, plus any semantic and context layers. You can also optionally append information to the `commons` agent system prompt using the `instructions` argument.

```python
import chatlas
import commons

biodiversity = commons.data_source(
    observations=observations,
    site_area=site_area,
    dictionary="dictionaries/biodiversity.yaml",
)

agent = commons.Commons(
    client=chatlas.ChatAnthropic(model="claude-sonnet-5"),
    data_sources={"biodiversity": biodiversity},
    semantic_layer=commons.semantic_layer("measures"),
    context_layer=commons.context_layer(files=["context/context.md"]),
    instructions="instructions.md",
)

agent.chat("How many species were observed at Oak Bluff?")
```

`data_sources` is one `DataSource` or a mapping of name to `DataSource`. Name the sources when a measure takes an injected connection, because commons injects by name. `semantic_layer()` accepts measures, modules, or paths to `.py` files and directories. `context_layer()` accepts a list of file paths.

Underneath, a `Commons` agent inherits from a `chatlas.Chat` object, but builds its own system prompt and tools. As such, it follows the `Chat` API and provides the `chat()` and `stream_async()` methods as ways to ask a question. The other `chatlas` entry points, such as `chat_async()`, `stream()`, and `chat_structured()`, are not supported and will raise `NotImplementedError` at this moment.

### Chat UI

The above examples will build a terminal-based agent without any UI attached. If you want to
use the agent in an interactive web UI, the `commons.ui` module will put the agent behind a 
[Shiny](https://shiny.posit.co/py/) chat. This requires installing the `shiny` extra group.

When developing locally, you can use `commons.ui.app(agent)` to return a complete Shiny app:

```python
app = commons.ui.app(agent)
```

`commons.ui.app()` shares one agent across every session, so it suits one visitor at a time.
For a deployed app, you should instead build the page with `commons.ui.theme()` and construct
the agent inside the Shiny server function, so each session gets its own agent:

```python
import shinychat
from shiny import App

app_ui = shinychat.page_chat("Biodiversity", id="chat", theme=commons.ui.theme())


def app_server(input, output, session):
    # The same construction as above; building it per session keeps each
    # session's citation and provenance state separate.
    commons.ui.server(
        "chat",
        commons.Commons(
            client=chatlas.ChatAnthropic(model="claude-sonnet-5"),
            data_sources={"biodiversity": biodiversity},
            semantic_layer=commons.semantic_layer("measures"),
            context_layer=commons.context_layer(files=["context/context.md"]),
            instructions="instructions.md",
        ),
    )


app = App(app_ui, app_server)
```

## Example application

[`demo.py`](https://github.com/posit-dev/commons/blob/main/pkg-py/demo.py), in the Python package's directory, is a fuller worked example: an agent over made-up forest canopy data with a semantic layer of measures and a context layer. Run it with `uv run shiny run demo.py` for the chat UI, or `uv run python demo.py` to ask the same questions from the terminal. [`demo.ipynb`](https://github.com/posit-dev/commons/blob/main/pkg-py/demo.ipynb) is the same agent in a notebook, with cells for reading what the agent registered and adding a measure of your own.


### Security and governance for commons agents

`commons` allows users to build and deploy agents that can run SQL queries against live databases and (soon) execute arbitrary Python code. This raises several security and governance questions: How do you make sure that an agent cannot delete production data? Can model-generated Python code interfere with a [Posit Connect](https://posit.co/products/enterprise/connect) deployment? Can an agent show users tables, rows, or business context that they shouldn't be able to see?

This page explains the boundaries that `commons` provides and the responsibilities that remain with the application author and server administrator.

## The `commons` harness

A `commons` agent is a `chatlas.Chat` object that carries a system prompt and a set of tools. The system prompt lists the available data sources and business context. Tools let the model retrieve more context, call a trusted calculation, or run a SQL query.

We assume that a model might make any request allowed by its tools. Application security should therefore not depend on the model following an instruction like “never reveal sensitive data.” Instead, only give an agent access to data that the current user of the application is allowed to see. The system prompt, tool arguments, and tool results are also sent to the model provider. The agent must therefore not have access to data that you [do not trust that provider to process](https://posit.co/blog/trust-llm-tools).

## SQL code execution

### Destructive actions

The agent can run one read-only statement at a time. `commons` parses each statement with [sqlglot](https://github.com/tobymao/sqlglot) and checks its structure against an allowlist of statement forms that only read, such as `SELECT` and the set operations. `commons` refuses every other form, stacked statements, and statements that `sqlglot` cannot parse, so it fails safe/closed.

For data frames and pins, which `commons` loads into its own DuckDB database, it also hardens the connection. It disables community and unsigned extensions, automatic extension install and load, external access, and the local filesystem. Then it sets `lock_configuration`, so a query cannot turn those settings back on.

These checks provide defense in depth, but they are not a database sandbox. When you supply a database connection, `commons` queries it as-is, and a statement that the parser accepts runs with all the permissions of that connection. The primary safeguard against destructive SQL is therefore database-enforced read-only access, and you should open the connection as a read-only user or role.

### Data access

The `tables` argument to `data_source()` controls which tables `commons` describes to the model, but it is not an authorization boundary. SQL written by the agent can query any object that the connection can reach.

On Posit Connect, [viewer OAuth integrations](https://docs.posit.co/connect/admin/access-controls/) can give an interactive application the current viewer's Snowflake or Databricks credentials. If the application creates its engine from those credentials, the warehouse continues to enforce that viewer's existing access policies, including row-level and column-level security.

For a Snowflake or Databricks source, `commons` snapshots the connection’s principal and namespace when it creates a Snowflake or Databricks data source, and its active and secondary roles as well on Snowflake, and rejects subsequent operations if that identity changes.

Viewer credentials are not automatic. `commons` uses the engine that the application supplies. When using viewer credentials, you should create the engine and the `commons` agent inside the Shiny server function so that each session has the correct database identity.

When viewer credentials are not available, it is recommended to use a service account with access only to the data that the application needs. Every viewer then has the same database permissions, so share the application only with users who are allowed that access.

## Python code execution

The Python agent currently has no tool that runs code written by the model, but this feature is expected in the near future. Currently, there is no concern for Python `commons` agents executing arbitrary Python code. When the feature is added, it will generally follow the R package's [R code execution](https://posit-dev.github.io/commons/r/articles/governance.html#r-code-execution) design.

## Permissioning facts

The rows returned by SQL are not the only sensitive information available to an agent. The description and details of each data dictionary, and its glossary entries up to a size cap, go into the system prompt. The full dictionary entry for a table, with a summary of sampled rows, goes to the model when a table is first described or queried. Context documents and the rest of the dictionary prose are available to the model through search.

Because of this, you should only include facts that you can share with both the application's viewers and its model provider. If one audience must not see a fact, use separate applications with separate context, or put the facts behind viewer credentials. Asking the model to hide the fact from that audience is not a reliable means of preventing unauthorized access to the information.

## Telemetry

The Python implementation of `commons` currently does not support telemetry or conversation logging, though this feature is planned to be added in the near future. When it is added, it will generally follow the R package's [Logging trajectories](https://posit-dev.github.io/commons/r/articles/governance.html#logging-trajectories) design.


### Feature parity with the R package

While both Python and R packages are first-class implementations of `commons`, the R implementation was developed first, and the Python package does not yet implement everything the R package does. This page lists the gaps that remain as of the current version and what a Python agent does instead.

| R-only feature | What a Python agent does instead |
|---|---|
| Snowflake semantic views and Databricks metric views as trusted calculations, answered `Verified answer` | Imports the catalog of a Snowflake or Databricks engine, but answers these questions with custom SQL, so `Cited` or `Untrusted` |
| Custom R code from the agent, run in a sandboxed worker process | Writes custom SQL only, with Python execution coming soon |
| `gt` tables and `ggplot` figures returned by measures, rendered in the chat | Renders data-frame results as tables, with no current plot support |
| `trajectory_read()` and `trajectory_review()`, for reading and annotating logged conversations | No equivalent yet |
| The commons agent skill for coding agents | Ships with the R package only |

This page will shrink (and eventually disappear) as the Python package matures.