Provider
Provider(name, model)A model provider interface for a Chat.
This abstract class defines the interface a model provider must implement in order to be used with a Chat instance. The provider is responsible for performing the actual chat completion, and for handling the streaming of the completion results.
Note that this class is exposed for developers who wish to implement their own provider. In general, you should not need to interact with this class directly.
Attributes
| Name | Description |
|---|---|
| model | Get the model used by the provider |
| name | Get the name of the provider |
| supported_builtin_tools | The provider-agnostic built-in tool classes (e.g. ToolWebSearch) that this |
| unsupported_builtin_tool_hint | Provider-specific context added to the error raised by |
Methods
| Name | Description |
|---|---|
| batch_poll | Poll the status of a submitted batch. |
| batch_result_turn | Convert a batch result to a Turn. |
| batch_retrieve | Retrieve results from a completed batch. |
| batch_status | Get the status of a batch. |
| batch_submit | Submit a batch of conversations for processing. |
| check_builtin_tool_support | Raise if this provider can’t translate tool into its API’s format. |
| close | Release resources held by this provider (e.g., HTTP connection pools, |
| close_async | Asynchronous variant of close(). |
| has_batch_support | Returns whether this provider supports batch processing. |
| list_models | List all available models for the provider. |
| set_model | Set the model used by the provider. |
| stream_content | Content to yield for chunk. |
| value_cost | Compute the cost for a completion. |
batch_poll
Provider.batch_poll(batch)Poll the status of a submitted batch.
Args: batch: Batch information returned from batch_submit
Returns: Updated batch information
batch_result_turn
Provider.batch_result_turn(result, has_data_model=False)Convert a batch result to a Turn.
Args: result: Individual BatchResult from batch_retrieve has_data_model: Whether the request used a structured data model
Returns: Turn object or None if the result was an error
batch_retrieve
Provider.batch_retrieve(batch)Retrieve results from a completed batch.
Args: batch: Batch information
Returns: List of BatchResult objects, one for each request in the batch
batch_status
Provider.batch_status(batch)Get the status of a batch.
Args: batch: Batch information
Returns: BatchStatus with processing status information
batch_submit
Provider.batch_submit(conversations, data_model=None)Submit a batch of conversations for processing.
Args: conversations: List of conversation histories (each is a list of Turns) data_model: Optional structured data model for responses
Returns: BatchInfo containing batch job information
check_builtin_tool_support
Provider.check_builtin_tool_support(tool)Raise if this provider can’t translate tool into its API’s format.
Called when a built-in tool is registered with a Chat, so that unsupported tools fail immediately rather than being silently dropped (or rejected by a cryptic API error) on the first request.
close
Provider.close()Release resources held by this provider (e.g., HTTP connection pools, database sessions).
This is best-effort and only closes synchronous resources; use close_async() to also release asynchronous resources. Providers only close resources they created themselves – clients or connections supplied by the caller are left open. Implementations must be safe to call multiple times.
The default implementation does nothing.
close_async
Provider.close_async()Asynchronous variant of close().
Releases both synchronous and asynchronous resources held by this provider. The default implementation calls close().
has_batch_support
Provider.has_batch_support()Returns whether this provider supports batch processing. Override this method to return True for providers that implement batch methods.
list_models
Provider.list_models()List all available models for the provider.
set_model
Provider.set_model(value)Set the model used by the provider.
The default implementation just updates the model name and returns the provider unchanged. Providers that dispatch to a different API flavor based on the model name (e.g., Posit AI) override this to return a different provider instance when the new model belongs to another flavor.
Returns
| Name | Type | Description |
|---|---|---|
| Provider | The provider to use for subsequent requests (possibly self). |
stream_content
Provider.stream_content(chunk, completion, turns=())Content to yield for chunk.
completion is the result of merging every chunk up to and including chunk (i.e. stream_merge_chunks runs first). Providers needing cross-chunk state read it from there rather than storing it on self: a single provider instance is shared across forked chats (Chat.__deepcopy__ keeps provider by reference), so several streams can be in flight at once.
turns is the full request history (prior turns plus the newest user turn) that produced this stream, for providers that need to resolve content referencing something outside completion itself (e.g. Anthropic’s document_index citations, which can reference a document from an earlier turn). Defaults to () for callers that don’t have it.
value_cost
Provider.value_cost(completion, tokens=None)Compute the cost for a completion.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| completion | ChatCompletionT |
The completion object from the provider. | required |
| tokens | tuple[int, int, int] | None | Optional pre-computed tokens tuple. If not provided, will be extracted from the completion. | None |
Returns
| Name | Type | Description |
|---|---|---|
| float | None | The cost in USD, or None if cost cannot be computed. |