Displays and results

When delivering experiences such as in a chatbot app, it’s strongly recommended to give your users:

A visual indication when a tool is requested by the model
A choice to approve or deny that request
A clear display of tool results

In tool calling, we saw how .chat() automatically handles 1 and 3 for you, providing a nice developer experience out of the box. However, when streaming in something like a chatbot app, you’ll need to do a bit more work to provide these features.

Content objects

To display tool calls when streaming, first set the content parameter to "all". This way, when a tool call occurs, the stream will include ContentToolRequest and ContentToolResult objects, with information about the tool call. These classes have smart defaults for methods such as _repr_markdown_() and _repr_html_(). As a result, they will render sensibly in Jupyter notebooks and other environments that support rich content displays. They also have methods for specific web frameworks like Shiny, giving you output more tailored for the framework you’re using.

For a quick example, here’s a Shiny chatbot that displays tool calls in a user-friendly way.

client.py

import requests
from chatlas import ChatAnthropic

chat_client = ChatAnthropic()

def get_current_weather(lat: float, lng: float):
    """Get the current temperature given a latitude and longitude."""

    lat_lng = f"latitude={lat}&longitude={lng}"
    url = f"https://api.open-meteo.com/v1/forecast?{lat_lng}&current=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m"
    res = requests.get(url)
    return res.json()["current"]

chat_client.register_tool(get_current_weather)

app.py

from client import chat_client
from shiny.express import ui

chat = ui.Chat(id="chat")
chat.ui(messages=["Hello! How can I help you today?"])

@chat.on_user_submit
async def _(user_input: str):
    response = await chat_client.stream_async(
      user_input,
      content="all"
    )
    await chat.append_message_stream(response)

Custom displays

To customize how a tool result is actually rendered, you can leverage the fact that the tool can return a ContentToolResult instance instead of a simple value. By subclassing this class and overriding it’s default methods, you can create custom, rich, interactive displays for your tool results in various contexts. Here’s an extension of the previous example to displays the weather result on an interactive map using ipywidgets and ipyleaflet.

Show code

from chatlas import ContentToolResult
import ipywidgets
from ipyleaflet import Map, CircleMarker
from shinywidgets import register_widget, output_widget

class WeatherToolResult(ContentToolResult):
    def tagify(self):
        if self.error:
            return super().tagify()

        args = self.arguments
        loc = (args["latitude"], args["longitude"])
        info = (
            f"<h6>Current weather</h6>"
            f"Temperature: {self.value['temperature_2m']}°C<br>"
            f"Wind: {self.value['wind_speed_10m']} m/s<br>"
            f"Time: {self.value['time']}"
        )

        m = Map(center=loc, zoom=10)
        m.add_layer(
            CircleMarker(location=loc, popup=ipywidgets.HTML(info))
        )

        register_widget(self.id, m)
        return output_widget(self.id)

def get_current_weather(lat: float, lng: float):
    """Get the current temperature given a latitude and longitude."""

    lat_lng = f"latitude={lat}&longitude={lng}"
    url = f"https://api.open-meteo.com/v1/forecast?{lat_lng}&current=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m"
    response = requests.get(url)
    json = response.json()
    return WeatherToolResult(value=json["current"])

Screenshot of a tool result as an interactive map

Custom model results

By default, tool results are formatted as a JSON string, which is suitable for most use cases. However, that might not be ideal for all scenarios, especially if your tool does something sophisticated like return an image for the model to consume. In such cases, you can use the ContentToolResult class to return the result in a different format. For example, if you want to pass the return value of the tool function directly to the model without any formatting, set the model_format parameter to "as_is":

import base64
import requests

import chatlas as ctl

def get_picture():
    "Returns an image"
    url = "https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png"
    bytez = requests.get(url).content
    res = [
        {
            "type": "image",
            "source": {
                "type": "base64",
                "media_type": "image/png",
                "data": base64.b64encode(bytez).decode("utf-8"),
            },
        }
    ]
    return ctl.ContentToolResult(value=res, model_format="as_is")

chat = ctl.ChatAnthropic()
chat.register_tool(get_picture)

res = chat.chat(
    "You have a tool called 'get_picture' available to you. "
    "When called, it returns an image. Tell me what you see in the image.",
    echo="text"
)

The image shows four translucent colored dice arranged together. There’s a red die in the foreground, a blue die in the upper
left, a green die in the upper right, and a yellow die at the bottom. All dice appear to be standard six-sided dice with white dots (pips) representing the numbers 1 through 6. The dice have a glossy, semi-transparent appearance that gives them a
colorful, vibrant look against the white background. The image has a shallow depth of field, creating a slight blur effect on
the dice that aren’t in the foreground, which emphasizes the red die in the center.