> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usefused.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Accept multimodal data

> Validate image, audio, video, file, text, and custom data across agents and tools.

Define a Pydantic contract once. Harnest applies it to agents, tools, JSON, SSE, WebSockets, SubAgents, and session messages.

## Choose a content type

| Content              | Type       | Configure with       |
| -------------------- | ---------- | -------------------- |
| Text                 | `Text`     | Pydantic field rules |
| Images               | `Image`    | `ImageConstraints`   |
| Audio                | `Audio`    | `AudioConstraints`   |
| Video                | `Video`    | `VideoConstraints`   |
| Files                | `File`     | `FileConstraints`    |
| Custom JSON          | `Data[T]`  | `DataConstraints`    |
| Generic stored media | `AssetRef` | Pydantic field rules |

Media constraints can limit MIME type, decoded byte size, dimensions, pixels, duration, pages, frame rate, sample rate, channels, animation, and archive expansion.

## Define the contract

Keep shared contracts in the root `models/` directory. Use `Annotated` to keep limits beside the field they protect.

```python models/vision.py theme={null}
from typing import Annotated

from pydantic import BaseModel

from harnest.content import Image, ImageConstraints


Screenshot = Annotated[
    Image,
    ImageConstraints(
        media_types=frozenset({"image/jpeg", "image/png"}),
        max_bytes=500 * 1024,
        max_width=1920,
        max_height=1080,
        max_pixels=2_073_600,
        animated=False,
    ),
]


class VisionInput(BaseModel):
    question: str
    screenshot: Screenshot


class VisionOutput(BaseModel):
    answer: str
```

Harnest inspects decoded bytes. Client-supplied metadata does not bypass the contract.

## Apply the contract

<Tabs>
  <Tab title="Agent input and output">
    ```python agent.py theme={null}
    from harnest.agent import Agent
    from harnest.model import LiteLLMModel
    from harnest.models.vision import VisionInput, VisionOutput


    root_agent = Agent(
        name="vision",
        model=LiteLLMModel("provider/vision-model"),
        input_schema=VisionInput,
        output_schema=VisionOutput,
    )
    ```
  </Tab>

  <Tab title="Server tool output">
    ```python tools/analyze.py theme={null}
    from harnest.models.vision import VisionOutput
    from harnest.tool import tool


    @tool(output_schema=VisionOutput)
    async def analyze() -> VisionOutput:
        return VisionOutput(answer="Ready")
    ```
  </Tab>

  <Tab title="Client tool output">
    ```python tools/browser.py theme={null}
    from pydantic import BaseModel

    from harnest.client_tool import client_tool
    from harnest.models.vision import Screenshot


    class CaptureResult(BaseModel):
        screenshot: Screenshot


    @client_tool(output_schema=CaptureResult)
    async def capture_viewport() -> CaptureResult:
        """Capture the visible browser viewport."""
        ...
    ```
  </Tab>
</Tabs>

Client-tool stubs execute on the client. Harnest validates the submitted result before the waiting agent or SubAgent resumes.

## Send transient media

Without a storage annotation, media is available only to the current model turn.

```json theme={null}
{
  "sessionId": "demo",
  "input": {
    "question": "What is visible?",
    "screenshot": {
      "type": "image",
      "mediaType": "image/jpeg",
      "data": "<base64>"
    }
  }
}
```

<Steps>
  <Step title="Harnest validates the bytes">
    The Pydantic field limits are applied to inspected content.
  </Step>

  <Step title="The adapter builds the model request">
    ADK or LangGraph receives the media only for the immediate provider call. This includes mid-turn tool results inside SubAgents.
  </Step>

  <Step title="Harnest clears the private lease">
    Success clears the bytes. A provider retry can reuse them before that point.
  </Step>
</Steps>

| Surface                         | Stored value                                                            |
| ------------------------------- | ----------------------------------------------------------------------- |
| Native checkpoint and history   | `{ "type": "image", "mediaType": "image/jpeg", "content": "attached" }` |
| Logs, traces, and audit records | No bytes or private correlation IDs                                     |
| Public events and `/messages`   | Safe attachment placeholder                                             |
| Final inline agent output       | Returned once on the authenticated response; not replayable             |

<CardGroup cols={2}>
  <Card title="Store and retrieve media" icon="database" href="/harnest/build/models-and-libraries/store-and-retrieve-media">
    Return durable references or make media replayable.
  </Card>

  <Card title="Build client tools" icon="browser" href="/harnest/build/agent-tools/client-tools">
    Resume an agent with browser or device data.
  </Card>
</CardGroup>