Introduction

rai-sdk is a Rust SDK for backend AI workflows. It puts OpenAI, Anthropic, and OpenRouter behind one typed client, so changing model or provider does not mean rewriting request construction, tool plumbing, or stream parsing.

What it solves

Each provider has its own request shape, its own tool-calling protocol, and its own streaming event format. Writing directly against them means three code paths that drift apart. rai-sdk normalizes all three behind a single API while keeping provider-specific escape hatches available.

It also handles the parts that are tedious to get right:

• Tool loops. When a model asks to call a tool, something has to execute it, append the result, and ask the model to continue — until it stops asking. generate() does that for you.
• Structured output. Getting valid JSON out of a model is not the same as getting JSON that matches your type. The SDK generates a JSON Schema from your Rust type, validates the response against it, then deserializes.
• Transient failures. Rate limits and timeouts are normal, not exceptional. Retries with exponential backoff and jitter are built in.
• Stream parsing. Server-sent events arrive split across arbitrary byte boundaries. The SDK buffers and reassembles them correctly.

Supported providers

All three are enabled by default. See Installation to compile only what you need.

Design notes

• Typestate builders. generate() does not exist as a method until the request has both a prompt and a model. Incomplete requests are a compile error, not a runtime one.
• Explicit _once variants. Methods ending in _once make exactly one provider call and never execute registered tools. The plain variants run the full loop. The distinction is in the name rather than a boolean argument.
• Errors carry provenance. Error records which provider failed and exposes category helpers such as is_retryable() and is_auth_error(), so callers can branch on kind instead of matching every variant.

Project status

Early and pre-1.0. The crate works, but the public API may change in breaking ways before 1.0. Pin an exact version if you need stability.

Where to go next

• Installation — add the crate and pick feature flags.
• Quickstart — a working request in a few lines.
• API reference on docs.rs — every public item.

Installation

Add the crate

cargo add rai-sdk

Or edit Cargo.toml directly:

[dependencies]
rai-sdk = "0.1"

Companion crates

The SDK is async and schema-driven, so most projects also need:

[dependencies]
rai-sdk = "0.1"
tokio = { version = "1", features = ["full"] }      # async runtime
serde = { version = "1", features = ["derive"] }    # structured output and tool args
serde_json = "1"                                    # tool return values
futures = "0.3"                                     # only for consuming raw streams

serde and serde_json are required for structured output and tool calling. futures is only needed if you consume stream() directly, since you need StreamExt to iterate it.

You do not need to depend on schemars separately. The SDK re-exports it as rai_sdk::schemars along with the JsonSchema derive, which keeps derive-macro versions in lockstep.

Minimum supported Rust version

Rust 1.86. The crate uses edition 2024 (which needs 1.85), and the dependency tree raises the effective floor to 1.86.

The MSRV is declared as rust-version in Cargo.toml and verified in CI, so it will not drift silently. Treat an MSRV increase as a breaking change.

Feature flags

To compile only one provider, turn the defaults off — but remember that the TLS backend is part of the default set, so you must name one:

[dependencies]
rai-sdk = { version = "0.1", default-features = false, features = ["anthropic", "rustls-tls"] }

Two things are worth knowing about how provider features interact with configuration:

• A feature controls whether provider support is compiled in. Credentials control whether it is usable at runtime.
• Requesting a provider whose feature is disabled fails with Error::ProviderNotEnabled. Requesting one that is compiled in but has no API key fails with Error::ProviderNotConfigured. The two are distinct so you can tell a build-configuration mistake from a deployment mistake.

Choosing a TLS backend

At least one TLS backend must be enabled when any provider is enabled. Enabling a provider with neither is a build error with an explanatory message. A build with no providers and no TLS backend is valid for consumers that only need the shared data types.

rustls-tls (default) needs no system OpenSSL, which makes Linux builds simpler. The cost is that it builds aws-lc-rs, which requires cmake and a C compiler. Most CI images have both; minimal containers often do not.

native-tls uses the operating system’s TLS stack — Security Framework on macOS, SChannel on Windows, OpenSSL on Linux — and avoids building aws-lc-rs with cmake. Choose it if your build environment lacks cmake, or if you want to honor the system trust store:

[dependencies]
rai-sdk = { version = "0.1", default-features = false, features = ["anthropic", "native-tls"] }

On Linux, native-tls links against the system OpenSSL, so you will need its development package (libssl-dev on Debian and Ubuntu) instead.

Cargo features are additive, so another dependency can cause both TLS backends to be compiled. rai-sdk explicitly uses rustls when both are available. To keep aws-lc-rs and cmake out of the dependency graph, disable default features and enable only native-tls, as in the example above.

Note that this crate also disables jsonschema’s default HTTP schema resolution. Schemas are generated locally from your Rust types, so a schema $ref can never trigger an outbound request — which keeps the TLS choice meaningful and removes a class of request-forgery risk. Internal $ref and $defs resolution is unaffected.

Verify the install

cargo build

Then set a key and run a bundled example from a checkout of the repository:

export OPENAI_API_KEY="sk-..."
cargo run --example basic_chat

See Configuration for the full list of environment variables.

Quickstart

Set a key

export OPENAI_API_KEY="sk-..."

Make a request

use rai_sdk::{ClientBuilder, Model};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()
        .model(Model::gpt4o_mini())
        .build()?;

    let response = client
        .request()
        .prompt("Explain Rust ownership in two sentences.")
        .generate()
        .await?;

    println!("{}", response.text());
    Ok(())
}

What each step does

ClientBuilder::new().from_env() reads credentials and settings from the environment. Anything you set explicitly on the builder afterwards takes precedence.

.model(Model::gpt4o_mini()) sets the default model. This also changes the builder’s type: only after a model is present does build() produce a client whose request() starts in a model-ready state. That is why the next step does not need to repeat the model.

.build()? constructs the HTTP client and validates configuration. It fails if the selected provider has no usable credentials.

.request().prompt(...) starts a request. prompt() accepts a &str, a String, a Message, or a full Prompt.

.generate().await? sends the request and, if tools are registered, runs the tool loop until the model produces a final answer. Use generate_once() for exactly one provider call with no tool execution.

response.text() concatenates the text content of the response.

Overriding the model per request

The client’s model is a default, not a constraint. Override it on a single request:

use rai_sdk::{ClientBuilder, Model};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()
        .model(Model::gpt4o_mini())
        .build()?;

    // Uses Anthropic for this one request; the client default is unchanged.
    let response = client
        .request()
        .model(Model::claude_sonnet_46())
        .prompt("Summarize the Rust borrow checker.")
        .generate()
        .await?;

    println!("{}", response.text());
    Ok(())
}

This requires credentials for whichever provider you name, so the example above needs ANTHROPIC_API_KEY in addition to OPENAI_API_KEY.

Tuning generation

Pass a GenerationConfig to control sampling and limits:

use rai_sdk::{ClientBuilder, GenerationConfig, Model};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()
        .model(Model::gpt4o_mini())
        .build()?;

    let response = client
        .request()
        .config(
            GenerationConfig::new()
                .with_temperature(0.2)
                .with_max_tokens(512),
        )
        .prompt("List three Rust testing tips.")
        .generate()
        .await?;

    println!("{}", response.text());
    Ok(())
}

Note that temperature and top_p are ignored for OpenAI reasoning (o-series) models, which do not accept them.

Next steps

• Configuration — every environment variable and its builder equivalent.
• Structured output — get a typed value instead of text.
• Tool calling — let the model call your code.
• Streaming — render tokens as they arrive.

Configuration

Configuration is client-scoped: credentials, endpoints, timeouts, and retry policy. Per-request settings such as temperature live in GenerationConfig instead.

Precedence

Config getters fall back to the environment when a field was not set programmatically, so an explicit value always wins over an environment variable. This means from_env() is a starting point, not a lock-in:

use rai_sdk::{ClientBuilder, Model};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()                                  // read everything available
        .openai_base_url("http://localhost:8080/v1") // then override one field
        .model(Model::gpt4o_mini())
        .build()?;
    let _ = client;
    Ok(())
}

Order matters within the builder chain: a later setter overwrites an earlier one, including one populated by from_env().

Credentials

A missing key is not a construction error. build() succeeds and the failure surfaces as Error::ProviderNotConfigured when you actually use that provider. This lets one binary support several providers and only require the keys for the ones it uses.

Endpoints

Base URLs are also the seam that makes the SDK testable without network access — point them at a local mock server.

OpenRouter attribution

OpenRouter uses attribution headers to identify the calling app.

The canonical variables win when both are present. OPENROUTER_CATEGORIES is comma-separated, and empty entries are trimmed away:

export OPENROUTER_CATEGORIES="productivity,agents"

Timeout

An unparseable value is ignored and the default is kept, rather than failing at startup.

Retries

Retry configuration is only populated from the environment when at least one of these variables is recognized; otherwise the built-in defaults apply. In code, use RetryConfig — see Retries and error handling.

Configuring entirely in code

Nothing requires environment variables. Skip from_env() and set everything explicitly, which is often preferable in tests and in services that get configuration from a secret manager:

use std::time::Duration;

use rai_sdk::{ClientBuilder, Model, RetryConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .openai_key(std::env::var("MY_APP_OPENAI_KEY")?)
        .timeout(30)
        .retry_config(
            RetryConfig::new()
                .with_max_retries(5)
                .with_initial_delay(Duration::from_millis(250)),
        )
        .model(Model::gpt4o_mini())
        .build()?;
    let _ = client;
    Ok(())
}

You can also build a Config directly and hand it to Client::new.

Handling secrets

• Never commit API keys. Use environment variables, a secret manager, or a git-ignored .env file the process loads itself.
• Do not log a Config. Its Debug output contains credentials.
• Keys are only read when requested, so a process that never calls a provider never touches its key.

Providers and models

A Model value carries both the provider and the wire model ID. Choosing a model therefore chooses a provider — there is no separate provider setting to keep in sync.

#![allow(unused)]
fn main() {
use rai_sdk::{Model, ProviderKind};

let model = Model::gpt4o_mini();
assert_eq!(model.provider(), ProviderKind::OpenAI);
}

OpenAI

Constructors cover the current GPT and reasoning families, for example:

#![allow(unused)]
fn main() {
use rai_sdk::Model;

let _ = Model::gpt4o_mini();
let _ = Model::gpt4o();
let _ = Model::gpt4_1();
let _ = Model::gpt5();
let _ = Model::gpt5_mini();
let _ = Model::o3();
let _ = Model::o4_mini();
}

Reasoning (o-series) models are detected by the SDK, which omits sampling parameters they reject such as temperature and top_p. You do not need to special-case that yourself.

Anthropic

#![allow(unused)]
fn main() {
use rai_sdk::Model;

let _ = Model::claude_sonnet_46();
let _ = Model::claude_opus_47();
let _ = Model::claude_haiku_45();
let _ = Model::claude_35_sonnet();
}

Anthropic model IDs are not vendor-prefixed, unlike OpenRouter’s.

OpenRouter

OpenRouter proxies many vendors behind one API, which makes it a good default when you want breadth without managing several accounts.

#![allow(unused)]
fn main() {
use rai_sdk::Model;

// Let OpenRouter pick.
let _ = Model::openrouter_auto();

// Curated constructors.
let _ = Model::openrouter_gpt5();
let _ = Model::openrouter_claude_sonnet_4_5();
let _ = Model::openrouter_gemini_25_flash();
let _ = Model::openrouter_deepseek_r1();
let _ = Model::openrouter_qwen3_coder();
}

OpenRouter IDs are vendor-prefixed (vendor/model).

Any OpenRouter model

The curated list will always lag the catalog, so pass an ID directly for anything not covered:

#![allow(unused)]
fn main() {
use rai_sdk::Model;

let model = Model::openrouter_custom("mistralai/mistral-large-2512");
}

The ID is passed through verbatim, so a typo surfaces as a provider error rather than a compile error.

Attribution

OpenRouter identifies calling apps through attribution headers. Set them once on the client:

use rai_sdk::{ClientBuilder, Model};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()
        .openrouter_http_referer("https://your-app.example")
        .openrouter_title("Your App")
        .model(Model::openrouter_auto())
        .build()?;

    let response = client
        .request()
        .prompt("Summarize OpenRouter in one paragraph.")
        .generate()
        .await?;

    println!("{}", response.text());
    Ok(())
}

See Configuration for the environment-variable equivalents.

Choosing a provider

• OpenAI — strongest structured-output support via native JSON Schema mode.
• Anthropic — long-context work and tool use.
• OpenRouter — breadth, fallback, and access to models you do not have direct accounts for. Note that per-vendor quirks leak through: Gemini models reached via OpenRouter reject schemas containing $schema, $defs, or $ref, which is why the SDK normalizes and inlines generated schemas. See Structured output.

Mixing providers in one process

One client has one default model, but each request can override it, and a client only needs credentials for the providers it actually uses. Check availability at runtime:

use rai_sdk::{ClientBuilder, Model, ProviderKind};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()
        .model(Model::gpt4o_mini())
        .build()?;

    if client.is_provider_available(ProviderKind::Anthropic) {
        let response = client
            .request()
            .model(Model::claude_sonnet_46())
            .prompt("Hello from Anthropic.")
            .generate()
            .await?;
        println!("{}", response.text());
    }

    Ok(())
}

Structured output

Structured output turns a model response into a typed Rust value. The SDK generates a JSON Schema from your type, asks the provider to conform to it, validates the response against the schema, and only then deserializes.

That last part matters: valid JSON is not the same as JSON matching your type. Validation happens before deserialization so failures come with schema-level diagnostics rather than an opaque serde error.

Basic usage

Derive Deserialize and JsonSchema, then call generate_structured::<T>():

use rai_sdk::{ClientBuilder, GenerationConfig, JsonSchema, Model};
use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize, Serialize, JsonSchema)]
struct Recipe {
    name: String,
    ingredients: Vec<String>,
    steps: Vec<String>,
    prep_time_minutes: u32,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()
        .model(Model::gpt4o_mini())
        .build()?;

    let structured = client
        .request()
        .config(GenerationConfig::new().with_temperature(0.2))
        .prompt("Return a simple chocolate cake recipe as JSON.")
        .generate_structured::<Recipe>()
        .await?;

    println!("{:#?}", structured.output);
    println!("raw text: {}", structured.response.text());
    Ok(())
}

The result is a StructuredOutput<T> with two fields: output (your typed value) and response (the underlying response, including usage).

Use JsonSchema from rai_sdk rather than depending on schemars yourself — the SDK re-exports both the trait and the derive so versions cannot mismatch.

With or without tools

Use the _once variant when you want a pure transformation and no tool side effects, even though the client has tools registered.

Validation failures

If the response does not satisfy the schema, you get an error instead of a partially-populated value. Distinguish that from transport problems by inspecting the error:

use rai_sdk::{ClientBuilder, JsonSchema, Model};
use serde::Deserialize;
#[derive(Debug, Deserialize, JsonSchema)]
struct Recipe { name: String }
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = ClientBuilder::new().from_env().model(Model::gpt4o_mini()).build()?;
match client
    .request()
    .prompt("Return a recipe as JSON.")
    .generate_structured::<Recipe>()
    .await
{
    Ok(structured) => println!("{:?}", structured.output),
    Err(error) if error.is_retryable() => eprintln!("transient: {error}"),
    Err(error) => eprintln!("did not match the schema: {error}"),
}
Ok(())
}

Lowering the temperature and describing the desired shape in the prompt both reduce validation failures.

Schema generation details

The generated schema is deliberately conservative, because strict providers reject anything unexpected:

• Subschemas are inlined. Generation runs with inline_subschemas = true and no top-level "$schema", so nested non-recursive types are emitted inline instead of producing "$defs"/"$ref".
• additionalProperties defaults to false on every object schema, without overriding a value you set explicitly.
• "$schema" keys are stripped wherever they appear.

The reason is Gemini: reached through OpenRouter, its response_schema rejects schemas containing "$schema", "$defs", or "$ref" with a 400 INVALID_ARGUMENT.

Recursive types are not supported

Inlining cannot represent a type that transitively contains itself, so schemars falls back to "$defs"/"$ref" for recursive types. The SDK does not resolve those references. A recursive structured-output type will therefore be rejected by strict providers. Flatten the shape — for example, by replacing nested self-references with an ID or a bounded depth — if you need Gemini compatibility.

JSON mode versus schema mode

For cases where you want valid JSON but do not care about its shape:

#![allow(unused)]
fn main() {
use rai_sdk::GenerationConfig;

let config = GenerationConfig::new().with_json_mode(true);
let _ = config;
}

A schema always wins over the json_mode flag. You can also supply a hand-written schema with with_json_schema(..), or generate one from a type without performing a request using with_json_schema_for::<T>().

Tool calling

Tools let the model call your code. You register typed handlers; when the model asks for one, the SDK validates the arguments, runs the handler, feeds the result back, and asks the model to continue — until it produces a final answer.

Defining a tool

A tool’s argument type supplies its JSON Schema, so the schema advertised to the provider and the type your handler receives cannot drift apart.

use rai_sdk::{ClientBuilder, JsonSchema, Model, Result, Tool, ToolContext};
use serde::{Deserialize, Serialize};
use serde_json::json;

#[derive(Debug, Deserialize, Serialize, JsonSchema)]
struct WeatherArgs {
    city: String,
    #[serde(default = "default_unit")]
    unit: String,
}

fn default_unit() -> String {
    "celsius".to_string()
}

async fn get_weather(args: WeatherArgs, _ctx: ToolContext) -> Result<serde_json::Value> {
    Ok(json!({
        "city": args.city,
        "temperature": 22,
        "unit": args.unit,
        "condition": "Sunny"
    }))
}

#[tokio::main]
async fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
    let weather_tool = Tool::new("get_current_weather")
        .description("Get the current weather in a city.")
        .handler(get_weather)?;

    let client = ClientBuilder::new()
        .from_env()
        .model(Model::gpt4o_mini())
        .tool(weather_tool)
        .build()?;

    let response = client
        .request()
        .prompt("What is the weather in Paris right now?")
        .generate()
        .await?;

    println!("{}", response.text());
    Ok(())
}

.handler() is fallible on purpose: it generates and validates the schema at build time, so an unrepresentable argument type fails when you construct the tool rather than mid-conversation.

Write a real description. It is the only thing telling the model when the tool applies.

Handler context

Handlers receive a ToolContext alongside the typed arguments:

The tool loop

generate() repeats: send the request, execute any requested tools, append results, send again. It stops when the model returns a final answer without tool calls.

Bound the loop with max_tool_rounds (default 8):

#![allow(unused)]
fn main() {
use rai_sdk::GenerationConfig;

let config = GenerationConfig::new().with_max_tool_rounds(3);
let _ = config;
}

Exceeding the limit fails with Error::ToolLoopLimitExceeded, which prevents a model that keeps calling tools from looping indefinitely.

Getting tool calls without executing them

generate_once() performs a single provider call and returns tool calls without running your handlers. Use it when you want to inspect, gate, or approve calls before they take effect:

use rai_sdk::{ClientBuilder, Model};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = ClientBuilder::new().from_env().model(Model::gpt4o_mini()).build()?;
let response = client
    .request()
    .prompt("What is the weather in Paris?")
    .generate_once()
    .await?;

for message in &response.messages {
    if message.has_tool_calls() {
        for call in &message.tool_calls {
            println!("model wants {} with {}", call.name, call.arguments);
        }
    }
}
Ok(())
}

Tool calls live on the individual Message values in response.messages, alongside has_tool_calls() for a quick check.

Argument validation

Arguments are validated against the tool’s schema before the handler runs. A validation failure is not a hard error: the SDK returns a structured tool-error message to the model describing each violation, so it can correct itself and call the tool again.

That means your handler only ever sees arguments that already satisfy the schema, and a confused model produces a retry rather than a failed request. Each violation is reported as a ToolArgumentIssue with the offending path, the schema keyword that rejected it, and a message.

Tighten schemas to get better self-correction. schemars attributes carry through:

#![allow(unused)]
fn main() {
use rai_sdk::JsonSchema;
use serde::Deserialize;

#[derive(Deserialize, JsonSchema)]
#[schemars(deny_unknown_fields)]
struct SearchArgs {
    #[schemars(description = "Name or email substring to search for", length(min = 1))]
    query: String,
    #[schemars(description = "Maximum results to return", range(min = 1, max = 10))]
    limit: Option<usize>,
}
}

Errors returned by your handler are also surfaced to the model as tool-error content rather than aborting generation. Return an error when a call genuinely cannot be satisfied and you want the model to react to that fact.

Per-request tools

Tools can be registered on the client (shared by every request) or per request:

Limitations

• Raw streaming rejects requests with registered tools. See Streaming.
• A provider that does not support tool calling produces Error::ToolProviderUnsupported.
• Tool names must be unique per client; registering a duplicate is an error.

Streaming

Streaming exists for two different reasons, and the SDK offers a method for each:

• You want to show tokens as they arrive — use stream() and handle events.
• You want a complete response but delivered over the streaming transport — use stream_accumulated(), which streams internally and hands you a finished Response. This is useful for long generations that would otherwise sit near a request timeout.

Accumulated streaming

use rai_sdk::{ClientBuilder, Model};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()
        .model(Model::gpt4o_mini())
        .build()?;

    let response = client
        .request()
        .prompt("Write a short launch announcement.")
        .stream_accumulated()
        .await?;

    println!("{}", response.text());
    Ok(())
}

The result is the same shape as generate() — only the transport differs.

Raw stream events

For incremental output, iterate the stream. This needs futures::StreamExt in scope.

use futures::StreamExt;
use rai_sdk::{ClientBuilder, Model, provider::ProviderStreamEvent};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()
        .model(Model::gpt4o_mini())
        .build()?;

    let mut stream = client
        .request()
        .prompt("Count from one to five.")
        .stream()
        .await?;

    while let Some(event) = stream.next().await {
        match event? {
            ProviderStreamEvent::Text(text) => print!("{text}"),
            ProviderStreamEvent::Done { .. } => println!(),
            _ => {}
        }
    }

    Ok(())
}

Each item is a Result, because a stream can fail partway through. Do not use while let Some(Ok(event)): that silently swallows mid-stream errors and looks like a clean early finish.

Event kinds

ProviderStreamEvent normalizes each provider’s SSE format:

Tool-call arguments arrive as fragments that are not individually valid JSON. Buffer all chunks for an id and only parse once Done arrives.

Match non-exhaustively (_ => {}) so new event kinds do not break your code.

Streaming and tools

The streaming methods reject requests that carry tools, rather than silently ignoring them, with Error::InvalidRequest. Executing a tool loop requires sending follow-up requests, which is incompatible with handing you a single continuous stream.

The check uses the request’s effective tool set, so one client can do both. Opt out per request to stream:

#![allow(unused)]
fn main() {
use rai_sdk::{ClientBuilder, Model};

async fn run(tool: rai_sdk::Tool) -> Result<(), Box<dyn std::error::Error>> {
let client = ClientBuilder::new()
    .from_env()
    .model(Model::gpt4o_mini())
    .tool(tool)
    .build()?;

// Tools run here.
let answer = client.request().prompt("Use a tool if needed.").generate().await?;

// And this request streams, because it opts out of tools.
let stream = client
    .request()
    .no_tools()
    .prompt("Just write prose.")
    .stream()
    .await?;
let _ = (answer, stream);
Ok(())
}
}

The rule applies in both directions: adding a tool with .tool(..) on a request makes it non-streamable even if the client has no tools, instead of quietly dropping it.

If you need incremental output and tool execution in one exchange, drive the loop yourself with generate_once(), executing calls between turns.

Cancellation

Dropping a stream aborts the upstream provider request. Every streaming method is driven entirely by its consumer: the provider’s HTTP response body is polled from inside the returned stream, never from a detached background task. Dropping the stream drops that body, closes the connection, and the provider stops generating.

That holds for the whole family — stream(), generate_stream_events(), stream_wire_events(), and stream_accumulated() — and it holds when the surrounding task is cancelled rather than the stream explicitly dropped, which is what a tokio::time::timeout or a web-framework client disconnect looks like. Nothing keeps running in the background.

Two consequences to plan for:

• A cancelled generation produces no terminal event, so no usage is reported. Providers still bill for what they generated before the abort, so metering cannot rely on the final usage event alone.
• Conversely, there is nothing to clean up. You do not need a cancellation token or an abort handle; letting the stream go out of scope is the whole mechanism.

Proxying a stream to your own clients

If your server holds the provider credentials and streams results on to a desktop or browser client, the events have to cross a wire. The wire module covers that case:

  client ──POST──▶ your server ──rai-sdk──▶ provider
         ◀──SSE─── WireStreamEvent ◀────────┘

stream_wire_events() yields WireStreamEvents, which serialize to a tagged JSON object — one SSE data: payload each:

#![allow(unused)]
fn main() {
use futures::StreamExt;
use rai_sdk::{ClientBuilder, Model};

async fn run() -> Result<(), Box<dyn std::error::Error>> {
let client = ClientBuilder::new().from_env().model(Model::gpt4o_mini()).build()?;

let mut events = client
    .request()
    .prompt("Explain SSE in one sentence.")
    .stream_wire_events()
    .await?;

while let Some(event) = events.next().await {
    // event: text_delta
    // data: {"type":"text_delta","text":"Server-sent"}
    println!("event: {}\ndata: {}\n", event.tag(), serde_json::to_string(&event)?);
}
Ok(())
}
}

On the receiving side, StreamAccumulator is the client-side counterpart of stream_accumulated(): feed it the parsed events and it hands back one Response, tool calls included.

use rai_sdk::wire::{StreamAccumulator, WireStreamEvent};

fn main() -> Result<(), Box<dyn std::error::Error>> {
let payloads = [
    r#"{"type":"message_start","protocol_version":1,"model":"gpt-4o-mini","provider":"openai"}"#,
    r#"{"type":"text_delta","text":"Hello, "}"#,
    r#"{"type":"text_delta","text":"world."}"#,
    r#"{"type":"usage","usage":{"prompt_tokens":9,"completion_tokens":3,"total_tokens":12}}"#,
    r#"{"type":"message_stop","finish_reason":"stop"}"#,
];

let mut accumulator = StreamAccumulator::new();
for payload in payloads {
    accumulator.push(serde_json::from_str::<WireStreamEvent>(payload)?)?;
}

let response = accumulator.finish()?;
assert_eq!(response.text(), "Hello, world.");
assert_eq!(response.usage.unwrap().total_tokens, Some(12));
Ok(())
}

examples/sse_proxy.rs runs the whole loop — an axum handler, SSE re-emission, and client-side reassembly — in one process.

Wire events

Unlike the other streaming methods, stream_wire_events() items are not Results. Once the stream is open, every outcome is an event:

A mid-stream provider failure arrives as error rather than as a truncated response, which is the point: a client that receives no terminal event at all knows its connection died instead. StreamAccumulator::finish() enforces the distinction — it returns the carried error for the first case and a stream-kind error naming the truncation for the second.

Versioning

The "type" strings and each event’s field names are a compatibility surface: a server and a client can be built from different rai-sdk versions. Renaming or removing one is a breaking change and will be called out in the changelog. Adding a variant is not, so match with a catch-all arm — WireStreamEvent and WireErrorKind are both #[non_exhaustive], and an unrecognized error kind deserializes into WireErrorKind::Other instead of failing.

WIRE_PROTOCOL_VERSION names the current revision of the framing and rides on every message_start. It is bumped only when a client must react to a framing change, never for additive variants.

Timeouts

Streaming does not exempt a request from the configured timeout. A long generation can still exceed AI_TIMEOUT_SECONDS; raise it for workloads that legitimately run long. See Configuration.

Multimodal prompts

A plain string prompt is shorthand. Underneath, a request carries a Prompt: a sequence of Message values, each with a role and either simple text or a list of content blocks.

Roles and messages

#![allow(unused)]
fn main() {
use rai_sdk::{Message, Prompt};

let prompt = Prompt::single(Message::system("You are a terse Rust expert."))
    .with_message(Message::user("Why does the borrow checker reject this?"));

assert_eq!(prompt.system_message(), Some("You are a terse Rust expert."));
}

Prompt::single starts from one message, Prompt::new takes a whole Vec<Message>, and with_message appends. A Vec<Message> also converts directly with .into(), so you rarely need to name Prompt at all.

Roles are System, User, Assistant, and Tool. Providers differ in how they handle system prompts — Anthropic takes it as a separate top-level field rather than a message — and the SDK handles that translation, so you can express it as a message consistently.

Multi-turn conversations

Build history by listing messages in order:

#![allow(unused)]
fn main() {
use rai_sdk::{Message, Prompt};

let prompt = Prompt::new(vec![
    Message::user("What is a lifetime?"),
    Message::assistant("A lifetime names how long a reference is valid."),
    Message::user("Show me a case where elision fails."),
]);

assert_eq!(prompt.messages.len(), 3);
}

If you already have the previous turns as ConversationTurn values, Prompt::with_history expands each turn into its user message, assistant message, and tool results for you — or use generate_with_history on the request builder.

Images

Use Message::user_multimodal with content blocks:

use rai_sdk::{ClientBuilder, ContentBlock, Message, Model, Prompt};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ClientBuilder::new()
        .from_env()
        .model(Model::gpt4o_mini())
        .build()?;

    let prompt = Prompt::single(Message::user_multimodal(vec![
        ContentBlock::text("Describe this image in one sentence."),
        ContentBlock::image_url("https://example.com/image.png"),
    ]));

    let response = client.request().prompt(prompt).generate().await?;
    println!("{}", response.text());
    Ok(())
}

Order is meaningful: put the instruction text before or after the image deliberately, since models attend to position.

URL versus inline data

ImageSource supports either a URL or base64 data with a media type. Use a URL when the provider can reach it, and base64 for local or private images that must travel in the request body.

Content block types

ContentBlock covers Text, Image, Audio, Video, and File.

Provider support is uneven, and this is the most important caveat in this chapter. OpenAI and OpenRouter currently serialize image content. The other block types exist in the common prompt model, but provider-specific serialization may be incomplete — a block a provider does not support may simply not reach the model rather than producing a loud error.

Verify behavior for your provider and model before relying on audio, video, or file blocks in production. Choose a model that documents support for the modality you need; multimodal capability is per-model, not per-provider.

Checking a prompt

#![allow(unused)]
fn main() {
use rai_sdk::{ContentBlock, Message, Prompt};

let prompt = Prompt::single(Message::user_multimodal(vec![
    ContentBlock::text("Hello"),
    ContentBlock::image_url("https://example.com/image.png"),
]));

assert!(prompt.is_multimodal());
}

Prompt::system_message() returns the first system message, if any.

Retries and error handling

Rate limits and timeouts are routine when calling model providers, not exceptional. The SDK retries them by default so you do not have to wrap every call.

Defaults

Delays grow exponentially (1s, 2s, 4s, …), are clamped at the maximum, and are randomized by jitter. Jitter matters under load: without it, many clients throttled at the same moment retry in lockstep and re-create the spike that throttled them.

Configuring retries

use std::time::Duration;

use rai_sdk::{ClientBuilder, Model, RetryConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let retry = RetryConfig::new()
        .with_max_retries(5)
        .with_initial_delay(Duration::from_millis(500))
        .with_max_delay(Duration::from_secs(30))
        .with_backoff_multiplier(2.0)
        .with_jitter(true);

    let client = ClientBuilder::new()
        .from_env()
        .model(Model::claude_sonnet_46())
        .retry_config(retry)
        .build()?;

    let response = client
        .request()
        .prompt("Give me three practical Rust error-handling tips.")
        .generate()
        .await?;

    println!("{}", response.text());
    Ok(())
}

The same values can come from the environment — see Configuration.

Disabling retries

use rai_sdk::{ClientBuilder, Model};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// For every request from this client.
let client = ClientBuilder::new()
    .from_env()
    .no_retry()
    .model(Model::gpt4o_mini())
    .build()?;

// Or for one request only.
let response = client
    .request()
    .no_retry()
    .prompt("One-shot request.")
    .generate()
    .await?;
println!("{}", response.text());
Ok(())
}

Disabling retries is the right choice when the caller already has its own retry policy, or when you are inside a request path with a hard latency budget and would rather fail fast. RetryConfig::none() is equivalent.

What is retried

Retried:

• Error::RateLimit — the provider throttled you
• Error::Timeout — the request exceeded the configured timeout
• Transient HTTP and transport failures

Not retried:

• Error::Auth — a bad key will stay bad
• Error::InvalidRequest — malformed requests fail deterministically
• Error::ModelNotAvailable, Error::ProviderNotConfigured, Error::ProviderNotEnabled
• Error::ContentFiltered — a policy decision, not a transient fault

Check with error.is_retryable() rather than enumerating variants.

Handling errors

Error exposes category helpers so you can branch on kind:

use rai_sdk::{ClientBuilder, Model};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = ClientBuilder::new().from_env().model(Model::gpt4o_mini()).build()?;
match client.request().prompt("Hello").generate().await {
    Ok(response) => println!("{}", response.text()),
    Err(error) if error.is_auth_error() => eprintln!("check your API key: {error}"),
    Err(error) if error.is_rate_limit() => eprintln!("still throttled after retries: {error}"),
    Err(error) => eprintln!("{} failure: {error}", error.kind_str()),
}
Ok(())
}

kind_str() is a better metric label than the full error message, which may contain request-specific detail.

Notable variants

• Error::ToolArguments — a model supplied arguments that failed schema validation. Normally handled internally and returned to the model for self-correction; see Tool calling.
• Error::ToolLoopLimitExceeded — the tool loop hit max_tool_rounds.
• Error::ToolProviderUnsupported — tool calling is not supported for that provider.
• Error::ProviderNotEnabled versus ProviderNotConfigured — a missing Cargo feature versus a missing API key. See Installation.
• Error::Request — a provider error with no more specific mapping, including malformed provider responses.

Interaction with timeouts

The retry budget multiplies the timeout: a 120-second timeout with 3 retries can take well over six minutes worst-case, including backoff. Size the timeout and retry count together against your own deadline instead of tuning them independently.

Examples

The repository ships runnable examples. Clone it and run them with cargo run --example.

git clone https://github.com/rmagatti/rai-sdk
cd rai-sdk
export OPENAI_API_KEY="sk-..."

These make real API calls and will consume credit.

basic_chat

cargo run --example basic_chat

The smallest complete request: build a client from the environment, send a prompt, print the text. Start here to confirm your credentials work.

Source: examples/basic_chat.rs

structured_output

cargo run --example structured_output

Derives JsonSchema on a struct and uses generate_structured to get a validated, typed value back instead of text. See Structured output.

Source: examples/structured_output.rs

tool_calling

cargo run --example tool_calling

Registers a typed tool and lets generate() run the loop: the model requests the call, the SDK executes the handler, feeds the result back, and the model produces a final answer. See Tool calling.

Source: examples/tool_calling.rs

sse_proxy

cargo run --example sse_proxy

The full server-side proxy loop in one process: an axum handler streams a generation with stream_wire_events(), re-emits each event as an SSE data: payload, and a client in the same binary parses them back and reassembles one Response with StreamAccumulator. Reach for this when your server holds the provider credentials and streams results on to a desktop or browser client. See Streaming.

Source: examples/sse_proxy.rs

Using a different provider

All four examples use Model::gpt4o_mini(). Change the constructor and set that provider’s key to try another:

#![allow(unused)]
fn main() {
use rai_sdk::Model;
let _ = Model::claude_sonnet_46();  // needs ANTHROPIC_API_KEY
let _ = Model::openrouter_auto();   // needs OPENROUTER_API_KEY
}

Seeing what the SDK is doing

The SDK emits tracing spans and events. Install a subscriber and set RUST_LOG to inspect requests, retries, and tool execution:

RUST_LOG=rai_sdk=debug cargo run --example tool_calling

tracing-subscriber is already a dev-dependency, so this works in the examples without adding anything.

Testing without API calls

The SDK’s own test suite is fully offline: base URLs are pointed at a mock HTTP server, so no test needs credentials. You can use the same approach in your project by setting OPENAI_BASE_URL (or the equivalent) at a local mock. See Configuration and Contributing.

Contributing

Contributions are welcome. The authoritative guide lives in the repository:

• CONTRIBUTING.md — setup, workflow, code style, and PR expectations
• CODE_OF_CONDUCT.md
• SECURITY.md — report vulnerabilities privately, never in a public issue

What CI enforces

cargo fmt --all --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
cargo test --doc --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features

Two policies are worth calling out because they surprise people:

• missing_docs is enforced. Every new public item needs documentation, or the build fails.
• Tests must be offline. The suite runs with no provider credentials and must never make live API calls. Provider behavior is tested against a mock HTTP server through the base-URL configuration. CI actively fails if credentials are present.

The crate also sets unsafe_code = "forbid", and the MSRV (currently 1.86) is read from Cargo.toml and verified in CI.

Working on this guide

The guide is an mdBook in docs/:

cargo install mdbook --locked
mdbook serve docs --open   # live reload while editing
mdbook build docs          # one-off build

Add a chapter by creating the file in docs/src/ and listing it in docs/src/SUMMARY.md — a page missing from SUMMARY.md will not appear in the book.

Pushes to main deploy the guide to GitHub Pages automatically. Pull requests build it without deploying, so a broken book is caught in review.

Reporting issues

Use the issue templates. For bugs, include the rai-sdk version, Rust version, OS, provider, and enabled features — provider-specific and feature-gating bugs are common and hard to reproduce without those details.

Never paste API keys into an issue, including inside logs or backtraces.