# Ox TypeScript SDK and Realtime API

SDK `0.1.0` private preview · Protocol `ox.realtime` version `1.0.0` · [SDK resources](/voice-sdk/index.html) · [JSON Schema](/voice-sdk/ox-realtime-v1.schema.json) · [Protocol inspector](/voice-inspector.html)

Use `@ox/sdk` from trusted server code for model inference, reusable media, Realtime sessions, and project administration. Browser code imports `@ox/sdk/browser` and receives only short-lived session descriptors. Permanent API keys and control-plane tokens never belong in browser or mobile code.

## 1. Install and initialize the SDK

### Before you begin

You need Node.js 20 or newer, an Ox API key, and the private-preview package artifact supplied with your access. Keep the API key in trusted server code. Never place it in browser JavaScript, mobile binaries, source control, URLs, or logs. The current public preview API origin is `https://www.amerint.co`; set it explicitly until the permanent Ox domain is assigned.

### Install the private preview

The package is not published to npm yet. Install the supplied artifact directly:

```bash
npm install /absolute/path/to/ox-sdk-0.1.0.tgz
```

> Private preview: repository contributors can create the same artifact with `npm install`, `npm run sdk:build`, and `npm pack --workspace @ox/sdk`. Customer applications should install only the artifact they receive from Ox.

### Create a server client

Create one client and reuse it. The example reads `OX_API_KEY` from the process environment and fails before making a request when the variable is missing.

```js
import { Ox } from "@ox/sdk";

const apiKey = process.env.OX_API_KEY;
if (!apiKey) throw new Error("OX_API_KEY is required");

const ox = new Ox({
  apiKey,
  baseUrl: process.env.OX_API_BASE_URL ?? "https://www.amerint.co",
});

const models = await ox.models.list();
if (models.data.length === 0) {
  throw new Error("No Ox models are enabled for this project");
}

console.log("Connected to Ox");
console.table(models.data.map(({ id }) => ({ id })));
```

Save this as `quickstart.mjs`. Load the key from your secret manager or a protected local environment file, then run:

```bash
node --env-file=.env quickstart.mjs
```

Or [download the Node quickstart](/starters/ox-quickstart.mjs) and run the same command after installing your private-preview SDK artifact.

### Verify the connection

A successful run prints `Connected to Ox` followed by the model IDs enabled for your project. A `401` means the server credential is missing or invalid. An empty model list means the project needs model access before it can run inference.

The SDK rejects permanent credentials in browser runtimes, uses a ten-minute request timeout, supports per-request cancellation and timeouts, and does not retry inference or session creation automatically.

## 2. Models, media, and chat completions

### Choose an enabled model

List the models enabled for the current project before selecting one. Upload image or video bytes once when several questions will reuse the same content.

```ts
import { readFile } from "node:fs/promises";

const models = await ox.models.list();
const model = models.data[0];
if (!model) throw new Error("No Ox models are enabled for this project");

const media = await ox.media.create(await readFile("inspection.mp4"), {
  contentType: "video/mp4",
});

const completion = await ox.chat.completions.create({
  model: model.id,
  media_ids: [media.id],
  messages: [{ role: "user", content: "Describe the first safety issue." }],
  temperature: 0,
});

console.log(completion.choices[0]?.message.content);
```

### Stream a response

For incremental output, set `stream: true` and consume the returned async iterable. Abort the request when the downstream client disconnects.

```ts
const stream = await ox.chat.completions.create({
  model: "Qwen/Qwen3-VL-30B-A3B-Instruct",
  messages: [{ role: "user", content: "Summarize the scene." }],
  stream: true,
  stream_options: { include_usage: true },
}, { signal: request.signal, timeoutMs: 60_000 });

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? "");
}
```

Media uploads accept `Blob`, `ArrayBuffer`, or an array-buffer view and are limited to 256 MiB. Use `ox.media.exists(media.id)` to check a content-addressed object without downloading it.

## 3. Mint a Realtime session on your server

### Keep the permanent key on the server

Create sessions only in authenticated server routes. Return the descriptor with `Cache-Control: no-store`; never return the permanent API key. Select `ox/voice-agent` for a conversational agent that can call customer functions while the microphone remains live.

```ts
const session = await ox.realtime.sessions.create({
  model: "ox/voice-agent",
  transport: "websocket",
  session: {
    instructions: "Help customers schedule service. Confirm before booking.",
    endpointing: { eagerness: "medium" },
  },
});

return Response.json(session, {
  headers: { "Cache-Control": "no-store" },
});
```

### Return a short-lived descriptor

The lower-level HTTP contract is `POST https://www.amerint.co/v1/realtime/sessions` with `Authorization: Bearer $OX_API_KEY`. A returned session secret is short-lived, single-use, and still sensitive.

## 4. Connect a browser voice agent

### Connect from a user gesture

```ts
import { connectBrowserRealtime } from "@ox/sdk/browser";

const call = await connectBrowserRealtime({
  session,
  workletBaseUrl: "https://www.amerint.co/worklets",
  configuration: {
      instructions: "Help customers schedule service. Confirm before booking.",
      language: "en",
      voice: "Ryan",
      endpointing: { eagerness: "medium" },
      reasoning: { model: "Qwen/Qwen3-Next-80B-A3B-Instruct", temperature: 0.3, max_output_tokens: 512 },
      tools: [{
        type: "function",
        name: "find_appointments",
        description: "Find open appointment times.",
        parameters: { type: "object", properties: { date: { type: "string" } }, required: ["date"] },
      }],
  },
  onToolCall: async ({ name, arguments: json }) => {
    const args = JSON.parse(json);
    if (name === "find_appointments") return findAppointmentsForSignedInUser(args);
    throw new Error("Unknown tool");
  },
});

call.on("event", console.log);
await call.start(); // microphone permission must follow a user gesture
```

### Use the hosted preview bundle

The hosted private-preview ESM bundle is also available at `https://www.amerint.co/voice-sdk/ox-realtime.js` for applications that cannot install the package yet.

> Supported options are `instructions`, up to 64 `history` messages, up to 32 function `tools`, `language`, `voice`, `reasoning` model/temperature/token limit, endpointing eagerness and silence bounds, and the fixed component identifiers returned by the API. Browser configuration is visible to and controllable by the signed-in user. For server-owned policy, send the same `session` object during minting only when health reports `server_persisted_agent_configuration: true`; the server then injects it before audio is admitted and the SDK will not overwrite it.

### Understand the runtime boundary

The SDK owns AudioWorklet capture, windowed-sinc resampling, exact 80 ms framing, capture-age and socket backpressure rejection, adaptive output jitter buffering, and local plus server-confirmed barge-in. Binary client frames are exactly 2560 bytes of 16000 Hz mono PCM16. Binary server frames are 24000 Hz mono PCM16. JSON and base64 audio are not accepted on the public hop.

Tool execution remains customer-owned. The model emits `response.function_call_arguments.done`; the SDK invokes the allow-listed handler, submits `conversation.item.create`, and sends `response.create` so the agent can continue the same turn. Microphone capture stays live while the tool is pending. Results time out after 15 seconds and are limited to 65536 bytes.

## 5. Connect with LiveKit WebRTC or SIP

### Connect a browser with WebRTC

Request WebRTC on the server, then pass only the returned descriptor to the browser. The adapter uses the official LiveKit client and enables microphone publishing by default.

```ts
// Trusted server
const session = await ox.realtime.sessions.create({
  model: "ox/voice-agent",
  transport: "webrtc",
  identity: signedInUser.id,
});
```

```ts
// Browser, called from a user gesture
import { connectLiveKitRealtime } from "@ox/sdk/browser";

const room = await connectLiveKitRealtime(session);
```

Import the same adapter from `@ox/sdk/livekit` when a dedicated subpath is preferable. Private-preview applications without package installation can import it from `https://www.amerint.co/voice-sdk/ox-livekit.js`. Pass `publishMicrophone: false` if the application will publish its own audio track. The participant token is scoped to one room and microphone publishing.

### Place an outbound SIP call

Outbound telephony uses the same room and bridge. The server validates E.164 destinations and dispatches the model participant before dialing.

```ts
const call = await ox.realtime.calls.create({ to: "+14155550100" });
```

## 6. Lifecycle and canonical events

### Server events

The relay converts provider-specific speech and transcript names into this single dialect. Customers should not branch on provider names.

| Event | State | Meaning |
| --- | --- | --- |
| `session.created` | connected | The upstream admitted the session. |
| `session.updated` | ready | The requested configuration is active. |
| `input.speech_started` | listening | Server VAD confirmed speech; stop stale assistant playout immediately. |
| `input.speech_stopped` | thinking | Server VAD closed the user turn. |
| `input.transcript.partial` | listening | The current full input hypothesis; replace the previous partial. |
| `input.transcript.final` | thinking | Final transcript for the user turn. |
| `response.created` | thinking | A response generation began. |
| `output.transcript.delta` | speaking | Append-only assistant transcript text. |
| `output.transcript.final` | speaking | Final assistant transcript for the response. |
| `output.audio.done` | speaking | No more binary PCM packets will be emitted for this response. |
| `response.function_call_arguments.done` | tool_wait | Execute the named customer function and submit its result. |
| `tool.started` | tool_wait | The agent began a function call. |
| `tool.finished` | thinking | The function result was accepted by the agent. |
| `response.cancelled` | listening | Barge-in cancelled the in-progress response. |
| `response.done` | ready | The response finished. |
| `error` | error | A bounded protocol or runtime error occurred. |
| `session.end` | closed | The session ended and will not emit more media. |
| `pong` | ready | Heartbeat response. |
| `pandan.relay.trace` | unchanged | Redacted relay-relative timing; contains no provider credential or wall clock. |
| `pandan.relay.provider` | unchanged | Resolved admission route without credentials. |
| `trace.stage` | unchanged | Redacted per-stage deadline timing from the Ox agent runtime. |

### Client events

| Event | Direction | Meaning |
| --- | --- | --- |
| `session.update` | client | Configure instructions, history, voice, endpointing, reasoning, and customer functions. |
| `conversation.item.create` | client | Return the result of a customer function call. |
| `response.create` | client | Continue generation after a tool result. |
| `session.close` | client | End the session cleanly. |
| `ping` | client | Application heartbeat. |

### State transitions

Normal state flow: `minted → connected → ready → listening → thinking → speaking → ready`. Interruption moves `speaking → listening`; a tool call moves `thinking → tool_wait → thinking`. `error` and `closed` are terminal for the current connection.

## 7. Barge-in, backpressure, and reconnect

### Interruptions and buffered audio

Stop local playout as soon as the capture worklet detects probable speech. Confirm the interruption on `input.speech_started`, discard all buffered audio from the cancelled generation, and never replay it. The SDK drops stale capture frames rather than building conversational lag. The relay closes a client that exceeds 1048576 buffered bytes.

### Reconnect safely

A session secret expires after 900 seconds, is single-use, and cannot reconnect after it has been claimed. Mint a new session and restore only customer-approved text history. Never replay microphone audio or a partially executed tool call automatically.

## 8. Realtime readiness

### Read the readiness response

`GET /v1/realtime/health` reports configuration readiness; it is not an active upstream-provider probe. The top-level `status` describes customer traffic admission: `ready` means the customer API can mint sessions, `degraded` means only the browser preview is available, and `unavailable` means realtime sessions cannot be established.

### Gate only the features you use

For a browser voice agent that calls customer tools, gate only on the fields that path requires:

```ts
const health = await ox.realtime.health.retrieve();

if (!health.customer_api_ready ||
    !health.capabilities.voice_agent ||
    !health.capabilities.customer_tools) {
  throw new Error(health.summary);
}
```

### Understand limitations

Missing optional features appear in `limitations`. In particular, `durable_state_unavailable` disables server-persisted agent configuration, durable session records, and control-plane quotas; it does not disable browser-configured voice, customer tools, or core realtime traffic. Require `durable_state` or `capabilities.server_persisted_agent_configuration` only when your application uses those features.

## 9. Conformance and inspection

### Validate the event contract

Run `npm run voice:conformance` against the versioned normal-turn, interruption, binary-audio, error, expiry, and reconnect fixtures. Attach `ProtocolInspector` to the SDK to record monotonic timing and state transitions. It redacts keys, authorization, cookies, session secrets, tokens, audio bodies, and function outputs before export. Do not use raw WebSocket logging in production.

## 10. Limits, measured latency, and capacity

### Hard protocol limits

Hard limits: 120-second sessions, 65536-byte control frames, 262144-byte pre-admission queue, 32 tools, 8192-byte schema per tool, and 30 session mints per project per minute when durable production state is enabled. Account-specific limits returned by the control plane override preview defaults.

### Measured warm latency

> Published measurement (2026-08-12T20:05:00Z): Warm n=10, concurrency 1, exact Node implementation of the browser WebSocket contract; not a physical browser or agent-tool benchmark.

| Warm metric | p50 | p95 |
| --- | ---: | ---: |
| Relay upstream connect | 105.585 ms | 188.833 ms |
| First input to first PCM | 258.054 ms | 306.638 ms |
| Estimated first input to audible | 282.168–290.228 ms | 330.643–338.703 ms |

### Cold-start and capacity scope

> Cold sample: 365526.377 ms to session creation. Scale-to-zero is not an interactive path. Customer traffic requires warm admitted capacity. Admission-controlled preview. No public concurrency SLO is promised; retry a 503 only with jittered backoff or use reserved capacity. These are scoped engineering observations, not an agent-tool latency distribution, concurrency promise, quality result, or production SLO.

## 11. Production session minting

### Use the hardened Next.js example

The [Next.js production example](/voice-sdk/session-mint-example.ts) authenticates the application user, requires same-origin Fetch Metadata, validates a double-submit CSRF token, applies a per-tenant quota before calling Ox, returns no-store JSON, and logs only request/session identifiers. Replace its explicit auth and quota adapters with your own durable implementations. Never log `Authorization`, cookies, CSRF values, session descriptors, tool arguments/results, or audio.

## 12. Project control plane

### Create project credentials

Project administration requires a separate control-plane client and credential. Never initialize it with a customer API key or expose its token to browser code.

```ts
import { OxControlPlane } from "@ox/sdk";

const token = process.env.OX_CONTROL_PLANE_ADMIN_TOKEN;
if (!token) throw new Error("OX_CONTROL_PLANE_ADMIN_TOKEN is required");

const controlPlane = new OxControlPlane({ token });
const project = await controlPlane.projects.create({
  name: "Acme Voice",
  contactEmail: "developer@example.com",
});
const issued = await controlPlane.apiKeys.create(project.id, {
  name: "Project automation",
  scopes: ["project:read", "project:write", "project:keys"],
});

await deliverKeyOnceToAuthorizedCustomer(issued.value);
```

`issued.value` is returned once. Do not log it, store it in source control, or send it to an unauthorized client. `projects` supports list, create, retrieve, suspend, and reactivate. `apiKeys` supports list, create, and revoke.

### Use a project-scoped management key

Customers never receive the global control-plane token. Bind the scoped key to its project in trusted server code:

```ts
import { OxProject } from "@ox/sdk";

const project = new OxProject({
  projectId: process.env.OX_PROJECT_ID!,
  apiKey: process.env.OX_PROJECT_MANAGEMENT_KEY!,
});

const agents = await project.agents.list();
const sessions = await project.sessions.list({ limit: 25 });
```

Management keys require explicit `project:read`, `project:write`, or `project:keys` scopes, cannot cross project boundaries, and cannot issue keys with scopes they do not possess. Realtime-only keys retain only `realtime` and cannot access management routes.

### Register webhooks and inspect usage

Register realtime-session and Hallikar incident webhooks and store the returned signing secret once. Verify delivery signatures against the unmodified request body before parsing JSON.

```ts
const endpoint = await controlPlane.webhooks.create(projectId, {
  url: "https://example.com/webhooks/ox",
  events: ["realtime.session.completed", "realtime.session.failed", "hallikar.incident.opened"],
});
await storeWebhookSecret(endpoint.secret);

const usage = await controlPlane.usage.list(projectId, { days: 30 });
const audit = await controlPlane.auditLogs.list(projectId, { limit: 50 });
const invitation = await controlPlane.dashboardInvitations.create(projectId);
```

Webhook endpoints support list, create, and disable. Dashboard invitation tokens are short-lived and should be delivered only to the intended project operator.

### Manage agents and inspect session history

Agents are reusable, project-scoped configurations. Creating an agent produces version 1; supplying a new configuration through `agents.update` appends an immutable version. Name-only updates do not create a configuration version.

```ts
const agent = await controlPlane.agents.create(projectId, {
  name: "Support",
  configuration: {
    instructions: "Resolve account questions and confirm before changing data.",
    voice: "Ryan",
    endpointing: { eagerness: "medium" },
  },
});

const updated = await controlPlane.agents.update(projectId, agent.id, {
  configuration: {
    ...agent.versions.at(-1)?.configuration,
    instructions: "Resolve account questions. Escalate refund requests.",
  },
});

const sessions = await controlPlane.sessions.list(projectId, {
  limit: 25,
  status: "completed",
  transport: "websocket",
});
const latest = sessions.data[0]
  ? await controlPlane.sessions.retrieve(projectId, sessions.data[0].id)
  : undefined;
```

Use `agents.list`, `agents.retrieve`, and `agents.archive` for lifecycle management. Archived agents remain available with `agents.list(projectId, { includeArchived: true })`. Session records contain project metadata, configuration, routing, trace identifiers, and metered audio usage when those fields have been recorded; they never contain the one-time session secret.

### Verify webhook signatures

```ts
import { verifyWebhook } from "@ox/sdk";

const rawBody = await request.text();
const secret = process.env.OX_WEBHOOK_SECRET;
if (!secret) throw new Error("OX_WEBHOOK_SECRET is required");

const valid = await verifyWebhook({
  secret,
  body: rawBody,
  headers: {
    id: request.headers.get("webhook-id") ?? "",
    timestamp: request.headers.get("webhook-timestamp") ?? "",
    signature: request.headers.get("webhook-signature") ?? "",
  },
});

if (!valid) return new Response("Invalid signature", { status: 401 });
const event = JSON.parse(rawBody);
```

## 13. Errors, cancellation, and compatibility

### Handle typed SDK errors

`OxApiError` exposes the HTTP status, bounded server error code/message, and request ID. `OxTimeoutError` exposes the configured timeout, and `OxConnectionError` represents transport or invalid-response failures.

```ts
import { OxApiError, OxTimeoutError } from "@ox/sdk";

try {
  await ox.models.list({ signal: request.signal, timeoutMs: 30_000 });
} catch (error) {
  if (error instanceof OxApiError) {
    reportRequestFailure(error.status, error.code, error.requestId);
  } else if (error instanceof OxTimeoutError) {
    reportTimeout(error.timeoutMs);
  }
}
```

### Respond by status and close code

HTTP `401` means the permanent server key is missing/invalid. `413` means the mint body is too large. `429` means the project mint quota was exceeded. `503` means the model, durable state, or admitted warm capacity is unavailable. WebSocket `1000` is a clean close; `1008` is policy/auth; `1009` is too large; `1011` is a runtime/relay failure. Retry only before a session is admitted, with capped exponential jitter, and mint a new secret for each attempt.

### Maintain compatibility

Integrations created before the Ox rename can continue importing `@pandan/realtime`; it re-exports `@ox/sdk`, `@ox/sdk/browser`, and `@ox/sdk/livekit`. New integrations should use the Ox package names and the current public preview origin, `https://www.amerint.co`.

## 14. Authentication and API keys

### Send a bearer token

Every authenticated customer API request sends the project credential in the HTTP `Authorization` header. The Realtime readiness endpoint is public so applications can inspect availability before exposing a voice action. Put the current preview origin and credential in a protected server environment, then construct URLs from the origin rather than repeating it throughout the application.

```bash
export OX_API_BASE_URL=https://www.amerint.co
export OX_API_KEY='[YOUR_OX_API_KEY]'
```

```http
Authorization: Bearer [YOUR_OX_API_KEY]
Accept: application/json
```

Never put a permanent credential in browser code, a public environment variable, a mobile binary, a URL, source control, analytics, or logs. Browser voice applications mint a short-lived Realtime descriptor on their own authenticated server and return only that descriptor to the client.

### Understand credential boundaries

Ox uses separate credentials for customer inference, project-scoped administration, and operator control-plane work.

| Credential | Intended use | Allowed location |
| --- | --- | --- |
| Project API key | Enabled inference and Realtime operations for one project | Trusted server only |
| Project management key | Project resources allowed by `project:read`, `project:write`, or `project:keys` | Trusted server only |
| Control-plane token | Cross-project provisioning and operator workflows | Isolated operator service only |
| Realtime session secret | One short-lived browser connection | Returned to one authenticated client, never persisted |

A managed project key cannot access another project or grant scopes it does not possess. A Realtime-only key cannot use project-management routes. Raw keys and webhook signing secrets are returned only at creation time; store them immediately in a server-side secret manager.

### Diagnose authentication failures

An HTTP `401` means the bearer credential is absent, malformed, expired, revoked, or unknown. An HTTP `403` means the credential is valid but lacks the required project scope. Do not retry either response automatically. Rotate a compromised key by issuing its replacement, updating the server secret, verifying traffic with the new key, and then revoking the old key.

## 15. HTTP API quickstart

### List the models enabled for your project

Start with `GET /v1/models`. This verifies the API origin and credential without spending an inference request.

```bash
curl --fail-with-body "$OX_API_BASE_URL/v1/models" \
  --header "Authorization: Bearer $OX_API_KEY" \
  --header "Accept: application/json"
```

The response is an OpenAI-compatible list. Choose an ID returned for your project rather than copying a model name from a model card or another account.

```json
{
  "object": "list",
  "data": [{ "id": "MODEL_ID", "object": "model" }]
}
```

### Create a chat completion

Replace `MODEL_ID` with an ID from the list response. The smallest useful request contains a model and one user message.

```bash
curl --fail-with-body "$OX_API_BASE_URL/v1/chat/completions" \
  --header "Authorization: Bearer $OX_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"model":"MODEL_ID","messages":[{"role":"user","content":"Describe what you can help with in one sentence."}],"temperature":0}'
```

Read the answer from `choices[0].message.content`. Preserve the `x-request-id` response header in server logs when reporting a failed request, but never log the authorization header or media bytes.

### Use an OpenAI-compatible client

Existing OpenAI client libraries can target Ox by setting their base URL to the preview origin plus `/v1`. Ox-specific reusable-media helpers and Realtime transport helpers remain in `@ox/sdk`.

```bash
npm install openai
# Or: python -m pip install openai
```

```js
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.OX_API_KEY,
  baseURL: (process.env.OX_API_BASE_URL ?? "https://www.amerint.co") + "/v1",
});

const models = await client.models.list();
const model = models.data[0];
if (!model) throw new Error("No models are enabled for this project");

const completion = await client.chat.completions.create({
  model: model.id,
  messages: [{ role: "user", content: "Reply with a short hello." }],
});
console.log(completion.choices[0]?.message.content);
```

```python
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OX_API_KEY"],
    base_url=os.getenv("OX_API_BASE_URL", "https://www.amerint.co") + "/v1",
)

models = client.models.list()
if not models.data:
    raise RuntimeError("No models are enabled for this project")

completion = client.chat.completions.create(
    model=models.data[0].id,
    messages=[{"role": "user", "content": "Reply with a short hello."}],
)
print(completion.choices[0].message.content)
```

## 16. Chat completions reference

### Build a request

`POST /v1/chat/completions` follows the OpenAI chat-completions shape and adds top-level `media_ids` for content uploaded through the Ox Media API.

| Field | Type | Notes |
| --- | --- | --- |
| `model` | string | Required. Use an ID returned by `GET /v1/models`. |
| `messages` | array | Required and non-empty. Roles are `developer`, `system`, `user`, `assistant`, or `tool`. |
| `media_ids` | string[] | Optional Ox extension for previously uploaded images or videos. |
| `temperature`, `top_p` | number | Optional sampling controls. Prefer changing one at a time. |
| `max_tokens`, `max_completion_tokens` | number | Optional output bounds supported for client compatibility. |
| `stop` | string or string[] | Optional stop sequence or sequences. |
| `tools`, `tool_choice` | object[] / string or object | Optional OpenAI-compatible function tools. |
| `response_format` | object | Optional structured-response configuration supported by the selected model. |
| `seed`, `user` | number / string | Optional compatibility fields. |
| `stream` | boolean | Set to `true` for server-sent events. |
| `stream_options.include_usage` | boolean | Requests a final streaming usage block when supported. |

Messages can contain plain text or multimodal content parts. Remote image and video URLs must be reachable by the inference service; use the Media API for private or repeatedly queried content.

```json
{
  "model": "MODEL_ID",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "video_url", "video_url": { "url": "https://example.com/clip.mp4" } },
      { "type": "text", "text": "What happens after the door opens?" }
    ]
  }],
  "temperature": 0
}
```

### Stream server-sent events

Set `stream: true` and read each `data:` event until `[DONE]`. Use `curl --no-buffer` during development so chunks appear as they arrive.

```bash
curl --fail-with-body --no-buffer "$OX_API_BASE_URL/v1/chat/completions" \
  --header "Authorization: Bearer $OX_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"model":"MODEL_ID","messages":[{"role":"user","content":"Summarize the scene."}],"stream":true,"stream_options":{"include_usage":true}}'
```

Each JSON event uses `object: "chat.completion.chunk"`; append `choices[0].delta.content` and stop on `[DONE]`. Cancel the upstream request when the downstream user disconnects.

### Use tools safely

Function definitions use the OpenAI-compatible `tools` and `tool_choice` fields. Treat generated arguments as untrusted input: parse JSON, validate it against the function schema, authorize the signed-in user again, apply timeouts, and return bounded results. Never let a model choose credentials, tenant IDs, or authorization policy.

## 17. Reusable media API

### Upload image or video bytes

Send raw bytes to `POST /v1/media` with an image or video `Content-Type`. The maximum object size is 256 MiB. The response returns a content-addressed ID; uploading identical content may return the same ID with `created: false`.

```bash
curl --fail-with-body "$OX_API_BASE_URL/v1/media" \
  --request POST \
  --header "Authorization: Bearer $OX_API_KEY" \
  --header "Content-Type: video/mp4" \
  --data-binary @inspection.mp4
```

```json
{
  "id": "media_EXAMPLE",
  "object": "media",
  "content_type": "video/mp4",
  "bytes": 1048576,
  "created": true
}
```

### Reuse a media ID

Pass one or more IDs at the top level of a chat-completion request. This avoids uploading the same private object before each question.

```json
{
  "model": "MODEL_ID",
  "media_ids": ["media_EXAMPLE"],
  "messages": [{ "role": "user", "content": "Identify the first safety issue." }],
  "temperature": 0
}
```

Check availability without downloading content by sending `HEAD /v1/media/{media_id}`. A `2xx` response means the object exists; `404` means it is unavailable to the current API path.

```bash
curl --fail-with-body --head "$OX_API_BASE_URL/v1/media/media_EXAMPLE" \
  --header "Authorization: Bearer $OX_API_KEY"
```

## 18. Quickstart

### Set the Vision API origin

The live-stream API is available under the versioned preview namespace on the same portal origin. Use a Hallikar preview API key supplied with Vision API access. The stable Ox inference and Realtime APIs remain under `/v1`; do not interchange their credentials with a Hallikar preview key.

```bash
export OX_VISION_API_BASE_URL=https://www.amerint.co/v1beta
export HALLIKAR_API_KEY='[YOUR_HALLIKAR_API_KEY]'
```

### Verify the model catalog

The model catalog is public and is the cheapest end-to-end connectivity check. Choose only a model whose `status` is `ready`.

```bash
curl --fail-with-body "$OX_VISION_API_BASE_URL/models" \
  --header "Accept: application/json"
```

### Create a live stream

An empty JSON object creates a managed LiveKit stream. The response contains the stream ID and a short-lived, publish-only LiveKit token. Publish video with the LiveKit SDK, wait until stream status reports a first frame, and then reference that stream from Chat Completion.

```bash
curl --fail-with-body "$OX_VISION_API_BASE_URL/streams" \
  --request POST \
  --header "Authorization: Bearer $HALLIKAR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{}'
```

### Ask about the latest frame

Replace `STREAM_ID` and `MODEL_ID` with values returned by the API. The `hallikar://` URL is resolved inside Hallikar and is never fetched from the public internet.

```bash
curl --fail-with-body "$OX_VISION_API_BASE_URL/chat/completions" \
  --request POST \
  --header "Authorization: Bearer $HALLIKAR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"model":"MODEL_ID","messages":[{"role":"user","content":[{"type":"text","text":"What is happening now?"},{"type":"image_url","image_url":{"url":"hallikar://streams/STREAM_ID?frame_index=-1"}}]}],"max_completion_tokens":128}'
```

## 19. The Stream

### Understand the lifecycle

A stream is an owned, leased live-video resource. `POST /streams` creates it, `GET /streams/{id}` reports frame availability, `POST /streams/{id}/keepalive` renews its lease, and `DELETE /streams/{id}` ends it. There is no public pause, resume, list-all, or `/infer` endpoint in the current contract.

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/streams` | Create a managed, pull, or customer-LiveKit stream. |
| `GET` | `/streams/{id}` | Read state, frame counters, timing, source type, and lease expiry. |
| `POST` | `/streams/{id}/keepalive` | Renew the lease and refresh any managed publish token. |
| `POST` | `/streams/{id}/viewer-token` | Mint a short-lived, subscribe-only LiveKit token. |
| `GET` | `/streams/{id}/preview` | Return a bounded frame or clip preview as JPEG data URLs. |
| `DELETE` | `/streams/{id}` | End the stream and release live resources. |

### Wait for usable frames

Poll status until `first_frame_at_ms` is non-null. `recent_fps` describes the current capture cadence, `retained_frame_count` counts frames still available for inference, and `evicted_frame_count` counts frames that left the ten-minute history window. Frame indices are lifetime indices, not positions inside the retained window.

```bash
curl --fail-with-body "$OX_VISION_API_BASE_URL/streams/STREAM_ID" \
  --header "Authorization: Bearer $HALLIKAR_API_KEY"
```

### Renew and delete explicitly

The current lease is five minutes. Renew around every four minutes and use the returned replacement publish token for managed LiveKit streams. Delete a stream when capture ends; do not wait for expiry as a normal cleanup path.

```bash
curl --fail-with-body "$OX_VISION_API_BASE_URL/streams/STREAM_ID/keepalive" \
  --request POST \
  --header "Authorization: Bearer $HALLIKAR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{}'

curl --fail-with-body "$OX_VISION_API_BASE_URL/streams/STREAM_ID" \
  --request DELETE \
  --header "Authorization: Bearer $HALLIKAR_API_KEY"
```

## 20. Camera sources

### Choose a source type

The create-stream body accepts one of four current camera paths. Omitting `source` creates a Hallikar-managed LiveKit room and returns publish credentials. Pull sources connect to RTSP or HLS. `livekit_room` subscribes to one camera or screen-share track in a customer-owned LiveKit room.

| Source | Request shape | Publish credentials returned |
| --- | --- | --- |
| Managed LiveKit | `{}` | Yes |
| RTSP camera | `{"source":{"type":"rtsp","url":"rtsp://..."}}` | No |
| HLS stream | `{"source":{"type":"hls","url":"https://...m3u8"}}` | No |
| Existing LiveKit room | `{"source":{"type":"livekit_room",...}}` | No |

### Connect an RTSP or HLS source

Hallikar validates the URL and probes reachability before accepting a pull source. A source URL may contain camera credentials, so the API never echoes it in responses. Store the URL as a secret and send it only from trusted server code.

```bash
curl --fail-with-body "$OX_VISION_API_BASE_URL/streams" \
  --request POST \
  --header "Authorization: Bearer $HALLIKAR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"source":{"type":"hls","url":"https://camera.example/live/index.m3u8"}}'
```

### Subscribe to a customer LiveKit room

The token must remain valid through the next lease interval and must allow the integration identity to join the named room. Set `track_source` to `camera` or `screen_share`. On keepalive, provide a refreshed `source_token` before the old token expires.

```json
{
  "source": {
    "type": "livekit_room",
    "url": "wss://your-project.livekit.cloud",
    "token": "[SHORT_LIVED_LIVEKIT_TOKEN]",
    "participant_identity": "camera-17",
    "track_source": "camera"
  }
}
```

## 21. Chat Completion

### Send text, images, video, or live stream media

`POST /chat/completions` follows the OpenAI Chat Completions request and response shape. Text-only calls work. Image and video content can use reachable HTTPS URLs, bounded data URLs, or `hallikar://streams/{id}` references owned by the same API identity.

```json
{
  "model": "MODEL_ID",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "Describe changes during the last ten seconds." },
      { "type": "video_url", "video_url": { "url": "hallikar://streams/STREAM_ID?start_offset_ms=-10000&end_offset_ms=0&max_fps=2" } }
    ]
  }],
  "max_completion_tokens": 256
}
```

### Select an exact frame or time range

Use `frame_index=-1` for the latest retained frame, `timestamp_ms` for a time measured from the first captured frame, or negative `offset_ms` for a position relative to the current stream time. Video requests accept start and end frame indices, timestamps, or offsets plus an optional `max_fps` sampling bound.

### Stream server-sent events

Set `stream: true` and read each `data:` record until `[DONE]`. A final usage event is best effort when `stream_options.include_usage` is true. Cancel the upstream request when the downstream client disconnects.

## 22. Models

### List live availability

`GET /models` returns the current model catalog in an OpenAI-compatible list. This endpoint is public. Availability is runtime state: use only IDs whose `status` is `ready`, and check again after a capacity or provider error.

```bash
curl --fail-with-body "$OX_VISION_API_BASE_URL/models" \
  --header "Accept: application/json"
```

```json
{
  "object": "list",
  "data": [{
    "id": "Qwen/Qwen3-VL-30B-A3B-Instruct",
    "object": "model",
    "owned_by": "hallikar",
    "status": "ready"
  }]
}
```

### Treat the catalog as admission state

A model appearing in repository configuration or a model card does not prove it is currently admitted. The live catalog is authoritative for request selection. A later request can still fail because availability changes between discovery and inference, so handle `429`, `502`, `503`, and `504` explicitly.

## 23. Glossary

### Core terms

| Term | Meaning |
| --- | --- |
| Stream | An owned, leased live-video resource with a bounded retained-frame history. |
| Source | The camera path feeding a stream: managed LiveKit, RTSP, HLS, or a customer LiveKit room. |
| Publish token | A short-lived LiveKit credential that can publish media to one managed stream. It cannot call the HTTP API. |
| Viewer token | A short-lived, subscribe-only LiveKit credential for watching a managed or pull stream. |
| Stream time | Milliseconds elapsed since the first captured frame, not since the stream resource was created. |
| Frame index | A zero-based lifetime frame number. Negative indices count backward from the latest retained frame. |
| Retained window | The bounded history still available for inference; currently ten minutes. |
| `hallikar://` reference | A private media selector resolved by Hallikar inside a chat-completion request. |
| Keepalive | A lease renewal that prevents an active stream from expiring. |
| Region pin | The optional `X-Hallikar-Region` header used to keep a stream and its follow-up calls in one region. |

## 24. Best practices

### Build for bounded live state

Create a stream immediately before capture, wait for `first_frame_at_ms`, renew the lease before four minutes, and delete it in a `finally` block. Treat publish, viewer, and customer-room tokens as credentials. Never log them, place them in URLs, or persist them longer than their job.

### Use the smallest media selection

Use one image frame when the question is about the current scene. Request a video segment only when temporal order matters, and set `max_fps` to the lowest cadence that preserves the event. This reduces latency, token usage, and provider-side media processing.

### Preserve ownership and region

Use the same bearer key for stream creation, status, preview, deletion, and any chat request that references the stream. A resource owned by another identity returns `404` rather than revealing its existence. If creation uses `X-Hallikar-Region`, send the same value on every lifecycle call.

### Handle failure by class

Do not retry `401`, `403`, or validation `422` responses without changing the request. Retry transient `429`, `502`, `503`, and `504` responses with capped exponential backoff and jitter. A failed keepalive near expiry requires a new stream; do not assume the old publish token remains valid.

## 25. LiveKit

### Publish a browser camera

Create a managed stream on your server and return only its `publish` descriptor to the signed-in browser. The browser joins the room and publishes video with the official LiveKit client. Never send the Hallikar API key to browser code.

```ts
import { Room, createLocalTracks } from "livekit-client";

const room = new Room();
await room.connect(stream.publish.url, stream.publish.token);
const [videoTrack] = await createLocalTracks({ video: true, audio: false });
await room.localParticipant.publishTrack(videoTrack);
```

### Watch what Hallikar receives

Call `POST /streams/{id}/viewer-token` on your server and return the short-lived `view` descriptor to an authorized viewer. Viewer tokens cannot publish media or data. This endpoint returns `409` for `livekit_room` sources because the customer-owned room controls its own subscribers.

### Refresh without interrupting capture

Call keepalive before the lease expires. For a managed stream, pass the replacement publish token to LiveKit with `room.updateToken(newToken)`. For a customer-owned LiveKit source, send a refreshed join token in `{"source_token":"..."}`; pull sources do not use either token.