Ox TypeScript SDK and Realtime API

Before you begin

You need Node.js 20 or newer, an Ox API key, and the private-preview package artifact supplied with your access. Keep the API key in trusted server code. Never place it in browser JavaScript, mobile binaries, source control, URLs, or logs. The current public preview API origin is https://www.amerint.co; set it explicitly until the permanent Ox domain is assigned.

Install the private preview

The package is not published to npm yet. Install the supplied artifact directly:

npm install /absolute/path/to/ox-sdk-0.1.0.tgz

Create a server client

Create one client and reuse it. The example reads OX_API_KEY from the process environment and fails before making a request when the variable is missing.

import { Ox } from "@ox/sdk";

const apiKey = process.env.OX_API_KEY;
if (!apiKey) throw new Error("OX_API_KEY is required");

const ox = new Ox({
  apiKey,
  baseUrl: process.env.OX_API_BASE_URL ?? "https://www.amerint.co",
});

const models = await ox.models.list();
if (models.data.length === 0) {
  throw new Error("No Ox models are enabled for this project");
}

console.log("Connected to Ox");
console.table(models.data.map(({ id }) => ({ id })));

Save this as quickstart.mjs. Load the key from your secret manager or a protected local environment file, then run:

node --env-file=.env quickstart.mjs

Or download the Node quickstart and run the same command after installing your private-preview SDK artifact.

Verify the connection

A successful run prints Connected to Ox followed by the model IDs enabled for your project. A 401 means the server credential is missing or invalid. An empty model list means the project needs model access before it can run inference.

The SDK rejects permanent credentials in browser runtimes, uses a ten-minute request timeout, supports per-request cancellation and timeouts, and does not retry inference or session creation automatically.

Choose an enabled model

List the models enabled for the current project before selecting one. Upload image or video bytes once when several questions will reuse the same content.

import { readFile } from "node:fs/promises";

const models = await ox.models.list();
const model = models.data[0];
if (!model) throw new Error("No Ox models are enabled for this project");

const media = await ox.media.create(await readFile("inspection.mp4"), {
  contentType: "video/mp4",
});

const completion = await ox.chat.completions.create({
  model: model.id,
  media_ids: [media.id],
  messages: [{ role: "user", content: "Describe the first safety issue." }],
  temperature: 0,
});

console.log(completion.choices[0]?.message.content);

Stream a response

For incremental output, set stream: true and consume the returned async iterable. Abort the request when the downstream client disconnects.

const stream = await ox.chat.completions.create({
  model: "Qwen/Qwen3-VL-30B-A3B-Instruct",
  messages: [{ role: "user", content: "Summarize the scene." }],
  stream: true,
  stream_options: { include_usage: true },
}, { signal: request.signal, timeoutMs: 60_000 });

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? "");
}

Media uploads accept Blob, ArrayBuffer, or an array-buffer view and are limited to 256 MiB. Use ox.media.exists(media.id) to check a content-addressed object without downloading it.

Keep the permanent key on the server

Create sessions only in authenticated server routes. Return the descriptor with Cache-Control: no-store; never return the permanent API key. Select ox/voice-agent for a conversational agent that can call customer functions while the microphone remains live.

const session = await ox.realtime.sessions.create({
  model: "ox/voice-agent",
  transport: "websocket",
  session: {
    instructions: "Help customers schedule service. Confirm before booking.",
    endpointing: { eagerness: "medium" },
  },
});

return Response.json(session, {
  headers: { "Cache-Control": "no-store" },
});

Return a short-lived descriptor

The lower-level HTTP contract is POST https://www.amerint.co/v1/realtime/sessions with Authorization: Bearer $OX_API_KEY. A returned session secret is short-lived, single-use, and still sensitive.

Connect from a user gesture

import { connectBrowserRealtime } from "@ox/sdk/browser";

const call = await connectBrowserRealtime({
  session,
  workletBaseUrl: "https://www.amerint.co/worklets",
  configuration: {
      instructions: "Help customers schedule service. Confirm before booking.",
      language: "en",
      voice: "Ryan",
      endpointing: { eagerness: "medium" },
      reasoning: { model: "Qwen/Qwen3-Next-80B-A3B-Instruct", temperature: 0.3, max_output_tokens: 512 },
      tools: [{
        type: "function",
        name: "find_appointments",
        description: "Find open appointment times.",
        parameters: { type: "object", properties: { date: { type: "string" } }, required: ["date"] },
      }],
  },
  onToolCall: async ({ name, arguments: json }) => {
    const args = JSON.parse(json);
    if (name === "find_appointments") return findAppointmentsForSignedInUser(args);
    throw new Error("Unknown tool");
  },
});

call.on("event", console.log);
await call.start(); // microphone permission must follow a user gesture

Use the hosted preview bundle

The hosted private-preview ESM bundle is also available at https://www.amerint.co/voice-sdk/ox-realtime.js for applications that cannot install the package yet.

Understand the runtime boundary

The SDK owns AudioWorklet capture, windowed-sinc resampling, exact 80 ms framing, capture-age and socket backpressure rejection, adaptive output jitter buffering, and local plus server-confirmed barge-in. Binary client frames are exactly 2560 bytes of 16000 Hz mono PCM16. Binary server frames are 24000 Hz mono PCM16. JSON and base64 audio are not accepted on the public hop.

Tool execution remains customer-owned. The model emits response.function_call_arguments.done; the SDK invokes the allow-listed handler, submits conversation.item.create, and sends response.create so the agent can continue the same turn. Microphone capture stays live while the tool is pending. Results time out after 15 seconds and are limited to 65536 bytes.

Connect a browser with WebRTC

Request WebRTC on the server, then pass only the returned descriptor to the browser. The adapter uses the official LiveKit client and enables microphone publishing by default.

// Trusted server
const session = await ox.realtime.sessions.create({
  model: "ox/voice-agent",
  transport: "webrtc",
  identity: signedInUser.id,
});

// Browser, called from a user gesture
import { connectLiveKitRealtime } from "@ox/sdk/browser";

const room = await connectLiveKitRealtime(session);

Import the same adapter from @ox/sdk/livekit when a dedicated subpath is preferable. Private-preview applications without package installation can import it from https://www.amerint.co/voice-sdk/ox-livekit.js. Pass publishMicrophone: false if the application will publish its own audio track. The participant token is scoped to one room and microphone publishing.

Place an outbound SIP call

Outbound telephony uses the same room and bridge. The server validates E.164 destinations and dispatches the model participant before dialing.

const call = await ox.realtime.calls.create({ to: "+14155550100" });

Server events

The relay converts provider-specific speech and transcript names into this single dialect. Customers should not branch on provider names.

Client events

State transitions

Normal state flow: minted → connected → ready → listening → thinking → speaking → ready. Interruption moves speaking → listening; a tool call moves thinking → tool_wait → thinking. error and closed are terminal for the current connection.

Interruptions and buffered audio

Stop local playout as soon as the capture worklet detects probable speech. Confirm the interruption on input.speech_started, discard all buffered audio from the cancelled generation, and never replay it. The SDK drops stale capture frames rather than building conversational lag. The relay closes a client that exceeds 1048576 buffered bytes.

Reconnect safely

A session secret expires after 900 seconds, is single-use, and cannot reconnect after it has been claimed. Mint a new session and restore only customer-approved text history. Never replay microphone audio or a partially executed tool call automatically.

Read the readiness response

GET /v1/realtime/health reports configuration readiness; it is not an active upstream-provider probe. The top-level status describes customer traffic admission: ready means the customer API can mint sessions, degraded means only the browser preview is available, and unavailable means realtime sessions cannot be established.

Gate only the features you use

For a browser voice agent that calls customer tools, gate only on the fields that path requires:

const health = await ox.realtime.health.retrieve();

if (!health.customer_api_ready ||
    !health.capabilities.voice_agent ||
    !health.capabilities.customer_tools) {
  throw new Error(health.summary);
}

Understand limitations

Missing optional features appear in limitations. In particular, durable_state_unavailable disables server-persisted agent configuration, durable session records, and control-plane quotas; it does not disable browser-configured voice, customer tools, or core realtime traffic. Require durable_state or capabilities.server_persisted_agent_configuration only when your application uses those features.

Validate the event contract

Run npm run voice:conformance against the versioned normal-turn, interruption, binary-audio, error, expiry, and reconnect fixtures. Attach ProtocolInspector to the SDK to record monotonic timing and state transitions. It redacts keys, authorization, cookies, session secrets, tokens, audio bodies, and function outputs before export. Do not use raw WebSocket logging in production.

Hard protocol limits

Hard limits: 120-second sessions, 65536-byte control frames, 262144-byte pre-admission queue, 32 tools, 8192-byte schema per tool, and 30 session mints per project per minute when durable production state is enabled. Account-specific limits returned by the control plane override preview defaults.

Measured warm latency

Cold-start and capacity scope

Use the hardened Next.js example

The Next.js production example authenticates the application user, requires same-origin Fetch Metadata, validates a double-submit CSRF token, applies a per-tenant quota before calling Ox, returns no-store JSON, and logs only request/session identifiers. Replace its explicit auth and quota adapters with your own durable implementations. Never log Authorization, cookies, CSRF values, session descriptors, tool arguments/results, or audio.

Create project credentials

Project administration requires a separate control-plane client and credential. Never initialize it with a customer API key or expose its token to browser code.

import { OxControlPlane } from "@ox/sdk";

const token = process.env.OX_CONTROL_PLANE_ADMIN_TOKEN;
if (!token) throw new Error("OX_CONTROL_PLANE_ADMIN_TOKEN is required");

const controlPlane = new OxControlPlane({ token });
const project = await controlPlane.projects.create({
  name: "Acme Voice",
  contactEmail: "developer@example.com",
});
const issued = await controlPlane.apiKeys.create(project.id, {
  name: "Project automation",
  scopes: ["project:read", "project:write", "project:keys"],
});

await deliverKeyOnceToAuthorizedCustomer(issued.value);

issued.value is returned once. Do not log it, store it in source control, or send it to an unauthorized client. projects supports list, create, retrieve, suspend, and reactivate. apiKeys supports list, create, and revoke.

Use a project-scoped management key

Customers never receive the global control-plane token. Bind the scoped key to its project in trusted server code:

import { OxProject } from "@ox/sdk";

const project = new OxProject({
  projectId: process.env.OX_PROJECT_ID!,
  apiKey: process.env.OX_PROJECT_MANAGEMENT_KEY!,
});

const agents = await project.agents.list();
const sessions = await project.sessions.list({ limit: 25 });

Management keys require explicit project:read, project:write, or project:keys scopes, cannot cross project boundaries, and cannot issue keys with scopes they do not possess. Realtime-only keys retain only realtime and cannot access management routes.

Register webhooks and inspect usage

Register realtime-session and Hallikar incident webhooks and store the returned signing secret once. Verify delivery signatures against the unmodified request body before parsing JSON.

const endpoint = await controlPlane.webhooks.create(projectId, {
  url: "https://example.com/webhooks/ox",
  events: ["realtime.session.completed", "realtime.session.failed", "hallikar.incident.opened"],
});
await storeWebhookSecret(endpoint.secret);

const usage = await controlPlane.usage.list(projectId, { days: 30 });
const audit = await controlPlane.auditLogs.list(projectId, { limit: 50 });
const invitation = await controlPlane.dashboardInvitations.create(projectId);

Webhook endpoints support list, create, and disable. Dashboard invitation tokens are short-lived and should be delivered only to the intended project operator.

Manage agents and inspect session history

Agents are reusable, project-scoped configurations. Creating an agent produces version 1; supplying a new configuration through agents.update appends an immutable version. Name-only updates do not create a configuration version.

const agent = await controlPlane.agents.create(projectId, {
  name: "Support",
  configuration: {
    instructions: "Resolve account questions and confirm before changing data.",
    voice: "Ryan",
    endpointing: { eagerness: "medium" },
  },
});

const updated = await controlPlane.agents.update(projectId, agent.id, {
  configuration: {
    ...agent.versions.at(-1)?.configuration,
    instructions: "Resolve account questions. Escalate refund requests.",
  },
});

const sessions = await controlPlane.sessions.list(projectId, {
  limit: 25,
  status: "completed",
  transport: "websocket",
});
const latest = sessions.data[0]
  ? await controlPlane.sessions.retrieve(projectId, sessions.data[0].id)
  : undefined;

Use agents.list, agents.retrieve, and agents.archive for lifecycle management. Archived agents remain available with agents.list(projectId, { includeArchived: true }). Session records contain project metadata, configuration, routing, trace identifiers, and metered audio usage when those fields have been recorded; they never contain the one-time session secret.

Verify webhook signatures

import { verifyWebhook } from "@ox/sdk";

const rawBody = await request.text();
const secret = process.env.OX_WEBHOOK_SECRET;
if (!secret) throw new Error("OX_WEBHOOK_SECRET is required");

const valid = await verifyWebhook({
  secret,
  body: rawBody,
  headers: {
    id: request.headers.get("webhook-id") ?? "",
    timestamp: request.headers.get("webhook-timestamp") ?? "",
    signature: request.headers.get("webhook-signature") ?? "",
  },
});

if (!valid) return new Response("Invalid signature", { status: 401 });
const event = JSON.parse(rawBody);

Handle typed SDK errors

OxApiError exposes the HTTP status, bounded server error code/message, and request ID. OxTimeoutError exposes the configured timeout, and OxConnectionError represents transport or invalid-response failures.

import { OxApiError, OxTimeoutError } from "@ox/sdk";

try {
  await ox.models.list({ signal: request.signal, timeoutMs: 30_000 });
} catch (error) {
  if (error instanceof OxApiError) {
    reportRequestFailure(error.status, error.code, error.requestId);
  } else if (error instanceof OxTimeoutError) {
    reportTimeout(error.timeoutMs);
  }
}

Respond by status and close code

HTTP 401 means the permanent server key is missing/invalid. 413 means the mint body is too large. 429 means the project mint quota was exceeded. 503 means the model, durable state, or admitted warm capacity is unavailable. WebSocket 1000 is a clean close; 1008 is policy/auth; 1009 is too large; 1011 is a runtime/relay failure. Retry only before a session is admitted, with capped exponential jitter, and mint a new secret for each attempt.

Maintain compatibility

Integrations created before the Ox rename can continue importing @pandan/realtime; it re-exports @ox/sdk, @ox/sdk/browser, and @ox/sdk/livekit. New integrations should use the Ox package names and the current public preview origin, https://www.amerint.co.

Send a bearer token

Every authenticated customer API request sends the project credential in the HTTP Authorization header. The Realtime readiness endpoint is public so applications can inspect availability before exposing a voice action. Put the current preview origin and credential in a protected server environment, then construct URLs from the origin rather than repeating it throughout the application.

export OX_API_BASE_URL=https://www.amerint.co
export OX_API_KEY='[YOUR_OX_API_KEY]'

Authorization: Bearer [YOUR_OX_API_KEY]
Accept: application/json

Never put a permanent credential in browser code, a public environment variable, a mobile binary, a URL, source control, analytics, or logs. Browser voice applications mint a short-lived Realtime descriptor on their own authenticated server and return only that descriptor to the client.

Understand credential boundaries

Ox uses separate credentials for customer inference, project-scoped administration, and operator control-plane work.

A managed project key cannot access another project or grant scopes it does not possess. A Realtime-only key cannot use project-management routes. Raw keys and webhook signing secrets are returned only at creation time; store them immediately in a server-side secret manager.

Diagnose authentication failures

An HTTP 401 means the bearer credential is absent, malformed, expired, revoked, or unknown. An HTTP 403 means the credential is valid but lacks the required project scope. Do not retry either response automatically. Rotate a compromised key by issuing its replacement, updating the server secret, verifying traffic with the new key, and then revoking the old key.

List the models enabled for your project

Start with GET /v1/models. This verifies the API origin and credential without spending an inference request.

curl --fail-with-body "$OX_API_BASE_URL/v1/models" \
  --header "Authorization: Bearer $OX_API_KEY" \
  --header "Accept: application/json"

The response is an OpenAI-compatible list. Choose an ID returned for your project rather than copying a model name from a model card or another account.

{
  "object": "list",
  "data": [{ "id": "MODEL_ID", "object": "model" }]
}

Create a chat completion

Replace MODEL_ID with an ID from the list response. The smallest useful request contains a model and one user message.

curl --fail-with-body "$OX_API_BASE_URL/v1/chat/completions" \
  --header "Authorization: Bearer $OX_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"model":"MODEL_ID","messages":[{"role":"user","content":"Describe what you can help with in one sentence."}],"temperature":0}'

Read the answer from choices[0].message.content. Preserve the x-request-id response header in server logs when reporting a failed request, but never log the authorization header or media bytes.

Use an OpenAI-compatible client

Existing OpenAI client libraries can target Ox by setting their base URL to the preview origin plus /v1. Ox-specific reusable-media helpers and Realtime transport helpers remain in @ox/sdk.

npm install openai
# Or: python -m pip install openai

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.OX_API_KEY,
  baseURL: (process.env.OX_API_BASE_URL ?? "https://www.amerint.co") + "/v1",
});

const models = await client.models.list();
const model = models.data[0];
if (!model) throw new Error("No models are enabled for this project");

const completion = await client.chat.completions.create({
  model: model.id,
  messages: [{ role: "user", content: "Reply with a short hello." }],
});
console.log(completion.choices[0]?.message.content);

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OX_API_KEY"],
    base_url=os.getenv("OX_API_BASE_URL", "https://www.amerint.co") + "/v1",
)

models = client.models.list()
if not models.data:
    raise RuntimeError("No models are enabled for this project")

completion = client.chat.completions.create(
    model=models.data[0].id,
    messages=[{"role": "user", "content": "Reply with a short hello."}],
)
print(completion.choices[0].message.content)

Build a request

POST /v1/chat/completions follows the OpenAI chat-completions shape and adds top-level media_ids for content uploaded through the Ox Media API.

Messages can contain plain text or multimodal content parts. Remote image and video URLs must be reachable by the inference service; use the Media API for private or repeatedly queried content.

{
  "model": "MODEL_ID",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "video_url", "video_url": { "url": "https://example.com/clip.mp4" } },
      { "type": "text", "text": "What happens after the door opens?" }
    ]
  }],
  "temperature": 0
}

Stream server-sent events

Set stream: true and read each data: event until [DONE]. Use curl --no-buffer during development so chunks appear as they arrive.

curl --fail-with-body --no-buffer "$OX_API_BASE_URL/v1/chat/completions" \
  --header "Authorization: Bearer $OX_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"model":"MODEL_ID","messages":[{"role":"user","content":"Summarize the scene."}],"stream":true,"stream_options":{"include_usage":true}}'

Each JSON event uses object: "chat.completion.chunk"; append choices[0].delta.content and stop on [DONE]. Cancel the upstream request when the downstream user disconnects.

Use tools safely

Function definitions use the OpenAI-compatible tools and tool_choice fields. Treat generated arguments as untrusted input: parse JSON, validate it against the function schema, authorize the signed-in user again, apply timeouts, and return bounded results. Never let a model choose credentials, tenant IDs, or authorization policy.

Upload image or video bytes

Send raw bytes to POST /v1/media with an image or video Content-Type. The maximum object size is 256 MiB. The response returns a content-addressed ID; uploading identical content may return the same ID with created: false.

curl --fail-with-body "$OX_API_BASE_URL/v1/media" \
  --request POST \
  --header "Authorization: Bearer $OX_API_KEY" \
  --header "Content-Type: video/mp4" \
  --data-binary @inspection.mp4

{
  "id": "media_EXAMPLE",
  "object": "media",
  "content_type": "video/mp4",
  "bytes": 1048576,
  "created": true
}

Reuse a media ID

Pass one or more IDs at the top level of a chat-completion request. This avoids uploading the same private object before each question.

{
  "model": "MODEL_ID",
  "media_ids": ["media_EXAMPLE"],
  "messages": [{ "role": "user", "content": "Identify the first safety issue." }],
  "temperature": 0
}

Check availability without downloading content by sending HEAD /v1/media/{media_id}. A 2xx response means the object exists; 404 means it is unavailable to the current API path.

curl --fail-with-body --head "$OX_API_BASE_URL/v1/media/media_EXAMPLE" \
  --header "Authorization: Bearer $OX_API_KEY"

Set the Vision API origin

The live-stream API is available under the versioned preview namespace on the same portal origin. Use a Hallikar preview API key supplied with Vision API access. The stable Ox inference and Realtime APIs remain under /v1; do not interchange their credentials with a Hallikar preview key.

export OX_VISION_API_BASE_URL=https://www.amerint.co/v1beta
export HALLIKAR_API_KEY='[YOUR_HALLIKAR_API_KEY]'

Verify the model catalog

The model catalog is public and is the cheapest end-to-end connectivity check. Choose only a model whose status is ready.

curl --fail-with-body "$OX_VISION_API_BASE_URL/models" \
  --header "Accept: application/json"

Create a live stream

An empty JSON object creates a managed LiveKit stream. The response contains the stream ID and a short-lived, publish-only LiveKit token. Publish video with the LiveKit SDK, wait until stream status reports a first frame, and then reference that stream from Chat Completion.

curl --fail-with-body "$OX_VISION_API_BASE_URL/streams" \
  --request POST \
  --header "Authorization: Bearer $HALLIKAR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{}'

Ask about the latest frame

Replace STREAM_ID and MODEL_ID with values returned by the API. The hallikar:// URL is resolved inside Hallikar and is never fetched from the public internet.

curl --fail-with-body "$OX_VISION_API_BASE_URL/chat/completions" \
  --request POST \
  --header "Authorization: Bearer $HALLIKAR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"model":"MODEL_ID","messages":[{"role":"user","content":[{"type":"text","text":"What is happening now?"},{"type":"image_url","image_url":{"url":"hallikar://streams/STREAM_ID?frame_index=-1"}}]}],"max_completion_tokens":128}'

Understand the lifecycle

A stream is an owned, leased live-video resource. POST /streams creates it, GET /streams/{id} reports frame availability, POST /streams/{id}/keepalive renews its lease, and DELETE /streams/{id} ends it. There is no public pause, resume, list-all, or /infer endpoint in the current contract.

Wait for usable frames

Poll status until first_frame_at_ms is non-null. recent_fps describes the current capture cadence, retained_frame_count counts frames still available for inference, and evicted_frame_count counts frames that left the ten-minute history window. Frame indices are lifetime indices, not positions inside the retained window.

curl --fail-with-body "$OX_VISION_API_BASE_URL/streams/STREAM_ID" \
  --header "Authorization: Bearer $HALLIKAR_API_KEY"

Renew and delete explicitly

The current lease is five minutes. Renew around every four minutes and use the returned replacement publish token for managed LiveKit streams. Delete a stream when capture ends; do not wait for expiry as a normal cleanup path.

curl --fail-with-body "$OX_VISION_API_BASE_URL/streams/STREAM_ID/keepalive" \
  --request POST \
  --header "Authorization: Bearer $HALLIKAR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{}'

curl --fail-with-body "$OX_VISION_API_BASE_URL/streams/STREAM_ID" \
  --request DELETE \
  --header "Authorization: Bearer $HALLIKAR_API_KEY"

Choose a source type

The create-stream body accepts one of four current camera paths. Omitting source creates a Hallikar-managed LiveKit room and returns publish credentials. Pull sources connect to RTSP or HLS. livekit_room subscribes to one camera or screen-share track in a customer-owned LiveKit room.

Connect an RTSP or HLS source

Hallikar validates the URL and probes reachability before accepting a pull source. A source URL may contain camera credentials, so the API never echoes it in responses. Store the URL as a secret and send it only from trusted server code.

curl --fail-with-body "$OX_VISION_API_BASE_URL/streams" \
  --request POST \
  --header "Authorization: Bearer $HALLIKAR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"source":{"type":"hls","url":"https://camera.example/live/index.m3u8"}}'

Subscribe to a customer LiveKit room

The token must remain valid through the next lease interval and must allow the integration identity to join the named room. Set track_source to camera or screen_share. On keepalive, provide a refreshed source_token before the old token expires.

{
  "source": {
    "type": "livekit_room",
    "url": "wss://your-project.livekit.cloud",
    "token": "[SHORT_LIVED_LIVEKIT_TOKEN]",
    "participant_identity": "camera-17",
    "track_source": "camera"
  }
}

Send text, images, video, or live stream media

POST /chat/completions follows the OpenAI Chat Completions request and response shape. Text-only calls work. Image and video content can use reachable HTTPS URLs, bounded data URLs, or hallikar://streams/{id} references owned by the same API identity.

{
  "model": "MODEL_ID",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "Describe changes during the last ten seconds." },
      { "type": "video_url", "video_url": { "url": "hallikar://streams/STREAM_ID?start_offset_ms=-10000&end_offset_ms=0&max_fps=2" } }
    ]
  }],
  "max_completion_tokens": 256
}

Select an exact frame or time range

Use frame_index=-1 for the latest retained frame, timestamp_ms for a time measured from the first captured frame, or negative offset_ms for a position relative to the current stream time. Video requests accept start and end frame indices, timestamps, or offsets plus an optional max_fps sampling bound.

Stream server-sent events

Set stream: true and read each data: record until [DONE]. A final usage event is best effort when stream_options.include_usage is true. Cancel the upstream request when the downstream client disconnects.

List live availability

GET /models returns the current model catalog in an OpenAI-compatible list. This endpoint is public. Availability is runtime state: use only IDs whose status is ready, and check again after a capacity or provider error.

curl --fail-with-body "$OX_VISION_API_BASE_URL/models" \
  --header "Accept: application/json"

{
  "object": "list",
  "data": [{
    "id": "Qwen/Qwen3-VL-30B-A3B-Instruct",
    "object": "model",
    "owned_by": "hallikar",
    "status": "ready"
  }]
}

Treat the catalog as admission state

A model appearing in repository configuration or a model card does not prove it is currently admitted. The live catalog is authoritative for request selection. A later request can still fail because availability changes between discovery and inference, so handle 429, 502, 503, and 504 explicitly.

Core terms

Build for bounded live state

Create a stream immediately before capture, wait for first_frame_at_ms, renew the lease before four minutes, and delete it in a finally block. Treat publish, viewer, and customer-room tokens as credentials. Never log them, place them in URLs, or persist them longer than their job.

Use the smallest media selection

Use one image frame when the question is about the current scene. Request a video segment only when temporal order matters, and set max_fps to the lowest cadence that preserves the event. This reduces latency, token usage, and provider-side media processing.

Preserve ownership and region

Use the same bearer key for stream creation, status, preview, deletion, and any chat request that references the stream. A resource owned by another identity returns 404 rather than revealing its existence. If creation uses X-Hallikar-Region, send the same value on every lifecycle call.

Handle failure by class

Do not retry 401, 403, or validation 422 responses without changing the request. Retry transient 429, 502, 503, and 504 responses with capped exponential backoff and jitter. A failed keepalive near expiry requires a new stream; do not assume the old publish token remains valid.

Publish a browser camera

Create a managed stream on your server and return only its publish descriptor to the signed-in browser. The browser joins the room and publishes video with the official LiveKit client. Never send the Hallikar API key to browser code.

import { Room, createLocalTracks } from "livekit-client";

const room = new Room();
await room.connect(stream.publish.url, stream.publish.token);
const [videoTrack] = await createLocalTracks({ video: true, audio: false });
await room.localParticipant.publishTrack(videoTrack);

Watch what Hallikar receives

Call POST /streams/{id}/viewer-token on your server and return the short-lived view descriptor to an authorized viewer. Viewer tokens cannot publish media or data. This endpoint returns 409 for livekit_room sources because the customer-owned room controls its own subscribers.

Refresh without interrupting capture

Call keepalive before the lease expires. For a managed stream, pass the replacement publish token to LiveKit with room.updateToken(newToken). For a customer-owned LiveKit source, send a refreshed join token in {"source_token":"..."}; pull sources do not use either token.