Python SDK

The Python SDK (opentracy) is the native entry point. Use it if you’re starting a new project or if you want features (auto-routing, distillation, trace ingestion) that aren’t part of the OpenAI API shape.

Install

pip install opentracy

One install pulls a platform-specific wheel with the Go engine binary, the ONNX embedder, and pre-trained routing weights bundled in. No extras needed for the core path.

pip install "opentracy[distill]"    # adds training deps (torch, unsloth, peft, trl)
pip install "opentracy[research]"   # adds sentence-transformers for the Python router backend
pip install "opentracy[server]"     # adds FastAPI + ClickHouse for self-hosting
pip install "opentracy[anthropic]"  # native Anthropic SDK path
pip install "opentracy[all]"        # everything

The four things you’ll do

1. One-off completion

Just a chat completion, no routing, no trace.

import opentracy as ot

resp = ot.completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "What's 2+2?"}],
    temperature=0,
)
print(resp.choices[0].message.content)

Full API: completion reference.

2. Explicit router with fallbacks

When you want deterministic rules (“try GPT-4o first, then Claude, then DeepSeek”), use the Router class:

router = ot.Router(
    model_list=[
        {"model_name": "smart", "model": "openai/gpt-4o"},
        {"model_name": "smart", "model": "anthropic/claude-sonnet-4-6"},
    ],
    fallbacks=[{"smart": ["deepseek/deepseek-chat"]}],
    strategy="round-robin",   # or "least-cost", "lowest-latency", "weighted-random"
    num_retries=2,
    timeout=60,
)

resp = router.completion(
    model="smart",   # logical alias, resolved to one of the deployments
    messages=[{"role": "user", "content": "..."}],
)

Full API: Router reference.

3. Semantic auto-router

Load the pre-trained router once; it picks the right model per prompt:

auto = ot.load_router(cost_weight=0.5)

decision = auto.route("Write a haiku about autumn")
print(decision.selected_model)      # e.g. "ministral-3b-latest"
print(decision.cluster_id)          # e.g. 87
print(decision.expected_error)      # e.g. 0.212
print(decision.all_scores)          # full score dict

Combined with ot.completion this becomes a cost-optimizing client:

def smart_call(prompt: str, api_key: str) -> str:
    d = auto.route(prompt)
    resp = ot.completion(
        model=d.selected_model,
        messages=[{"role": "user", "content": prompt}],
        api_key=api_key,
    )
    return resp.choices[0].message.content

Full API: load_router reference.

4. Distillation

The one-call path — ot.distill() runs the full 4-phase pipeline in-process and returns a callable Student. Needs opentracy[distill] and a CUDA GPU.

import opentracy as ot

student = ot.distill(
    dataset="tickets.jsonl",          # path, list[dict], or a callable
    teacher="openai/gpt-4o",
    student="llama-3.2-1b",
    steps=100,
    quantize="q4_k_m",                # or None to skip GGUF export
)

print(student("Classify: refund please"))       # local inference, $0

# Ship it behind a logical name — app code never changes
student.deploy("ticket-classifier")
resp = ot.completion(model="ticket-classifier", messages=[...])

Full API: ot.distill reference. For the long-running, queued REST flow against a self-hosted engine (ClickHouse-backed jobs, UI observability), use Distiller instead — same engine, different deployment shape.

Async

Everything that has a sync version has async:

import asyncio
import opentracy as ot

async def main():
    resp = await ot.acompletion(
        model="openai/gpt-4o-mini",
        messages=[{"role": "user", "content": "hello"}],
    )
    print(resp.choices[0].message.content)

asyncio.run(main())

acompletion shares its request-preparation path with the sync version, so force_engine, force_direct, fallbacks, and engine-prefix handling all behave identically.

Trace ingestion

If you have existing logs from another LLM provider and want to use them for dataset building or distillation in OpenTracy, you can import them directly:

from opentracy import add_trace, add_traces, import_traces

# Single trace
add_trace({
    "prompt": "Classify: ...",
    "response": "billing",
    "model": "openai/gpt-4o",
    "total_cost_usd": 0.00025,
    "latency_ms": 340,
    "metadata": {"source": "legacy-log-export"},
})

# Batch
add_traces([{...}, {...}, {...}])

# From a JSONL file
import_traces("path/to/exported-traces.jsonl")

Engine routing opt-in

By default the SDK calls providers directly. To route through an OpenTracy engine (for observability, aliases, etc.), set the env var once:

export OPENTRACY_ENGINE_URL="http://localhost:8080"

From that point on, ot.completion(...) routes through the engine. Per-call overrides:

# Always engine (even if OPENTRACY_ENGINE_URL is unset):
ot.completion(..., force_engine=True)

# Always direct (even if OPENTRACY_ENGINE_URL is set):
ot.completion(..., force_direct=True)

Why isn’t this automatic? Because silently routing through whatever happens to be listening on localhost:8080 is a footgun. Opt-in is explicit.

13 providers via `create_client`

If you want a first-class LLMClient object (for profiling, or to fit into custom routing code), create_client covers every provider:

c = ot.create_client("openai",   "gpt-4o-mini")       # dedicated class
c = ot.create_client("deepseek", "deepseek-chat")     # UnifiedClient wrapper
c = ot.create_client("together", "meta-llama/Llama-3")# UnifiedClient wrapper

out = c.generate("Hello", max_tokens=64, temperature=0.0)
print(out.text, out.latency_ms, out.tokens_used)

Five providers have dedicated classes (OpenAI, Anthropic, Google, Groq, Mistral); the remaining seven (DeepSeek, Perplexity, Cerebras, Sambanova, Together, Fireworks, Cohere) route through a UnifiedClient that speaks the OpenAI-chat protocol. Bedrock is registered but raises a clear error on construction — AWS SigV4 is not handled by UnifiedClient yet; use ot.completion(force_engine=True) instead.

Public API

Everything import opentracy as ot exposes publicly:

# Core
ot.completion, ot.acompletion, ot.Router, ot.ModelResponse, ot.StreamChunk, ot.parse_model
# Multi-provider
ot.create_client, ot.LLMResponse
# Pricing
ot.model_cost, ot.get_model_info, ot.supported_models
# Trace ingestion
ot.add_trace, ot.add_traces, ot.import_traces
# Distillation — one-call + REST client
ot.distill, ot.DistillError, ot.Student, ot.StudentError
ot.Distiller, ot.TrainingClient, ot.DistillerError
# Local alias registry (distilled students map to logical model names)
ot.set_alias, ot.unset_alias, ot.list_aliases, ot.get_alias
# Version
ot.__version__

Lazy research APIs (load_router, UniRouteRouter, RouterEvaluator, LLMJudge, …) resolve via __getattr__ — they import the first time you touch them, so they don’t slow down the initial import opentracy.

Legacy code using import lunar_router as lr keeps working via a backwards-compat shim that redirects to opentracy and emits a DeprecationWarning. New code should use import opentracy as ot.

Self-host

Run engine + ClickHouse + UI locally or in your cloud.

API Reference

Every parameter and return value.

Get Started

Concepts

Guides

Install

The four things you’ll do

1. One-off completion

2. Explicit router with fallbacks

3. Semantic auto-router

4. Distillation

Async

Trace ingestion

Engine routing opt-in

13 providers via `create_client`

Public API

Next

Self-host

API Reference

​Install

​The four things you’ll do

​1. One-off completion

​2. Explicit router with fallbacks

​3. Semantic auto-router

​4. Distillation

​Async

​Trace ingestion

​Engine routing opt-in

​13 providers via create_client

​Public API

​Next

Self-host

API Reference

Install

The four things you’ll do

1. One-off completion

2. Explicit router with fallbacks

3. Semantic auto-router

4. Distillation

Async

Trace ingestion

Engine routing opt-in

13 providers via `create_client`

Public API

Next