Skip to main content
A dataset is a named, versioned bundle of rows shaped for training or evaluation. Each row is a (prompt, response, metadata) tuple. Datasets are how you go from “a million raw traces” to “500 high-quality examples of ticket classification I want to distill a model on”. Datasets are the artifact that makes everything downstream work. You can’t distill without one. You can’t run a rigorous evaluation without one.

​
Three ways a dataset is born

​
1. Auto-clustered from traces (the fast path)

The engine clusters your traces by prompt embedding similarity, names each cluster with an LLM, and offers each named cluster as a candidate dataset:
In the UI you click Promote to dataset, give it a name, and pick the rows to include. Under the hood, this is the shortest path from running traffic to training data.

​
2. Uploaded from a file

If you already have labeled data — a JSONL, a CSV, a Hugging Face dataset — you can upload it directly:

​
3. Generated from prompts (synthetic)

Start with a list of prompts you care about. The engine asks a teacher model to generate N responses per prompt, judges them, and keeps the top ones. Useful when you have prompt ideas but no labeled responses yet.

​
What a dataset row looks like

Rows that came from traces retain a source_trace_id — you can always follow back to the original request.

​
Curation: filtering the bad rows

Raw traces have noise. A good dataset is curated: you keep the useful rows and drop the rest. The engine ships a curation pipeline with three stages:
1

Judge

An LLM judge (configurable — defaults to a cheap model like openai/gpt-4o-mini) scores each row on helpfulness, relevance, and format. Rows below a threshold are flagged.
2

Filter

Apply rules: drop rows with errors, drop rows outside the target cluster, drop rows above a length limit, drop rows with flagged PII.
3

Review

Human review in the UI for the top slice — usually 50–100 borderline rows. Not required but cheap insurance for your first distillation.
Each stage is implemented in opentracy.distillation.curation and you can run them standalone if you’re building a pipeline manually. See the API reference for details.

​
Versioning

Datasets are immutable once frozen. If you curate more rows or change the judge, you get a new version:
Every distillation job records the dataset version it trained on, so you can always reproduce a result or compare student models trained on different data.

​
Two things you do with a dataset

Distill

Hand the dataset to the distillation pipeline. A student model gets fine-tuned on the teacher’s labels and comes out as a LoRA adapter you can serve.

Evaluate

Pick a dataset as the benchmark. Run any model against it and compare accuracy, cost per row, and latency — including models you’re considering swapping in via an alias.

​
Common mistakes

Don’t distill with < 200 rows. Below that threshold the student tends to overfit and doesn’t generalize beyond the training prompts. 500–2000 is the sweet spot for most tasks.
Don’t mix unrelated clusters in one dataset. If you put “SQL generation” and “customer emails” in the same dataset, the student learns neither well. One dataset = one coherent task.
Don’t skip curation on the first run. Raw traces include failures, refusals, truncated outputs. Let the judge drop those before training — otherwise the student learns the noise.