This is the prompt-learning loop: dataset → objective → optimizer → validation. It needs three things from you — labeled examples, a starter prompt and an objective. We'll walk through prompt compression, but the same loop optimizes anything you can score.
A dataset is a list of inputs with known answers. Here is a name-extraction set: passages of text and the people mentioned in them.
| inputs.text | answer |
|---|---|
| "After the merger, Priya Natarajan moved the Denver office to a four-day week; Marcus Hale objected in the March board meeting." | ["Priya Natarajan", "Marcus Hale"] |
| "Rainfall in Portland exceeded the September average, the National Weather Service reported." | [] |
| "Dr. Elena Vasquez will present the findings; Tuesday's session is chaired by Tom Bright of Unilever." | ["Elena Vasquez", "Tom Bright"] |
| "The Lakers signed Charles Okafor to a two-year deal, sources told Reuters." | ["Charles Okafor"] |
Run a prompt over the dataset and score each output against the answer. Now every prompt has a number — or a vector of numbers. That's what the optimizer sees. It never needs to understand the task; it only needs (prompt, score) pairs.
| prompt | f1 | template_tokens |
|---|---|---|
| Extract the full names of all people mentioned in the text below. Return only real people (not places, companies, months …). Output JSON: {"names": [...]} | 0.97 | 97 |
| List every person's full name in the text as JSON {"names": [...]}. Skip organizations and places. | 0.96 | 31 |
| People named, JSON {"names":[]} | 0.93 | 9 |
| Names: | 0.61 | 2 |
The starter prompt is the root of the search tree — usually whatever you have in production today. The objective says what "better" means. Scores are vectors; the objective is a small function that turns a vector into a scalar, so it's cheap to swap.
Maximize accuracy minus a per-token penalty, or hold accuracy above a floor and minimize tokens. f1 − 0.001·template_tokens, or a token budget that tightens each round.
Exact match, set-F1, or an LLM judge asked "is this equivalent to the reference?" Pure quality, no cost term.
Fraction of outputs that parse as valid JSON, follow a schema, or stay within a length limit. Great for tool-call formatters.
Output tokens, prompt tokens, wall time. Combine with a quality metric to trade them off explicitly.
from bpto import AnthropicClient, Dataset, LinearObjective, Task, combine, template_tokens, token_count
from tasks.compression import Entities, set_f1
task = Task(
root=open("prompt.txt").read(), # your current prompt, with {text} placeholder
description="extracts the names of all people mentioned in a passage",
dataset=Dataset.from_jsonl("train.jsonl"), # {"inputs": {"text": ...}, "answer": [...]}
schema=Entities,
scorer=combine(set_f1(), template_tokens(), token_count()),
objective=LinearObjective(f1=1.0, template_tokens=-0.002),
client=AnthropicClient("claude-opus-5"),
)The optimizer grows a tree. An LLM proposes rewrites of a node (random variants, or directed: "make it shorter", "be more precise", or a reflection on the examples the parent got wrong). Each child is evaluated on the dataset. A selector decides which node to expand next — greedy, a Pareto pool, or a Bayesian surrogate over prompt embeddings that predicts where the good children are.
For a two-objective problem like compression, the answer isn't one prompt — it's a front. bpto keeps every non-dominated (tokens, accuracy) pair so you can choose the operating point your product needs.
You are an information extraction system. Read the passage below carefully and extract the full names of every person who is mentioned. Include each person only once, preserve the order in which they first appear, and do not include organisations, places, or pronouns. Titles such as Dr. or Professor are not part of the name. If no people are mentioned, return an empty list.
Passage:
{text}
Extract human names from: {text}
Nothing above was specific to shortening prompts. The optimizer sees a root, a dataset, a scorer and an objective. Change the objective and the same tree search does something else:
Completions are cached on (prompt, config, schema), so re-scoring the whole tree under a new objective costs nothing.
Real products aren't one prompt. A ReAct-style agent is a loop of several prompts, each with its own job, each run several times per task. Every box below is a prompt you can optimize — with its own objective — using the same loop.
Because the optimizer only ever sees (prompt, score), a module's "prompt" can be anything text-shaped: a system prompt, a few-shot block, a tool description, or the instructions inside a chain-of-thought template.
Today all of this runs as the open-source bpto library. The promptcompression.ai Studio will put the same steps — upload data, set a starter prompt and objective, search, pick from the front — behind a point-and-click interface.