The objective is flat almost everywhere, jumps when it moves, and lives on a thin slice of the space that nobody can write down. Here's what that means and how the current methods deal with it.
Prompt learning is machine learning applied to prompts. The analogy is exact enough to be useful:
| training a model | prompt learning |
|---|---|
| weights | the prompt text |
| training set | labeled (input, answer) examples |
| loss function | the objective: accuracy, tokens, format, latency, or a weighted / constrained combination |
| optimizer (SGD, Adam) | prompt optimization: an LLM proposes rewrites, a selector decides which to evaluate and expand |
| validation set | a held-out set the optimizer never sees |
| overfitting | a prompt that wins on 30 examples and loses on the next 30 |
Where the analogy breaks is the optimizer row, and that break is the subject of this page. Weights are continuous and the loss is differentiable; prompts are discrete text and the objective has no gradient. Everything else transfers — which is why the discipline that comes with it (held-out sets, budgets, reporting fronts instead of single numbers) transfers too.
Not to be confused with prompt tuning (Lester et al. 2021) or prefix tuning, which learn continuous "soft prompt" vectors by gradient descent. Prompt learning in the sense used here keeps the model frozen and the prompt human-readable; the output is a string you can paste into a config file.
Prompt optimization means: given a task and a scoring function, find the string that scores best. The space is every possible string. The score is what an LLM does when you feed it that string, averaged over a dataset. That's a well-posed optimization problem — and almost none of the usual machinery applies.
Change one word and the score usually doesn't move at all. The model reads "Extract the names" and "Extract all the names" the same way. Gradient descent has nothing to follow: the surface is piecewise flat, and the pieces are wide. The only way to learn anything is to make a real edit and pay for a full evaluation.
Drop the word "only" and precision collapses. Add "as JSON" and format compliance goes from 60% to 99%. The landscape is a plateau riddled with cliffs and thousands of tiny basins. Local search gets stuck in the first basin it finds; every basin looks like an optimum from inside.
Almost every string is a useless prompt. The good ones live on a thin manifold of fluent, task-relevant text — and there is no formula for its boundary. You can't grid-search it, you can't sample it uniformly, and you can't project onto it. The only thing that reliably stays on the manifold is another language model generating text. So every practical method has the same shape: an LLM proposes candidates, a scorer evaluates them, and a selector decides where to look next. The differences are all in the proposer and the selector.
Every point on the landscape costs one LLM call per dataset example. Scores carry sampling noise. Small datasets overfit: a prompt that wins on 30 examples may lose on the next 30. So the budget is measured in rollouts — unique (prompt, example) calls — and the question every method has to answer is: what do I evaluate next, given what I've already paid for?
Three families, each getting one thing right.
DSPy separates the program (a pipeline of typed modules) from the prompt text each module ends up using. Its optimizers then fill in the text. MIPROv2 proposes a pool of candidate instructions and bootstrapped few-shot demonstrations for each module, then runs Bayesian optimization over the discrete choice of which instruction and which demos to combine — a surrogate model predicts which combinations will score well, so the expensive full evaluations go where they're most informative.
GEPA (Agrawal et al., 2025) treats optimization as evolution, with two ideas that matter. First, the mutation operator is reflective: an LLM reads the parent prompt, the examples it failed on and textual feedback about why, then rewrites. A scalar reward says "0.7"; text feedback says "you extracted the company name as a person" — far more information per rollout. Second, the parent pool is a per-example Pareto front: a candidate stays alive if it's the best on any single training example, and parents are sampled in proportion to how many examples they win. That preserves diversity — a prompt that's mediocre on average but nails a hard case is kept around to breed from.
Reference numbers from the paper on Qwen3-8B (test accuracy, %): HotpotQA 42.3 → 62.3, HoVer 35.3 → 52.3, PUPA 80.8 → 91.9, IFBench 36.9 → 38.6, within ~7k rollouts — beating MIPROv2 and GRPO-style RL at a fraction of the rollouts.
Bayesian optimization is the standard answer to "expensive, noisy, black-box objective". Embed each evaluated prompt as a vector, fit a Gaussian process from embedding → score, and use an acquisition function (expected improvement, UCB, Thompson sampling) to choose the next candidate: the one with the best combination of predicted score and uncertainty. The GP handles noise natively — it knows a single high score on a small sample might be luck.
Applied to a prompt tree, BO answers two questions: which node should I expand next? (fit the surrogate to a node's descendants' value, so it learns which regions breed good children) and which of these freshly-proposed children are worth a full evaluation? (pre-screen by predicted score, evaluate only the top ones).
bpto implements GEPA-style selection and BO selection as peers on the same tree, with the same mutation operators, cache and budget, so they can be compared at equal rollouts with one ingredient swapped at a time. The experiment log is in the repo; the standing conclusions so far: