Prompt compression: 97 tokens → 6–29, GEPA vs Bayesian optimization
12 seeds × 2,000 rollouts per arm · 100 training / 200 held-out examples per seed · task: extract people's names from a passage · model under optimization: Amazon Nova Micro; rewrites by Nova Lite · total cost $0.41 · September 2026.
Both arms use the same mutation operator (an LLM reads a minibatch of the parent's failures and rewrites), the same evaluation gate, the same budget. They differ only in which parent gets expanded and how many children are proposed and screened. The objective was constrained: minimize template tokens subject to training F1 ≥ a floor that tightened from root − 0.15 to root − 0.05 over the run.
Best prompt per seed (held-out F1 in parentheses; root held-out F1 0.97–0.99)
| seed | GEPA | tok | BO | tok |
|---|---|---|---|---|
| 0 | List all unique full names of people in: {text}, ignoring titles and organizations. | 19 (0.982) | Extract full names from: {text} | 6 (0.904) |
| 1 | List unique full names from this text, ignoring titles: {text} | 14 (0.907) | Extract full names from text, excluding titles: {text} | 12 (0.910) |
| 2 | Extract human names from: {text} | 6 (0.965) | Extract only personal names from this passage: {text} | 12 (0.951) |
| 3 | Extract all unique full names from: {text}, ignoring titles and organizations. | 18 (0.982) | Extract personal names from {text}, avoid titles and organizations. | 15 (0.902) |
| 4 | Extract names from: {text}. List only people, no titles or orgs. | 14 (0.880) | Extract full human names from: {text} | 8 (0.891) |
| 5 | List all names of people in this passage, excluding titles: {text} | 15 (0.994) | Extract all human names from: {text} | 8 (0.952) |
| 6 | Identify all people's full names mentioned in: {text}. List them once, in order, without titles. | 22 (0.992) | Extract only person names from: {text} | 8 (0.987) |
| 7 | Extract full names from this text, omitting titles and entities: {text} | 16 (0.990) | Identify human names in: {text}. | 6 (0.942) |
| 8 | Extract all full names of individuals from: {text} | 11 (0.916) | List all full person names from the following text.\nPassage:\n{text} | 15 (0.953) |
| 9 | Extract full names from: {text}. Ignore titles and locations. | 14 (0.916) | List names in this text, excluding titles and organizations: {text} | 15 (0.972) |
| 10 | Identify all full names of individuals in this passage, avoiding titles and organizations: {text} | 23 (0.996) | Identify and list all names of people mentioned in the text below, ignoring titles and non-person entities. Names:\n{text} | 29 (0.946) |
| 11 | Identify all full names of people in this passage and list them in order: {text} | 18 (0.897) | Extract full names from: {text}. Exclude titles. | 10 (0.987) |
Summary: GEPA 15.8 ± 1.3 tokens, held-out F1 0.951; BO 12.0 ± 1.8 tokens, held-out F1 0.941. Paired difference 3.8 ± 2.0 tokens, BO shorter on 8 of 12 seeds. At the accuracy the floor targeted (root − 0.05 to − 0.10) BO is 3–5 tokens shorter with the ±2 SE band clear of zero; at stricter bars (within 0.03 of root) GEPA's Pareto pool hedges better. BO makes ~3× the reflector calls (3 children per round vs 1); a "GEPA + 3 children, random keep-1" control is planned.
Source: experiments/2026-09-12-compression-v2-gepa-vs-bo (NOTES.md, results.jsonl, analysis.md).