How this page works
This page is written in two layers. When a formula matters, the exact mathematical version appears on the left. Beside it is the same idea in everyday language. You can follow only the plain-language side and still understand the main story.
y = f(x; θ)
In the simplest terms, the model receives some information, processes it using numbers learned during training, and produces an answer. The rest of this page explains how those learned numbers are organised and how we check that the answers deserve to be trusted.
Three labels appear throughout the page. Measured means the number came from an experiment. Retracted means an earlier conclusion was later shown to be wrong or unsupported, so we keep it visible as a warning. Load-bearing marks a decision that other parts of the system depend on.
Part 1 — The basic idea
A small language model trained for one job
Frugal models are deliberately small: roughly nine million adjustable values. That is tiny compared with today's large general-purpose AI systems, which can contain hundreds of billions of values. The aim is not to know everything; it is to do a focused job efficiently.
Each model is trained from scratch for its task. It does not begin with a general model that has already read the internet. Its vocabulary and behaviour come from the material supplied for that particular job.
That choice defines what Frugal is good at. It is not intended to answer general-knowledge questions from memory. Instead, it is designed to work with the information placed in front of it — for example, finding a date in a document, choosing a category, or deciding what action the document calls for.
“Frugal” refers to three practical goals: the model should be inexpensive to train, inexpensive to run, and practical to keep under your own control. In many cases it can run on ordinary hardware without sending private documents to an external AI service.
What a task looks like
A task starts with a simple description of what is needed and the fields the model should return. A field might be text, a number, a date, an identifier, or one choice from a fixed list. The same platform can extract information, classify something, choose an action, work out an answer, or generate text.
The normal workflow has four steps: prepare the training material, train the model, test it on examples it has not seen before, and then use it on real documents or tasks.
Part 2 — How the model works
2.1 How the model reads a sequence
Many modern language models use attention, which lets each word compare itself with many other words. That is powerful, but it becomes increasingly expensive as the input gets longer. Frugal uses a cheaper running-memory approach instead.
h_t = a_t ⊙ h_{t−1} + (1 − a_t) ⊙ x_t
a_t = σ(W_a · ·) forget gate, full precision
bias 2.0 ⇒ a ≈ 0.88 at init
The model carries a running summary as it reads. For each new token, it decides how much of the previous summary to keep and how much new information to mix in. The control that makes this choice is called the forget gate.
At the start of training it is biased toward remembering: about 88% of the previous summary is kept at each step. The model therefore begins by holding on to context and learns when information can safely be forgotten.
Because the model only needs this running summary, the memory needed to produce each new token stays roughly constant instead of growing with the whole prompt. That is a major reason it can run on ordinary hardware.
Reading the same information much faster
A running summary sounds as though every step must wait for the previous one. Fortunately, the calculation can be regrouped so that many parts are processed at the same time without changing the underlying result.
(a₂, x₂) ∘ (a₁, x₁) = (a₂·a₁, x₂ + a₂·x₁) Kogge–Stone parallel prefix scan O(log T) sequential steps shifts doubling 1, 2, 4, …
A useful analogy is adding a long list of numbers: you can add several groups at once and then combine the group totals. The Frugal recurrence has a similar property, so it can be evaluated in parallel.
For a sequence of 512 tokens, this reduces what looks like 512 dependent steps to about nine rounds of parallel combination — roughly 57 times fewer sequential stages, while implementing the same equation.
We discovered that generation was still using the slow version of this calculation. A 384-token prompt took 4.3 seconds before producing its first new token, and a 2,048-token invoice took about 23 seconds. The parallel version completed the same calculation in 65 milliseconds — a 66× difference. The mathematics was fine; the implementation was calling the slower route.
⚠ The two methods are mathematically equivalent, but computers can produce tiny last-decimal differences when operations are grouped differently. For that reason, benchmark scores from before and after this implementation change are not treated as directly comparable.
2.2 Storing most learned weights in just three states
A conventional neural network often stores each learned weight as a 32-bit floating-point number. Frugal compresses most of these weights to only three possibilities: negative, zero, or positive.
scale = mean(|W|) ≥ 1e-5
Ŵ = clip(round(W / scale), −1, +1) · scale
Ŵ ∈ {−scale, 0, +scale}
log₂ 3 ≈ 1.58 bits per weight
For each block of weights, the model first works out a typical size, called the scale. Every weight is then rounded to the nearest of three values: minus that scale, zero, or plus that scale.
Only three states need about 1.58 bits of information instead of 32. In a properly packed implementation this can greatly reduce how much data must be moved through memory, which matters more than it may sound for small CPU-based models.
How it can still learn after rounding
Rounding makes learning awkward because small changes disappear until a threshold is crossed. Training therefore uses a standard approximation that lets the model behave as though the rounding step were smooth when it calculates how each weight should change.
w_used = W + (Ŵ − W).detach() forward: value is Ŵ backward: gradient flows to W
In other words: use the rounded value when producing an answer, but send the learning signal back to the precise underlying value. This is intentionally approximate, but it allows the compact model to train.
The model therefore keeps a precise training copy and a three-level working copy. The precise value can move gradually during training; the working value changes only when the precise value crosses a rounding boundary.
A few sensitive components remain full precision, including the tokenizer, expert router, forget gate and uncertainty layer.
The important engineering lesson is that ordinary PyTorch and NumPy arrays do not automatically realise the storage saving: they still hold ternary values in large numeric formats. In the measured implementation, performance was limited mainly by moving weights through memory, not by doing arithmetic.
A genuinely compact representation packs four two-bit weights into one byte, uses small integer inputs and accumulates the result before scaling once at the end. Against NumPy this measured 1.45–1.85× faster, with the advantage increasing on larger matrices — exactly what we would expect if memory traffic is the real bottleneck.
⛔ We also tested a literally “multiply-free” version that used a branch for every weight: add, skip or subtract. It was 5× slower. The practical benefit comes from compact storage, not from avoiding the processor's multiply instruction.
Another inefficiency was later removed: the code kept recalculating rounded weights during generation even though the underlying weights cannot change at that time. That unnecessary work cost 2.13× a decode step, meaning more than half of generation effort was being spent recomputing a constant.
2.3 Using a few specialists at a time
r = W_r x , p = softmax(r)
(g, idx) = topk(p, k) , g ← g / Σ g
y = Σ_{j∈top-k} g_j · E_{idx_j}(x)
Instead of sending every token through one large general-purpose sub-network, the model has several smaller “experts.” A router chooses the two most relevant experts for each token and combines their outputs.
This lets the model contain more specialised capacity without paying the full computation cost on every token, because most experts are inactive for any one piece of text.
Without a safeguard, one expert can start receiving most of the work simply because it gained an early advantage. The remaining experts then get too little practice. A small balancing penalty helps prevent that collapse.
L_aux = E · Σ_e f_e · P_e f_e = fraction of tokens routed to e P_e = mean router probability for e L_aux = 1 when routing is uniform weight = 0.01
The penalty looks at both how often each expert was selected and how strongly the router preferred it. It encourages the workload to remain reasonably spread out rather than concentrating on a single expert.
The penalty is intentionally small — one hundredth of the main scale — because the goal is to prevent collapse, not to force perfectly even usage when the work itself is uneven.
2.4 Measuring when the model is unsure
The final layer does not treat every learned weight as one perfectly known number. Instead, each weight is represented by a central estimate and a range of plausible values.
σ = softplus(ρ) init ρ = −5 w = μ_w + σ_w ⊙ ε , ε ~ N(0, I)
You can picture each weight as a small cloud rather than a single point. Sampling from those clouds produces several slightly different versions of the same trained model.
Training is arranged so this randomness does not block learning: a fixed random draw is shifted and scaled by values the model can learn. The uncertainty ranges begin very narrow and widen only when the data supports that uncertainty.
KL(q ‖ p) = Σ [ log(σ_p/σ)
+ (σ² + μ²)/(2σ_p²) − ½ ]
weight kl_w = 1 / n_tokens
A regularising penalty keeps the model from becoming unjustifiably certain. In simple terms, the model must earn confidence from the data rather than assuming it.
With little training data, that cautious prior matters more; with a large amount of evidence, the data has more influence. This is how the model is encouraged to remain appropriately uncertain on weak evidence.
Two different kinds of uncertainty
p̄ = (1/S) Σ_s p_s total = H[p̄] aleatoric = (1/S) Σ_s H[p_s] epistemic = I = H[p̄] − (1/S) Σ_s H[p_s]
Running the model several times helps separate two situations. If all runs agree but the answer itself is ambiguous, the uncertainty is in the task or data. If the runs disagree with one another, the model itself lacks enough knowledge. The technical terms are aleatoric and epistemic uncertainty.
The second kind — disagreement between model samples — is the more useful warning signal. It tends to rise when the model is outside the kind of examples it learned from and can therefore support a “send this to a human” rule.
It is tempting to turn the two uncertainty types into automatic advice such as “collect more data” or “fix ambiguous labels.” We tested that interpretation at field level, and the evidence did not support it.
Across 25 fields, the proportion labelled aleatoric stayed in a narrow range of 0.763–0.979 and did not separate easy fields from hard ones. Even a field with 100% accuracy looked similar to one with only 25.7% accuracy on that measure.
The reason is structural: disagreement between well-trained model samples is usually small, while uncertainty over wording is almost always present. We still report both values, but we no longer treat either one as a diagnosis by itself.
2.5 Copying values instead of making them up
For a small model working with documents, the most damaging mistake is often not choosing the wrong visible value — it is inventing a value that never appeared in the document. In one date study, 51.8% of errors were invented values, compared with only 2.1% that selected the wrong date that was actually present.
α = softmax(scores) over source positions
context = Σ_i α_i · src_h_i
p_gen = σ(W_g · [q_h ; context])
copy_dist(w) = Σ_{i : src_id_i = w} α_i
p(w) = p_gen·gen_probs(w)
+ (1 − p_gen)·copy_dist(w)
The model therefore has two ways to produce the next token: generate it from its vocabulary, or point back to a token already present in the source document. A learned switch decides how much to use each route.
Pointing is especially useful for values such as account numbers, identifiers and dates, where exact copying matters and invention is unacceptable.
Teaching the model when copying is required
Simply providing a copy mechanism is not enough. If the model can receive the same reward by guessing a common value, it may still learn to generate instead of locate. Extra training signals therefore push it toward copying whenever the correct value is provably present in the source.
L_gate = BCE(p_gen[t], y_gate[t])
L_ptr = −log Σ_{s ∈ occurrences(t)} α[t, s]
L = L_lm + λ_gate·L_gate + λ_ptr·L_ptr
When the training data proves that a value appears in the document, the model is taught both to choose the copy route and to point near a valid occurrence. If the same value appears several times, any correct occurrence is accepted.
The rule is straightforward: if the value is known to be in the source, teach copy; if the output is punctuation, a fixed option or an explicit “missing” marker, teach generate; if we cannot reliably determine whether copying is possible, do not force either behaviour.
That last case matters. Treating a failed text match as proof that the value is absent would actively teach the model to invent. Uncertain supervision is therefore left out rather than converted into a misleading lesson.
An early version supervised value tokens but not the JSON punctuation around them. Because the same generation-versus-copy switch is shared across output types, the strong pressure to copy values also pushed punctuation toward copying, even though punctuation usually was not available in the source.
As the model tried to copy punctuation that was not there, the relevant probability approached zero and the training loss exploded to NaN. Every stage failed mathematically even though the run status still said succeeded. That failure led to stricter validity checks and explicit supervision for structural output.
2.6 Turning raw text into model-sized pieces
base = 256 byte tokens + specials merge = most frequent adjacent pair repeat until |vocab| = vocab_size default vocab_size = 2048
The tokenizer starts with all 256 possible byte values, so any text can be represented. It then repeatedly joins common neighbouring byte patterns into larger symbols. Frequent words become short representations; unusual words remain split into smaller pieces.
Because the system begins from bytes, an unfamiliar character, another writing system or an emoji can still be represented. It may require more tokens, but it is never silently treated as an unknown character.
On MuSiQue-style Wikipedia text, a vocabulary of 2,048 symbols averaged 3.08 characters per token. That preserves all text but splits names more heavily, so a fixed 512-token window contains less source material than it would with a larger vocabulary. Vocabulary size is therefore chosen deliberately before training, because every later model inherits that choice.
2.7 What training is trying to improve
L = L_task + kl_w · KL(q ‖ p) + moe_aux_weight · L_aux AdamW, lr 3e-3, weight decay 0.05 cosine annealing over all steps gradient clipping at norm 1.0 kl_w = 1 / (len(dataset) · seq_len)
Training combines three goals: produce the correct answer, avoid unjustified certainty, and keep the expert router from collapsing onto one specialist. The optimiser changes the model to reduce the combined penalty from those goals.
The learning rate starts relatively large and gradually becomes smaller, so training can move quickly at first and then make finer adjustments. Gradient clipping also limits any single update, preventing one unusual example from causing an extreme jump.
The epoch budget is more than a spending limit. It also sets the length of the learning-rate schedule, so changing the budget changes how the model learns. A “longer run” is therefore not exactly the same experiment with more time attached.
That is one reason the default budget is 150 epochs rather than 1,000. With a 1,000-epoch schedule, the first 100 epochs would still be the warm-up period, while many of our studies finish before that point.
Part 3 — The statistical guards
Why the safeguards matter
The model architecture above mostly uses established ideas. The safeguards below come from our own experimental mistakes: cases where a result looked convincing until we discovered that the measurement process itself had created or exaggerated the effect.
The recurring lesson is that software can run exactly as written and still answer the wrong scientific question. Ordinary unit tests do not catch that. These checks are meant to test the experiment, not only the code.
3.1 What counts as genuine learning
Before calling a score “good,” we first ask what score could be achieved without learning anything useful. For example, if 90% of examples belong to one category, always choosing that category already scores 0.90.
trivial_floor = score of the most
frequent observed answer
MEASURED_SEED_NOISE = 0.05
COMPETENCE_MARGIN = 2 × 0.05 = 0.10
beats_trivial ⟺ score ≥ floor + margin
We call that easy baseline the floor. A model must beat it before we can claim that the model has learned something beyond the simplest possible shortcut.
Even a small win above the floor may just be random variation. Repeating the same training recipe produces slightly different results because the starting point is random. We measured that variation at about 0.05, so our default requirement is to clear the floor by twice that amount.
A “guess the most common answer” shortcut has passed earlier checks three times. Most recently, a reasoning task used a floor of 0.00, allowing one repeated answer across 200 questions to be marked as better than trivial. The floor is now based on the most common observed answer instead.
⚠ The randomness is not always evenly spread across behaviours. In one repeat, accuracy on values that were truly present barely moved, while the model's willingness to say “absent” changed by 0.107. That is why evaluation also looks at the specific behaviours that matter, not only one overall score.
3.2 Setting targets that have evidence behind them
Before spending on a training run, the platform states what level of performance would count as a useful result. The important question is where that target comes from.
locatability = fraction of values
present in the document
ATTAINMENT_OF_LOCATABILITY = 0.80
measured: 0.6126 ÷ 0.768
target = 0.80 × locatability
For extraction, we can cheaply measure how many correct answers are actually visible in the source documents. In one study, 76.8% were locatable and the model correctly extracted 61.3%, meaning it recovered about 80% of what was available to find.
That measured 80% conversion becomes an anchor: on a new task, we can first measure how much information is locatable, then use the observed conversion rate to estimate a realistic target.
“A target formula needs a measured attainment anchor, or it is a guess wearing a decimal point.”
A number such as 0.75 can look precise even when no experiment justifies it. At present, extraction has a measured 0.80 anchor and classification has a reference-model anchor. The reason and generate objectives do not yet have equivalent evidence-based targets.
provisional_target = floor + margin
goal.stop_reason:
if target_kind == "provisional":
return None # keep going
When no such anchor exists, we use a weaker requirement: beat the trivial baseline by more than normal run-to-run noise. That is honest, but it should be read as a minimum bar rather than a performance promise.
The improvement loop is not allowed to stop merely because it clears that minimum. Otherwise a model only slightly better than doing nothing could be declared complete. The run continues to its real stopping conditions and reports how far it actually went.
Each prediction also carries a stated uncertainty range based on how it was estimated. The platform then records predicted versus achieved performance. On the invoice programme, the estimator predicted 0.58 and the measured result was 0.6126, within the expected range and slightly better than forecast.
Some targets cannot be learned reliably because the labels themselves are inconsistent. For example, the same date may be written in several formats. When the desired output is not determined consistently by the input, more data alone cannot solve the problem. Such fields are capped rather than pretending unlimited training will fix them.
3.3 Deciding when training has gone far enough
Stopping too early leaves capability unused; stopping too late can waste money or encourage memorisation. The stopping rule therefore matters as much as the training recipe itself, and an earlier version stopped far too aggressively.
stop after N epochs with no
new best held-out score
regret = (true best over 100 epochs)
− (best-so-far at the stop)
DEFAULT_EPOCHS = 150
default_patience = max(20, 0.2 × budget)
The system keeps the best model seen so far and stops only after a chosen number of evaluation rounds without improvement. We judge the stopping rule by regret: how much better the result could have been if training had continued.
Because the best checkpoint is saved throughout the run, the stopping rule mainly controls how long we keep paying for training rather than directly deciding the final model quality.
| Patience | Mean regret | Worst case |
|---|---|---|
| 6 (the old default) | 0.1481 | 0.8083 |
| 20 (now) | 0.0025 | 0.0121 |
The worst cases matter. A patience setting of six could discard most of the performance that would have appeared later. A simpler patience of 20 matched more complicated trend detectors within experimental noise, so we prefer the simpler rule with a measured safety margin.
⛔ Two more elaborate approaches were rejected: detecting bends in the score curve, which mostly tracked noise, and trying to predict the future peak, which was unstable even between runs that differed only by random seed.
The early-training “dead zone”
Some exact-match tasks score zero for several epochs because the model cannot yet produce a valid answer format. A naïve patience rule mistakes that early flat period for a finished plateau and stops before meaningful learning has begun.
One reasoning run stopped at epoch 7 of 40 with a parse rate of 0.0. The same recipe allowed to continue reached a parse rate of 1.0 by epoch 25. Removing that premature stop alone moved the task score from 0.037 to 0.807.
The fix is to require a minimum amount of training and a basic readiness signal before the stopper is allowed to act. If the score is still at its floor and the training loss is clearly falling, the run is not treated as finished.
A related “still improving?” check had two hidden problems: it read a value from a file location that was never populated, and it required the final epoch to be the best epoch. It therefore returned false for all 15 runs in one study, even when substantial improvement was still occurring.
That false signal led the system toward the expensive recommendation “collect more real data,” even though the actual fix was simply to allow the existing training run to continue.
3.4 Choosing which trained version to keep
During training, the platform repeatedly tests the model on held-back examples and saves the best checkpoint. The score used for that choice has a large effect on everything reported afterwards.
Until August 2026, that decision used parse_rate × mean(field_accuracy). It multiplied two noisy quantities together while ignoring the more stable measure we actually cared about.
| Quantity | Mean epoch-to-epoch change | |
|---|---|---|
| parse rate | 0.069 | dominated the signal, carried least information |
| field accuracy | 0.021 | the informative one |
| their product | 0.056 | what was actually used |
The result was not merely a noisier chart; it sometimes selected the wrong checkpoint. In one run, accuracy improved from 0.675 to 0.712, but an earlier temporary spike in parse rate caused the system to keep a weaker model at 0.689. That was enough to explain a 0.058 gap between otherwise identical runs.
Selection now uses accuracy on the fields and cases that are genuinely present. Invalid output still fails naturally because it produces no correct field values, so removing the parse-rate multiplier does not remove the penalty for broken output.
3.5 Treating equivalent answers as equivalent
A model that returns “11/12/2025” when the reference says “11 Dec 2025” has found the correct date in a different format. Marking that as wrong would understate the model and could send the next experiment in the wrong direction.
| Type | How it is compared |
|---|---|
| number | parsed to a decimal (strip $ and commas); correct if |pred − gold| < 0.01 |
| date | compared as (day, month, year), so any format of the same date matches. A wrong date still fails |
| id | strip everything non-alphanumeric, ignore case |
| text / enum | ignore case and collapse spacing; region abbreviations resolve to one canonical value |
Comparisons are therefore chosen according to the kind of value being graded. One real dataset declared dates, amounts and identifiers as plain text, so they were initially compared character-for-character. Re-grading with appropriate comparisons improved some fields by 0.133 and 0.066.
The platform now infers additional structure from the field labels and applies a more suitable comparator only when doing so cannot make a correctly declared field worse.
Two formatting failures can also make otherwise correct JSON unreadable: copied raw line breaks and repeated blocks from greedy decoding. A conservative repair step that recovers key–value pairs, escapes control characters and keeps the first copy of a repeated key improved field accuracy from 0.675 to 0.759 and rescued 9 of 67 documents without retraining.
⚠ We still report the raw parse rate separately from the repaired rate. Combining them into one “success” number would hide whether the model itself is becoming more or less disciplined about output format.
3.6 Making sure the test is not accidentally easy
A benchmark is only useful if the model cannot pass it for the wrong reason. Three kinds of accidental shortcut have already produced misleading results in this programme.
Leakage
If test examples also appear in training, the score measures memory rather than generalisation. Duplicate checks therefore normalise harmless differences such as spacing and capitalisation. One corpus listed as 2,109 documents actually included the 67 test documents; the clean training count is 2,080.
A closed question set
A reasoning task once scored 0.807, but its generator had created every possible question before splitting the data. Every test question had therefore appeared in training. The score itself was real; calling it evidence of reasoning was not.
Shortcuts — the exam answerable without the skill
An internal image dataset was solved perfectly by counting coloured pixels rather than recognising the intended shapes. After the first fix, another accidental clue — the size of the shape's footprint — still gave away the answer. Both datasets looked successful until we deliberately searched for shortcuts.
We now measure a shortcut baseline: the score achieved by a deliberately simple method that exploits obvious accidental clues. On real CIFAR-10 photographs, the model scored 0.7634 against a shortcut score of 0.1521, making the result much harder to explain as a cheap trick.
The practical rule is simple: identify anything the model could use as an unintended clue, then vary or remove it. A dataset you created yourself is especially likely to contain patterns you did not intend.
Sealed splits and a canary
Final test sets are sealed with a recorded fingerprint, purpose and origin, and every opening is logged. We also run a small “canary” task with a known expected result, so a broken evaluation pipeline can reveal itself before a real experiment is trusted.
3.7 Comparing results on equal terms
A score means very little unless the conditions that produced it travel with it. Changes in training time, data, hardware, grading rules or opportunity to try multiple checkpoints can all change the apparent winner.
| Condition | What happened when it was ignored |
|---|---|
| Processor architecture | Same code, same data, same random seed: parse rate 0.881 on Intel-compatible against 0.955 on ARM. Never compare across machines. |
| Epoch budget | The same five studies read as a step at 14 epochs, flat at 40, and a gentle rise at 100. Three budgets, three conclusions. The cap was the finding — twice. |
| Field count | A mid-run score of 0.8301 covered 16 fields; the final 0.7442 covered 28. Read without denominators, the conclusion reverses. |
| Wall-clock time | Three byte-identical cloud configurations ran 561, 521 and 386 seconds per epoch. Rented machines differ. Cost is counted, never timed. |
For example, comparing the best point from a short run with the best point from a long run gives the longer run more chances to benefit from a lucky high score. Fair comparisons therefore need equal opportunity, not just the same headline metric.
Part 4 — What the experiments showed
4.1 Extracting information from documents
| Finding | Numbers | Standing |
|---|---|---|
| Copy supervision works — teaching the model to copy rather than invent | 0.5253 → 0.6126 (+0.087) | The one change clearly shown to help |
| The mechanism beats volume. 695 documents with copy supervision outperformed 2,109 without | 695 > 2,109 | More data did not replace the need for the copy mechanism |
| ⚠ But the mechanism is recall, not honesty: misses collapsed while fabrication rose | — | Recorded this way to avoid overstating the result |
| No capacity ceiling going from 16 to 28 fields | flat | No evidence that the task must be split because the model is too small |
| Fabrication dominates the error profile | 51.8% invented vs 2.1% wrong-but-visible | This identified invention as the main problem to address |
| More data helped, but every run shared one epoch cap | 0.3418 → 0.4560 → 0.6296 | ⚠ Provisional — being repeated without the old training limit |
4.2 Sorting text into categories
Across five classification studies, Frugal performed about as well as a conventional word-counting baseline: mean 1.009 relative to that reference, with standard deviation 0.044 and a range of 0.947–1.069. We record the working constant as 1.00, deliberately rounding down because the variation does not justify claiming more precision.
| Examples per category | Score relative to reference |
|---|---|
| 375 | 0.838 |
| 750 | 1.015 — first reaches the reference level |
| 1,500 | 1.069 |
Based on those results, the default data requirement was reduced from 1,500 to 750 examples per category.
An earlier rule said tasks with a reference score below 0.80 should be refused because performance appeared to drop sharply there. When the same study was repeated without the 14-epoch training limit, that sharp drop disappeared, and the very task the rule would have rejected reached 0.947 of the reference. The refusal rule has been retired.
⚠ There is still a mild pattern in that region, so we avoid replacing one overstatement with another. The evidence supports “a gradual rise with one stronger stretch,” not a clean threshold.
4.3 How much training time is useful
Across 15 classification runs using budgets of 150, 300 and 600 epochs, the average scores were 0.8843, 0.8662 and 0.8731. Those differences are within normal variation, so longer training did not consistently help classification in this setting. That is not a universal rule: in the vision work, increasing 100 to 300 epochs improved the control by 4.9 standard deviations.
4.4 Working with images
| Finding | Numbers |
|---|---|
| Genuine recognition on real photographs | 0.7634 vs a 0.1521 shortcut floor |
| Augmentation — small shifts and occasional mirroring | 0.7653 → 0.8426 (+0.077) |
| Why it works: the model had memorised its training images | train error 0.00033 → 155× higher |
⛔ The cheaper pooled design — a toy test promised +0.157 | −0.0025 at full scale |
| ⚠ Real run-to-run variation, three identical runs | 0.7592 / 0.7552 / 0.7739 → sd ≈ 0.010 |
The three repeated image runs show why small differences need replication: normal run-to-run variation was about 0.010, enough for a single run to look like a loss, no change or a win. Two further changes improved the score by only 0.0076, which is too small to separate confidently from that noise, and the proposed explanation for the improvement did not survive testing.
Part 5 — What we learned from mistakes
What earlier results taught us
We keep these corrections in the permanent record. Removing an incorrect result would make the page look cleaner, but it would also make the same mistake easier to repeat. Every issue below passed the software tests that existed at the time.
| Claim | What actually caused it |
|---|---|
| A performance step at reference ≈ 0.80 | A 14-epoch training cap |
e2e-reason demonstrates reasoning at 0.807 | Every test question also appeared in training |
| A perfect 1.0000 on an image task | Counting coloured pixels; then, after the fix, shape extent |
The pooled design gives +0.157 | A toy test; at full scale it is −0.0025 |
| Aleatoric share diagnoses what to fix | It has no separating power — a 100%-accurate field scored 0.905 |
| Synthetic supplementation is harmful (−3.4 pts) | The test set contained almost none of the rare fields it targeted — the question is still open |
| Curriculum staging helps | Finished 0.0286 below ordinary training; removed as default |
| Confidence calibration can time the stop | 15–20× noisier than the score already in use |
| All comparisons before 4 August 2026 | The stopper kept the wrong model and seeds were not fixed |
| A copy-supervision A/B result | Its masking made valid copying impossible; loss went to NaN while the run reported succeeded |
| Stage 1 of E1 teaches comprehension (validation 0.35 → 0.62) | The learning rate restarted and 4,324 new boards arrived at the same time; a control run was added before the result landed |
| E1's stage-1 floor is 0.712 | A rate quoted beside a reward ratio; the floor in the right units is 0.989, leaving one point of room |
| Asking good questions is the fair ceiling for a dialogue | Fewer than half of an agent's unknown scores are visible to its partner, so asking barely beats random; telling your best scores is the rule to beat |
- Register the prediction and the decision rule before seeing the result. A threshold chosen afterwards is a description, not a test.
- Record the full recipe beside every number — epochs, data, machine, grading method. The one condition you fail to record is the one that turns out to matter.
- Three points minimum for a trend. Two show a rise, not a slope.
- Use measures that cannot be won by guessing the commonest answer.
- Examine the errors before designing the remedy. Error analysis prevented two badly-aimed experiments here.
- Prove your safety checks can actually fire. Several silently never did.
- A numerical fit is a clue, not a cause.
Any regularly repeated operation will eventually become a problem if its cost grows with the entire history. The dashboard hit this three times. In the clearest case, one status request took 275 seconds and reached 1,226 seconds in the worst case because it searched 1.3 million stored objects to find about 2,500 relevant ones. The lasting rule is that routine work should have a fixed upper bound on how much history it must scan.
Part 6 — The new multi-agent strategy
The new multi-agent question: can conversation create better joint decisions?
The current multi-agent programme no longer treats a hand-designed chain of tasks as evidence of “emergence.” Those earlier workflow, lattice and decomposition studies are still useful engineering evidence, but they tested systems whose routes and responsibilities were largely designed in advance. The new study asks a harder question: can identical Frugal agents discover a useful way to cooperate through conversation itself?
Dialogue-Induced Collaborative Optimization is the active emergence experiment. Two copies of the same frozen Frugal model receive different pieces of the same problem and communicate through a deliberately limited dialogue. Their model weights do not change during the episode. What may improve is the quality of their shared solution as useful messages accumulate.
The task: DialOp Assignment
The main benchmark is DialOp Assignment. Two agents must agree on a one-to-one matching between eight reviewers and eight papers. Each agent sees only part of the underlying 8 × 8 preference table, and the numbers visible to one agent are on a different scale from the numbers visible to the other. Neither agent has all the information needed for the best decision, so useful cooperation requires them to decide what to share, what to ask, what to revise and when to agree.
The benchmark is attractive because the final answer can be scored exactly. We can also compare the dialogue with strong alternatives — including one agent given both private views — rather than only comparing it with an artificially information-starved individual.
What the agents are allowed to do
Both participants use the same Frugal checkpoint. We do not pre-assign one model to be a “critic,” another to be a “questioner,” or another to be an “optimizer,” because that would build the organisation into the experiment. On each turn an agent can share information, ask a question, revise its private candidate, make a proposal, accept or reject. The conversation transcript is public, while each agent's private information and private candidate remain private unless it chooses to reveal them.
A simple Conductor keeps the conversation legal, alternates speakers, records cost and applies the stopping rules. It is deliberately not an intelligent manager: it does not choose who should speak, repair answers, reveal hidden scores or pick the best earlier proposal after the fact.
The study uses a four-level claim ladder rather than one vague “emergence score.” First, does the team use distributed information better than an isolated agent? Second, does genuine back-and-forth help more than simply pooling summaries or exchanging proposals? Third — the primary target — does the raw quality of the agents' candidate solution improve over successive dialogue rounds more than it does in matched controls? Fourth, and strongest, can the dialogue team beat a single agent that receives both private views and the same model-call and communication budget?
Level 4 is allowed to be false. The experiment is designed so a negative result remains informative rather than weakening the comparison until the team appears to win.
Teaching the members before the experiment
A model trained from scratch has no idea that a message is for anything, so the members are taught in steps. The steps are allowed to be strict about form — a legal answer, a complete matching, a message that fits the budget, the best matching for the information you hold — because none of that is the claim under test. They are never allowed to teach policy — what to say, when to ask, who does which part — because that is the claim, and a habit installed early would still be there in the final experiment. Relaxing the teaching later does not remove it: each generation is frozen and the next one learns from its traces.
| Stage | What it teaches | How we know it is time to move on |
|---|---|---|
| 0 — form | Produce a legal record and the best matching for your own view | Legal answers at least 95% of the time, and matchings better than random |
| 1 — comprehension | Given your view and facts a partner has stated, produce the best matching for both together | Changing one stated number moves the answer in the direction it should |
| 2 — self-play | Nothing about what to say. Frozen members play fresh boards; the conversations that went better are kept and the next generation learns from them | When a new generation no longer improves |
| 3 — freeze | — | The comparison arms run on this frozen generation |
The first member, trained for 36 hours, produced a legal record 99.2% of the time and found good matchings from its own view: 0.860 on a scale where random scores 0.697 and the best possible answer scores 1.0. That is about half the distance from random to perfect, and it found the best matching on only 1.3% of boards. So it is competent, not excellent.
Its messages were another matter. Only 17% stated a true fact. 37% named a score the member could not even see, and 44% gave the wrong number for a score it could. Nothing in stage 0 rewarded telling the truth, because which fact to state was deliberately left random, and the model learned the sentence shape without learning to make it true.
Stage 1 is training as this is written, and its validation score rose from 0.31 to 0.62 against stage 0's peak of 0.35. That looks like comprehension. But three things changed at once when stage 1 started: the learning-rate schedule restarted, 4,324 new boards arrived, and the partner's statements appeared in the prompt. Only the last is the treatment. We checked that the stage-1 answers are not simply easier to hit — they are not; the tie structure is identical — so the jump is real learning, and most of it may be plain extra training. A control run was therefore started the same day: the same starting weights, the same schedule, the same boards, and an empty transcript. Whatever it gains is training; whatever stage 1 gains beyond it is the transcript. A registered floor was also found to be in the wrong units the same morning — a rate of 0.712 quoted beside a reward ratio whose true floor is 0.989 — and corrected before the run landed.
Controls and communication limits
The primary dialogue has a communication budget that is intentionally smaller than the cost of simply dumping both private tables into the transcript. This forces the agents to communicate selectively. Every treatment is compared on the same episodes, and the study records model calls, generated tokens, generated bytes and deterministic counted cost.
Two rules about form were added in September. First, a statement must be true. If a member says “reviewer 3 on paper 5 is 649,” that cell must be one it can see, at the number it sees; otherwise the turn is illegal, exactly as an incomplete matching is. This says nothing about which fact to state or when — that remains the agents' business — but without it the channel the agents learn from would be four parts noise to one part information. Second, a question may be given its own small allowance so that asking does not cost a third of a speaker's whole budget while carrying no information itself. Under the original prices, telling always beat asking, so a policy that asks could never have appeared. A question that smuggles a fact inside it is still charged to the main budget. The number of turns and the size of the question allowance are fixed by the unsealed pilot, before any real result is seen, and before self-play begins — the members learn the shape of the prompt they train on, so the shape must be settled first.
A second set of test boards is also generated, on which the two agents' knowledge overlaps less and cooperation has more to win. It is reported beside the main set and never mixed into it. On the main boards a perfect single agent already scores 0.89, so the room for any team gain is about eleven points; on the harder set it is about eighteen. If an effect exists, that is where it will be visible first.
| Comparison | Plain-language purpose |
|---|---|
S-local | What can one agent do with only its own information? |
S-pooled | What can one agent do if it receives both private views in a single call? |
S-pooled-reflect | If one agent gets the same overall call and byte budget, is extra thinking alone enough? |
T-one-shot | Does a single exchange of compressed information work as well as a conversation? |
T-proposal-only | Do words and questions add value beyond exchanging candidate solutions? |
T-dialogue | The main treatment: full alternating conversation. |
T-corrupt | If realistic-looking partner messages have the wrong meaning, does performance fall? This tests whether message content actually matters. |
Measuring how good the cooperation becomes
“Does cooperation emerge?” is a yes-or-no question. “How good can it get?” needs a ceiling to measure against. Before any dialogue has been run, we fixed one by computing, exactly, what a conversation of a given length could achieve on each board under four different policies. A team's real score is then placed on that scale, so the result is a position rather than a pass or a fail.
| Rung | Who decides what is said | After 1 statement | After 4 | After 8 |
|---|---|---|---|---|
| Alone | Nobody — the agent uses only its own view | 0.871 | 0.871 | 0.871 |
| Random | The partner states true facts, chosen at random. What a team that only shares should reach | 0.885 | 0.917 | 0.948 |
| Fair ceiling | The better of two simple rules a real agent could learn: ask for the unknown score you expect to matter most, or volunteer your highest scores | 0.928 | 0.977 | 0.991 |
| Omniscient | An oracle that knows the whole table picks the most useful facts. Not reachable; it says how much there was to win | 0.958 | 0.991 | 0.994 |
Three things follow. The budget is not what limits quality. Within the communication limit the study already has — about eight statements — the ceiling is 0.99, against 0.87 for an agent alone, so twelve points of room exist without loosening anything. What gets said matters more than how much. At four statements, saying true things at random buys under five points; saying the right true things buys twelve. Asking on its own is nearly worthless here, because fewer than half of the scores an agent cannot see are visible to its partner, so most well-aimed questions draw no answer. That is why the fair ceiling is the better of asking and telling: a scale that used asking alone would have been beaten by a team that simply announced its best scores. A team's selection efficiency is the share of the distance from the random rung to the fair ceiling that it actually covers. Above one means the conversation is doing something neither simple rule can.
The 134 recorded human conversations on this task averaged 0.921 over about eighteen messages each. That is the random rung. People talking at length did about as well as stating true facts without choosing them, so “better than humans” is a modest target here, and the scale is deliberately not capped by the human reference.
To see whether room or policy is the limit, the same frozen team is also run under four budgets — half the normal limit, the normal limit, double, and unlimited — so quality against room becomes a curve rather than a single point. Only the normal limit counts as the claim; the others are diagnostics, and the unlimited setting fails the bandwidth rule by design. The curve is redrawn for every self-play generation. How good the cooperation becomes is that curve moving upward while the room stays fixed, and a generation that no longer moves it is the signal to stop.
How the experiment progresses
Work is staged. Phase 0 audits and reproduces the benchmark. Phase 1 builds the dialogue runtime. Phase 2 checks that a single Frugal member can produce legal matchings and genuinely use transcript information. Phase 3 runs a small unsealed pilot and freezes the communication budget, turn count and statistical thresholds. Phase 4 is the sealed E1 evaluation. Only after E1 is complete does the programme move to a second problem to test whether any learned collaboration pattern generalises.
The research definition, benchmark, control arms and staged plan are fixed, and the dialogue runtime, dataset sealing, exact scoring and fresh episode generator are built. The stage-0 member has passed its two gates. The stage-1 member and its control run are training and land within a day of each other. Ten amendments to the plan were registered and built this week: the truthfulness rule, the question allowance, the harder second board set, the training control, the quality scale and its budget ladder, behaviour measurements that read a policy directly from the transcripts, a fairer way of choosing which trained version to keep, a second version of the stage-1 test in the shape self-play will produce, and a written bar the members must clear alone before any team is paid for. Two settings remain for the unsealed pilot to fix — the turn count and the question allowance — and self-play, the pilot and the sealed E1 experiment are still to come.
Part 7 — Current working settings
Current defaults — useful settings, not universal laws
| Decision | Current setting | Reason |
|---|---|---|
| Extraction target | 0.80 × locatability | Measured conversion rate; being repeated without the old training limit |
| Classification target | 1.00 of reference | Mean 1.009, variation 0.044; rounded down deliberately |
| Data for that target | 750 per category | Where performance first reaches the reference level |
| Label-inconsistent field cap | 0.30 | The labels are too inconsistent for the input alone to determine one answer |
| Training budget | 150 epochs | High enough that it should rarely stop a promising run purely for cost |
| Stopping patience | max(20, 0.2 × budget) = 30 | Average missed improvement 0.0025; worst case 0.0121 |
| Seed-noise floor | 0.05, one-sided | Measured run-to-run variation; decision margins are based on it |
| Curriculum staging | off | Measured 0.0286 worse than ordinary single-stage training |
| Extraction corpus | 2,080 clean documents | The old 2,109 included the test set |
| Loop limits | 3 attempts · 96 h · $100 | Places a clear limit on repeated automated attempts |
Part 8 — Questions still open
What is still unresolved
- The extraction target. The 0.80 conversion factor was measured under the same kind of training limit that caused problems elsewhere. The invoice data ladder has not yet been repeated without that cap, so 0.80 remains provisional.
- The image-performance ceiling. We do not yet know whether a longer training budget produces a genuinely better final model or simply moves the best point later in training. This is harder to separate because the budget also controls the learning-rate schedule.
- Rare fields. The observed deficit was +0.2493, but later analysis showed that much of it came from the model declining to answer rather than lacking the underlying capability. The true cost is therefore estimated between +0.02 and +0.24, not the raw headline number.
- Synthetic extra training data. This is still unanswered because the original test set barely contained the rare fields that the synthetic data was intended to improve.
- How much classification data is enough. We know 750 examples per category can reach the current reference level, but we have not yet mapped the point where adding more data stops being useful.
- Whether useful collaboration truly emerges from dialogue — and how good it can become. E1 asks whether two identical frozen Frugal agents can progressively improve a joint solution through limited conversation, and whether that interaction adds value beyond pooling information, exchanging proposals or simply giving the same compute to one agent. The ceiling is now measured: within the current budget a well-chosen conversation could reach 0.99 against 0.87 alone, and people reached 0.92. Whether trained members choose well is exactly what is not yet known, and the result is not assumed in either direction.
The main lesson from the programme so far is simple: a result only means what we think it means when we can explain exactly how it was produced. Many of our strongest corrections came from discovering that the experimental machinery had shaped the outcome more than the idea being tested.
Methods referenced: BitNet b1.58 · Mamba/Griffin selective recurrence · Kogge–Stone prefix scan · Switch/GShard load balancing ·
Bayes by Backprop · pointer-generator networks · straight-through estimation · AdamW · RMSNorm