Small models, measured
Frugal builds small AI tools, each trained for a specific job, and tests how well they work. This page explains what Frugal offers, what three months of testing have taught us, and what we are working on next.
A small AI tool for a specific job
Large AI systems can do many things, but that flexibility can bring extra cost, energy use and waiting time. Many business tasks need a much narrower set of skills. Frugal builds a specialist: a small AI model, or trained computer program, for a job such as reading invoices, sorting customer messages or identifying objects in photos. Once trained, it can run on an ordinary laptop without sending documents to an online AI service.
The question behind all of it: how small can a model be and still do a real job well, and can we prove it?
Say what the model should read and what answer you need.
Use your documents, public collections of data, or examples written by a larger AI. Each needs an answer we can check.
It learns from the examples on a rented computer chosen for the job. The computer shuts down when the work ends.
Test it on examples it has never seen. Compare its score with guessing and simple shortcuts.
The result is a specialist you can download and run on a compatible computer. You manage the work through the control room, a website where you set up jobs, watch learning progress, read reports and compare attempts. An assistant can explain the results and suggest what to try next.
What makes a Frugal model different
Small by design
A model has about 9–25 million internal settings, far fewer than a large chatbot can have. Its main settings use just three values: −1, 0 and +1. Doubling the amount of text roughly doubles the reading work, helping keep long documents affordable.
Learning to copy accurately
A built-in copying feature can take reference numbers, dates and amounts directly from a document. We are testing whether teaching it when to copy reduces made-up answers. Requiring exact copying every time is different: it can reject correct answers that use a different format, so we are removing that requirement.
Flagging uncertain answers
The model estimates how sure it is about each answer, so doubtful answers can be sent to a person. Our tests found that it became better at recognising uncertainty as it learned.
Checking more than the score
We compare each result with a shortcut score: what guessing or exploiting a clue could achieve without learning the task. Final test examples are kept separate, access is recorded, and each result identifies the software version used.
You approve proposed actions
A model can learn to prepare an email or calendar entry. A separate program carries out the action only when the model is confident enough and a person confirms.
Text, pictures and teams
The platform supports text, pictures (added in August) and teams of models (added in September). These parts are kept separate to reduce the risk that changing one will disrupt another.
Work that can resume
Jobs can pause, resume and recover from some failures. Before a study starts, we record its plan and what counts as success. That lets readers judge the result against the original goal.
A plain-language control room
See progress charts, readable reports and a map linking related experiments. Ask the assistant to explain a result, or watch two models exchange messages live or in saved recordings from different stages of learning.
Frugal, in full
Frugal helps businesses build and own small AI tools for specific jobs. Describe the task, gather examples, teach the model and check the results through one website. Learning takes place on rented computers; the finished model can work offline on an ordinary computer. You can see how it was built, compare its results with simple shortcuts and ask an assistant for help. The platform also supports teams of models and studies of how well they work.
From a sentence to a plan
- Help writing instructions: describe the job and a larger AI drafts a plan, including the details to return, such as a date or amount. We call each of these details a field.
- Five kinds of job: find details in documents, sort items into categories, prepare an action, work out an answer or write text. Each has its own way of measuring success.
- A choice of model sizes: choose a setup suited to the job's difficulty and the examples available.
- A realistic goal: the platform estimates what the examples might allow the model to learn, then asks you to confirm the target.
- Time and cost estimates: see a likely range before starting. Jobs too large for the computer's memory are stopped before they begin.
Your documents, public data, or a teacher
- Upload your own: provide documents with the correct answers marked, visible only to authorised users.
- Find public examples: search data libraries including Hugging Face, Kaggle, Zenodo, OpenML, UCI and data.gov.au. The platform checks their size, contents and terms of use.
- Check suitability first: test a small sample to see whether it contains the information your task needs before downloading the full collection.
- AI-written practice material: Claude, ChatGPT or Gemini can write examples. A review step rejects examples that do not follow the required answer rules.
- Spot problems early: a report flags inconsistent answer names, easy shortcuts, examples that are too long and test examples accidentally included in the practice material.
- Keep usage rights visible: the model includes a record of each data source's licence and whether it permits business use.
Small, cheap, and honest about doubt
- Simple internal settings using −1, 0 and +1, with a reading design that keeps a fixed amount of working information as it moves through a document.
- A copying feature for taking reference numbers, dates and amounts directly from the page.
- Work shared inside the model: only the relevant parts work on each piece of text, reducing the work needed.
- A confidence estimate: each answer includes how sure the model is. Tests check whether its uncertain answers really are more likely to be wrong.
Each job on its own machine
- A computer suited to the job: each stage uses a rented computer chosen for total cost, with specialist AI hardware where needed. Computers shut down when there is no work waiting.
- Stop when progress stalls: the platform checks separate practice-test examples and can stop learning when scores stop improving, while allowing time for learning to begin.
- Pause and resume after the last completed pass through the practice examples.
- Recover from failures: where safe, a failed attempt can restart from its best saved model. Software faults are reported instead of repeatedly retried.
- Record the software used: every result names its software version, making differences between experiments visible.
Scores you can trust
- Separate learning and testing: use 70% of examples for learning, 20% for progress checks and 10% for the final test. Every access to the final test is recorded.
- A comparison with shortcuts: show what guessing or exploiting an easy clue would score alongside the model's result.
- Check misleading success: flag models that always say information is missing, always choose the same category or express confidence that their results do not justify.
- Test changes in presentation: remove labels, shuffle lines and rename the details the model is asked to find.
- Readable reports: download reports as text, web pages, PDF or Markdown, with every number checked against the recorded results.
- Compare experiments fairly: flag differences in software or in the rules used to turn a model's predictions into answers.
Make it better on purpose
- Practise difficult cases: collect mistakes and create more examples of the details the model struggles to read.
- Learn from your corrections: review answers, fix them and give the model further training on those fixes.
- Reuse a successful model as the starting point for specialists in related jobs.
- Try again within limits: work toward a target with a maximum number of attempts, time and spending, then return control to you.
- Teach in stages: prepare examples, demonstrate, coach, gradually reduce help and finally test the skill.
Take it anywhere
- A downloadable specialist that runs offline without specialist AI hardware. It needs Python and NumPy, two freely available software tools; optional add-ons can make it faster.
- Read whole folders of documents and flag uncertain answers for someone to check.
- Prepare actions: draft an email, calendar entry or link to open. A separate program acts only when confidence is high enough and a person confirms.
- A directory of specialists that a larger AI can use to find the right model for a job.
Many small models, one question
- Connect models into a team: let specialists share a job, and measure both the team's result and each member's performance.
- Understand the teamwork: see the cost of handing work between models, where errors begin and what each member adds.
- Watch conversations: see two models discuss a decision live, or replay conversations from saved stages of learning.
- Plan studies in advance: record the methods being compared, the steps and the pass marks before starting. Only planned jobs can join the study.
- See related experiments: a study map shows how the recorded attempts connect.
See everything as it happens
- Live progress: charts show whether mistakes are decreasing and scores on separate examples are improving, alongside time and cost so far.
- One overview per job: what it is for, what it found, why it stopped and what to do next.
- A history of team activity you can replay in the order it happened.
- A choice of screen layouts: Classic and Workspace, plus a phone layout. Install the website as an app and use your browser's text-size settings.
Hardware kept busy, costs in view
- A waiting list for jobs: you or the study plan set priorities, and longer-waiting jobs move up the list. Jobs needing specialist AI hardware start when a suitable computer is free.
- Alarms when a machine sits idle while work waits, or a planned step is never submitted.
- Cost tracking for each attempt and organisation, including rented computers and charges for using larger AI models to write examples.
- A computer overview: see active jobs, waiting jobs and problems starting them.
A guide that explains
- Answers questions about runs, results and costs, grounded in the platform's own documentation.
- Fills in forms and schedules work after your confirmation, while respecting the platform's limits on job size.
- Proposes data fixes but never applies them; only a person can.
Built for more than one organisation
- Controlled sign-in: staff use Google accounts and clients need an invitation. Access permissions are checked whenever the platform is used.
- Organisations kept apart: each sees only its own runs and costs, with roles including view-only.
- Tools for administrators: a second person approves invitations, access is reviewed daily, and administrators can check usage and computers.
- A protected activity record: records of system access and data viewed are kept for 12 months.
What experiments taught us about small models
Grouped by what the model works on: language, pictures and actions, plus teams of models working together.
Language · reading and understanding text
Tests used real invoices and counted details actually present in each document. In a separate comparison, teaching the model to copy raised accuracy from 0.53 to 0.61.
The task and model design stayed the same as we added examples. A value of 4× means four times as much reported uncertainty on wrong answers. In a test using more examples, this reached 10.2×.
Strengths
- Copying helps it find the right details. A model taught to copy using 695 documents beat one taught without copying using 2,109.
- It can sort text into categories. Across five collections of examples, it roughly matched an established method, averaging 1.01 times that method's score, with about 750 examples per category.
- It reads familiar layouts well. Our best reader scores 0.97 on new documents presented in formats it learned during practice.
Weaknesses
- It can make up answers. About half of its wrong dates appear nowhere in the document. In reports filed with the US financial regulator, the SEC, made-up values account for 51–68% of each model's errors. The larger model makes up the fewest.
- Uncommon details are harder to learn. When a detail appears in only about 20% of documents, its accuracy drops by about 0.21. Reference numbers suffer most.
- Questions with several steps remain difficult. Even on individual steps from a public comparison test, the model scored only 0.09, mostly because it copied answers incorrectly.
Each pair of bars shows one model. A field is a detail we ask it to find, such as a date. Learning only one name for each field left the larger reader scoring 0.01 when names changed. Practising several names raised scores to 0.28–0.31; practising 12 names raised them to 0.59–0.67. None met the goal of staying within 0.05 of its score on familiar names. Bigger models did not do better here. Remaining errors mostly involve misspelling the requested field name, sometimes beside the correct answer, or making up the answer. The chart's generalist practises several document types; N-PORT and 13F are types of financial report read by the specialists.
Vision · understanding pictures
We slightly shifted or mirrored the practice photos to make memorising them less useful. This raised accuracy by 0.077, about eight times the usual variation between repeated attempts.
Strengths
- It learns to recognise picture content. Its score of 0.76 is about five times the shortcut score of 0.15.
- Varied examples make a big difference. Small changes to practice photos helped the model recognise new photos instead of relying on memory.
- More practice helped. Longer training improved results, showing that the earlier scores were not the best this design could achieve.
Weaknesses and lessons
- Some practice pictures had an unintended shortcut. Counting the tiny dots that make up an image gave a perfect score of 1.0 without recognising its content. Our usual check missed this.
- A promising small test did not hold up. A design change that improved a simplified test by 0.16 reduced the full test's score by 0.003.
- Tested on one photo set only, and never on scanned documents.
Action · doing things
Teams · several small models working together
When members were switched off one at a time, the win came from one strong specialist, not from teamwork.
Two copies of the same model practised solving puzzles together by exchanging messages. They learned to reach agreement in every game, up from 14–69%, but their messages helped no more than randomly chosen messages. A simple rule for choosing useful facts did much better.
A simple teaching rule supplied examples of useful messages. After 46 rounds, ending 3 October, the model finished at 0.892, just below the 0.90 target. Its best score was 0.898 at round 44, and improvement was slowing. Messages helped its partner by almost as much as the rule's messages did: 0.0245 versus 0.0248 on the study's measure of benefit. The final agreed answer scored 0.960 versus the rule's 0.969. This shows that the small model can learn useful communication from examples.
Each specialist learned one type of document; the generalist learned an equal mix of all three. All specialists comfortably passed their readiness targets of 0.76–0.79. Yet each scored within about 0.01 of the generalist, and only two scored higher. Specialising has not yet produced a clear advantage for a team to build on. The larger model leads on all three document types.
Only about a fifth of the specialists' mistakes are shared by all three. If we could always choose the correct answer whenever any member had it, the score would be 0.974, above the best member's 0.948. This is an ideal result, not a result achieved by a working team, and it is only slightly above the larger model's 0.971. Requiring every answer to be copied exactly from the document made all models worse, by 0.07–0.20. A likely reason is that correct answers sometimes need a different format: for example, 1234.56 when the document says 1,234.56.
What we asked, and what came back
Each study's question and pass mark were written down before it ran. A clean failure is still a result, and several of the most useful lessons came from failures.
Text
| Study | The question | Models | What we found |
|---|---|---|---|
| Learning to copy | Does teaching the model to copy reduce made-up answers? | Invoice readers | Win Accuracy rose from 0.53 to 0.61. In these tests, teaching copying helped more than adding documents. |
| More examples | Does accuracy keep rising with more practice documents? | Invoice readers | Win Scores rose from 0.34 to 0.46 to 0.63 as we doubled the examples twice. |
| Sorting text | Can a small model match an established method? | 5 data collections | Win It roughly matched the comparison method: 1.01 times its score on average, with about 750 examples per category. |
| Time spent learning | Does more practice always help? | Text-sorting models | Finding corrected Beyond a point, extra practice only added cost. An apparent jump in earlier results came from a limit we had set on training. We withdrew that claim twice. |
| Teaching in stages | Is it better to teach a few details first, then add more? | Invoice readers | Fail Teaching all details together scored 0.029 higher, so staged teaching is no longer the default. |
| Uncommon details | How much harder is it to read details that rarely appear? | Invoice readers | Partly tested Accuracy fell by 0.21, with reference numbers affected most. We have not yet tested whether made-up practice examples can recover the loss. |
| Questions with steps | Can a small model handle one step from a public comparison test? | 1 small model | Fail It scored 0.09 against a target of 0.11. Errors involved copying; changing how its predictions became a final answer raised the score to 0.12. |
| Reading text in pieces | Does recognising a wider range of word fragments help? | 1 small model | Made it worse No. The score fell from 0.08 to 0.03. |
| Addition | Can a small model add numbers, and what trips it up? | 1 small model | Win, with a surprise Reading numbers one digit at a time raised the score from 0.01 to 0.99. |
Pictures
| Study | The question | What we found |
|---|---|---|
| Recognising pictures | Does the model learn what a picture shows? | Win It scored 0.76, about five times the score achievable through a simple shortcut. |
| Checking the examples | Can the model succeed without recognising the picture? | Problem in our data Yes. Counting the tiny dots in the practice images gave a perfect 1.0. We corrected the examples and the checks. |
| Varied photos | Do slightly changed copies of each photo help? | Win Accuracy rose by 0.077, the largest picture improvement, by reducing reliance on memorised images. |
| A simpler design | Does a design that worked in a small, simplified test work at full scale? | Fail A gain of 0.16 in the simplified test became a loss of 0.003 in the full test. |
| More practice | Does spending three times longer learning help? | Win The score rose by 0.038, showing that the earlier results could still be improved. |
Teams of models
| Study | The question | What we found |
|---|---|---|
| Passing work along | Does adding more models to a chain improve the answer? | Test needs redesign We changed both the chain's length and the difficulty of its steps, so their effects could not be separated. Errors built up: the fifth model lost about 85% of its measured ability. |
| A planned team | Can small specialists beat a larger model on real financial reports? | Higher score The team scored 0.97 against 0.63, but checks showed that one strong specialist accounted for the success. |
| A public comparison | Can one small model compete on a public test used to compare AI systems? | Not yet It scored 0.97 on our familiar formats but nearly 0 on the public test. It had relied on memorised field names. Further attempts to fix this are described below. |
| New names, larger model | Does practising different names for the same detail help our best reader? | Below target Accuracy on new names rose from 0.013 to 0.278, while familiar names stayed at 0.975. It missed the 0.92 target. On 29 September we chose it to replace the earlier model, A3, in the final test because it handles new wording much better. |
| Practising 12 names | Do 12 names and short descriptions for each detail help with unfamiliar wording? | Partly New-name scores rose from 0.25–0.35 to 0.59–0.67. None came within 0.05 of its familiar-name score, the required target. The larger reader scored 0.590; the smaller model taught an equal mix of document types did best at 0.673. |
| Copying with new names | When names change, does requiring exact copies of text, reference numbers and dates help? | No clear difference The average score fell by 0.003, and answers more often used the wrong format. On 6 October we decided to use exact-copy rules in the final test only if they first do at least as well on practice tests. |
| Ready for a team? | Are the three specialists good enough to test together? | Passed All comfortably passed. But each scored within about 0.01 of a same-sized model taught several document types. |
| Different mistakes? | Could the specialists cover for one another's errors? | Some room to improve Always choosing a correct answer when any specialist had it would score 0.974. That compares with 0.948 for the best member and 0.971 for one larger model. No working team has achieved that ideal score. |
| Always copying exactly | Does requiring every answer to be copied exactly remove errors? | Fail All models got worse, losing 0.07–0.20. Exact copying probably blocks correct answers that need a different format from the document. |
| Copying selected details | Does requiring exact copies only for text, reference numbers and dates help? | Slightly worse Scores fell by 0.005–0.015. Models made up fewer answers but left more gaps. This needs another check on the public test's practice examples before the final test. |
| Understanding the errors | Are wrong answers misread or made up? | Mostly made up Between 51% and 68% of each model's errors are answers that appear nowhere in the document, even in a different format. |
| New names, four models | Can the small models handle unfamiliar names for details? | Fail Scores fell from 0.92–0.95 with familiar names to 0.25–0.35 with new ones. Changing the surrounding instructions no longer caused problems. |
| Discussion, first attempt | Can two copies of one model make a better decision by talking? | Stopped after review The test design was flawed: each model had to solve the whole puzzle itself, and the allowed messages did not include "I accept". |
| Discussion, redesigned | If a separate program handles solving the puzzle, can the models learn useful discussion? | No success They learned when to agree, but their messages were no more useful than random ones. |
| Finding the cause | Why did the models fail to learn what to say? Checked 27 September. | Cause found The score used to guide learning gave too little credit to useful messages. A more detailed score reflected their usefulness about twice as clearly. |
| Six short trials | Can more detailed feedback teach useful messages? | Learning failed All six attempts failed within one or two rounds, and most models stopped sending messages. We traced this to the teaching method. |
| Checking the fix | Does the repaired teaching method change behaviour as intended? | Passed The model changed when instructed and stayed steady otherwise. One strict check failed because of how it measured change. A check of actual behaviour passed, with scores of 0.862 to 0.869 over five rounds. |
| Learning from examples | Can a model learn useful messages by copying examples from a good rule? | Nearly After 46 rounds, about 89% of messages were as useful as the rule's, just below the 90% target. The best score was 0.898. The help given to its partner almost matched the rule's, showing that a small model can learn this skill. |
| Six trials after the fix | Does more detailed feedback work once the teaching method is repaired? | No All models kept talking and agreeing, but scores of 0.516–0.555 missed the 0.60 target; chance scores 0.50. These feedback methods were not enough, so we moved to teaching with examples. |
| Apparently missing answers | When an answer seems missing under a new field name, what did the model write? | Cause found In 87% of these cases, it misspelled the requested field name, sometimes beside the correct value. For example, irs_processing_sit drops the last letter of "site". It rarely used another field's name. |
| Had copying been taught? | Did these models receive lessons on when to copy? | Earlier view corrected No. The copying feature was available, but they had not been taught when to use it. We therefore cannot blame their failures on copying lessons. The next study tests whether those lessons help. |
| Learning from errors (T2) | Does practising past mistakes with new wording help? | Mixed Focusing practice on naming mistakes beat ordinary practice for both models, by 0.046 and 0.038. It was the only method to improve both unchanged starting models, by 0.025 and 0.040. Against extra practice on familiar examples, it won for only one model. Each method was tried once, and normal variation could explain gains this small. Tests on two more models are queued. |
| Copying and new wording | Does teaching copying help with unfamiliar names? | Under way We are comparing four setups: the current method, lessons in copying, copying field names from the request, and no copying feature. We also repeat the current method with a different random starting setup. Two versions are learning and two are waiting; all face familiar and new names. |
| A message tutor (T1 step 0) | Can a teaching program score every possible message without too much cost? | Yes It can calculate exact scores under its chosen rule, taking about 0.9 hours per round. The original rule chooses a best-scoring message only 42% of the time, leaving room for a coached model to do better. |
| First coaching test (T1-Q) | Can a tutor teach messages that are better than the original rule's? | Fail All measures fell in round one: the share of messages as good as the rule's dropped from 0.89 to 0.40. An automatic limit on behaviour change stopped training at round 7. Too many choices received the same top score to guide learning clearly, and some scored well only because of the tutor's assumptions about later moves. |
| Revised coaching (T1-Q2) | Does copying the original examples, with a few clearly better replacements, work better? | Initial check We are trying one round first. The full test starts only if the model maintains its score, makes clear choices and keeps sending useful messages. |
| Asking useful questions | How often does the partner have the information needed to answer? | Measured For the model taught by examples, the share is 0.371, below random questions at 0.408. A good question-selection rule reaches 0.499, so the target is 0.450. |
| When talking helps | Which puzzles leave more room for useful discussion to improve the result? | Mostly answered A bigger board or less information for each model made little difference: communication added 0.063–0.069. Only puzzles selected using stricter requirements increased the benefit. |
| More demanding puzzles | How much does communication help on puzzles selected to need it? | Answered Good messages added 0.107 to the score instead of 0.070, a 53% larger benefit, confirmed on new puzzles. Each puzzle takes about 17 seconds to create, so we made a reusable collection of 4,328. |
The original method scored the answer the pair agreed on. That score changed little with the messages they exchanged, giving weak guidance about what to say. Scoring each partner's own best answer reflected message quality about twice as clearly. Combining the two scores weakened that connection. The mixed versions below give the individual-answer score increasing weight. Replaying messages to measure their effect gave the strongest connection, but was costly and varied between attempts. The short trials listed above tested these ideas.
Teaching works, but how we coach matters
Sharing useful information. By practising examples from a good teaching rule, the model learned to send messages that were just as useful 89.2% of the time, narrowly missing the 90% target. Our first attempt to improve on that rule made results worse: too many choices were treated as equally good, making the model less decisive. The revised method keeps most of the original examples and replaces a few with clearly better messages. An initial one-round check is under way.
Understanding new wording. Practising 12 names for each detail roughly doubled accuracy on unfamiliar names to 0.59–0.67. A larger model did no better. Extra practice on naming mistakes also helped, but we need repeated tests to distinguish the benefit from normal variation. Misspelling the requested name remains a major problem.
Copying accurately. We are removing the requirement to copy every answer exactly, because some correct answers need a different format. A separate study compares four ways of teaching or using copying to see whether they help with unfamiliar wording.
Managing the work. A new job queue started on 7 October. It sets priorities, starts jobs when suitable computers are free and raises an alarm if work is waiting while computers sit idle. Jobs can join a study only if its recorded plan includes them.
Try one round on a standard processor. Begin all 20 rounds only if scores hold, choices stay clear and messages remain useful.
Compare copying lessons, copying names from the request and no copying with the current method. Repeat the current method to check normal variation.
Try the same three practice methods on two more models to see whether the improvements hold up.
Where we started, where we got to, where we are going
- Small models can read invoices, sort text, recognise photos and flag doubtful answers.
- We have measured where they struggle: made-up answers, uncommon details and unfamiliar wording.
- The platform checks results against shortcuts and can recover from some job failures.
- We have not tested how well models prepare actions such as calendar entries.
- Models still struggle with unfamiliar documents and wording.
- We have not shown a clear benefit from teamwork. So far, one strong member can explain a team's success, and communication is only starting to match a good teaching rule.
- Some completed experiments still need written conclusions.
- Messages: more learning from examples (study T1-C) and an initial check of revised coaching (T1-Q2), using standard processors.
- New wording: the copying study, followed by repeated tests of learning from mistakes on two more models.
- Computers: all four machines with specialist AI processors are busy, with work organised by the new queue.
The biggest lesson. Many disappointing or misleading results came from how we taught and tested the models. A limit on practice created a false pattern. Some pictures and questions could be answered through shortcuts or memory. Some feedback failed to reward useful messages, and one teaching method led models to stop talking. We found these problems by testing our assumptions, and the platform now checks for them automatically.
The end goal. Use a public comparison test, with the final examples kept separate from practice, to answer two questions: how close can one Frugal model get to the best systems, and can a team of small models that learn from their mistakes beat one larger model?
Small models can do useful work on clearly defined tasks. The next challenge is to make them more dependable when the wording changes, and to find out whether several can work better together. Our results make better teaching a promising next step.