You want an app you point at dinner that tells you what dinner was. That app exists now, several of them do, and they work better than they have any right to — as long as you know which part of the number is measurement and which part is an educated guess.
This page explains what happens between the shutter and the number. We are building one of these apps, so treat the enthusiasm accordingly — but everything below is checkable, and where the technology is weak we say so, because you will find out in week two anyway.
What actually happens when you take the photo
Every app in this category runs the same four stages. They differ in how good each stage is and in how much of it they show you.
Segmentation — cutting the plate apart
The model separates the photograph into distinct food regions: this pile is rice, that slab is salmon, those green slices are avocado. This is the stage that has improved most in the last two years and it is now genuinely good on a plate where the foods are visually separate.
Identification — naming each region
Each region gets a label and a confidence. "Grilled salmon" is easy. "Some kind of brown sauce" is not, and a sauce can carry more calories than the protein it is poured over.
Portion estimation — the hard one
The model estimates how much of each food is there, in grams. A photograph is flat: it has no depth information, so volume is inferred from apparent size, from the plate as a reference object, and from what portions of that food usually look like. This is where nearly all of the error lives.
Lookup — grams to calories
Once you have "165 g of grilled salmon", the calories are a database query and are not really in question. The uncertainty was already spent in stage three.
Where photo estimation is reliable
- Packaged and branded food, where the item is recognisable and the portion is fixed by the packet.
- Plates where foods are physically separate — a protein, a starch, a vegetable, not touching.
- Foods with a stable density and a familiar shape: an egg, a slice of bread, a chicken breast, a banana.
- Anything you eat repeatedly. Correct it once and a well-built app should stop getting that meal wrong.
- Relative change over a week. Even with real error on each meal, the direction of a week is usually right, and the direction is what most people are actually after.
Where it breaks, and why
- Oil, butter and sauce. Invisible in a photograph and calorically enormous. The same stir-fry can differ by 300 kcal depending on how much oil went in the pan, and no camera can see it.
- Mixed dishes. A curry, a stew, a rice bowl with everything folded through it. Segmentation needs visible boundaries and a stew has none.
- Depth. A bowl photographed from above shows its surface, not how deep the food goes. Two identical-looking bowls can hold very different amounts.
- Food outside the training data. A model trained mostly on Western restaurant plates will misread a night-market meal, a convenience-store lunch box or a table of side dishes. This is a data problem, not a physics problem, and it is fixable — but only if the app is actually trained for where you eat.
- Anything already eaten. Photograph the plate before you start. Half a meal in a photograph is a guess about the other half.
So how far off is it?
Honestly? Enough that any app quoting you a single accuracy percentage should make you suspicious. Accuracy depends almost entirely on what is on the plate — a packet of crisps is nearly exact, a homemade curry is a guess with a photograph attached. There is a fuller answer, including what the published research does and does not support, on the accuracy page.
The useful question is not "how accurate is it" but "how quickly can I fix it when it is wrong". An app that is 15% off and corrects in one tap beats an app that is 8% off and buries the edit control three screens deep, because you will use the first one in March and you will have deleted the second one by February.
What to check before you trust one
- Does it show you how it read the plate? If it just produces a number, you have no way of knowing whether it thought your chicken was tofu. Visible segmentation is not a gimmick — it is the only audit trail you get.
- How many taps to correct a portion? Try it on the first meal. If correcting is harder than logging manually, the app has not actually solved the problem it advertises.
- Can you set fractional portions? Two thirds of a bowl is the single most common real-world correction, and an app offering only whole servings will annoy you daily.
- What is free, in specifics? "Free to download" is not a free tier. Find the number of free scans before you invest an evening in setup.
- Does it work without an account? Being asked to create a login before you have seen a single result is a sign the product is optimising for signups rather than for you.
What we are building
Plateknow is an iOS app that does the four stages above and shows you the second one. You watch the plate get cut apart, which is what makes the number believable, and every result card carries a correction control that is always visible rather than hidden behind an edit mode. Portions adjust in fractions. There is a mode that sets no calorie goal at all and simply records the day, because a lot of people want to see their food without being given a target.
It is not on the App Store yet. Submission is planned for December.
Before the App Store
Get it before it is public — and tell us what to build.
Early access on TestFlight, and no charge during the beta.
Do these apps need an internet connection?
Most do, because the model runs on a server rather than on the phone. That has a privacy implication worth knowing about: your food photographs leave your device. Any app in this category should tell you plainly whether that happens and how long the images are kept.
Can I photograph a recipe or a menu instead of the food?
Some apps read text as well as images, which works well for a packaged label and badly for a restaurant menu — a menu tells you the dish name, not the portion or how much oil the kitchen used.
Is a barcode scanner better?
For packaged food, yes, straightforwardly. A barcode is an exact match to a known product. Photo estimation is for the food that has no barcode, which for most people is most of what they eat.
Will it work for the food I eat if I am not American?
Ask before you commit an evening to it. This is the most common failure and the least discussed. A model is only as good as the plates it was trained on.