Building AI calorie tracking from a photo: lessons from VitaCal

Zayin Krige, 2026-09-25. 6 min read.

AI calorie tracking from a photo works by sending a resized meal photo to a vision model, asking it for every food on the plate with calories and macros in a fixed JSON shape, checking that JSON on a server, and letting the user correct the result before anything is logged. In VitaCal, the calorie tracker I build and run, the model is Google's gemini-3.8-flash with thinking set to low, called from a Firebase Cloud Function. The model call is a few lines. The work is validation, correction, deleting the photo and bounding the bill.

The pipeline, step by step

1. Capture and upload

The app is Flutter, for the reasons in my Flutter vs React Native comparison. Gallery images are resized to 1024 pixels wide at JPEG quality 80, and the camera plugin shoots at its high resolution preset. The photo goes to Cloud Storage under a per-user analysis prefix, with a 60 second upload timeout. The function refuses any path outside that user's own prefix.

2. The Gemini call

The function sends two parts: the prompt text and the image as inline data. Every call carries safety settings that block harassment, hate speech, sexually explicit and dangerous content at medium and above. Transient errors get up to three attempts with exponential backoff starting at one second. A model-not-found error skips straight to a fallback model, gemini-3.1-flash-lite, so a retired model name degrades the service instead of stopping it. The function times out at 60 seconds and the app stops waiting at 75.

3. Structured output, checked on the server

The prompt allows exactly three reply shapes: a list of items, a clarification request, or an error. Each item has a name, calories for the visible portion (not per 100g), optional protein, carbohydrate and fat in grams, and a confidence between 0 and 1. The photo path does not use the API's schema mode: the server strips markdown fences, parses the object and normalises it: calories are rounded and clamped to 0 to 99,999, macros are clamped at zero, confidence is clamped to 0 to 1, and any item without a name or with zero calories is dropped. If nothing valid survives, the call fails loudly rather than returning an empty meal.

4. Correction before logging

Two mechanisms carry uncertainty to the user. When the model is below 0.8 confidence on what a food is, the prompt requires two or three alternatives, each with full nutrition; tapping one replaces the item's name, calories and macros. When the photo physically cannot answer a question that changes calories a lot, such as hidden oil or a portion with no scale reference, the model returns one or two tappable questions with two to four quantified options instead of items. The user answers at most two rounds of questions, or taps for a best estimate, which forbids further questions. Every calorie field stays editable.

5. Favourites

The confirm step has a "Save as a favourite" toggle, off by default. A favourite stores the item names joined, with calories and macros summed. Today offers up to three favourites last used within two hours of the current time, each relogged in one tap with no model call.

Choose the model on your own data

I benchmarked candidates on 60 real meal photos that users had logged, measuring drift from the calories each user actually saved. gemini-3.8-flash answered 55 of 60 with a median drift of 4% and a 90th percentile of 23%. gemini-3.1-flash-lite answered 59 of 60 at about a third of the price, but its median drift was 15% and its 90th percentile 68%. On a calorie tracker that number is the product, so photos stay on the dearer model.

On 60 typed meal descriptions the same two models were level, so the text path runs the cheaper one. Text results do not transfer to photos. On the public Nutrition5k dataset (100 dishes, forced estimate, overhead cafeteria photography rather than handheld phone photos), gemini-3.8-flash answered all 100 and gemini-2.5-flash answered 68. On the 68 dishes both answered, the mean absolute percentage error was 30% against 61%, and median latency was 3.1 seconds against 12.6.

Privacy: two copies, two retention rules

The analysis copy is deleted when the analysis ends, by the function and by the app, or when a pending clarification exchange ends. A scheduled job runs every five minutes and deletes any analysis copy older than ten minutes, so a crash midway cannot leave one behind: every analysis copy is gone within fifteen minutes of upload. When the user saves the meal, a second copy is stored with it so it can show on the Today screen. That copy is kept until the account is deleted, and account deletion removes it.

The function keeps a short audit record of each analysis: which file, which model, and the foods and calories returned. It never contains the photo, the prompt or the model's raw output.

Cost control

• Measure cost per call, not per token. Measured from billing data, a photo costs about $0.0036. gemini-3.8-flash has dearer tokens than gemini-2.5-flash and still costs less per photo, because capped thinking uses far fewer of them.
• Meter every analysis. Free accounts get 5 AI meal analyses a week and Pro accounts 30, with a rate limit of 10 calls a minute per user and App Check enforced on the function.
• Charge only for results. A failed analysis is not charged. A clarification chain costs one analysis: the first round pays and the follow-ups ride free against a server-side counter that expires after 30 minutes, because a round number sent by the client can be forged.
• Never benchmark on the production key. One benchmark run on the live key spent more than half of that month's Gemini bill, and two days later the spend cap tripped and meal analysis stopped for real users. Benchmarks now run in a separate project with its own key and cap.

Failure modes worth designing for

• Not food, or too blurry: the model returns an error shape and the user sees its message.
• Text in the photo trying to change the model's behaviour: the prompt's first rules tell it to ignore it.
• No JSON in the reply, or no valid items: the function records the error, deletes the photo and returns a failure.
• A retired model: gemini-2.5-flash already returns 404 for new projects, and a test keeps retired models out of every fallback chain.

Output in the user's language

The app ships in 21 languages and the prompt is written in English, so the prompt has to say which language to answer in. The app sends its locale; the server reduces it to a two or three letter language subtag and discards anything else, because a free-text locale would be a prompt injection route. For non-English users a clause is appended last, telling the model to write names, questions and messages in that language while keeping every JSON key and unit in English. Chinese is pinned to Simplified characters and Serbian to Cyrillic. English prompts get no clause, so they stay byte-identical.

Health sync

Each logged item can be written to the phone's health store when the user is on Pro and has nutrition export switched on. On iOS it goes to HealthKit as separate dietary energy, protein, carbohydrate and fat samples. On Android it goes to Health Connect as one meal record with the name and meal type.

What I would tell anyone building this

Treat the model's reply as untrusted input, give uncertainty a place in the interface, benchmark on your own users' photos, and decide how long every copy of the photo lives before you ship. More of my work is on the homepage.