I love to bake and I love eating baked goods. But my problem is most recipes just make way too much. A standard cupcake recipe yields a full dozen, most cookie recipes make 24, and given my complete lack of portion control, I really only need to be making two (maaayybe four) at a time.
So back during covid, I built a simple recipe calculator. It did basic math to scale ingredient amounts up or down for smaller batch sizes. It was written in plain javascript and pretty reliable for the task.
Fast forward to a few weeks ago, a friend was hyping up Chrome’s Built-in AI, showing off all his project ideas. That got me thinking.. could I turn my basic scaling calculator into an intelligent recipe adapter? I’ve baked enough to know what can be tweaked and how, but I wanted to see if on device/in browser AI could suggest smart swaps (e.g. for dairy, eggs, or gluten free) on the fly while keeping recipe ratios intact.
The thing is, Chrome Built-in AI only work in Chrome. As a loyal Vivaldi user, I had to look elsewhere. That’s when I found WebGPU AI. It was perfect because it runs locally directly in the browser, meaning guaranteed privacy cause the data never leaves the device, and server costs are also zero. Plus, once the model weights land in CacheStorage & IndexedDB, it even work offline.
Building with AGY
Up until now, my experience with AI assisted coding was pretty much limited to just basic inline autocomplete. And even that I’ve often kept off. I found it a bit too distracting, sometimes halfway through a thought, a suggestion would come up, and I’d instinctively stop thinking to read it.. completely losing my own train of thought. :/
But since this whole project was an AI experiment anyway, I decided to go all in. I downloaded Antigravity IDE and the CLI and jumped right into the deep end.
I started by creating a project spec file detailing my core requirements like main goals, preferred tech stack, and acceptance criteria, etc. I also created a file containing ingredient substitutions that I know are good, formatted in clean JSON so the agent could use it as reference. I gave everything to AGY, and it got straight to work. It parsed the requirements doc, generated an implementation plan, and asked for the green light to start on the subtasks.
Watching the agent work was pretty neat. I spent the afternoon reading its steps and thoughts in the terminal output. It mapped out my repo, optimized my existing functions, iterated on its own code, and added comments across the codebase as a nice touch.
I think many would agree with me that reading existing code is never as fun as writing your own. And while I am amazed at all the code AGY outputted, it just doesn’t sit right with me to not understand how it works. But reviewing code is kinda boring, which is why I decided to turn it into this post.. so I can learn and note down my findings at the same time. :)
Understanding WebGPU and WebLLM
Simply put, WebGPU is a modern web API that provides low overhead, direct access to your device’s GPU hardware. While its predecessor (WebGL) was designed primarily for 3D graphics rendering, WebGPU introduces first class compute shaders written in WGSL (WebGPU Shading Language), which enables general purpose GPU computing (GPGPU) directly inside the browser!
LLMs rely heavily on massive matrix multiplications. While a CPU processes tasks sequentially across a few cores, GPUs contain thousands of small arithmetic logic units (ALUs) capable of executing matrix operations in parallel. WebGPU maps these tensor operations directly onto modern native hardware APIs like Vulkan (Linux/Windows), Metal (macOS/iOS), and Direct3D 12 (Windows).
To bridge WebGPU with AI models, we integrate @mlc-ai/web-llm. WebLLM is a high performance in browser LLM inference engine built on the Apache TVM compiler ecosystem.
MLC LLM & Model Conversion
Before a model can run in WebLLM, it has to go through MLC LLM (Machine Learning Compilation). You can’t just drop raw PyTorch or HuggingFace weights into the browser, as browsers don’t have Python or CUDA.
MLC LLM acts as an offline compiler that converts open weight models into browser ready static assets through:
- Weight Quantization
- Raw 16-bit float weights (
fp16) are compressed into 4-bit representation (q4f16orq4f32), shrinking a 2GB model down to ~350MB so it fits in browser memory.
- Graph Optimization via TVM Unity
- The model’s computation graph (Self Attention, RMSNorm, Linear projections) is translated into TVM’s Intermediate Representation (Relax IR). Sequential math operations are fused together into single, optimized WebGPU compute kernels.
- Web Artifact Export
- The compilation outputs three main files:
params_shard_*.bin: Quantized weight files split into~25MBbinary chunks for streaming and caching in browserCacheStoragevia Caches API.mlc-chat-config.json: Model configuration, context window size, and quantization parameters.- Wasm Runtime Module: A compiled WebAssembly library containing tokenization and engine control logic.
Here is how WebLLM uses these compiled assets inside the browser:
- Model & Shader Loading
- Loads the pre-compiled WGSL compute kernels generated during compilation.
- WebAssembly Control Plane
- The lightweight Wasm engine manages tokenization, KV-Cache memory, and dispatches WebGPU compute commands (
GPUCommandEncoder).
- VRAM Buffer Binding & Quantization
- Binary weight shards (
params_shard_*.bin) are transferred fromCacheStoragedirectly into GPU VRAM buffers.
- Token Generation Loop
- Compute passes execute matrix math entirely within VRAM, keeping data transfer between the GPU and JavaScript to a minimum.
flowchart TD
A["CacheStorage (Binary Weights)"] -->|Load Shards| C["GPU VRAM (Buffer Allocation)"]
B["Wasm Control Engine"] -->|Tokenize & Dispatch| D["WebGPU Compute Pass (WGSL)"]
C --> D
D -->|Parallel MatMul| E["GPU Hardware Execution Units"]
E -->|Sampled Token| F["Web UI Stream Callback"]Another critical thing to note is keeping the AI engine as a global singleton. Large model weights take time to download and parse. If we recreate the WebLLM engine on every user click, the app will freeze as VRAM reloads over and over. Best to initialize the engine once globally, store the instance, and reuse it for every prompt.
// Keep the engine as a single long lived instance
let webllmEngine = null;
async function getEngine(modelId) {
if (webllmEngine) return webllmEngine;
webllmEngine = await webllm.CreateMLCEngine(modelId);
return webllmEngine;
}How WebGPU Runs in the Browser
Before calling WebGPU APIs, we have to check if the entry point, navigator.gpu, exists. It’s the global object that bridges JavaScript and the underlying GPU hardware. Checking its existence ensures the browser actually supports WebGPU and is running in a secure context as required by WebGPU (e.g., https or localhost).
Once we validate the navigator’s existence, we can use it to request a GPUAdapter via navigator.gpu.requestAdapter(). With the adapter, we can then inspect adapter.features to see what hardware capabilities our GPU supports.
Why does this inspection matter? To run computational workloads, we must turn the GPUAdapter into a GPUDevice by calling adapter.requestDevice(). The GPUDevice is the actual logical connection to the GPU that provides the computational sandbox and resource management required to run work. Features are strictly opt-in when requesting a device, meaning even if your system supports 16-bit floats, the browser will block you from using shader-f16 unless you explicitly ask for it during device creation. If you try to request a feature in requiredFeatures that the adapter.features set doesn’t contain, requestDevice() throws an error and refuses to initialize. Inspecting adapter.features beforehand prevents the app from crashing on machines with older GPUs, fallback drivers, or missing browser flags.
But what is shader-f16 anyway? Standard GPU operations historically ran on 32-bit floating point numbers (f32). shader-f16 is a specialized WebGPU extension that allows shaders to process numbers using half precision (f16). And with it comes significant benefits for in browser AI, such as 2x lower memory usage & faster inference speed. Note: spotty support on Linux :c
If shader-f16 is supported, our model loader selects 16-bit quantised models like gemma3-1b-it-q4f16_1-MLC or Qwen2.5-0.5B-Instruct-q4f16_1-MLC. If missing (e.g., on Linux or older drivers), it falls back to 32-bit quantised variants (q4f32_1) like SmolLM2-360M or Qwen2.5-0.5B, so the app still functions without throwing a shader validation error.
// Check for GPU shader support and select candidate models
let hasF16 = false;
try {
const adapter = await navigator.gpu.requestAdapter();
hasF16 = adapter && adapter.features && adapter.features.has("shader-f16");
} catch (e) {
console.warn("Could not check WebGPU adapter features:", e);
}
const candidateModels = hasF16
? ["gemma3-1b-it-q4f16_1-MLC", "Qwen2.5-0.5B-Instruct-q4f16_1-MLC", "SmolLM2-360M-Instruct-q4f32_1-MLC"]
: ["Qwen2.5-0.5B-Instruct-q4f32_1-MLC", "SmolLM2-360M-Instruct-q4f32_1-MLC"];What the Agent Implemented
Once the WebGPU capability checks were in place, AGY built out the core feature pipeline. So what did the initial AI integration actually look like?
At a high level, it came down to three main pieces: streaming download progress, crafting JSON prompts, and updating the app state.
Streaming Model Progress
Downloading model weights takes time. Even a tiny 500 million parameter model requires fetching roughly 300 megabytes of data into local IndexedDB storage.
Without feedback, a user might think the app froze. AGY handled this by hooking into WebLLM’s initProgressCallback. It passed live status messages straight to the UI so users could track download progress in real time.
// Stream loading progress directly into the modal UI
webllmEngine = await webllm.CreateMLCEngine(selectedModel, {
initProgressCallback: (report) => {
if (onStatusUpdate) onStatusUpdate(report.text);
},
appConfig
});Initial JSON Prompting
Small models get off track quickly if you don’t give them tight guardrails. AGY set up an initial prompt structure aimed at forcing the LLM to output clean JSON instead of long conversational answers. It set the temperature to 0.3 to keep responses focused. It also passed a simple JSON template directly inside the user prompt.
const systemPrompt = "You are a helpful assistant. Output valid JSON only.";
const userPrompt = `Return a JSON array of up to 3 substitutes for "${ingredientName}".
Format: [{"name": "string", "ratio": 1, "unit": "ratio", "desc": "string"}]
Do not include markdown code blocks or extra text.`;
const reply = await engine.chat.completions.create({
messages: [
{ role: "system", content: systemPrompt },
{ role: "user", content: userPrompt }
],
temperature: 0.3, // Lower temperature reduces hallucination
});State Updates & DOM Management
After generating a response, the agent needed to put those new ingredients into the app. It wrote a function to parse the incoming text, append new options to the global SUBSTITUTIONS map, and trigger a fresh render of the ingredient list.
// 1. Update global state / database
SUBSTITUTIONS[key] = [...existing, ...newAISubs];
// 2. Re-render UI list
updateAdjustedRecipe();
// 3. Find updated container in fresh DOM
const freshContainer = document.querySelector(`[data-ingredient-index="${index}"]`);
showSubstituteModal(index, freshContainer, line, selectedSubIdx);On paper, this initial implementation worked. The WebGPU engine loaded, the model responded, and new ingredients appeared on screen. But once I started testing it with real recipes, several subtle flaws surfaced.
What I Had to Fix Myself
Structured Prompting & Sanitisation
I’ve found that small models (0.5B to 1B parameters) get confused easily. Before AGY generated any code, I provided a reference file of valid substitutions. Baking swaps fall into two categories: simple 1-to-1 swaps (like plain yogurt for buttermilk) and multi-component mixtures (like milk plus vinegar for buttermilk).
When the agent wrote the initial prompt, it ignored my dual schema layout. Worse, it used concrete ingredient names in the prompt’s schema example. Because 0.5B models rely heavily on pattern matching, the LLM hallucinated the example ingredient whenever it got stuck. Ask it to substitute baking soda, and it would suggest plain yogurt! haha
I fixed this by rewriting the prompt with generalised schemas, strict JSON output rules, and explicit recipe context so the model knows what role the ingredient plays in the overall dish.
You are an expert chef and baker assistant.
Your task is to provide accurate cooking and baking substitutions in strict JSON format only. Do not include markdown code blocks, backticks, or any conversational text outside the JSON array.
### Instructions:
1. Provide up to 3 realistic, high-quality substitutes for "${ingredientName}".
2. Prioritize functional equivalents (e.g., binding, leavening, moisture, acidity, structure).
3. Exclude any substitutes that match this list: ${existingStr}.
4. If no safe or functional substitute exists, return [{ "name": "No substitute found", "ratio": 0, "unit": "ratio", "desc": "No substitute found", "isAI": "true" }].
Use the original recipe's ingredient list as context to determine what the best substitute for "${ingredientName}" is. The original recipe's ingredient list is as follows: ${ingredientsContext}.
### Output Constraints:
- Return ONLY a raw JSON array of objects.
- Do NOT use Markdown code fences.
- Do NOT include introductory text, explanations, or conclusions.
### Schema Requirements:
Each object in the array must strictly match ONE of these structure types:
1. Single Ingredient Substitute:
[
{
"name": "Substitute Name",
"ratio": <number: multiplier relative to original amount, e.g. 1 for 1:1, 0.5 for half amount>,
"unit": "<string: use 'ratio' for proportional scaling, or explicit unit like 'cup', 'tbsp', 'tsp'>",
"desc": "<string: short 1-sentence tip on texture/flavour impact and any required prep like softened, melted, or gelled>"
}
]
2. Multi-Component Combination Mix Substitute:
[
{
"name": "Mix Name (e.g. AP Flour + Cornstarch Mix)",
"desc": "<string: short 1-sentence tip on texture/flavour impact and any required prep like softened, melted, or gelled>",
"components": [
{ "name": "Component 1 Name", "ratio": <number: use 'ratio' for proportional scaling, or explicit unit like 'cup', 'tbsp', 'tsp'>, "unit": "<string: use 'ratio' for proportional scaling, or explicit unit like 'cup', 'tbsp', 'tsp'>" },
{ "name": "Component 2 Name", "ratio": <number: use 'ratio' for proportional scaling, or explicit unit like 'cup', 'tbsp', 'tsp'>, "unit": "<string: use 'ratio' for proportional scaling, or explicit unit like 'cup', 'tbsp', 'tsp'>" }
]
}
]
Respond ONLY with the raw valid JSON array.Edge Cases & Errors
Downloading multi-gigabyte models into CacheStorage fills up browser storage fast. The agent didn’t account for storage limits, causing downloads to break when disks filled up. I added error handling to catch QuotaExceededError exceptions and clear stale WebLLM caches (caches.delete()) automatically so new downloads have room to finish.
Another subtle bug was WindowSizeConfigurationError. Prebuilt WebLLM configs often set both context_window_size and sliding_window_size as positive numbers. These represent two incompatible ways of managing GPU attention. Fixed context looks back across all tokens in a sequence, while sliding window only tracks a fixed local neighbourhood. Having both active at the same time throws a configuration error. I fixed this by overriding sliding_window_size: -1 inside appConfig.
UI Usability & Experience
In the agent’s original UI flow, the modal called closeAllModals() the instant the model finished generating.
From a user standpoint, it was terrible. I would click “Generate,” wait a few seconds for the local model to process, and then.. the modal just disappeared, leaving me wondering if it even worked! haha. I refactored the async handler to keep the modal open and refresh the content in place, so the user can actually read and select the generated substitutions.
I also added some visual distinctions. Since swapping out ingredients does come with some risk (and an even greater one if it’s AI generated), users should be made aware. I decided to add an ✨ AI Substitute badge to generated options.
Finally, the agent was appending new AI suggestions to the bottom of the list. It didn’t feel right to have to scroll down to see the new options. I updated the array handling so that fresh AI suggestions are prepended to the top of the list (SUBSTITUTIONS[key] = [...parsedSubs, ...existing]), so the top recommendations catch your eye right away.
Comparing Models, Qwen2.5-0.5B vs Gemma (1B / 2B)
I tried two main model families inside the browser, and they gave wildly different results.
Qwen2.5-0.5B is super fast. It loads in seconds and runs on almost any hardware setup. But it struggles with complex instructions. When asked for JSON, it often hallucinated repetitive loops or broke schema rules.
Gemma (1B / 2B) is an excellent instruction follower. Gemma produced clean JSON arrays and filtered out duplicate ingredients reliably. However, it requires significantly more VRAM, loads slowly, and occasionally triggered browser IndexedDB storage quota limits or out-of-memory errors on machines with smaller drives.
To prevent stray markdown formatting from breaking the app, I added a regex extraction fallback before parsing the JSON response.
// Extract JSON payload safely from raw LLM output
const rawText = reply.choices[0].message.content.trim();
const jsonMatch = rawText.match(/\[\s*\{.*\}\s*\]/s);
const parsedSubs = jsonMatch ? JSON.parse(jsonMatch[0]) : JSON.parse(rawText);What’s Next?
I’m probably not going to play around more with this anytime soon, but here are some ideas of what could be added next..
Offloading to Web Workers
Right now, WebLLM runs on the main browser thread. Moving the Wasm engine and WebGPU compute pipeline into a dedicated Web Worker will ensure the UI stays 60fps silky smooth even while generating long responses.
Vision / OCR Recipe Scanning
Integrating lightweight in browser vision models (like WebGPU based Florence-2 or MobileVLM) so users can snap a photo of a physical recipe card or cookbook page and automatically import ingredients. I still have some special handwritten recipe cards, so it would be nice to digitize them!
Local Storage Management UI
Adding a settings toggle in the UI that allows users to manage cached model weights or switch between model tiers (0.5B vs 1B vs 2B) depending on their GPU hardware and battery status. Also maybe an easier way to clear cached models?
Wrapping Up
I’m sure plenty of people have gone through a similar experience. It started off amazingly, AGY completed the initial WebGPU wrapper faster than I could read the WebLLM documentation. It really did feel like magic! But then it started feeling a bit uncomfortable.
I looked at my codebase and realized I had no idea how parts of it worked. It’s already hard enough remembering code I wrote years ago, but with the addition of generated code for brand new technical concepts, I truly had no idea where to even begin.
I’m a strong believer that real intuition comes from making mistakes. We learn by failing, internalizing the reason for our failures, and fixing them. When an AI agent solves the problem and just hands us the result, we’re building shallow knowledge. Sure, we get a working feature today, but we lose the deep understanding needed to maintain it in a few months.
Would I do agentic coding again? Sure, it’s an incredible speed booster for many tasks. But moving forward, I would probably limit it to only automating tasks I already understand. When exploring new technical concepts, I’d rather do the manual digging myself. I might move slower, but the knowledge will actually stick. :)