Build for a browser

This page has the details for a program that runs in a browser. See the tutorial to build and run one, and WebAssembly for how the parts fit together.

Select the build

yzma-loader.js selects the best build that the browser can run.

A page can set the choice with globalThis.yzmaMode. The values are auto, which is the default, webgpu, and cpu. A page URL also accepts ?mode=cpu or ?mode=webgpu.

A machine with an integrated GPU and a discrete GPU gives the browser a choice. A page picks one with globalThis.yzmaPowerPreference, which takes high-performance or low-power. A page URL also accepts ?gpu=high-performance or ?gpu=low-power. With no value the browser picks. The loader tests that GPU and makes llama.cpp ask for the same one.

With webgpu the loader still falls back to the CPU when the browser cannot run that build. A slow page is better than a page that does not work.

Ask llama.cpp which device does the work, not the browser. A page can have WebGPU while llama.cpp finds no device.

backend := llamawasm.Backend()
device := llamawasm.GPUDevice()

Headers for more than one thread

The faster CPU build needs SharedArrayBuffer. A browser gives that only to a page with these headers.

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

wasm/serve/main.go sends them. When a host sends no headers, the loader uses the single thread build. llamawasm.Threaded() reports the selection.

A host such as GitHub Pages sends no headers. A page there gets them from a service worker such as coi-serviceworker. Such a worker must not send the model download through respondWith. Firefox stops a service worker that holds a response open for a long time, and the download then fails with TypeError: Error in input stream. Let the browser make the cross origin request instead, for example with event.stopImmediatePropagation() in a listener that runs before the worker’s listener.

An isolated page can get a model from another origin only when that origin sends the CORS headers. Hugging Face sends them. A model on a host with no CORS headers must be copied to the page’s origin.

Threads

llama.cpp asks for four threads unless a caller changes it. A machine with more cores then loses a lot of speed.

So ContextDefaultParams and MtmdContextParamsDefault send llamawasm.Threads(), which the JavaScript glue reads from the machine.

The threads have no effect on an image. The projector used 30.4 seconds on four threads and 33.3 seconds on sixteen. The GPU makes an image fast, not the CPU.

Speed

The numbers are on the Benchmarks page. With SmolLM-135M.Q2_K on an Intel Core i9-13900HX in Chrome, the build with more threads gives 93.3 tokens a second. WebGPU gives 66.2 on an RTX 4070 and 19.6 on the Intel graphics of the same machine.

To measure a build, run ./benchmarks/run.sh --backend wasm in the yzma repository for Node. For a browser, which WebGPU needs, paste benchmarks/browser-bench.js in the page’s console. Set gpu in the script to pick the GPU.

On a small model the CPU with more threads is faster than the GPU, because each operation is too small to be worth the transfer to the GPU. The GPU does better on a larger model. Test both with ?mode=cpu and ?mode=webgpu.

An image gives a different result. A photo of 960 by 720 through the SmolVLM-256M Q8_0 projector takes 42.7 seconds on the CPU with more threads and 1.6 seconds with WebGPU.

A projector does many calculations at the same time, which is what a GPU is built for. That makes the GPU 25 times faster. A page with images needs WebGPU more than a page with text only.

The CPU builds give the same text each time. The GPU gives the same text for the first tokens and then different text, because the shaders do the calculations in a different order.

Images

The page decodes the image, not llama.cpp. It draws the file on a canvas and sends the pixels to the program.

const bitmap = await createImageBitmap(file);
context.drawImage(bitmap, 0, 0, width, height);
const { data } = context.getImageData(0, 0, width, height); // RGBA
worker.postMessage({ kind: "describe", prompt, width, height, rgba: data.buffer }, [data.buffer]);

So every format that the browser can read works, and the WebAssembly build needs no image library. The Go side removes the alpha byte and sends the RGB to mtmd.

The multimodal calls have an Mtmd prefix, because the same package has the llama calls too.

mctx, err := llamawasm.MtmdInitFromFile("/models/mmproj.gguf", model, 0, onGPU)
bitmap, err := llamawasm.MtmdBitmapInit(width, height, rgb)
chunks, err := llamawasm.MtmdInputChunksInit()
llamawasm.MtmdTokenize(mctx, chunks, prompt, true, true, []llamawasm.MtmdBitmap{bitmap})
nPast, err := llamawasm.MtmdHelperEvalChunks(mctx, ctx, chunks, 0, 0, nBatch, true)

Then the usual loop of SamplerSample and Decode follows, the same as for text.

The prompt must hold one marker for each image. MtmdMarker returns the model’s marker.

This downloads two files, the model and its projector.

Tool calling

pkg/template and pkg/message are pure Go, so they build for WebAssembly. The browser gets the same tool calling as a host.

tmpl := llamawasm.ModelChatTemplate(model, "")
prompt, err := template.ApplyWithTools(tmpl, messages, tools, true)

message.ParseToolCalls reads the calls out of the answer. The program runs them, appends a message.Tool and a message.ToolResponse, and renders again for the final answer.

A model must be trained for tool calls to make one. Qwen2.5-0.5B-Instruct is about the smallest that works.

llamawasm.ChatApplyTemplate takes one message only. Use pkg/template for a conversation with turns.

The shim has no end of turn token, so the WebAssembly build uses the end of sequence token’s text and tries a short list of the usual markers. A host build reads the token itself.

Typed decisions

The exp/decide package also builds for the browser. It runs a System One model that answers a typed question with a probability for each option. The wasm/decide example is a page for it.

DecideMany reads the state once and then asks each question. For this it needs a context where the sequences share one cache. Set these ContextParams fields to make one.

Both need a llama.cpp module with ABI 10 or later, which is v0.6.0 or later. An older module returns llamawasm.ErrNoKVUnified when KVUnified is set. exp/decide then decodes each question on its own, so it still works, only slower.

Clef and Decision 2.0 models read a decision head through an extended batch. This needs a module with ABI 11 or later, which is v0.7.0 or later.

WebGPU settings

f16 shaders and NVIDIA

The llama.cpp backend needs shader-f16 and reports no device without it. In a browser the backend uses the browser’s adapter and sets no options.

• An Intel integrated GPU supports f16 and the WebGPU build works.
• A discrete NVIDIA card does not support f16 in a browser. Dawn has the vulkan_enable_f16_on_nvidia option, and llama.cpp sets it outside a browser but not in one. A page cannot set it, because it is a browser flag. Start Chrome with this command:

google-chrome --enable-dawn-features=vulkan_enable_f16_on_nvidia

On Linux this switch alone is not enough. See Vulkan in Chrome on Linux.

The GPU self test

A GPU that llama.cpp accepts can still compute wrong values. The model’s answer is then random tokens, and nothing in the text shows that the fault is the GPU and not a weak model. So the loader measures the device.

Before it gives the module to the page, yzma-loader.js runs one small matrix multiply on the GPU and the same one on the CPU. It compares the two results with the normalized mean squared error. A good device gives about 3e-8 and noise gives about 1. The limit is 1e-2. The test needs no model and takes a few milliseconds.

When the test fails, the loader drops the GPU build and uses a CPU build. globalThis.yzmaGPUReject holds the reason. A Go program gets the same answer from llamawasm.BackendOK().

if !llamawasm.BackendOK() {
	// The GPU computed wrong values, so the page runs on the CPU.
}

BackendOK is true for a build with only the CPU, and for a module older than ABI 8, which has no test.

Test WebGPU without a browser

The yzma repository can run the WebGPU build outside a browser, with the same WebGPU code that the browsers use.

make test-wasm-dawn NODE=/path/to/node26
make test-wasm-wgpu

Set GPU=high-performance or GPU=low-power to pick the GPU. DAWN_FLAGS passes flags to Dawn, for example backend=vulkan,adapter=NVIDIA. DAWN_FLAGS=backend=opengles runs the same OpenGL ES path as Chrome on Linux. Dawn then uses its compatibility mode, as Chrome does.

These tests need a GPU, so they are not in CI.

Vulkan in Chrome on Linux

Chrome on Linux keeps Vulkan off. WebGPU then uses ANGLE’s OpenGL ES backend, in Dawn’s compatibility mode. chrome://gpu shows Vulkan: Disabled, and the first adapter in Dawn Info is an OpenGLES backend line with (Compatibility Mode) at the end.

This path gives one of two results, and neither is good.

• On many cards the adapter has no shader-f16. llama.cpp then finds no device and the loader uses the CPU. The page is slow but correct.
• On an Intel Xe with Mesa the adapter has shader-f16, llama.cpp takes it, and it computes wrong values. The self test catches this case and falls back to the CPU. See issue #341.

These three switches give Chrome the Vulkan backend. Close all Chrome windows first, and make sure no Chrome process keeps running in the background.

google-chrome --enable-unsafe-webgpu --enable-features=Vulkan \
  --enable-dawn-features=vulkan_enable_f16_on_nvidia

All three are needed. --use-angle=vulkan is not. If a switch seems to do nothing, check the command line in chrome://version.

On Chrome 154 with Ubuntu 24.04, an Intel Raptor Lake, and an RTX 4070, the three switches give a Vulkan adapter for each GPU, both with shader-f16. Run this in any page’s console to list them.

for (const p of ["low-power", "high-performance"]) {
  const a = await navigator.gpu.requestAdapter({ powerPreference: p });
  console.log(p, a && a.info.vendor, a && a.info.architecture,
    a && a.features.has("shader-f16"));
}

A good result names a GPU on each line, not swiftshader, and ends with true.

Issue #341 was tested with only the last two switches. On Ubuntu 22.04 with Mesa 23.2.1, Dawn then still gave the OpenGL ES adapter. Such a machine can try all three. Without them the self test falls back to the CPU.

Firefox

Firefox runs the WebGPU build, but WebGPU is not on by default. Set both of these in about:config and restart the browser.

JSPI arrived in Firefox 153, so 153 or later needs no other switches.

Firefox 154 gave llama.cpp wrong values. Firefox 156 gives the right text, but very slowly.

The RTX 4070 is no faster than the Intel graphics, so the time goes to Firefox and not to the GPU. Auto mode uses the CPU in Firefox. Mode webgpu still selects the GPU, which makes it easy to test a new Firefox.

Limits

• WebGPU needs an adapter with f16 shaders, and Chrome or Edge 137 or later. Firefox 153 or later can run it, but the loader uses the CPU there because it is faster. Every other browser uses the CPU with SIMD.
• A browser does not expose subgroup matrix instructions, which llama.cpp uses only outside a browser. So the GPU is slower in a page than the same backend on a desktop.
• Some drivers expose an adapter that llama.cpp accepts and that computes wrong values. The loader then uses a CPU build.
• An operation larger than maxStorageBufferBindingSize goes back to the CPU.
• A JavaScript ArrayBuffer holds at most 2 GB, so a larger model must come in splits.
• pkg/llamawasm has text generation, embeddings, and images. It doesn’t support audio, video, LoRA adapters, saving state to a file, or quantization. The extended batch has no embedding input. It saves context state in memory.
• The shim has no grammar sampler, so a tool call cannot be forced by a grammar as it can on a host.