Chat

Keep a conversation with a model.

The hello program sends one prompt. A chat program keeps the conversation, so the model remembers what came before.

Before you start

Download a model that follows instructions:

yzma model get -u https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-fp16.gguf

Run the example

$ go run ./examples/chat/ -model ./models/qwen2.5-0.5b-instruct-fp16.gguf
USER> Are you ready to go?

Yes, I'm ready to go! What would you like to do?

USER> Let's go to the zoo


Great! Let's go to the zoo. What would you like to see?

USER> I want to feed the llama


Sure! Let's go to the zoo and feed the llama. What kind of llama are you interested in feeding?

See the code.

What is different

The chat template. An instruction model expects a marker before each message. ChatApplyTemplate adds the markers. Most GGUF files hold their own template, and ModelChatTemplate reads it.

tmpl := llama.ModelChatTemplate(model)
prompt := llama.ChatApplyTemplate(tmpl, messages, true)

See Chat templates.

The context keeps the conversation. The example adds each question and each reply to the list of messages. It then applies the template to the whole list, and decodes only the text that is not in the context yet. So the model sees the complete history, but each turn decodes only the new part.

formatted, _ := chatTemplate(true)
text, ok := strings.CutPrefix(formatted, decoded)

If the new history does not start with the old one, the example clears the memory and decodes everything again.

The sampler is not greedy. A chat needs some variation. The example uses a chain with a temperature sampler, a top-k sampler, a top-p sampler, and a min-p sampler.

The flags

Flag	Default	What it does
`-model`	none	The model file. This flag is required.
`-lib`	`YZMA_LIB`	The directory with the `llama.cpp` libraries.
`-p`	none	One prompt. Omit this flag for a chat session.
`-sys`	none	The system prompt.
`-template`	none	The name of a template, when the model has none.
`-temp`	0.8	The temperature. A higher value gives more variation.
`-top-k`	40	Keeps the 40 best tokens.
`-top-p`	0.9	Keeps the tokens that make 90 percent of the probability.
`-min-p`	0.1	Removes a token that is much worse than the best one.
`-c`	4096	The size of the context in tokens.
`-t`	0	The CPU threads. 0 uses a count from the model size.
`-n`	-1	How many tokens to generate. -1 means the context size.
`-b`	2048	The logical batch size.
`-ub`	2048	The physical batch size.
`-cmoe`	false	Keeps all Mixture of Experts weights in the CPU.
`-ncmoe`	0	Keeps the Mixture of Experts weights of the first N layers in the CPU.
`-v`	false	Shows the `llama.cpp` log messages.

When the context is full

A long conversation fills the context. You have three choices.

Make the context larger with -c. The model has a limit that ModelNCtxTrain gives.
Remove the oldest messages with MemorySeqRm.
Clear the context with MemoryClear and start again.

Next steps

Go to Vision to ask a question about an image.