Typed decisions

The exp/decide package runs System One models. You give one of these models a state and a typed question, and it gives back a probability for each option instead of text. One decode gives the answer, so it is fast, and the probabilities are calibrated so you can trust them as a confidence.

Use it to route a ticket to a team, to check a fact against a text, or to rate a reply.

The package is experimental. Its API can change in a later release.

The models

The config file holds the token ids and the calibration temperatures. Download it from the same model page as the GGUF file. Laya, Julia-1, Kev and Clef need llama.cpp v0.6.0 or later. pplx-decider, d1, Decision 2.0 and Decision 3.0 need llama.cpp v0.7.0 or later.

Some of these families work in a different way.

• d1 and Decision 3.0 take a nil state. Decision 3.0 also takes a question with no text.
• Clef and Decision 2.0 have a decision head in the llama.cpp graph. yzma marks the question and the options in an extended batch.
• Clef reads all the questions of DecideMany in one prompt, so one answer depends on the others.
• For Decision 2.0, the confidence is 1 minus the normalized entropy.

Three kinds of question

Ask a question

package main

import (
	"fmt"
	"log"

	"github.com/hybridgroup/yzma/exp/decide"
	"github.com/hybridgroup/yzma/pkg/llama"
)

func main() {
	if err := llama.Load(""); err != nil {
		log.Fatal(err)
	}
	llama.LogSet(llama.LogSilent())
	if err := llama.Init(); err != nil {
		log.Fatal(err)
	}
	defer llama.Close()

	d, err := decide.Open("lev-Q8_0.gguf", decide.Options{})
	if err != nil {
		log.Fatal(err)
	}
	defer d.Close()

	q := decide.ChoiceDesc("Which team should handle this?",
		decide.Option{Name: "billing", Description: "Charges, invoices, refunds"},
		decide.Option{Name: "technical", Description: "Bugs, outages"},
	)

	res, err := d.Decide("I was charged twice for order A-104.", q, "")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(res.Answer, res.TopProbability)
}

The state can be a string or any value that encodes as JSON. The last argument of Decide is a category, such as theme_routing or general_sentiment. A Jev-Style model has a calibration temperature for each category. An empty category uses the global temperature.

Result holds the answer, the options, a probability for each option, and the number of input tokens. res.Probability("billing") gives the probability of one option.

Many questions about one state

DecideMany reads the state once and then asks each question.

results, err := d.DecideMany(state, []decide.Question{q1, q2, q3}, "")

Options.ManyMode picks how.

Options

The llama-server API

decide.ParseRequest reads the JSON of the /v1/systemone API that llama-server serves, and Decider.Answer answers it. So a Go program can take the same requests as the server.

In the browser

The package also builds with TinyGo for the browser and runs on pkg/llamawasm. Asking many questions about one state needs a llama.cpp module with ABI 10 or later. An older module still works, and decodes each question on its own. See the wasm/decide example.

A model that loads with decide.Open, such as Julia-1 or Laya, needs yzma 1.29.1 or later in the browser. Clef and Decision 2.0 need yzma 1.30.0 and a llama.cpp module with ABI 11 or later, which is v0.7.0 or later. Jeyzma runs these models in a page.

The example

The decide example runs all of the model families from the command line.

go run ./examples/decide/ -readout gguf -model ~/models/lev-Q8_0.gguf \
    -state '{"ticket": "I was charged twice for order A-104. Please refund the duplicate."}' \
    -question "Which team should handle this?" \
    -options '{"billing": "Charges, invoices, refunds", "technical": "Bugs, outages", "other": ""}'

-readout takes jev, jevk5, decider or gguf. Leave out -options for a true or false question, and add -type score with a JSON list of levels for a score question.