Module lookup 

Prompt-lookup speculative decoding.

A normalized transcript is nearly its input, so the input is already a good draft of the output. Guess the next K tokens by finding where the text just emitted last appeared in the input, validate the whole guess in one batched decode, keep the longest prefix the model agrees with, and drop the KV entries behind the rest.

Reading tokens is near-free next to generating them, so checking K guesses costs about what generating one does and every correct guess is free. Acceptance FALLS as K rises while wall-clock keeps improving, because the longer accepted runs outweigh the wasted drafts: tune this on wall-clock, never on the acceptance rate.

Re-exports

use llama_cpp_2::model::LlamaModel; 🔒

use llama_cpp_2::sampling::LlamaSampler; 🔒

use llama_cpp_2::token::LlamaToken; 🔒

use super::NormalizeError; 🔒

use super::Session; 🔒

use super::decode_piece; 🔒

Constants

K

Draft length, chosen on wall-clock. Acceptance FALLS as this rises while wall-clock keeps improving, so it is not tuned on the acceptance rate.

NGRAM

Match width. Two beats one decisively; three gains nothing.

Functions

draft

The tokens following the most recent occurrence of generated’s last ngram tokens within source, capped at k.

generate

Greedy generation, speculating from source.