Skip to main content

decode_piece

Function decode_piece 

Source
fn decode_piece(decoder: &mut Decoder, bytes: &[u8]) -> String
Expand description

Decode one token’s bytes through a decoder the whole generation reuses. The decoder must persist across tokens: a piece can end mid-character, and a fresh decoder would replace both halves.