Chapter 4: LLM inference

Temperature and entropy: how sampling reshapes the distribution

Most apps that run a language model have a temperature setting. It divides the model's scores before softmax, so the next word becomes more sure or more varied. Interviewers ask what it does to the chances and to the entropy, at very low and very high values, and with top-p.

18 minInteractive lessonChapter 4: LLM inferenceLesson 2 of 12

What you will learn

  • Compute a distribution and its entropy at several temperatures
  • Explain the limits T to 0 and T to infinity
  • Explain how temperature interacts with top-p

Key terms

logit
a raw score the model gives each possible next word
softmax
turns scores into chances that add up to 1
temperature T
a number above 0 that divides every score first
entropy H
how unsure a choice is, measured in bits

Prerequisites

Take Greedy, beam, top-k and top-p decoding first. It comes right before this lesson in the course.

Where this lesson picks up: In the decoding lesson, greedy took the top word, and top-p kept the fewest top words whose chances add up to at least p. Both rules used the model's chances as they came, so how do you make those chances sharper or flatter?

What comes next

Even with a good temperature, a model can say the same phrase again and again, so how do you stop it?

Next lesson: Repetition, penalties, constrained decoding and determinism

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.