Aditya Venkatesh's Personal Website

What is Temperature in LLMs?

When it comes to LLMs, temperature is a key parameter that controls the randomness or creativity of the text the model generates. It acts like a dial that influences how adventurous or predictable the model’s word choices are when producing language, essentially shaping the style and variety of the output.

How Temperature Works

Temperature works by adjusting the probability distribution over possible next words before the model selects its output. Internally, the model computes logits—raw, unnormalized scores for each potential token. Temperature rescales these logits, effectively sharpening or flattening the probability distribution. A low temperature value (close to zero) makes the distribution peak sharply around the highest-probability words, making the output highly predictable and deterministic. Conversely, a higher temperature flattens the distribution, increasing the chances of selecting less likely tokens and thus boosting output diversity and creativity.

Why Temperature Matters

This parameter is crucial because it allows users to balance coherence and inventiveness based on their needs. For example, when factual accuracy and reliability are paramount—such as in technical writing or data-driven responses—a low temperature is preferred to generate consistent and precise text. On the other hand, creative tasks like storytelling, brainstorming, or poetry benefit from a higher temperature, which encourages novel and varied outputs. Adjusting temperature helps tailor the model’s behavior to different applications, making it a versatile tool in managing language generation.

Practical Implications and Examples

Typically, temperature values range from 0 to 1, though values above 1 are sometimes used for even more randomness. A common low setting might be around 0.2 to 0.5, yielding safe and focused text. Mid-range values around 0.7 strike a balance, offering natural yet slightly inventive language. At high values near 1 or beyond, outputs become more unpredictable and sometimes quirky, which can be exciting but may also introduce nonsensical or irrelevant content.