MIROSTAT v2 EXPLAINED
VARIABLES
- tau
your target negative log prob or target 'surprise' value that it tries to average towards. e.g 9.95 tau. think of it astarget_log_prob
. - observed_surprise
the negative log probability of the chosen token. a funky way to measure how far from the top probability you are. e.g 4.29 'surprise value' - error
this is determiningobserved_surprise - tau
. whatever value it gets (if its negative or positive) will be subtracted from mu later in an effort to get mu closer to the tau. - mu
mu's final value, after error correction, is the 'surprise threshold' used for picking the next token. by default its 2x tau value i think, that doesn't last for more than one token though.
mu = mu - (eta * error)
- eta
learning rate at which it factors in the last token, as seen in the mu calculation above. if eta = 1.0, the calculation can be simplified as:
mu = mu - error
that's because mu will be identical to the last token's observed_surprise before 'error correction'.
DEMONSTRATION OF MATHS
so lets say we have:
- 9.95 tau, mu 19.9
and lets say it picked a token with an observed_surprise of 6.0
6.0 (observed_surprise) - 9.95 (tau) = -3.95 (error)
if mu is 19.9 right now...
mu = mu - (1.0 * -3.95)
then it becomes 23.85, and that's used as the next 'surprise threshold' (mu) of what's acceptable to choose from, because 6.0 wasn't very surprising compared to what the tau targets. it tries to swing back and forth hitting that target tau on average.