Perplexity

The perplexity is a trivial measure to make the Entropy more intuitive:

\[\P{X} = 2^{\H{X}}\]

The perplexity of a random variable is the size of a uniform distribution that would have the same entropy. For example, a distribution with 2 bits of entropy has a perplexity of 4, and so could be said to be “as random” as a four-sided die.

The conditional perplexity is defined in the natural way:

\[\P{X | Y} = 2^{\H{X | Y}}\]

We can see that the xor distribution is “4-way” perplexed:

In [1]: from dit.other import perplexity

In [2]: from dit.example_dists import Xor

In [3]: perplexity(Xor())
Out[3]: 4.0

API

perplexity(dist, rvs=None, crvs=None)[source]

Parameters:

dist (Distribution) – The distribution from which the perplexity is calculated.
rvs (list, None) – The indexes of the random variable used to calculate the perplexity. If None, then the perpelxity is calculated over all random variables.
crvs (list, None) – The indexes of the random variables to condition on. If None, then no variables are condition on.

Returns:

P – The perplexity.

Return type:

float