Rényi Entropy¶

The Rényi entropy is a spectrum of generalizations to the Shannon Entropy:

$\RE{\alpha}{X} = \frac{1}{1-\alpha} \log_2 \left( \sum_{x \in \mathcal{X}} p(x)^\alpha \right)$
In [1]: from dit.other import renyi_entropy

In [2]: from dit.example_dists import binomial

In [3]: d = binomial(15, 0.4)

In [4]: renyi_entropy(d, 3)
Out[4]: 2.6611840717104625


Special Cases¶

For several values of $$\alpha$$, the Rényi entropy takes on particular values.

$$\alpha = 0$$¶

When $$\alpha = 0$$ the Rényi entropy becomes what is known as the Hartley entropy:

$\RE{0}{X} = \log_2 |X|$
In [5]: renyi_entropy(d, 0)
Out[5]: 4.0


$$\alpha = 1$$¶

When $$\alpha = 1$$ the Rényi entropy becomes the standard Shannon entropy:

$\RE{1}{X} = \H{X}$
In [6]: renyi_entropy(d, 1)
Out[6]: 2.9688513169509623


$$\alpha = 2$$¶

When $$\alpha = 2$$, the Rényi entropy becomes what is known as the collision entropy:

$\RE{2}{X} = - \log_2 p(X = Y)$

where $$Y$$ is an IID copy of X. This is basically the surprisal of “rolling doubles”

In [7]: renyi_entropy(d, 2)
Out[7]: 2.7607270851693615


$$\alpha = \infty$$¶

Finally, when $$\alpha = \infty$$ the Rényi entropy picks out the probability of the most-probable event:

$\RE{\infty}{X} = - \log_2 \max_{x \in \mathcal{X}} p(x)$
In [8]: renyi_entropy(d, np.inf)
Out[8]: 2.275104563096674


General Properies¶

In general, the Rényi entropy is a monotonically decreasing function in $$\alpha$$:

$\RE{\alpha}{X} \ge \RE{\beta}{X}, \quad \beta > \alpha$

Further, the following inequality holds in the other direction:

$\RE{2}{X} \le 2 \cdot \RE{\infty}{X}$

API¶

renyi_entropy(dist, order, rvs=None, rv_mode=None)[source]

Compute the Renyi entropy of order order.

Parameters: dist (Distribution) – The distribution to take the Renyi entropy of. order (float >= 0) – The order of the Renyi entropy. rvs (list, None) – The indexes of the random variable used to calculate the Renyi entropy of. If None, then the Renyi entropy is calculated over all random variables. rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’. H_a – The Renyi entropy. float ditException – Raised if rvs or crvs contain non-existant random variables. ValueError – Raised if order is not a non-negative float.