Rényi Entropy

The Rényi entropy is a spectrum of generalizations to the Shannon Entropy:

\[\RE{\alpha}{X} = \frac{1}{1-\alpha} \log_2 \left( \sum_{x \in \mathcal{X}} p(x)^\alpha \right)\]

In [1]: from dit.other import renyi_entropy

In [2]: from dit.example_dists import binomial

In [3]: d = binomial(15, 0.4)

In [4]: renyi_entropy(d, 3)
Out[4]: 2.6611840717104625

Special Cases

For several values of \(\alpha\), the Rényi entropy takes on particular values.

\(\alpha = 0\)

When \(\alpha = 0\) the Rényi entropy becomes what is known as the Hartley entropy:

\[\RE{0}{X} = \log_2 |X|\]

In [5]: renyi_entropy(d, 0)
Out[5]: 4.0

\(\alpha = 1\)

When \(\alpha = 1\) the Rényi entropy becomes the standard Shannon entropy:

\[\RE{1}{X} = \H{X}\]

In [6]: renyi_entropy(d, 1)
Out[6]: 2.968851316950962

\(\alpha = 2\)

When \(\alpha = 2\), the Rényi entropy becomes what is known as the collision entropy:

\[\RE{2}{X} = - \log_2 p(X = Y)\]

where \(Y\) is an IID copy of X. This is basically the surprisal of “rolling doubles”

In [7]: renyi_entropy(d, 2)
Out[7]: 2.7607270851693615

\(\alpha = \infty\)

Finally, when \(\alpha = \infty\) the Rényi entropy picks out the probability of the most-probable event:

\[\RE{\infty}{X} = - \log_2 \max_{x \in \mathcal{X}} p(x)\]

In [8]: renyi_entropy(d, np.inf)
Out[8]: 2.2751045630966735

General Properies

In general, the Rényi entropy is a monotonically decreasing function in \(\alpha\):

\[\RE{\alpha}{X} \ge \RE{\beta}{X}, \quad \beta > \alpha\]

Further, the following inequality holds in the other direction:

\[\RE{2}{X} \le 2 \cdot \RE{\infty}{X}\]

API

See also Sibson Mutual Information for the Sibson (\(\alpha\)-) generalization of mutual information built from Rényi divergence.

renyi_entropy(dist, order, rvs=None)[source]

Compute the Renyi entropy of order order.

Parameters:

dist (Distribution) – The distribution to take the Renyi entropy of.
order (float >= 0) – The order of the Renyi entropy.
rvs (list, None) – The indexes of the random variable used to calculate the Renyi entropy of. If None, then the Renyi entropy is calculated over all random variables.

Returns:

H_a – The Renyi entropy.

Return type:

float

Raises:

ditException – Raised if rvs or crvs contain non-existant random variables.
ValueError – Raised if order is not a non-negative float.