Negentropy

The negentropy [Bri53] measures how far a distribution is from uniformity. It is defined as the difference between the entropy of a uniform distribution over the same alphabet and the entropy of the distribution itself:

\[\N{X} = \sum_{i} \log_2 |\mathcal{X}_i| - \H{X}\]

where \(|\mathcal{X}_i|\) is the cardinality of the alphabet of the \(i\)th random variable. Since the uniform distribution maximizes the entropy, the negentropy is non-negative, and is zero if and only if the distribution is uniform.

Unlike most measures in dit, the negentropy depends on the cardinality of the alphabet and not just the probabilities. For example, the xor distribution is one bit away from uniform:

In [1]: from dit.other import negentropy

In [2]: from dit.example_dists import Xor

In [3]: negentropy(Xor())
Out[3]: 1.0

API

negentropy(dist, rvs=None)[source]

Compute the negentropy of a distribution.

The negentropy is the difference between the entropy of a uniform distribution over the same alphabet and the entropy of the distribution itself. It is a non-negative quantity which is zero if and only if the distribution is uniform, and quantifies how far the distribution is from uniformity.

Parameters:

dist (Distribution) – The distribution from which the negentropy is calculated.
rvs (list, None) – The indexes of the random variables used to calculate the negentropy. If None, then the negentropy is calculated over all random variables.

Returns:

N – The negentropy.

Return type:

float

Examples

>>> d = dit.example_dists.Xor()
>>> dit.other.negentropy(d)
1.0

Raises:: ditException – Raised if rvs contain non-existant random variables.