Cumulative Residual Entropy

The cumulative residual entropy [RCVW04] is an alternative to the differential Shannon entropy. The differential entropy has many issues, including that it can be negative even for simple distributions such as the uniform distribution; and that if one takes discrete estimates that limit to the continuous distribution, the discrete entropy does not limit to the differential (continuous) entropy. It also attempts to provide meaningful differences between numerically different random variables, such as a die labeled [1, 2, 3, 4, 5, 6] and one lebeled [1, 2, 3, 4, 5, 100].

Note

The Cumulative Residual Entropy is unrelated to Residual Entropy.

\[\CRE{X} = -\int_{0}^{\infty} p(|X| > x) \log_{2} p(|X| > x) dx\]

In [1]: from dit.other import cumulative_residual_entropy

In [2]: d1 = dit.Distribution([1, 2, 3, 4, 5, 6], [1/6]*6)

In [3]: d2 = dit.Distribution([1, 2, 3, 4, 5, 100], [1/6]*6)

In [4]: cumulative_residual_entropy(d1)
Out[4]: 2.068318255702844

In [5]: cumulative_residual_entropy(d2)
Out[5]: 22.6726800460167

Generalized Cumulative Residual Entropy

The genearlized form of the cumulative residual entropy integrates over the intire set of reals rather than just the positive ones:

\[\GCRE{X} = -\int_{-\infty}^{\infty} p(X > x) \log_{2} p(X > x) dx\]

In [6]: from dit.other import generalized_cumulative_residual_entropy

In [7]: generalized_cumulative_residual_entropy(d1)
Out[7]: 2.068318255702844

In [8]: d3 = dit.Distribution([-2, -1, 0, 1, 2], [1/5]*5)

In [9]: cumulative_residual_entropy(d3)
Out[9]: 0.9065649754771961

In [10]: generalized_cumulative_residual_entropy(d3)
Out[10]: 1.692878689342031

Conditional Cumulative Residual Entropy

The conditional cumulative residual entropy \(\CRE{X|Y}\) is a distribution with the same probability mass function as \(Y\), and the outcome associated with \(p(y)\) is equal to the cumulative residual entropy over probabilities conditioned on \(Y = y\). In this sense the conditional cumulative residual entropy is more akin to a distribution over \(\H{X|Y=y}\) than the single scalar quantity \(\H{X|Y}\).

\[\CRE{X | Y} = - \int_{0}^{\infty} p(|X| > x | Y) \log_{2} p(|X| > x | Y) dx\]

Conditional Generalized Cumulative Residual Entropy

Conceptually the conditional generalized cumulative residual entropy is the same as the non-generalized form, but integrated over the entire real line rather than just the positive:

\[\GCRE{X | Y} = - \int_{-\infty}^{\infty} p(X > x | Y) \log_{2} p(X > x | Y) dx\]

API

cumulative_residual_entropy(dist, extract=False)[source]

The cumulative residual entropy is an alternative to the Shannon differential entropy with several desirable properties including non-negativity.

Parameters:

dist (Distribution) – The distribution to compute the cumulative residual entropy of each index for.
extract (bool) – If True and dist.outcome_length() is 1, return the single GCRE value rather than a length-1 array.

Returns:

CREs – The cumulative residual entropy for each index.

Return type:

ndarray

Examples

>>> d1 = Distribution([1, 2, 3, 4, 5, 6], [1/6]*6)
>>> d2 = Distribution([1, 2, 3, 4, 5, 100], [1/6]*6)
>>> cumulative_residual_entropy(d1)
2.0683182557028439
>>> cumulative_residual_entropy(d2)
22.672680046016705

generalized_cumulative_residual_entropy(dist, extract=False)[source]

The generalized cumulative residual entropy is a generalized from of the cumulative residual entropy. Rather than integrating from 0 to infinity over the absolute value of the CDF.

Parameters:

dist (Distribution) – The distribution to compute the generalized cumulative residual entropy of each index for.
extract (bool) – If True and dist.outcome_length() is 1, return the single GCRE value rather than a length-1 array.

Returns:

GCREs – The generalized cumulative residual entropy for each index.

Return type:

ndarray

Examples

>>> generalized_cumulative_residual_entropy(uniform(-2, 3))
1.6928786893420307
>>> generalized_cumulative_residual_entropy(uniform(0, 5))
1.6928786893420307

Conditional Forms

conditional_cumulative_residual_entropy(dist, rv, crvs=None)[source]

Returns the conditional cumulative residual entropy.

Parameters:

dist (Distribution) – The distribution to compute the conditional cumulative residual entropy of.
rv (list, None) – The possibly joint random variable to compute the conditional cumulative residual entropy of. If None, then all variables not in crvs are used.
crvs (list, None) – The random variables to condition on. If None, nothing is conditioned on.

Returns:

CCRE – The conditional cumulative residual entropy.

Return type:

Distribution

Examples

>>> from itertools import product
>>> events = [ (a, b) for a, b, in product(range(5), range(5)) if a <= b ]
>>> probs = [ 1/(5-a)/5 for a, b in events ]
>>> d = Distribution(events, probs)
>>> print(conditional_cumulative_residual_entropy(d, 1, [0]))
Class:    Distribution
Alphabet: (-0.0, 0.5, 0.91829583405448956, 1.3112781244591329, 1.6928786893420307)
Base:     linear

x p(x) -0.0 0.2 0.5 0.2 0.918295834054 0.2 1.31127812446 0.2 1.69287868934 0.2

conditional_generalized_cumulative_residual_entropy(dist, rv, crvs=None)[source]

Returns the conditional cumulative residual entropy.

Parameters:

dist (Distribution) – The distribution to compute the conditional generalized cumulative residual entropy of.
rv (list, None) – The possibly joint random variable to compute the conditional generalized cumulative residual entropy of. If None, then all variables not in crvs are used.
crvs (list, None) – The random variables to condition on. If None, nothing is conditioned on.

Returns:

CCRE – The conditional cumulative residual entropy.

Return type:

Distribution

Examples

>>> from itertools import product
>>> events = [ (a-2, b-2) for a, b, in product(range(5), range(5)) if a <= b ]
>>> probs = [ 1/(3-a)/5 for a, b in events ]
>>> d = Distribution(events, probs)
>>> print(conditional_generalized_cumulative_residual_entropy(d, 1, [0]))
Class:    Distribution
Alphabet: (-0.0, 0.5, 0.91829583405448956, 1.3112781244591329, 1.6928786893420307)
Base:     linear

x p(x) -0.0 0.2 0.5 0.2 0.918295834054 0.2 1.31127812446 0.2 1.69287868934 0.2