Cumulative Residual Entropy

The cumulative residual entropy [RCVW04] is an alternative to the differential Shannon entropy. The differential entropy has many issues, including that it can be negative even for simple distributions such as the uniform distribution; and that if one takes discrete estimates that limit to the continuous distribution, the discrete entropy does not limit to the differential (continuous) entropy. It also attempts to provide meaningful differences between numerically different random variables, such as a die labeled [1, 2, 3, 4, 5, 6] and one lebeled [1, 2, 3, 4, 5, 100].

Note

The Cumulative Residual Entropy is unrelated to Residual Entropy.

\[\CRE{X} = -\int_{0}^{\infty} p(|X| > x) \log_{2} p(|X| > x) dx\]
In [1]: from dit.other import cumulative_residual_entropy

In [2]: d1 = dit.ScalarDistribution([1, 2, 3, 4, 5, 6], [1/6]*6)

In [3]: d2 = dit.ScalarDistribution([1, 2, 3, 4, 5, 100], [1/6]*6)

In [4]: cumulative_residual_entropy(d1)
Out[4]: 2.068318255702844

In [5]: cumulative_residual_entropy(d2)
Out[5]: 22.672680046016705

Generalized Cumulative Residual Entropy

The genearlized form of the cumulative residual entropy integrates over the intire set of reals rather than just the positive ones:

\[\GCRE{X} = -\int_{-\infty}^{\infty} p(X > x) \log_{2} p(X > x) dx\]
In [6]: from dit.other import generalized_cumulative_residual_entropy

In [7]: generalized_cumulative_residual_entropy(d1)
Out[7]: 2.068318255702844

In [8]: d3 = dit.ScalarDistribution([-2, -1, 0, 1, 2], [1/5]*5)

In [9]: cumulative_residual_entropy(d3)
Out[9]: 0.9065649754771961

In [10]: generalized_cumulative_residual_entropy(d3)
Out[10]: 1.6928786893420307

Conditional Cumulative Residual Entropy

The conditional cumulative residual entropy \(\CRE[X|Y]\) is a distribution with the same probability mass function as \(Y\), and the outcome associated with \(p(y)\) is equal to the cumulative residual entropy over probabilities conditioned on \(Y = y\). In this sense the conditional cumulative residual entropy is more akin to a distribution over \(\H[X|Y=y]\) than the single scalar quantity \(\H[X|Y]\).

\[\CRE{X | Y} = - \int_{0}^{\infty} p(|X| > x | Y) \log_{2} p(|X| > x | Y) dx\]

Conditional Generalized Cumulative Residual Entropy

Conceptually the conditional generalized cumulative residual entropy is the same as the non-generalized form, but integrated over the entire real line rather than just the positive:

\[\GCRE{X | Y} = - \int_{-\infty}^{\infty} p(X > x | Y) \log_{2} p(X > x | Y) dx\]

API

cumulative_residual_entropy(dist, extract=False)[source]

The cumulative residual entropy is an alternative to the Shannon differential entropy with several desirable properties including non-negativity.

Parameters:
  • dist (Distribution) – The distribution to compute the cumulative residual entropy of each index for.
  • extract (bool) – If True and dist.outcome_length() is 1, return the single GCRE value rather than a length-1 array.
Returns:

CREs – The cumulative residual entropy for each index.

Return type:

ndarray

Examples

>>> d1 = ScalarDistribution([1, 2, 3, 4, 5, 6], [1/6]*6)
>>> d2 = ScalarDistribution([1, 2, 3, 4, 5, 100], [1/6]*6)
>>> cumulative_residual_entropy(d1)
2.0683182557028439
>>> cumulative_residual_entropy(d2)
22.672680046016705
generalized_cumulative_residual_entropy(dist, extract=False)[source]

The generalized cumulative residual entropy is a generalized from of the cumulative residual entropy. Rather than integrating from 0 to infinity over the absolute value of the CDF.

Parameters:
  • dist (Distribution) – The distribution to compute the generalized cumulative residual entropy of each index for.
  • extract (bool) – If True and dist.outcome_length() is 1, return the single GCRE value rather than a length-1 array.
Returns:

GCREs – The generalized cumulative residual entropy for each index.

Return type:

ndarray

Examples

>>> generalized_cumulative_residual_entropy(uniform(-2, 3))
1.6928786893420307
>>> generalized_cumulative_residual_entropy(uniform(0, 5))
1.6928786893420307

Conditional Forms

conditional_cumulative_residual_entropy(dist, rv, crvs=None, rv_mode=None)[source]

Returns the conditional cumulative residual entropy.

Parameters:
  • dist (Distribution) – The distribution to compute the conditional cumulative residual entropy of.
  • rv (list, None) – The possibly joint random variable to compute the conditional cumulative residual entropy of. If None, then all variables not in crvs are used.
  • crvs (list, None) – The random variables to condition on. If None, nothing is conditioned on.
  • rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.
Returns:

CCRE – The conditional cumulative residual entropy.

Return type:

ScalarDistribution

Examples

>>> from itertools import product
>>> events = [ (a, b) for a, b, in product(range(5), range(5)) if a <= b ]
>>> probs = [ 1/(5-a)/5 for a, b in events ]
>>> d = Distribution(events, probs)
>>> print(conditional_cumulative_residual_entropy(d, 1, [0]))
Class:    ScalarDistribution
Alphabet: (-0.0, 0.5, 0.91829583405448956, 1.3112781244591329, 1.6928786893420307)
Base:     linear

x p(x) -0.0 0.2 0.5 0.2 0.918295834054 0.2 1.31127812446 0.2 1.69287868934 0.2

conditional_generalized_cumulative_residual_entropy(dist, rv, crvs=None, rv_mode=None)[source]

Returns the conditional cumulative residual entropy.

Parameters:
  • dist (Distribution) – The distribution to compute the conditional generalized cumulative residual entropy of.
  • rv (list, None) – The possibly joint random variable to compute the conditional generalized cumulative residual entropy of. If None, then all variables not in crvs are used.
  • crvs (list, None) – The random variables to condition on. If None, nothing is conditioned on.
  • rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.
Returns:

CCRE – The conditional cumulative residual entropy.

Return type:

ScalarDistribution

Examples

>>> from itertools import product
>>> events = [ (a-2, b-2) for a, b, in product(range(5), range(5)) if a <= b ]
>>> probs = [ 1/(3-a)/5 for a, b in events ]
>>> d = Distribution(events, probs)
>>> print(conditional_generalized_cumulative_residual_entropy(d, 1, [0]))
Class:    ScalarDistribution
Alphabet: (-0.0, 0.5, 0.91829583405448956, 1.3112781244591329, 1.6928786893420307)
Base:     linear

x p(x) -0.0 0.2 0.5 0.2 0.918295834054 0.2 1.31127812446 0.2 1.69287868934 0.2