Union Information

Finn & Lizier [FL20] decompose joint entropy using pointwise maxima and minima of the marginal surprisals \(h(x_i) = -\log_2 p(x_i)\):

\[\begin{split}H^{\cup}(X_{0:n}) &= \mathbb{E}\bigl[\max_i h(x_i)\bigr] \\ H^{\cap}(X_{0:n}) &= \mathbb{E}\bigl[\min_i h(x_i)\bigr] \\ H^{+}(X_{0:n}) &= \H{X_{0:n}} - H^{\cup}(X_{0:n})\end{split}\]

These are the union entropy, intersection entropy, and synergistic entropy. The unique entropy of one group relative to another is \(H^{\cup}(X,Y) - H(Y)\).

In [1]: from dit.multivariate import union_entropy, intersection_entropy, synergistic_entropy, unique_entropy

In [2]: from dit.example_dists import Xor

In [3]: d = Xor()

In [4]: union_entropy(d)
Out[4]: 1.0

In [5]: intersection_entropy(d)
Out[5]: 1.0

In [6]: synergistic_entropy(d)
Out[6]: 1.0

In [7]: unique_entropy(d, [[0], [1]])
Out[7]: 0.0

The related partial entropy decomposition \(H_{\mathrm{mos}}\) is documented with the Partial Information Decomposition.

API

union_entropy(dist, rvs=None, crvs=None)[source]

Compute the union entropy H(X_1 t X_2 t … t X_n).

The expected surprise of the most surprising marginal realisation:

H(X_1 t … t X_n) = E[ max(h(x_1), …, h(x_n)) ]

Parameters:

  • dist (Distribution) – The distribution from which the union entropy is calculated.

  • rvs (list, None) – The random variable groups. If None, each variable is its own group.

  • crvs (list, None) – Variables to condition on (not supported; must be None or empty).

Returns:

Hu – The union entropy.

Return type:

float

intersection_entropy(dist, rvs=None, crvs=None)[source]

Compute the intersection entropy H(X_1 u X_2 u … u X_n).

The expected surprise of the least surprising marginal realisation:

H(X_1 u … u X_n) = E[ min(h(x_1), …, h(x_n)) ]

Parameters:

  • dist (Distribution) – The distribution from which the intersection entropy is calculated.

  • rvs (list, None) – The random variable groups. If None, each variable is its own group.

  • crvs (list, None) – Variables to condition on (not supported; must be None or empty).

Returns:

Hi – The intersection entropy.

Return type:

float

synergistic_entropy(dist, rvs=None, crvs=None)[source]

Compute the synergistic entropy H(X_1 + X_2 + … + X_n).

How much more information the joint distribution provides beyond what the marginals can share:

H(X_1 + … + X_n) = H(X_1, …, X_n) - H(X_1 t … t X_n)

Equivalently: E[ h(x_1, …, x_n) - max(h(x_1), …, h(x_n)) ]

Parameters:

  • dist (Distribution) – The distribution from which the synergistic entropy is calculated.

  • rvs (list, None) – The random variable groups. If None, each variable is its own group.

  • crvs (list, None) – Variables to condition on (not supported; must be None or empty).

Returns:

Hs – The synergistic entropy.

Return type:

float

unique_entropy(dist, rvs=None, crvs=None)[source]

Compute the unique entropy H(X_1 X_2).

How much more information the first rv group provides relative to the second, on average:

H(X Y) = H(X t Y) - H(Y) = E[ max(h(x) - h(y), 0) ]

Parameters:

  • dist (Distribution) – The distribution from which the unique entropy is calculated.

  • rvs (list, None) – Exactly two random variable groups [rv_a, rv_b]. Returns H(rv_a \ rv_b).

  • crvs (list, None) – Variables to condition on (not supported; must be None or empty).

Returns:

Hu – The unique entropy.

Return type:

float

Raises:

ValueError – If rvs does not contain exactly two groups.