CAEKL Mutual Information

The Chan-AlBashabsheh-Ebrahimi-Kaced-Liu mutual information [CABE+15] is one possible generalization of the Mutual Information.

\(\J{X_{0:n}}\) is the smallest \(\gamma\) such that:

\[\H{X_{0:n}} - \gamma = \sum_{C \in \mathcal{P}} \left[ \H{X_C} - \gamma \right]\]

for some non-trivial partition \(\mathcal{P}\) of \(\left\{0:n\right\}\). For example, the CAEKL mutual information for the xor distribution is \(\frac{1}{2}\), because the joint entropy is 2 bits, each of the three marginals is 1 bit, and \(2 - \frac{1}{2} = 3 (1 - \frac{1}{2})\).

In [1]: In [1]: from dit.multivariate import caekl_mutual_information as J

A more concrete way of defining the CAEKL mutual information is:

\[\J{X_{0:n}} = \min_{\mathcal{P} \in \Pi} ~ \operatorname{I}_\mathcal{P}\left[X_{0:n}\right]\]

where \(\operatorname{I}_\mathcal{P}\) is the total_correlation of the partition:

\[\operatorname{I}_\mathcal{P}\left[X_{0:n}\right] = \sum_{C \in \mathcal{P}} \H{X_C} - \H{X_{0:n}}\]

and \(\Pi\) is the set of all non-trivial partitions of \(\left\{0:n\right\}\).

API

caekl_mutual_information(dist, rvs=None, crvs=None, rv_mode=None)[source]

Calculates the Chan-AlBashabsheh-Ebrahimi-Kaced-Liu mutual information.

Parameters

  • dist (Distribution) – The distribution from which the CAEKL mutual information is calculated.

  • rvs (list, None) – A list of lists. Each inner list specifies the indexes of the random variables used to calculate the total correlation. If None, then the total correlation is calculated over all random variables, which is equivalent to passing rvs=dist.rvs.

  • crvs (list, None) – A single list of indexes specifying the random variables to condition on. If None, then no variables are conditioned on.

  • rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.

Returns

J – The CAEKL mutual information.

Return type

float

Examples

>>> d = dit.example_dists.Xor()
>>> dit.multivariate.caekl_mutual_information(d)
0.5
>>> dit.multivariate.caekl_mutual_information(d, rvs=[[0], [1]])
0.0
Raises

ditException – Raised if dist is not a joint distribution or if rvs or crvs contain non-existant random variables.