Copy Mutual Information

The copy mutual information [KCM19] is a measure capturing the portion of the Mutual Information between \(X\) and \(Y\) which is due to \(X=Y\):

\[\begin{split}\op{I^{copy}}[X \to Y] = \sum_{x \in \mathcal{X}} p(X = x) \begin{cases} d_{KL}\left(p(Y=x|X=x)||p(Y=x)\right) & \textrm{if} p(Y=x|X=x) > p(Y=x) \\ 0 & \textrm{otherwise} \end{cases}\end{split}\]

Consider the binary symmetric channel. With probabilities \(\leq \frac{1}{2}\), the input (\(X\)) is largely copied to the output (\(Y\)); while when the probabilities \(\geq \frac{1}{2}\), the output is largely opposite the input. We therefore expect the mutual information to be “copy-like” for \(0 \leq p \leq \frac{1}{2}\), while the mutual information should be not “copy-like” for \(\frac{1}{2} \leq p \leq 1\):

In [1]: In [1]: from dit.divergences import copy_mutual_information as Icopy

API

copy_mutual_information(dist, X, Y, rv_mode=None)[source]

Computes the copy mutual information. Roughly, it is the portion of the mutual information which results from X = Y.

Parameters

  • dist (Distribution) – The distribution of interest.

  • X (iterable) – The indicies to consider as X.

  • Y (iterable) – The indicies to consider as Y.

  • rv_mode (str, None) – Specifies how to interpret crvs and rvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random varible names. If None, then the value of self._rv_mode is consulted, which defaults to ‘indices’.

Returns

Icopy – The copy mutual information of x.

Return type

float