Sibson Mutual Information

Sibson (or \(\alpha\)-) mutual information generalizes Shannon mutual information. At \(\alpha = 1\) it equals Shannon MI; at \(\alpha = \infty\) it equals maximal leakage from \(X\) to \(Y\).

\[I_\alpha(X;Y) = \min_{Q_Y} D_\alpha(P_{XY} \| P_X \otimes Q_Y) = \frac{\alpha}{\alpha - 1} \log_2 \sum_y \left(\sum_x P(x)\, P(y|x)^\alpha\right)^{1/\alpha}\]

The measure is asymmetric in \((X, Y)\): the first argument is the source whose marginal \(P(x)\) appears in the sum.

In [1]: from dit.other import sibson_mutual_information, maximal_leakage

In [2]: from dit.example_dists import Xor

In [3]: from dit import Distribution

In [4]: d = Distribution(["00", "11"], [0.5, 0.5])

In [5]: sibson_mutual_information(d, [0], [1], 2)
Out[5]: 1.0000000000000002

In [6]: maximal_leakage(d, [0], [1])
Out[6]: 1.0

Conditional variants

Two conditional Sibson measures from Esposito et al. (2021) are provided:

sibson_conditional_mutual_information_y_given_z — minimizes over \(Q_{Y|Z}\); reduces to unconditional Sibson MI when \(Z\) is constant.
sibson_conditional_mutual_information_z — minimizes over \(Q_Z\); symmetric in \(X\) and \(Y\).

sibson_mutual_information(dist, rvs_X, rvs_Y, order)[source]

Compute the Sibson mutual information of order order.

This is asymmetric in (X, Y): rvs_X is the source variable whose marginal appears in the definition (Wu et al., Verdu).

Parameters:

dist (Distribution) – The joint distribution.
rvs_X (list) – Indexes of the random variables defining X.
rvs_Y (list) – Indexes of the random variables defining Y.
order (float > 0) – The order alpha.

Returns:

I_a – Sibson mutual information in bits.

Return type:

float

Raises:

ValueError – If order is not positive.

sibson_mutual_information_pmf(p_xy, order)[source]

Compute the Sibson mutual information of order order.

Parameters:

p_xy (array_like, shape (n_x, n_y)) – Joint PMF with axis 0 indexing X and axis 1 indexing Y.
order (float > 0) – The order alpha. Use 1 for Shannon MI and numpy.inf for maximal leakage.

Returns:

I_a – Sibson mutual information in bits.

Return type:

float

Raises:

ValueError – If order is not positive.

maximal_leakage(dist, rvs_X, rvs_Y)[source]

Maximal leakage from X to Y.

Equivalent to Sibson mutual information of order infinity.

Parameters:

dist (Distribution) – The joint distribution.
rvs_X (list) – Indexes of the source random variables.
rvs_Y (list) – Indexes of the observed random variables.

Returns:

L – Maximal leakage in bits.

Return type:

float

sibson_conditional_mutual_information_y_given_z(dist, rvs_X, rvs_Y, rvs_Z, order)[source]

Conditional Sibson MI minimizing over Q_{Y|Z} (Esposito et al., Def. 3).

Reduces to unconditional Sibson MI when Z is constant (Esposito et al., Def. 3).

Parameters:

dist (Distribution) – The joint distribution over X, Y, and Z.
rvs_X (list) – Indexes defining each variable group.
rvs_Y (list) – Indexes defining each variable group.
rvs_Z (list) – Indexes defining each variable group.
order (float > 0) – The order alpha.

Returns:

I_a – Conditional Sibson mutual information in bits.

Return type:

float

sibson_conditional_mutual_information_z(dist, rvs_X, rvs_Y, rvs_Z, order)[source]

Conditional Sibson MI minimizing over Q_Z (Esposito et al., Def. 4).

Symmetric in X and Y.

Parameters:

dist (Distribution) – The joint distribution over X, Y, and Z.
rvs_X (list) – Indexes defining each variable group.
rvs_Y (list) – Indexes defining each variable group.
rvs_Z (list) – Indexes defining each variable group.
order (float > 0) – The order alpha.

Returns:

I_a – Conditional Sibson mutual information in bits.

Return type:

float