DeWeese-like Measures

Mike DeWeese has introduced a family of multivariate information measures based on a multivariate extension of the data processing inequality [JEC17]. The general idea is the following: local modification of a single variable can not increase the amount of correlation or dependence it has with the other variables. Consider, however, the triadic distribution:

Note

TODO: confirm the canonical reference for Mike DeWeese’s construction. The formulation appears in [JEC17], but a standalone DeWeese publication may exist.

In [1]: from dit.example_dists import dyadic, triadic

In [2]: print(triadic)
Class:    Distribution
Alphabet: (('0', '1', '2', '3'), ('0', '1', '2', '3'), ('0', '1', '2', '3'))
Base:     linear

x                 p(X0,X1,X2)
('0', '0', '0')   1/8
('0', '2', '2')   1/8
('1', '1', '1')   1/8
('1', '3', '3')   1/8
('2', '0', '2')   1/8
('2', '2', '0')   1/8
('3', '1', '3')   1/8
('3', '3', '1')   1/8

This particular distribution has zero Co-Information:

In [3]: from dit.multivariate import coinformation

In [4]: coinformation(triadic)
Out[4]: 0.0

Yet the distribution is a product of a giant bit (coinformation \(1.0\)) and the xor (coinformation \(-1.0\)), and so there exists within it the capability of having a coinformation of \(1.0\) if the xor component were dropped. This is exactly what the DeWeese construction captures:

\[\ID{X_0 : \ldots : X_n} = \max_{p(x'_i | x_i)} \I{X'_0 : \ldots : X'_n}\]
In [5]: from dit.multivariate import deweese_coinformation

In [6]: deweese_coinformation(triadic)
Out[6]: 0.9999999999999999

DeWeese version of the Total Correlation, Dual Total Correlation, and CAEKL Mutual Information are also available, and operate on an arbitrary number of variables with optional conditional variables.

API

deweese_coinformation(dist, rvs=None, crvs=None, niter=None, deterministic=False)

Compute the DeWeese coinformation.

Parameters:

  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

Returns:

val – The value of the DeWeese coinformation.

Return type:

float

deweese_total_correlation(dist, rvs=None, crvs=None, niter=None, deterministic=False)

Compute the DeWeese total correlation.

Parameters:

  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

Returns:

val – The value of the DeWeese total correlation.

Return type:

float

deweese_dual_total_correlation(dist, rvs=None, crvs=None, niter=None, deterministic=False)

Compute the DeWeese dual total correlation.

Parameters:

  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

Returns:

val – The value of the DeWeese dual total correlation.

Return type:

float

deweese_caekl_mutual_information(dist, rvs=None, crvs=None, niter=None, deterministic=False)

Compute the DeWeese caekl mutual information.

Parameters:

  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

Returns:

val – The value of the DeWeese caekl mutual information.

Return type:

float