DeWeese-like Measures

Mike DeWeese has introduced a family of multivariate information measures based on a multivariate extension of the data processing inequality [JEC17]. The general idea is the following: local modification of a single variable can not increase the amount of correlation or dependence it has with the other variables. Consider, however, the triadic distribution:

Note

TODO: confirm the canonical reference for Mike DeWeese’s construction. The formulation appears in [JEC17], but a standalone DeWeese publication may exist.

In [1]: from dit.example_dists import dyadic, triadic

In [2]: print(triadic)
Class:    Distribution
Alphabet: (('0', '1', '2', '3'), ('0', '1', '2', '3'), ('0', '1', '2', '3'))
Base:     linear

x                 p(X0,X1,X2)
('0', '0', '0')   1/8
('0', '2', '2')   1/8
('1', '1', '1')   1/8
('1', '3', '3')   1/8
('2', '0', '2')   1/8
('2', '2', '0')   1/8
('3', '1', '3')   1/8
('3', '3', '1')   1/8

This particular distribution has zero Co-Information:

In [3]: from dit.multivariate import coinformation

In [4]: coinformation(triadic)
Out[4]: 0.0

Yet the distribution is a product of a giant bit (coinformation \(1.0\)) and the xor (coinformation \(-1.0\)), and so there exists within it the capability of having a coinformation of \(1.0\) if the xor component were dropped. This is exactly what the DeWeese construction captures:

\[\ID{X_0 : \ldots : X_n} = \max_{p(x'_i | x_i)} \I{X'_0 : \ldots : X'_n}\]
In [5]: from dit.multivariate import deweese_coinformation

In [6]: deweese_coinformation(triadic)
Out[6]: 1.0

DeWeese version of the Total Correlation, Dual Total Correlation, and CAEKL Mutual Information are also available, and operate on an arbitrary number of variables with optional conditional variables.

API

deweese_coinformation(dist, rvs=None, crvs=None, niter=None, deterministic=False)

Compute the DeWeese coinformation.

Parameters:

  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

Returns:

val – The value of the DeWeese coinformation.

Return type:

float

deweese_total_correlation(dist, rvs=None, crvs=None, niter=None, deterministic=False)

Compute the DeWeese total correlation.

Parameters:

  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

Returns:

val – The value of the DeWeese total correlation.

Return type:

float

deweese_dual_total_correlation(dist, rvs=None, crvs=None, niter=None, deterministic=False)

Compute the DeWeese dual total correlation.

Parameters:

  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

Returns:

val – The value of the DeWeese dual total correlation.

Return type:

float

deweese_caekl_mutual_information(dist, rvs=None, crvs=None, niter=None, deterministic=False)

Compute the DeWeese caekl mutual information.

Parameters:

  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

Returns:

val – The value of the DeWeese caekl mutual information.

Return type:

float