DeWeese-like Measures

Mike DeWeese has introduced a family of multivariate information measures based on a multivariate extension of the data processing inequality. The general idea is the following: local modification of a single variable can not increase the amount of correlation or dependence it has with the other variables. Consider, however, the triadic distribution:

In [1]: In [1]: from dit.example_dists import dyadic, triadic

This particular distribution has zero coinformation:

In [2]: In [3]: from dit.multivariate import coinformation

Yet the distribution is a product of a giant bit (coinformation \(1.0\)) and the xor (coinformation \(-1.0\)), and so there exists within it the capability of having a coinformation of \(1.0\) if the xor component were dropped. This is exactly what the DeWeese construction captures:

\[\ID{X_0 : \ldots : X_n} = \max_{p(x'_i | x_i)} \I{X'_0 : \ldots : X'_n}\]
In [3]: In [5]: from dit.multivariate import deweese_coinformation

DeWeese version of the total_correlation, dual_total_correlation, and caekl_mutual_information are also available, and operate on an arbitrary number of variables with optional conditional variables.

API

deweese_coinformation(dist, rvs=None, crvs=None, niter=None, deterministic=False, rv_mode=None)

Compute the DeWeese coinformation.

Parameters
  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

  • rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.

Returns

val – The value of the DeWeese coinformation.

Return type

float

deweese_total_correlation(dist, rvs=None, crvs=None, niter=None, deterministic=False, rv_mode=None)

Compute the DeWeese total correlation.

Parameters
  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

  • rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.

Returns

val – The value of the DeWeese total correlation.

Return type

float

deweese_dual_total_correlation(dist, rvs=None, crvs=None, niter=None, deterministic=False, rv_mode=None)

Compute the DeWeese dual total correlation.

Parameters
  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

  • rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.

Returns

val – The value of the DeWeese dual total correlation.

Return type

float

deweese_caekl_mutual_information(dist, rvs=None, crvs=None, niter=None, deterministic=False, rv_mode=None)

Compute the DeWeese caekl mutual information.

Parameters
  • dist (Distribution) – The distribution of interest.

  • rvs (iter of iters, None) – The random variables of interest. If None, use all.

  • crvs (iter, None) – The variables to condition on. If None, none.

  • niter (int, None) – If specified, the number of optimization steps to perform.

  • deterministic (bool) – Whether the functions to optimize over should be deterministic or not. Defaults to False.

  • rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.

Returns

val – The value of the DeWeese caekl mutual information.

Return type

float