Information-Theoretic Couplings

A coupling of marginal distributions \(P_1, \ldots, P_k\) is a joint distribution whose marginals match the \(P_i\). dit constructs couplings that optimize multivariate information measures subject to those marginal constraints.

These routines are distinct from optimal-transport couplings such as the Earth Mover’s Distance, which minimize expected ground-metric cost [CABE+15].

Coupling constructors

min_residual_entropy_coupling(dists, *, niter=50)

Coupling with minimal residual entropy (variation of information).

Parameters:

• dists (list of Distribution) – Marginal distributions to couple.

• niter (int) – Number of basin-hopping iterations.

Returns:

dist – A joint distribution with the prescribed marginals and approximately minimal residual entropy.

Return type:

Distribution

max_total_correlation_coupling(dists, *, niter=50)

Coupling with maximal total correlation (multi-information).

With fixed marginals, this is equivalent to minimum joint entropy.

Parameters:

• dists (list of Distribution) – Marginal distributions to couple.

• niter (int) – Number of basin-hopping iterations.

Returns:

dist – A joint distribution with the prescribed marginals and approximately maximal total correlation.

Return type:

Distribution

max_dual_total_correlation_coupling(dists, *, niter=50)

Coupling with maximal dual total correlation (binding information).

Parameters:

• dists (list of Distribution) – Marginal distributions to couple.

• niter (int) – Number of basin-hopping iterations.

Returns:

dist – A joint distribution with the prescribed marginals and approximately maximal dual total correlation.

Return type:

Distribution

max_caekl_coupling(dists, *, niter=50)

Coupling with maximal CAEKL mutual information.

Parameters:

• dists (list of Distribution) – Marginal distributions to couple.

• niter (int) – Number of basin-hopping iterations.

Returns:

dist – A joint distribution with the prescribed marginals and approximately maximal CAEKL mutual information.

Return type:

Distribution

Scalar summaries

coupling_min_residual_entropy(dists, *, niter=25)

Minimum residual entropy over couplings with the given marginals.

Parameters:

• dists (list of Distribution) – Marginal distributions to couple.

• niter (int) – Number of basin-hopping iterations.

Returns:

R – The residual entropy of min_residual_entropy_coupling().

Return type:

float

coupling_metric(dists, p=1.0)

Residual entropy of the minimum-entropy coupling with the given marginals.

Note

This uses MinEntOptimizer (minimum joint entropy), not a direct minimization of residual entropy. For the latter, use coupling_min_residual_entropy() or min_residual_entropy_coupling().

Parameters:

• dists (list of Distribution) – The distributions to consider as marginals

• p (float) – The p-norm used when evaluating residual entropy on the coupling.

Returns:

cm – The residual entropy of the minimum joint-entropy coupling.

Return type:

float

The legacy coupling_metric() returns the residual entropy of the minimum joint-entropy coupling (not a direct minimization of residual entropy).