Information Measures

dit supports many information measures, ranging from as standard as the Shannon entropy to as exotic as Gács-Körner common information. We organize these quantities into the following groups.

We first have the Shannon-like measures. These quantities are based on sums and differences of entropies, conditional entropies, or mutual informations of random variables:

Common informations quantify shared information through constructions (common random variables, auxiliary channels, sufficient statistics) rather than sums of entropies; in the bivariate case they do not reduce to the mutual information:

This next group of measures can not be represented on information diagrams, and can not really be directly compared to the measures above:

There are also measures of “distance” or divergence between two (and in some cases, more) distributions:

Several measures of shared information are related to the ability of two (or more) agents to agree upon a secret key in the face of an eavesdropper: