Generalized Divergences

Several one-parameter families extend the Kullback-Leibler Divergence. dit implements the \(\alpha\)-divergence, Hellinger divergence, Rényi divergence, Tsallis divergence, and a generic Csiszár \(f\)-divergence [CT06].

In [1]: from dit.divergences import alpha_divergence, hellinger_divergence, renyi_divergence, tsallis_divergence

In [2]: p = dit.Distribution(['0', '1'], [3/4, 1/4])

In [3]: q = dit.Distribution(['0', '1'], [1/2, 1/2])

In [4]: renyi_divergence(p, q, alpha=2)
Out[4]: 0.32192809488736235

In [5]: tsallis_divergence(p, q, alpha=2)
Out[5]: 0.25

API

alpha_divergence(dist1, dist2, alpha=1.0, rvs=None)[source]

The alpha divergence of dist1 and dist2, as used in Information Geometry. Note there is more than one inequivalent definition of “alpha divergence” in the literature, this one comes from http://en.wikipedia.org/wiki/Information_geometry .

Parameters:

  • dist1 (Distribution) – The first distribution in the alpha divergence.

  • dist2 (Distribution) – The second distribution in the alpha divergence.

  • alpha (float, 1) – The divergence is a one parameter family in alpha.

  • rvs (list, None) – The indexes of the random variable used to calculate the alpha divergence between. If None, then the alpha divergence is calculated over all random variables.

Returns:

dkl – The alpha divergence between dist1 and dist2.

Return type:

float

Raises:

ditException – Raised if either dist1 or dist2 doesn’t have rvs or, if rvs is None, if dist2 has an outcome length different than dist1.

hellinger_divergence(dist1, dist2, alpha=1.0, rvs=None)[source]

The Hellinger divergence of dist1 and dist2.

Parameters:

  • dist1 (Distribution) – The first distribution in the Hellinger divergence.

  • dist2 (Distribution) – The second distribution in the Hellinger divergence.

  • alpha (float, 1) – The divergence is a one parameter family in alpha.

  • rvs (list, None) – The indexes of the random variable used to calculate the Hellinger divergence between. If None, then the Hellinger divergence is calculated over all random variables.

Returns:

dkl – The Hellinger divergence between dist1 and dist2.

Return type:

float

Raises:

ditException – Raised if either dist1 or dist2 doesn’t have rvs or, if rvs is None, if dist2 has an outcome length different than dist1.

renyi_divergence(dist1, dist2, alpha=1, rvs=None)[source]

The Renyi divergence of dist1 and dist2.

Parameters:

  • dist1 (Distribution) – The first distribution in the Renyi divergence.

  • dist2 (Distribution) – The second distribution in the Renyi divergence.

  • alpha (float, 1) – The divergence is a one parameter family in alpha.

  • rvs (list, None) – The indexes of the random variable used to calculate the Renyi divergence between. If None, then the Renyi divergence is calculated over all random variables.

Returns:

dkl – The Renyi divergence between dist1 and dist2.

Return type:

float

Raises:

ditException – Raised if either dist1 or dist2 doesn’t have rvs or, if rvs is None, if dist2 has an outcome length different than dist1.

tsallis_divergence(dist1, dist2, alpha=1.0, rvs=None)[source]

The Tsallis divergence of dist1 and dist2.

Parameters:

  • dist1 (Distribution) – The first distribution in the Tsallis divergence.

  • dist2 (Distribution) – The second distribution in the Tsallis divergence.

  • alpha (float, 1) – The divergence is a one parameter family in alpha.

  • rvs (list, None) – The indexes of the random variable used to calculate the Tsallis divergence between. If None, then the Tsallis divergence is calculated over all random variables.

Returns:

dkl – The Tsallis divergence between dist1 and dist2.

Return type:

float

Raises:

ditException – Raised if either dist1 or dist2 doesn’t have rvs or, if rvs is None, if dist2 has an outcome length different than dist1.

f_divergence(dist1, dist2, f, rvs=None)[source]

The Csiszar f-divergence of dist1 and dist2. Note that it is typically more accurate to use a specialized divergence function when available due to roundoff errors and small probability effects.

Parameters:

  • dist1 (Distribution) – The first distribution in the f-divergence.

  • dist2 (Distribution) – The second distribution in the f-divergence.

  • f (function) – The auxillary function defining the f-divergence

  • rvs (list, None) – The indexes of the random variable used to calculate the f-divergence between. If None, then the f-divergence is calculated over all random variables.

Returns:

dkl – The f-divergence between dist1 and dist2.

Return type:

float

Raises:

ditException – Raised if either dist1 or dist2 doesn’t have rvs or, if rvs is None, if dist2 has an outcome length different than dist1.