Constructors and examples

Besides the Distribution constructors on Construction, dit ships helpers for building joints and a catalog of named examples.

dit.distconst

These are imported at the package root:

In [1]: import dit

In [2]: d = dit.uniform_distribution(2, 2)

In [3]: print(d)
Class:    Distribution
Alphabet: ((0, 1), (0, 1))
Base:     linear

x        p(X0,X1)
(0, 0)   0.25
(0, 1)   0.25
(1, 0)   0.25
(1, 1)   0.25

In [4]: xor = dit.insert_rvf(d, lambda x: (int(x[0]) ^ int(x[1]),))

In [5]: print(xor)
Class:    Distribution
Alphabet: ((0, 1), (0, 1), (0, 1))
Base:     linear

x           p(X0,X1,X2)
(0, 0, 0)   0.25
(0, 1, 1)   0.25
(1, 0, 1)   0.25
(1, 1, 0)   0.25

Kirkwood and Lancaster approximations

Both approximations truncate an interaction expansion of \(p\) at subsets of at most \(k\) variables, and share the coefficients

\[c_S = (-1)^{k-|S|} \binom{n-|S|-1}{k-|S|}, \qquad |S| \leq k.\]

The Kirkwood approximation [KYKG07, Wat60] combines the marginals geometrically, and the Lancaster approximation [Lan69] arithmetically:

\[\tilde{p}_k(x) = \prod_{|S| \leq k} p(x_S)^{c_S}, \qquad p_k(x) = \sum_{|S| \leq k} c_S \, p(x_S) \prod_{i \notin S} p(x_i).\]

The default order is \(k = n-1\). For three variables these are \(p(x, y) p(x, z) p(y, z) / \left(p(x) p(y) p(z)\right)\) and \(p(x, y) p(z) + p(x, z) p(y) + p(y, z) p(x) - 2 p(x) p(y) p(z)\). Both return dense, unvalidated distributions.

The Kirkwood approximation is nonnegative but not normalized. Its total mass is the normalizer of the Kirkwood and Ouroboros Mutual Informations. For the giant bit, every outcome off the diagonal has a vanishing pairwise marginal:

In [6]: from dit.example_dists import giant_bit

In [7]: dit.kirkwood_approximation(giant_bit(3, 2)).pmf
Out[7]: array([1., 0., 0., 0., 0., 0., 0., 1.])

The order-\(k\) Lancaster approximation keeps the Lancaster interactions of at most \(k\) variables in the expansion \(p = \prod_i p(x_i) \sum_T \delta_T(x_T)\). At the default order it is \(p - \Delta_L p\), where \(\Delta_L p = \prod_i \left(p^*_{X_i} - p_{X_i}\right)\) is the Lancaster interaction measure [SGB13]. It sums to one and matches every marginal of \(p\) on at most \(k\) variables, but it may be negative:

In [8]: dit.lancaster_approximation(dit.Distribution(['000', '010', '100', '111'], [1/4] * 4)).pmf
Out[8]: 
array([ 0.3125, -0.0625,  0.1875,  0.0625,  0.1875,  0.0625,  0.0625,
        0.1875])

For \(n \leq 3\) the default-order Lancaster approximation reproduces \(p\) whenever \(p\) factors nontrivially, though not only then. For \(n > 3\) it can miss factorizations such as \((X_0, X_1) \perp (X_2, X_3)\), which Streitberg’s partition-lattice correction repairs [Str90]. No canonical literature source is known for Lancaster orders below \(n-1\).

dit.example_dists

Parametric and hand-built examples:

  • Circuits: Xor(), And(), Or(), Rdn(), Unq(), and related PID illustrations

  • Counts: uniform(), bernoulli(), binomial(), multinomial()

  • Structured joints: giant_bit(), n_mod_m(), dyadic(), triadic(), pr_box()

Empirical constructors under dit.example_dists.empirical (titanic(), penguins(), blood_types(), congress(), student(), car(), bach(), corelli()) fetch their source data at call time and return the estimated joint. The music examples additionally require dit[music].

API

uniform(outcomes, base=None)[source]

Produces a uniform distribution over outcomes.

Parameters:

  • outcomes (iterable) – The set of outcomes with which to construct the distribution.

  • base (float, 'linear', 'e') – The desired base for the distribution probabilities.

Returns:

d – A uniform distribution over outcomes.

Return type:

Distribution

Raises:

  • ditException – Raised if the elements of outcomes do not all have the same length.

  • TypeError – Raised if outcomes is not iterable.

uniform_distribution(outcome_length, alphabet_size, base=None)[source]

Returns a uniform distribution.

Parameters:

  • outcome_length (int) – The length of the outcomes.

  • alphabet_size (int, list of lists) – The alphabets used to construct the outcomes of the distribution. If an integer, then the alphabet for each random variable will be the same, consisting of integers from 0 to k-1 where k is the alphabet size. If a list, then the elements are used as the alphabet for each random variable. If the list has a single element, then it will be used as the alphabet for each random variable.

  • base (float, 'linear', 'e') – The desired base for the distribution probabilities.

Returns:

d – A uniform distribution.

Return type:

Distribution.

Examples

Each random variable has the same standardized alphabet: [0,1] >>> d = dit.uniform_distribution(2, 2)

Each random variable has its own alphabet. >>> d = dit.uniform_distribution(2, [[0,1],[1,2]])

Both random variables have [‘H’,’T’] as an alphabet. >>> d = dit.uniform_distribution(2, [[‘H’,’T’]])

mixture_distribution(dists, weights, merge=False)[source]

Create a mixture distribution: \(\sum p_i d_i\)

Parameters:

  • dists ([Distribution]) – List of distributions to mix. Each distribution is assumed to have the same base and sample space.

  • weights ([float]) – List of weights to use while mixing dists. The weights are assumed to be probability represented in the base of the distributions.

  • merge (bool) – If True then distributions will be mixed even if they do not share the same sample space. The idea is that each of the input distributions is reinterpreted on a common, merged sample space. If False, then an exception will be raised if incompatible distributions are mixed.

Returns:

mix – The mixture distribution.

Return type:

Distribution

Raises:

  • DitException – Raised if there dists and weights have unequal lengths.

  • InvalidNormalization – Raised if the weights do not sum to unity.

  • InvalidProbability – Raised if the weights are not valid probabilities.

  • IncompatibleOutcome – Raised if the sample spaces for each distribution are not compatible.

product_distribution(dist, rvs=None, base=None)[source]

Returns a new distribution which is the product of marginals.

Parameters:

  • dist (distribution) – The original distribution.

  • rvs (sequence) – A sequence whose elements are also sequences. Each inner sequence defines the marginal distribution used to create the new distribution.

  • base (float, 'linear', 'e') – The desired base for the distribution probabilities.

Returns:

d – The product distribution.

Return type:

Distribution

kirkwood_approximation(dist, rvs=None, order=None)[source]

Construct the unnormalized generalized Kirkwood superposition approximation of a distribution.

For variables \(X_0, \ldots, X_{n-1}\) the order-\(k\) approximation truncates the Möbius (interaction) expansion of \(\log p\) at subsets of size at most \(k\) [KYKG07, Wat60]:

\[\tilde{p}_k(x) = \prod_{1 \leq |S| \leq k} p(x_S)^{c_S}, \qquad c_S = (-1)^{k-|S|} \binom{n-|S|-1}{k-|S|}.\]

The default order \(k = n-1\) is the standard Kirkwood approximation, e.g. \(p(x, y) p(x, z) p(y, z) / \left(p(x) p(y) p(z)\right)\) for three variables. Order 1 is the product of marginals and order 0 is constant one. The approximation is not normalized: with \(Z\) its total mass, kirkwood_mutual_information() is \(D_{KL}(p \| \tilde{p}_k / Z)\). Wherever a marginal used by the product vanishes, the approximation is zero. See lancaster_approximation() for the additive analogue.

Parameters:

  • dist (Distribution) – The distribution to approximate.

  • rvs (list, None) – A list of lists. Each inner list specifies the indexes of the random variables forming one variable of the approximation. If None, then each random variable is its own variable.

  • order (int, None) – The largest marginal size k used by the approximation. Must satisfy 0 <= k <= n - 1. If None, defaults to n - 1.

Returns:

approx – The dense, unnormalized approximation over the outcomes of dist coalesced to rvs.

Return type:

Distribution

Raises:

ditException – Raised if order is out of range.

Examples

>>> d = dit.example_dists.giant_bit(3, 2)
>>> dit.kirkwood_approximation(d).pmf
array([1., 0., 0., 0., 0., 0., 0., 1.])
lancaster_approximation(dist, rvs=None, order=None)[source]

Construct the order-k Lancaster approximation of a distribution, the additive analogue of the Kirkwood approximation.

Lancaster’s expansion of \(X_0, \ldots, X_{n-1}\) writes the joint distribution relative to the product of its marginals [Lan69]:

\[p(x) = \prod_i p(x_i) \sum_{T} \delta_T(x_T), \qquad \delta_T(x_T) = \sum_{S \subseteq T} (-1)^{|T|-|S|} \frac{p(x_S)}{\prod_{i \in S} p(x_i)},\]

where \(\delta_\emptyset = 1\) and \(\delta_{\{i\}} = 0\). \(\prod_i p(x_i) \, \delta_{\{0, \ldots, n-1\}}\) is the Lancaster interaction measure \(\Delta_L p = \prod_i (p^*_{X_i} - p_{X_i})\) [SGB13]. Truncating the expansion at \(|T| \leq k\) gives

\[p_k(x) = \sum_{|S| \leq k} c_S \, p(x_S) \prod_{i \notin S} p(x_i), \qquad c_S = (-1)^{k-|S|} \binom{n-|S|-1}{k-|S|},\]

with the same coefficients as kirkwood_approximation(), but combined arithmetically rather than geometrically. The default order \(k = n-1\) is \(p - \Delta_L p\), e.g. for three variables

\[p_2(x, y, z) = p(x, y) p(z) + p(x, z) p(y) + p(y, z) p(x) - 2 p(x) p(y) p(z).\]

Orders 0 and 1 are both the product of marginals. Every order sums to one, and its marginals of size at most k agree with those of \(p\), but it may be negative. For \(n \leq 3\), the default order reproduces \(p\) whenever \(p\) factors nontrivially, although the converse fails. For \(n > 3\) even that implication fails, which motivates Streitberg’s correction [Str90]. No canonical literature source is known for orders below \(n-1\).

Parameters:

  • dist (Distribution) – The distribution to approximate.

  • rvs (list, None) – A list of lists. Each inner list specifies the indexes of the random variables forming one variable of the approximation. If None, then each random variable is its own variable.

  • order (int, None) – The largest interaction size k kept by the approximation. Must satisfy 0 <= k <= n - 1. If None, defaults to n - 1.

Returns:

approx – The dense, possibly signed, approximation over the outcomes of dist coalesced to rvs.

Return type:

Distribution

Raises:

ditException – Raised if order is out of range.

Examples

>>> d = dit.Distribution(['000', '010', '100', '111'], [1/4] * 4)
>>> dit.lancaster_approximation(d).pmf
array([ 0.3125, -0.0625,  0.1875,  0.0625,  0.1875,  0.0625,  0.0625,  0.1875])
insert_rvf(d, func, index=-1)[source]

Returns a new distribution with an added random variable at index index.

The new random variable must be a function of the other random variables. By this, we mean that the entropy of the new random variable conditioned on the original random variables should be zero.

Parameters:

  • dist (Distribution) – The distribution used to construct the new distribution.

  • func (callable | list of callable) – A function which takes a single argument—the value of the previous random variables—and returns a new random variable. Note, the return value will be added to the outcome using __add__, and so it should be a hashable, orderable sequence (as every outcome must be). If a list of callables is provided, then multiple random variables are added simultaneously and will appear in the same order as the list.

  • index (int) – The index at which to insert the random variable. A value of -1 is will append the random variable to the end.

Returns:

d – The new distribution.

Return type:

Distribution

Examples

>>> d = dit.Distribution(['00', '01', '10', '11'], [1/4]*4)
>>> def xor(outcome):
...    return str(int(outcome[0] != outcome[1]))
...
>>> d2 = dit.insert_rvf(d, xor)
>>> d.outcomes
('000', '011', '101', '110')
distribution_from_bayesnet(digraph, nodes=None, sample_space=None, attr='dist')[source]

Returns a distribution built from a Bayesian network.

Each node represents a random variable X_i. Every node must store its conditional probability distribution P(X_i | Y_i) where Y_i represents the parents of X_i. If a node has no in-degree, then it must store the probability distribution P(X_i).

Parameters:

  • digraph (NetworkX digraph) – A directed graph, representing the Bayesian graphical model.

  • nodes (list, None) – The order of the nodes that will determine the random variable order. If None, then we use sorted(digraph.nodes()), which assumes the nodes are sortable. The reason we assume they are sortable is because the parent values must correspond to the node order and thus, we need an unambiguous ordering that the user could have known ahead of time.

  • sample_space (SampleSpace) – If provided, this specifies the outcomes of the distribution to be constructed. The distributions stored on the nodes are assumed to be compatible with this space. If functions are stored on the nodes, then this parameter must be provided.

  • attr (str) –

    The attribute for each node that holds the conditional distributions. The attribute value can take a variety of forms.

    It can be a function. The function must take two arguments. The first is the value of random variable for the current node. The second is a dictionary of keyed by parents whose values are the values of the random variables corresponding to the parents. The function should return the probability P(node_val|parent_vals).

    It can be a list, such as [parents, dists], that holds the parents and the conditional distributions: dists[i] = P(X | Y_i = parents[i]). It can also be a dict-like structure so that dists[y] is a distribution representing P(X | Y = y)`. If the node has no in-degree, then the attribute value should store the distribution only. When using distributions, each should have random variable names assigned that match the nodes in the graph, or alternatively, all nodes in the graph should be integers and then random variable names are not necessary. The order of elements within these lists (or the dict) does not matter, but for nodes that have parents, the order of the random variables that specify the parents, must match the order of nodes. So for example, if the node order is [2, 1, 0] and node 1 has parents 0 and 2. Then the parents for node 1 will be such that the first element corresponds to node 2 and the second to node 0, since 2 precedes 0 in the node order. All distributions will be assumed to have the same base, and this base will determine the base of the constructed distribution.

Returns:

dist – The joint distribution.

Return type:

Distribution

Examples

The Xor logic gate has the following structure:

>>> g = nx.DiGraph()
>>> g.add_edge(0, 2)
>>> g.add_edge(1, 2)

Let’s add distributions to it using functions.

>>> uniform = lambda node_val, parents: 0.5
>>> def xor(node_val, parents):
...     if '1' == parents[0] == parents[1]:
...         desired_output = '1'
...     else:
...         desired_output = '0'
...     return int(node_val == desired_output)
...
>>> g.nodes[0]['dist'] = uniform
>>> g.nodes[1]['dist'] = uniform
>>> g.nodes[2]['dist'] = xor
>>> ss = ['000', '001', '010', '011', '100', '101', '110', '111']
>>> d = dit.distribution_from_bayesnet(g, sample_space=ss)

Alternatively, we could add distributions using Distribution objects.

>>> uniform = dit.uniform_distribution(1, 2)
>>> sample_space1 = [(0,), (1,)]
>>> one = dit.Distribution(sample_space1, [0, 1])
>>> zero = dit.Distribution(sample_space1, [1, 0])
>>> sample_space2 = [(0, 0), (0, 1), (1, 0), (1, 1)]
>>> xor = [ sample_space2, [zero, one, one, zero]]
>>> g.nodes[0]['dist'] = uniform
>>> g.nodes[1]['dist'] = uniform
>>> g.nodes[2]['dist'] = xor
>>> d = dit.distribution_from_bayesnet(g)

We can add noise whenever the output would normally be 1.

>>> noisy = dit.Distribution(sample_space, [.1, .9])
>>> dists = [zero, noisy, noisy, zero]
>>> g.nodes[2]['dist'][1] = dists
>>> d = dit.distribution_from_bayesnet(g)