Inference

dit.inference estimates distributions and information quantities from samples. Plug-in counts live alongside Miller–Madow-style and NSB-style entropy estimators, plus k-nearest-neighbor / Kraskov–Stögbauer–Grassberger estimators for differential entropy and total correlation.

The kNN / KSG implementations optionally use scikit-learn (installed with the dit[optional] extra) when it is available.

From samples

distribution_from_data() builds a joint over words of length L from a sequence of symbols. dist_from_timeseries() treats each column of a multivariate series as a variable and appends a history_length past together with the present.

In [1]: from dit.inference import distribution_from_data, entropy_0, entropy_1

In [2]: data = [0, 0, 0, 1, 1, 1]

In [3]: d = distribution_from_data(data, L=1, base='linear')

In [4]: d.outcomes
Out[4]: (0, 1)

In [5]: entropy_0(data)
Out[5]: 1.0

Estimators

API

distribution_from_data(d, L, trim=True, base=None)[source]

Returns a distribution over words of length L from d.

The returned distribution is the naive estimate of the distribution, which assigns probabilities equal to the number of times a particular word appeared in the data divided by the total number of times a word could have appeared in the data.

Roughly, it corresponds to the stationary distribution of a maximum likelihood estimate of the transition matrix of an (L-1)th order Markov chain.

Parameters:

  • d (list) – A list of symbols to be converted into a distribution.

  • L (integer) – The length of the words for the distribution.

  • trim (bool) – If true, then words with zero probability are trimmed from the distribution.

  • base (int or string) – The desired base of the returned distribution. If None, then the value of dit.ditParams[‘base’] is used.

dist_from_timeseries(observations, history_length=1, base='linear')[source]

Infer a distribution from time series observations. For each variable, infer a history_length past and a single observation present.

Parameters:

  • observations (list of tuples, ndarray) – A sequence of observations in time order.

  • history_length (int) – The history length to utilize.

  • base (float, str) – The base to use for the distribution. Defaults to ‘linear’.

Returns:

ts – A distribution with the first half of the indices as the pasts of the various time series, and the second half their present values.

Return type:

Distribution

entropy_0(data, length=1)[source]

Estimate the entropy of length length subsequences in data.

Parameters:

  • data (iterable) – An iterable of samples.

  • length (int) – The length to group samples into.

Returns:

h0 – An estimate of the entropy.

Return type:

float

Notes

This returns the naive estimate of the entropy.

entropy_1(data, length=1)[source]

Estimate the entropy of length length subsequences in data.

Parameters:

  • data (iterable) – An iterable of samples.

  • length (int) – The length to group samples into.

Returns:

h1 – An estimate of the entropy.

Return type:

float

Notes

If M is the alphabet size and N is the number of samples, then the bias of this estimator is:

B ~ M/N

entropy_2(data, length=1)[source]

Estimate the entropy of length length subsequences in data.

Parameters:

  • data (iterable) – An iterable of samples.

  • length (int) – The length to group samples into.

Returns:

h2 – An estimate of the entropy.

Return type:

float

Notes

If M is the alphabet size and N is the number of samples, then the bias of this estimator is:

B ~ (M+1)/(2N)

differential_entropy_knn(data, rvs=None, k=4, noise=1e-10)[source]

Compute the differential entropy of data using a k-nearest neighbors density estimator.

Parameters:

  • data (np.ndarray) – The data.

  • rvs (list) – The columns of data to use as the random variable. If None, use all.

  • k (int) – The number of nearest neighbors to use.

Returns:

h – The estimated entropy.

Return type:

float

Notes

The entropy is returned in units of bits.

total_correlation_ksg(data, rvs, crvs=None, k=4, noise=1e-10)

Compute the total correlation from observations. The total correlation is computed between the columns specified in rvs, given the columns specified in crvs. This utilizes the KSG kNN density estimator, and works on discrete, continuous, and mixed data.

Parameters:

  • data (np.array) – Real valued time series data.

  • rvs (iterable of iterables) – The columns for which the total correlation is to be computed.

  • crvs (iterable) – The columns upon which the total correlation should be conditioned.

  • k (int) – The number of nearest neighbors to use in estimating the local kernel density.

  • noise (float) – The standard deviation of the normally-distributed noise to add to the data.

Returns:

tc – The total correlation of rvs given crvs.

Return type:

float

Notes

The total correlation is computed in bits, not nats as most KSG estimators do.

This implementation uses scikit-learn.