Skip to content

Drift Calculation

cumulative_based

ks_drift(reference_sample, test_sample, filename)

Perform Kolmogorov-Smirnov (KS) drift detection between a reference sample and a test sample.

This function applies KS tests across the dimensions of the input data. It compares each dimension of the test sample against the corresponding dimension of the reference sample and summarizes the resulting statistics and p-values. Optionally, the results are saved to a JSON file.

Parameters:

Name Type Description Default
reference_sample ndarray

Two-dimensional reference sample, where each column corresponds to one dimension.

 required
test_sample ndarray

Two-dimensional test sample, where each column corresponds to one dimension.

 required
filename str

Path to the JSON file where drift results will be saved. If an empty string is provided, no file is created.

 required

Returns:

Type Description
dict

Dictionary containing the aggregated KS drift results.

  • "magnitude": median KS statistic across dimensions.
  • "p_value": median p-value across dimensions.
  • "p_value_median": median p-value across dimensions.
  • "p_value_mean": mean p-value across dimensions.
  • "p_value_min": minimum p-value across dimensions.
  • "p_value_max": maximum p-value across dimensions.

Raises:

Type Description
ValueError

If reference_sample and test_sample have mismatched dimensions.
IOError

If the results cannot be written to filename.
Notes

The KS test is a non-parametric test that compares the empirical distribution functions of two samples.

This implementation assumes that both input samples are two-dimensional arrays with the same number of columns.

density_based

js_drift(reference_sample, test_sample, filename)

Compute Jensen-Shannon (JS) divergence between two distributions.

This function estimates distributional drift between a reference sample and a test sample using Jensen-Shannon divergence. Results can optionally be exported to a JSON file.

Parameters:

Name Type Description Default
reference_sample ndarray

Reference sample distribution.

 required
test_sample ndarray

Test sample distribution.

 required
filename str

Output path used to save the drift results. If empty, results are not exported.

 required

Returns:

Type Description
dict

Dictionary containing the drift results.

  • "magnitude": Jensen-Shannon divergence magnitude.
Notes

Jensen-Shannon divergence is a symmetric measure of the difference between two probability distributions.

Input samples are converted to float64 NumPy arrays before computation.

If filename is provided, the results are exported in JSON format.

kl_drift(reference_sample, test_sample, filename, eps=1e-12)

Compute Kullback-Leibler (KL) divergence between two distributions.

This function estimates distributional drift between a reference sample and a test sample using Kullback-Leibler divergence. Input distributions are normalized and smoothed to avoid numerical instability. Results can optionally be exported to a JSON file.

Parameters:

Name Type Description Default
reference_sample ndarray

Reference probability distribution.

 required
test_sample ndarray

Test probability distribution.

 required
filename str

Output path used to save the drift results. If empty, results are not exported.

 required
eps float

Small smoothing constant used to avoid division by zero and logarithms of zero. Defaults to 1e-12.

 1e-12

Returns:

Type Description
dict

Dictionary containing the drift results.

  • "magnitude": Kullback-Leibler divergence magnitude.

Raises:

Type Description
ValueError

If the input distributions do not have the same shape.
Notes

Kullback-Leibler divergence is computed as:

.. math::

\sum P(x) \log \frac{P(x)}{Q(x)}

Input distributions are normalized to sum to 1 before computation.

Smoothing is applied using eps to improve numerical stability.

log_likelihood_drift(reference_sample, test_sample, filename, K=1000, n_jobs=10, alpha=1e-12)

Perform log-likelihood drift detection between a reference sample and a test sample.

This function estimates distributional drift using a log-likelihood approach and evaluates statistical significance through permutation testing. Results can optionally be exported to a JSON file.

Parameters:

Name Type Description Default
reference_sample ndarray

Reference sample distribution.

 required
test_sample ndarray

Test sample distribution.

 required
filename str

Output path used to save the drift results. If empty, results are not exported.

 required
K int

Number of permutations used for significance estimation. Defaults to 1000.

 1000
n_jobs int

Number of parallel jobs used during permutation testing. Defaults to 10.

 10
alpha float

Smoothing parameter used in the log-likelihood computation. Defaults to 1e-12.

 1e-12

Returns:

Type Description
dict

Dictionary containing the drift results.

  • "magnitude": log-likelihood drift magnitude.
  • "p_value": permutation-test p-value.
Notes

Permutation testing is used to estimate whether the observed drift magnitude significantly differs from the null distribution generated by random resampling.

vector_based

mmd_drift(reference_sample, test_sample, filename, K=100, n_jobs=10)

Perform Maximum Mean Discrepancy (MMD) drift detection between a reference sample and a test sample.

This function estimates distributional drift using MMD with an RBF kernel and evaluates statistical significance through permutation testing. Results can be saved to a JSON file and intermediate results can be used to resume interrupted computations.

Parameters:

Name Type Description Default
reference_sample ndarray

Reference sample data.

 required
test_sample ndarray

Test sample data to compare against the reference sample.

 required
filename str

Output path used to save the drift results. If empty, results are not exported.

 required
K int

Number of permutations used for significance estimation. Defaults to 100.

 100
n_jobs int

Number of parallel jobs used during permutation testing. Defaults to 10.

 10

Returns:

Type Description
dict

Dictionary containing the drift results.

  • "magnitude": MMD drift magnitude.
  • "p_value": permutation-test p-value.

Raises:

Type Description
RuntimeError

If some permutations are not completed.
Notes

MMD is computed using an RBF kernel. Permutation testing is used to estimate whether the observed drift magnitude significantly differs from the null distribution generated by random resampling.

cos_drift(reference_sample, test_sample, filename, K=100, n_jobs=10)

Detect drift between a reference sample and a test sample using cosine distance.

This function estimates semantic drift from the cosine distance between reference and test samples and evaluates statistical significance through permutation testing. Results can be exported to a JSON file.

Parameters:

Name Type Description Default
reference_sample ndarray

Reference sample data.

 required
test_sample ndarray

Test sample data to compare against the reference sample.

 required
filename str

Output path used to save the drift results. If empty, results are not exported.

 required
K int

Number of permutations used for significance estimation. Defaults to 100.

 100
n_jobs int

Number of parallel jobs used during permutation testing. Defaults to 10.

 10

Returns:

Type Description
dict

Dictionary containing the drift results.

  • "magnitude": cosine drift magnitude.
  • "p_value": permutation-test p-value.

Raises:

Type Description
ValueError

If the backup file exists but the number of completed permutations does not match K.
RuntimeError

If some permutations are not completed.
Notes

Permutation testing is used to estimate whether the observed cosine drift magnitude significantly differs from the null distribution generated by random resampling.