What is a split-conformal prediction interval?

The procedure

Split the labelled data into a training part and a disjoint calibration part. Fit the estimator on the training part only. Compute absolute residuals on the calibration part. Take the finite-sample corrected empirical quantile of those residuals, and report the predicted value ± q for each test point. In SKSurrogate this is split_conformal_regression_interval(regressor, X_train, y_train, X_calibration, y_calibration, X_test, alpha=0.1). The estimator is cloned and fitted only on the training data, and the function does not accept test labels — by construction, the interval cannot peek at the answer it is meant to bracket.

What the guarantee covers

The marginal coverage statement assumes that calibration examples and future examples are exchangeable — drawn from the same distribution, in any order — and that the fitted model was chosen independently of the calibration set. If a hyperparameter search used the calibration rows, the correction is computed on data the model has already seen and the guarantee no longer applies. Coverage is marginal: it holds on average over draws, not per subgroup, not per region of feature space, and not under distribution shift.

Checking it in practice

A guarantee you never test is a claim, not a measurement. Given held-out labels, prediction_interval_diagnostics(y_holdout, lower, upper, alpha=0.1) reports nominal and empirical coverage, a Wilson score interval for the empirical coverage (which describes finite-sample uncertainty and is not the conformal guarantee), mean and median width, and the mean Winkler score for a central two-sided interval. An optional groups argument adds bounded per-group coverage counts and Wilson intervals; small groups are highly uncertain, and marginal coverage does not imply subgroup coverage. Unbounded intervals stay in the coverage count, while width and interval score are None if any interval is unbounded — a detail that matters when a model is badly miscalibrated.

Conformalized quantile regression

When two conditional-quantile estimators are available — typically at α/2 and 1−α/2 — the conformal correction can widen or narrow the base interval according to the calibration residuals (split_conformal_quantile_interval). Predicted bounds are ordered before calibration, and the result includes an empty_interval_mask: a negative correction can produce an empty set when the corrected lower bound exceeds the upper one. The correction is infinite when the requested rank exceeds the calibration size, which is the honest answer to "you asked for 99.9 % coverage from 40 calibration points".

Limitations to state when you report one

• Marginal, not conditional: no per-subgroup coverage claim follows.
• Exchangeability, not i.i.d. forever: drift voids the guarantee.
• The model must not be selected on the calibration split.
• Interval width is a property of the data and model, not a quality score; a narrow interval on a bad model is still narrow.

The API reference and the full caveats are in the tracking and evaluation guide; the deployed model bundle and its interval diagnostics are described in the control-plane guide. The measurement half of the workflow — what a single split can and cannot tell you — is on nested cross-validation.