What is nested cross-validation?

Nested cross-validation keeps the step that chooses a model separate from the step that estimates its generalisation error. The inner loop selects; the outer loop scores the selection procedure on data the inner loop never saw. Without that separation, the reported score is the score you optimised, and it is optimistic by construction.

Why a single split misleads

When the same folds are used to tune and to score, every configuration is evaluated on the same data and the winner's score inherits the selection. The bias grows with the number of candidates and shrinks with the size of the data — which is exactly the regime where expensive problems live. Nested CV answers a different question: not "how well does the chosen configuration score on these folds" but "how well does this selection procedure score on unseen folds".

What SKSurrogate's `nested_cv_evaluation` does

nested_cv_evaluation computes inner-CV scores and outer-fold estimates for the estimator you supply, then evaluates it on the untouched validation partition. It stores fold-level predictions and metrics, mean outer scores, repeated-CV confidence intervals and the final validation score in task metadata. Mixed-type DataFrame features are preprocessed inside each training fold, and unseen categories are ignored at scoring time rather than being folded into the vocabulary — the leakage that quietly inflates a score. Group-aware splitters preserve group labels through the evaluation path.

Two limits are stated plainly in the documentation and should be repeated here: the function does not search among estimators or tune hyperparameters, and it does not replace the search layer. It is the evaluation half — the part that scores what the search produced without re-using the search's own optimised number. Tuning happens in SurrogateRandomCV / AML; the evaluation asks how that result behaves on partitions it never touched.

Reading the result honestly

Results carry an evaluation protocol: registered partition fingerprints, the CV splitter specification, the repeat seed policy, the preprocessing path, the scoring identity and whether a validation set was used. When two results are compared with compare_repeated_cv_results, known mismatches are rejected and the output reports whether alignment could be verified — opaque custom scorers, missing fingerprints and older result dictionaries remain explicitly unverified, which is a signal, not a gap to paper over.

The paired comparison reports the candidate-minus-reference mean and median repeat-score difference, a percentile bootstrap interval and paired Cohen's d_z. It is deliberately descriptive: the resampling unit is a complete repeat-level score difference, repeats reuse observations, and the interval summarizes variation across repeats. It is not an independent-sample significance test, and excluding zero is not a significance claim. BCa intervals do not resolve dependence between repeats either; for a confirmatory comparison, define the independent unit and the comparison family before looking at the data.

Time series

For ordered observations, pass TimeSeriesSplit for both the inner and outer loops and choose a gap that excludes observations between the training and test windows. Rows must already be in chronological order: the package does not infer timestamps or sort them, and it does not construct lag, rolling or horizon features — doing that without future leakage is the caller's responsibility.

What it does not settle

Nested CV estimates the procedure's error on the distribution your folds were drawn from. It says nothing about a shifted deployment population, and it does not produce prediction intervals — that is split conformal. Reporting one number from one split is still the most common way to overstate a result; the API reference is in the tracking and evaluation guide.