What is surrogate-based optimization?

The mechanism

For a function f that can only be evaluated a limited number of times, one iteration is: sample a point in the feasible set; evaluate f there; add the pair to the evaluation set E; fit a surrogate to E; minimise the surrogate to propose the next point; evaluate f at that proposal. Repeat until the budget is exhausted, then return the best evaluated pair. The package's own description of this loop, including the MinEvals random-evaluation phase before the surrogate engages, is in the surrogate search guide.

Three design choices carry most of the behaviour: how the initial and subsequent points are sampled, which model is fitted, and how the model is minimised. SKSurrogate ships three samplers — a random point in the feasible set, a cube around the last point (with init_radius 2.0 and a contraction of 0.9 by default), and a sphere around the last point — and accepts user-defined sampling classes with the same signature. The surrogate is any scikit-learn regressor; the default is an RBF Gaussian process.

When it pays, and when it does not

The case for a surrogate is economic, not algorithmic: it pays when one evaluation costs minutes and a model fit costs milliseconds. The package publishes what happens when that condition is not met. On six public datasets at a budget of 30 evaluations, the surrogate-guided search is at parity with Optuna-TPE, which is top on five of the six; the whole table spans about half a point of accuracy. At 60 evaluations the surrogate arm leads all six datasets. The honest summary is that on a flat, three-parameter landscape your choice among these tools will not move your accuracy much — the benchmark page has the tables.

What the surrogate buys instead is evaluation efficiency. On the 1,220-pipeline space the search reached within 1 % of its own final best in a median of 6 evaluations, against 11–17 for the other arms at identical final accuracy. That advantage does not transfer to wall-clock time: the Gaussian-process fit grows as O(n³), so per-fit cost rose from 3 ms to 174 ms across a run and 49 % of a 120 s budget went to search overhead (22 % fitting, 27 % acquisition).

The response surface matters more than the search

Inside the same search loop, changing only the fitted surface changes the result. An un-tuned RBF Gaussian process beat the legacy cubic Hilbert polynomial on all six datasets at both budgets, with seed-level wins of 20W/17T/5L at B=30 and 25W/13T/4L at B=60 (exact binomial p ≈ 4×10⁻³ and p ≈ 10⁻⁴). That is why v0.2 made the RBF Gaussian process the default and kept HilbertRegressor available. A controlled synthetic study explains the mechanism: the polynomial regressor is exact on in-span objectives but truncates elsewhere, the stock Gaussian process has no noise term and interpolates noise, and a tree ensemble is the robust generalist.

How SKSurrogate implements it

SurrogateRandomCV is a BaseSearchCV subclass, so it has the call shape of RandomizedSearchCV: pass an estimator, a param_distributions mapping of tuples (numeric ranges) and lists (categorical values), and n_iter. For pipeline design, AML adds an evolutionary search over transformers and estimators (fit(X, y, strategy="eoa")) before the local surrogate refinement. Every evaluation is recorded, which is what makes the result an artifact rather than a single returned winner.

What it does not do

The surrogate is an approximation of the observed surface, not a certificate: it can be confidently wrong between samples, and no coverage guarantee is claimed for the search itself. It does not remove the need for a held-out evaluation, and it does not make a flat objective steep. Compare with a single-split score using nested cross-validation, and quantify deployed uncertainty with conformal intervals.