Honest benchmark: surrogate search vs TPE at small budgets

Equal evaluation budgets, six public datasets, seven seeds per cell. TPE is top on five of six; the interesting result is which response surface you fit.

All methods optimize the same estimator (StandardScaler → RandomForestClassifier(50)) over the same shared discrete grid (224 configurations across 3 parameters), with stratified 5-fold CV inside every search and one shared held-out 80/20 outer split per dataset-seed. Budgets are measured in candidate-model evaluations — a hardware-independent currency. Seven seeds per cell; 6 datasets from OpenML (pinned by did) plus one bundled synthetic set, B = 30 and B = 60 evaluations.

Arm	What it is
SRS + RBF-GP	The package's search with its v0.2 default surface (stock un-tuned `GaussianProcessRegressor`)
SRS, no surrogate	Identical outer loop, surrogate never engages — the ablation that isolates the surrogate's contribution
random / Optuna-TPE	Uniform sampling and TPE as the standard baselines (`scikit-optimize` is archived upstream; recorded as “not runnable” rather than silently dropped)

Mean final CV accuracy, B = 30

dataset	SRS + RBF-GP	no surrogate	random	Optuna-TPE
arrhythmia	0.7020	0.6910	0.6941	0.7036
credit-g	0.7423	0.7391	0.7405	0.7436
galaxy3	0.6473	0.6448	0.6484	0.6507
ionosphere	0.9352	0.9347	0.9347	0.9337
optdigits	0.9403	0.9392	0.9380	0.9405
spambase	0.9280	0.9269	0.9272	0.9286

TPE is top on five of six datasets, and the whole table spans roughly half a point — a RandomForest(50) over three parameters is a flat landscape where no search method separates. On four of six datasets, uniform random reaches its own near-best in as few as 2 evaluations.

The caveat, in full

Seven seeds per cell; budgets of 30–60 evaluations; the fixed-pipeline suite's space is only three parameters wide; D-track objectives are smooth and low-dimensional (d ≤ 3); the wall-clock arm uses a different currency (seconds, hardware-dependent) and is explicitly not comparable to the evaluation-count tables. These are small-budget results, and TPE remains competitive. SKSurrogate's edge is in stability, surface quality and evidence — not a universal accuracy crown.

Reproduce it

pip install sksurrogate
python -m benchmarks.run --dataset all --budget 30      # full five-method suite (~1–3 h laptop-scale)
python -m benchmarks.dtrack.d2_compare                  # the regime-map study

Protocol contract + errata history: benchmarks/DESIGN.md
Raw B=30/B=60 records (462 runs): 10.5281/zenodo.23629024
D-track raw data + report: 10.5281/zenodo.23652613
Full write-up: An Honest Benchmark of My Own AutoML Package