SPOT applied to non-stochastic optimization problems: an experimental study

Most parameter tuning methods feature a number of parameters themselves. This also holds for the Sequential Parameter Optimization [1] Toolbox (SPOT). It provides default values, which are reasonable for many problems, but these defaults are set to favor robustness over performance. By default, a Random Forest (RF) [2] model is used for the surrogate optimization. The RF model is built rather fast. It runs robustly (i.e. it does not crash) and can handle non-ordered parameters (i.e. factors) very well. However, the RF model does provide poor optimization performance for a number of problems, due to the inbuilt discontinuities. It would often be more reasonable to use Kriging models [4]. These usually perform well for small and medium sized decision space dimensions. For use with the SPOT package, there are several existing packages that provide Kriging methods that often fit the required problem well (DiceKriging, mlegp, etc.). However, these methods have one thing in common, they are not robust. Especially when several design points (samples in the decision space) are close to each other, those functions often fail. Hence, in SPOT versions greater 1.0, a Kriging model based on the Matlab code by Forrester et.al. [3] was introduced.