Particle Swarm Optimisation

The problem The general problem tackled by PSO is that of minimising (or maximising) a function, e.g. finding the combination of tuning parameters that gives the best RMSECV for a support vector machine. A useful analogy is that of finding the top of a hill. If the hill has a simple shape and a single summit, you can arrive at the top using the strategy of always walking uphill. This is how the standard deterministic search methods work, typically estimating the gradient of the function at the current point and moving in a direction that is steeply upwards. Many real life problems are not as easy as this, being more akin to the task of finding the summit of the highest mountain in a country you have never visited before and for which no map exists. If you start from some point in the country and walk uphill you will arrive at a summit, but which one? If the country is small enough and you have enough time you could do a systematic search, checking all the points on a suitably chosen grid. If it is large (here the analogy breaks down, because a large optimisation problem typically involves a lot of parameters, i.e. a lot of dimensions, rather than a large range of two parameters) then this may not be feasible. It is this problem of maximising a function of many variables that may have many local maxima that methods like PSO aim to tackle.