In this Letter we present the controlled coupling of a single nitrogen vacancy center to a plasmonic structure. With the help of an atomic force microscope, a single nanodiamond containing a single nitrogen vacancy center and two gold nanospheres are assembled step-by-step. We show that both the excitation rate and the radiative decay rate of the color center are enhanced by about 1 order of magnitude, while the single photon character of the emission is maintained. Hot spots between diamond and gold nanoparticles provide an efficient near-field coupling, despite the mismatch in size and shape. Our approach provides hybrid systems as important building blocks for novel nanophotonic light sources in advanced plasmonic devices stable even at room temperature.