Deconstructing the Galaxy Stellar Mass Function with UKIDSS and CANDELS: The Impact of Colour, Structure and Environment