Spectral decomposition for operator self-similar processes and their generalized domains of attraction