A general perspective on time observables