Quantum Theory as an Emergent Phenomenon: Foundations and Phenomenology