Not all parameters are born equal: Attention is mostly what you need