A Closer Look at the Limitations of Instruction Tuning