When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks