Autonomous Vehicles (AVs) are equipped with various sensors and controlled by Autonomous Driving Systems (ADSs) to provide high-level autonomy. When interacting with the environment, AVs suffer from a broad attack surface, and the sensory data are susceptible to anomalies caused by faults, sensor malfunctions, or attacks, which may jeopardize traffic safety and result in serious accidents. Most of the current works focus on anomaly detection of specific attacks, such as GPS spoofing or traffic sign attacks. There are no works on scenario-aware anomaly detection for ADSs. In this paper, focusing on the lane-following scenario, we introduce a novel transformer-based one-class classification model to identify time series anomalies and adversarial image examples. It can detect GPS spoofing, traffic sign recognition and lane detection attacks with high efficiency and accuracy. We further design a Swin-transformer model to enhance the detection performance. Experiments on Baidu Apollo and two public data sets (GTSRB and Tusimple) show that compared with the state-of-the-art methods, our method, on average, improves the detection performance by 9.7%, 14.7% and 15.7% for GPS spoofing, traffic sign recognition and lane detection attacks, respectively.