Beyond Over-Protection: A Targeted Approach to Spectre Mitigation and Performance Optimization