Algebraic monodromy and obstructions to formality