Two-dimensional materials provide a unique platform to explore the full potential of magnetic proximity-driven phenomena, which can be further used for applications in next-generation spintronic devices. Of particular interest is to understand and control spin currents in graphene by the magnetic exchange field of a nearby ferromagnetic material in graphene-ferromagnetic-insulator (FMI) heterostructures. Here, we present the experimental study showing the strong modulation of spin currents in graphene layers by controlling the direction of the exchange field due to FMI magnetization. Owing to clean interfaces, a strong magnetic exchange coupling leads to the experimental observation of complete spin modulation at low externally applied magnetic fields in short graphene channels. Additionally, we discover that the graphene spin current can be fully dephased by randomly fluctuating exchange fields. This is manifested as an unusually strong temperature dependence of the nonlocal spin signals in graphene, which is due to spin relaxation by thermally induced transverse fluctuations of the FMI magnetization.