We investigate how to model Markovian evolution of coupled harmonic oscillators, each of them interacting with a local environment. When the coupling between the oscillators is weak, dissipation may be modeled using local Lindblad terms for each of the oscillators in the master equation, as is commonly done. When the coupling between oscillators is strong, this model may become invalid. We derive a master equation for two coupled harmonic oscillators that are subject to individual heat baths modeled by a collection of harmonic oscillators and show that this master equation in general contains nonlocal Lindblad terms. We compare the resulting time evolution with that obtained for dissipation through local Lindblad terms for each individual oscillator and show that the evolution is different in the two cases. In particular, the two descriptions give different predictions for the steady state and for the entanglement between strongly coupled oscillators. This shows that when describing strongly coupled harmonic oscillators, one must take great care in how dissipation is modeled and that a description using local Lindblad terms may fail. This may be particularly relevant when attempting to generate entangled states of strongly coupled quantum systems.