Machining instability, namely chatter, occurs due to the interaction between the structural dynamics and the cutting process. The process damping generated at the tool-workpiece interface is an important parameter of that interaction. A significant enhancement to the chatter simulation model in milling is presented. It includes tracking of the interference between the tool flank and the generated wavy surface, which is the source of process damping. Results of simulation runs performed to determine the limits of stability are presented for sharp tools as well as for tools with various amount of flank wear. The phase relationship between the ploughing force and tool vibrations is explained using these simulations. It is also shown that the improved model accurately predicts the increase in the limit of stability due to tool wear, as well as the effect of the wave length of the machined surface undulations on process damping. Cutting tests of aluminum confirmed the simulation results.