Realism in image synthesis increases significantly when real-world lighting captured into environment maps (EM) is used to illuminate rendered scenes. Tremendous progress in the development and accessibility of high dynamic range (HDR) video cameras enable direct capturing of HDR video environment maps (VEM). In this work we present an interactive system for fully dynamic lighting of a scene using captured HDR VEM. The key component of our system is an algorithm for efficient decomposition of HDR VEM into a set of representative directional light sources, which can be used for the direct lighting computation with shadows on graphics hardware. The resulting lights have favorable properties in terms of flickering reduction and their number can be adaptively changed to keep a constant framerate while good spatial distribution (stratification) properties are maintained. We can handle a large number of light sources with shadows using novel techniques, which reduce the cost of BRDF-based shading and visibility computations.