Natural human activity recognition (HAR) via millimeter wave (mmWave) sensing is a key to the human-computer interaction (HCI), e.g., activity assistance and living state monitoring. Prior work has shown the feasibility of HAR by utilizing mmWave radar, but it falls short of two real-world issues: poor recognition accuracy in the noisy environment and unable to give real-time response due to long latency. In this paper, we propose m-Activity, which can realize HAR while reducing noise caused by environmental multi-path effects, and operate fluently at runtime. m-Activity first distills the human-orientated movements from the noisy background environment and then classify the movements using a custom-designed lightweight neural network called HARnet. To drive the above methods, we propose a simple but efficient response mechanism to enable real-time recognition. We prototype m-Activity on a commodity mmWave radar chip and evaluate its recognition performance over 5 pre-defined human activities within the detection range of 3m, which results in off-line accuracy of 93.25%, and real-time accuracy of 91.52%. Furthermore, we validate m-Activity’s ability under a complex real-world scenario, i.e., fitness center, which is full of severe multi-path effects caused by various strong metal reflectors.