Development of a health monitoring system for construction has become an important task for structural damage identification. Since the building structure has many uncertain factors, the method of using modal analysis to identify the structural damage is of low accuracy. A method applying a grey system to the structural damage identification is presented. The grey system for the frequency variance rate and the stiffness change is established with the first-order single-argument grey system (GM(1, 1)). GM(1, 1) prediction is used to reflect the global function of the structural dynamic fingerprints and find the relation of the frequency variance rate and the stiffness change. Vibration tests for frame structures were carried out, with many cases considered, including single-damage and multi-damage ones with different degrees and locations. The results show that for shear buildings, the damage degree and location can be determined by using the grey system and measuring the frequency change.