The total optimal search criterion in solving the mixed integer linear model with GNSS carrier phase observations

Existing algorithms for GPS ambiguity determination can be classified into three categories, i.e. ambiguity resolution in the measurement domain, the coordinate domain and the ambiguity domain. There are many techniques available for searching the ambiguity domain, such as FARA (Frei and Beutler in Manuscr Geod 15(4):325–356, 1990), LSAST (Hatch in Proceedings of KIS’90, Banff, Canada, pp 299–308, 1990), the modified Cholesky decomposition method (Euler and Landau in Proceedings of the sixth international geodetic symposium on satellite positioning, Columbus, Ohio, pp 650–659, 1992), LAMBDA (Teunissen in Invited lecture, section IV theory and methodology, IAG general meeting, Beijing, China, 1993), FASF (Chen and Lachapelle in J Inst Navig 42(2):371–390, 1995) and modified LLL Algorithm (Grafarend in GPS Solut 4(2):31–44, 2000; Lou and Grafarend in Zeitschrift für Vermessungswesen 3:203–210, 2003). The widely applied LAMBDA method is based on the Least Squares Ambiguity Search (LSAS) criterion and employs an effective decorrelation technique in addition. G. Xu (J Glob Position Syst 1(2):121–131, 2002) proposed also a new general criterion together with its equivalent objective function for ambiguity searching that can be carried out in the coordinate domain, the ambiguity domain or both. Xu’s objective function differs from the LSAS function, leading to different numerical results. The cause of this difference is identified in this contribution and corrected. After correction, the Xu’s approach and the one implied in LAMBDA are identical. We have developed a total optimal search criterion for the mixed integer linear model resolving integer ambiguities in both coordinate and ambiguity domain, and derived the orthogonal decomposition of the objective function and the related minimum expressions algebraically and geometrically. This criterion is verified with real GPS phase data. The theoretical and numerical results show that (1) the LSAS objective function can be derived from the total optimal search criterion with the constraint on the fixed integer ambiguity parameters, and (2) Xu’s derivation of the equivalent objective function was incorrect, leading to an incorrect search procedure. The effects of the total optimal criterion on GPS carrier phase data processing are discussed and its practical implementation is also proposed.