The growing number of miniaturized satellites or small-body space debris is a challenging problem for autonomous ground-based space object observation. Although most space objects larger than 10 cm in diameter have been catalogued by North American Aerospace Defense Command, the precise orbital information of each space object (based on six orbital parameters) remains important and should be maintained periodically due to orbital perturbations. In the past decades, modern ground-based Electro-optic telescopes equipped with electronic detectors have been widely used in astrometry engineering.The tracking performance of this equipment primarily depends on the size and brightness of the space target. Moreover, in the real-time observation procedure based on STARE tracking mode in a short exposure time, the space object and stellar background will similarly appear in the point-spread function with different levels of signal-to-noise ratio under the variable conditions of background interference, which is difficult to recognize. The aim of present work is to achieve high-sensitivity detection and improved tracking ability for non-Gaussian and dynamic backgrounds with a simple mechanism and computational efficiency. To overcome this limitation, we emphasize robust tracking of small size satellite and faint object via a state estimation technique. We proposed a neural-network based adaptive running Gaussian average algorithm to extract a moving space object from the stellar background and its interference. The algorithm was integrated to a Track-before-Detect (TBD) framework which used Monte-Carlo based particle filter. The integrated algorithms were adopted to track the space object. Three sequential astronomical image datasets taken by the Asia-Pacific Ground-Based Optical Space Object Observation System (APOSOS) telescopes under different conditions were used to evaluate the tracking strategy. The results showed that the scheme achieved a satisfying tracking performance.