Software backdoor analysis based on sensitive flow tracking and concolic execution

In order to effectively detect and analyze the backdoors, this paper introduces a method named Backdoor Analysis based on Sensitive flow tracking and Concolic Execution (BASEC). BASEC uses sensitive flow tracking to effectively discover backdoor behaviors, such as stealing secret information and injecting evil data into system, with less false negatives. With concolic execution on predetermined path, the backdoor trigger condition can be extracted and analyzed to achieve high accuracy. BASEC has been implemented and experimented on several software backdoor samples widespread on the Internet, and over 90% of them can be detected. Compared with behavior-based and system-call-based detection methods, BASEC relies less on the historical sample collections, and is more effective in detecting software backdoors, especially those injected into software by modifying and recompiling source codes.

[1]  Koushik Sen,et al.  Concolic testing , 2007, ASE.

[2]  Christopher Krügel,et al.  Behavior-based Spyware Detection , 2006, USENIX Security Symposium.

[3]  Xuxian Jiang,et al.  Stealthy malware detection and monitoring through VMM-based “out-of-the-box” semantic view reconstruction , 2010, TSEC.

[4]  James Newsome,et al.  Dynamic Taint Analysis for Automatic Detection, Analysis, and SignatureGeneration of Exploits on Commodity Software , 2005, NDSS.

[5]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[6]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[7]  Alexander Egyed,et al.  Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering , 2007, ASE 2007.

[8]  Yi-Min Wang,et al.  Detecting stealth software with Strider GhostBuster , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[9]  Sencun Zhu,et al.  Detecting Software Theft via System Call Based Birthmarks , 2009, 2009 Annual Computer Security Applications Conference.

[10]  Zhenkai Liang,et al.  Automatically Identifying Trigger-based Behavior in Malware , 2008, Botnet Detection.

[11]  Somesh Jha,et al.  Semantics-aware malware detection , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[12]  Dawn Xiaodong Song,et al.  Automatic protocol reverse-engineering: Message format extraction and field semantics inference , 2013, Comput. Networks.

[13]  Rupak Majumdar,et al.  Tools and Algorithms for the Construction and Analysis of Systems , 1997, Lecture Notes in Computer Science.

[14]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[15]  Robert K. Cunningham,et al.  A taxonomy of computer worms , 2003, WORM '03.

[16]  Alessandro Orso,et al.  Dytan: a generic dynamic taint analysis framework , 2007, ISSTA '07.