Towards reducing the attack surface of software backdoors

Backdoors in software systems probably exist since the very first access control mechanisms were implemented and they are a well-known security problem. Despite a wave of public discoveries of such backdoors over the last few years, this threat has only rarely been tackled so far. In this paper, we present an approach to reduce the attack surface for this kind of attacks and we strive for an automated identification and elimination of backdoors in binary applications. We limit our focus on the examination of server applications within a client-server model. At the core, we apply variations of the delta debugging technique and introduce several novel heuristics for the identification of those regions in binary application that backdoors are typically installed in (i.e., authentication and command processing functions). We demonstrate the practical feasibility of our approach on several real-world backdoors found in modified versions of the popular software tools ProFTPD and OpenSSH. Furthermore, we evaluate our implementation not only on common instruction set architectures such as x86-64, but also on commercial off-the-shelf embedded devices powered by a MIPS32 processor.

[1]  Angelos D. Keromytis,et al.  Adaptive defenses for commodity software through virtual application partitioning , 2012, CCS.

[2]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[3]  David A. Wagner,et al.  Defeating UCI: Building Stealthy and Malicious Hardware , 2011, 2011 IEEE Symposium on Security and Privacy.

[4]  Loïc Duflot,et al.  CPU bugs, CPU backdoors and consequences on security , 2008, Journal in Computer Virology.

[5]  Andrew C. Myers,et al.  Secure program partitioning , 2002, TOCS.

[6]  Halvar Flake,et al.  Structural Comparison of Executable Objects , 2004, DIMVA.

[7]  David Brumley,et al.  TIE: Principled Reverse Engineering of Types in Binary Programs , 2011, NDSS.

[8]  Ken Thompson,et al.  Reflections on trusting trust , 1984, CACM.

[9]  Chao Zhang,et al.  A Framework to Eliminate Backdoors from Response-Computable Authentication , 2012, 2012 IEEE Symposium on Security and Privacy.

[10]  Niels Provos,et al.  Preventing Privilege Escalation , 2003, USENIX Security Symposium.

[11]  Zhenkai Liang,et al.  Automatically Identifying Trigger-based Behavior in Malware , 2008, Botnet Detection.

[12]  Berk Sunar,et al.  Trojan Detection using IC Fingerprinting , 2007, 2007 IEEE Symposium on Security and Privacy (SP '07).

[13]  Herbert Bos,et al.  Howard: A Dynamic Excavator for Reverse Engineering Data Structures , 2011, NDSS.

[14]  Yuanyuan Zhou,et al.  Designing and Implementing Malicious Hardware , 2008, LEET.

[15]  Steven Hand,et al.  Privilege separation made easy: trusting small libraries not big processes , 2008, EUROSEC '08.

[16]  Farinaz Koushanfar,et al.  A Survey of Hardware Trojan Taxonomy and Detection , 2010, IEEE Design & Test of Computers.

[17]  Milo M. K. Martin,et al.  Overcoming an Untrusted Computing Base: Detecting and Removing Malicious Hardware Automatically , 2010, 2010 IEEE Symposium on Security and Privacy.

[18]  Tatu Ylönen,et al.  The Secure Shell (SSH) Authentication Protocol , 2006, RFC.

[19]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[20]  Cliff Changchun Zou,et al.  A chipset level network backdoor: bypassing host-based firewall & IDS , 2009, ASIACCS '09.

[21]  Harish Patil,et al.  Pin: building customized program analysis tools with dynamic instrumentation , 2005, PLDI '05.

[22]  Simha Sethumadhavan,et al.  Silencing Hardware Backdoors , 2011, 2011 IEEE Symposium on Security and Privacy.

[23]  H. T. Kung,et al.  DISTROY: Detecting Integrated Circuit Trojans with Compressive Measurements , 2011, HotSec.

[24]  Abhay K. Bhushan,et al.  The File Transfer Protocol , 1971, Request for Comments.

[25]  Christoforos E. Kozyrakis,et al.  Nemesis: Preventing Authentication & Access Control Vulnerabilities in Web Applications , 2009, USENIX Security Symposium.

[26]  Mark Handley,et al.  Wedge: Splitting Applications into Reduced-Privilege Compartments , 2008, NSDI.

[27]  David Brumley,et al.  Privtrans: Automatically Partitioning Programs for Privilege Separation , 2004, USENIX Security Symposium.

[28]  Debin Gao,et al.  BinHunt: Automatically Finding Semantic Differences in Binary Programs , 2008, ICICS.

[29]  Douglas Kilpatrick,et al.  Privman: A Library for Partitioning Applications , 2003, USENIX Annual Technical Conference, FREENIX Track.