A Comparative Study of Software Secrets Reporting by Secret Detection Tools

Background: According to GitGuardian's monitoring of public GitHub repositories, secrets sprawl continued accelerating in 2022 by 67% compared to 2021, exposing over 10 million secrets (API keys and other credentials). Though many open-source and proprietary secret detection tools are available, these tools output many false positives, making it difficult for developers to take action and teams to choose one tool out of many. To our knowledge, the secret detection tools are not yet compared and evaluated. Aims: The goal of our study is to aid developers in choosing a secret detection tool to reduce the exposure of secrets through an empirical investigation of existing secret detection tools. Method: We present an evaluation of five open-source and four proprietary tools against a benchmark dataset. Results: The top three tools based on precision are: GitHub Secret Scanner (75%), Gitleaks (46%), and Commercial X (25%), and based on recall are: Gitleaks (88%), SpectralOps (67%) and TruffleHog (52%). Our manual analysis of reported secrets reveals that false positives are due to employing generic regular expressions and ineffective entropy calculation. In contrast, false negatives are due to faulty regular expressions, skipping specific file types, and insufficient rulesets. Conclusions: We recommend developers choose tools based on secret types present in their projects to prevent missing secrets. In addition, we recommend tool vendors update detection rules periodically and correctly employ secret verification mechanisms by collaborating with API vendors to improve accuracy.

[1]  Bradley Reaves,et al.  SecretBench: A Dataset of Software Secrets , 2023, 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR).

[2]  Bradley Reaves,et al.  What Challenges Do Developers Face About Checked-in Secrets in Software Artifacts? , 2023, 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE).

[3]  Jens Dietrich,et al.  SecretHunter: A Large-scale Secret Scanner for Public Git Repositories , 2022, 2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom).

[4]  L. Williams,et al.  What are the Practices for Secret Management in Software Artifacts? , 2022, IEEE Cybersecurity Development.

[5]  Yuanyuan Zhang,et al.  Automated Detection of Password Leakage from Public GitHub Repositories , 2022, 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE).

[6]  Md. Rayhanur Rahman,et al.  Why secret detection tools are not enough: It’s not just about false positives - An industrial case study , 2022, Empirical Software Engineering.

[7]  Akond Rahman,et al.  Different Kind of Smells: Security Smells in Infrastructure as Code Scripts , 2021, IEEE Security & Privacy.

[8]  Sneha Kumar Kasera,et al.  Secrets in Source Code: Reducing False Positives using Machine Learning , 2020, 2020 International Conference on COMmunication Systems & NETworkS (COMSNETS).

[9]  Md. Rayhanur Rahman,et al.  Share, But be Aware: Security Smells in Python Gists , 2019, 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[10]  Chris Parnin,et al.  The Seven Sins: Security Smells in Infrastructure as Code Scripts , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[11]  Marco Tulio Valente,et al.  What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform , 2018, J. Syst. Softw..

[12]  Christina Shenvi.  Whispers. , 2018, Annals of emergency medicine.

[13]  IEEE Xplore , 2018, IEEE Communications Standards Magazine.

[14]  David Wang,et al.  Feature scaling , 2018, Radiopaedia.org.

[15]  Vahid Garousi,et al.  Guidelines for including the grey literature and conducting multivocal literature reviews in software engineering , 2017, Inf. Softw. Technol..

[16]  Audris Mockus,et al.  Patterns of folder use and project popularity: a case study of github repositories , 2014, ESEM '14.

[17]  Skye Hardesty.  SpringerLink. , 2006, Issues in Science and Technology Librarianship.

[18]  J. H. Zar,et al.  Spearman Rank Correlation , 2005 .

[19]  Bernard Rous,et al.  The ACM digital library , 2001, CACM.

[20]  Matthew A. Jaro,et al.  Advances in Record-Linkage Methodology as Applied to Matching the 1985 Census of Tampa, Florida , 1989 .

[21]  Bradley Reaves,et al.  Characterizing the Security of Github CI Workflows , 2022, USENIX Security Symposium.

[22]  Slim Trabelsi,et al.  Optimizing Leak Detection in Open-source Platforms with Machine Learning Techniques , 2021, ICISSP.

[23]  Bradley Reaves,et al.  How Bad Can It Git? Characterizing Secret Leakage in Public GitHub Repositories , 2019, NDSS.

[24]  Ekaba Bisong.  Google BigQuery , 2019, Building Machine Learning and Deep Learning Models on Google Cloud Platform.

[25]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[26]  William E. Winkler,et al.  String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage. , 1990 .