An expert system for checking the correctness of memory systems using simulation and metamorphic testing

Abstract During the last few years, computer performance has reached a turning point where computing power is no longer the only important concern. This way, the emphasis is shifting from an exclusive focus on the optimisation of the computing system to optimising other systems, like the memory system. Broadly speaking, testing memory systems entails two main challenges: the oracle problem and the reliable test set problem. The former consists in deciding if the outputs of a test suite are correct. The latter refers to providing an appropriate test suite for determining the correctness of the system under test. In this paper we propose an expert system for checking the correctness of memory systems. In order to face these challenges, our proposed system combines two orthogonal techniques – simulation and metamorphic testing – enabling the automatic generation of appropriate test cases and deciding if their outputs are correct. In contrast to conventional expert systems, our system includes a factual database containing the results of previous simulations, and a simulation platform for computing the behaviour of memory systems. The knowledge of the expert is represented in the form of metamorphic relations, which are properties of the analysed system involving multiple inputs and their outputs. Thus, the main contribution of this work is two-fold: a method to automatise the testing process of memory systems, and a novel expert system design focusing on increasing the overall performance of the testing process. To show the applicability of our system, we have performed a thorough evaluation using 500 memory configurations and 4 different memory management algorithms, which entailed the execution of more than one million of simulations. The evaluation used mutation testing, injecting faults in the memory management algorithms. The developed expert system was able to detect over 99% of the critical injected faults, hence obtaining very promising results, and outperforming other standard techniques like random testing.

[1]  William J. Dally,et al.  Memory access scheduling , 2000, Proceedings of 27th International Symposium on Computer Architecture (IEEE Cat. No.RS00201).

[2]  Katsuhiko Sato,et al.  Universal Test Interface for Embedded-DRAM Testing , 1999, IEEE Des. Test Comput..

[3]  Rajeev Balasubramonian,et al.  Avoiding information leakage in the memory controller with fixed service policies , 2015, 2015 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[4]  Natalie D. Enright Jerger,et al.  Achieving predictable performance through better memory controller placement in many-core CMPs , 2009, ISCA '09.

[5]  Clifford R. Hollander,et al.  DART: An Expert System for Computer Fault Diagnosis , 1981, IJCAI.

[6]  Zuohua Ding,et al.  Testing Central Processing Unit scheduling algorithms using Metamorphic Testing , 2013, 2013 IEEE 4th International Conference on Software Engineering and Service Science.

[7]  Huai Liu,et al.  How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? , 2014, IEEE Transactions on Software Engineering.

[8]  Onur Mutlu,et al.  Ramulator: A Fast and Extensible DRAM Simulator , 2016, IEEE Computer Architecture Letters.

[9]  Kai Li,et al.  The PARSEC benchmark suite: Characterization and architectural implications , 2008, 2008 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[10]  Ding-Ming Kwai,et al.  A hybrid built-in self-test scheme for DRAMs , 2015, VLSI Design, Automation and Test(VLSI-DAT).

[11]  Khaled Salah,et al.  Implementation and verification of a generic universal memory controller based on UVM , 2015, 2015 10th International Conference on Design & Technology of Integrated Systems in Nanoscale Era (DTIS).

[12]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[13]  David W. Nellans,et al.  Handling the problems and opportunities posed by multiple on-chip memory controllers , 2010, 2010 19th International Conference on Parallel Architectures and Compilation Techniques (PACT).

[14]  Ad J. van de Goor,et al.  Generating march tests automatically , 1994, Proceedings., International Test Conference.

[15]  Bertrand Meyer,et al.  On the number and nature of faults found by random testing , 2011, Softw. Test. Verification Reliab..

[16]  Manuel Núñez,et al.  A Methodology for Designing Energy-aware Systems for Computational Science , 2015, ICCS.

[17]  José Antenor Pomilio,et al.  Novel expert system for defining power quality compensators , 2015, Expert Syst. Appl..

[18]  Onur Mutlu,et al.  Self-Optimizing Memory Controllers: A Reinforcement Learning Approach , 2008, 2008 International Symposium on Computer Architecture.

[19]  Kuo-Hsing Cheng.  Dynamic Random Access Memory , 1999, The VLSI Handbook.

[20]  José F. Martínez,et al.  Understanding and mitigating refresh overheads in high-density DDR4 DRAM systems , 2013, ISCA.

[21]  Faye A. Briggs,et al.  A study of performance impact of memory controller features in multi-processor server environment , 2004, WMPI '04.

[22]  Hiren D. Patel,et al.  A framework for scheduling DRAM memory accesses for multi-core mixed-time critical systems , 2015, 21st IEEE Real-Time and Embedded Technology and Applications Symposium.

[23]  Mario Piattini,et al.  Mutation Testing , 2014, IEEE Software.

[24]  Wei-Fen Lin,et al.  Reducing DRAM latencies with an integrated memory hierarchy design , 2001, Proceedings HPCA Seventh International Symposium on High-Performance Computer Architecture.

[25]  Baowen Xu,et al.  Application of Metamorphic Testing to Supervised Classifiers , 2009, 2009 Ninth International Conference on Quality Software.

[26]  Matthew Poremba,et al.  NVMain: An Architectural-Level Main Memory Simulator for Emerging Non-volatile Memories , 2012, 2012 IEEE Computer Society Annual Symposium on VLSI.

[27]  Bruce Jacob,et al.  Memory Systems: Cache, DRAM, Disk , 2007 .

[28]  Kees G. W. Goossens,et al.  Modeling and Verification of Dynamic Command Scheduling for Real-Time Memory Controllers , 2016, 2016 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS).

[29]  Benoît Dupont de Dinechin,et al.  Time-critical computing on a single-chip massively parallel processor , 2014, 2014 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[30]  Robert M. Hierons,et al.  A methodology for validating cloud models using metamorphic testing , 2015, Ann. des Télécommunications.

[31]  Tsong Yueh Chen,et al.  Metamorphic Testing: A New Approach for Generating Next Test Cases , 2020, ArXiv.

[32]  Seth H. Pugsley,et al.  Memory bandwidth reservation in the cloud to avoid information leakage in the memory controller , 2014, HASP@ISCA.

[33]  Jan Reineke,et al.  Ascertaining Uncertainty for Efficient Exact Cache Analysis , 2017, CAV.

[34]  Nihar R. Mahapatra,et al.  The processor-memory bottleneck: problems and solutions , 1999, CROS.

[35]  Brad Calder,et al.  Discovering and Exploiting Program Phases , 2003, IEEE Micro.

[36]  Mor Harchol-Balter,et al.  ATLAS: A scalable and high-performance scheduling algorithm for multiple memory controllers , 2010, HPCA - 16 2010 The Sixteenth International Symposium on High-Performance Computer Architecture.

[37]  Mark G. Karpovsky,et al.  Pseudo-exhaustive word-oriented DRAM testing , 1995, Proceedings the European Design and Test Conference. ED&TC 1995.

[38]  James M. Bieman,et al.  Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels , 2016, Softw. Test. Verification Reliab..

[39]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[40]  Cheng-Wen Wu,et al.  Simulation-based test algorithm generation for random access memories , 2000, Proceedings 18th IEEE VLSI Test Symposium.

[41]  Nitin,et al.  Understanding and Analyzing the Impact of Memory Controller's Scheduling Policies on DRAM's Energy and Performance , 2015 .

[42]  David W. Nellans,et al.  Micro-pages: increasing DRAM efficiency with locality-aware data placement , 2010, ASPLOS XV.

[43]  Xuan Liu,et al.  A New Method for Constructing Metamorphic Relations , 2012, 2012 12th International Conference on Quality Software.

[44]  Bruce Jacob,et al.  DRAMSim2: A Cycle Accurate Memory System Simulator , 2011, IEEE Computer Architecture Letters.

[45]  Akhtar Hussain,et al.  An expert system for acoustic diagnosis of power circuit breakers and on-load tap changers , 2015, Expert Syst. Appl..

[46]  Aamer Jaleel,et al.  DReAM: Dynamic Re-arrangement of Address Mapping to Improve the Performance of DRAMs , 2015, MEMSYS.

[47]  Carlos Miguel Herrero-Jimenez.  An expert system for the identification of environmental impact based on a geographic information system , 2012, Expert Syst. Appl..

[48]  Onur Mutlu,et al.  BLISS: Balancing Performance, Fairness and Complexity in Memory Access Scheduling , 2016, IEEE Transactions on Parallel and Distributed Systems.

[49]  Sally A. McKee,et al.  Hitting the memory wall: implications of the obvious , 1995, CARN.

[50]  Brian Rogers,et al.  Scaling the bandwidth wall: challenges in and avenues for CMP scaling , 2009, ISCA '09.

[51]  Mohamed O. Kayed,et al.  A Novel Approach for SVA Generation of DDR Memory Protocols Based on TDML , 2014, 2014 15th International Microprocessor Test and Verification Workshop.

[52]  Seth H. Pugsley,et al.  USIMM : the Utah SImulated Memory Module , 2012 .

[53]  Joaquín Bautista,et al.  An expert system to minimize operational costs in mixed-model sequencing problems with activity factor , 2018, Expert Syst. Appl..

[54]  Dongmei Zhang,et al.  An Application of Metamorphic Testing for Testing Scientific Software , 2016, 2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET).

[55]  Cheng-Wen Wu,et al.  A Programmable BIST Core for Embedded DRAM , 1999, IEEE Des. Test Comput..

[56]  Ad J. van de Goor,et al.  Automating the verification of memory tests , 1994, Proceedings of IEEE VLSI Test Symposium.

[57]  Mohamed Hassan,et al.  MCXplore: Automating the Validation Process of DRAM Memory Controller Designs , 2018, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[58]  Stephan Merz,et al.  Model Checking , 2000 .

[59]  Lionel C. Briand,et al.  Random Testing: Theoretical Results and Practical Implications , 2012, IEEE Transactions on Software Engineering.

[60]  Pawel Gepner,et al.  Multi-Core Processors: New Way to Achieve High System Performance , 2006, PARELEC.

[61]  Howard Leo Kalter,et al.  Processor-based built-in self-test for embedded DRAM , 1998, IEEE J. Solid State Circuits.

[62]  Manoranjan Satpathy,et al.  MSimDRAM: Formal Model Driven Development of a DRAM Simulator , 2016, 2016 29th International Conference on VLSI Design and 2016 15th International Conference on Embedded Systems (VLSID).