Achieving autonomous power management using reinforcement learning

System level power management must consider the uncertainty and variability that come from the environment, the application and the hardware. A robust power management technique must be able to learn the optimal decision from past events and improve itself as the environment changes. This article presents a novel on-line power management technique based on model-free constrained reinforcement learning (Q-learning). The proposed learning algorithm requires no prior information of the workload and dynamically adapts to the environment to achieve autonomous power management. We focus on the power management of the peripheral device and the microprocessor, two of the basic components of a computer. Due to their different operating behaviors and performance considerations, these two types of devices require different designs of Q-learning agent. The article discusses system modeling and cost function construction for both types of Q-learning agent. Enhancement techniques are also proposed to speed up the convergence and better maintain the required performance (or power) constraint in a dynamic system with large variations. Compared with the existing machine learning based power management techniques, the Q-learning based power management is more flexible in adapting to different workload and hardware and provides a wider range of power-performance tradeoff.

[1]  Sanjay Ranka,et al.  Using game theory for scheduling tasks on multi-core processors for simultaneous optimization of performance and energy , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing.

[2]  Ying Tan,et al.  Stochastic Modeling and Optimization for Robust Power Management in a Partially Observable System , 2007, 2007 Design, Automation & Test in Europe Conference & Exhibition.

[3]  Tajana Simunic,et al.  Utilizing Predictors for Efficient Thermal Management in Multiprocessor SoCs , 2009, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[4]  Tajana Simunic,et al.  Temperature management in multiprocessor SoCs using online learning , 2008, 2008 45th ACM/IEEE Design Automation Conference.

[5]  Tajana Simunic,et al.  Temperature Aware Dynamic Workload Scheduling in Multisocket CPU Servers , 2011, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[6]  Rich Caruana,et al.  Greedy Attribute Selection , 1994, ICML.

[7]  Massoud Pedram,et al.  Fine-grained dynamic voltage and frequency scaling for precise energy and performance tradeoff based on the ratio of off-chip access to on-chip computation times , 2005 .

[8]  Tajana Simunic,et al.  Temperature Aware Task Scheduling in MPSoCs , 2007, 2007 Design, Automation & Test in Europe Conference & Exhibition.

[9]  Ying Tan,et al.  A Framework of Stochastic Power Management Using Hidden Markov Model , 2008, 2008 Design, Automation and Test in Europe.

[10]  Tajana Simunic,et al.  Evaluating the impact of job scheduling and power management on processor lifetime for chip multiprocessors , 2009, SIGMETRICS '09.

[11]  George Theocharous,et al.  Machine Learning for Adaptive Power Management , 2006 .

[12]  Rajarshi Das,et al.  A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation , 2006, 2006 IEEE International Conference on Autonomic Computing.

[13]  Tajana Simunic,et al.  Predict and act: dynamic thermal management for multi-core processors , 2009, ISLPED.

[14]  Jin Qian,et al.  PARAID: A gear-shifting power-aware RAID , 2007, TOS.

[15]  Tajana Simunic,et al.  JETC: Joint energy thermal and cooling management for memory and CPU subsystems in servers , 2012, IEEE International Symposium on High-Performance Comp Architecture.

[16]  Engin Ipek,et al.  Dynamic Multicore Resource Management: A Machine Learning Approach , 2009, IEEE Micro.

[17]  Ying Tan,et al.  Workload prediction and dynamic voltage scaling for MPEG decoding , 2006, Asia and South Pacific Conference on Design Automation, 2006..

[18]  Massoud Pedram,et al.  Fine-Grained Dynamic Voltage and Frequency Scaling for Precise Energy and Performance Trade-Off Based on the Ratio of Off-Chip Access to On-Chip Computation Times , 2004, DATE.

[19]  Niraj K. Jha,et al.  An energy-aware framework for dynamic software management in mobile computing systems , 2008, TECS.

[20]  David Levine,et al.  Managing Power Consumption and Performance of Computing Systems Using Reinforcement Learning , 2007, NIPS.

[21]  Tajana Simunic,et al.  Temperature-aware MPSoC scheduling for reducing hot spots and gradients , 2008, 2008 Asia and South Pacific Design Automation Conference.

[22]  Chenyang Lu,et al.  Introduction to Control Theory And Its Application to Computing Systems , 2008 .

[23]  Li Shang,et al.  Thermal vs Energy Optimization for DVFS-Enabled Processors in Embedded Systems , 2007, 8th International Symposium on Quality Electronic Design (ISQED'07).

[24]  Yung-Hsiang Lu,et al.  Joint Power Management of Memory and Disk Under Performance Constraints , 2006, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[25]  Onur Mutlu,et al.  Self-Optimizing Memory Controllers: A Reinforcement Learning Approach , 2008, 2008 International Symposium on Computer Architecture.

[26]  Rajesh Gupta,et al.  SleepServer: A Software-Only Approach for Reducing the Energy Consumption of PCs within Enterprise Environments , 2010, USENIX Annual Technical Conference.

[27]  Wei Liu,et al.  Adaptive power management using reinforcement learning , 2009, 2009 IEEE/ACM International Conference on Computer-Aided Design - Digest of Technical Papers.

[28]  Allen C.-H. Wu,et al.  A predictive system shutdown method for energy saving of event-driven computation , 1997, 1997 Proceedings of IEEE International Conference on Computer Aided Design (ICCAD).

[29]  Luca Benini,et al.  Event-driven power management , 2001, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[30]  Giovanni De Micheli,et al.  Multicore thermal management with model predictive control , 2009, 2009 European Conference on Circuit Theory and Design.

[31]  Mark D. Pendrith On Reinforcement Learning of Control Actions in Noisy and Non-Markovian Domains , 1994 .

[32]  Mani B. Srivastava,et al.  Power management in energy harvesting sensor networks , 2007, TECS.

[33]  Qinru Qiu,et al.  Dynamic thermal management for multimedia applications using machine learning , 2011, 2011 48th ACM/EDAC/IEEE Design Automation Conference (DAC).

[34]  Y. Charlie Hu,et al.  Program counter-based prediction techniques for dynamic power management , 2006, IEEE Transactions on Computers.

[35]  Kai Ma,et al.  Temperature-constrained power control for chip multiprocessors with online model estimation , 2009, ISCA '09.

[36]  Rajesh K. Gupta,et al.  Leakage aware dynamic voltage scaling for real-time embedded systems , 2004, Proceedings. 41st Design Automation Conference, 2004..

[37]  Srivaths Ravi,et al.  Energy-optimizing source code transformations for operating system-driven embedded software , 2007, TECS.

[38]  Radu Marculescu,et al.  On-chip traffic modeling and synthesis for MPEG-2 video applications , 2004, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[39]  Wei Liu,et al.  Enhanced Q-learning algorithm for dynamic power management with performance constraint , 2010, 2010 Design, Automation & Test in Europe Conference & Exhibition (DATE 2010).

[40]  Ben H. H. Juurlink,et al.  Leakage-aware multiprocessor scheduling for low power , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[41]  Yung-Hsiang Lu,et al.  A Homogeneous Architecture for Power Policy Integration in Operating Systems , 2009, IEEE Transactions on Computers.

[42]  Diana Marculescu,et al.  Power Management of Voltage/Frequency Island-Based Systems Using Hardware-Based Methods , 2009, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[43]  John Wilkes,et al.  UNIX Disk Access Patterns , 1993, USENIX Winter.

[44]  Eun Jung Kim,et al.  Predictive dynamic thermal management for multicore systems , 2008, 2008 45th ACM/IEEE Design Automation Conference.

[45]  Radu Marculescu,et al.  Designing Heterogeneous Embedded Network-on-Chip Platforms With Users in Mind , 2010, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[46]  Tajana Simunic,et al.  System-Level Power Management Using Online Learning , 2009, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[47]  Sudhanva Gurumurthi,et al.  Accelerating enterprise solid-state disks with non-volatile merge caching , 2010, International Conference on Green Computing.

[48]  Hai Zhou,et al.  Parallel CAD: Algorithm Design and Programming Special Section Call for Papers TODAES: ACM Transactions on Design Automation of Electronic Systems , 2010 .