Deep reinforcement learning for efficient measurement of quantum devices

Deep reinforcement learning is an emerging machine learning approach which can teach a computer to learn from their actions and rewards similar to the way humans learn from experience. It offers many advantages in automating decision processes to navigate large parameter spaces. This paper proposes a novel approach to the efficient measurement of quantum devices based on deep reinforcement learning. We focus on double quantum dot devices, demonstrating the fully automatic identification of specific transport features called bias triangles. Measurements targeting these features are difficult to automate, since bias triangles are found in otherwise featureless regions of the parameter space. Our algorithm identifies bias triangles in a mean time of less than 30 minutes, and sometimes as little as 1 minute. This approach, based on dueling deep Q-networks, can be adapted to a broad range of devices and target transport features. This is a crucial demonstration of the utility of deep reinforcement learning for decision making in the measurement and operation of quantum devices.

[1]  P. T. Eendebak,et al.  Computer-automated tuning of semiconductor double quantum dots into the single-electron regime , 2016, 1603.02274.

[2]  Sahar Daraeizadeh,et al.  Designing high-fidelity multi-qubit gates for semiconductor quantum dots through deep reinforcement learning , 2020, 2020 IEEE International Conference on Quantum Computing and Engineering (QCE).

[3]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[4]  Dario Tamascelli,et al.  Coherent transport of quantum states by deep reinforcement learning , 2019, Communications Physics.

[5]  Jacob M. Taylor,et al.  Machine learning techniques for state recognition and auto-tuning in quantum dots , 2017, npj Quantum Information.

[6]  Dino Sejdinovic,et al.  Quantum device fine-tuning using unsupervised embedding learning , 2020, New Journal of Physics.

[7]  D. DiVincenzo,et al.  Quantum computation with quantum dots , 1997, cond-mat/9701055.

[8]  Pankaj Mehta,et al.  Reinforcement Learning in Different Phases of Quantum Control , 2017, Physical Review X.

[9]  Saeed Fallahi,et al.  Notch filtering the nuclear environment of a spin qubit. , 2016, Nature nanotechnology.

[10]  Werner Wegscheider,et al.  Automated Tuning of Double Quantum Dots into Specific Charge States Using Neural Networks , 2019 .

[11]  K. W. Chan,et al.  Autonomous Tuning and Charge-State Detection of Gate-Defined Quantum Dots , 2019, Physical Review Applied.

[12]  Andreas D. Wieck,et al.  Closed-loop control of a GaAs-based singlet-triplet spin qubit with 99.5% gate fidelity and low leakage , 2019, Nature Communications.

[13]  P. T. Eendebak,et al.  Loading a quantum-dot based “Qubyte” register , 2019, npj Quantum Information.

[14]  Matthias Troyer,et al.  Solving the quantum many-body problem with artificial neural networks , 2016, Science.

[15]  Xin Wang,et al.  Transferable control for quantum parameter estimation through reinforcement learning , 2019, 1904.11298.

[16]  E. R. MacQuarrie,et al.  Autotuning of double dot devices in situ with machine learning. , 2020, Physical review applied.

[17]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[18]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[19]  Barteld Kooi,et al.  Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , 2011, Adaptive Agents and Multi-Agent Systems.

[20]  K. Itoh,et al.  A quantum-dot spin qubit with coherence limited by charge noise and fidelity higher than 99.9% , 2018, Nature Nanotechnology.

[21]  Andreas D. Wieck,et al.  A machine learning approach for automated fine-tuning of semiconductor spin qubits , 2019, Applied Physics Letters.

[22]  J. P. Dehollain,et al.  A two-qubit logic gate in silicon , 2014, Nature.

[23]  Jacob M. Taylor,et al.  Self-consistent measurement and state tomography of an exchange-only spin qubit. , 2013, Nature nanotechnology.

[24]  Jordi Arbiol,et al.  A singlet-triplet hole spin qubit in planar Ge , 2020, Nature Materials.

[25]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[26]  Hartmut Neven,et al.  Universal quantum control through deep reinforcement learning , 2018, npj Quantum Information.

[27]  L. Vandersypen,et al.  Supporting Online Material for Coherent Control of a Single Electron Spin with Electric Fields Materials and Methods Som Text Figs. S1 and S2 References , 2022 .

[28]  J. R. Petta,et al.  Radio frequency charge sensing in InAs nanowire double quantum dots , 2012, 1205.6494.

[29]  Chris Toumey.  Reality, fantasy and civility in molecular assemblers , 2018, Nature Nanotechnology.

[30]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[31]  Jacob M. Taylor,et al.  Resonantly driven CNOT gate for electron spins , 2018, Science.

[32]  Daniel Braun,et al.  Improving the dynamics of quantum sensors with reinforcement learning , 2019, New Journal of Physics.

[33]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[34]  D. P. DiVincenzo,et al.  Coherent spin manipulation in an exchange-only qubit , 2010, 1005.0273.

[35]  José Miguel Hernández-Lobato,et al.  Taking gradients through experiments: LSTMs and memory proximal policy optimization for black-box quantum control , 2018, ISC Workshops.

[36]  D. Ritchie,et al.  Sensitive radio-frequency measurements of a quantum dot by tuning to perfect impedance matching , 2015, 1510.06944.

[37]  Austen Lamacraft,et al.  Quantum Ground States from Reinforcement Learning , 2020, MSML.

[38]  Ritchie,et al.  Measurements of Coulomb blockade with a noninvasive voltage probe. , 1993, Physical review letters.

[39]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[40]  G. A. D. Briggs,et al.  Sensitive radiofrequency readout of quantum dots using an ultra-low-noise SQUID amplifier , 2018, Journal of Applied Physics.

[41]  Leo P. Kouwenhoven,et al.  Rapid Detection of Coherent Tunneling in an InAs Nanowire Quantum Dot through Dispersive Gate Sensing , 2018, Physical Review Applied.

[42]  Maud Vinet,et al.  Level Spectrum and Charge Relaxation in a Silicon Double Quantum Dot Probed by Dual-Gate Reflectometry. , 2016, Nano letters.

[43]  Carl E. Rasmussen,et al.  PILCO: A Model-Based and Data-Efficient Approach to Policy Search , 2011, ICML.

[44]  Peng Wei,et al.  Open quantum system control based on reinforcement learning , 2019, 2019 Chinese Control Conference (CCC).

[45]  Andrew S. Dzurak,et al.  Fidelity benchmarks for two-qubit gates in silicon , 2018, Nature.

[46]  Tom Schaul,et al.  Prioritized Experience Replay , 2015, ICLR.

[47]  Liuqi Yu,et al.  Spectroscopy of Quantum Dot Orbitals with In-Plane Magnetic Fields. , 2018, Physical review letters.

[48]  Dino Sejdinovic,et al.  Machine learning enables completely automatic tuning of a quantum device faster than human experts , 2020, Nature Communications.

[49]  Barry C. Sanders,et al.  Learning in quantum control: High-dimensional global optimization for noisy quantum dynamics , 2016, Neurocomputing.

[50]  Liuqi Yu,et al.  Hyperfine-phonon spin relaxation in a single-electron GaAs quantum dot , 2017, Nature Communications.

[51]  Justyna P. Zwolak,et al.  QFlow lite dataset: A machine-learning approach to the charge states in quantum dot experiments , 2018, PloS one.

[52]  Demis Hassabis,et al.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play , 2018, Science.

[53]  V. Sørdal,et al.  Deep reinforcement learning for quantum Szilard engine optimization , 2019, Physical Review A.

[54]  Dong-Ling Deng,et al.  Machine Learning Detection of Bell Nonlocality in Quantum Many-Body Systems. , 2017, Physical review letters.

[55]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[56]  Hartmut Neven,et al.  Universal quantum control through deep reinforcement learning , 2019 .

[57]  Michael A. Osborne,et al.  Efficiently measuring a quantum device using machine learning , 2018, npj Quantum Information.

[58]  J. P. Dehollain,et al.  An addressable quantum dot qubit with fault-tolerant control-fidelity. , 2014, Nature nanotechnology.

[59]  Mengdi Wang,et al.  Model-Based Reinforcement Learning with Value-Targeted Regression , 2020, L4DC.

[60]  Florian Marquardt,et al.  Reinforcement Learning with Neural Networks for Quantum Feedback , 2018, Physical Review X.

[61]  Zheng An,et al.  Deep reinforcement learning for quantum gate control , 2019, EPL (Europhysics Letters).

[62]  Pascal Poupart,et al.  Bayesian Reinforcement Learning , 2010, Encyclopedia of Machine Learning.

[63]  Lu-Ming Duan,et al.  Efficient representation of quantum many-body states with deep neural networks , 2017, Nature Communications.

[64]  Kamyar Azizzadenesheli,et al.  Efficient Exploration Through Bayesian Deep Q-Networks , 2018, 2018 Information Theory and Applications Workshop (ITA).

[65]  Alpha Lee,et al.  Automatic virtual voltage extraction of a 2xN array of quantum dots with machine learning. , 2020, 2012.03685.

[66]  Akash Sengupta,et al.  Using Reinforcement Learning to find Efficient Qubit Routing Policies for Deployment in Near-term Quantum Computers , 2018, 1812.11619.

[67]  Pieter Abbeel,et al.  Benchmarking Model-Based Reinforcement Learning , 2019, ArXiv.

[68]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.