Seamless Utilization of Heterogeneous XSede Resources to Accelerate Processing of a High Value Functional Neuroimaging Dataset

We describe the technical effort used to process a voluminous high value human neuroimaging dataset on the Open Science Grid with opportunistic use of idle HPC resources to boost computing capacity more than 5-fold. With minimal software development effort and no discernable competitive interference with other HPC users, this effort delivered 15,000,000 core hours over 7 months. The CamCAN Lifespan Neuroimaging Dataset [1,2], Cambridge (UK) Centre for Ageing and Neuroscience, was acquired and processed beginning in December 2016. The referee consensus solver deployed to the Open Science Grid [3,4] was used for this task. The dataset includes demographic and screening measures, a high-resolution MRI scan of the brain, and whole-head magnetoencephalographic (MEG) recordings during eyes closed rest (560 sec), a simple task (540 sec), and passive listening/viewing (140 sec). The data were collected from 619 neurologically normal individuals, ages 18-87. The processed results from both resting and task recordings are completed and available for download at http://stash.osgconnect.net/+krieger/. These constitute ≈3.0 TBytes of data including the location within the brain (1 mm resolution), time stamp (1 msec resolution), and 80 msec time course for each of 3.7 billion validated neuroelectric events, i.e. mean 6.1 million events for each of the 619 participants. Primary processing utilized the referee consensus solver deployed on the Open Science Grid (OSG). The pool of computing elements used by the solver was boosted using XSede allocations on Bridges (Pittsburgh Supercomputing Center) and Comet (San Diego Supercomputing Center). Computing elements were added directly to the OSG's HTcondor pool. This required no changes to the solver and only minor changes to the workflow management software which was fully operational in two days. HTcondor's glidein mechanism was used to add the booster computing elements to the HTcondor pool. A script was deployed on each machine which queues Glidein's only when there are idle computing elements. Each Glidein's lifetime is limited to two hours, insuring that these jobs do not significantly delay other users of these important supercomputing resources. This opportunistic use of Brides and Comet delivered more than a 3-fold increase in the number of core hours available for this effort.