A virtual filesystem approach to storage, analysis and delivery of volumetric image data for connectomics

Biology and medicine are increasingly driven by analyses of 3D and time series imagery for studies that are not possible with 2D images. Structural data required for building spatially realistic cell and connectomics models are particularly demanding of both resolution and spatial extent. Image capture methods for optical and electron microscopy at gigapixel per second rates are now routine. In combination these factors can currently produce hundreds of terabytes per specimen at data densities up to 1 PB per cubic mm of tissue. New techniques are needed to economically handle these speeds and data scales and to distribute results for on-demand analyses by researchers and students nationwide. A virtual volume file system (VVFS) approach to these problems is suggested by trends in the economics of computation and data storage along with typical data access patterns. In recent years improvements in the speed and cost of computation have dramatically outpaced gains in storage cost and performance. This is particularly true in GPGPU computation where data bandwidth is often the limiting factor for overall throughput. The essence of this VVFS mechanism is to apply on-the-fly computation to replace redundant data storage in critical operations such as registration, rendering and automated recognition. This is accomplished using the Linux Filesystem in UserSpace (FUSE) mechanism to provide file compatible interfaces to programs that operate from data files. This interface produces the appropriate content on-demand as applications such as TensorFlow or other analysis systems access the virtual files. The VVFS provides a flexible framework for connecting multiple program units into large scale applications while also reducing redundant data storage. By moving computation directly into the access path it minimizes data traffic while processing only those parts of the virtual data which end user applications consume.