IKAROS: An HTTP-Based Distributed File System, for Low Consumption & Low Specification Devices

We present the design of IKAROS: an HTTP-based distributed file system, which provides file access scalability and targets a large variety of operating systems and storage systems. IKAROS bypasses the server bottleneck enabling clients to access storage directly, while supporting the usage of multiple types of meta-data. It enables low-consumption, low-specification and low-cost devices to achieve a high throughput data transfer, responding to highly demanding applications. We present data transfer results comparing IKAROS, NFS, PVFS2 and HDFS on a Small Office/Home Office Network Attached Storage infrastructure. We show that IKAROS architecture satisfies and outperforms the data rate demands of high performance applications. We also present experimental results which compare IKAROS and GridFTP using the European Grid Infrastructure. IKAROS performs better in most cases while being competitive at the rest.

[1]  Dimitrios Katsaros,et al.  Architectural Requirements for Cloud Computing Systems: An Enterprise Cloud Approach , 2011, Journal of Grid Computing.

[2]  Tao Yang,et al.  The Panasas ActiveScale Storage Cluster - Delivering Scalable High Bandwidth Storage , 2004, Proceedings of the ACM/IEEE SC2004 Conference.

[3]  Peter J. Braam,et al.  The Coda Distributed File System , 1998 .

[4]  Eugenio Cesario,et al.  XtreemFS: a case for object-based storage in Grid data management , 2007 .

[5]  Matei Ripeanu,et al.  The case for a versatile storage system , 2010, OPSR.

[6]  Robert B. Ross,et al.  On the duality of data-intensive file system design: Reconciling HDFS and PVFS , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[7]  Liang Hu,et al.  Integrating Local Job Scheduler - LSFTM with GfarmTM , 2005, ISPA.

[8]  Parag Agrawal,et al.  The case for RAMClouds: scalable high-performance storage entirely in DRAM , 2010, OPSR.

[9]  Robert L. Grossman,et al.  Distributing the Sloan Digital Sky Survey Using UDT and Sector , 2006, 2006 Second IEEE International Conference on e-Science and Grid Computing (e-Science'06).

[10]  Depei Qian,et al.  MapReduce Workload Modeling with Statistical Approach , 2011, Journal of Grid Computing.

[11]  Flavia Donno,et al.  Static and Dynamic Data Models for the Storage Resource Manager v2.2 , 2009, Journal of Grid Computing.

[12]  Andrew J. Hutton,et al.  Lustre: Building a File System for 1,000-node Clusters , 2003 .

[13]  W. Allcock,et al.  GridFTP protocol specification , 2002 .

[14]  Federica Fanzago,et al.  High-Energy Physics on the Grid: the ATLAS and CMS Experience , 2008, Journal of Grid Computing.

[15]  A. Rowstron,et al.  Past: persistent and anonymous storage in a peer-to-peer networking environment , 2001 .

[16]  Robert B. Ross,et al.  PVFS: A Parallel File System for Linux Clusters , 2000, Annual Linux Showcase & Conference.

[17]  Zhao Zhang,et al.  Toward loosely coupled programming on petascale systems , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[18]  Osamu Tatebe,et al.  Gfarm v2: A Grid file system that supports high-performance distributed and parallel data computing , 2005 .

[19]  Peter Honeyman,et al.  Exporting storage systems in a scalable manner with pNFS , 2005, 22nd IEEE / 13th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST'05).

[20]  Carlos Maltzahn,et al.  Ceph: a scalable, high-performance distributed file system , 2006, OSDI '06.

[21]  Liang Hu,et al.  Implementing Data Aware Scheduling In Gfarm(R) Using LSF(TM) Scheduler plugin Mechanism , 2005, GCA.

[22]  GhemawatSanjay,et al.  The Google file system , 2003 .

[23]  Nikolai Joukov,et al.  GreenFS: making enterprise computers greener by protecting them better , 2008, Eurosys '08.

[24]  Yannis Cotronis,et al.  Design and Implementation of the Mobile Grid Resource Management System , 2012, Comput. Sci..

[25]  Lesandro Ponciano,et al.  Assessing Green Strategies in Peer-to-Peer Opportunistic Grids , 2012, Journal of Grid Computing.

[26]  Y. Wu,et al.  PhEDEx high-throughput data transfer management system , 2006 .

[27]  Moreno Marzolla,et al.  Design and implementation of the gLite CREAM job management service , 2010, Future Gener. Comput. Syst..

[28]  Ian T. Foster,et al.  Making a case for distributed file systems at Exascale , 2011, LSAP '11.

[29]  Thomas W. Doeppner Distributed file systems and distributed memory , 1996, CSUR.

[30]  Zhao Zhang,et al.  Towards Loo on , 2008 .

[31]  Frank B. Schmuck,et al.  GPFS: A Shared-Disk File System for Large Computing Clusters , 2002, FAST.

[32]  Mike Loukides,et al.  Managing NFS and NIS , 1991 .

[33]  Antony I. T. Rowstron,et al.  PAST: a large-scale, persistent peer-to-peer storage utility , 2001, Proceedings Eighth Workshop on Hot Topics in Operating Systems.

[34]  Douglas Thain,et al.  Chirp: a practical global filesystem for cluster and Grid computing , 2008, Journal of Grid Computing.

[35]  Peter Honeyman,et al.  Direct-pNFS: scalable, transparent, and versatile access to parallel file systems , 2007, HPDC '07.