Distributed Metaserver Mechanism and Recovery Mechanism Support in Quantcast File System

With the need of data storage increases tremendously nowadays, distributed file system becomes the most important data storage system in cloud computing. In distributed file system development, there are many researchers work hard to refine the architecture to provide scalability and reliability. In our work, we propose a distributed metaserver system including metaserver scale-out, metadata replication, metaserver recovery, and metaserver management recovery mechanisms. In our experiments, the proposed system can increase the capacity of metadata and increase the reliability by fault tolerance mechanism. The overhead of read/write data is very little in the proposed system as well.

[1]  Sriram Rao,et al.  A The Quantcast File System , 2013, Proc. VLDB Endow..

[2]  Andrew J. Hutton,et al.  Lustre: Building a File System for 1,000-node Clusters , 2003 .

[3]  Cheng Huang,et al.  Erasure Coding in Windows Azure Storage , 2012, USENIX Annual Technical Conference.

[4]  S.Suganthi,et al.  Cassandra-A Decentralized Structured Storage System , 2017 .

[5]  Jincai Chen,et al.  A Dynamic Metadata Equipotent Subtree Partition Policy for Mass Storage System , 2007, 2007 Japan-China Joint Workshop on Frontier of Computer Science and Technology (FCST 2007).

[6]  Samuel Lang,et al.  GIGA+: scalable directories for shared file systems , 2007, PDSW '07.

[7]  Kai Ren,et al.  Giga + TableFS on PanFS : Scaling Metadata Performance on Cluster File Systems , 2013 .

[8]  Carlos Maltzahn,et al.  Ceph: a scalable, high-performance distributed file system , 2006, OSDI '06.

[9]  Dimitris S. Papailiopoulos,et al.  Simple regenerating codes: Network coding for cloud storage , 2011, 2012 Proceedings IEEE INFOCOM.

[10]  F. Moore,et al.  Polynomial Codes Over Certain Finite Fields , 2017 .

[11]  Kai Ren,et al.  TABLEFS: Enhancing Metadata Efficiency in the Local File System , 2013, USENIX Annual Technical Conference.

[12]  Prashant Malik,et al.  Cassandra: a decentralized structured storage system , 2010, OPSR.

[13]  Yueming Lu,et al.  A Zones-Based Metadata Management Method for Distributed File System , 2013, ISCTCS.

[14]  Kannan Ramchandran,et al.  A Solution to the Network Challenges of Data Recovery in Erasure-coded Distributed Storage Systems: A Study on the Facebook Warehouse Cluster , 2013, HotStorage.

[15]  Hairong Kuang,et al.  The Hadoop Distributed File System , 2010, 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST).

[16]  John Kubiatowicz,et al.  Erasure Coding Vs. Replication: A Quantitative Comparison , 2002, IPTPS.

[17]  Gregory R. Ganger,et al.  Object-based storage , 2003, IEEE Commun. Mag..

[18]  GhemawatSanjay,et al.  The Google file system , 2003 .