Distributed File Information Management System Based on Hadoop

There are two main problems to store the system data on single machine: limited storage space and low reliability. The concept of distribution solves the two problems fundamentally. Many independent machines are integrated as a whole. As a result, these separated resources are integrated together. This paper focuses on developing a system, based on SSH, XFire and Hadoop, to help users store and manage the distributed files. All the files stored in HDFS should be encrypted to protect users privacy. In order to save resources, system is designed to avoid uploading the duplicate files by checking the files MD5 string.