Increasing performance of parallel and distributed systems in high performance computing using weight based approach

High performance computing (HPC), large scale instruments and continuously increasing simulation tools are generating data at a huge rate that are difficult to be effectively managed and analyzed. Implementation of MapReduce model provides a way for processing huge volumes of data through the use of large number of commodity computers. MapReduce and Hadoop have been initially used for processing web data. But recently they have been used for processing more complex scientific applications. The proposed system helps to understand the impact of file system, network and programming modes on performance. The performance an application can obtain is largely work load dependent. Design of every MapReduce system has to include the Key factors like High speed, Quick Response, Accurate result. The proposed work is to improve the scheduling and management functionality of Parallel and Distributed Computing. The proposed technique Weight based Approach improves the performance by improving job scheduling strategy.

[1]  Jun Wang,et al.  Supporting HPC Analytics Applications with Access Patterns Using Data Restructuring and Data-Centric Scheduling Techniques in MapReduce , 2013, IEEE Transactions on Parallel and Distributed Systems.

[2]  Lavanya Ramakrishnan,et al.  MARIANE: Using MApReduce in HPC environments , 2014, Future Gener. Comput. Syst..

[3]  Madhusudhan Govindaraju,et al.  MARLA: MapReduce for Heterogeneous Clusters , 2012, 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012).

[4]  Matthew T. O'Keefe,et al.  The Global File System: A File System for Shared Disk Storage , 1997 .

[5]  Madhusudhan Govindaraju,et al.  DELMA: Dynamically ELastic MapReduce Framework for CPU-Intensive Applications , 2011, 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[6]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[7]  Lavanya Ramakrishnan,et al.  Evaluating Hadoop for Data-Intensive Scientific Operations , 2012, 2012 IEEE Fifth International Conference on Cloud Computing.

[8]  Madhusudhan Govindaraju,et al.  LEMO-MR: Low Overhead and Elastic MapReduce Implementation Optimized for Memory and CPU-Intensive Applications , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.