> On Fri, 10 Jan 2014 03:36:58 PM reza azimi wrote: snip >> Our workload are mainly MPI based benchmarks and we want to test some >> hadoop benchmarks in future. > > According to a presentation I saw at the Slurm User Group back in > September > Intel are working on Hadoop support in Slurm in such a way that you will > not > need a modified Hadoop stack. Not sure when that code will land though. >
Allow me to provide a little insight into the Hadoop scene. Think of Hadoop V1 as a monolithic MapReduce (MR) engine, think of Hadoop V2 (released Oct '13) as a scheduler with application frameworks, one of which is a Hadoop-1 compatible MR framework. That is, Hadoop is moving beyond just MR. Graph processing is one example and MPI applications can "run" under Hadoop-2 (scaling and efficiency is another issue). There are a slew of other application frameworks that are being developed to work within the Hadoop big data environment. The resource manager in Hadoop-2 is called YARN (Yet Another Resource Negotiator) and is different than the traditional HPC resource manager. It offers many services that help to create big data applications. The basic Hadoop Install now has three major components (all present in the distribution tar ball): HDFS (Hadoop Distributed File System, YARN, and the MP framework. As Hadoop-2 begins to gain traction, I think it is important to remember that Hadoop no longer means MR (H!=MR). There is a lot more to this developing story, but I believe, Hadoop-2 will allow some HPC type applications to run on Hadoop clusters. Efforts such as SLUM integration of MR libraries will bring MR to HPC clusters, but it all depends where the data live. -- Doug > Hope this is useful! > > Happy new year all! :-) > > Chris > -- > Christopher Samuel Senior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 > http://www.vlsci.org.au/ http://twitter.com/vlsci > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -- > Mailscanner: Clean > -- Doug -- Mailscanner: Clean _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
