[StarCluster] New Grid Engine Hadoop Integration HOWTO

[Paul McDonagh]

Thanks Rayson,

This and the previous email are a couple of really good suggestions. I'll try 'em out and see what happens.

Best,
Paul.

On Jun 1, 2012, at 14:52, Rayson Ho wrote:

> If you are running Hadoop on StarCluster, you may also be interested
> in this new method contributed by Prakashan Korambath of UCLA.
> 
> http://gridscheduler.sourceforge.net/howto/GridEngineHadoop.html
> 
> The difference between the original SGE 6.2u5 method vs the new one is
> that with Prakashan's approach, Grid Engine is used for resource
> allocation, and the Hadoop job scheduler/Job Tracker is used to handle
> all the MapReduce operations. A Hadoop cluster is created on demand
> with Prakashan's approach, but in the original SGE 6.2u5 method Grid
> Engine replaces the Hadoop job scheduler.
> 
> As standard Grid Engine PEs are used in this new approach, one can
> call "qrsh -inherit" and use Grid Engine's method to start Hadoop
> services on remote nodes, and thus get full job control, job
> accounting, and cleanup at terminate benefits like any other tight PE
> jobs!
> 
> Rayson
> 
> ================================
> Open Grid Scheduler / Grid Engine
> http://gridscheduler.sourceforge.net/
> 
> Scalable Grid Engine Support Program
> http://www.scalablelogic.com/
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster