[StarCluster] Windows AMIs as cluster nodes?

[Jacob Barhak]

Hi Giovanni,

You are not the only windows user. I myself use windows for development
environment - yet my code runs on multiple platforms.

I am far from being an expert in the internals of StarCluster, yet from my
limited experience it seems that SGE and its configuration to run on
windows environment will be your biggest obstacle since this is how the
cluster nodes communicate jobs. Also there are some Linux specific
references that may need a fix such as mounting NFS - this is at the
StarCluster or plugin code. And you should add lots of testing to this
project - it does not seem trivial.

Other experts on the list here may confirm this, yet this does not seem to
be a small project. Perhaps you should consider converting your code to be
mutli-platform and run on Linux environment - it may take less time and
effort - especially if you are using a portable language such as Python.

I know that there are some efforts from the Azure cloud - perhaps these
efforts will mature at some point to provide a StarCluster equivalent for
Windows - yet for now it seems StarCluster on the Amazon cloud in the
dominant option in the market if you want to create a virtual cluster - at
least I know of no other competitors that are close enough with similar
features.

Unless you have special constraints and lots of resources to back you I
suggest using Linux AMIs - you can still keep a windows front end to run
StarCluster for Launch and for development.

I hope I am presenting a good picture here.

                Jacob



On Mon, Dec 9, 2013 at 2:42 PM, Giovanni <pip.giovanni at gmail.com> wrote:

> Hello,
>
> I have recently started looking at AWS for our parallel/distributed
> processing needs and came across Starcluster and would very much like to
> use it. However, we are a Windows shop and do our processing on Windows
> machines.
>
> I've installed Starcluster on Win 7 and have run the demo smoothly.
>
> The big question now - is it feasible to create a Windows AMI that
> Starcluster can use and build a cluster with? And if so, what does the
> Windows AMI need? I've gotten as far as creating a Windows instance with
> passwordless SSH server and the "root" user and I can get SC to connect to
> it. My understanding is that I now have to configure Sun Grid Engine
> queuing system (open-source edition), the network file system, and OpenMPI.
> Anything else?
>
> Does anyone have any experience with this? Is it doable?
>
> Giovanni
>
> _______________________________________________
> StarCluster mailing list
> StarCluster at mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20131209/a3d03a26/attachment.htm