[StarCluster] plugin shell help
Justin Riley
jtriley at MIT.EDU
Sat Oct 30 18:32:36 EDT 2010
Dan,
The next version uses a ClusterManager object rather than the cluster.*
methods. The new ClusterManager is automatically loaded in the shell as
variable 'cm'. So with the latest github code it would be:
$ starcluster shell
$ cm.run_plugin("shortreadplugin", "myfirstcluster")
You can pretty much replace any cluster.* method call with cm.* if
you're using the latest github code.
~Justin
On 10/30/10 2:57 PM, Dan Tenenbaum wrote:
> Looks like there are a number of files in this fork that don't exist
> in the 0.91 that I downloaded.
>
> So just downloaded your whole fork and built that, then got this in
> the shell:
>
> In [1]: cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
> ---------------------------------------------------------------------------
> AttributeError Traceback (most recent call
> last)
>
> /Library/Python/2.6/site-packages/StarCluster-0.9999-py2.6.egg/starcluster/commands/shell.pyc
> in <module>()
> ----> 1
> 2
> 3
> 4
> 5
>
> AttributeError: 'module' object has no attribute 'run_plugin'
>
> In [2]:
>
> Dan
>
>
> On Sat, Oct 30, 2010 at 11:45 AM, Dan Tenenbaum <dtenenba at fhcrc.org
> <mailto:dtenenba at fhcrc.org>> wrote:
>
> Keeping this on the list.
> BTW, the new cluster.py has no syntax errors, so something else
> must be happening.
> Dan
>
>
> On Sat, Oct 30, 2010 at 11:44 AM, Dan Tenenbaum
> <dtenenba at fhcrc.org <mailto:dtenenba at fhcrc.org>> wrote:
>
> Thanks. That gave me a different error, below.
> The first time this happened, I did:
> sudo easy_install iptools, but it is still happening.
>
> Dan
>
> $ starcluster shell
> Leopard libedit detected.
> ERROR: An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (405, 0))
>
> ---------------------------------------------------------------------------
> ImportError Traceback (most
> recent call last)
>
> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster
> in <module>()
> 3 __requires__ = 'StarCluster==0.91'
> 4 import pkg_resources
> ----> 5 pkg_resources.run_script('StarCluster==0.91',
> 'starcluster')
> 6
> 7
>
> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc
> in run_script(self, requires, script_name)
> 440 ns.clear()
> 441 ns['__name__'] = name
> --> 442
> self.require(requires)[0].run_script(script_name, ns)
> 443
> 444
>
> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc
> in run_script(self, script_name, namespace)
> 1165 )
> 1166 script_code =
> compile(script_text,script_filename,'exec')
> -> 1167 exec script_code in namespace, namespace
> 1168
> 1169 def _has(self, path):
>
> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster
> in <module>()
> 1
> ----> 2 #!/usr/bin/python
>
> 3 # EASY-INSTALL-SCRIPT: 'StarCluster==0.91','starcluster'
>
> 4 __requires__ = 'StarCluster==0.91'
> 5 import pkg_resources
> 6 pkg_resources.run_script('StarCluster==0.91',
> 'starcluster')
>
> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cli.py
> in <module>()
> 36
> 37
> ---> 38
> 39
> 40
>
> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cluster.py
> in <module>()
> 10
> 11
> ---> 12
> 13
> 14
>
> ImportError: cannot import name iptools
>
>
> On Sat, Oct 30, 2010 at 11:38 AM, Marc Resnick
> <mresnick at mit.edu <mailto:mresnick at mit.edu>> wrote:
>
> The version here
> http://github.com/jtriley/StarCluster/blob/experimental/starcluster/cluster.py
> may have those bugs fixed, as I remember that one as well.
>
>
> Marc
>
> On Sat, Oct 30, 2010 at 2:31 PM, Dan Tenenbaum
> <dtenenba at fhcrc.org <mailto:dtenenba at fhcrc.org>> wrote:
> > I already did that. ;)
> > Now I'm getting another error:
> > In [1]:
> cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
> >>>> Using private key
> /Users/dante/.ec2/bioc-keypair.pem.rsa (rsa)
> >
> ---------------------------------------------------------------------------
> > AttributeError Traceback
> (most recent call last)
> >
> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc
> > in <module>()
> > ----> 1
> > 2
> > 3
> > 4
> > 5
> >
> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
> > in run_plugin(plugin_name, cluster_tag, cfg)
> > 175 plugins = {}
> > 176 plugins[plugin_name] = plug
> > --> 177 plugins = cl.load_plugins(plugins)
> > 178 master = cl.master_node
> > 179 for p in plugins:
> >
> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
> > in load_plugins(self, plugins)
> > 314 plugs = []
> > 315 for plugin in plugins:
> > --> 316 setup_class = plugin.get('setup_class')
> > 317 plugin_name = plugin.get('__name__')
> > 318 mod_name =
> '.'.join(setup_class.split('.')[:-1])
> > AttributeError: 'str' object has no attribute 'get'
> > Hmm....
> > Dan
> >
> > On Sat, Oct 30, 2010 at 11:27 AM, Marc Resnick
> <mresnick at mit.edu <mailto:mresnick at mit.edu>> wrote:
> >>
> >> Yes, that's the bug I told Justin about this Summer...
> I think I
> >> patched it, but he may have forgotten to pull from my fork.
> >>
> >> You can fix it by editing the run_plugin method, and
> replacing
> >>
> >> cl = get_cluster(cluster_tag)
> >>
> >> with
> >>
> >> cl = get_cluster(cluster_tag, cfg)
> >>
> >>
> >>
> >> Marc
> >>
> >>
> >> On Sat, Oct 30, 2010 at 2:20 PM, Dan Tenenbaum
> <dtenenba at fhcrc.org <mailto:dtenenba at fhcrc.org>> wrote:
> >> > Hi Marc,
> >> > I am running 0.91.
> >> > I was doing a couple things wrong. I figured them out
> but I am still
> >> > having
> >> > this problem.
> >> > One thing I did wrong was have the
> >> > plugins=shortreadplugin
> >> > line under the global section; it needed to be under the
> >> > [cluster smallcluster]
> >> > section.
> >> > When I start the cluster now, I can see the output
> from my plugin:
> >> >>>> Running plugin shortreadplugin
> >> >>>> about to echo
> >> > But, when I run the shell and put in this command:
> >> >
> cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
> >> > I get the same error described below.
> >> > Dan
> >> >
> >> > On Sat, Oct 30, 2010 at 11:11 AM, Marc Resnick
> <mresnick at mit.edu <mailto:mresnick at mit.edu>> wrote:
> >> >>
> >> >> Hey Dan,
> >> >>
> >> >> Are you running the latest version? I saw that bug a
> few months ago, I
> >> >> thought I remember Justin fixing it.
> >> >>
> >> >>
> >> >> Marc
> >> >>
> >> >> On Sat, Oct 30, 2010 at 2:00 PM, Dan Tenenbaum
> <dtenenba at fhcrc.org <mailto:dtenenba at fhcrc.org>>
> >> >> wrote:
> >> >> > Hi,
> >> >> > I am not much of a python programmer and I'm
> trying to figure out the
> >> >> > plugin
> >> >> > system.
> >> >> > I created a ~/.starcluster/plugins directory and
> put a file in it
> >> >> > called
> >> >> > shortread.py.
> >> >> > Here are its contents:
> >> >> >
> >> >> > from starcluster.clustersetup import ClusterSetup
> >> >> > from starcluster.logger import log
> >> >> > class ShortReadPlugin(ClusterSetup):
> >> >> > def __init__(self, arg1):
> >> >> > self.arg1 = arg1
> >> >> > log.info <http://log.info>('arg1 = %s' % arg1)
> >> >> > def run(self, nodes, master, user,
> user_shell, volumes):
> >> >> > log.info <http://log.info>("about to echo")
> >> >> > master.ssh.execute("echo 'hey'")
> >> >> > As you can see, it's just a test.
> >> >> > In my config, I put this at the bottom:
> >> >> > [plugin shortreadplugin]
> >> >> > SETUP_CLASS = ShortReadPlugin
> >> >> > arg1=foo
> >> >> > And above in the [global] section I put:
> >> >> > plugins=shortreadplugin
> >> >> > I'm a bit confused about whether this is right
> because in your
> >> >> > example:
> >> >> > http://web.mit.edu/stardev/cluster/docs/plugins.html
> >> >> > ...you have the SETUP_CLASS as
> ubuntu.PackageInstaller. I'm not sure
> >> >> > where
> >> >> > the "ubuntu" fits into this. Is it like java where
> I need to have a
> >> >> > folder
> >> >> > called ubuntu where the class resides?
> >> >> > Anyway, I fired up my cluster and went to test
> this with "starcluster
> >> >> > shell"
> >> >> > and got the following output:
> >> >> > In [5]: cluster.run_plugin('shortreadplugin',
> 'myfirstcluster', cfg)
> >> >> >
> >> >> >
> >> >> >
> ---------------------------------------------------------------------------
> >> >> > TypeError
> Traceback (most recent call
> >> >> > last)
> >> >> >
> >> >> >
> >> >> >
> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc
> >> >> > in <module>()
> >> >> > ----> 1
> >> >> > 2
> >> >> > 3
> >> >> > 4
> >> >> > 5
> >> >> >
> >> >> >
> >> >> >
> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
> >> >> > in run_plugin(plugin_name, cluster_tag, cfg)
> >> >> > 170 def run_plugin(plugin_name, cluster_tag, cfg):
> >> >> > 171 ec2 = cfg.get_easy_ec2()
> >> >> > --> 172 cl = get_cluster(cluster_tag)
> >> >> > 173 cl.load_receipt()
> >> >> > 174 plug = cfg.get_plugin(plugin_name)
> >> >> > TypeError: get_cluster() takes exactly 2 arguments
> (1 given)
> >> >> >
> >> >> > It's hard for me to understand what's happening
> here. I of course am
> >> >> > not
> >> >> > calling get_cluster directly. Is there a bug in
> run_plugin() or is
> >> >> > there
> >> >> > something wrong in my setup, or both?
> >> >> > Thanks
> >> >> > Dan
> >> >> >
> >> >> > _______________________________________________
> >> >> > StarCluster mailing list
> >> >> > StarCluster at mit.edu <mailto:StarCluster at mit.edu>
> >> >> > http://mailman.mit.edu/mailman/listinfo/starcluster
> >> >> >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20101030/a30cc693/attachment.htm
More information about the StarCluster
mailing list