[StarCluster] plugin shell help
Dan Tenenbaum
dtenenba at fhcrc.org
Sat Oct 30 15:40:57 EDT 2010
OK, so I've gone back to vanilla StarCluster with the patch Marc suggests
above (adding an argument to the call to get_cluster() in run_plugins().
Then I get the weird:
AttributeError: 'str' object has no attribute 'get'
I have put some print statements in to try and clarify things and here is
the output:
before load_plugins, plugins is
{'shortreadplugin': {'setup_class': 'bioconductor.ShortReadPlugin', 'arg1':
'foo', '__name__': 'shortreadplugin'}}
plugin:
shortreadplugin
So this tells me that I am passing a hash to load_plugins(), but
load_plugins() is receiving a string! Why? Is this some python thing?
Sorry for spamming the list so much with this but I'm sort of under the gun
on this project....
Thanks
Dan
On Sat, Oct 30, 2010 at 12:16 PM, Dan Tenenbaum <dtenenba at fhcrc.org> wrote:
> Something else is going on....the shell doesn't seem to recognize any of
> the methods in "cluster" as belonging to ClusterManager. I'm not enough of a
> python developer to know why this might be.
> Dan
>
>
>
> On Sat, Oct 30, 2010 at 12:05 PM, Marc Resnick <mresnick at mit.edu> wrote:
>
>> Got me stumped. If I remember correctly, that function worked for me
>> the last time I tried it...
>>
>> On Sat, Oct 30, 2010 at 2:57 PM, Dan Tenenbaum <dtenenba at fhcrc.org>
>> wrote:
>> > Looks like there are a number of files in this fork that don't exist in
>> the
>> > 0.91 that I downloaded.
>> > So just downloaded your whole fork and built that, then got this in the
>> > shell:
>> > In [1]: cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
>> >
>> ---------------------------------------------------------------------------
>> > AttributeError Traceback (most recent call
>> last)
>> >
>> /Library/Python/2.6/site-packages/StarCluster-0.9999-py2.6.egg/starcluster/commands/shell.pyc
>> > in <module>()
>> > ----> 1
>> > 2
>> > 3
>> > 4
>> > 5
>> > AttributeError: 'module' object has no attribute 'run_plugin'
>> > In [2]:
>> > Dan
>> >
>> > On Sat, Oct 30, 2010 at 11:45 AM, Dan Tenenbaum <dtenenba at fhcrc.org>
>> wrote:
>> >>
>> >> Keeping this on the list.
>> >> BTW, the new cluster.py has no syntax errors, so something else must be
>> >> happening.
>> >> Dan
>> >>
>> >> On Sat, Oct 30, 2010 at 11:44 AM, Dan Tenenbaum <dtenenba at fhcrc.org>
>> >> wrote:
>> >>>
>> >>> Thanks. That gave me a different error, below.
>> >>> The first time this happened, I did:
>> >>> sudo easy_install iptools, but it is still happening.
>> >>> Dan
>> >>> $ starcluster shell
>> >>> Leopard libedit detected.
>> >>> ERROR: An unexpected error occurred while tokenizing input
>> >>> The following traceback may be corrupted or invalid
>> >>> The error message is: ('EOF in multi-line statement', (405, 0))
>> >>>
>> >>>
>> ---------------------------------------------------------------------------
>> >>> ImportError Traceback (most recent call
>> >>> last)
>> >>>
>> >>>
>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster
>> >>> in <module>()
>> >>> 3 __requires__ = 'StarCluster==0.91'
>> >>> 4 import pkg_resources
>> >>> ----> 5 pkg_resources.run_script('StarCluster==0.91', 'starcluster')
>> >>> 6
>> >>> 7
>> >>>
>> >>>
>> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc
>> >>> in run_script(self, requires, script_name)
>> >>> 440 ns.clear()
>> >>> 441 ns['__name__'] = name
>> >>> --> 442 self.require(requires)[0].run_script(script_name, ns)
>> >>> 443
>> >>> 444
>> >>>
>> >>>
>> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc
>> >>> in run_script(self, script_name, namespace)
>> >>> 1165 )
>> >>> 1166 script_code =
>> >>> compile(script_text,script_filename,'exec')
>> >>> -> 1167 exec script_code in namespace, namespace
>> >>> 1168
>> >>> 1169 def _has(self, path):
>> >>>
>> >>>
>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster
>> >>> in <module>()
>> >>> 1
>> >>> ----> 2 #!/usr/bin/python
>> >>> 3 # EASY-INSTALL-SCRIPT: 'StarCluster==0.91','starcluster'
>> >>> 4 __requires__ = 'StarCluster==0.91'
>> >>> 5 import pkg_resources
>> >>> 6 pkg_resources.run_script('StarCluster==0.91', 'starcluster')
>> >>>
>> >>>
>> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cli.py
>> >>> in <module>()
>> >>> 36
>> >>> 37
>> >>> ---> 38
>> >>> 39
>> >>> 40
>> >>>
>> >>>
>> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cluster.py
>> >>> in <module>()
>> >>> 10
>> >>> 11
>> >>> ---> 12
>> >>> 13
>> >>> 14
>> >>> ImportError: cannot import name iptools
>> >>>
>> >>> On Sat, Oct 30, 2010 at 11:38 AM, Marc Resnick <mresnick at mit.edu>
>> wrote:
>> >>>>
>> >>>> The version here
>> >>>>
>> >>>>
>> http://github.com/jtriley/StarCluster/blob/experimental/starcluster/cluster.py
>> >>>> may have those bugs fixed, as I remember that one as well.
>> >>>>
>> >>>>
>> >>>> Marc
>> >>>>
>> >>>> On Sat, Oct 30, 2010 at 2:31 PM, Dan Tenenbaum <dtenenba at fhcrc.org>
>> >>>> wrote:
>> >>>> > I already did that. ;)
>> >>>> > Now I'm getting another error:
>> >>>> > In [1]: cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
>> >>>> >>>> Using private key /Users/dante/.ec2/bioc-keypair.pem.rsa (rsa)
>> >>>> >
>> >>>> >
>> ---------------------------------------------------------------------------
>> >>>> > AttributeError Traceback (most recent
>> call
>> >>>> > last)
>> >>>> >
>> >>>> >
>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc
>> >>>> > in <module>()
>> >>>> > ----> 1
>> >>>> > 2
>> >>>> > 3
>> >>>> > 4
>> >>>> > 5
>> >>>> >
>> >>>> >
>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
>> >>>> > in run_plugin(plugin_name, cluster_tag, cfg)
>> >>>> > 175 plugins = {}
>> >>>> > 176 plugins[plugin_name] = plug
>> >>>> > --> 177 plugins = cl.load_plugins(plugins)
>> >>>> > 178 master = cl.master_node
>> >>>> > 179 for p in plugins:
>> >>>> >
>> >>>> >
>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
>> >>>> > in load_plugins(self, plugins)
>> >>>> > 314 plugs = []
>> >>>> > 315 for plugin in plugins:
>> >>>> > --> 316 setup_class = plugin.get('setup_class')
>> >>>> > 317 plugin_name = plugin.get('__name__')
>> >>>> > 318 mod_name =
>> '.'.join(setup_class.split('.')[:-1])
>> >>>> > AttributeError: 'str' object has no attribute 'get'
>> >>>> > Hmm....
>> >>>> > Dan
>> >>>> >
>> >>>> > On Sat, Oct 30, 2010 at 11:27 AM, Marc Resnick <mresnick at mit.edu>
>> >>>> > wrote:
>> >>>> >>
>> >>>> >> Yes, that's the bug I told Justin about this Summer... I think I
>> >>>> >> patched it, but he may have forgotten to pull from my fork.
>> >>>> >>
>> >>>> >> You can fix it by editing the run_plugin method, and replacing
>> >>>> >>
>> >>>> >> cl = get_cluster(cluster_tag)
>> >>>> >>
>> >>>> >> with
>> >>>> >>
>> >>>> >> cl = get_cluster(cluster_tag, cfg)
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> Marc
>> >>>> >>
>> >>>> >>
>> >>>> >> On Sat, Oct 30, 2010 at 2:20 PM, Dan Tenenbaum <
>> dtenenba at fhcrc.org>
>> >>>> >> wrote:
>> >>>> >> > Hi Marc,
>> >>>> >> > I am running 0.91.
>> >>>> >> > I was doing a couple things wrong. I figured them out but I am
>> >>>> >> > still
>> >>>> >> > having
>> >>>> >> > this problem.
>> >>>> >> > One thing I did wrong was have the
>> >>>> >> > plugins=shortreadplugin
>> >>>> >> > line under the global section; it needed to be under the
>> >>>> >> > [cluster smallcluster]
>> >>>> >> > section.
>> >>>> >> > When I start the cluster now, I can see the output from my
>> plugin:
>> >>>> >> >>>> Running plugin shortreadplugin
>> >>>> >> >>>> about to echo
>> >>>> >> > But, when I run the shell and put in this command:
>> >>>> >> > cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
>> >>>> >> > I get the same error described below.
>> >>>> >> > Dan
>> >>>> >> >
>> >>>> >> > On Sat, Oct 30, 2010 at 11:11 AM, Marc Resnick <
>> mresnick at mit.edu>
>> >>>> >> > wrote:
>> >>>> >> >>
>> >>>> >> >> Hey Dan,
>> >>>> >> >>
>> >>>> >> >> Are you running the latest version? I saw that bug a few months
>> >>>> >> >> ago, I
>> >>>> >> >> thought I remember Justin fixing it.
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> Marc
>> >>>> >> >>
>> >>>> >> >> On Sat, Oct 30, 2010 at 2:00 PM, Dan Tenenbaum
>> >>>> >> >> <dtenenba at fhcrc.org>
>> >>>> >> >> wrote:
>> >>>> >> >> > Hi,
>> >>>> >> >> > I am not much of a python programmer and I'm trying to figure
>> >>>> >> >> > out the
>> >>>> >> >> > plugin
>> >>>> >> >> > system.
>> >>>> >> >> > I created a ~/.starcluster/plugins directory and put a file
>> in
>> >>>> >> >> > it
>> >>>> >> >> > called
>> >>>> >> >> > shortread.py.
>> >>>> >> >> > Here are its contents:
>> >>>> >> >> >
>> >>>> >> >> > from starcluster.clustersetup import ClusterSetup
>> >>>> >> >> > from starcluster.logger import log
>> >>>> >> >> > class ShortReadPlugin(ClusterSetup):
>> >>>> >> >> > def __init__(self, arg1):
>> >>>> >> >> > self.arg1 = arg1
>> >>>> >> >> > log.info('arg1 = %s' % arg1)
>> >>>> >> >> > def run(self, nodes, master, user, user_shell, volumes):
>> >>>> >> >> > log.info("about to echo")
>> >>>> >> >> > master.ssh.execute("echo 'hey'")
>> >>>> >> >> > As you can see, it's just a test.
>> >>>> >> >> > In my config, I put this at the bottom:
>> >>>> >> >> > [plugin shortreadplugin]
>> >>>> >> >> > SETUP_CLASS = ShortReadPlugin
>> >>>> >> >> > arg1=foo
>> >>>> >> >> > And above in the [global] section I put:
>> >>>> >> >> > plugins=shortreadplugin
>> >>>> >> >> > I'm a bit confused about whether this is right because in
>> your
>> >>>> >> >> > example:
>> >>>> >> >> > http://web.mit.edu/stardev/cluster/docs/plugins.html
>> >>>> >> >> > ...you have the SETUP_CLASS as ubuntu.PackageInstaller. I'm
>> not
>> >>>> >> >> > sure
>> >>>> >> >> > where
>> >>>> >> >> > the "ubuntu" fits into this. Is it like java where I need to
>> >>>> >> >> > have a
>> >>>> >> >> > folder
>> >>>> >> >> > called ubuntu where the class resides?
>> >>>> >> >> > Anyway, I fired up my cluster and went to test this with
>> >>>> >> >> > "starcluster
>> >>>> >> >> > shell"
>> >>>> >> >> > and got the following output:
>> >>>> >> >> > In [5]: cluster.run_plugin('shortreadplugin',
>> 'myfirstcluster',
>> >>>> >> >> > cfg)
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> ---------------------------------------------------------------------------
>> >>>> >> >> > TypeError Traceback (most
>> recent
>> >>>> >> >> > call
>> >>>> >> >> > last)
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc
>> >>>> >> >> > in <module>()
>> >>>> >> >> > ----> 1
>> >>>> >> >> > 2
>> >>>> >> >> > 3
>> >>>> >> >> > 4
>> >>>> >> >> > 5
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
>> >>>> >> >> > in run_plugin(plugin_name, cluster_tag, cfg)
>> >>>> >> >> > 170 def run_plugin(plugin_name, cluster_tag, cfg):
>> >>>> >> >> > 171 ec2 = cfg.get_easy_ec2()
>> >>>> >> >> > --> 172 cl = get_cluster(cluster_tag)
>> >>>> >> >> > 173 cl.load_receipt()
>> >>>> >> >> > 174 plug = cfg.get_plugin(plugin_name)
>> >>>> >> >> > TypeError: get_cluster() takes exactly 2 arguments (1 given)
>> >>>> >> >> >
>> >>>> >> >> > It's hard for me to understand what's happening here. I of
>> >>>> >> >> > course am
>> >>>> >> >> > not
>> >>>> >> >> > calling get_cluster directly. Is there a bug in run_plugin()
>> or
>> >>>> >> >> > is
>> >>>> >> >> > there
>> >>>> >> >> > something wrong in my setup, or both?
>> >>>> >> >> > Thanks
>> >>>> >> >> > Dan
>> >>>> >> >> >
>> >>>> >> >> > _______________________________________________
>> >>>> >> >> > StarCluster mailing list
>> >>>> >> >> > StarCluster at mit.edu
>> >>>> >> >> > http://mailman.mit.edu/mailman/listinfo/starcluster
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >>
>> >>>> >> >
>> >>>> >> >
>> >>>> >>
>> >>>> >
>> >>>> >
>> >>>>
>> >>>
>> >>
>> >
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20101030/5a1f81da/attachment.htm
More information about the StarCluster
mailing list