OK, so I've gone back to vanilla StarCluster with the patch Marc suggests above (adding an argument to the call to get_cluster() in run_plugins().<div><br></div><div>Then I get the weird:</div><div>AttributeError: 'str' object has no attribute 'get'</div>
<div><br></div><div>I have put some print statements in to try and clarify things and here is the output:</div><div><br></div><div><div>before load_plugins, plugins is </div><div>{'shortreadplugin': {'setup_class': 'bioconductor.ShortReadPlugin', 'arg1': 'foo', '__name__': 'shortreadplugin'}}</div>
<div>plugin:</div><div>shortreadplugin</div></div><div><br></div><div>So this tells me that I am passing a hash to load_plugins(), but load_plugins() is receiving a string! Why? Is this some python thing?</div><div><br></div>
<div>Sorry for spamming the list so much with this but I'm sort of under the gun on this project....</div><div><br></div><div>Thanks</div><div>Dan</div><div><br></div><div><br><div class="gmail_quote">On Sat, Oct 30, 2010 at 12:16 PM, Dan Tenenbaum <span dir="ltr"><<a href="mailto:dtenenba@fhcrc.org">dtenenba@fhcrc.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Something else is going on....the shell doesn't seem to recognize any of the methods in "cluster" as belonging to ClusterManager. I'm not enough of a python developer to know why this might be.<div>
Dan</div><div><div></div><div class="h5">
<div><br><div><br><br><div class="gmail_quote">On Sat, Oct 30, 2010 at 12:05 PM, Marc Resnick <span dir="ltr"><<a href="mailto:mresnick@mit.edu" target="_blank">mresnick@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Got me stumped. If I remember correctly, that function worked for me<br>
the last time I tried it...<br>
<div><div></div><div><br>
On Sat, Oct 30, 2010 at 2:57 PM, Dan Tenenbaum <<a href="mailto:dtenenba@fhcrc.org" target="_blank">dtenenba@fhcrc.org</a>> wrote:<br>
> Looks like there are a number of files in this fork that don't exist in the<br>
> 0.91 that I downloaded.<br>
> So just downloaded your whole fork and built that, then got this in the<br>
> shell:<br>
> In [1]: cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)<br>
> ---------------------------------------------------------------------------<br>
> AttributeError Traceback (most recent call last)<br>
> /Library/Python/2.6/site-packages/StarCluster-0.9999-py2.6.egg/starcluster/commands/shell.pyc<br>
> in <module>()<br>
> ----> 1<br>
> 2<br>
> 3<br>
> 4<br>
> 5<br>
> AttributeError: 'module' object has no attribute 'run_plugin'<br>
> In [2]:<br>
> Dan<br>
><br>
> On Sat, Oct 30, 2010 at 11:45 AM, Dan Tenenbaum <<a href="mailto:dtenenba@fhcrc.org" target="_blank">dtenenba@fhcrc.org</a>> wrote:<br>
>><br>
>> Keeping this on the list.<br>
>> BTW, the new cluster.py has no syntax errors, so something else must be<br>
>> happening.<br>
>> Dan<br>
>><br>
>> On Sat, Oct 30, 2010 at 11:44 AM, Dan Tenenbaum <<a href="mailto:dtenenba@fhcrc.org" target="_blank">dtenenba@fhcrc.org</a>><br>
>> wrote:<br>
>>><br>
>>> Thanks. That gave me a different error, below.<br>
>>> The first time this happened, I did:<br>
>>> sudo easy_install iptools, but it is still happening.<br>
>>> Dan<br>
>>> $ starcluster shell<br>
>>> Leopard libedit detected.<br>
>>> ERROR: An unexpected error occurred while tokenizing input<br>
>>> The following traceback may be corrupted or invalid<br>
>>> The error message is: ('EOF in multi-line statement', (405, 0))<br>
>>><br>
>>> ---------------------------------------------------------------------------<br>
>>> ImportError Traceback (most recent call<br>
>>> last)<br>
>>><br>
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster<br>
>>> in <module>()<br>
>>> 3 __requires__ = 'StarCluster==0.91'<br>
>>> 4 import pkg_resources<br>
>>> ----> 5 pkg_resources.run_script('StarCluster==0.91', 'starcluster')<br>
>>> 6<br>
>>> 7<br>
>>><br>
>>> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc<br>
>>> in run_script(self, requires, script_name)<br>
>>> 440 ns.clear()<br>
>>> 441 ns['__name__'] = name<br>
>>> --> 442 self.require(requires)[0].run_script(script_name, ns)<br>
>>> 443<br>
>>> 444<br>
>>><br>
>>> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc<br>
>>> in run_script(self, script_name, namespace)<br>
>>> 1165 )<br>
>>> 1166 script_code =<br>
>>> compile(script_text,script_filename,'exec')<br>
>>> -> 1167 exec script_code in namespace, namespace<br>
>>> 1168<br>
>>> 1169 def _has(self, path):<br>
>>><br>
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster<br>
>>> in <module>()<br>
>>> 1<br>
>>> ----> 2 #!/usr/bin/python<br>
>>> 3 # EASY-INSTALL-SCRIPT: 'StarCluster==0.91','starcluster'<br>
>>> 4 __requires__ = 'StarCluster==0.91'<br>
>>> 5 import pkg_resources<br>
>>> 6 pkg_resources.run_script('StarCluster==0.91', 'starcluster')<br>
>>><br>
>>> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cli.py<br>
>>> in <module>()<br>
>>> 36<br>
>>> 37<br>
>>> ---> 38<br>
>>> 39<br>
>>> 40<br>
>>><br>
>>> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cluster.py<br>
>>> in <module>()<br>
>>> 10<br>
>>> 11<br>
>>> ---> 12<br>
>>> 13<br>
>>> 14<br>
>>> ImportError: cannot import name iptools<br>
>>><br>
>>> On Sat, Oct 30, 2010 at 11:38 AM, Marc Resnick <<a href="mailto:mresnick@mit.edu" target="_blank">mresnick@mit.edu</a>> wrote:<br>
>>>><br>
>>>> The version here<br>
>>>><br>
>>>> <a href="http://github.com/jtriley/StarCluster/blob/experimental/starcluster/cluster.py" target="_blank">http://github.com/jtriley/StarCluster/blob/experimental/starcluster/cluster.py</a><br>
>>>> may have those bugs fixed, as I remember that one as well.<br>
>>>><br>
>>>><br>
>>>> Marc<br>
>>>><br>
>>>> On Sat, Oct 30, 2010 at 2:31 PM, Dan Tenenbaum <<a href="mailto:dtenenba@fhcrc.org" target="_blank">dtenenba@fhcrc.org</a>><br>
>>>> wrote:<br>
>>>> > I already did that. ;)<br>
>>>> > Now I'm getting another error:<br>
>>>> > In [1]: cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)<br>
>>>> >>>> Using private key /Users/dante/.ec2/bioc-keypair.pem.rsa (rsa)<br>
>>>> ><br>
>>>> > ---------------------------------------------------------------------------<br>
>>>> > AttributeError Traceback (most recent call<br>
>>>> > last)<br>
>>>> ><br>
>>>> > /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc<br>
>>>> > in <module>()<br>
>>>> > ----> 1<br>
>>>> > 2<br>
>>>> > 3<br>
>>>> > 4<br>
>>>> > 5<br>
>>>> ><br>
>>>> > /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc<br>
>>>> > in run_plugin(plugin_name, cluster_tag, cfg)<br>
>>>> > 175 plugins = {}<br>
>>>> > 176 plugins[plugin_name] = plug<br>
>>>> > --> 177 plugins = cl.load_plugins(plugins)<br>
>>>> > 178 master = cl.master_node<br>
>>>> > 179 for p in plugins:<br>
>>>> ><br>
>>>> > /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc<br>
>>>> > in load_plugins(self, plugins)<br>
>>>> > 314 plugs = []<br>
>>>> > 315 for plugin in plugins:<br>
>>>> > --> 316 setup_class = plugin.get('setup_class')<br>
>>>> > 317 plugin_name = plugin.get('__name__')<br>
>>>> > 318 mod_name = '.'.join(setup_class.split('.')[:-1])<br>
>>>> > AttributeError: 'str' object has no attribute 'get'<br>
>>>> > Hmm....<br>
>>>> > Dan<br>
>>>> ><br>
>>>> > On Sat, Oct 30, 2010 at 11:27 AM, Marc Resnick <<a href="mailto:mresnick@mit.edu" target="_blank">mresnick@mit.edu</a>><br>
>>>> > wrote:<br>
>>>> >><br>
>>>> >> Yes, that's the bug I told Justin about this Summer... I think I<br>
>>>> >> patched it, but he may have forgotten to pull from my fork.<br>
>>>> >><br>
>>>> >> You can fix it by editing the run_plugin method, and replacing<br>
>>>> >><br>
>>>> >> cl = get_cluster(cluster_tag)<br>
>>>> >><br>
>>>> >> with<br>
>>>> >><br>
>>>> >> cl = get_cluster(cluster_tag, cfg)<br>
>>>> >><br>
>>>> >><br>
>>>> >><br>
>>>> >> Marc<br>
>>>> >><br>
>>>> >><br>
>>>> >> On Sat, Oct 30, 2010 at 2:20 PM, Dan Tenenbaum <<a href="mailto:dtenenba@fhcrc.org" target="_blank">dtenenba@fhcrc.org</a>><br>
>>>> >> wrote:<br>
>>>> >> > Hi Marc,<br>
>>>> >> > I am running 0.91.<br>
>>>> >> > I was doing a couple things wrong. I figured them out but I am<br>
>>>> >> > still<br>
>>>> >> > having<br>
>>>> >> > this problem.<br>
>>>> >> > One thing I did wrong was have the<br>
>>>> >> > plugins=shortreadplugin<br>
>>>> >> > line under the global section; it needed to be under the<br>
>>>> >> > [cluster smallcluster]<br>
>>>> >> > section.<br>
>>>> >> > When I start the cluster now, I can see the output from my plugin:<br>
>>>> >> >>>> Running plugin shortreadplugin<br>
>>>> >> >>>> about to echo<br>
>>>> >> > But, when I run the shell and put in this command:<br>
>>>> >> > cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)<br>
>>>> >> > I get the same error described below.<br>
>>>> >> > Dan<br>
>>>> >> ><br>
>>>> >> > On Sat, Oct 30, 2010 at 11:11 AM, Marc Resnick <<a href="mailto:mresnick@mit.edu" target="_blank">mresnick@mit.edu</a>><br>
>>>> >> > wrote:<br>
>>>> >> >><br>
>>>> >> >> Hey Dan,<br>
>>>> >> >><br>
>>>> >> >> Are you running the latest version? I saw that bug a few months<br>
>>>> >> >> ago, I<br>
>>>> >> >> thought I remember Justin fixing it.<br>
>>>> >> >><br>
>>>> >> >><br>
>>>> >> >> Marc<br>
>>>> >> >><br>
>>>> >> >> On Sat, Oct 30, 2010 at 2:00 PM, Dan Tenenbaum<br>
>>>> >> >> <<a href="mailto:dtenenba@fhcrc.org" target="_blank">dtenenba@fhcrc.org</a>><br>
>>>> >> >> wrote:<br>
>>>> >> >> > Hi,<br>
>>>> >> >> > I am not much of a python programmer and I'm trying to figure<br>
>>>> >> >> > out the<br>
>>>> >> >> > plugin<br>
>>>> >> >> > system.<br>
>>>> >> >> > I created a ~/.starcluster/plugins directory and put a file in<br>
>>>> >> >> > it<br>
>>>> >> >> > called<br>
>>>> >> >> > shortread.py.<br>
>>>> >> >> > Here are its contents:<br>
>>>> >> >> ><br>
>>>> >> >> > from starcluster.clustersetup import ClusterSetup<br>
>>>> >> >> > from starcluster.logger import log<br>
>>>> >> >> > class ShortReadPlugin(ClusterSetup):<br>
>>>> >> >> > def __init__(self, arg1):<br>
>>>> >> >> > self.arg1 = arg1<br>
>>>> >> >> > <a href="http://log.info" target="_blank">log.info</a>('arg1 = %s' % arg1)<br>
>>>> >> >> > def run(self, nodes, master, user, user_shell, volumes):<br>
>>>> >> >> > <a href="http://log.info" target="_blank">log.info</a>("about to echo")<br>
>>>> >> >> > master.ssh.execute("echo 'hey'")<br>
>>>> >> >> > As you can see, it's just a test.<br>
>>>> >> >> > In my config, I put this at the bottom:<br>
>>>> >> >> > [plugin shortreadplugin]<br>
>>>> >> >> > SETUP_CLASS = ShortReadPlugin<br>
>>>> >> >> > arg1=foo<br>
>>>> >> >> > And above in the [global] section I put:<br>
>>>> >> >> > plugins=shortreadplugin<br>
>>>> >> >> > I'm a bit confused about whether this is right because in your<br>
>>>> >> >> > example:<br>
>>>> >> >> > <a href="http://web.mit.edu/stardev/cluster/docs/plugins.html" target="_blank">http://web.mit.edu/stardev/cluster/docs/plugins.html</a><br>
>>>> >> >> > ...you have the SETUP_CLASS as ubuntu.PackageInstaller. I'm not<br>
>>>> >> >> > sure<br>
>>>> >> >> > where<br>
>>>> >> >> > the "ubuntu" fits into this. Is it like java where I need to<br>
>>>> >> >> > have a<br>
>>>> >> >> > folder<br>
>>>> >> >> > called ubuntu where the class resides?<br>
>>>> >> >> > Anyway, I fired up my cluster and went to test this with<br>
>>>> >> >> > "starcluster<br>
>>>> >> >> > shell"<br>
>>>> >> >> > and got the following output:<br>
>>>> >> >> > In [5]: cluster.run_plugin('shortreadplugin', 'myfirstcluster',<br>
>>>> >> >> > cfg)<br>
>>>> >> >> ><br>
>>>> >> >> ><br>
>>>> >> >> ><br>
>>>> >> >> > ---------------------------------------------------------------------------<br>
>>>> >> >> > TypeError Traceback (most recent<br>
>>>> >> >> > call<br>
>>>> >> >> > last)<br>
>>>> >> >> ><br>
>>>> >> >> ><br>
>>>> >> >> ><br>
>>>> >> >> > /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc<br>
>>>> >> >> > in <module>()<br>
>>>> >> >> > ----> 1<br>
>>>> >> >> > 2<br>
>>>> >> >> > 3<br>
>>>> >> >> > 4<br>
>>>> >> >> > 5<br>
>>>> >> >> ><br>
>>>> >> >> ><br>
>>>> >> >> ><br>
>>>> >> >> > /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc<br>
>>>> >> >> > in run_plugin(plugin_name, cluster_tag, cfg)<br>
>>>> >> >> > 170 def run_plugin(plugin_name, cluster_tag, cfg):<br>
>>>> >> >> > 171 ec2 = cfg.get_easy_ec2()<br>
>>>> >> >> > --> 172 cl = get_cluster(cluster_tag)<br>
>>>> >> >> > 173 cl.load_receipt()<br>
>>>> >> >> > 174 plug = cfg.get_plugin(plugin_name)<br>
>>>> >> >> > TypeError: get_cluster() takes exactly 2 arguments (1 given)<br>
>>>> >> >> ><br>
>>>> >> >> > It's hard for me to understand what's happening here. I of<br>
>>>> >> >> > course am<br>
>>>> >> >> > not<br>
>>>> >> >> > calling get_cluster directly. Is there a bug in run_plugin() or<br>
>>>> >> >> > is<br>
>>>> >> >> > there<br>
>>>> >> >> > something wrong in my setup, or both?<br>
>>>> >> >> > Thanks<br>
>>>> >> >> > Dan<br>
>>>> >> >> ><br>
>>>> >> >> > _______________________________________________<br>
>>>> >> >> > StarCluster mailing list<br>
>>>> >> >> > <a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
>>>> >> >> > <a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
>>>> >> >> ><br>
>>>> >> >> ><br>
>>>> >> >><br>
>>>> >> ><br>
>>>> >> ><br>
>>>> >><br>
>>>> ><br>
>>>> ><br>
>>>><br>
>>><br>
>><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div></div>
</div></div></blockquote></div><br></div>