<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Thanks.</div><div class=""><br class=""></div><div class="">Yes, I started checking before and after line execution, but nothing different. So a relaunch:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">>>> IPCluster has been started on SecurityGroup:@sc-toma for user 'sgeadmin'</div><div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">with 159 engines on 5 nodes.</div></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""></div><div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">In IP[y]: Notebook</div><div class=""><br class=""></div><div class="">Through IP[y]: Notebook, I’m assigning 150 engines.</div><div class=""><br class=""></div><div class="">So using the following test code:</div><div class=""><br class=""></div><div class=""><div class="">from IPython.parallel import Client</div><div class="">ipclient = Client(packer = 'pickle')</div><div class="">dview = ipclient[:]</div><div class="">lview = ipclient.load_balanced_view()</div></div><div class=""><br class=""></div><div class="">Here I am getting an array of 150 elements as expected when using len(ipclient.ids)</div><div class=""><br class=""></div><div class="">Now executing code:</div><div class=""><br class=""></div><div class="">%%px --local</div><div class=""><div class="">import socket</div><div class="">import pandas as pd</div><div class="">def distribute(a):</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>return socket.getfqdn()</div></div><div class=""><br class=""></div><div class=""><div class="">a = pd.DataFrame(range(0,10000))</div><div class=""><br class=""></div></div><div class="">version 1a:</div><div class="">dview.map(distribute, a[0]).get()</div><div class=""><br class=""></div><div class="">version 1b:</div><div class="">lview.map(distribute, a[0]).get()</div><div class=""><br class=""></div><div class="">--This results in an output of ‘master’ in each element.</div><div class=""><br class=""></div><div class="">version 2:</div><div class=""><div class="">dview.scatter('a', a)</div><div class="">dview.execute('b = distribute(a[0])', block=True)</div><div class="">dview.gather('b', block=True)</div></div><div class=""><br class=""></div><div class="">--This also results in an output of ‘master’ in each element.</div><div class=""><br class=""></div><div class="">verifying with len(ipclient.ids) confirms that I have all the engines in place.</div><div class=""><br class=""></div><div class="">A.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 1, 2015, at 11:14 AM, MinRK <<a href="mailto:benjaminrk@gmail.com" class="">benjaminrk@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Can you perhaps share a code sample? Have you verified that all the engines are registered with the Client (`Client.ids`) before submitting the tasks?<div class=""><br class=""></div><div class="">-MinRK</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Jul 1, 2015 at 9:09 AM, Austin So <span dir="ltr" class=""><<a href="mailto:austin.so@tomabio.com" target="_blank" class="">austin.so@tomabio.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">I’ve been trying to figure out what I’m doing wrong here, and if it is an issue within the starcluster config file. I’ve exhausted all possible implementations in my code that I could think of.</div><div class=""><br class=""></div><div class="">During set up, 255 engines have been recorded to have been set-up by starcluster upon launch that are available to IPcluster.</div><div class=""><br class=""></div><div class="">Within IPython Notebooks, I’m trying to distribute a function across all my nodes and engines (5@r3.8xlarge).</div><div class=""><br class=""></div><div class="">So when the line of code is running, I’m looking at qhost, and I see that only the Master is showing a CPU load. I look at the Cloud Metrics, and I see that only the Master is showing a CPU load. At the suggestion of a friend, I returned a socket.fqdn() call to identify if the results were processed by the master or one of the nodes. All results returned were generated by the Master.</div><div class=""><br class=""></div><div class="">Any hints to identify where the source of the problem lies would be greatly appreciated.</div><div class=""><br class=""></div><div class="">Best</div><span class="HOEnZb"><font color="#888888" class=""><div class=""><br class=""></div><div class="">Austin</div><div class=""><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""></span></div>
</span></div>
</div><br class=""></font></span></div><br class="">_______________________________________________<br class="">
StarCluster mailing list<br class="">
<a href="mailto:StarCluster@mit.edu" class="">StarCluster@mit.edu</a><br class="">
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" rel="noreferrer" target="_blank" class="">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br class="">
<br class=""></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""><div apple-content-edited="true" class=""><div class=""><span style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></span></div>
</span></div>
</div><br class=""></div></body></html>