<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Cedar,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I completely agree with David. We routinely use blast in a software pipeline build on top of EC2. We started by using an NFS share, but we are currently transitioning
to ephemeral storage.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Our plan is to put the nr database (and other file-based data libraries) on local SSD ephemeral storage for each node in the cluster. You may want to consider
pre-packaging the compressed libraries on a custom StarCluster AMI, then use a plug-in to mount ephemeral storage and decompress the blast libraries into ephemeral storage. The avoids the download from S3 each time you start a node, which added 10-20 minutes
in our case. Plus, it eliminates one more possible point of failure during cluster initialization. To us, it is worth the extra cost of maintaining a custom AMI and the extra size of the AMI itself.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Steve<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> starcluster-bounces@mit.edu [mailto:starcluster-bounces@mit.edu]
<b>On Behalf Of </b>David Stuebe<br>
<b>Sent:</b> Friday, May 09, 2014 12:49 PM<br>
<b>To:</b> starcluster@mit.edu<br>
<b>Subject:</b> Re: [StarCluster] Fast shared or local storage? (Cedar McKay)<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Hi Cedar<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Beware of using NFS – it may not be posix compliant in ways that seem minor but have caused problems for HDF5 files. I don't know what the blast db file structure
is or how they organize their writes, but it can be a problem in some circumstances. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">I really like the suggestions of using the ephemeral storage. I suggest you create a plugin that moves the data to the drive from S3 on startup when you add a
node. That should be simpler than the on demand caching which although elegant may take you some time to implement. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">David<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Thanks for the very useful reply. I think I'm going to go with the s3fs option and cache to local ephemeral drives. A big blast database is split into many parts,
and I'm pretty sure that every file in a blast db isn't read every time, so this way blasting can proceed immediately. The parts of the blast database download from s3 on demand, and cached locally. If there was much writing, I'd probably be reluctant to use
this approach because the s3 eventual consistency model seems to require tolerance of write fails at the application level. I'll write my results to a shared nfs volume.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">I thought about mpiBlast and will probably explore it, but I read some reports that it's xml output isn't exactly the same as the official NCBI blast output,
and may break biopython parsing. I haven't confirmed this, and will probably compare the two techniques.
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Thanks again!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Cedar<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>