<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">It’s quite fast. Using the AWS CLI tools, I clock downloads at ~180-200 MB/s (megabytes per second) from S3 to an r3 instance. Similar for uploads. The CLI tools run up to 10 threads per download, thus the speed. YMMV of course, especially with smaller instances. Keep in mind S3 does not provide real random access (though byte-range headers are supported), so it cannot replace NFS/EBS in all cases.<div><br></div><div>Nik<br><div><br><div><div>On Oct 6, 2014, at 6:19 PM, John Readey <<a href="mailto:jreadey@hdfgroup.org">jreadey@hdfgroup.org</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;">
<div>Is anyone pulling in data on demand using the S3 API’s? I’m curious how the performance compares to using NFS.</div>
<div><br>
</div>
<div>John</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);">
<span style="font-weight:bold">From: </span>Jennifer Staab <<a href="mailto:jstaab@cs.unc.edu">jstaab@cs.unc.edu</a>><br>
<span style="font-weight:bold">Date: </span>Monday, October 6, 2014 at 6:08 PM<br>
<span style="font-weight:bold">To: </span>greg <<a href="mailto:margeemail@gmail.com">margeemail@gmail.com</a>>, "<a href="mailto:starcluster@mit.edu">starcluster@mit.edu</a>" <<a href="mailto:starcluster@mit.edu">starcluster@mit.edu</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [StarCluster] Possible to bring in data stored on S3?<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Besides the others mentioned, AWS has their own "command line interface" CLI see
<a href="http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html">
here</a> for installation instructions and <a href="http://docs.aws.amazon.com/cli/latest/">
here</a> for usage. AWS CLI provides tools for accessing S3. If you don't want to use AWS CLI, there are many SDKs available depending on programming language or platform see
<a href="http://aws.amazon.com/tools/">here</a>.<br>
<br>
Good Luck.<br>
<br>
Jennifer<br>
<br>
On 10/6/14 3:29 PM, greg wrote:<br>
</div>
<blockquote cite="mid:CAOpDiKyBKR=g673oWfwB3FFQKBuNfiUmRxNiToLWxNgLzTwJgQ@mail.gmail.com" type="cite">
<pre wrap="">Hi all,
Is it possible to bring data into my cluster from Amazon S3? I didn't
see it in the manual.
Thanks for reading!
Greg
_______________________________________________
StarCluster mailing list
<a class="moz-txt-link-abbreviated" href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><a class="moz-txt-link-freetext" href="http://mailman.mit.edu/mailman/listinfo/starcluster">http://mailman.mit.edu/mailman/listinfo/starcluster</a></pre>
</blockquote>
<br>
</div>
</div>
</blockquote>
</span>
</div>
_______________________________________________<br>StarCluster mailing list<br><a href="mailto:StarCluster@mit.edu">StarCluster@mit.edu</a><br>http://mailman.mit.edu/mailman/listinfo/starcluster<br></blockquote></div><br></div></div></body></html>