[StarCluster] Possible to bring in data stored on S3?

[Jennifer Staab]

I agree with Nik, it seems reasonably fast in my experience.  Note that 
S3 doesn't have a file system, it's just an indexed object.  So you can 
write your object once and read it many times, but there isn't any 
updating, file locking, etc.  I typically use S3 for storage and 
archiving or to provide a static data reference for things that aren't 
going to change, but need to be widely available.  I know those that use 
S3 for static webpages, or for storing logs, etc.  If your data is going 
to change often, you might want to do things a different way.

Good Luck.

Jennifer

On 10/6/14 9:19 PM, John Readey wrote:
> Is anyone pulling in data on demand using the S3 API's?  I'm curious 
> how the performance compares to using NFS.
>
> John
>
> From: Jennifer Staab <jstaab at cs.unc.edu <mailto:jstaab at cs.unc.edu>>
> Date: Monday, October 6, 2014 at 6:08 PM
> To: greg <margeemail at gmail.com <mailto:margeemail at gmail.com>>, 
> "starcluster at mit.edu <mailto:starcluster at mit.edu>" 
> <starcluster at mit.edu <mailto:starcluster at mit.edu>>
> Subject: Re: [StarCluster] Possible to bring in data stored on S3?
>
>     Besides the others mentioned, AWS has their own "command line
>     interface" CLI see here
>     <http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html>
>     for installation instructions and here
>     <http://docs.aws.amazon.com/cli/latest/> for usage. AWS CLI
>     provides tools for accessing S3. If you don't want to use AWS CLI,
>     there are many SDKs available depending on programming language or
>     platform see here <http://aws.amazon.com/tools/>.
>
>     Good Luck.
>
>     Jennifer
>
>     On 10/6/14 3:29 PM, greg wrote:
>>     Hi all,
>>
>>     Is it possible to bring data into my cluster from Amazon S3?  I didn't
>>     see it in the manual.
>>
>>     Thanks for reading!
>>
>>     Greg
>>     _______________________________________________
>>     StarCluster mailing list
>>     StarCluster at mit.eduhttp://mailman.mit.edu/mailman/listinfo/starcluster
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20141006/b0b1b2bd/attachment.htm