[StarCluster] Fast shared or local storage? (Cedar McKay)

[Rayson Ho]

The "64GB limit" in s3fs should only affect writes to S3, as AWS
limits 10,000 parts in a multipart upload operation, and s3fs by
default uses 10MB part size. Thus the max filesize is 100GB, but the
s3fs developers limit it to 64GB (they say it is a nicer number). That
code was removed from the latest git version, and the limit now is
10000 * multipart size (so if multipart_size is set to a larger number
than 10MB, then the max file size could be larger than 100GB).

On the other hand, S3 supports concurrent reads from S3 via "Range
GET"s, which is what is used by s3fs. As S3 does not limit the number
of concurrent GETs, the 100GB limit shouldn't apply to reads from S3.

Rayson

==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html




On 5/13/14, MacMullan, Hugh <hughmac at wharton.upenn.edu> wrote:
> I remember having to modify the s3fs source to handle files greater than ...
> I think 64GB, FYI, in case that's important. If anyone runs into issues with
> big files and s3fs, poke the code, or if you need further details (no idea
> if that's still a problem), just let me know.
>
> -Hugh
>
> On May 13, 2014, at 16:07, "Cedar McKay"
> <cmckay at uw.edu<mailto:cmckay at uw.edu>> wrote:
>
> Great, thanks for all the info guys. I ended up implementing mounting my
> read only databases as an s3fs volume, then designating the ephemeral
> storage as the cache. Hopefully this will give me the best of all worlds;
> fast local storage, and lazy downloading.
>
> I haven't tested much yet, but If I have problems with this setup I'll
> probably just skip the s3fs thing and just load the database straight onto
> the ephemeral storage as you suggested.
>
> best,
> Cedar
>
>
>
> On May 12, 2014, at 2:12 PM, Steve Darnell
> <darnells at dnastar.com<mailto:darnells at dnastar.com>> wrote:
>
> Hi Cedar,
>
> I completely agree with David. We routinely use blast in a software pipeline
> build on top of EC2. We started by using an NFS share, but we are currently
> transitioning to ephemeral storage.
>
> Our plan is to put the nr database (and other file-based data libraries) on
> local SSD ephemeral storage for each node in the cluster. You may want to
> consider pre-packaging the compressed libraries on a custom StarCluster AMI,
> then use a plug-in to mount ephemeral storage and decompress the blast
> libraries into ephemeral storage. The avoids the download from S3 each time
> you start a node, which added 10-20 minutes in our case. Plus, it eliminates
> one more possible point of failure during cluster initialization. To us, it
> is worth the extra cost of maintaining a custom AMI and the extra size of
> the AMI itself.
>
> Best regards,
> Steve
>
> From: starcluster-bounces at mit.edu<mailto:starcluster-bounces at mit.edu>
> [mailto:starcluster-bounces at mit.edu<mailto:bounces at mit.edu>] On Behalf Of
> David Stuebe
> Sent: Friday, May 09, 2014 12:49 PM
> To: starcluster at mit.edu<mailto:starcluster at mit.edu>
> Subject: Re: [StarCluster] Fast shared or local storage? (Cedar McKay)
>
>
> Hi Cedar
>
> Beware of using NFS – it may not be posix compliant in ways that seem minor
> but have caused problems for HDF5 files. I don't know what the blast db file
> structure is or how they organize their writes, but it can be a problem in
> some circumstances.
>
> I really like the suggestions of using the ephemeral storage. I suggest you
> create a plugin that moves the data to the drive from S3 on startup when you
> add a node. That should be simpler than the on demand caching which although
> elegant may take you some time to implement.
>
> David
>
>
> Thanks for the very useful reply. I think I'm going to go with the s3fs
> option and cache to local ephemeral drives. A big blast database is split
> into many parts, and I'm pretty sure that every file in a blast db isn't
> read every time, so this way blasting can proceed immediately. The parts of
> the blast database download from s3 on demand, and cached locally. If there
> was much writing, I'd probably be reluctant to use this approach because the
> s3 eventual consistency model seems to require tolerance of write fails at
> the application level. I'll write my results to a shared nfs volume.
>
> I thought about mpiBlast and will probably explore it, but I read some
> reports that it's xml output isn't exactly the same as the official NCBI
> blast output, and may break biopython parsing. I haven't confirmed this, and
> will probably compare the two techniques.
>
> Thanks again!
>
> Cedar
>
>