[StarCluster] HDF5 Users

[John Readey]

Hello,
  I just came across the StarCluster project and was wondering how popular the HDF5 file format is with StarCluster users.  One challenge with using HDF5 in a cluster is that HDF5 over NFS is not supported.  Currently parallel usage of HDF5 now is restricted to MPI-I/O with parallel filesystems such as Lustre or  GPFS.
  At the HDFGroup we've been proto-typing a REST-based service implementation of HDF5 - see http://www.hdfgroup.org/pubs/papers/RESTful_HDF5.pdf for a description of the API.  I can imagine this service being a good solution  for cluster based data I/o.  In this scenario each node in the cluster would read/write data via http requests (rather than using NFS).   Performance is one aspect that would be critical in a cluster environment.  It's unknown at this point how much overhead the http request/response processing would add.
  I touched on this idea in my presentation at this summer's HDF Workshop.  You can see the slides here: http://www.slideshare.net/HDFEOS/jxr?ref=http://hdfeos.org/workshops/ws17/slideshare.php.  From the architecture slide you can see that clients can access the REST API directly or use the HDF5 library (for C/Fortran applications).  The library backend will convert the calls to http requests.
  If anyone is interested in this approach it would be great to discuss.  Please reply to this thread or contact me directly.

John Readey
HDF Group
http://www.hdfgroup.org/
Jreadey AT hdfgroup.org


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/starcluster/attachments/20140916/f218f42a/attachment.htm