Life Sciences Collaborative Access Team

9700 S. Cass Ave, Bldg 436A
Argonne, IL 60439
Main Office:    630-252-0222
21-ID-D:    630-343-6851 21-ID-E:    630-343-6852
21-ID-F:    630-343-6853 21-ID-G:    630-343-6854

Lustre at LS-CAT

LS-CAT uses the Lustre file system to manage data storage. This file system was developed by Cluster File Systems, Inc (now owned by Sun) for use at supercomputer facilities and, happily, is perfect for our situation. On the server side we have one computer, "piano", that acts as a "Meta Data Server" (or an "MDS" as the lustre geeks like to say). This handles all the directory information but does not store any of the actual data. The meta data is continually backed up on a second server "forte" using the DRBD block device. The plan is to make piano and forte into a fail-over pair we'll call pianoforte to provide a highly available system, but we are not there yet.

The data are normally striped across the data servers (aka Object Storage Targets [OST]) and hence even large files can be read and written very quickly as the load is distributed across many fast servers.

This file system model scales very well: as we add more OST's the access speed increases and is limited by the ethernet connections to the clients and these connection can be bonded so that multiple ports work together to form a single fast connection.

The illustration below shows the Lustre implementation at LS-CAT as of early 2008. Each of our stations has a "V" computer that writes the data directly from the detector to the Lustre file system. There are also two processing computers per experimental station (the "G" computers). As of this writing there are only 3 OST's (the "P" computers) and these provide a total of 11 TB of storage.

Our Lustre system is several orders of magnitude smaller than other Lustre systems in production use: we have plenty of room to grow before we'll run into any limits.