ACCRE is developing a new high-performance data archiving system called Logistical Storage, or L-Store. See our brochure at left and our performance graphs from SuperComputing 06 where L-Store achieved sustained data transfers at 3.3 GigaBytes/s. This work is supported with funding from NSF grant PHY-0619847 and by the Vanderbilt Center for the Americas. For more information please see the L-Store Wiki and REDDnet Wiki.

L-Store provides a flexible logistical storage framework for distributed, scalable, and secure access to data for a wide spectrum of users. L-Store is planned to be used on the REDDnet infrastructure. It is designed to provide: virtually unlimited scalability in both raw storage and associated file system metadata; a decentralized management system; security; fault tolerant metadata support; user controlled replication and striping of data on a file and directory level; scalable performance in both raw data movement and metadata queries; a virtual file system interface in both a web and command line form; and support for the concept of geographical locations for data migration to facilitate quicker access. These features are accomplished using two fundamental technologies designed to handle such wide-ranging data storage and access needs:

DISTRIBUTED METADATA
Distributed Hash Tables (DHT), developed for peer-to-peer systems, are used as a scalable mechanism for managing metadata by distributing all metadata across a collection or ring of metadata servers. The DHT supports a single operation: given a key (the filename), it maps the key onto a server using a hash function. The hash function evenly distributes the keys throughout the available key space and each DHT server is responsible for a fraction of the key space. The client only needs to know a single DHT server to perform a key lookup since each DHT server keeps information concerning key to node mappings for routing to nearby DHT servers.

LOGISTICAL NETWORKING
A data transfer protocol is a standard format used to transfer data between computers on a network. L-Store utilizes the Internet Backplane Protocol (IBP) developed by the Logistical Computing and Internetworking (LoCI) Lab at the University of Tennessee, Knoxville. IBP enables the movement of large data sets via the simultaneous transfer of data fragments rather than requiring the sequential transfer of the entire data set. Mirroring, data striping, fault tolerance, and recovery features are also supported by IBP. The requisite software can be installed on any machine running a Unix/Linux operating system, effectively transforming the machine into a storage depot.

L-STORE ARCHITECTURE
L-Store is designed to provide a clean, functional delineation between its layers with each function designed to scale independently of the other building blocks. The figure below shows the L-store components and how they fit together. The best way to understand this is to see how a file is absorbed into L-Store. First, each file is associated with a directory, name, list of attributes, and authentication/authorization objects. Then through the use of IBP, the file is broken up into multiple slices and stored on different media.

L-SERVER - This server controls the L-Store system and is how the client interacts with L-Store. Multiple L-Servers are supported with transparent load balancing.
DIRECTORY RING - The directory ring ascribes each directory a unique directory ID (DID). The DID is constructed by hashing the full path of the directory.
FILE RING - After the files parent DID is resolved, it is then combined with the hash of the filename (FID) and stored in the file ring.
SLICE RING - IBP splits a file up into multiple slices. Each of these slices is given a unique Slice ID (SID), and stored as part the file's metadata. Separating the slice metadata, which include information about the storage media, from the file information gives L-Store the ability to detect hot spots on the medium and automatically replicate or move slices to eliminate the hot spot.
JUKEBOX - L-Store has intentionally separated out where the media is stored from how it is accessed. This separation allows one to treat tape, disk, DVD, or memory in the same way. A Jukebox is composed of three components: media, movers, and a media changer. A Jukebox can have multiple media and movers.
MEDIA CHANGER - The media changer is responsible for keeping track of media space usage and attributes, tracking media, and setting up the connection between media and movers. The Jukebox Manager probes the Media Changer's inventory for satisfying specific media attribute requests.
MOVERS and MEDIA - A mover presents an IBP depot to the client and is strictly for moving data to and from the different media. For directly-attached disks there would typically be a fixed mover; for example, the machine attached to the disk. In SAN environments, the relationship between mover and media can be more dynamic.
JUKEBOX MANAGER - The Jukebox Manager is designed to keep track of each media items different attributes. There can be multiple Jukebox Managers, each keeping track of one or more Jukeboxes. A Jukebox Manager handles requests for different attributes and space requirements made for all write operations.