Storage controllers: choosing between local disk and the network 'cloud'
Different types of storage services are available to repositories, from local
disks to the distributed 'cloud', offering choices of scale, bandwidth
and cost. Rather than adopt a single storage approach, with growing
data volumes and data types it is likely repositories will choose a
combination of services, or 'hybrid' storage. If there are storage
options, we have to manage copy and
transfer of content from the repository to the chosen locations.
Preserv has developed a storage controller
for EPrints software.
Extended abstract From
the Desktop to the Cloud: Leveraging Hybrid Storage Architectures in
, updated April 2009, accepted for Open Repositories Conference 2009 (OR09)
Introduces the EPrints storage controller, which allows repositories
using this software to integrate with emerging network, storage and
cloud services. For a less technical approach, see the furniture
removal analogy in this blog
(February 2009) to help you understand these
cloud-storage controller developments.
The EPrints storage controller has been successfully
for storing content in Amazon S3/Cloudfront, and will be
implemented and available from EPrints version 3.2
Two more storage plug-ins are available so far for the
storage controller: the local storage plug-in that also supports the
legacy local disk layout, and a plug-in for the Sun STK5800 server.
Find out how to write a storage plug-in
Presentation From open
storage to smart storage: enabling EPrints repository preservation
(slides), Sun Preservation and Archiving Special Interest Group
, May 2008
First description of the storage controller for EPrints repository
software. Supports a pluggable storage layer for repositories,
providing the ability to store objects in different locations based on a set of rules, e.g. using
metadata or type. For example, a generated thumbnail could be stored locally while the original image is stored in Amazon S3 and in a local archival server. Another example would be storing files of a certain size or classification offsite and sending these to a tape queue for backup.
Listing Fedora Commons (the repository software) as a storage service in the slide above led to some confusion initially. Using one repository software as a front-end (EPrints) to another (Fedora) offers intriguing possiblities, particularly in this case where the softwares have complementary strengths in terms of interfaces and data management. Keeping two repositories in sync when both are trying to perform similar operations could be tricky, however, but is possible when dealing with input of new items.
Alternative storage controllers
As part of Fedora Commons, the Akubra project
is implementing a plug-in based storage abstraction.
DuraSpace, a joint DSpace/Fedora project, seeks to offer a commercial service to mediate between the respective repository softwares and storage services:
"'DuraSpace' (is) a new web-based
service that will allow institutions to easily distribute content to
multiple storage providers, both 'cloud-based' and institution-based.
The idea behind DuraSpace is to provide a trusted, value-added service
layer to augment the capabilities of generic storage providers by making
stored digital content more durable, manageable, accessible and
sharable." Full DSpace press release
For a perspective on early progress with Duraspace from a Preserv team leader, see this blog entry
(25 February 2009) applauding the concept of durability for repository content and storage flexibility, but cautious about offering too many new services: "Let's do it in the cloud - but lets work really hard at articulating the benefits that the cloud end user will enjoy and stop relying on general talk about value-added services. I think researchers/end-users will forgive us for not having finished implementing something yet, but they won't forgive us for a lack of imagination."