Many groups are now exploring the development of scalable, decentralized file systems, such as OceanStore, FreeHaven, Freenet, and MojoNation. They are better characterized as serverless or peer to peer than the strictly client / server organization of traditional systems. The challenges of extreme decentralization and very large scale are new and difficult while real experience is hard to come by. While existing projects and proposed systems have divergent goals and strategies, they face many common problems and can learn from each others prototypes. It would benefit everyone to have a forum for clarifying goals, sharing new ideas, soliciting feedback and getting help with problems. Ideally, the discussion would help solidify this new field, eventually leading to some agreement on a taxonomy of goals, problems and solutions. A mailing list focused on technical issues and steering clear of fuzzier topics, more like coderpunks than cypherpunks, could provide such a service. A list dedicated to the exchange of ideas *between* these groups would supplement the multitude of existing project-specific discussions and enrich them all.
The charter of the Global-Scale Distributed Storage Systems mailing list is:
The purpose of the mailing list is to foster discussion of design and implementation issues related to the development of scalable, decentralized storage systems of literally global scope. The emphasis should be on technical descriptions and critique of mechanisms providing efficiency, reliability, security, and similar properties. Discussion of goals and semantics is also desirable, while acknowledging that a diversity of systems will be built and evaluated. Messages with primarily political, legal or philosophical content are discouraged.
This mailing list is hosted by ibiblio using Mailman software. Here is the webpage for subscriptions and member may email submissions to bluesky@lists.ibiblio.org.
The list was formerly hosted at franklin.oit.unc.edu. The subscriber information was transferred to the new host on Monday 18-November-2002 and there should be no interruption in service.
Non-members can view the archive on ibiblio or at mail-archive (the latter appears to be incomplete).
"When a user requests a large file from a content provider's site, the user's Swarmcast application automatically locates and downloads chunks of the file from the nearest, least busy computers that have previously downloaded the file. The user's computer downloads simultaneously from as many nodes in the network as possible, maximizing their available download capacity..."Dan Moniz adds on 11-Jul-2001 that Swarmcast is open source.
[It] is a more complex and sophisticated system which has concepts similar to Freenet. Here is an excerpt of their ideas on ``nomadic'' data:In a system as large as OceanStore, locality is of extreme importance. Thus, we have as a goal that data can be cached anywhere, anytime. We call this policy promiscuous caching. Data which is allowed to flow freely is called nomadic data. Note that nomadic data is an extreme consequence of separating information from its physical location. Although promiscuous caching complicates data coherence and location, it provides great flexibility to optimize locality and to trade off consistency for availability.OceanStore keeps careful track of where data is located as it flows through the network. This allows its data lookup algorithm to be highly reliable and fault tolerant. My concern is that it may turn out to be costly to keep location information up to date, and since they don't have any code on their web page, it's hard to judge how the efficiency tradeoffs will turn out.