Return-path: Subject: A Distributed Network Cache Service To: cypherpunks@cyberpass.net, eternity@internexus.net Date: Fri, 29 Aug 1997 14:16:21 -0700 (PDT) From: Mike Duvos I've been thinking a bit about the micropayment-based distributed network file system of the future. It should be scalable, with the addition of servers being a simple process. Anyone should be able to run a server. Kidnapping a server should disclose nothing about the contents of files stored there. Which files are stored on which servers should be concealed from mere users of the network. Anyone should be able to use the system by connecting to the server nearest them. Deterministic routing between servers should be used by servers to provide files to other servers. The system should provide for replication of files on a fixed number of servers, with "k of n" parts being sufficient for reconstruction. Aside from replication done for reliability, there should be only one copy of a file in the system at any time. If multiple users submit the same file, there should be no more copies than if one user submitted it. The architecture of the system should be completely independent of any specific file system organization, and any user should be able to sync all or any part of his file system with the network file system, or cache against the network file system on his local machine. The system should be completely secure, even from operators of servers. The person submitting a file receives a secret necessary for its reconstruction, and the file may only be accessed by the owner and those to whom the secret is communicated. The war room at the Pentagon and TCMay should be able to use the same server as an extension of their local file systems without any security worries. Let's consider the following service... Bob has an string of 100,000 octets he wants to make available to Alice and a few of their closest friends. Bob makes a secure connection to a Trusted Agent doing business with the network file system, and gives the Trusted Agent the string, and a micropayment of $1/MB or 10 cents. Working in a secure box, the Trusted Agent batch compresses and scrambles Bob's string into a much smaller string of seemingly random bits, and gives Bob two 128 bit values, a TAG, which will identify Bob's data to Bob, and to anyone else Bob may wish to share his data with, and a CONTEXT, which will unravel Bob's data back into its original form. The Trusted Agent then signs Bob's data, declaring that the resulting digital coccoon produced from Bob's data is associated with the TAG, and that the CONTEXT necessary to recover the data has been communicated to Bob and all copies destroyed. The agent gives Bob the TAG and CONTEXT, and sends the processed data and the TAG to the distributed file system, and it is stored on one or more of its servers for the next 100 days, after which it expires. For the next 100 days, Bob and all of his friends may use the 256 bit TAG/CONTEXT pair in place of Bob's data and may serve that data for free any number of times off any network file system server. They simply make a secure connection to any server, and send the TAG. The server returns to them the seemingly random stream of bits, after which a freeware program takes that data and the CONTEXT, and regenerates the original data. After 100 days, bob may fork over another 10 cents for another 100 days, or his data will disappear. The TAG assigned to an Octet String is unique, and no two strings have the same TAG, and no string is ever assigned multiple tags, no matter how often it may be entered and purged from the system. Now, almost everything you could want from a network wide distributed file system can be carried on top of this simple service, which might be considered a system of temporary lodging for popular Octet Strings. You rent your Octet String a room, where you and people you authorize may have unlimited visits for 100 days, and at the end, your Octet String checks out for parts unknown. One can do an almost unlimited number of useful things with such a service which, given a fast network connection, allows you to store any file in 256 bits for a period of time ranging from 100 days to forever, for a small fee. Bob could back up his entire file system to the network, sending each individual file or directory as an Octet String, with hard links to files replaced with TAG/CONTEXT pairs. The TAG/CONTEXT pair for Bob's root directory would then be used to access Bob's file system off any network server. If Bob changed some files, he could resync his FS with the server by only updating a few Octet Strings. If Bob synced his FS with the network daily, and jotted down the TAG/CONTEXT pair for his root directory, Bob could mount his file system as it appeared at backup time for any of the last 100 days, with no replication of material on the servers that contained it. Bob could write a "compression" program, Bobzip, which would produce an "archive" where each file only took 256 bits of space, and which could be "unBobzipped" on any machine connected to the Network for the next 100 days. Bob could mail this small "archive" to his friends. If browsers understood URLs referencing the distributed file system, Usenet binaries could be replaced with appropriate pointers in Usenet articles, ending the endless replication of binaries on thousands of machines, and reducing Usenet back to a managable volume of data. Users could boot their machines off the Network, and instantly see gigabytes of software available to them. After syncing their cache file system with the network, they could drop their PC off the roof, buy another one, and by typing in just one TAG/CONTEXT pair, see all their files again. Cache proxys could be set up which would grab files from the servers and provide NFS access to them, permitting other hosts to export NNTP, HTTP, and FTP access to the distributed network data. I would imagine a typical useful node in such a distributed system would need about 10 gig of disk, a fast network connection, and a reasonably fast processor. Such a node could hold 100 meg of compressed data for each of the 100 days until the space was reused. At full utilization, each node would generate in excess of $100 a day of revenue at $1/MB. -- Mike Duvos $ PGP 2.6 Public Key available $ enoch@zipcon.com $ via Finger $ {Free Cypherpunk Political Prisoner Jim Bell} ********** Return-path: Date: Fri, 29 Aug 1997 15:29:41 -0700 (PDT) From: Lucky Green Subject: Re: A Distributed Network Cache Service To: Mike Duvos Cc: cypherpunks@cyberpass.net, eternity@internexus.net On Fri, 29 Aug 1997, Mike Duvos wrote: > The Trusted Agent then signs Bob's data, declaring that > the resulting digital coccoon produced from Bob's data is > associated with the TAG, and that the CONTEXT necessary to > recover the data has been communicated to Bob and all copies > destroyed. Which is of course where the protocol will fail. Any such system must be resistant to operator compromise. A clean solution for a distributed data haven resistant to machine/operators compromise would be to use a design similar to the Anonymous Mailbox Servers I gave a talk on at HIP'97. Unfortunately, our project leader is very busy with his daytime job and it probably will be a while before we will see some demo sites up and running. But it will happen :-) --Lucky ********** Return-path: X-Authentication-Warning: fma66.fma.com: majordomo set sender to owner-espam@lists.espace.net using -f X-Orig-From: Mike Duvos X-E$Pam-Source: owner-cypherpunks@cyberpass.net Date: Sun, 31 Aug 1997 15:31:59 -0400 To: espam@intertrader.com From: Robert Hettinga Subject: Adding Memory to the Net Sender: owner-espam@lists.espace.net Reply-To: e$@thumper.vmeng.com --------------------------------------------------------------------- This mail is brought to you by the e$pam mailing list --------------------------------------------------------------------- To: cypherpunks@cyberpass.net Date: Sun, 31 Aug 1997 11:15:41 -0700 (PDT) From: Mike Duvos Sender: owner-cypherpunks@cyberpass.net X-Loop: cypherpunks@cyberpass.net A few people have commented on my "Distributed Network Cache Service" suggestion since I posted it, and I would like to respond to their comments. The basic idea here is to add memory to the Net in a reliable way, so that the Net provides memory services in the way it now universally provides data transport services. The Net would then serve compressed encrypted Octet Strings to machines connected to it, and provide a consistant view of which Octet Strings were available at any point in time which was independent of the access point. Adding Octet Strings to the universe of available ones, with a specified lifetime, would involve a micropayment. Computers connected to the Net could then employ either Octet Strings or fixed length data consisting of a cryptographic hash and decryption key to describe any chunk of data, which would reduce the storage requirements of hosts accordingly. Replication and synchronization of data would be the responsiblity of the totality of machines providing the cache service, and would augment the storage capacity of ordinary hosts which often replicate data excessively and synchronize it poorly. As Lucky Green correctly points out, since the cache service would effectively serve prepared digital coccoons whose contents were not visible to it, protocols would be needed to ensure that data was not spoofed. However, this is a "who do cache servers trust to give them data" issue, and not a "how do we serve the data and provide a Network-wide consistant view of the database" issue. Such a system, implemented efficiently, could carry the Eternity service, reliable network news, and serve as a distribution point for all freeware. It could have terabytes of storage, using a fraction of the resources now consumed by the endlessly replicated news spools of hosts on the Net. It would be as uncensorable as Usenet, and survive the destruction or compromise of a large number of cache servers in the system. It would be reliable, and no ones data would be visible to anyone else. I think such a service is the "World File System" reduced to its most basic principles, upon which endless other useful services may be based. Comments? -- Mike Duvos $ PGP 2.6 Public Key available $ enoch@zipcon.com $ via Finger $ {Free Cypherpunk Political Prisoner Jim Bell} --------------------------------------------------------------------- Where people, networks and money come together: Consult Hyperion http://www.hyperion.co.uk info@hyperion.co.uk --------------------------------------------------------------------- Like e$? Help pay for it! See Or, for e$/e$pam sponsorship, --------------------------------------------------------------------- ********** Return-path: X-Authentication-Warning: fma66.fma.com: majordomo set sender to owner-espam@lists.espace.net using -f X-Orig-From: lutz@taranis.iks-jena.de (Lutz Donnerhacke) X-E$Pam-Source: owner-cypherpunks@cyberpass.net Date: Mon, 1 Sep 1997 08:08:45 -0400 To: espam@intertrader.com From: Robert Hettinga Subject: Re: Adding Memory to the Net Sender: owner-espam@lists.espace.net Reply-To: e$@thumper.vmeng.com --------------------------------------------------------------------- This mail is brought to you by the e$pam mailing list --------------------------------------------------------------------- From: lutz@taranis.iks-jena.de (Lutz Donnerhacke) Newsgroups: iks.lists.cypherpunks Subject: Re: Adding Memory to the Net Date: 1 Sep 1997 11:44:48 GMT Organization: IKS GmbH Jena Sender: owner-cypherpunks@cyberpass.net Reply-To: lutz@taranis.iks-jena.de (Lutz Donnerhacke) X-Loop: cypherpunks@cyberpass.net * Mike Duvos wrote: >The basic idea here is to add memory to the Net in a reliable >way, so that the Net provides memory services in the way it now >universally provides data transport services. The Net would then Try to find SMMP, Simple Memory Managment Ptotocol. Dejanews (old database) may help. Originated to Kristian Koehntopp. --------------------------------------------------------------------- Where people, networks and money come together: Consult Hyperion http://www.hyperion.co.uk info@hyperion.co.uk --------------------------------------------------------------------- Like e$? Help pay for it! See Or, for e$/e$pam sponsorship, --------------------------------------------------------------------- ********** Return-path: X-Authentication-Warning: fma66.fma.com: majordomo set sender to owner-espam@lists.espace.net using -f X-Orig-From: bureau42 Anonymous Remailer X-E$Pam-Source: owner-cypherpunks@cyberpass.net Date: Wed, 3 Sep 1997 10:27:50 -0400 To: espam@intertrader.com From: Robert Hettinga Subject: Re: Adding Memory to the Net Sender: owner-espam@lists.espace.net Reply-To: e$@thumper.vmeng.com --------------------------------------------------------------------- This mail is brought to you by the e$pam mailing list --------------------------------------------------------------------- From: bureau42 Anonymous Remailer Comments: This message did not originate from the Sender address above. It was remailed automatically by anonymizing remailer software. Please report problems or inappropriate use to the remailer administrator at . Subject: Re: Adding Memory to the Net To: cypherpunks@algebra.com Sender: owner-cypherpunks@cyberpass.net X-Loop: cypherpunks@cyberpass.net -----BEGIN PGP SIGNED MESSAGE----- This is an interesting proposal. It would be easier for people who are new to this business if there were more specifics about how this would be implemented. I'm sure you have particular techniques in mind for splitting the data across servers, etc., but everybody else doesn't necessarily know what they are. Examples are nice because you can go over them one bit at a time. Here are some ignorant comments and questions. Mike Duvos wrote: > Bob makes a secure connection to a Trusted Agent doing business... Why is the Trusted Agent necessary? What does the agent do that Bob couldn't do himself? It seems like Bob could split up the data and pay the servers to hold it just as well as the agent. And he wouldn't have to trust anybody. > For the next 100 days, Bob and all of his friends may use the 256 > bit TAG/CONTEXT pair in place of Bob's data and may serve that data > for free any number of times off any network file system server. Or the people getting the data could also pay small fee for good service. Like a tip, if storing the data itself is the dominant cost. This makes it harder to attack the servers by saturating them with requests for data. Usenet has traditionally been free for posters. This model could continue if there were users who were willing to pay to see other people's posts. Servers would keep the data around so long as there seemed to be people asking for it. You could imagine the price going up over time as fewer people want to see it. At some point not enough people may want to pay as much money so it goes away. In this model the server operators would just be information speculators. And, if The Bad Guys started knocking out servers, there would be a powerful financial incentive to start serving the information that was eliminated since its market price should go up. > After 100 days, bob may fork over another 10 cents for another 100 > days, or his data will disappear. The TAG assigned to an Octet > String is unique, and no two strings have the same TAG, and no > string is ever assigned multiple tags, no matter how often it may be > entered and purged from the system. It would be nice if the ten cents sells a promise by the server to make the data available for some period of time in a way which is independently auditable. If the server stops carrying the data, Bob can publish his proof that he was defrauded. Because the facts would not be in doubt, it would be unlikely to happen frequently. > Users could boot their machines off the Network, and instantly see > gigabytes of software available to them. After syncing their cache > file system with the network, they could drop their PC off the roof, > buy another one, and by typing in just one TAG/CONTEXT pair, see all > their files again. This is very cool. You can hide 256 bits pretty easily, which makes The Great Satan's job harder. > Aside from replication done for reliability, there should be only > one copy of a file in the system at any time. If multiple users > submit the same file, there should be no more copies than if one > user submitted it. Would this really be worth the effort? If Bob wants to pay to put the file up twice, why not let just let him? Just Another Cypherpunk -----BEGIN PGP SIGNATURE----- Version: 2.6.2 iQEVAwUBNAuAZv+1rw4IkyBFAQG3WQf9EBR/peMkdc+0WT5W6O+L9KMf69artRy9 Fg8kCObbHlYEn/rCd0DRJ+Hd9YLfBz8aeo7YqaU3Lawe9+WW5hH0HbSYZe6vXeBN M5VKW4HJpcY4zapc1DCdCPW72KnWZjlrOsP3X+r1yi1KrLlsORNkl3M85dhS5vLz YRoZIIZE51vmrnV3mCA0dClljP2+e4c7FmxtO36spi7n/PPVwkP7xtHqsy762Ex/ iyxyRXCWnPMFHy0tJRWi25HeGqC2IRg22zQ5zk36c6z/JPS92Yabix/rnVWcazP4 3OnmT+nF3xy3mtdx9eMbPTBvxY28uOq+q2k/0/KjNmDjVRPRl2mlvw== =quQD -----END PGP SIGNATURE----- --------------------------------------------------------------------- Where people, networks and money come together: Consult Hyperion http://www.hyperion.co.uk info@hyperion.co.uk --------------------------------------------------------------------- Like e$? Help pay for it! See Or, for e$/e$pam sponsorship, ---------------------------------------------------------------------