tazzbit writes: The storage vendors have been crowing about data deduplication technology for some time now, but a new open source project, Opendedup, brings it to Linux and its hypervisors — KVM, Xen and VMware. The new deduplication-based file system called SDFS (GPL v2) is scalable to eight petabytes of capacity with 256 storage engines, which can each store up to 32TB of deduplicated data. Each volume can be up to 8 exabytes and the number of files is limited by underlying file system. Opendedup runs in user space making it platform independent, easier to scale and cluster and can integrate with other user space services like Amazon S3.
Programmers used to batch environments may find it hard to live without
giant listings; we would find it hard to use them.
-- D.M. Ritchie