zsync — Optimised rsync over HTTP

Colin Phipps

(work in progress)

Abstract

This document describes the thinking behind zsync, a new file transfer program which implements efficient download of only the content of a file which is not already known to the receiver. zsync uses the rsync algorithm, but implemented on the client side, so that only one-off pre-calculations are required on the server, and no special server software or new protocol is required to use zsync.


Table of Contents

1. The Problem
File Transfer
Existing Methods for Partial File Transfer
Compressed Files
The Ideal Solution
2. zsync Theory
Rsync on the Client Side
The zsync Control File
Checksum Transmission
Weak Checksum
Strong Checksum
Match Continuation
rsync Speed
Negative Hash Table
Networking
Comparison with rsync
Empirical Results
3. Compressed Content
Looking Inside
Mapping Deflated Files
Is It Worthwhile?
Compressed Delta Transfer
Compression Results
Recompression
4. Implementation
libzsync
Control File Generation
zsync Client
Security Considerations
Work To Do
Bibliography