#1 RFE: Parallelize file reads during patching

David Coppit — Fri, 10 Oct 2014 14:14:51 -0000

What is the underlying reason for this request? I assume it's for better performance. On my system I have an SSD, and for me the performance bottleneck is CPU. But I wouldn't be surprised if even for spinning disks the bottleneck is CPU.

RFE: Parallelize file reads during patching

tromer — Thu, 09 Sep 2004 09:15:52 -0000

During patching, librsync currently reads old blocks
sequentially, waiting for each to be retrieved before
retrieving the next one. In highly mixed deltas (such
as those generated by the test script of bug 1022764, a
re-sorted database, a filesystem restored from tar,
etc.), many of these operations require a seek in the
underlying basis stream. When this stream is backed by
a file residing on hard disk, this can make patching
extremely slow.

If librsync is changed to perform many reads in
parallel, it will give the storage system an
opportunity to order the physical disk accesses more
efficiently through the usual mechanisms: kernel
elevator algorithm, parallel access to disks on a RAID
device, TCQ on SCSI or SATA drives, etc.

On the librsync side, this will require replacing the
rs_copy_cb callback interface by a more complicated
interface, that supports submission of multiple read
requests and querying which was completed. The
application will then do whatever it takes to implement
that; in rdiff's case, it probably means threads or forks.

Some of this functionality can be separated into a
separate library, which may be of independent interest.

Recent changes to feature-requests

#1 RFE: Parallelize file reads during patching

RFE: Parallelize file reads during patching