Swarm Merging

From VuzeWiki
Jump to: navigation, search

Since 5501_B34 Vuze has supported 'swarm merging' across torrents that share common files. Vuze automatically detects when two or more of your incomplete downloads share one or more files of identical size and will attempt to merge the torrent swarms to download the file faster or, possibly, complete an existing file with bad availability.

If merging completes successfully you will be informed via a popup indicating how much data was transferred between the file clones - likewise, if it fails, a popup will indicate that the merging process has been abandoned.

Since 5501_B36 a new column is available that shows the overall number of bytes that have been merged into a download from another download that shares one or more common files:

RTENOTITLE

This shows two downloads, one with no seeds and another that is well seeded. Data is being exchanged between the downloads (3.35MB has been transfered from the second download/swarm and 400KB in the opposite direction), The overall download of the content is proceeding at 41.6 kB/sec + 25.6 kB/sec -> 67.2 kB/sec, demonstrating a good increase in download rate. Also, as the download proceeds, the swarm with no seeds will have a good chance of being 're-born' as fresh content is made available to it.

Since 5711_B15 you can enable swarm merging to include complete files as well via a sub-option in Tools->Options->Files. This will result in all copies of a file being merged as opposed to the existing 'at least one copy' approach when not enabled. This is of benefit when you want to seed already completed files into other incomplete downloads. At the same time that this option was added the piece selection algorithm was also extended to prioritize the acquisition of pieces in one file that are unavailable in another, thus speeding up the cross-pollination of dead swarms.

Usage[edit]

Once you have incomplete downloads that share a common file running within Vuze, everything is automatic. So to exploit this feature you just need to get the downloads running. Points to note:

  1. The downloads must be incomplete and actively downloading (once a download is complete you have all the data so merging is not required; if one of the downloads isn't in a downloading state the swarms can't be merged). Since 5711_B15 complete and actively seeding (queued doesn't count) files will also be considered for merging if you have enabled this option.
  2. You (yes you!) need to arrange for the multiple downloads to be added to Vuze (see below for some information on finding duplicates)
  3. Each download requires its own storage when running - data is replicated between the copies but there will indeed be separate per-download copies of the data.

Finding Duplicate Files[edit]

Swarm discoveries can help you detect torrents containing files of the same size as existing files in two ways:

  • Right-click on a file and select 'Swarm Discoveries: By File Size':


SwarmDiscoveriesBySizeMenu.png


  • You can also directly enter the file size in bytes by right-clicking on the Swarm Discoveries side bar entry and selecting from the menu:


SwarmDiscoveriesBySize.png


Note that this is only available for files above a certain size.

Download-specific decentralized chat channels can also be used to report other torrents that (may) contain duplicate files. Keep you eye out for a flashing message notification icon in the status area:

ChatStatusIcon.png

Double-clicking this will open any chats with outstanding messages (or right-click and select one of interest)

Small Print[edit]

The test for files being potentially equivalent is solely based on the file sizes. While it is possible to use the file names themselves to derive some kind of 'match metric' (e.g. same file type, common words in file name) this has not been done and the exercise is left to the reader.

Due to the way that multi-file torrents are handled within torrents there are generally block alignment issues between common files that can cause merging difficulty with the first and last blocks of files.

Example[edit]

Here is an example involving the download of the popular Libre Office software by torrent.

The torrent is available here: http://download.documentfoundation.org/libreoffice/stable/4.4.0/win/x86/LibreOffice_4.4.0_Win_x86.msi.torrent

Opening this in Vuze will, assuming you haven't disabled the 'open torrent options dialog' (see Tools->Options->Files), show the following dialog:


SwarmMergeEx1.png


You can see the torrent contains a single file of size 213.56MB. To search for other torrents that potentially contain the same file select it and hit the 'Swarm It!' button. This will open a Swarm Discoveries view in the client that searches for the file:


SwarmMergeEx2.png


In this case it has found a bundle that contains both the original file and a 'Help Pack'. When you select this result and hit the 'Download' link in the 'Actions' column it will add this torrent as a second download to the torrent options dialog:


SwarmMergeEx3.png


Hitting the 'OK' button will add both downloads to Vuze and merge the swarms from the two downloads:


SwarmMergeEx4.png


You can see that the aggregated download rate for the file is around 2.2MB/sec with 1.2MB/sec coming from the original torrent and an additional 1.0MB/sec coming from the second one.