The problem
Different tick data sources store data in different formats. To provide a couple of extreme examples:
- Dukascopy stores the data in a proprietary compressed format, using one file for each hour of each day for each symbol - 24 files per day work out to a total of 120 files per week or 480 files per month per symbol.
- On the opposite end, TrueFx stores the data in files that cover a whole month of data for each symbol.
For cases like TrueFx (#2) which have the data concentrated in larger files, the download speed tends to be better even though the processing speed suffers because a single thread can be used for each large file.
However, for situations like Dukascopy (#1 above), the Tick Data Suite has to download 24 files for each day worth of tick data for each symbol. These files have to be decompressed, they have to be decoded, the data has to be processed (sorted, duplicates removed, spikes detected and filtered) and finally converted to the Tick Data Suite format and compressed using a proprietary compression algorithm.
As you can probably imagine, this process can take a lot of bandwidth and download time because instead of having the data concentrated in a larger file there have to be a lot of small requests and responses. On top of that, the whole data process chain mentioned above tends to use a lot of CPU power. This also means that having a slower CPU (e.g. laptop, VPS etc.) will result in longer download times.
By default, the Tick Data Suite is configured with moderate settings that should not have a major impact on your bandwidth usage or CPU load for a typical PC. That means you can still do other complex activities (such as playing an online game) while you're downloading data. However, it is possible to optimize the download speed at the expense of higher bandwidth usage and higher CPU loads.
The solution
Fortunately, it is easy to increase the download speed.
In order to do that, open the Settings dialog located in the ☰ popup menu in Tick Data Manager.
Once the dialog is opened, increase Max processing threads and Max concurrent downloads to the desired values and click OK.
The Max concurrent downloads setting governs how many files can be downloaded in parallel. This refers to the files for a single symbol that were discussed above (the 24 files per day for Dukascopy). It's important to note that the Max concurrent downloads setting will not allow downloading data for several symbols at once. Increasing this setting to 8 (which is the maximum) simply means that a Dukascopy EURUSD tick data download will download 8 EURUSD files at a time, each of them spanning one hour.
The Max processing threads setting governs the maximum amount of threads that can run at the same time in order to process the data that is being downloaded. For Dukascopy, oftentimes downloading is faster than processing so it makes sense to have more processing threads than concurrent downloads.
Finally, it must be mentioned that the new configuration does not apply to any tick data downloads that are ongoing or queued. If you have such downloads, you must cancel all of them (remove them from the queue) then restart them.