Importing tick data

Modified on Thu, 07 Sep 2017 at 02:40 PM

Video guide

Step by step guide with screenshots

In order to import tick data, first of all a new custom tick data source needs to be added. Start by running the Tick Data Manager and select Add new... in the Source dropdown.

In the Add new tick data source dialog that comes up, fill in the Name of the new tick data source (e.g. the name of the MT5 server if you're importing MT5 data) then fill in the Description if you wish. The description is optional and only serves to better identify the source or to store notes related to it.

The Path will be filled in for you automatically while you type the symbol name, you don't need to modify it (and cannot modify it).

Leave the GMT set to 0 and the DST set to None for now, unless you know exactly the GMT and DST configuration of the source data.

Click OK when done.

After clicking OK, the new tick data source will be added and selected in the dropdown. Click the Add symbol... button to add a new custom tick data symbol to your new source.

In the Add symbol dialog, the easiest way to configure the details is by clicking the Autodetect from CSV... button, browsing to the CSV file and finally clicking Open. The Tick Data Suite will open the CSV file, identify the symbol configuration and fill it in for you. Note that this step does not import any data, it merely "glances" at the file to identify the details.

When satisfied with the configuration, click the Add symbol button.

Alternatively, you can manually fill in the data - type the Name, select the number of Digits (fractional decimals) and enable the Ask price available checkbox if the source data has ask as well as bid prices. As before, the Path field is automatically filled as you type the symbol name and cannot be edited.

The newly added symbol will show up in the list of symbols. Click the ... button to its right to open the Symbol operations dialog.

In the Import tab, click the Browse... button, locate the CSV file then click the Import... button to open the Import tick data dialog.

The file format should be automatically detected shortly after the Import tick data dialog shows up, at which point the settings will be configured for you.

If the autodetection fails, chances are there is something wrong with the source file. In the unlikely event that the file format is not identified and the file is ok, you can manually configure the field separator, the date/time columns and formats as well as the price and volume columns.

The bottom part of the dialog is divided in two areas: the upper one shows a raw file excerpt while the lower one dynamically updates the parsed excerpt as the configuration is modified. If the bottom area does not show anything, the import configuration is not correct and you won't be able to start the import process. In case you wish to re-run the automatic detection process all you need to do is click the Autodetect button in the lower left corner.

When satisfied with the import configuration, click the Start import button to queue the tick data import task.

You may see errors in the log area of the Tick Data Manager if the file has problems. The line or the date is mentioned for such errors (although after a certain number of times they are only printed in the debug log); if you want, you can inspect the source file to check and/or correct it then import it once more.

Note that the import process may take a long time, depending on the amount of ticks stored in the source CSV file.

When the import process is finished, the data has been imported to your tick data repository and thoroughly compressed; consequently, the source CSV file is no longer needed. If you have no other use for it, it is safe to delete it at this point.

Identifying the source data GMT offset and DST configuration

If you do not correctly configure the GMT and DST of your new tick data source, the GMT/DST configuration in your Tick Data Settings in MT4 will not result in an accurate time offset. Since many brokers fail to post the GMT and DST configuration they are using on their website, the Tick Data Suite features an automatic way of identifying it by checking the imported data against the existing Dukascopy data.

In order for this to work, the symbol you run the autodetection on must be named exactly like its Dukascopy counterpart and you must have Dukascopy data that overlaps a significant part of the data imported for your custom symbol, preferably all of it although you can get away with a minimum of one year overlap. For example, if you imported GBPCAD data for the interval 2015-2017 and you want to run the GMT/DST autodetection on this symbol, you should download the GBPCAD Dukascopy data for the same period to ensure best results.

To start the GMT/DST autodetection process, click the ... button next to the imported symbol to open the Symbol operations dialog, select the Manage tab and click the Autodetect GMT & DST  button. This will queue a GMT/DST automatic detection task that will go through the symbol data at key dates and compare it against the reference Dukascopy data in order to identify the best match and obtain the GMT offset and DST configuration of your custom tick data source.

When the task is finished, the result will be displayed in the Tick Data Manager log pane. The difference displayed is the average difference between the inspected M1 bars of the custom symbol (realigned to UTC by means of the detected GMT and DST) versus the M1 bars of the Dukascopy symbol. If the difference is minimal and the autodetection task is confident of the results, it will automatically update the GMT and DST of your custom tick data source for you.

If you do not see the message that informs you the values have been updated and you believe the values to be correct (you can divide the difference by the symbol pip to get an idea), you should update the source configuration manually by clicking the Properties button next to the Source dropdown and selecting the detected values in the GMT and DST dropdowns.

Using the imported data in Metatrader 4 backtests

In order to use the newly imported data in MT4, start by selecting the MT4 symbol in the backtest pane; after that all you need to do is open the Tick data settings dialog and select the new tick data source in the Source dropdown. If the symbol you imported is a Forex symbol that does not have an out of the ordinary name, chances are it will be automatically selected for you. If it's a CFD or the Symbol dropdown remains blank, manually select the symbol from the dropdown.

If your MT4 terminal was running while you were importing the data, you must click the Refresh button to make sure the new tick data source and symbol are present in the dropdowns.

Once the new tick data source and the symbol are selected, configure any other settings that you may want to change, click the Ok button and finally click the Start button to start your backtest using the newly imported tick data.