Symptoms
You run two backtests with seemingly identical conditions and the results produced are different.
The two backtests can be:
- Consecutive in the same terminal.
- Using the data from the Tick Data Suite in two separate terminals, connected to the same broker/account or not.
- Using the Dukascopy data from the Tick Data Suite in one case and the data from Tickstory in another case.
Cause
Without exception, this is caused by differences in the backtest configuration or environment.
Resolution
1. If you're running the same backtest repeatedly in the same MT4 terminal and getting different results:
- First thing you should check is the spread - make sure it is not set to Current, in which case each backtest will be using a different spread if you're backtesting during the week on a variable spread broker.
- Quite commonly, this can be an issue that stems from the MT4 backtesting engine using the current price for converting the backtest results to the backtest currency of your choice; for example, if you backtest GBPCAD with EUR as backtest currency, the current EURCAD price will be used to determine the tick value. Since the market price fluctuates, the results will be slightly different from one backtest to another. To avoid this issue, simply use the quote currency (CAD in the above example) as the backtest currency.
- Another issue might be slippage - if you enabled slippage but disabled the Reproducible slippage checkbox in the Slippage tab of the Tick data settings dialog, you will get different slippage for every backtest run. You need to enable the Reproducible slippage checkbox if you want consistent results.
- A very rare case may be if your EA is reading bar data from symbols other than the backtest symbol or runs indicators on such symbols (e.g. your backtest runs on EURUSD but has an indicator running on USDCAD or for instance reads iClose for USDJPY), you need to have HST data that is in sync with your tick data. In order to do that, you need to run a backtest of any EA on any timeframe using any date range on each of the symbols involved using the same GMT and DST that you intend to use in your main backtest.
2. If you're comparing two backtests obtained with different terminals or different data, I suggest backtesting the attached EA using the same settings in both terminals or on the same data sets. The EA will print some data specific to each backtest in the backtest journal - copy the results for each backtest somewhere and compare them (you can use something like the online Text Compare! tool if you don't have some software to help you with that). Some particular cases:
- If the key prices displayed are different, you are most likely using different GMT and/or DST settings.
- If the number of ticks in the backtest are different between data sets, keep in mind that the Tick Data Suite applies duplicate tick removal unless you manually disabled it. Duplicate tick removal is the process which identifies series of 2 or more consecutive ticks that occur within the same minute and have identical ask/bid prices and only keeps the first tick in such a series, removing the others as they are not needed and in the vast majority of cases have absolutely no impact on a backtest other than increased duration.
- Other details that I recommend looking at, in order of most commonly causing the difference: the spread, the commission, stop/freeze levels, tick value, profit calculation mode, stopout, leverage, margin calculation.
- Note that EAs that are almost continuously in the market (e.g. Martingale, grid etc.) will typically exhibit huge differences if even a single trade closes at a different time or price - such small changes can cascade throughout the whole backtest resulting in dramatic differences at the end.
- If all the details displayed by the attached EA at the end of each backtest are identical but you're still getting different results with your EA, I recommend ensuring that you are using the same version of the EA in both cases.