Shifting the tick data to the past

Modified on Wed, 10 Feb 2021 at 01:09 AM

Why shift?


Some EAs are programmed to cheat backtests. There are quite a few ways to achieve such an outcome and one of the most frequently used is to hardcode dates or periods of time that have a significant impact on the backtest results.


For example, an EA might stop trading during a time interval where most of its activity results in losses; a trend-following EA might have trend start dates hardcoded and open positions on these dates; another EA may have some particular dates blacklisted.


All the above examples have the same outcome: the backtest results are changed in favor of the EA.


There is however a special case when an EA has a legitimate reason to skip some hardcoded dates & hours: if the EA has some sort of news event avoidance in its live trading (for instance it avoids Nonfarm Payrolls by means of the FFCal indicator) it might also hardcode past dates with news events that it would have avoided in live trading. The alternative would be to make use of a news database in backtesting which is certainly possible but it's not at all easy to implement and unfortunately it is beyond the skills of most EA developers.


The 28 year shift to the past


The only tick data shift that is realistically possible is to the past by 28 years. The main reason for that is the fact that shifting by this value will result in a 100% match between the dates and the days of the week.

If we would shift by a different value - for instance one year - we would have tick data on weekends while it would be missing during some days of the week, the triple swap would be charged on the wrong day and so on.


Sure, it would in theory be possible to shift the data to the future by 28 years and it would still result in a 100% date to day of the week match but we would run into the y2k38 problem. Some of you may remember the y2k problem; y2k38 (also known as the Epochalypse) is going to be something similar - perhaps even worse. Long story short, sometime during 2038 unix epoch timestamps will exceed the range of a 32 bit integer variable which will affect various systems in various ways.


The procedure


If you have Tick Data Suite v2.2.37 or above, all you need to do in order to enable shifting data to the past by 28 years is enable the Shift data to the past checkbox in the Expert tab of the Tick data settings dialog.


Some things have to be taken into consideration:

  1. In order to prevent accidentally forgetting this checkbox enabled for an extended period of time it is configured as a one-shot.
    In other words, as soon as you run a backtest with the data shift setting enabled it will automatically disable itself.
  2. If you wish to enable the Use date feature in the MT4 strategy tester pane, you must also shift the Start and End dates to the past by 28 years.
    For example, if you wish to backtest 2010.01.01 to 2020.01.01 with the data shift enabled, the interval you should configure is 1982.01.01-1992.01.01.


Outcome


If an EA has no hardcoded dates at all, the backtest results should be close to identical if you run the same EA with the 28 year shift enabled or disabled. That is, of course, assuming all the other settings remain unchanged.


It's important to keep in mind that as mentioned above, some EAs may have a legitimate reason to hardcode dates for news event avoidance. But seeing that this would only apply to EAs that also have news avoidance in their live trading it's still pretty easy to tell if an EA is trying to cheat or not.