Everything you need to know about backtesting: Historical Data

This article you will show you what type of data you need to load into your MetaTrader, to get reliable backtest results. Are you looking for a reliable Historical data source? Would you like to know how to install historical tick data? Continue reading, and you will get to know everything…

In our previous post we showed the proper settings of MT4, we introduced the use of the Strategy Tester, and we came up with a rally car example to illustrate the whole process. In this article, we will tell you how to download and use good quality historical forex data.

Historical Forex Data

“The type of ”fuel”, you should “fill” into your robot”

While the proper settings of MT4 can be done very easily, downloading and installing historical data is a more complicated process.
We usually have two problems with the data:
– We don’t have enough historical data.
– We have low quality historical data.

Insufficient historical forex data

You may have experienced that you wanted to start the test from one year ago, but in spite of the correct date setting your test just started from a few months ago. MetaTrader 4 always downloads the data from a central server, which can be run by your broker or by MetaQoutes, depending on which broker’s platform you use. It doesn’t matter which broker you choose, you will be able to download only a few months of data, because the providers always delete the old ones. The reason for this is because good quality forex data takes up a lot of storage space, and the brokers want to reduce the traffic of their servers. The available quantity of historical forex data depends on the timeframe you chose, therefore we can’t tell you how much data you will be able to download.
Although the data is always deleted from the server, it won’t be deleted from your computer. If you use a MetaTrader continuously, then you can collect the data: Let’s say you install a MT4 today, and you always download the last 3 months’ data from the server, then a year later, you will be able to test on 12+3=15 months of data.

Low quality forex data

The quality of historical forex data is also important, because if we test on low quality data, then we will have the same problem as in Control point testing. (Here we showed the example of the solider whose map does not contain all of the mines)

aknamezo

What is tick data?
Tick data is the data that contains every single change in price. Non-tick data are also available, like 1 minute candlestick data. If we want to get a candlestick, we need to know at least its 4 points : High-, Low-, Open-, and Close-prices. As we introduced in our previous post, our result can be influenced by the route that the price defines, before it reaches the closing price of the position. So if we want to get reliable results, we need to know far more points of the candlestick.
The tick data contains every single value of the change in the price, including the open and close prices of the candlestick as well, giving us more reliable backtest results.

We can get feedback about the quality of our data by viewing the upper part of the “Report” tab.

modelling-quality

The situation that can be seen above is one of my personal “favorites”: The platform was unable to detect the data quality, therefore it indicated n/a.
According to our experience, the best quality that we can achieve with downloaded MetaTrader4 data is 90%.

How to get 99.9% modelling quality on MT4

If we want to get reliable backtest results, we need to use good quality historical data. Good quality historical data means it contains all the ticks, there are no gaps in it, and it is broker specific. In our previous article we showed that there are brokers who change the price 25 000 times daily, while there are others who do it 75 000 times in a day. This disparity is almost like the difference between the “Every tick” and “Control points” simulation method. But that’s not all! Since the forex market is not a centralized market, there could be an important difference between the prices of two brokers.

historical-data-downloading

In the picture above you can see the charts of two companies in the same time, on the same M15 timeframe. There is such a huge difference between the prices that even the signals of the indicators are different: While the big candlestick closed below the MA in the left picture, it closed above it on the right.
As can be seen at the bottom of the pictures, running the same EA on both charts gives us completely different results.
The tick volumes and the price differences can cause such a huge difference in the performance of the robot that we experience that it is profitable at one broker while resulting in loss at another.
And we are not able to foretell which provider is better. Obviously, the one where the EA works best! That is why we recommend testing your own EA on the tick data of different broker companies.

Where can we download good quality tick data?

Definitely not from the server of the broker, because they can upgrade the data quality afterwards by deleting the spikes, or they can delete some ticks, to reduce the size of the files.
In the last few years the Dukascopy’s source was a good solution. With the use of this source, we were able to reach 99% backtest results, but this data was from Dukascopy’s STP system, which is directly not available from MetaTrader4.
Forex Broker Stars offers a better solution: It not only provides 99% quality , it also provides broker specific tick data.

Testing our EA on different broker’s data, we can get to know our EA’s sensitivity to data quality, and we can decide which broker is the most suitable for it.

It is worth mentioning that there are other data sources as well, but we won’t introduce them, because they have only 1 minute data (instead of tick data), or we have no experience using them:
ForexTester
HistData

Importing Tick Data into MetaTrader 4

We can usually download tick data in CSV file format. The MT4 is unable to read such files directly, so we need to convert them into another format.

The data importing issue is not the only problem we face when we use an external data source: MT4 tries to download tick data immediately after if it is started, and this overwrites our imported files.

This means there is no sense in copying the data from one platform to another, because the MT4 will overwrite them immediately after start up.

TickStory and TickDataSuite provide good solutions to this issue. Both softwares are able to convert the CSV files to a proper format, and they are able to change the operation of the MT4 so as not to overwrite our new files.

What is the difference between TickStory and Tick Data Suite?

Price: While the price of Tick Story is a one-time $23.95 fee, the price of TickDataSuite is a one-time $97 single fee plus a monthly $10 fee.

Downloading data: Only Tick Story has an internal data downloader tool, which uses the mentioned Dukascopy data source. It is a good advantage because we don’t need to install and sign up into the Dukascopy trading platform. Unfortunately Tick Data Suite doesn’t have this tool, but both programs are able to handle Forex Broker Stars tick data.

Handling: Tick Data Suite provides us a separate script with we can convert our CSV files into the proper format for our MT4. To avoid any overrides we need to paste a new exe file into our platform’s folder, and then start it with this exe.
TickStory has a built in data downloader and data converter tool, which provides protection against overrides as well, but this could be a disadvantage of the program: It has a lot of features, and it can be a bit difficult to understand and use.

However, independent of which program we choose, the key of our success is the use of good quality tick data!

Summary:

We don’t need to find any “holly-grail” if we want to be successful. Rather we need to pay attention to every single detail like money management, the signals of the indicators, psychology, and the strategy backtesting. In order to have reliable backtest results, we need to pay attention to every advice mentioned above.

Drop a comment

Your email address will not be published. Required fields are marked *

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close