If you’ve been researching futures, commodities, currencies, or stocks trading, you’ve probably been exposed to backtest results. These results supposedly show what a trading method would have done in the past, if it had followed it. They are one of the reasons the US government requires the phrase “past performance is no guarantee of future results” when discussing business systems or approaches.
The backtests are hypothetical and may or may not have been traded for real money. It is even possible that it is IMPOSSIBLE to perform the operations shown in a backtest.
As a seasoned systems developer, in 5 minutes with my Tradestation test software, I can build a system in any market with a jaw-dropping backtest, it would look so good. META, would fail in the future – practically guaranteed! Believe me, many developers create systems in exactly this way and then try to sell you their “secret”.
So why are backtests so unreliable? Four reasons come to mind.
1) Optimization: Most test software has an optimization function, which will select the best set of parameters, based on the above data. Most developers abuse this feature. What worked best in the past is unlikely to be the best in the future. Consequently, falling in love with these over-optimized backtests, you think you’re buying a Mercedes, but you’re actually getting a Yugo.
two) Hindsight bias: It is difficult to create a system without “peaking” in the data. Since maximum data cannot be reached in the future, a developer who does this during development is, in effect, cheating. Many times, people do not even realize that they are doing this; It can be such a subtle mistake.
3) Software limitations: The software itself has limitations that allow for unrealistic or unattainable fills. For example, systems with market closing orders are very likely to include unrealistic executions, as the order can be submitted after the market closes (and is never filled), but the software still thinks it was filled.
4) No real-time performance – Developers post their own backtest results, make them appear real, and have no independent real-time verification of their results. Can you really trust the backtests of the same person who is trying to sell you the system?
What is the solution to this backtesting dilemma? Simple, if you are dealing with a CTA (commodity trading advisor), hedge fund, or mutual fund, make sure you see real records audited in real time. If you are dealing with a systems developer, make sure the results you see are independently verified and verified, preferably with results based on real money accounts (not demo). I list two of these websites at the bottom.
It’s easy to be seduced by extraordinary backtest results. Just remember that those results may not be real.
Websites like worldcupa investments.com or Collect2.com are just two of many websites that offer third-party verification of trade results.
