One of the greatest appeals and advantages of mechanical trading systems is the ability to evaluate their historical performance by “backtesting” the strategies on historical price data. While we may have just a handful of months of actual performance data available, computers and backadjusted data make it possible to see what a system “would have done” going back years and years.
The problem, of course, is that the system has been designed on this very same data. Whether intentional or not, because systems are designed on past data, they are often the victims of what we call “curve fitting”, making the ability to backtest results one of the biggest disadvantages of trading systems also. Because the future may look nothing like the past in a particular market – the “fitting” of parameters onto the past “curve” of data may cause big problems on the future data curve, causing the system to be out of phase and potentially causing investors losses.
The easiest way to understand “curve fitting” is through a simple example. Imagine a system that buys or sells Soybean futures on a breakout above or below the market high or low for the past X number of days. When testing the system on past data, the testing may show $5,000 in profits when using a 10 day high/low, $10,000 in profits when using a 20 day high/low, and $20,000 when using a 30 day high/low.
If you were the developer, which value would you use in designing the system, 10, 20, or 30? I would guess most people would use the 30 value, as it gives the highest profit. Now a developer will look at more than just profit, and test for lowest drawdown or most winning months, for example; but whatever your goal for the system, it is human nature to design a system whose parameters produce results as close as possible to those desired. The problem is, just because one parameter worked on the past data does not mean it will work on the future, unknown data. So how would a developer attempt to avoid such a problem?
Another example of curve fitting is adjusting parameters after the fact. Imagine a trading model which is doing well, but suffers a rather big loss on three out four Wednesday afternoons since live trading began. A trader might look at those results and come up with a brilliant plan, code the system not to take any trades on Wednesdays after 1:30 pm. Running the code backwards after putting in the new logic would result in those Wednesday losing trades going away, and voila – you have curve fitting.
To combat curve fitting – developers use many tricks of the trade such as testing on out-of-sample data, not optimizing parameters for the best backtested results (instead using logic based parameters – such as a 5 day moving average to align with the week), and making sure there are as few parameters as possible in the system (more parameters equals more degrees of freedom which equals more things to potentially go wrong).
Next week, we’ll delve into how an investor can make sure they are trading a “Curve-fit free” Trading System.
IMPORTANT RISK DISCLOSURE
You should fully understand the risks associated with trading futures, options on futures, commodity trading systems and retail off-exchange foreign currency transactions (“Forex”) before making any trades. Trading futures, options on futures, Forex and commodity trading systems involves substantial risk of loss and is not suitable for all investors. The ability to withstand losses and to adhere to a particular trading program in spite of trading losses are material points which can adversely affect investor returns. You should carefully consider whether trading is suitable for you in light of your circumstances, knowledge, and financial resources. You may lose all or more than your initial investment. Opinions, market data, and recommendations are subject to change without notice. Past performance is not necessarily indicative of future results.
The returns for trading systems listed throughout this website are hypothetical in that they represent returns in a model account. The model account rises or falls by the average single contract profit and loss achieved by clients trading actual money pursuant to the listed system’s trading signals on the appropriate dates (client fills), or if no actual client profit or loss available – by the hypothetical single contract profit and loss of trades generated by the system’s trading signals on that day in real time (real-time) less slippage, or if no real time profit or loss available – by the hypothetical single contract profit and loss of trades generated by running the system logic backwards on backadjusted data (backadjusted).
The hypothetical model account begins with the initial capital level listed, and is reset to that amount each month. The percentage returns reflect inclusion of commissions, fees, slippage, and the cost of the system. The monthly cost of the system is subtracted from the net profit/loss prior to calculating the percentage return.
If and when a trading system has an open trade, the returns are marked to market on a daily basis, using the backadjusted data available on the day the computer backtest was performed for backtested trades, and the closing price of the then front month contract for real time and client fill trades. For a trade which spans months, therefore, the gain or loss for the month ending with an open trade is the marked to market gain or loss (the month end price minus the entry price, and vice versa for short trades).
The actual percentage gains/losses experienced by investors will vary depending on many factors, including, but not limited to: starting account balances, market behavior, the duration and extent of investor’s participation (whether or not all signals are taken) in the specified system and money management techniques. Because of this, actual percentage gains/losses experienced by investors may be materially different than the percentage gains/losses as presented on this website.
Please read carefully the CFTC required disclaimer regarding hypothetical results below. HYPOTHETICAL PERFORMANCE RESULTS HAVE MANY INHERENT LIMITATIONS, SOME OF WHICH ARE DESCRIBED BELOW. NO REPRESENTATION IS BEING MADE THAT ANY ACCOUNT WILL OR IS LIKELY TO ACHIEVE PROFITS OR LOSSES SIMILAR TO THOSE SHOWN; IN FACT, THERE ARE FREQUENTLY SHARP DIFFERENCES BETWEEN HYPOTHETICAL PERFORMANCE RESULTS AND THE ACTUAL RESULTS SUBSEQUENTLY ACHIEVED BY ANY PARTICULAR TRADING PROGRAM. ONE OF THE LIMITATIONS OF HYPOTHETICAL PERFORMANCE RESULTS IS THAT THEY ARE GENERALLY PREPARED WITH THE BENEFIT OF HINDSIGHT. IN ADDITION, HYPOTHETICAL TRADING DOES NOT INVOLVE FINANCIAL RISK, AND NO HYPOTHETICAL TRADING RECORD CAN COMPLETELY ACCOUNT FOR THE IMPACT OF FINANCIAL RISK OF ACTUAL TRADING. FOR EXAMPLE, THE ABILITY TO WITHSTAND LOSSES OR TO ADHERE TO A PARTICULAR TRADING PROGRAM IN SPITE OF TRADING LOSSES ARE MATERIAL POINTS WHICH CAN ALSO ADVERSELY AFFECT ACTUAL TRADING RESULTS. THERE ARE NUMEROUS OTHER FACTORS RELATED TO THE MARKETS IN GENERAL OR TO THE IMPLEMENTATION OF ANY SPECIFIC TRADING PROGRAM WHICH CANNOT BE FULLY ACCOUNTED FOR IN THE PREPARATION OF HYPOTHETICAL PERFORMANCE RESULTS AND ALL WHICH CAN ADVERSELY AFFECT TRADING RESULTS.
The information contained in the reports within this site is provided with the objective of “standarizing” trading systems account performance and is intended for informational purposes only. It should not be viewed as a solicitation for the referenced system or vendor. While the information and statistics within this website are believed to be complete and accurate, we cannot guarantee their completeness or accuracy. As past performance does not guarantee future results, these results may have no bearing on, and may not be indicative of, any individual returns realized through participation in this or any other investment.