Backtesting - is it a valid concept ?

I wonder what others think about the concept of backtesting ?
One the one hand it perports to find the best set of parameters for a given method and thereby tune it to the period tested.
If the results are good, then these parameters can then be used with some (?) confidence for live trades.
On the other hand one could argue that backtesting is just curve fitting and therefore simply 20:20 hindsight.

Seems to me that for a backtest to have any validity, it must produce a smoothly increasing equity (profit) curve with no wild girations. This at least demonstrates that the method and parameters found worked fairly consistently over the period tested.
Of course a lot depends on the period(s) selected.
Again to have any validity, I think that a test must either be run over a considerable time period, embracing many different market conditions and yet still remaining consistent, or it must be run on selected market conditions with a view to using the optimised parameters from the test in that type of market.

To disprove the theory of curve fitting, one suggestion is to take a period of time, say 5 years, and run your test over the first half of the data to get some results.
These results can then be run over the second half of the data as if it were the unknown future. i.e. a kind of Forward Testing.

One thing's for sure, the Software vendors must love things like backtesting. What an opportunity to develop and sell a nice technical product, using the math co-processor and the disk drive of your PC to the full. A nice 'gadget'.
Lovely jubbly - at least for the vendors.

Enough from me. What do others think? In particular anyone who has used a backtested system in the real market sucessfully.

Glenn
Glenn is offline

Glenn,

Before you ask about back testing being of any value , you have to ask yourself another more important question..
DOES PAST CORRELATE tO FUTURE ?

You can only have three answers .


1) yes it does..
2) No it does not
3) it does to some degree
if you think number 3 is the answer then your back testing will be effective to the same degree ..
Grey1 is offline Commercial chat room

One more question
"and for how long" which I suppose is really a subset of "some degree"
__________________
Rognvald

Grey1
"DOES PAST CORRELATE tO FUTURE ?"

Agree the question, hence my thought about Forward Testing as a means of finding out.
But are you suggesting that one's degree of belief in backtesting is only a guarantee of good backtests, rather than good future results ?

Rognvald
"and for how long" ?
Again I'm thinking that forward testing may help to determine this to some degree i.e. within the bounds of available historical data.

One way or another most traders are looking for something which works enough of the time to enable money management to produce a profit.
So how do they find something which works 'enough of the time' ?
And yes, how will they know when it stops working (apart from the obvious) ?

Glenn

Hi Glenn,

Solve that and you've solved the worlds problems.

In short there is no answer.

JonnyT
__________________
Trading Mentor