Can NT cross validate in sample?

[grapgsuh39]

NT is powerful. I'm surprised that NT doesn't offer in sample cross validation. Or I'm surprised that none of my searches (I've searched high and low) revealed this ability or extensions or anyone else wanting it.

Cross validation prevents the optimizer from over fitting curves to data.
Definition of cross validation ... https://en.wikipedia.org/wiki/Cross-...8statistics%29

1) NT can optimize/train on one set of data. 2) During optimization NT should check the fitness (over fitting & under fitting) using a cross validation set of data which is "in sample". 3) Then as a third step, it should apply the parameters to out of sample data that was not used for optimizing or validating.

NT does steps 1 and 3 (room for improvement) but it doesn't seem to do step 2, cross validation. The room for improvement is this manual work should be automated ... "you would need to save your input parameters and re-run the test for the second date range and compare the results" in this thread ...

http://ninjatrader.com/support/forum...+out+of+sample. Also, Walk forward is not the same as cross validation.

With in sample cross validation NT optimizer will avoid over fitting.

[grjvedxu89]

Am I overlooking this capability? or
Is there a trivial (not labor intensive) work around for this? or
Is there an addon for crossvalidation? or
Has NT user built one? or

Can NT give guidance on where to start adding it? Perhaps extending @GeneticOptimizer.cs? Would it be possible to code a Performance Type to test half way into the walkforward range to simulate a cross validation test? Then the performance type would sum the peroformance of the optimizer plus the cross validation (half walk forward) to get a total performance?

[Hebgakhgad]

Thank you for your post.

Currently Cross-Validation is not used or an option. I am not aware of an add on for this as I would venture to guess it requires creating a new Strategy Analyzer in order to perform this function. The Strategy Analyzer code is not exposed other than the Optimization Types/Fitnesses which would not offer us the needed options for the Cross-Validation.

I will forward this to development as a suggestion and for their insight.

[gwjqqbrp11]

After years of research, now cross-validation is considered a required part of training/cross/test routines in all forms of data analysis, even trading. Given that, I'm stunned that NT doesn't have cross-validation.

Having Cross-validation will take NT optimizer to another level of reliability.

I've built my own Performance Type. So I'm getting the feeling that there just might be a way to quasi-cross-validate.

The key question is how can Type look into half of the Walkforward range?

NT has some dynamite folks like Bertrand, Josh, et al. So I'm hoping they can get me started with above question.