Add AIC (Akaike Information Criteria)

[villyc]

We need to give users information about how to avoid over fitting when doing the time series fitting. For this, we need a new routine for the time series fitting form. There are two options for how to do this, a simple and a more complicated which involves a series of runs. I'll describe both:

The simple version

We calculate the

AIC = 2 x number of search parameters + number of data points x ln (SS)

The number of search parameters is the number of vulnerabilities we estimate + the number of spline points + the number of years we find primary production anomalies for. The number of data points is more tricky. In principle it is:

Relative biomass = # data points –1 
Absolute biomass = # data points 
Catches = # data points

This is however likely to overestimate the real number of datapoints because of correlation between them. Numbers in a data series are not independent observations. Carl suggests that we divide the number of datapoints with 5. I'm inclined to say that each time series gives us 1-2 data points. In any case, I'll give directions for what we do, still discussing. So, we need to make this an entry with a default value, which users can overrule. On the form, we thus need: Data points: AIC = The easy way is to just write the AIC in the iterations field after a search is done.

The more complicated version

see the attached spreadsheet. I made the analysis here, and it can be done fairly simple as a batch run. For this we need to run through:

First make a run with time series loaded, calculate SS, then AIC with no search parameters

If vulnerability is checked then Reset vulnerabilities If anomaly is checked then reset the forcing function in use

If vulnerability then first run sensitivity, and find out the order in which to include predators in search. Search for vulnerability for the most sensitive predator; record SS and AIC; reset vulnerability ; search for the two most sensitive predators, etc until we search for all consumers.

If anomaly search is checked (but not vulnerability search): search for 0 spline points (done already, it's the same results as before we set any search parameters), then try 2, 3, 4, 5, 10, 15, 20, 25, 30, 35, up to number of years – 5. Then make an anomaly search with spline = 0, I.e. Search for primary production anomaly for all years.

Finally if both vulnerability and anomaly are searched, the leave the anomaly as is, I.e. If it is e.g., 3 spline points then we only do those 3 spline points. We now do exactly the same as for the vulnerability search above, only we reset both vulnerabilities and the forcing function before each run.

Perhaps we can just dump the results from this to a csv file?

[jeroens]

AIC complicated version