Importing lots of Time Series

[jeroens]

Some EwE5 databases may contain a large amount of time series, which can cause a myriad of issues in EwE6:

• >10000 TS may cause out of memory issues when loading TS
• >32000 TS will cause database errors (Sequence exceeds integer range)

We may need a whole different strategy for dealing with TS, internally resembling the way EwE5 works:

• Dataset definitions read from DB, not from loaded TS (minor change)
• Load only the dataset of TS from DB in memory that a user is interested in (big change)
• Apply TS per entire dataset (minor change)
• Use different data type for sequence fields (minor change)

[jeroens]

The aformentioned changes have been implemented:

• Fixed database error that limited the number of TS in the database to 32768 (potential occurences on other database tables are fixed as well)
• The user will now decide which datasets to load
• From loaded datasets, the user can apply TS (or entire datasets)

[carieh]

Using the model Jeroen sent meFirst the model took a long time to load, and then the time series (I only selected a small fraction of the ones in the database) it took a really long time to load them all, and the screen went semi-transparent (i.e. you could see the window behind it open instead of the Ecosim interface). Then it stopped responding when I tried to open the interface. Ha you can read this message in the background. Please remember I only tried to load a small fraction of what was in the model: Ollie MARE model. Basically this is still a freeze.

[jeroens]

Hopefully it is better now the TS user interface has changed. This will first show up in version 6.0.3.10. It is really painful for EwE to handle such an amount of data, no matter how we look at it. Please re-test, I am happy with the results now.

[carieh]

AAA_Mare model gets an error when importing. I ignore, but I am not sure of the significance. and then the program freezes. (this happened 3 times). Version 6.0.3.11 :-(

[jeroens]

The error is fixed and was indeed not significant.

The 'freeze' is not real, the program will eventually move on. This particular AAA Mares model contains an insane 50.000+ time series, spread over 7000+ datasets. The database import logic simply has a *LOT* of work to do - and takes forever to complete, but it WILL complete, yippee!

I managed to speed up the import process a little. There is nothing I can fix any further here ;)

Won't get any better after build 6.0.3.11