Repository navigation
Memory optimization #67
Description
Hello, I am facing some memory problems when reading big databases (which I would like to store completly in the memory). I noticed that the wfdb functions generally return float64 numpy arrays. However, if I understood correctly from the link below, the signals are coded on maximum 32 bits.
https://www.physionet.org/physiotools/wag/signal-5.htm
Here are my questions / remarks:
- does this mean it is safe to convert all signals to 32bits ?
- if this is safe, would it be possible to make your functions return float32 arrays ? Or even better to return the minimum size format depending on the header format field ?
G.
Metadata
Metadata
Assignees
Labels
No labels
Activity
cx1111 commentedon Aug 1, 2017
I think the second option sounds reasonable. Can anyone think of any instances where returning numpy arrays with dtype=floatXX !=64 will do any harm with the average user's scripts?
cx1111 commentedon Aug 14, 2017
I'm nearly done with this, though I have some uncertainties regarding default logic and usability.
When digital values are returned, it is easy to use the numpy dtype corresponding to the same resolution or higher, ie. int16 for fmt 16 signals. But when physical values are requested, it may be a bit more ambiguous.
Consider a fmt 16 or fmt 32 signal. Converting those to float16 and float32 will make them lose some resolution if indeed the input signal was really captured with 16 or 32 bits.
In the end, I wonder what values the
returnresargument I am adding should be allowed to be. I thought of:The default will be 64 in any case. Regarding my previous comment, people could easily create overflows/underflows if they unintentionally use a low resolution dtype.
cx1111 commentedon Aug 14, 2017
I've decided to allow it to be set to either 64, 32, 16 or 8. The user can mind the dtype since different cases may warrant different options.
Regarding your questions, for all formats aside from 32, it will be safe to convert to float32 with insignificant precision loss. And even with fmt 32 signals, it is unlikely that we have any (at this point in time on physionet) in which the capturing device actually recorded with 32 bit precision, so float32 would be generally safe in those cases too.
One last note, we have some ridiculously long signals in certain databases (many many days long). When processing these kinds of signals, reading them part by part will be inevitable since this package reads all values into memory.
I'll close this issue once I push the changes.
Gijom commentedon Aug 16, 2017
Great, many thanks for your prompt changes!
cx1111 commentedon Aug 16, 2017
Sure, no problem. Let me know if my changes actually did anything :)
cx1111 commentedon Aug 31, 2017
I should go and edit the code to use np.add/subtract/divide/multiply during adc/dac conversion when only one version of the signal (digital/physical) is desired, to prevent extra memory use via temporary variable allocation.
Add output_dir argument to csv_to_wfdb. Fixes #67.
Add output_dir argument to csv_to_wfdb. Fixes #67.
Add output_dir argument to csv_to_wfdb. Fixes #67.
Add output_dir argument to csv_to_wfdb. Fixes #67.
Add write_dir argument to csv_to_wfdb. Fixes #67. (#492)
Adds changelog for v4.2.0. (#529)