Repository navigation
Open question: support for non-IEEE 754 float point types #23
Description
Relates to webmachinelearning/webnn#252
Some accelerators use non-standard float point types (e.g. bfloat16 and TF32). They are important to achieve high performance (e.g. by using Nvidia's tensor cores), and/or reduce resource usage (e.g. FP32->FP16 reduces memory usage by half).
How could MLLoader leverage these types? Some ideas:
- Do it transparently, auto convert based on the accelerator
- Should the API allow JS code to specify acceptable quantization levels (e.g. use bf16 but not fp16)
- What if the chip doesn't support the model's declared data type (e.g. BF16 chip + FP32 model)
Reacted by josephrocca
Activity
josephrocca commentedon Feb 13, 2022
Another factor is download time. IIUC, the current tfjs format (for example) doesn't support float16, and so tfjs-converter converts weights to float32. This isn't ideal because it doubles the model size. I think it makes more sense to always optimistically serve the model in its "native" floating point format and for conversion to be done at run time based on the device's hardware.
[-]Open question: support for non-IETF 754 float point types[/-][+]Open question: support for non-IEEE 754 float point types[/+]