Repository navigation
Develop MVP of model bundle #3482
Description
Is your feature request related to a problem? Please describe.
Thanks for the interesting technical discussion with @ericspod @wyli @atbenmurray @rijobro , as we still have many unclear requirements and unknown use cases, we plan to develop the model package feature step by step, May adjust the design based on feedback during the development.
For the initial step, the core team aligned to develop a small but typical example for inference first, it will use JSON config files to define environments, components and workflow, save the config and model into TorchScript model. then other projects can easily reconstruct the exact same python program and parameters to reproduce the inference. When the small MVP is ready, will share and discuss within the team for the next steps.
I will try to implement the MVP referring to some existing solutions, like NVIDIA Clara MMAR, ignite online package, etc. Basic task steps:
- Include metadata (env / sys info, changelog, version, input / output data format, etc), configs, model weights, etc. in a model package example for review. PR: [WIP] 486 Add example model package tutorials#487
- Search specified python packages and build python instances from dictionary config with
name/path&args. PR: 3482 AddConfigComponentfor config parsing #3720 - Recursively parse configs in a dictionary with dependencies and executable code, for example:
{"dataset": {"<name>": "Dataset", "<args>": {"data": "$load_datalist()"}}, "dataloader": {"<name>": "DataLoader", "<args>": {"data": "@dataset"}}}. PR: 3482 Add ReferenceResolver to manage config items #3818, 3482 3829 AddConfigParserto recursively parse config content #3822 - Add support to save the raw config dictionaries into TorchScript model. PR: Adding Torchscript utility functions #3138 .
- Add schema mechanism to verify the content of folder structure, metadata.json content, etc. Refer to: https://json-schema.org/, https://github.com/Julian/jsonschema. PR: 3482 Add schema verification for
metadata#3865 - Add support to verify network input / output with fake data (data input is from
metadata). - Add mechanism to easily load JSON config files with override (can add YAML support in future), similar to example: https://hydra.cc/docs/tutorials/basic/your_first_app/config_groups/. PR: 3482 add
runAPI for common training, evaluation and inference #3832 - Add support to refer to other config item in the same config file or other confg files, referring to Hydra ideas: https://hydra.cc/docs/advanced/overriding_packages/.
- Complete the inference example MMAR for spleen segmentation task. PR: Add inference example for bundle tutorials#604
- Write user manual and detailed documentation. PR: MONAI Archive Specification #3834 3482 - update bundle doc #3982
- [*Optional] Upload the spleen MMAR example to
huggingface(Pretrained Models #3451). - [*Optional] Support relative config level in the reference ID, for example:
"test": "@###data#1", 1#means current level, 2##means upper level, etc. PR: 3482 Add support for relative IDs in the config content #3974 - [*Optional] Support customized
ConfigItemandReferenceResolverin theConfigParser. PR: 3482 Add support for customized ConfigItem and resolver #3980 -
_requires_keyword for config component (adds a_requires_key forConfigComponent#3942). - Import statement in bundle config (support of 'import' statement in bundle config #3966).
- Config python logging properties in a file. PR: 3482 Add support to set config file for logging module #3994
- [*Optional] Specify rank ID for component to run only on some ranks, for example, saving checkpoint in rank 0.
Activity
dbericat commentedon Dec 13, 2021
@CPBridge this is the current PR to expand model metadata.
Nic-Ma commentedon Dec 14, 2021
Hi @wyli @ericspod @rijobro @atbenmurray ,
I updated the ticket description to make it more clear.
Thanks.
wyli commentedon Dec 14, 2021
We've discussed that the MVP should focus on the minimal structure, including model weights, basic environment/system info, maintaining changelog/versioning, how to ensure accessibility for both human and machine.
Nic-Ma commentedon Dec 14, 2021
Hi @wyli ,
Sure, I added these items to the MVP task list.
Thanks.
dbericat commentedon Dec 14, 2021
@MMelQin @gigony @GreySeaWolf @CPBridge @mocsharp @joshliberty @vikashg thoughts on proposed MVP?
ericspod commentedon Dec 14, 2021
We should discuss a clear definition of requirements and objectives. We want to define a format of a single file or multiples files which contains the model weights at least with secondary information describing how to use it for various use cases. This will allow a human or a program to determine what sort of model it is, how to use the model, and what tasks to use it for. For our MVP we want to consider the starting position of what the model weight storage and metadata storage would look like and if it would achieve that objective to some degree.
The base level set of requirements I would suggest are:
One use case for this information is a human user looking into how the model is used in a particular task. They would want a clear idea of what inputs are expected and what the outputs mean. Whatever format this information is can be either easily read by a human or easily converted into a convenient format using included tools.
A second use case is a deployment environment which automatically constructs whatever infrastructure is needed around a model to present it through a standard interface. This would require generating transform sequences automatically to pre- and post-process data, and load the model through some script defining the workflow. This would used by MONAI Deploy to automatically generate a MAP from the package, or another script automatically create a Docker image to serve the model as a command line tool or through Flask, or another script to interface with existing hosting services to upload the model and whatever other information is needed.
Nic-Ma commentedon Dec 15, 2021
Hi @ericspod ,
Thanks very much for your detailed description.
I will try to prepare a draft model package for discussion according to your summary and our basic ideas, and then develop the necessary features to support it.
Thanks.
wyli commentedon Dec 16, 2021
I'd strongly recommend that the MVP should support the essential features for fast prototyping of model packages. The following uses the popular
detectron2package as an example:Hybrid configs with python + yaml/json/xml
Here is a workflow config of mixing .py and .yaml, and the essential mechanism is a lazy instantiation of objects.
Compositional Configuration
An extension of
RCNN-FPNcould be constructed by referring to the base config and overriding some of the options.These are mainly to ensure a good flexibility and extensibility of the packaging.
Nic-Ma commentedon Dec 16, 2021
Hi @wyli ,
Thanks for your great suggestions.
These are very good references for config parsing logic in steps 3 and 4 of this ticket, we definitely need similar features for flexibility and extensibility.
Thanks.
Nic-Ma commentedon Dec 16, 2021
Hi @wyli @ericspod @atbenmurray @rijobro ,
I tried to prepare a draft model package structure for the first step of this ticket in the tutorial PR.
Project-MONAI/tutorials#487
Could you please help take a look and share some feedback for the model package structure?
(1) I didn't implement any config-parsing related logic so far, I think that's later steps.
(2) I am not very sure to put some description strings in
metadata.jsonorREADME.md, so I put in both sides, would like to see your ideas.Thanks in advance.
18 remaining items
Nic-Ma commentedon Feb 19, 2022
Another related PR relating to the configuration parsing: #3822.
Thanks.
Nic-Ma commentedon Feb 21, 2022
Related PR for common training, evaluation and inference: #3832 .
Thanks.
Nic-Ma commentedon Feb 21, 2022
As we are still developing the
ConfigParserin PR #3822, I compared it with the HydraOmegaConf: https://omegaconf.readthedocs.io/en/latest/index.htmlThere are several interesting features of it we still not support:
dot listinstead of dict keys:https://omegaconf.readthedocs.io/en/latest/usage.html#from-a-dot-list
https://omegaconf.readthedocs.io/en/latest/usage.html#from-command-line-arguments
Added draft support in PR: 3482 add
runAPI for common training, evaluation and inference #3832default valueif the target node is not existing:https://omegaconf.readthedocs.io/en/latest/usage.html#default-values
https://omegaconf.readthedocs.io/en/latest/usage.html#mandatory-values
We tried to support it with TorchScript saving.
Similiar to our "@" mark with ID, but also support relative path.
Draft support in PR: 3482 add
runAPI for common training, evaluation and inference #3832I think we can try to support 3 with enhanced idea: use the "@" mark to define a placeholder, then parser gives the value.
And we can try to support 4 to make the base config similar to "base class with abstractmethod" that must be overridden.
For other features we may consider later.
Thanks.
ericspod commentedon Feb 21, 2022
I have a PR now on the specification document #3834 which should have been a draft PR but oh well. It's very minimal compared to what @Nic-Ma prototyped here and lacks details about intent, use cases, design expectations, etc.
wyli commentedon Feb 25, 2022
I think #3822 is now intuitive and flexible after a few rounds of refactoring. We may want to provide a system-wide flag to optionally disable
eval()because it is too powerful and unsafe..Nic-Ma commentedon Feb 28, 2022
I put all the links of related PRs to the task steps in the above ticket description.
Thanks.
Nic-Ma commentedon Mar 7, 2022
Related PR for inference example bundle: Project-MONAI/tutorials#604.
Thanks.
[-]Develop MVP of model package[/-][+]Develop MVP of model bundle[/+]wyli commentedon Mar 20, 2022
follow-up from the dev meeting:
ericspod commentedon Mar 21, 2022
Revising this tutorial to use our bundle would be a good exemplar: https://github.com/Project-MONAI/tutorials/blob/e3eea8704f5d002002f79cffec112c5c280476b4/modules/transfer_mmar.ipynb
Nic-Ma commentedon Mar 22, 2022
After sharing with the internal NVIDIA Clara team, I put 2 missing features in the task feature.
Thanks.
Nic-Ma commentedon Mar 30, 2022
Feedback from MONAI deploy team:
Welcome more testing and feedback later.
Thanks.
Nic-Ma commentedon Apr 4, 2022
Most of the tasks are completed, I created several new tickets to track the left items, and let's close this big first ticket now.
Thanks.