Remove the dependency on Metaflow

[tdruez]

The benefits from using Metaflow are now outweighed by the issues and lack of customization (logging for example) forcing us into complex workarounds.

Since we do not use much of the Metaflow features (we've only used Metaflow for the structure of the Pipelines), we do not have a lot of code that depends on it.
The required code changes will mostly be needed on the Pipelines side.

Those change will also allow us to have more control over the execution of Pipelines (logging, profiling, branching, ...)
It will remove 1 level of indirection in the execution stack since Metaflow is always run as a subprocess call (it cannot be called within Python by design).

[mjherzog]

@tdruez Does this mean that most of what we are using from Metaflow is the approach to structuring a Pipeline without really using much Metaflow code?

[tdruez]

@mjherzog Yes, we are basing our Pipelines on the Metaflow Flow structure but inherit from its execution approach (through subprocess) that is not great in the ScanCode.io context (asynchronous tasks).