Handling of run-specific temporary directories when using as library

[stefan6419846]

Short Description

Running scancode as a library will generate run-specific temporary directories, which are never cleaned up.

Possible Labels

• new feature

Select Category

• Enhancement
• Add License/Copyright
• Scan Feature
• Packaging
• Documentation
• Expand Support
• Other

Describe the Update

There is a documented/automated way to clean the temporary directories after each run (or it happens automatically) as soon as they are not used any more.

How This Feature will help you/your organization

Writing custom wrappers around scancode functionality without using the CLI requires manual cleanup of temporary directories, which seems to be undocumented and might unnecessarily use/block space on the hard drive.

Possible Solution/Implementation Details

There should be clear documentation for the public(?) API in scancode.api that manual cleanup as in https://github.com/nexB/scancode-toolkit/blob/a29ae817d95296146ba8ac5bd94ed043d541cdc6/src/scancode/cli.py#L1021-L1035 is required.

Alternatively, scancode should automatically register the corresponding directories for cleanup during exit or provide a corresponding context manager to simplify usage.

Example/Links if Any

I am currently using atexit.register(fileutils.delete, scancode_config.scancode_temp_dir) in every parallel processing unit after noticing the behavior.

Can you help with this Feature

Maybe. I am not sure about general use cases, but for now scancode does not seem to have any explicit documentation on library usage or on a public API anyway.

[pombredanne]

This all makes sense, thank you! I am all for this within reason. My experience with typing hints is that it is easy to get carried away with complex expressions to the point it makes some functions and definition signatures look like spaghetti and is difficult to read. (No offense meant to spaghetti and FSM followers).

Also if you are working on extensions to ScanCode, may be there is some stuff that can make sense upstream too?

If your code is at https://github.com/stefan6419846/license_tools/blob/master/license_tools/scancode_tools.py for instance then there is a recent new plugin that may do similar things (collect the ELF needed libraries) in https://github.com/nexB/elf-inspector
Food for thoughts!

[stefan6419846]

This all makes sense, thank you! I am all for this within reason. My experience with typing hints is that it is easy to get carried away with complex expressions to the point it makes some functions and definition signatures look like spaghetti and is difficult to read. (No offense meant to spaghetti and FSM followers).

I am totally aware of that and should of course be handled with care (for example introducing type aliases where appropriate), although this probably should have been commented at #3487 as the current issue is about general library usage and temporary directory cleanup.

Also if you are working on extensions to ScanCode, may be there is some stuff that can make sense upstream too?

For now, the repository you mention below is mostly just some basic CLI wrapper around ScanCode tightly fitted to my own needs of quickly getting a first impression on the licensing status of any package. I just implemented a straight-forward API I actually can remember quickly and which prevents me from doing manual work. (This does not replace the actual review and an in-depth analysis later, but proved to be valuable anyway. Thanks for your great work on this tool!) I might add other integrations later (unrelated to ScanCode) if I find it useful.

Regarding checking in extensions to upstream: Yes, I am open to this. If you find something useful in the current code you would like to see in your upstream version, feel free to say so. I already thought about opening an issue/PR for replacing the current dictionaries with dataclasses to improve library experience as well, but due to time constraints and no real urgency as well as no real difference for CLI users, I did not yet further follow-up on this.

If your code is at https://github.com/stefan6419846/license_tools/blob/master/license_tools/scancode_tools.py for instance then there is a recent new plugin that may do similar things (collect the ELF needed libraries) in https://github.com/nexB/elf-inspector

As already mentioned: Yes, this is my current code which wraps ScanCode.

The ELF/shared object analysis is something which I just integrated to get some basic insight at all and is just a very basic starting point as statically linked stuff is missed as well. There are some rough plans for further automated analyis to basically resolve everything on system package manager level to be able to analyze the actual sources, but no implementations yet. Nevertheless, thanks for pointing out the new extension - I will probably have a look at it and how it might be helpful for my use cases.

[AyanSinhaMahapatra]

This was also #2034 (comment)