Remove MD5 use to allow FIPS compliance.

[RayGozer]

Short Description

Systems with FIPS enabled cannot run scancode-toolkit due to use of MD5 algorithm. Causes error: "ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS".

Possible Labels

• new feature

Select Category

• Enhancement

Describe the Update

Certain systems are required to enabled FIPS due to contractual requirements. With FIPS enabled, any use of MD5 results in an error when using scancode-toolkit.

File "venv/lib/python3.8/site-packages/licensedcode/match_hash.py", line 49, in tokens_hash
return md5(as_bytes).digest()
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

MD5 can be replaced with SHA1, which is FIPS compliant.

How This Feature will help you/your organization

We cannot use scancode-toolkit without this modification.

Possible Solution/Implementation Details

Assuming the use of MD5 is arbitrary, it may be possible to replace it with SHA1.

Example/Links if Any

I have no public links available.

Can you help with this Feature

Maybe. I am waiting for authorization to submit a pull request.

[mjherzog]

For reference FIPS = Federal Information Processing Standards: https://www.nist.gov/standardsgov/compliance-faqs-federal-information-processing-standards-fips

[pombredanne]

@RayGozer

Are you saying that your system Python is modified in a way to reject/fail when the hash.md5 standard library code is called? That's news and I find it a little bit brutal... feels like prohibiting any knife in the house because they can cut skin ;) ....

Now, let's make sure we can run on a FIPS system alright.

FWIW, in the place that failed for you, we do not use MD5 as a secure hash but as a fast enough non-crypto hash function that works consistently across OSes.

Using SHA1 or SHA256 to the same effect is not an issue in "venv/lib/python3.8/site-packages/licensedcode/match_hash.py" and hopefully this should have only marginal performance impact.

But we still need to compute whole-file MD5 checksums otherwise as they are sometimes available in upstream package repository and consist of a good enough but weak similarity clue for matching files. We do not rely on them, but we cannot eliminate them. We are using them also as a fast enough non-crypto hash function in some fingerprinting algos for an upcoming matching tool.

If you submit a patch please ensure that we can:

1. test if we are a FIPS system (or deal with exceptions alright)
2. have unit tests that can mock this condition
3. we can use sha1 always in licensedcode/match_hash.py
4. for the checkms computed at the file level with the --info option you could either have a try/except if md5 fails or some other tests that you may know of in here https://github.com/nexB/commoncode/blob/main/src/commoncode/hash.py and IMHO the correct strategy and UX in these case would be to return and empty MD5 and a top level scan error or warning that MD5 is disabled on this FIPS system.

See also the calling site at https://github.com/nexB/scancode-toolkit/blob/a410f6ebb93a9b22343d7737897b61f3cd2a6ac3/src/scancode/api.py#L320

There are also other places where we deal with MD5 from external sources or because they are used and needed there
See https://github.com/nexB/scancode-toolkit/search?l=Python&q=md5

As an example they are used in RPMs and Cocoapods and Debian https://github.com/nexB/scancode-toolkit/blob/a410f6ebb93a9b22343d7737897b61f3cd2a6ac3/src/packagedcode/debian.py#L466

In all cases this is never relied upon for its cryptographic features and we cannot eliminate the upstream usage of these.... we are not computing hashes there so you may not care.

[RayGozer]

Thanks for the quick response! Yes, FIPS is brutal. I was reading some other python experiences and it may be possible to keep md5, but usedforsecurity must be explicitly set to False: hashlib.md5(b"foo", usedforsecurity=False).hexdigest(). cupy/cupy#5988. I'm still looking into the change.

Thank you for the information on upstream usage.

[DennisClark]

Here is a somewhat interesting essay regarding MD5 and supply chain security in general:
https://nakedsecurity.sophos.com/2022/11/30/serious-security-md5-considered-harmful-to-the-tune-of-600000/

[pombredanne]

@RayGozer re:

Yes, FIPS is brutal. I was reading some other python experiences and it may be possible to keep md5, but usedforsecurity must be explicitly set to False: hashlib.md5(b"foo", usedforsecurity=False).hexdigest(). cupy/cupy#5988. I'm still looking into the change.

Thanks. This makes sense! Tell me how to best proceed. I really welcome your help there!

[AyanSinhaMahapatra]

This was resolved by using hashlib.md5(b"foo", usedforsecurity=False).hexdigest() wherever necessary, please reopen if we missed something.