CodeGate doesn't distinguish Python built-in modules and external packages

Describe the issue

CodeGate isn't aware of the built-in Python modules, and may treat imports of these as references to external packages.

The specific case I've encountered is hashlib - at one time it was an external package and so exists in PyPI and thus also in our data set, but the external package was archived and moved to built-in ages ago. When CodeGate encounters import hashlib in code, it finds the archived package in the vector DB, and reports it as archived/deprecated.

Insight report - https://www.insight.stacklok.com/report/pypi/hashlib
PyPI entry - https://pypi.org/project/hashlib/20081119/

CodeGate behavior:

Steps to Reproduce

Reference the app.py file from the codegate-demonstration repo using Copilot or Continue chat.

Operating System

MacOS (Arm)

IDE and Version

VS Code 1.96.2

Extension and Version

Any

Provider

GitHub Copilot

Model

Any

Logs

2025-01-08T21:19:27.008Z [debug    ] Found matching packages in sqlite-vec database matched_packages=['hashlib (crates)', 'hashlib (pypi)', 'invokehttp (pypi)'] module=codegate pathname=/app/src/codegate/pipeline/codegate_context_retriever/codegate.py
2025-01-08T21:19:27.008Z [debug    ] Final context message          context_message=Context: hashlib is a Rust package available on Crates ecosystem.  However, this package is found to be archived and no longer maintained. For additional information refer to https://www.insight.stacklok.com/report/crates/hashlib - Package offers this functionality: Provide various hash algorithms under a same abstraction layer.
hashlib is a Python package available on PyPI ecosystem.  However, this package is found to be deprecated and no longer recommended for use. For additional information refer to https://www.insight.stacklok.com/report/pypi/hashlib - Package offers this functionality: Secure hash and message digest algorithm library

Additional Context

No response

There's a potential secondary issue here too, where CodeGate is reporting this as both a Crates and PyPI package even though this is a Python file, shall I open a separate issue for this?

@ptelang

@ptelang is this covered by #475 ?

@ptelang

@ptelang retest

There's a potential secondary issue here too, where CodeGate is reporting this as both a Crates and PyPI package even though this is a Python file, shall I open a separate issue for this?

This issue is fixed in the latest version by this PR.

Currently, Codegate cannot identify libraries like hashlib which were external earlier but are now built into python.

We can address this issue when the projects functionality is implemented. Codegate can then read the dependency files (e.g. requirements.tx, pyproject.toml, etc.) to detect cases like hashlib and prevent the false positive.

@ptelang

@ptelang this is fixed now?

@lukehinds

@lukehinds, Part of this issue was fixed.

"hashlib" getting raised as a bad package is a corner case and is very rare. So, I think we can close this.

@ptelang

closing per request by @ptelang

Navigation Menu

Repository navigation

CodeGate doesn't distinguish Python built-in modules and external packages #518

Describe the issue

Steps to Reproduce

Operating System

IDE and Version

Extension and Version

Provider

Model

Logs

Additional Context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CodeGate doesn't distinguish Python built-in modules and external packages #518

Description

Describe the issue

Steps to Reproduce

Operating System

IDE and Version

Extension and Version

Provider

Model

Logs

Additional Context

Activity

danbarr commented on Jan 8, 2025

lukehinds commented on Jan 10, 2025

lukehinds commented on Jan 13, 2025

ptelang commented on Jan 13, 2025

ptelang commented on Jan 13, 2025

lukehinds commented on Jan 20, 2025

ptelang commented on Feb 3, 2025

jhrozek commented on Feb 17, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions