Repository navigation
Fix: Make LocalFileSystem.match() work for bare relative globs in the current working directory - #41388
Fix: Make LocalFileSystem.match() work for bare relative globs in the current working directory#41388Naman-Katiyar wants to merge 3 commits into
Conversation
… current working directory Fixes apache#41375 LocalFileSystem._list() returned nothing for an empty directory prefix because os.path.exists('') is False, so FileSystem.match() found no files for bare relative globs such as 'users.json*' in the current working directory. Normalize an empty prefix to '.' and strip the leading './' from the yielded paths so they line up with the caller's relative pattern. Signed-off-by: Naman Katiyar <namankatiyar120@gmail.com>
…rking directory Adds a regression test for the LocalFileSystem fix: matching a bare relative glob such as 'users.json*' in the current working directory previously returned no files. The test reuses the existing tmpdir_as_cwd helper. Signed-off-by: Naman Katiyar <namankatiyar120@gmail.com>
There was a problem hiding this comment.
📋 Review Summary
This PR fixes LocalFileSystem._list() when matching bare relative globs in the current working directory (where dir_or_prefix is '') by normalizing the empty prefix to '.' and stripping the leading './' from yielded paths. However, there is a critical IndentationError in sdks/python/apache_beam/io/localfilesystem.py that prevents the module from being imported.
🔍 General Feedback
- The approach of normalizing
''to'.'inLocalFileSystem._list()and stripping the leading'.' + os.sepprefix cleanly resolves the issue and preserves relative paths forFileSystem.match(). - Please fix the indentation inside the
tryblock inLocalFileSystem._list()so that the module parses valid Python and yieldsFileMetadataunconditionally regardless ofstrip_dot_prefix.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
| try: | ||
| for f in list_files(dir_or_prefix): | ||
| try: | ||
| if strip_dot_prefix and f.startswith('.' + os.sep): |
There was a problem hiding this comment.
🔴 Critical: if strip_dot_prefix ... is indented at the same level (8 spaces) as try: on line 133, which raises an IndentationError on import. It needs to be indented to 10 spaces (and line 135 to 12 spaces).
| if strip_dot_prefix and f.startswith('.' + os.sep): | |
| if strip_dot_prefix and f.startswith('.' + os.sep): |
| for f in list_files(dir_or_prefix): | ||
| try: | ||
| if strip_dot_prefix and f.startswith('.' + os.sep): | ||
| f = f[len('.' + os.sep):] |
There was a problem hiding this comment.
🔴 Critical: Along with indenting the if statement on line 134 to 10 spaces, this body statement must be indented to 12 spaces so that yield FileMetadata(...) on line 136 remains outside the if block and executes when strip_dot_prefix is False.
| f = f[len('.' + os.sep):] | |
| f = f[len('.' + os.sep):] |
|
Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Indent the strip_dot_prefix check and its body inside the inner try block, leaving the yield outside the if so it still runs when strip_dot_prefix is False. Signed-off-by: Naman Katiyar <namankatiyar120@gmail.com>
Fixes #41375
LocalFileSystem._list() returned nothing for an empty directory prefix because os.path.exists('') is False, so FileSystem.match() found no files for bare relative globs such as 'users.json*' in the current working directory. This change normalises an empty prefix to '.' and strips the leading './' from the yielded paths so they line up with the caller's relative pattern. A regression test matching a bare relative glob in the current working directory is included.