[bug][github actions] codecov base commit does not match the "merge commit" that the branch used to generate coverage

[Ryang20718]

On codecov reports (github actions), we see the base commit used to compare the PR is rather stale. example PR is Ryang20718/cflag#28

i.e if my branch was branched off commit C whilst A is head, if I submit a pull request to run in github actions (for CI), we run on the merged commit.

A --> main head
|
B
|
C --> user/ryang/user_branch

Thus, the state of our branch is actually the following diagram.

A --> main head --> user/ryang/user_branch
|
B
|
C

This leads to a couple of unexpected bugs found in code coverage status checks. I.e because we run all of our tests on the merged commit, there may be a decent amount of files changed between commits C and A which we have inherited because we are running on the merged commit.

Thus, project level coverage may differ drastically even though the files we have touched have 100% patch coverage.

I have tried a couple of ways of bypassing this by overriding the commit to use the merged commit however, that doesn't work. If you manually rebase your PR and resubmit CI, then the base commit for the codecov report is accurate again. However, this is not a great developer experience as we'd like to make our codecov status checks blocking, but currently this is a bug

It seems like https://community.codecov.com/t/selected-base-commit-when-pr-is-based-on-an-older-commit-of-the-target-branch/4101 is a similar issue

I tried setting commit_parent in codecov-action similar to codecov/codecov-action#597, but to no avail
and also setting commit_parent to the HEAD of main, but to no avail

even though main has a more recent commits with coverage

Logs from attempt to use parent_commit

=> Running git config --global --add safe.directory /home/runner/work/cflag/cflag
/usr/bin/git config --global --add safe.directory /home/runner/work/cflag/cflag
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v create-commit --parent-sha de19c68b335e6e17e1e5b3204935d3c5e5d5b716 --git-service github -C 37d7897188310ffc193ea3340b682505f9ca2db8
info - 2024-11-16 00:[48](https://github.com/Ryang20718/cflag/actions/runs/11865673830/job/33071259742#step:4:49):48,182 -- ci service found: github-actions
debug - 2024-11-16 00:48:48,185 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-16 00:48:48,185 -- Loading config from .codecov.yml
debug - 2024-11-16 00:48:48,188 -- Starting create commit process --- {"codecov_yml_path": ".codecov.yml", "verbose": true, "auto_load_params_from": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "create-commit", "parent_sha": "de19c68b335e6e17e1e5b320[49](https://github.com/Ryang20718/cflag/actions/runs/11865673830/job/33071259742#step:4:50)35d3c5e5d5b716", "git_service": "github", "commit_sha": "37d7897188310ffc193ea3340b682505f9ca2db8", "pull_request_number": "28", "branch": "ryang/rebase", "fail_on_error": false, "slug": "Ryang20718/cflag"}
info - 2024-11-16 00:48:48,597 -- Process Commit creating complete
debug - 2024-11-16 00:48:48,597 -- Commit creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=202, text='{\"status\":\"queued\"}\\n')"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v create-report'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v create-report --git-service github -C 37d7897188310ffc193ea3340b682[50](https://github.com/Ryang20718/cflag/actions/runs/11865673830/job/33071259742#step:4:51)5f9ca2db8
info - 2024-11-16 00:48:49,317 -- ci service found: github-actions
debug - 2024-11-16 00:48:49,320 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-16 00:48:49,320 -- Loading config from .codecov.yml
debug - 2024-11-16 00:48:49,324 -- Starting create report process --- {"codecov_yml_path": ".codecov.yml", "verbose": true, "auto_load_params_from": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "create-report", "git_service": "github", "commit_sha": "37d7897188310ffc193ea3340b682505f9ca2db8", "code": "default", "pull_request_number": "28", "fail_on_error": false, "slug": "Ryang20718/cflag"}
info - 2024-11-16 00:48:49,806 -- Process Report creating complete
debug - 2024-11-16 00:48:49,806 -- Report creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=202, text='{\"status\":\"queued\"}\\n')"}
info - 2024-11-16 00:48:49,806 -- Finished creating report successfully --- {"response": "{\"status\":\"queued\"}\n"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v do-upload'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v do-upload -f torchscript_input_files_coverage.dat -F torchscript_input_files --git-service github -C 37d7897188310ffc193ea3340b682505f9ca2db8
info - 2024-11-16 00:48:50,523 -- ci service found: github-actions
debug - 2024-11-16 00:48:50,526 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-16 00:48:50,526 -- Loading config from .codecov.yml
debug - 2024-11-16 00:48:50,530 -- Starting upload processing --- {"codecov_yml_path": ".codecov.yml", "verbose": true, "auto_load_params_from": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "do-upload", "flags": ["torchscript_input_files"], "git_service": "github", "commit_sha": "37d7897188310ffc193ea3340b682505f9ca2db8", "report_code": "default", "network_root_folder": "/home/runner/work/cflag/cflag", "files_search_root_folder": "/home/runner/work/cflag/cflag", "files_search_exclude_folders": [], "disable_search": false, "disable_file_fixes": false, "build_code": "11865673830", "build_url": "https://github.com/Ryang20718/cflag/actions/runs/11865673830", "job_code": "ci", "name": "11865673830", "branch": "ryang/rebase", "pull_request_number": "28", "env_vars": {}, "plugin_names": ["xcode", "gcov", "pycoverage"], "dry_run": false, "use_legacy_uploader": false, "handle_no_reports_found": false, "report_type": "coverage", "network_filter": null, "network_prefix": null, "gcov_args": null, "gcov_ignore": null, "gcov_include": null, "gcov_executable": null, "swift_project": null, "fail_on_error": false, "slug": "Ryang20718/cflag"}
debug - 2024-11-16 00:48:50,530 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"], "cli_config": {}}
debug - 2024-11-16 00:48:50,530 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-11-16 00:48:50,530 -- Running xcode plugin...
warning - 2024-11-16 00:48:50,531 -- xcrun is not installed or can't be found.
debug - 2024-11-16 00:48:50,531 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-11-16 00:48:50,531 -- Running gcov plugin...
warning - 2024-11-16 00:48:50,532 -- No gcov data found.
debug - 2024-11-16 00:48:50,532 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
warning - 2024-11-16 00:48:50,533 -- coverage.py is not installed or can't be found.
debug - 2024-11-16 00:48:50,533 -- Collecting relevant files
info - 2024-11-16 00:48:50,542 -- Found 2 coverage files to report
info - 2024-11-16 00:48:50,542 -- > /home/runner/work/cflag/cflag/onboard_av_binary_coverage.dat
info - 2024-11-16 00:48:50,542 -- > /home/runner/work/cflag/cflag/torchscript_input_files_coverage.dat
debug - 2024-11-16 00:48:50,545 -- Selected uploader to use: <class 'codecov_cli.services.upload.upload_sender.UploadSender'>
debug - 2024-11-16 00:48:50,546 -- Sending upload request to Codecov
info - 2024-11-16 00:48:50,983 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/Ryang20718/cflag/commit/37d7897188310ffc193ea3340b682505f9ca2db8
debug - 2024-11-16 00:48:50,983 -- Upload request to Codecov complete. --- {"response": {"raw_upload_location": "https://storage.googleapis.com/codecov/shelter/github/Ryang20718%3A%3A%3A%3Acflag/37d7897188310ffc193ea3340b682505f9ca2db8/c39a133b-565c-46e3-b6bc-d590a640d796.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJWQHUGIBILH4J7Q6ZUSCIFNEOLYSNDS7L3B4N5SIBQ2J4YLYE5CRFCD%2F20241116%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20241116T004850Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=abf385f33cff4a5bfb2bf19cacdd6f640cf9e1d629645a9bd3f65b1dea1ce3d5", "url": "https://app.codecov.io/github/Ryang20718/cflag/commit/37d7897188310ffc193ea3340b682505f9ca2db8"}}
debug - 2024-11-16 00:48:50,983 -- Sending upload to storage
info - 2024-11-16 00:48:[51](https://github.com/Ryang20718/cflag/actions/runs/11865673830/job/33071259742#step:4:52),164 -- Process Upload complete
debug - 2024-11-16 00:48:51,164 -- Upload result --- {"result": "RequestResult(error=None, warnings=[], status_code=200, text='')"}

[Ryang20718]

cc @thomasrockhu-codecov

From a user's perspective this seems to be a bug. Wondering what your thoughts on this are and if there are any work arounds that I'm missing?

I'm really pushing hard to adopt codecov and am happy to help get this feature in if needed

[thomasrockhu-codecov]

@Ryang20718 sorry I had a stomach bug yesterday. I'm going to escalate this to our product team to take a look

[Ryang20718]

@thomasrockhu-codecov let me know if there's any additional information I can help provide. We're currently in the middle of switching to codecov and I would love to get these edges ironed out!

[thomasrockhu-codecov]

@Ryang20718 sorry I forgot to respond. We made some changes here which should go back to the way we were pulling commits initially for merged commits. Do you mind checking with the vanilla action v5?

[Ryang20718]

@thomasrockhu-codecov
Tried with codecov-action v5 in this run URL for this PR

Tried with merge commit sha

      - name: Checkout repository
        uses: actions/checkout@v4

      - uses: codecov/codecov-action@v5
        with:
          files: torchscript_input_files_coverage.dat
          flags: torchscript_input_files
          verbose: true
          commit_parent: ${{ github.event.pull_request.merge_commit_sha }} 
          token: ${{ secrets.CODECOV_TOKEN }}
          # codecov_yml_path: .codecov.yml

      - uses: codecov/codecov-action@v5
        with:
          files: onboard_av_binary_coverage.dat
          flags: onboard_av_binary
          verbose: true
          commit_parent: ${{ github.event.pull_request.merge_commit_sha }} 
          token: ${{ secrets.CODECOV_TOKEN }}

==> Running create-commit
      ./codecov --verbose create-commit -t <redacted> --git-service github --parent-sha 6cdbd778f354568e7915e152a7c43dd426d8b2e2 --sha bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d
info - 2024-11-21 20:55:57,653 -- ci service found: github-actions
debug - 2024-11-21 20:55:57,657 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:55:57,659 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:55:57,662 -- Loading config from /home/runner/work/cflag/cflag/.codecov.yml
debug - 2024-11-21 20:55:57,666 -- Starting create commit process --- {"verbose": true, "auto_load_params_from": null, "codecov_yml_path": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "create-commit", "git_service": "github", "parent_sha": "6cdbd778f354568e7915e152a7c43dd426d8b2e2", "commit_sha": "bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d", "pull_request_number": "28", "branch": "ryang/rebase", "fail_on_error": false, "slug": "Ryang20718/cflag"}
info - 2024-11-21 20:55:58,016 -- Process Commit creating complete
debug - 2024-11-21 20:55:58,017 -- Commit creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=202, text='{\"status\":\"queued\"}\\n')"}
 
==> Running create-report
      ./codecov --verbose create-report -t <redacted> --git-service github --sha bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d
info - 2024-11-21 20:55:58,767 -- ci service found: github-actions
debug - 2024-11-21 20:55:58,770 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:55:58,773 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:55:58,776 -- Loading config from /home/runner/work/cflag/cflag/.codecov.yml
debug - 2024-11-21 20:55:58,782 -- Starting create report process --- {"verbose": true, "auto_load_params_from": null, "codecov_yml_path": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "create-report", "git_service": "github", "commit_sha": "bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d", "code": "default", "pull_request_number": "28", "fail_on_error": false, "slug": "Ryang20718/cflag"}
info - 2024-11-21 20:55:59,315 -- Process Report creating complete
debug - 2024-11-21 20:55:59,315 -- Report creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=202, text='{\"status\":\"queued\"}\\n')"}
info - 2024-11-21 20:55:59,315 -- Finished creating report successfully --- {"response": "{\"status\":\"queued\"}\n"}
 
==> Running do-upload
      ./codecov --verbose do-upload -t <redacted> --file onboard_av_binary_coverage.dat --flag onboard_av_binary --gcov-executable gcov --git-service github --sha bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d
info - 2024-11-21 20:56:00,074 -- ci service found: github-actions
debug - 2024-11-21 20:56:00,077 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:56:00,080 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:56:00,082 -- Loading config from /home/runner/work/cflag/cflag/.codecov.yml
debug - 2024-11-21 20:56:00,087 -- Starting upload processing --- {"verbose": true, "auto_load_params_from": null, "codecov_yml_path": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "do-upload", "flags": ["onboard_av_binary"], "gcov_executable": "gcov", "git_service": "github", "commit_sha": "bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d", "report_code": "default", "network_root_folder": "/home/runner/work/cflag/cflag", "files_search_root_folder": "/home/runner/work/cflag/cflag", "files_search_exclude_folders": [], "disable_search": false, "disable_file_fixes": false, "build_code": "1[196](https://github.com/Ryang20718/cflag/actions/runs/11961496503/job/33347848788#step:5:203)1496503", "build_url": "https://github.com/Ryang20718/cflag/actions/runs/11961496503", "job_code": "ci", "name": "11961496503", "branch": "ryang/rebase", "pull_request_number": "28", "env_vars": {}, "plugin_names": ["xcode", "gcov", "pycoverage"], "dry_run": false, "use_legacy_uploader": false, "handle_no_reports_found": false, "report_type": "coverage", "network_filter": null, "network_prefix": null, "gcov_args": null, "gcov_ignore": null, "gcov_include": null, "swift_project": null, "fail_on_error": false, "slug": "Ryang20718/cflag"}
debug - 2024-11-21 20:56:00,088 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"], "cli_config": {}}
debug - 2024-11-21 20:56:00,088 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-11-21 20:56:00,088 -- Running xcode plugin...
warning - 2024-11-21 20:56:00,088 -- xcrun is not installed or can't be found.
debug - 2024-11-21 20:56:00,088 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-11-21 20:56:00,088 -- Running gcov plugin...
warning - 2024-11-21 20:56:00,090 -- No gcov data found.
debug - 2024-11-21 20:56:00,090 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
warning - 2024-11-21 20:56:00,090 -- coverage.py is not installed or can't be found.
debug - 2024-11-21 20:56:00,090 -- Collecting relevant files
info - 2024-11-21 20:56:00,101 -- Found 2 coverage files to report
info - 2024-11-21 20:56:00,101 -- > /home/runner/work/cflag/cflag/torchscript_input_files_coverage.dat
info - 2024-11-21 20:56:00,101 -- > /home/runner/work/cflag/cflag/onboard_av_binary_coverage.dat
debug - 2024-11-21 20:56:00,104 -- Selected uploader to use: <class 'codecov_cli.services.upload.upload_sender.UploadSender'>
debug - 2024-11-21 20:56:00,105 -- Sending upload request to Codecov
info - 2024-11-21 20:56:00,446 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/Ryang20718/cflag/commit/bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d
debug - 2024-11-21 20:56:00,447 -- Upload request to Codecov complete. --- {"response": {"raw_upload_location": "https://storage.googleapis.com/codecov/shelter/github/Ryang20718%3A%3A%3A%3Acflag/bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d/655ef708-fcea-4876-98d2-0bdecc5ed896.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJWQHUGIBILH4J7Q6ZUSCIFNEOLYSNDS7L3B4N5SIBQ2J4YLYE5CRFCD%2F20241121%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20241121T205600Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=f4d90076c86462bbfe75d9e6a792db722dde96b105399302128e578a18ef7d1b", "url": "https://app.codecov.io/github/Ryang20718/cflag/commit/bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d"}}
debug - 2024-11-21 20:56:00,447 -- Sending upload to storage
info - 2024-11-21 20:56:00,689 -- Process Upload complete
debug - 2024-11-21 20:56:00,689 -- Upload result --- {"result": "RequestResult(error=None, warnings=[], status_code=[200](https://github.com/Ryang20718/cflag/actions/runs/11961496503/job/33347848788#step:5:207), text='')"}

but still 3 commits behind

[Ryang20718]

also tried using a commit from main directly, but same result (report is still not using latest main to compare as base)

      - uses: codecov/codecov-action@v5
        with:
          files: onboard_av_binary_coverage.dat
          flags: onboard_av_binary
          verbose: true
          commit_parent: de19c68b335e6e17e1e5b3204935d3c5e5d5b716 #commit from main
          token: ${{ secrets.CODECOV_TOKEN }}

[Ryang20718]

side note

codecov_yml_path is broken on v5 using the vanilla v5 action

[thomasrockhu-codecov]

@Ryang20718 yeah, I'm about to push a fix for the codecov_yml_path I'm going to push a fix after the US holiday, however note that that's not related to coverage.

[thomasrockhu-codecov]

@Ryang20718 perhaps I'm not understanding, is this what your commit structure is like

         --- C' --- D' (feature)
       /            ^ HEAD
A --- B   --- C --- D  (main)
                    ^HEAD

And on D' you are expecting coverage to be generated between D' and D or D' and B?