Skip to content

feat(ci): validate workflow actions against repository policy #3624

Open
#3634
Open
feat(ci): validate workflow actions against repository policy#3624
#3634
Labels
github_actionsPull requests that update GitHub Actions codestate:staleInactive item at risk of automatic closure.topic:testing

Description

@purp
purp
opened on Sep 23, 2026

User Story

As an OpenShell maintainer, I want pull-request CI to detect workflow action references that violate the repository GitHub Actions policy, so that release and maintenance workflows do not fail at startup after merge.

Problem Statement

Workflow linting currently checks syntax and security findings but does not validate external uses: references against the effective Actions permissions for the repository. A workflow can therefore pass pull-request checks and merge while referencing an action revision that GitHub rejects before any job starts.

Impact / Why This Matters

Maintainers discover these conflicts only when the affected workflow is triggered, which may happen during a release. The current workaround is to run the workflow after merge or manually compare every action SHA with a large policy allowlist. That is late, error-prone, and can block time-sensitive release work.

Proposed Design

When a pull request changes a GitHub Actions workflow or action definition, a required CI check should validate its external action and reusable-workflow references against the effective policy that GitHub will enforce for OpenShell. The check should report each rejected reference and enough policy context to choose an approved revision or request approval. Changes that do not affect workflow references should avoid unnecessary work.

Acceptance Criteria

  • Pull requests that add or change external action or reusable-workflow references receive a policy compatibility check.
  • The check evaluates the effective repository policy rather than relying on a separately maintained copy of the allowlist.
  • A disallowed owner, repository, tag, or SHA fails before merge with an actionable annotation identifying the reference.
  • Enterprise-owned, GitHub-owned, verified Marketplace, explicitly allowed, local, and reusable-workflow references are handled consistently with GitHub enforcement.
  • Policy lookup or evaluation failures produce a clear diagnostic and cannot silently report success.
  • Existing workflow syntax and security checks continue to run.

Alternatives Considered

Relying on manual review or post-merge workflow dispatch keeps the failure late and depends on maintainers noticing exact SHA-level allowlist mismatches. Maintaining a second allowlist in the repository risks drift from the GitHub policy and would not reliably predict enforcement.

Agent Investigation

Release Tag run 35874219573 failed with startup_failure before creating jobs because oras-project/setup-oras@38de303aac69abb66f3e6255b7198bff35f323e3 was not allowed. The effective repository policy allowed the same action only at 1d808f7d7f6995cc68b7bf507bfe5c5446e1dc9d.

Activity

  1. purp
    added
    topic:testing
    github_actionsPull requests that update GitHub Actions code
    on Sep 23, 2026

  2. purp commented on Sep 23, 2026

    @purp
    purp
    on Sep 23, 2026
    CollaboratorAuthor

    🏗️ build-plan

    Implementation Plan

    Issue type: feat
    Complexity: Medium
    Confidence: Medium — implementation is bounded, but rollout needs two least-privilege credentials and Marketplace verification has no documented API.

    Summary

    Add a fail-closed workflow-reference policy evaluator for changed workflow and composite-action uses: references. Use an unprivileged pull_request/merge_group request workflow and a trusted default-branch workflow_run consumer. The consumer fetches candidate YAML as inert API data, reads the live effective Actions policy, and publishes actionable file/line annotations plus a stable required status on the candidate SHA.

    Keep Actionlint and Zizmor unchanged. Never execute a PR checkout, candidate action, dependency, script, or configuration while policy credentials are present. This split supports fork and Dependabot PRs without exposing secrets to candidate code.

    Scope

    • tasks/scripts/github_actions_policy.py: structurally discover workflow/composite-action references with source locations; compare base/head occurrence counts for PRs; scan the complete merge-group tree; fetch actions/permissions and selected-actions; implement local, repository-owner, enterprise-owner, GitHub-owned, explicit pattern, SHA-pinning, Marketplace, external-action, and reusable-workflow semantics; fail closed on malformed or incomplete inputs and API/network errors.
    • tasks/scripts/github_actions_policy_test.py: deterministic parser, matcher, policy-classification, source-fetch, trusted-event binding, and failure-mode tests using injected fixtures.
    • tasks/test.toml: add the focused checker test task and include it in mise run test.
    • .github/workflows/workflow-policy-request.yml: emit a secretless, permissionless request run for PR and merge-queue revisions without checking out candidate code.
    • .github/workflows/workflow-policy.yml: consume only successful runs from the exact request workflow, check out the trusted default-branch implementation, bind the event to the current PR or merge-group SHA, provide credentials only to the checker step, and publish OpenShell / Workflow Policy.
    • CI.md, architecture/build.md, and .agents/skills/watch-github-actions/SKILL.md: document the gate, trust boundary, credentials, live-policy inputs, failure diagnosis, and rollout.

    Implementation Steps

    1. Build and test the structural YAML parser and policy matcher, including quoted/flow keys, duplicates, local references, action subpaths, reusable workflows, wildcard/negative patterns, SHA pinning, and fail-closed dynamic or malformed uses: values.
    2. Add injected GitHub policy, enterprise, source, and Marketplace clients. Resolve the repository owner's enterprise organizations live with read:enterprise; bind Marketplace metadata to the exact case-insensitive owner/repository; reject incomplete PR file lists, truncated trees, oversized definitions, missing content, and ambiguous metadata.
    3. Add the unprivileged request workflow and trusted workflow_run consumer. Validate the request workflow path/event, resolve exactly one current PR when needed, skip stale runs, scan all definitions for merge groups, emit ::error file=...,line=..., and publish a stable commit status on every evaluated outcome.
    4. Update CI/architecture documentation and the Actions-monitoring skill, then run the contributor checks.
    5. Configure the two credentials, add OpenShell / Workflow Policy to the required ruleset, and verify rollout with disposable allowed/disallowed PR changes.

    Test Plan

    • Unit tests cover structural YAML discovery, changed-reference counting, policy categories, tags versus SHAs, reusable workflows, case-insensitive and negative patterns, exact Marketplace binding, enterprise resolution, source completeness limits, trusted workflow-run binding, stale PR runs, and merge-group full-tree scanning.
    • Run mise run test:github-actions-policy, mise run pre-commit, mise run test, and mise run ci. Existing Workflow Security CI continues to run Actionlint and Zizmor.
    • Use a live-policy canary against the repository settings. After merge, use a disposable PR to prove the deployed credentials and required status accept an approved SHA and reject a disallowed SHA.

    Risks & Open Questions

    • ACTIONS_POLICY_READ_TOKEN must be a fine-grained token scoped only to NVIDIA/OpenShell with repository Administration(read).
    • ACTIONS_ENTERPRISE_READ_TOKEN must be a classic token with only read:enterprise; GitHub's fine-grained tokens and GITHUB_TOKEN do not expose the required viewer-enterprise membership graph.
    • GitHub exposes verified_allowed but no documented API for verified Marketplace identities. The checker consumes exact repository-bound embedded Marketplace metadata and fails closed if it is missing or ambiguous.
    • The default-branch trust boundary is load-bearing: candidate files remain inert data, and credentials exist only on the trusted consumer's checker step.
    • The gate takes effect only after merge. Secret and ruleset configuration plus a post-merge canary PR are part of rollout.

    Documentation Impact

    Update CI.md, architecture/build.md, and .agents/skills/watch-github-actions/SKILL.md. No published product docs, gateway configuration docs, release-canary playbook, or LSM compatibility changes are needed.

    Revision 2 — replaced pull_request_target with a secretless request workflow and trusted workflow_run consumer; added merge-queue, Dependabot, enterprise-membership, and source-completeness handling.

  3. purp commented on Sep 23, 2026

    @purp
    purp
    on Sep 23, 2026
    CollaboratorAuthor

    Implemented in #3634.

    The PR adds a secretless pull_request/merge_group request workflow and a trusted default-branch workflow_run consumer. The checker structurally finds executable uses: references, compares only added occurrences for PRs, scans the complete queued tree for merge groups, reads the live repository and enterprise policy inputs, and publishes the required OpenShell / Workflow Policy status with file/line diagnostics. It fails closed on incomplete or ambiguous inputs.

    Verification completed:

    • mise run test:github-actions-policy — 36 passed
    • mise run pre-commit — passed
    • mise run ci — passed
    • live read-only canary against current upstream/main — 362 references evaluated, 0 rejected
    • commit signature verified before push

    Post-merge rollout still requires configuring ACTIONS_POLICY_READ_TOKEN, ACTIONS_ENTERPRISE_READ_TOKEN, adding OpenShell / Workflow Policy to the required ruleset, and running a disposable allowed/disallowed canary PR.

  4. github-actions commented on Oct 8, 2026

    @github-actions
    github-actionsbot
    on Oct 8, 2026 – with GitHub Actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  5. github-actions
    added
    state:staleInactive item at risk of automatic closure.
    on Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    github_actionsPull requests that update GitHub Actions codestate:staleInactive item at risk of automatic closure.topic:testing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions