Skip to content

Add certificate-based authentication support to Microsoft Graph files… - #69564

Closed
bujjibabukatta wants to merge 3 commits into
apache:mainfrom
bujjibabukatta:fix/#69335
Closed

bujjibabukatta wants to merge 3 commits into
apache:mainfrom
bujjibabukatta:fix/#69335

Conversation

@bujjibabukatta

@bujjibabukatta bujjibabukatta commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

Support certificate-based auth for Microsoft Graph Filesystem

Closes #69335

What

The Microsoft Graph filesystem (msgraph://, sharepoint://, onedrive://, msgd://
schemes, backed by airflow.providers.microsoft.azure.fs.msgraph.get_fs) only supported
client-secret authentication. KiotaRequestAdapterHook (used by the Microsoft Graph
operator/sensor) already supports certificate-based auth via certificate_path /
certificate_data connection extras — this PR brings the filesystem integration to
parity, so the same msgraph connection can be reused across the hook, operator,
sensor, and filesystem regardless of which auth method it uses.

Why this needed a workaround, not a one-line fix

The filesystem is built on the third-party msgraphfs package, whose AbstractMSGraphFS
is implemented on top of authlib's AsyncOAuth2Client. That client only understands
OAuth2 client-secret grants — it has no concept of azure-identity credential objects
like CertificateCredential. I traced this by walking msgraphfs's and authlib's
source directly:

  • authlib's default ClientAuth.DEFAULT_AUTH_METHODS only registers
    client_secret_basic, client_secret_post, and none — not private_key_jwt, and
    msgraphfs never calls register_client_auth_method() to add it. So passing
    token_endpoint_auth_method="private_key_jwt" through oauth2_client_params would
    fail at request time.
  • Even if it didn't, Azure AD's certificate-based client-assertion flow requires an
    x5t (certificate thumbprint) JWT header that generic RFC 7523 private_key_jwt
    signing doesn't add automatically — omitting it produces
    AADSTS700027: certificate ... not registered on application.
  • This is a known upstream limitation — the issue author already filed
    acsone/msgraphfs#13 about it.

Given that, this PR fetches a single access token up front via
azure.identity.CertificateCredential.get_token() (the same credential class already
used in KiotaRequestAdapterHook.get_credentials) and hands it to authlib as a static
bearer token in oauth2_client_params, instead of a client_secret.

Known limitation (documented in the docs and code comment): because authlib isn't
driving token acquisition in this path, it can't transparently refresh the token via the
certificate once it expires (~60–90 min, standard Azure AD access token lifetime). Tasks
that run longer than that need to re-resolve the ObjectStoragePath to get a fresh
filesystem/token. This is called out explicitly in the docs and in the AirflowException
users would otherwise silently hit.

How

  • When certificate_path or certificate_data is set on the connection, it now takes
    precedence over client_secret.
  • certificate_password reuses the same connection field as client_secret
    (conn.password), matching KiotaRequestAdapterHook.get_credentials's existing
    semantics — no new connection fields needed, and the existing msgraph connection
    form (which already has "Certificate path" / "Certificate data" widgets from the hook)
    works unchanged.
  • Certificate auth failures are wrapped in a clear AirflowException instead of
    surfacing a raw azure-identity/msgraphfs traceback.

Files changed

  • providers/microsoft/azure/src/airflow/providers/microsoft/azure/fs/msgraph.py — new
    _get_certificate_token() helper + branching logic in get_fs()
  • providers/microsoft/azure/tests/unit/microsoft/azure/fs/test_msgraph.py — 5 new
    tests: cert-path token flow, cert-data + password flow, precedence over
    client_secret, failure handling, existing client-secret tests unaffected
  • providers/microsoft/azure/docs/filesystems/msgraph.rst — new "Certificate-based
    Authentication" section, including the token-refresh limitation
  • providers/microsoft/azure/newsfragments/69335.feature.rst — new feature note

Testing done

  • ruff check / ruff format --diff — clean
  • mypy (isolated, no repo plugin) — clean
  • Added unit tests covering: cert-path auth, cert-data + password auth, precedence
    over client_secret, credential-failure error handling, and confirmed the existing
    client-secret-only tests are unaffected
  • Manually exercised get_fs() end-to-end (mocked CertificateCredential/MSGDriveFS)
    against 4 scenarios to confirm the exact oauth2_client_params built in each case

Not in scope / follow-ups

  • Automatic token refresh for certificate auth mid-task is not implemented — it would
    require either patching msgraphfs upstream or subclassing MSGDriveFS's internal
    client construction. Flagging for maintainers in case there's a preferred direction
    here; happy to follow up in a separate PR once acsone/msgraphfs#13 is resolved
    upstream.

Important

🛠️ Maintainer triage note for @bujjibabukatta · by @potiuk · 2026-07-08 15:38 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed:

  • Pre-commit / static checks failing (CI image checks / Static checks). See the contributor guide.

Full criteria: Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

DEFAULT_SCOPE = "https://graph.microsoft.com/.default"


def _get_certificate_token(

@dabla dabla Jul 9, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Maybe we should create a module level factory method in msgraph hook module which returns a CertificateCredential and have the get_credentials method in KiotaRequestAdapterHook and this module call it, that way we avoid code duplication.

For the rest looking good to me, just need to address static checks errors.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks! Pulled the cert-building logic into a shared
build_certificate_credential() factory in hooks/msgraph.py — both
get_credentials and fs/msgraph.py's _get_certificate_token call it now,
just passing in the sync vs async CertificateCredential class. No more
duplicated construction logic.

Ran ruff/mypy on both files locally and re-verified the fs module delegates
correctly. Existing tests didn't need changes since they mock at the
_get_certificate_token() boundary.

Pushed!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, this was a bad idea of mine, cause I see you have to pass the class as well, as msgraph hook uses the async aio version of it while the fs one uses the sync one, so this refactor doesn't make much sense. I would revert that change as it doesn't offer much honestly and I don't like the fact that we need to pass the class.

So I would revert that last change, if static checks are green, we can merge it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please check now?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change doesn't seem to be reverted

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove the build_certificate_credential method, as it doesn't offer much benefit as hook and fs use different classes anyway.

@potiuk
potiuk marked this pull request as draft July 20, 2026 11:54
@potiuk

potiuk commented Jul 20, 2026

Copy link
Copy Markdown
Member

@bujjibabukatta This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

  • ❌ Merge conflicts: This PR has merge conflicts with main and cannot be merged as-is. Rebase locally and resolve them: git fetch upstream main && git rebase upstream/main, resolve the conflicts, then force-push.

Note: Your branch is 312 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

  • Fix each issue listed above.
  • Make sure static checks pass locally (prek run --from-ref main --stage pre-commit).
  • Mark the PR as "Ready for review" when you're done.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.


Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dabla dabla closed this Aug 10, 2026
@dabla

dabla commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

This PR can be closed as it has been done in this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support certificate-based auth for Microsoft Graph Filesystem

4 participants