Skip to content

test(auth): enable mTLS and actor_token WIF integration tests by default - #14554

Merged
macastelaz merged 38 commits into
googleapis:oauth2-bound-tokensfrom
macastelaz:enable-mtls-wif-integration-tests
Oct 16, 2026
Merged

macastelaz merged 38 commits into
googleapis:oauth2-bound-tokensfrom
macastelaz:enable-mtls-wif-integration-tests

Conversation

@macastelaz

@macastelaz macastelaz commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Enables the 4 mTLS + actor_token Workload Identity Federation integration tests in ITWorkloadIdentityFederationTest to run by default in CI without requiring GOOGLE_API_CERTIFICATE_CONFIG or external Secret Manager secrets:

  1. Default mTLS Certificate Config Fallback: Falls back to the checked-in testresources/mtls/certificate_config.json (test_cert.pem / test_key.pem) when GOOGLE_API_CERTIFICATE_CONFIG is unset, removing the assumeTrue skip.
  2. Signed OIDC JWTs with cnf.x5t#S256 and may_act: Targets oidc-jwt-1 (OIDC_JWT_AUDIENCE) and uses the GOOGLE_APPLICATION_CREDENTIALS service account private key (whose public key is registered in oidc-jwt-1's jwksJson) to sign OIDC JWTs containing RFC 8705 certificate thumbprint binding (cnf.x5t#S256) and delegation (may_act) claims.

…401 recovery

- Pin mTLS HttpTransportFactory across multi-step STS and IAM token exchanges so both requests use the exact same certificate snapshot within a single refresh cycle.
- Add 401 Unauthorized recovery with automatic certificate reload from X509Provider and single-retry coordination in IdentityPoolCredentials and ImpersonatedCredentials.
- Preserve custom non-default HttpTransportFactory instances when X509Provider is configured.
- Add comprehensive unit tests across IdentityPoolCredentialsTest, ImpersonatedCredentialsTest, and OAuth2UtilsTest.
…dentials and address review findings

- Explicitly scope inner sourceCredentials to CLOUD_PLATFORM_SCOPE in ExternalAccountCredentials.buildImpersonatedCredentials and ImpersonatedCredentials.refreshAccessToken so STS issues tokens authorized to call IAM generateAccessToken even when downstream target scopes are configured via createScoped.
- Ensure public no-arg ImpersonatedCredentials.refreshAccessToken delegates without overriding source credential transport settings.
- Preserve custom actorTokenSupplier in IdentityPoolCredentials.Builder copy constructor when credentialSource is present.
- Ensure HTTP response is closed in a finally block in ImpersonatedCredentials.refreshAccessToken.
- Attach initial 401 exception as suppressed when the 401 retry attempt fails in IdentityPoolCredentials.refreshWithRetry.
- Add unit tests in IdentityPoolCredentialsTest and ImpersonatedCredentialsTest covering scoped impersonation, custom actorTokenSupplier preservation, and retry exception chaining.
…ration tests

- Add MtlsPipelineLocalTest providing hermetic in-process socket tests over JDK HttpsServer with client certificate authentication (peer cert verification, 401 retry with cert rotation, concurrent refreshes, atomic token read, IAM impersonation mTLS transport pinning, and 401 retry with fresh cert).
- Add ITWorkloadIdentityFederationTest extensions for certificate-bound workload + actor token JSON config and programmatic mTLS token suppliers covering both direct STS and Service Account Impersonation.
- Fix OAuthException to safely handle null HTTP error response content.
…tlsPipelineLocalTest, and ITWorkloadIdentityFederationTest
…th mTLS impersonation, and retry torn cert rotations
…als and retry bare IOExceptions on split cert writes
… and single cert read on impersonated refresh
…als and CertificateIdentityPoolSubjectTokenSupplier
…LS pinning

- Replace the ThreadLocal 401 signal with CachedStsTokenRejectedException.
- Only clear the cached STS token on IAM failure when this refresh used it and
  it is still the cached token.
- Scope the external source before taking the pinned impersonation path, and
  clear any access token copied by createScoped().
- Simplify isRetryableTransportException and drop redundant retry checks.
- Pass the cycle transport factory through for Aws/Pluggable impersonation.
- Javadoc and comment clarifications.
- Add tests for concurrent STS cache handling, standalone
  ImpersonatedCredentials 401 retries, transport factory upgrade, source
  scoping, isInvalidGrantException, and Aws/Pluggable cycle factories.
- Clear the copied access token before building the impersonation source credentials, so a rebuilt source (e.g. after deserialization) mints a fresh STS token instead of sending the cached service account token to IAM.
- Pin serialVersionUID for CertificateIdentityPoolSubjectTokenSupplier and OAuthException to their released computed values so previously serialized instances still load.
… cert-bound-oauth-integration-tests

# Conflicts:
#	google-auth-library-java/oauth2_http/java/com/google/auth/oauth2/OAuthException.java
#	google-auth-library-java/oauth2_http/javatests/com/google/auth/oauth2/OAuthExceptionTest.java
@macastelaz
macastelaz requested review from a team as code owners September 30, 2026 02:33

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces robust support for per-cycle mTLS transport pinning, certificate rotation, and automatic retry recovery across both STS and IAM token exchanges in Google Workload Identity Federation. Key changes include extracting leaf certificates directly from pinned KeyStores, caching intermediate STS tokens in ImpersonatedCredentials to prevent redundant exchanges, and implementing a retry mechanism in IdentityPoolCredentials that reloads rotated certificates upon encountering 401 Unauthorized or 400 invalid_grant errors. Additionally, helper utilities for comparing KeyStore entries and parsing structured OAuth exceptions were added, along with a comprehensive hermetic integration test suite (MtlsPipelineLocalTest) to validate the entire mTLS pipeline. There are no review comments, so I have no feedback to provide.

@macastelaz
macastelaz requested review from lsirac and nbayati October 15, 2026 19:47

@nbayati nbayati left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a small nit.

assumeTrue(
certConfigPath != null && !certConfigPath.isEmpty(),
"Skipping mTLS test: GOOGLE_API_CERTIFICATE_CONFIG env variable is not set.");
if (certConfigPath == null || certConfigPath.isEmpty()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Cloudtop, GOOGLE_API_CERTIFICATE_CONFIG is set by default to /etc/gcloud/certificate_config.json (which only has a pkcs11 config for device certificates and no workload section.

Since these tests are not testing environment discovery, could we drop getMtlsCertificateConfigPath() and use DEFAULT_MTLS_CERT_CONFIG_PATH directly so they run hermetically everywhere?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1e02910. Dropped getMtlsCertificateConfigPath() and updated the tests to use DEFAULT_MTLS_CERT_CONFIG_PATH directly.

@macastelaz
macastelaz merged commit 47374cf into googleapis:oauth2-bound-tokens Oct 16, 2026
274 of 277 checks passed
macastelaz added a commit that referenced this pull request Oct 30, 2026
…on for Workload Identity Federation (#14781)

### Summary

Merges the `oauth2-bound-tokens` feature branch into `main`, adding
end-to-end support for certificate-bound OAuth 2.0 tokens and automatic
mTLS certificate rotation in Workload Identity Federation
(`IdentityPoolCredentials` and `ImpersonatedCredentials`).

#### Included Feature Branch PRs
- #13955 — `feat(oauth2): Extract actor tokens for cert-bound OAuth2 STS
exchange`
- #14212 — `feat(oauth2): implement IAM impersonation mTLS transport
pinning and 401 recovery`
- #14220 — `test(oauth2): add mTLS in-process socket tests and live GCP
WIF integration tests`
- #14430 — `feat(oauth2): support non-mTLS token URLs and unbound actor
tokens in IdentityPoolCredentials`
- #14554 — `test(auth): enable mTLS and actor_token WIF integration
tests by default`

#### Key Capabilities
- **Actor Token Support for Hybrid Certificate-Bound WIF
(`IdentityPoolActorTokenSupplier`,
`FileIdentityPoolSubjectTokenSupplier`)**:
- Supports extracting both a primary X.509 leaf certificate subject
token and a secondary file-sourced `actor_token` (text or JSON via
`actor_token_field_name` and `actor_token_type`) for STS token exchange.
- Caches file-sourced tokens using file metadata (`mtime`, size, file
key) and JWT `exp` claims with a 5-minute expiration margin to avoid
redundant disk reads.
- **Per-Cycle mTLS Transport Pinning & Certificate Rotation
(`IdentityPoolCredentials`, `ImpersonatedCredentials`,
`MtlsHttpTransportFactory`)**:
- Snapshots the `KeyStore` once per refresh cycle so leaf certificate
extraction, the STS token exchange, and downstream IAM Service Account
impersonation (`generateAccessToken` / `generateIdToken`) share the
exact same certificate and private key snapshot.
- Automatically recovers from `401 Unauthorized`, OAuth `invalid_grant`,
and transient TLS/KeyStore load failures during non-atomic split
cert/key writes by reloading the `KeyStore` once and retrying when a
certificate change is detected (`OAuth2Utils.hasCertificateChanged`).
- Supports standalone `ImpersonatedCredentials` configured with
`MtlsProvider` as well as delegated `ImpersonatedCredentials` chains.
- **Test Coverage**:
- Comprehensive unit tests across `IdentityPoolCredentialsTest`,
`ImpersonatedCredentialsTest`,
`FileIdentityPoolSubjectTokenSupplierTest`,
`MtlsHttpTransportFactoryTest`, `OAuth2UtilsTest`, and
`OAuthExceptionTest`.
- In-process mTLS socket tests (`MtlsPipelineLocalTest`) and live GCP
Workload Identity Federation integration tests
(`ITWorkloadIdentityFederationTest`).
  
  ### Manual & Live Verification
End-to-end live GCP and real-TLS socket fault-injection testing was
executed against `oauth2-bound-tokens` (`65d592bd783`):
- **Full verification report, logs, and test harness**:
https://paste.googleplex.com/6755399436525184
- **Scenarios verified (15 / 15 passed)**:
1. **Live Direct mTLS STS Exchange (`sts.mtls.googleapis.com`)**:
File-based JSON `subject_token` + `actor_token` with RFC 8705
`cnf.x5t#S256` certificate binding and atomic single-supplier file
reads.
2. **Serialization Round-Trip (`readObject`) + `createScoped()`**:
Deserialization of `IdentityPoolCredentials` with mTLS state
reconstruction across multiple live STS refresh cycles.
3. **Full 3-Hop Live mTLS Chain (`STS mTLS` -> `IAM Impersonation mTLS`
-> `GCS mTLS`)**: Cert-bound workload + actor token exchange at
`sts.mtls.googleapis.com`, service account impersonation at
`iamcredentials.mtls.googleapis.com`, and bucket access at
`storage.mtls.googleapis.com` (`HTTP 200`).
4. **Programmatic Builder (`Suppliers` + `MtlsHttpTransportFactory`)
Live 3-Hop mTLS**: End-to-end `STS mTLS` -> `IAM mTLS` -> `GCS mTLS`
(`HTTP 200`) and verification of `OAuth2Utils` mTLS helpers
(`hasMtlsKeyStore`, `isKeylessMtlsTransportFactory`,
`isDefaultOrMtlsTransportFactory`).
5. **Non-mTLS Unbound Actor Token over Public Endpoints**: Unbound
`subject_token` + `actor_token` exchange over public
`sts.googleapis.com` -> `iamcredentials.googleapis.com` ->
`storage.googleapis.com` (`HTTP 200`).
6. **X.509 Certificate Workload Identity Federation**: Leaf certificate
extraction and `trust_chain` handshake against `sts.mtls.googleapis.com`
via `CertificateIdentityPoolSubjectTokenSupplier`.
7. **`PluggableAuthCredentials` Impersonation Transport
Reconciliation**: Rebuilding `PluggableAuthCredentials` with a custom
`HttpTransportFactory` and verifying both STS and
`ImpersonatedCredentials` use the reconciled transport against live STS,
IAM, and GCS (`HTTP 200`).
8. **Review Feedback Edge Cases**:
`FileIdentityPoolSubjectTokenSupplier` TEXT format subject/actor token
handling and `OAuthException.createFromJsonErrorResponse` parsing across
live STS 400 errors, nested `google.rpc.Status` JSON, and empty-body
HTTP error fallbacks.
9. **Real-TLS Socket Fault Injection**: Mid-cycle cert rotation pinning
between STS and IAM, `401 Unauthorized` IAM recovery (both fresh and
cached STS token eviction), `400 invalid_grant` STS cert rotation
recovery, and 8-thread concurrent mTLS refresh.

---------

Co-authored-by: Jin Seop Kim <jinseop@google.com>
Co-authored-by: Blake Li <blakeli@google.com>
Co-authored-by: Sakthivel Subramanian <179120858+sakthivelmanii@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants