The conformance suite

The conformance suite is 13 adversarial tests. It runs against a live Subact ID instance through its public endpoints only and references no server code. Each test guards one server check: remove the check and the test fails. Where a test provokes a refusal, it also checks that the refusal was written to the audit ledger.

The maintainers own the suite, and it is a required CI job.

The tests

#	Test	What it asserts
1	The subject is always the human	sub is the human; act.sub and client_id are the agent. A subject token whose subject is an agent is invalid_grant.
2	Scope at exchange is the intersection	The token holds only user ∩ agent ∩ requested scopes. An empty intersection is invalid_scope.
3	Scope only narrows on refresh	A scope the task does not hold is invalid_scope, even when the user and agent allow it. A refresh keeps the task_id and issues a new jti.
4	The audience must be allowed, and a task is bound to it	An audience outside allowed_audiences is invalid_target, and so is a refresh for another audience.
5	A token never outlives its task	With a one-minute task, exp is never past the task’s expiry, at issue or at a refresh 20 seconds in.
6	A client assertion is accepted once, from the registered key	A replayed assertion, a forged signature and an unknown agent are each invalid_client.
7	Revocation and disabling take effect at once	After a kill switch, the agent revoking its own grant, or disabling the agent, refresh is refused. After a kill switch or disabling, the token introspects inactive with the reason.
8	A subject token is trusted only from the identity provider	Expired, foreign-signed, wrong-audience and wrong-issuer subject tokens are each invalid_grant.
9	A task dies with the human it acts for	A person disabled or deleted at the identity provider fails the next refresh with access_denied. Needs poll mode.
10	The audit ledger is sealed	New records are sealed within a minute. The suite rebuilds each leaf, folds its audit path to the root, and checks each checkpoint’s signature against the JWKS and its link to the previous one.
11	A high-risk audience says so in the token	introspect_required is true for an audience in high_risk_audiences and absent for any other.
12	No refusal hands back the credential it refused	No subject token, client assertion or task grant appears in an error body, whole or in part.
13	A human the control plane will not act for is refused	After PUT /admin/sponsors/{key}/block, the token introspects inactive and refresh and a new exchange are access_denied, while the identity provider still reports the person as active.

Against the five invariants: tests 1 to 3 hold the first two, test 5 the third, tests 9, 10 and 13 and every test’s ledger check the fourth, and test 12 the fifth.

Running it

The suite runs a stub for everything the instance calls out to: the identity provider (discovery, JWKS, the admin users API and the token endpoint the sponsor check uses) and every agent’s JWKS. Point the instance at one HTTPS base URL for all of these. The suite listens there with a certificate the instance must trust. The URL must be HTTPS, because agent JWKS are only fetched over HTTPS.

Variable	Meaning
SUBACTID_CONFORMANCE_URL	Base URL of the instance under test
SUBACTID_CONFORMANCE_ADMIN_KEY	The instance’s SubactId:Admin:ApiKey
SUBACTID_CONFORMANCE_STUB_URL	The HTTPS base URL the instance uses for the identity provider and agent JWKS
SUBACTID_CONFORMANCE_STUB_PFX	A PKCS#12 file with no password for that URL’s host. The stub keeps its signing key next to it, so repeated runs against one instance use the same key

SUBACTID_CONFORMANCE_URL=https://subactid.internal.example.com \
SUBACTID_CONFORMANCE_ADMIN_KEY=$SUBACTID_ADMIN_KEY \
SUBACTID_CONFORMANCE_STUB_URL=https://localhost:5199 \
SUBACTID_CONFORMANCE_STUB_PFX=./stub.pfx \
dotnet test tests/SubactId.Conformance

The suite registers its own agents and writes to the ledger, so run it only against an instance you are willing to write to. If configuration is missing or the instance does not answer, every test fails with the reason. The CI job fails if the suite runs no tests.

Instance settings the suite relies on

The shipped defaults satisfy all of these except the checkpoint interval.

Setting	Required value	Why
SubactId:Audit:Aggregation:Window	At most one minute (default PT1M)	The suite looks back one minute for denials that name nobody
SubactId:Audit:Checkpoint:Interval	Well under one minute; CI uses PT5S	Test 10 waits at most one minute for its records to be sealed
SubactId:Agents:MinTaskTtl	At most PT1M (default PT1M)	Test 5 registers a one-minute task
SubactId:Tokens:DefaultTaskTtl	At least PT1M (default PT30M)	Test 5 refreshes 20 seconds into its task
SubactId:RateLimit:Burst	At least 120 (default 120)	The suite sends its requests from one address in a few seconds, and 120 leaves room for all of them
SubactId:UpstreamIdp:SponsorKeyClaim	sub (default)	Test 13 blocks a person by the subject it put in their token

SubactId:UpstreamIdp:SponsorCheck:CacheTtl can be anything: test 9 uses a different person for each outcome.

Signals mode

Test 9 disables a person at the stub’s admin API, which a signals-mode instance never asks. The other 12 tests do not depend on the sponsor check mode. CI runs them a second time against an instance in signals mode, with test 9 excluded by name, and fails if fewer than 12 run.

Test 13 covers the same rule as test 9 through Subact ID’s own block list, which every instance enforces in either mode.

What the suite does not cover

No test posts to /backchannel-logout, /scim/v2 or /events. Those receivers exist only when configured, so a test for them would fail a conformant instance that has not enabled them. Integration tests against a real database cover them instead.