From Code Diff to Exploit Replay: Building Autonomous Security Tests for Pull Requests

A pull request changes three lines in a shared repository helper. The visible route tests stay green. That helper feeds twenty endpoints, including a tenant export that once allowed cross-customer access. The security-relevant question is not how small the diff is; it is whether the old exploit's assumptions now intersect the changed symbol.

Running a full pentest on every pull request would be slow and noisy. Semantic selection is narrower: map changed code, configuration, and dependencies to routes, trust boundaries, protected effects, and historical findings, then replay only the invariants whose assumptions moved.

An autonomous agent can perform that mapping and execute bounded tests while the author still has context. The surrounding controls matter as much as speed. Pull-request code is untrusted, policy intent may be ambiguous, and an unusual response is not automatically an exploit. Test selection must be explainable, execution isolated, and blocking reserved for evidence tied to a protected effect.

Key Takeaways

• A line diff is only the starting signal. Security test selection needs a semantic impact graph across routes, policies, data, dependencies, jobs, and infrastructure.
• Historical findings should become versioned exploit-replay artifacts with setup, invariant, action, evidence, cleanup, and selection metadata.
• Autonomous agents add value by connecting code context to targeted runtime proof, not by running every scanner and payload on every PR.
• PR code and test agents must execute in ephemeral, least-privileged environments without production secrets or unrestricted egress.
• Merge gates should block deterministic protected-effect violations and failing known exploit regressions, while ambiguous policy remains a review task.
• Post-merge and scheduled validation still matter because no diff classifier can model every implicit runtime dependency.

Why File-Based Security Selection Is Too Shallow

Many PR pipelines use path rules:

if files under /auth changed -> run authentication tests
if package manifest changed -> run dependency scan
if Terraform changed -> run infrastructure checks

These rules are useful, but they miss semantics. A utility under /shared may build SQL for the billing service. A model serialization change can expose fields through several APIs. A database migration can remove the constraint that made a race condition harmless. A framework upgrade can alter middleware defaults. A cache helper can affect authorization even though its filename contains no security keyword.

Path rules also over-trigger. A documentation update under an authentication package should not provision a full identity lab. A formatting-only change should not launch a dynamic pentest. Coarse routing either wastes time or becomes so narrow that teams stop trusting its coverage.

Semantic selection asks a different question:

Which security-relevant behaviors can change because of this PR,
including behaviors reached indirectly through shared code or configuration?

Answering it requires relationships, not filenames.

Start With Security Invariants

Exploit replay works when the system knows what must remain true. Examples include:

• A user from Tenant Alpha cannot read or mutate Tenant Beta's objects.
• An authorization code produces at most one successful token issuance.
• A resource server rejects tokens issued for another audience.
• A fetcher cannot contact a restricted destination through redirects.
• A non-admin cannot approve, refund, export, or delete protected resources.
• A file path remains inside an approved workspace after canonicalization.
• A patch removes the vulnerable effect without breaking the intended operation.

These are more durable than individual payloads. Routes, libraries, and identifiers can change while the invariant remains. Each replay artifact should link a concrete test to one or more invariants, and each changed code path should link back to the invariants it can affect.

Invariant Ownership

An inferred invariant is not automatically policy. Product, engineering, and security need an owner who can approve expected behavior. If a service intentionally allows support staff to inspect customer metadata, a generic cross-tenant denial rule would generate a false blocker. The replay registry must record exceptions and their exact scope rather than force the agent to guess.

The Semantic Impact Graph

The graph connects code and configuration to runtime security behavior.

Useful Node Types

• Files, symbols, classes, functions, and modules.
• Routes, resolvers, event handlers, jobs, and commands.
• Authentication and authorization controls.
• Data models, queries, storage paths, caches, and search indexes.
• External network sinks and subprocesses.
• Secrets, cloud roles, service accounts, and infrastructure resources.
• Dependencies and resolved versions.
• Security invariants, findings, and replay tests.
• Protected effects such as data disclosure, state change, token issuance, payment, or code execution.

Useful Edges

• Calls, imports, implements, wraps, or overrides.
• Protects or is protected by.
• Reads, writes, serializes, caches, publishes, or executes.
• Receives identity from or forwards identity to.
• Configures or is configured by.
• Deploys to or assumes permission from.
• Fixed by, regressed by, or tested by.

The graph does not need perfect whole-program analysis to be useful. It can combine compiler or language-server references, route metadata, API schemas, ORM definitions, infrastructure plans, ownership files, tests, runtime traces, and historical findings. Every edge should carry provenance and confidence. A direct call edge is stronger than a name-based inference.

The Diff-to-Replay Pipeline

A practical pipeline has ten stages.

Each stage should be inspectable. A developer needs to know why a test was selected, what changed, which invariant it protects, and why the result blocks or does not block merge.

Stage 1: Normalize More Than Source Lines

The pipeline should compare the effective build, not just handwritten files.

Include:

• Source and test changes.
• Dependency manifests and lockfiles.
• Generated clients, schemas, and policy bundles.
• Database migrations and ORM metadata.
• Container images and base-image references.
• Infrastructure-as-code plans.
• CI workflows, runner permissions, and deployment configuration.
• Feature flags and security defaults.
• API contract changes.

A lockfile-only change may be the most security-relevant part of a PR. A generated policy file may differ even when its source template looks harmless. If generation happens after review, test selection must consume the generated delta too.

Stage 2: Classify Security-Relevant Semantics

The classifier looks for changed behavior, not vulnerability labels.

High-value signals include:

• New or modified routes and object references.
• Authentication, session, token, password-reset, SSO, or MFA logic.
• Roles, permissions, tenant filters, sharing, and policy composition.
• SQL, ORM, search, cache, storage, and serialization behavior.
• User-controlled URLs, files, templates, commands, and deserialization.
• State transitions, idempotency, locking, counters, and one-time artifacts.
• Payment, approval, export, invitation, and account-recovery workflows.
• Cloud IAM, Kubernetes RBAC, network egress, and secret access.
• Dependency upgrades affecting parsers, middleware, clients, or cryptography.

The output should say "authorization helper composition changed" or "redirect owner moved from wrapper to delegated client," not merely "security file changed."

Stage 3: Expand Impact Without Exploding Scope

Graph traversal needs limits. Starting from a changed shared function and selecting every route in the monorepo can make the pipeline unusable.

Use bounded expansion based on:

• Direct callers and known runtime consumers.
• Protected-effect proximity.
• Service ownership and deployment boundary.
• Historical evidence that the node affected a finding.
• Changed interface or behavior versus internal refactor.
• Confidence of each graph edge.
• Test cost and PR risk budget.

If an authorization repository method changes its tenant predicate, all callers that access tenant-owned objects deserve attention. If only logging changes behind an unchanged interface, dynamic authorization replay may be unnecessary unless the diff alters control flow.

Low-confidence wide impacts can schedule broader asynchronous review rather than block the PR immediately.

The Exploit Replay Registry

A replay is more than a saved HTTP request. It is a portable security test artifact.

id: AUTHZ-TENANT-0042
invariant: tenant-object-isolation
affected_nodes:
  - repository.invoice.find
  - route.invoice.export
preconditions:
  - tenant_alpha_member
  - tenant_beta_invoice
setup: fixture://two-tenant-invoice
action: replay://cross-tenant-export
oracle:
  response: denied
  export_job_created: false
  artifact_created: false
evidence:
  - http_trace
  - job_table_snapshot
  - storage_canary
cleanup: cleanup://invoice-run
safety: non_production_test_data
selection_tags: [authorization, tenant, export, jobs]
owner: appsec-and-billing

The registry should also store:

• Original root cause and fix commit.
• Required application and schema version range.
• Last passing run and environment.
• Determinism and flake history.
• Runtime cost and required capabilities.
• Secrets and data classification.
• Sibling variants and superseded artifacts.
• Expiration or review date.

Without maintenance metadata, replay libraries become graveyards of broken scripts. A failing artifact may indicate a vulnerability, an intentional contract change, fixture drift, or test rot. The pipeline must distinguish them.

Selecting the Right Tests

Selection can be expressed as a score:

priority =
  impact_of_invariant
  * reachability_confidence
  * change_proximity
  * historical_relevance
  * determinism
  / execution_cost

The exact formula is organization-specific. The important behavior is transparent ranking.

Run the smallest set that gives strong evidence, then widen when the impact graph or first result justifies it.

End-to-End Selection Example

Suppose a pull request replaces a tenant-scoped object loader with a new generic repository helper. The line diff touches one method. Semantic classification labels the change as object lookup and authorization-context propagation. The impact graph then finds four direct callers, an export job invoked by one caller, and two historical findings linked to the old loader.

The selector retrieves the tenant-object-isolation invariant and chooses cross-tenant read, update, and export replays. It does not select JWT rotation or SSRF tests because no credible graph path connects them to the change. The environment provisions two synthetic tenants, one private object, and an export canary. Execution shows direct reads are still denied, but the asynchronous export job is created for the wrong tenant and writes an artifact.

The result is not "repository helper looks risky." It is a deterministic violation with a trace from changed symbol to route, queue message, worker query, and protected artifact. The PR check blocks, points to the worker's missing tenant context, and links the replay identifier. After the fix, the same three cases run again; the export job is no longer created, and the artifact canary remains empty.

This example shows why semantic selection can be both narrower and deeper than a full scan. It avoids unrelated tests while following the one changed abstraction far enough to catch an effect outside the request that initiated it.

Provisioning an Ephemeral Test World

The PR environment needs enough realism to prove effects without inheriting production risk.

Required Properties

• Built from the exact candidate commit and resolved dependencies.
• Disposable database, cache, queues, object storage, and identity fixtures.
• Synthetic users, tenants, files, tokens, and financial values.
• No production credentials or copied customer datasets.
• Restricted egress with test-owned canaries and approved dependencies only.
• Least-privileged cloud or cluster identity.
• Resource quotas, timeouts, and a kill switch.
• Complete run correlation and reliable cleanup.

Pull Request Code Is Untrusted

This point is often missed. The code under test has not been approved. A malicious or compromised PR can try to read CI secrets, alter test results, contact external endpoints, or persist in a shared runner. Security testing infrastructure must assume the candidate code is hostile.

Use isolated workers, short-lived credentials, read-only source access where possible, no privileged host mounts, no reusable cloud tokens, and explicit egress policy. Fork-origin PRs require even stricter treatment and may run only non-secret static analysis until a trusted approval step.

The agent itself also needs scoped tools. Code-reading does not require production network access. Test execution does not require repository write permission. Evidence generation does not require raw secret access. Separate capabilities and log each action.

Runtime Proof: Observe the Protected Effect

An exploit replay should verify what matters beyond the response.

Examples:

• Authorization: row, object, job, event, or file remains unchanged for a denied principal.
• Race condition: exactly one state transition and credential family exist.
• SSRF: restricted canary and egress logs show no connection.
• JWT confusion: wrong-context token cannot invoke the protected action.
• Injection: harmless marker demonstrates data-plane impact without extracting unrelated records.
• File boundary: write remains inside disposable workspace and no external path changes.
• Cloud permission: test identity cannot perform the forbidden control-plane action.

The oracle should come from fixture truth, policy, or independent state observation, not from the same potentially vulnerable response. Agents can gather evidence, but a reviewed invariant decides the outcome.

Merge Gate Design

A good gate separates evidence states.

Do not let the agent hide provisioning errors as passing security checks. A required replay that never ran is not a pass. Conversely, do not block on a speculative natural-language concern without a policy basis or high-confidence code proof.

PR Feedback Format

Keep the comment close to engineering work:

• Why selected: changed symbol and impact path.
• Invariant: what must remain true.
• Observed: concise response and effect evidence.
• Impact: protected boundary crossed.
• Root cause: smallest relevant code or configuration location.
• Fix direction: actionable pattern, not a generic warning.
• Replay: check identifier and rerun status.

Long evidence belongs in a secured artifact linked from the check. Never place secrets, raw tokens, customer data, or unsafe payload details in public PR comments.

Handling Flakes, Drift, and Intentional Change

Security gates lose trust when replay tests are unstable.

Determinism First

Use barriers for race tests, fixture-owned identifiers for authorization, controlled redirectors for SSRF, synthetic issuers for JWT, and observable state markers. Avoid sleeps, random production discovery, and dependencies on third-party uptime.

Quarantine Is Not Closure

If a critical replay flakes, investigate immediately. A temporary quarantine must retain visibility, owner, reason, and deadline. Repeatedly rerunning until green turns a security failure into probability management.

Version the Oracle

Intentional policy changes require owner approval and an updated invariant before the old test is removed. Preserve history: what access changed, who approved it, what new negative boundaries replace the old ones, and whether data migration or token revocation is needed.

Detect Test Tampering

Candidate code should not be able to modify the trusted oracle, replay registry, or result-signing path without review. Security test definitions can live in protected repositories or require code-owner approval. The pipeline should report when a PR changes both a control and the test that would catch its failure.

Human Judgment and Agent Autonomy

Agents are strong at repetitive graph traversal, fixture generation, protocol mutation, and evidence collection. Humans remain necessary for:

• Approving business and trust invariants.
• Resolving legitimate exceptions.
• Evaluating chained impact and organizational context.
• Authorizing destructive or production-adjacent tests.
• Reviewing auto-fixes to security-sensitive code.
• Accepting risk and defining expiration.

The right division is not human versus automation. It is machine-scale execution under human-approved policy. An agent may suggest that a new export should be tenant-bound; only an accountable owner can establish that as the product contract.

Post-Merge and Scheduled Coverage

PR replay is a targeted snapshot. It can miss:

• Runtime configuration applied only after deployment.
• Service interactions outside the ephemeral environment.
• Delayed jobs and scheduled sync.
• Infrastructure policy composed at organization level.
• Dependency or base-image changes introduced by rebuilds.
• New threat intelligence after merge.
• Impact edges absent from the graph.

Run selected smoke replays after deployment, broader suites on a schedule, and full authorized assessments at risk-based intervals. Feed new findings back into the registry so the PR system improves over time. Continuous validation is a loop, not one merge check.

Metrics That Reward Security, Not Activity

Useful measures include:

• Percentage of critical invariants linked to executable replays.
• Percentage of verified findings converted to regression artifacts.
• Median time from PR update to focused security result.
• Replay selection precision: selected tests that were genuinely relevant.
• Critical replay execution success versus provisioning failure.
• Flake rate and mean time to repair test drift.
• Verified regressions prevented before merge.
• False-block rate and time to resolve policy ambiguity.
• Coverage of shared helpers, trust boundaries, and protected effects.
• Post-merge issues that should have been selected by the PR graph.

Raw number of tests, requests, or agent actions is not a security outcome. A five-minute replay that blocks a real tenant bypass is more valuable than thousands of generic checks that nobody can interpret.

FAQ

Does diff-aware security mean testing only changed lines?

No. The diff seeds a semantic impact graph. Tests may cover unchanged routes, services, or jobs that consume changed shared code, configuration, schemas, or dependencies.

Should every pull request run a full autonomous pentest?

No. Use the impact graph to select focused invariant and exploit-replay tests. Broader scheduled and post-deployment validation covers uncertainty and implicit dependencies.

What makes an exploit replay different from a scanner rule?

A scanner rule usually identifies a pattern or signal. A replay provisions prerequisites, performs a bounded action, checks an independent security invariant and protected effect, preserves evidence, and verifies the fix over time.

Can an AI agent decide whether a finding blocks merge?

It can apply a pre-approved evidence policy. It should not invent product policy or silently elevate ambiguous behavior into a blocker. Human owners define invariants, exceptions, severity rules, and risk acceptance.

How do we keep PR security tests from exposing CI secrets?

Treat candidate code as untrusted. Use ephemeral isolated runners, short-lived least-privileged credentials, no production datasets, restricted egress, separated agent capabilities, protected oracles, and secret-free handling for untrusted forks.

Historical Exploits Can Become Change Context

A verified finding contains more than a report: it can supply an invariant, a safe replay, a protected-effect oracle, and a map of the code and configuration on which the path depends. When a pull request touches one of those dependencies, the author can see why the old scenario is relevant and observe it against synthetic state while the change is still fresh.

Reproducible evidence may justify a merge gate; a plausible but unverified relationship is better kept as review context. Over time, this registry gives the pipeline a practical memory of how the product has failed and which kinds of changes can reopen those paths. The diff starts the analysis, but it is not the whole security boundary.

#ApplicationSecurity #ExploitReplay #AutonomousPentesting #PullRequestSecurity #DevSecOps #SecureSDLC #CodeSecurity