HEAVEN
An Autonomous
Penetration-Testing Framework

HEAVEN at a Glance

HEAVEN is a production-grade platform, not a prototype. The numbers below grow with each release, so they are written as floors rather than fixed counts.

Why HEAVEN Exists

The problem it was built for

Security is a lopsided contest. A defender has to cover every way in across the whole estate; an attacker needs one. As systems have grown into tangles of cloud services, microservices, container fleets, and identity providers, the ground a defender holds keeps growing while the attacker's job stays the same size. A yearly pen test and a signature scanner cannot keep pace with people who weaponise a fresh disclosure within hours.

Two responses fall short on their own. Traditional scanners are fast but shallow, they flag patterns and leave a human to sort the real from the noise. Recent LLM-driven agents are clever but unsafe, they will state things that are not true, and acting on a hallucinated command against a live system can cause an outage or break scope. HEAVEN was built to sit between the two: automate the repeatable work at machine scale, but never report a finding a real detector has not confirmed, and never take an action outside the agreed boundary.

What HEAVEN is

HEAVEN is a production-grade penetration-testing platform that automates the time-consuming parts of an engagement so the operator can focus on judgement. It runs three ways from the same dataset, a CLI for scriptable and CI use, a web console, and a REST-plus-WebSocket API, and it covers the full arc of an assessment from first recon to a client-ready report. The rest of this paper is about how it is built, why those choices matter, and what happened when it was pointed at a real target.

Contributions

• A verify-first autonomous loop that uses an LLM for breadth of ideas but gates every finding behind a real detector, which removes hallucination as an operational risk.
• A hybrid CVSS risk model trained on 315,648+ real NVD CVEs that only ranks findings with no published score, leaving the client-facing severity computed exactly from the CVSS vector.
• A safety layer, authorisation gate, credential redaction, and tamper-evident audit log, that makes autonomous action defensible on a live, sensitive estate.
• A real deployment: verification on live DVWA and use as the technical instrument in an MSc dissertation assessing a UK social-housing provider.

Literature Review

HEAVEN sits at the meeting point of four strands of prior work: reinforcement learning for automated testing, the recent turn to large language models as planners, the study of how these systems are benchmarked, and the adversarial-robustness literature that explains why a detector's headline accuracy can be misleading. Reading them together is what motivates HEAVEN's central design choice, that a model proposes and a deterministic detector confirms.

Reinforcement learning for automated testing

The first attempts to automate penetration testing framed it as a sequential decision problem. Schwartz and Kurniawati (2019) modelled a network as a Markov decision process and trained an agent to choose actions that lead to compromise, and Ghanem and Chen (2020) and Hu, Beuran, and Tan (2020) carried the idea into multi-host networks and deep reinforcement learning. The line is theoretically clean but runs into a practical wall: an agent trained in a simulator learns the simulator, and the gap between that and a real, messy target is large. That sim-to-real gap is a recurring theme in the survey literature, and it is one reason HEAVEN validates on real targets rather than on an abstract network model.

Large language models as planners

The current wave puts a language model in the planner's seat. Deng et al.'s PentestGPT showed an LLM could drive a web-app engagement when the task was split into reasoning, generation, and parsing; Kong et al.'s VulnBot spread the work across specialised agents; and Tran et al.'s DCIPHER automated the prompt coordination between them. Ginige et al. (2025) continued the line with AutoPentester. The capability is real, and so is the risk: Fang et al. (2024) demonstrated LLM agents autonomously exploiting one-day vulnerabilities, which sharpens the safety question rather than settling it. The common failure mode across these systems is confident wrong output, an agent asserting a vulnerability, a command, or a CVE that does not exist, which is exactly what a live engagement cannot tolerate.

How these systems are measured

A growing body of work asks whether the numbers reported for these tools mean what they appear to. Benchmarks such as AutoPenBench (Gioacchini et al.) and Cybench (Zhang et al.) brought reproducibility to LLM evaluation, and Bakker and Hastings found that fully autonomous agents underperform human-assisted ones on real capture-the-flag tasks, which is direct evidence that a benchmark score overstates readiness. The concern is not unique to offensive tooling. In the detection literature, work the present authors contributed to, Kumar, Chasmawala, et al.'s Accuracy Is Not Robustness (ICETCS), shows that a phishing detector reporting 97 to 99 percent accuracy can collapse to the high sixties once an attacker edits the features under their control, and that clean accuracy predicts neither the size of that drop nor which model resists best. The lesson transfers directly: a security tool has to be measured under the conditions it will actually face, and its evaluation has to be reproducible and honest about what it does not cover.

Positioning HEAVEN

Against that background, HEAVEN's contribution is not another LLM agent but a discipline about truth wrapped around one. It uses a model for breadth of ideas and a deterministic detector for every reported finding, so hallucination is removed as an operational risk rather than merely reduced. And it takes the measurement critique seriously: rather than a single demo number, it ships a five-tier benchmark suite, scored through one precision, recall, and F1 layer, that anyone can reproduce, which is the evidence the rest of this paper rests on.

The Ideas HEAVEN Is Built On

Propose with a model, confirm with a detector

The single most important rule in HEAVEN is that an LLM's assertion is never reported as fact. Inside the loop, a vuln-hypothesis agent lets the model propose where a weakness might be, and that hypothesis is handed to a real detector to confirm or reject. A proposal nothing can back up is dropped. This is what lets HEAVEN use a language model for its strength, breadth of ideas across an unfamiliar target, without inheriting its weakness. It is the practical answer to the hallucination problem the field keeps running into.

Proof is active, not assumed

A finding either carries its own proof or it is not reported as confirmed. SQL injection is proven with an sqlmap dump; remote code execution by dropping and reading back a canary file; SSRF and XXE out-of-band through an in-house OAST collaborator that needs no Burp Collaborator or interactsh dependency. Every confirmed finding ships with a defensible evidence package: the request and response, a copy-pasteable curl repro, the detection rationale, remediation, and CWE, OWASP, and MITRE references.

Safety is a layer, not a setting

Three properties make HEAVEN fit to run on a live, sensitive estate, and they sit over the engines rather than beside them. An authorisation gate refuses any destructive action without explicit sign-off and blocks anything outside agreed scope, so the operator cannot stray. Credentials seen during testing are redacted before anything is written to disk. The engagement log is a keyed hash, append-only, so a record cannot be altered afterwards without detection. These are the conditions under which a provider holding tenant data can reasonably let the work run at all.

It should run anywhere, with or without a key

Every AI feature has a deterministic fallback, so scanning, the console, reports, and ML scoring all work with zero API keys. When a model is wanted, HEAVEN is provider-agnostic across Anthropic, OpenAI, Gemini, and DeepSeek, and can point its entire AI layer at a local model through Ollama or any OpenAI-compatible server, which keeps findings on the host under an NDA. That the loop still runs with no external service is what makes an assessment reproducible.

HEAVEN Architecture

One engagement dataset drives three interfaces: a 62+ command CLI for scriptable and CI-friendly workflows, a 30+ page React console (scan launcher, live findings, combined risk, kill chain, reports), and a REST-plus-WebSocket API across 99+ RBAC-protected routes. Running the same evidence three ways removes the transcribe-between-tools step that usually breaks an audit trail.

Beneath the interfaces sits an asynchronous, dependency-aware orchestrator that sequences the work as a resumable, checkpointed task graph with stealth timing from 1 to 5. It coordinates five engine groups, reconnaissance, vulnerability detection, verified exploitation and post-exploitation, machine-assisted analysis, and reporting, over a storage layer. Storage is an async PostgreSQL 29+ table schema with a partitioned audit log, plus a zero-config SQLite fallback that presents the same interface where a single file is one engagement. The security layer, JWT RBAC across admin, operator, viewer, and auditor roles, an AES-256-GCM credential vault, an HMAC-signed audit log, and LLM credential redaction, sits over every engine so none of them can act on an unauthorised target.

The package layout

The framework is organised into clear engine groups, each a set of modules under the main package: recon, vulnscan, postex, ai, ml, mitre, devsecops, db, security, api, and cli, with the React console as a separate front end. That separation is what lets each capability grow on its own, and it is why the module and command counts are floors that rise with each release rather than fixed numbers.

HEAVEN Capabilities, End to End

HEAVEN covers the whole arc of an assessment across 15+ scan modes. The tables below group the coverage; each area is an engine family that keeps expanding.

Reconnaissance

nmap, web crawling, DNS brute-force, certificate transparency, Shodan, Active Directory enumeration, cloud across AWS, GCP, and Azure, containers and Kubernetes through the Docker socket and K8s API and RBAC, IoT and SCADA, Git secrets, and email OSINT. It also fingerprints defences, firewall, IDS/IPS, and WAF detection, with an adaptive evasion re-probe when it meets one.

Vulnerability detection

Active Directory and identity

Authenticated LDAP assessment covering Kerberoasting and AS-REP roasting (via LDAP and credential-free Kerberos pre-auth), DCSync rights, unconstrained and constrained delegation, and SMB-signing, SMBv1, and NTLMv1 posture. It checks AD CS certificate-template abuse from ESC1 to ESC4 and ESC8 web-enrolment relay, and maps the NTLM coercion surface, PetitPotam, PrinterBug, DFSCoerce, bind-only, so it never actually triggers coercion. It also does BloodHound-style path analysis and SSO testing across OAuth 2.0 and SAML.

Verified exploitation and post-exploitation

Active proof rather than guesses: sqlmap SQLi dumps, RCE canary drop-and-read, and the in-house OAST collaborator for out-of-band SSRF and XXE. Post-exploitation ships self-contained privilege-escalation engines for both Linux (GTFOBins-scored SUID, sudo, and capabilities, docker and lxd escape, writable /etc/passwd, cron and PATH hijack) and Windows (unquoted service paths, writable service binaries, SeImpersonate and SeBackup token privileges, AlwaysInstallElevated, autologon and registry credentials, UAC posture), with no linPEAS or WinPEAS download. A loot harvester collects SSH keys and cloud credentials with secrets redacted and plaintext never persisted, and a credential-reuse loop feeds SSH, SMB, and PsExec lateral movement with pass-the-hash, all tagged to an ATT&CK kill chain.

Cloud, API, and offline analysis

Credential-free public storage-bucket exposure across S3, GCS, and Azure Blob, proven from the provider's own response rather than guessed, plus a cloud-metadata SSRF catalogue that turns an SSRF into confirmed credential theft, and an authenticated account audit including a read-only IAM privilege review. API security follows the OWASP API Top 10, BOLA and IDOR, broken auth, mass assignment, and excessive data exposure across REST and GraphQL. Offline, HEAVEN analyses binaries, firmware, pcaps, documents, and media, and scores Android APK and iOS IPA files against the OWASP Mobile Top 10.

CVE intelligence

A curated offline CVE database of ~150+ version-matched entries, with a live fallback: any product or version not held locally is looked up in real time against NVD and CIRCL, merged and de-duplicated, version-confirmed, then flagged for KEV membership, scored with EPSS, and correlated to an Exploit-DB PoC, and disk-cached with a seven-day TTL. It degrades gracefully offline.

Risk Scoring That Earns Trust

The score a client sees is computed exactly from each finding's CVSS vector using the reference formula. HEAVEN reports CVSS v4.0, the current standard, and shows the calibrated CVSS v3.1 score alongside. Published CVE scores come straight from NVD and OSV. The machine-learning model only steps in for a finding that has no published score, and even then it is a secondary ranking signal pinned to the authoritative severity, never the badge itself.

A hybrid predictor, measured honestly

The predictor has two paths. When a finding carries CVSS metrics, a 13-feature ExtraTreesRegressor predicts the base score at 5-fold cross-validated R²=0.91. When it does not, a TF-IDF text model (a TfidfVectorizer with Ridge regression) reads the finding's own vulnerability type and description; it is trained on the real-finding population of the dataset, 315,648+ CVEs carrying a non-zero CVSS score.

For the text path the honest figure is not a single R-squared but a set of numbers measured on the population the model actually scores. It reaches an exact-score R-squared of 0.64, a Spearman rank correlation of 0.80 against the true score, and it lands the correct CVSS severity band 99 percent of the time within one level. The band-accuracy and the rank correlation are the figures that matter, because the model is used to order unscored findings, not to stamp a precise number on them. Pushing the exact-score R-squared higher would mean leaking the CVSS formula's own sub-scores into the features, which would be measuring memorisation rather than prediction, so the tool reports what the model is honestly for: getting the ranking and the band right.

HEAVEN CVSS predictor, figures as published in the project. The exact-formula path is used whenever a finding carries a CVSS vector; the model only ranks the unscored remainder, and provenance is documented in the model card.

Ordering on top of the score

Once a base score exists, HEAVEN layers on EPSS exploit-probability and CISA KEV membership, an asset-criticality multiplier so a crown-jewel host outranks a peripheral one, and empirical Bayesian priors learned from an operator's own past engagements. The result is an ordering that reflects real-world exploitation likelihood, not just theoretical severity. The models are fetched once with SHA-256 verification rather than bundled, and without them CVSS falls back gracefully to each finding's own base score.

Autonomous Reasoning, Grounded

HEAVEN's AI layer is more than a planner. It runs the observe→plan→act loop, a recon agent, an attack-chain planner, an LLM false-positive review, an AI remediation generator, and a cross-engagement knowledge graph that lets it reason about multi-hop attack paths rather than one step at a time. Every one of these has a deterministic fallback, so the loop runs with no key at all.

Grounded, not free-floating

The knowledge graph is what turns a foothold into a plan. When HEAVEN compromises a host, it can query the graph for what that host can reach next and which weaknesses those neighbours carry, which is how it reasons two or three hops ahead. This is the multi-hop planning the research literature describes, running in a shipped tool. The false-positive review is a second stage on top of the verify loop: a two-stage confirmation pass discards anything below 0.40 confidence outright, with an optional LLM second opinion on what remains.

Built for the Pipeline

Traditional pen testing is a once-a-year event; a modern pipeline ships several times a day, which leaves long stretches where nobody knows the security state. HEAVEN is built to close that gap. It runs scheduled re-scans with differential alerts through its watch command, so only what changed since the last baseline is surfaced. It carries Semgrep SAST, a dependency audit against OSV.dev, and CycloneDX SBOM generation, and it forwards to Jira and Linear for ticketing and to Splunk and Elastic for SIEM.

Because it exports SARIF, HEAVEN drops straight into GitHub code scanning, and its eight report formats mean the same engagement can produce a client PDF, a compliance-mapped HTML, and a machine-readable JSON or Burp XML from one run. The shift-left payoff is the usual one: catch an issue in development where a fix costs little, not in production where it costs a great deal.

The Controls That Make Autonomy Defensible

Letting a machine act on a live network is only reasonable if it cannot exceed its authority, cannot leak what it sees, and cannot quietly rewrite the record. HEAVEN's security layer is designed around those three needs.

Measured, Reproducible, Across Five Tiers

The evaluation is deliberately not a single demo. HEAVEN is scored across five tiers, the web application, the API surface, the network and service layer, static source analysis, and full autonomous authenticated coverage, and every tier runs through the same precision, recall, and F1 metrics layer against a labelled ground truth. Four of the five run with no Docker and no network egress, so anyone can reproduce them, and they are enforced as a floor in continuous integration so a regression cannot slip through. The point of running the same scoring across such different surfaces is that a number means the same thing everywhere: a reported finding is counted true only when it maps to a labelled entry, so precision is measured honestly rather than by quietly ignoring whatever else the scan reported.

The headline result, across all five

Every figure is measured and reproducible; the recall fractions are detections against the labelled ground truth on each target. The perfect scores are on controlled, labelled surfaces and are described honestly as such below, they are not a claim of parity with commercial suites across every vulnerability class.

Web tier: detection and attribution

The native web benchmark drives the real crawler, injection scanner, misconfiguration scanner, and out-of-band prober against a DVWA-faithful surface that reproduces MySQL comment semantics, and adds a misconfiguration and out-of-band surface for SSRF, XXE, CORS, open redirect, weak JWT, and insecure cookies. It covers 12 vulnerability categories and reports 17 findings, all real, with zero false positives on reflective or escaped endpoints, and it attributes each finding to the correct parameter rather than, say, the submit button. Crucially, SSRF and XXE are proven out-of-band: the target calls back to HEAVEN's own OAST collaborator, so they are confirmed interactions, not heuristics. The whole run finishes in about 13 seconds with no Docker and no network.

Network tier: real CVEs on a real VM

Against a running Metasploitable-2 host, HEAVEN surfaced all 12 must-find criticals with 50 of 50 reported findings mapping to a labelled entry and zero false positives. The twelve are the ones a competent network assessment has to find: the vsftpd 2.3.4 backdoor (CVE-2011-2523), the Samba usermap RCE (CVE-2007-2447), distccd (CVE-2004-2687), the UnrealIRCd backdoor (CVE-2010-2075), the ingreslock root bind shell, dRuby and Java RMI exposures, a world-readable NFS export, and default credentials on Tomcat manager, PostgreSQL, VNC, and SSH. Version-unconfirmed service CVEs are folded into one honest low-confidence roll-up rather than asserted as confirmed, which is what lets precision be measured against everything the scan legitimately reports.

SAST tier: a genuinely hard corpus

The static engine was scored against the industry-standard OWASP Benchmark v1.2, 2,740 Java cases where roughly half are real vulnerabilities and half are safe lookalikes built specifically to trip a scanner. HEAVEN scored a Youden index of 1.000, detecting all 1,415 real vulnerabilities with zero false positives across the full corpus, and every one of the eleven categories scored a perfect 1.000. This is not benchmark tuning: nothing in the detection path is benchmark-aware. The benchmark's safe lookalikes differ from the real bugs only by facts a sound dataflow analysis can decide, a tainted value assigned in a dead branch, read back from a collection under a different key, or discarded before the sink, and a weak algorithm named in configuration rather than in code, and HEAVEN performs that analysis generically over the real Java syntax tree, so the same logic holds on arbitrary code. For contrast, a purely pattern-based engine such as FindSecBugs scores about 0.42 Youden on the same corpus, because it cannot fold the dead code or resolve the configuration.

The engineering the numbers forced

The perfect scores are the end of a debugging story, not the start of one, and the benchmark is what surfaced the bugs that separate a demo from a usable tool. The first end-to-end run reported one injectable parameter 188 times, once per payload, which a fix to strip the query string from a finding's identity collapsed from 1,653 findings to 35 on a two-URL scan. Auth cookies were not being sent, so scanners hit protected pages unauthenticated until the cookie jar was corrected, taking endpoint discovery behind the login from 0 to 17. Two subtler fixes are the ones worth naming, because they are exactly the false-positive controls the literature review argues for: a reproduce-before-report rule that requires a concurrent divergence to recur on a confirmation burst before it is emitted, which took DVWA precision from 93 to 98 percent with recall unchanged, and a truth-value confirmation that requires a blind-SQLi oracle to survive a literal-swapped variant, so that a real oracle depending on the condition's truth value is kept and a transient coincidence is dropped, taking precision from 97.6 to 100 percent with recall still at 100.

Honest caveats

The evaluation states its own limits plainly. The DVWA runs used security level low behind an authenticated session, the canonical functional benchmark for a scanner, not a hardened production app. The benchmark target ran under CPU emulation, so wall-clock times are slower than on native hardware, though the findings are unaffected. The coverage spans the classes scored above and is not a claim of parity with commercial suites across every vulnerability class; the repository ships an honest head-to-head template with adapters for Burp, ZAP, and sqlmap rather than inventing competitor numbers. And the out-of-band SSRF and XXE proof requires the target to reach HEAVEN's collaborator, which holds for lab targets by default and needs a routable bind address for a remote one. Naming these is the same discipline as the rest of the tool.

HEAVEN in the Field: an MSc Dissertation

HEAVEN was the technical instrument for the author's MSc dissertation, a Cyber Essentials readiness study of a UK social-housing provider holding a large amount of tenant data. This is the part that separates the framework from a benchmark exercise. It was run under written authority, following the NIST SP 800-115 testing lifecycle and the BCS Code of Conduct, against a live estate spanning a public website, a head-office internal network, and cloud servers. The academic supervisor asked that HEAVEN serve as the instrument precisely because it produces a replicable evidence trail a conventional scanner cannot.

The framework ordered findings using its machine-learning severity estimate together with CVSS, EPSS, and the KEV catalogue, and recorded the request and response behind every finding so each one could be reproduced. The same three guardrails described above governed the work: the authorisation gate refused anything outside scope, credentials were redacted before anything touched disk, and the keyed-hash append-only log kept the record tamper-evident. A point worth stating plainly is that the model never acted on its own; it suggested which checks to run next, and the framework only performed actions that were already specified and authorised.

The shape of the result did real diagnostic work. The most common single class was sensitive file and path exposure, with 33 findings, followed by missing security headers and cleartext services, which told a small team that a handful of root causes sat behind the 167 findings rather than 167 unrelated ones, so the remediation plan could be short. A retest of eighteen findings from the provider's earlier November 2025 penetration test, chosen to span the severity range and every asset class, found nine truly closed, three partially closed, four that could not be judged without intrusive testing, and two re-opened on items previously marked done, which is exactly why the study argues that verifying a fix matters more than recording one.

Every Finding, Placed in Context

A finding is more useful when it is tied to a framework a client already reports against. HEAVEN maps every finding to MITRE ATT&CK techniques and to Lockheed Cyber Kill Chain phases, and it can pull threat intelligence over a TAXII feed. Its methodology documentation carries mappings against the pen-testing standards, OWASP Testing Guide v4.2, PTES, and NIST SP 800-115, and against the compliance frameworks a publication-grade tool is expected to address: Cyber Essentials and Cyber Essentials Plus v3.3, ISO/IEC 27001:2022 with Amendment 1:2024, PCI DSS v4.0.1, CIS Critical Security Controls v8.1, NIST Cybersecurity Framework 2.0, and SOC 2.

What makes the mapping honest rather than decorative is that it is live and never fabricated. A control lights up as exercised only when the specific HEAVEN detector it names actually produced a finding in the current engagement, and the summary counts are computed from the rows so they cannot drift from the detail. Where a control cannot be evidenced from a network or credentialed scan, the row says so explicitly, marked manual for host and endpoint tests, organizational for governance and policy, or physical for physical controls, so an auditor sees coverage and its honest limits at a glance and an operator knows exactly what still has to be done by hand.

The combined-risk view is where this pays off in an engagement. HEAVEN correlates findings that are individually modest into a materially worse combined issue, then chains them into an end-to-end attack path where each step yields a capability the next one uses, and it ranks the single break-the-chain fixes that collapse the most paths. That narrative appears in the web console, the CLI, and the PDF and HTML reports, which is what turns a flat list of issues into a short, ordered remediation plan.

How HEAVEN Answers the Field's Open Problems

The literature on autonomous penetration testing keeps returning to the same handful of unsolved problems. HEAVEN does not claim to close them, but each of its design choices maps onto one, and it is worth being precise about where it helps and where it does not.

Where HEAVEN does not solve the problem

Two of the field's hardest problems remain open for HEAVEN too, and it would be dishonest to claim otherwise. The sim-to-real gap is sidestepped rather than solved: HEAVEN is validated on real targets instead of a simulator, which is the right call for a tool but does not advance the science of transfer learning. And calibrated confidence, making a stated confidence genuinely mean a matching probability of being real, is still an open research question; HEAVEN's per-finding confidence scoring is a step toward it, not a full answer. Naming these honestly is part of what a publishable account of the tool has to do.

Honest Limits and Responsible Use

Limitations

• The framework is validated on a single live estate and a controlled benchmark, not across many organisations, so its generalisation is demonstrated but not yet broad.
• The ML predictor is a ranking aid for unscored findings, not a replacement for the CVSS vector, and its honest R² on routed findings is 0.63, which is why band-accuracy is the figure reported.
• Autonomy is bounded by design: interpretation stays with the operator, which is a deliberate limit on how far the tool goes on its own.

Dual use and oversight

A tool that can walk an estate at machine speed is a defensive asset in one set of hands and a weapon in another. HEAVEN is intended for authorised testing and education only, and running it against systems without explicit written permission is illegal in most jurisdictions. The design reflects that: the authorisation gate is not a convenience, it is the mechanism that keeps the tool inside the law and the engagement. The direction of regulation, across the EU AI Act, the US National Cybersecurity Strategy, and UK guidance, is toward mandatory human oversight for high-consequence automation, and HEAVEN's operator-in-the-loop model is built to meet that rather than resist it.

Where HEAVEN Goes Next

Near term

• Broaden validation across more organisations and estate types to turn a demonstrated result into a general one.
• Publish the DVWA benchmark methodology in full, test split, sample size, and band definition, so the numbers are independently reproducible.
• Extend the combined-risk chaining with the knowledge graph so more of the path-collapse analysis is automated.

Medium term

• Work toward calibrated confidence, so a reported number genuinely reflects the probability a finding is real.
• Grow the offline CVE database and the training population as the threat landscape moves, keeping the intelligence current by design.
• Deepen the local-model path so the full AI layer runs on-host at production quality, which matters most under strict NDAs.

Longer term

• Move toward a formally verified safety layer, a guard that provably cannot be overridden by the generative parts of the system.
• Explore certification for use in regulated environments, the way safety-critical software is certified in other fields.
• Keep the human-and-machine split at the centre: the tool carries scale and breadth, the operator keeps judgement, novel technique, and accountability.

Conclusion

HEAVEN is an argument, made in software, that autonomous penetration testing becomes useful only when it is careful. It automates the repeatable work of an engagement at real scale, 2877+ tests, 219+ modules, 62+ commands, 99+ routes, and a risk model trained on 315,648+ real CVEs, but it earns trust by refusing to report a finding a detector has not confirmed, refusing to act outside the agreed scope, and keeping a record a client can rely on.

The evaluation backs the design. HEAVEN was proven on a live target and then used, under written authority, as the technical instrument in a real assessment of a social-housing provider, where it found 167 issues across three scopes, ordered them into a short remediation plan, and held back its own uncertain results rather than inflating the count. That is the difference between a system that is clever in the abstract and one that is safe enough to run on an estate holding other people's data.

The open problems in the field, sim-to-real transfer and calibrated confidence chief among them, remain open for HEAVEN too, and this account names them plainly rather than papering over them. But the direction is clear. The credible future of this work is augmentation: a machine carrying scale, breadth, and repetition, a person holding judgement and responsibility. HEAVEN is a working instance of that split, and its field results are the evidence that the split works.

References

1. Bakker, I. and Hastings, J. Autonomous Penetration Testing: Solving Capture-the-Flag Challenges with LLMs. arXiv:2508.01054.
2. BDO. (2024) The EU AI Act: Key Takeaways. BDO Insights.
3. Clifford Chance. (2025) Who is Responsible for Agentic AI? Clifford Chance Thought Leadership.
4. Deng, G. et al. PentestGPT: A Large Language Model-Based Automatic Penetration Testing Tool. arXiv:2308.06782.
5. Fang, R., Bindu, R., Gupta, A. and Kang, D. (2024) LLM Agents can Autonomously Exploit One-Day Vulnerabilities. arXiv:2404.08144.
6. Ghanem, M.C. and Chen, T.M. (2020) Reinforcement Learning for Efficient Network Penetration Testing. Information, 11(1), pp. 1–23.
7. Ginige, T. et al. (2025) AutoPentester: An LLM-Based Autonomous Penetration Testing Framework. Proceedings of IEEE TrustCom 2025. Piscataway, NJ: IEEE.
8. Gioacchini, L. et al. AutoPenBench: An Open Benchmark for Automated Penetration Testing. arXiv:2410.03225.
9. Hu, Z., Beuran, R. and Tan, Y. (2020) Automated Penetration Testing Using Deep Reinforcement Learning. IEEE European Symposium on Security and Privacy Workshops, pp. 2–10.
10. Kong, H. et al. VulnBot: Autonomous Penetration Testing for a Multi-Agent Collaborative Framework. arXiv:2501.13411.
11. Miao, Y. et al. (2023) GAN-Based Autonomous Penetration Testing for Web Applications. Sensors, 23(18), p. 8014.
12. Nguyen, T. et al. Reinforcement Learning for Automated Penetration Testing: A Systematic Review. arXiv:2507.02969.
13. NIST. (2008) SP 800-115: Technical Guide to Information Security Testing and Assessment. National Institute of Standards and Technology.
14. Schwartz, J. and Kurniawati, H. (2019) Autonomous Penetration Testing Using Reinforcement Learning. arXiv:1905.05965.
15. Shao, Y. et al. An Empirical Evaluation of LLMs for Solving Offensive Security Challenges. arXiv:2507.00829.
16. Tran, N. et al. D-CYPHER: Dynamic Collaborative Intelligent Agents for Enhanced Reasoning in Offensive Security. arXiv:2502.10931.
17. Zhang, A.K. et al. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. arXiv:2408.08926.