External Developer Integration Shows How safelabs-eval Is Moving Beyond Its Own Repository

An External Integration Puts Open Source AI Security Tooling to the Test

Open source AI security projects often begin with self authored code and documentation. A more meaningful milestone comes when an unrelated developer studies the interfaces closely enough to build against them. That is what happened recently with safelabs-eval, an agentic AI red teaming framework created by Waqar Javed at Safe Labs AI.

An independent maintainer of EvalPort, an open interchange specification for portable LLM evaluation data, reviewed safelabs-eval’s public prompt schemas, runner, and scoring models and proposed a dedicated adapter between the two projects. The maintainer explicitly stated that the structures mapped unusually cleanly into EvalPort’s TestCase, Grader, and ResultSet model.

From GitHub Proposal to Merged Adapter

The discussion did not stop at a conceptual proposal. Javed reviewed the mapping decisions, including how safelabs-eval’s PASS, FAIL, VULNERABLE, and UNCERTAIN verdicts should be represented, where the adapter should live, and how raw response data should be preserved.

The EvalPort maintainer then implemented the adapter in the external EvalPort repository and merged it through a pull request. The final adapter converts safelabs-eval prompts into portable test cases, maps detectors into graders, preserves scoring metadata, and translates evaluation results into EvalPort compatible result sets. It also supports the reverse direction, allowing third party EvalPort suites to be brought into safelabs-eval workflows.

Testing Against the Real Package

According to the public integration record, the adapter completed 29 tests successfully, with zero failures or skips. Importantly, the maintainer reported that two of those were integration tests against the real installed safelabs-eval package rather than only mocked objects or fixtures.

The tests exercised real PromptEntry, EvalRecord, EvalResult, ScoringResult, and VerdictLevel structures and validated suites and result sets against EvalPort’s own validation functions. That kind of external interoperability work is valuable because it demonstrates that the framework’s public interfaces are understandable and usable outside the organization that created them.

Why Interoperability Matters for AI Security

Agentic AI security evaluation is still fragmented. Different teams use different orchestration frameworks, model providers, test suites, and result formats. Interoperability makes it easier to compare results, move test cases between systems, and avoid locking security evidence into one proprietary workflow.

For safelabs-eval, the EvalPort adapter is an early example of that principle. It does not prove widespread industry adoption, but it does establish a concrete external use case where another maintainer independently reviewed the code, built an adapter around the public interfaces, tested it against the actual package, and merged the integration into a separate project.

A Broader Research Program

The integration sits alongside Javed’s broader research on reproducible agentic AI security testing. His work has examined detector reliability, false positive calibration, cross model generalization, and cross framework portability. One controlled study ran 7,020 trials across six models, six execution conditions, and five attack families, finding that attack family and model choice explained substantially more outcome variance than framework choice in that experimental design.

The emerging theme is less about promoting one framework and more about making security evaluation inspectable, verifying what payload reached the model, preserving raw evidence, documenting uncertainty, and enabling results to move across tools.

Reference links:

1. safelabs-eval GitHub repository
2. EvalPort adapter discussion
3. EvalPort safelabs-eval adapter
4. Cross-Framework Portability research paper
5. Waqar Javed Profile