Smart Contract Security Benchmarking: Why Honest Results Beat Perfect Dashboards

Smart contract security benchmarking is easy to make look impressive and hard to make genuinely useful. A recent founder post about the security product Velmère captures the distinction: the team stopped treating its steadily greener internal test dashboard as proof of product progress and started asking whether it was simply becoming more effective at passing tests it had designed itself.

That question matters far beyond Solidity analyzers. Every AI tool, developer platform, fraud system, content-moderation product, and security SaaS eventually faces the same temptation: optimize the measurement until the measurement looks great. The stronger move is to build a benchmark that can embarrass the product, expose uncertainty, and show customers what will happen when the tool encounters unfamiliar real-world work.

In the Velmère post on r/SaaS, the founder described moving evaluation to pinned public smart-contract datasets. In one initial SmartBugs checkpoint, 29 cases matched vulnerability families the current engine claimed to support. Of those, 22 could be run with the compiler and abstract-syntax-tree evidence required by the system, and all 22 were detected. The remaining seven were withheld rather than labelled safe. A separate run over 29 OpenZeppelin Contracts 5.0.2 control candidates produced two alerts, but the founder explicitly declined to call them false positives before independent review.

That restraint is the story. Not the 22/22 detection figure, but the refusal to turn a partial result into an accuracy claim.

The real problem with green internal dashboards

Internal regression suites are essential. They catch bugs introduced by new releases, preserve hard-won fixes, and make engineering work repeatable. But they are not automatically an independent measure of how a product performs outside the environment that created it.

The danger is feedback-loop evaluation. A team identifies a bug, creates a test, changes the engine until the test passes, and celebrates an improvement. That is a legitimate regression fix. Yet when thousands of such cases become the primary product scorecard, the suite can gradually reflect the team’s own assumptions, data shapes, labelling habits, and implementation choices more than the customer environment.

For a smart-contract analyzer, this effect can be especially strong. The system may become highly effective on familiar compiler versions, known code idioms, supported vulnerability taxonomies, and evidence patterns. It may still fail, abstain, time out, or generate noisy reports when facing contracts assembled from different libraries, old Solidity versions, unusual inheritance structures, proxy patterns, generated code, or project-specific business logic.

A dashboard that says "98% passing" therefore leaves several critical questions unanswered:

• Passing which tests, written by whom, and for what purpose?
• Are inputs independent of the product team’s development loop?
• Which vulnerability categories and compiler versions are represented?
• How many test cases could not be executed or analysed?
• What happens on safe controls rather than known-vulnerable samples?
• How much human reviewer time does each alert consume?

If those answers are vague, the dashboard may be useful internally while being a weak basis for external product claims.

Why smart contract security benchmarking needs separate scores

The Velmère example illustrates a key measurement mistake: collapsing detection, execution, and correctness into one headline number. Those are different properties and should be reported separately.

A tool that detects a known vulnerability only when compilation succeeds is not equivalent to a tool that detects it across all repository conditions. Similarly, a tool that flags suspicious patterns in a vulnerable corpus has not necessarily demonstrated that its alerts are precise enough for a security team to trust.

Detection is not end-to-end coverage

The 22 executable SmartBugs cases detected by the engine represent a useful finding. Within that specific, supported, executable slice, the system found the labelled issues. But the figure does not establish performance across all 29 selected cases, much less the entire SmartBugs corpus or the broader universe of deployed Solidity code.

The seven withheld cases are not failures in the same way as missed vulnerabilities. They may be cases the engine cannot compile, cannot map to its AST requirements, or cannot support under its current detection model. Still, those outcomes matter to a customer. A vulnerability scanner that cannot reach a conclusion is not providing a clean bill of health; it is signalling that the case needs a different workflow.

That is why an honest report should show at least three separate rates:

1. Execution or analysability rate: How often can the product actually process the requested code and produce the evidence needed for a decision?
2. Detection rate among adjudicated, supported cases: Of the vulnerable cases the product could legitimately analyse, how often did it flag the labelled weakness?
3. End-to-end benchmark coverage: Of the full declared benchmark, how many cases resulted in a correct detection, correct non-alert, abstention, execution failure, unsupported classification, or incorrect result?

A tool can have a strong conditional detection rate and weak overall coverage. Both facts can be true, and customers need both.

Abstention is a product behavior, not a reporting footnote

Security products are often rewarded for decisiveness. Yet a system that reports "safe" when it lacks enough evidence may create the most dangerous outcome: false assurance.

Withholding a result is therefore not necessarily a defect. It can be a safer operating mode, particularly where a tool depends on a specific compiler, source completeness, build settings, dependency graph, or AST structure. The product should distinguish between "no issue found in a completed analysis" and "analysis could not establish a result."

The important caveat is that abstention cannot become a hiding place. If a product withholds results often, it may be correctly cautious but operationally limited. That limitation needs a visible metric, a clear explanation, and a remediation path. Can the user supply a compiler version? Is a dependency missing? Is the category unsupported? Does the case need manual review? Good benchmark reporting turns each abstention into an actionable state rather than a silent exclusion.

What the SmartBugs checkpoint can and cannot prove

SmartBugs is a meaningful starting point because it was created specifically to evaluate automated smart-contract analysis tools. Its curated dataset contains Solidity contracts annotated with tagged vulnerabilities, while the broader SmartBugs framework exists to provide a more uniform way to run analysis tools and datasets. (github.com)

That makes a pinned SmartBugs checkpoint far more credible than an unpublished internal test set. It gives other researchers, customers, and competitors a defined corpus they can inspect and reproduce. Pinning is important: benchmark repositories evolve, dependencies change, and a score without a fixed commit, container image, tool version, and execution configuration can become impossible to recreate.

Still, a benchmark is evidence, not a universal truth machine.

The selected slice is narrower than the corpus

The founder’s reported checkpoint began with 29 relevant cases in vulnerability families the engine currently supports. That is a sensible engineering filter for diagnosing whether implemented detectors work. It is not a product-wide accuracy denominator.

The next planned step—running the full corpus and retaining unsupported categories in the denominator—is deliberately less flattering but much more useful. It exposes the gap between supported-family effectiveness and real benchmark applicability. A customer choosing a tool needs to know both: whether it is excellent at a given category and whether it covers the kinds of problems that matter in their codebase.

SmartBugs documentation identifies SB Curated as a dataset of vulnerable Solidity contracts, and current project documentation lists it as 143 contracts. That scale can support repeatable checks, but it also underscores why teams should avoid treating a single benchmark suite as the entirety of modern Solidity risk. (github.com)

Labels are useful, but context remains essential

A labelled vulnerability in a benchmark represents a known target. In production, however, exploitability can depend on configuration, initialization, privilege boundaries, deployed dependencies, transaction sequencing, upgrade paths, and economic assumptions. A pattern-matching alert can be technically plausible but not exploitable in context; a codebase can also contain an economic or authorization flaw that a classic taxonomy does not express well.

This is one reason security tools should publish the granularity of their claims. Does an alert mean a vulnerable line, a vulnerable function, a suspicious pattern, a reachable exploit path, or a policy violation? Does the tool detect root causes or symptoms? Does it use source code, bytecode, build artefacts, traces, or a mix? The answer changes how users should interpret both benchmark and production results.

The two OpenZeppelin alerts are the most important number

The community reaction focused on the right issue. Two alerts among 29 OpenZeppelin control candidates are not merely an awkward blemish next to a 22/22 detection result. They are the beginning of the precision question.

OpenZeppelin Contracts is widely used as a library of reusable Solidity components and describes its project as a secure, reliable foundation maintained through risk-management, review, and incident-response practices. It also uses semantic versioning and warns developers that upgradeable-contract storage layouts across major versions should be treated as incompatible. (github.com)

That does not make every OpenZeppelin contract an automatic ground-truth negative. A library can contain intentionally flexible primitives, examples, configuration-sensitive code, or patterns that are safe only under documented assumptions. An alert could identify a true issue, a questionable design choice, a detector limitation, or an actual false positive. Until someone reviews the alert and records a verdict, "two alerts" is the only defensible label.

Why alert adjudication is the bottleneck

A commenter asked who would read the two alerts and how long each review would take. The founder acknowledged that the checkpoint preserved aggregate counts but lacked enough per-alert adjudication data to classify the reports later or provide a defensible minutes-per-alert figure.

That exchange reveals a common operational truth: the bottleneck is often not finding issues. It is proving whether an issue is real, prioritizing it, and explaining it to an engineer who must decide what to do next.

A useful alert record should preserve:

• A stable alert ID and the exact scanner version.
• Repository, commit, file, function, line range, and relevant code excerpt.
• Detector rule, severity, confidence, and stated preconditions.
• Compiler version, build configuration, and dependency-resolution result.
• Evidence trail: call path, data-flow explanation, AST node, symbolic trace, or reproduction steps.
• Reviewer identity or role, verdict, reasoning, and review duration.
• Final status: true positive, false positive, duplicate, accepted risk, not reproducible, needs context, or unresolved.

Without this trail, a team can calculate alerts per contract but not precision, reviewer burden, disagreement rate, or time-to-triage. Worse, it cannot learn systematically from disagreements.

Precision is not a vanity metric

A high false-positive rate changes user behavior. Developers begin ignoring findings, security teams add manual gates, and integration into CI becomes less valuable because every pull request creates noise. A scanner that catches many known issues but requires extensive human filtering may still help an audit team, yet it is a different product from a trusted blocking control.

Recent empirical work reinforces the need for caution. One benchmark study of eight smart-contract static-analysis tools, using 788 files and 10,394 labelled vulnerabilities, reported that existing tools missed roughly half the vulnerabilities in its benchmark and that precision did not exceed 10%. Those results do not describe Velmère specifically, but they demonstrate why limited detection checks should never be converted into broad accuracy promises. (arxiv.org)

A better scorecard for automated security products

The strongest lesson from this case is not "use SmartBugs." It is "report the whole decision pipeline." Founders should create a benchmark card that makes it difficult to confuse an appealing subset score with real-world performance.

Here is a practical scorecard for a smart-contract security product.

1. Define the evaluation population

State the full corpus size, dataset commit or release, licensing status, date of the run, and inclusion rules. If only certain vulnerability families are supported, show the total corpus and the supported subset side by side.

For example: "143 SB Curated contracts total; 29 cases matched currently supported detector families; 22 completed analysis; seven abstained due to compilation or required-evidence limitations." This wording does not hide the limitation while still communicating what the tool accomplished.

2. Separate tool outcomes

Every test case should land in one mutually exclusive bucket:

This taxonomy prevents a team from quietly treating unresolved, unsupported, or failed cases as successes.

3. Publish conditional and end-to-end metrics

Conditional recall can be useful: among supported, executable, labelled cases, what proportion did the tool detect? But it must sit beside end-to-end recall or coverage across the declared test population.

For safe controls, precision requires adjudication. Until alerts are reviewed, publish alert rate and unresolved-alert count, not precision. For practical use, add median and 90th-percentile triage time, because an average can conceal a few deeply complex reports.

4. Record uncertainty and disagreement

Security review is not always binary. Two reviewers may disagree about reachability, severity, exploitability, or whether a finding reflects a product bug versus a deliberate trade-off. Keep disagreement visible.

A mature program can use two independent reviewers for a sample of findings, document a tie-break process, and report inter-reviewer agreement. This is more work, but it helps determine whether a supposedly objective detector is actually producing ambiguous reports that consume senior security attention.

5. Make runs reproducible

Use pinned data revisions, locked dependencies, container images, exact compiler versions, configuration files, execution logs, and scripts that reproduce the table. SmartBugs 2.0 itself was designed as a modular execution framework to simplify tool execution, reproducibility, and larger experiments, which makes the broader ecosystem’s emphasis on repeatable evaluation clear. (ieeexplore.ieee.org)

Reproducibility does not mean a rival can recreate every proprietary component. It means an independent party can understand what was run, against what inputs, under what conditions, and whether the reported totals follow from those conditions.

How to build an evaluation ladder instead of one benchmark

No single corpus can assess every failure mode. The better approach is an evaluation ladder: each rung answers a different question and makes different kinds of product weakness visible.

Layer one: internal regression tests

These should be fast, broad, and run on every relevant change. Their job is to stop known regressions. Include tiny minimized examples, previously fixed customer issues where sharing is permitted, parser edge cases, build failures, and rule-specific negative cases.

Do not use this layer alone for external marketing claims. It is optimized for rapid development feedback, not independence.

Layer two: pinned public benchmark datasets

Public corpora such as SmartBugs create a common reference point. Run complete datasets where possible, publish filtering logic where not, and retain unsupported and abstained outcomes in the results table.

This layer tests whether the product generalizes beyond its own tests, but it can still be vulnerable to overfitting if the benchmark becomes the main development target. Keep a documented split between development, validation, and holdout data where feasible.

Layer three: vetted controls and clean-code suites

A security tool cannot be assessed only on vulnerable examples. It needs deliberately selected controls: audited contracts, widely used libraries, internally reviewed projects, and generated variants with known safe properties.

The controls must be independently reviewed and version-pinned. The key word is independently. Calling a contract "clean" because it is popular or because the tool previously said so creates circular validation.

Layer four: historical, independently audited code

The Velmère founder’s proposed replay of historical code already subjected to independent audit is an especially valuable next step. It can test whether the product would have surfaced documented findings before they were known to the tool team.

To make this meaningful, preserve the pre-fix commit, audit report, scope statement, relevant compiler settings, and a mapping from audit finding to code location. Exclude findings that rely solely on off-chain business context unless the benchmark explicitly measures that category. Also distinguish issues the scanner could theoretically detect from ones that require protocol-level judgment.

Layer five: live customer calibration

Finally, benchmark production-like repositories under consent and strong confidentiality controls. Measure execution success, alert volume, severity distribution, customer-confirmed findings, dismissals, remediation time, and missed issues discovered by later audits or incidents.

This is where teams learn whether their lab metrics correspond to customer value. It is also where privacy, security, and data-retention practices must be robust enough to earn trust.

What founders should say—and avoid saying—in public

Technical buyers do not need a claim that a product is flawless. They need a claim with boundaries.

The founder’s refusal to market 22/22 as "100% accuracy" is a useful template. Accuracy is often the wrong umbrella metric anyway, especially when true negatives dominate, categories vary in difficulty, and not every case receives an adjudicated verdict.

Instead, a credible product update might say:

On a pinned checkpoint of 29 SmartBugs cases in vulnerability families currently supported by our engine, 22 were fully analysable and all 22 generated detections. Seven were abstentions due to incomplete execution or evidence requirements. In a 29-contract OpenZeppelin 5.0.2 control lane, two alerts remain under independent review; we are not reporting precision until they are adjudicated.

This is not weak marketing. It gives a technical buyer enough detail to ask intelligent follow-up questions and gives the team a baseline it can improve without moving the goalposts.

Avoid these common phrases unless the evidence is much stronger than a narrow checkpoint:

• "100% accurate"
• "No false positives"
• "Auditor replacement"
• "Finds every vulnerability"
• "Secures any Solidity contract"
• "AI-verified safe"

Each creates an expectation that a partial benchmark rarely supports. Security buyers remember overclaims long after they forget a launch post.

The broader AI evaluation lesson

Although this story involves smart-contract analysis, the same dynamics apply to AI products. A support agent can excel on internally authored test conversations but fail on messy customer histories. A coding agent can complete benchmark issues while struggling with a repository’s undocumented build system. A fraud model can show strong offline performance while generating a review queue too large for analysts to handle.

The pattern is consistent: the product metric must include the human and operational system around the model.

For AI security products, that means measuring not only detection quality but evidence quality, calibration, abstentions, escalation paths, and reviewer time. A system that says "I do not have enough evidence" at the right moments may be safer than one that always sounds confident. But it must make uncertainty legible and useful rather than merely refusing work.

This is also why an evaluation plan belongs in the product roadmap, not just in a research appendix. Better benchmark instrumentation changes engineering priorities. If teams see that most abstentions come from old compiler versions, they can invest in compiler support. If reviewers spend 45 minutes disproving one detector’s alerts, they can tighten preconditions or improve evidence. If certain vulnerability families are rarely detected, product positioning can become more accurate before a customer discovers the gap.

Practical next steps for a security SaaS team

A founder does not need a research lab to start. The following 30-day plan can turn a vague testing story into a defensible evaluation program.

1. Freeze one baseline. Pick a public dataset revision, pin the tool version and environment, and save raw outputs.
2. Create a complete outcome schema. Ensure every case is classified as detected, missed, abstained, unsupported, failed, clean, alerted, or unresolved.
3. Add stable alert IDs. Make every alert traceable to source, build configuration, detector, evidence, and final reviewer decision.
4. Build a reviewer queue. Assign alerts to qualified internal reviewers or independent auditors, define verdict labels, and capture elapsed review time.
5. Publish a scope statement. List supported Solidity versions, vulnerability families, required artefacts, known exclusions, and intended use in CI or audit workflows.
6. Run a controls lane. Use vetted controls rather than assuming well-known code is automatically safe.
7. Add historical audited projects. Replay pre-fix versions and map tool output to findings that were independently documented.
8. Report what got worse. If the full-corpus denominator reduces the headline number, publish it. That is often the number that improves product decisions.

The result will not be a perfect score. It will be something more durable: a measurement system that helps the team prioritize fixes and helps buyers understand risk.

Conclusion: the benchmark should be allowed to disappoint you

The most credible part of the Velmère benchmark update is that it treats missing information as missing information. Seven cases did not become silent passes. Two control alerts did not become confirmed false positives. A 22/22 result did not become a universal accuracy claim.

That discipline is rare because it makes product progress look slower in the short term. But it creates a better long-term advantage. Teams that track execution limits, abstentions, unresolved alerts, review time, false positives, and historical audit replay are building an evidence system—not just a launch metric.

For founders building automated security and AI tools, that is the standard worth adopting. Smart contract security benchmarking should make a product’s blind spots visible enough to improve, not invisible enough to sell.

FAQ

What is smart contract security benchmarking?

Smart contract security benchmarking is the structured evaluation of security tools against defined Solidity or EVM code corpora. Strong benchmarking measures more than vulnerability detections: it includes execution success, missed issues, abstentions, controls, false positives, evidence quality, and human review effort.

Is 22 out of 22 detected vulnerabilities the same as 100% accuracy?

No. It can describe a 100% detection rate within a narrow, executable, supported subset, but it does not measure unsupported cases, analysis failures, abstentions, false positives, or performance on unrelated code. It is a conditional result, not a universal accuracy claim.

Why are false positives so important in smart contract scanners?

False positives consume expert review time and can train developers to ignore alerts. A scanner’s practical value depends on whether findings are understandable, reproducible, and sufficiently precise for the team’s workflow—not only on how many vulnerable samples it flags.

Should security tools abstain when they lack enough evidence?

Often, yes. An explicit abstention is safer than incorrectly declaring code safe. But the tool should clearly explain why it abstained, quantify how often it happens, and give users a path to resolve the missing compiler, source, configuration, or evidence requirements.

Can public datasets replace independent smart contract audits?

No. Public datasets are valuable for repeatable tool evaluation, but audits also assess project-specific architecture, protocol economics, deployment configuration, privilege assumptions, and exploitability. Benchmark results should inform tool selection and workflow design, not replace independent security judgment.