01
Hold point
hold point, n. A step in an inspection and test plan that work may not pass without a signature.
A bug is not a bug until it fails a test.
ShipReview audits a whole Next.js app built with coding agents, before your agency hands it to the client. Three models from three vendors read the code separately, and we run every claim they make as a test in a sandbox. You get back only what failed for real, each with its failing test and a fix that passes it, a plain list of what we did not check, and a verdict: ship or hold.
Read the sample report in preparation
- Next.js and TypeScript
- One Postgres database
- Up to 200,000 source tokens
- Price fixed before work starts
Security / tenant isolation
A member of one organisation can export another organisation’s invoices.
- Label
- Reproduced
- Location
- app/api/invoices/export/route.ts:18-41
- Frozen test, claim line
- claim(res.status === 403, 'CLAIM: org A cannot export org B invoices')
- Result
- original 200 with 14 rows of another tenant patched 403 pass
- Regression
- 9 existing tests still pass
- Verdict
- Hold until the diff is applied and rechecked
02
What comes back
Every line in the report points to a record.
The report is written for two readers. Page one is for the person who signs the handover and may be forwarded to the client. The rest is for the developer who fixes the code. Below is a shortened copy with our notes beside it.
-
A · Page one
HoldOne reproduced high-severity finding is open.
AVerdict. HOLD while any reproduced finding of high or critical severity is open. Otherwise SHIP, which means exactly this: no reproduced high-severity defect is open within the checked scope.
-
B · Findings by label
Findings by label Id Severity Label Finding F-001 high reproduced Invoice export ignores the caller’s organisation F-002 medium code_supported Session cookie set without Secure in production F-003 medium unverified Two invoices can receive the same number under concurrent writes F-004 low improvement_proposal Invoice list runs one query per customer F-005 high rejected SQL injection in search. The query is parameterised. BFindings by label. Sorted by label, then severity. Each finding lists which models raised it and any model that disagreed. Rejected claims stay in the list so you can see what was ruled out.
-
C · Frozen test, F-001
// __shipreview__/repro/F-001.test.ts sha256:9c4e…1a7f frozen const { orgA, orgB } = await seedTwoTenants() precondition(orgB.invoices.length === 14, 'PRECONDITION: org B has 14 invoices') const res = await exportInvoices({ as: orgA.member, org: orgB.id }) claim(res.status === 403, 'CLAIM: org A cannot export org B invoices')CThe frozen test. Written in your app’s own test style. Setup is marked PRECONDITION; the one assertion that states the defect is marked CLAIM. Once it fails on your code for that reason, we record its hash and it can no longer change.
-
D · Patch, F-001
--- a/app/api/invoices/export/route.ts +++ b/app/api/invoices/export/route.ts @@ -24,1 +24,2 @@ export async function GET(req, { params }) - const rows = await listInvoices(params.orgId) + if (params.orgId !== session.orgId) return forbidden() + const rows = await listInvoices(session.orgId)Patch checks Frozen test, patched copy pass · 403 Existing tests pass · 9 of 9, no new failure Type check pass Passed specified checks
DThe patch and its checks. A diff for you to apply. The repair step may not touch the frozen test, your tests, test config, package scripts or the lockfile. “Passed specified checks” means these three lines. It does not mean the fix is the best one.
-
E · Coverage
Coverage by model vendor Vendor Files sent Status Anthropic 64 of 64 completed OpenAI 64 of 64 completed Google 64 of 64 completed - blockedlint: the app has no lint script
- not checkedpayment webhook: it calls an outside service and the sandbox has no network
- excluded2 secret files dropped before upload
ECoverage. Which files each model was actually sent, which checks ran, which were blocked and why. A file we listed but did not send to a model is shown as not reviewed.
-
F · Cost
Cost by stage Stage Models Sandbox Deterministic checks $0.00 $0.31 Independent reviews $11.40 $0.00 Reproduce and repair $4.85 $0.62 FCost. What the audit spent on models and sandbox time, per stage, from our ledger. The price you pay was fixed by the quote and does not move with it.
Shortened. Field names are the report’s own; F-001 matches the record above, the other values are illustrative.
03
Evidence labels
The label is the state field of each finding in findings.json.
Five labels. A test earns the first one.
Agreement between models never upgrades a finding. Only a failing test does.
| Label | What it takes to earn it | What you should do with it |
|---|---|---|
| Reproducedreproduced | A frozen test failed on your code for the reason claimed, and a model from another vendor approved what the test expects. | Fix it before handover. If a patch passed its checks, start there. |
| Code supportedcode_supported | The cited code was read and is consistent with the claim. It was not run. | Read it and decide whether it matters for this client. If it does, ask for a reproduction or write the test. |
| Unverifiedunverified | Plausible, but context was missing or the check did not finish. | Treat it as a question for the developer, not as a defect. |
| Improvement proposalimprovement_proposal | No defect is claimed, or a performance gain has not been measured. | Optional. Schedule it or leave it. |
| Rejectedrejected | We tested or inspected the claim and it did not hold. | Nothing. It is listed so you can see what was ruled out. |
04
How an audit runs
Stages 2 and 3 run at the same time. Neither sees the other’s output.
The audit follows an inspection and test plan.
Six stages, in order. Reproduction and the patch check are hold points: the audit does not move past them until the record in the fourth column exists.
| No. | Activity | Acceptance criterion | Record produced | H / W / R |
|---|---|---|---|---|
| 1 | Snapshot and inventory Unpack the upload, drop secret files, list routes and the files that touch the database or authentication. | Next.js and TypeScript, one package.json, a lockfile, Postgres as the only service, 200,000 source tokens or fewer. Otherwise no quote. | Snapshot id, file manifest, quote | R |
| 2 | Deterministic checks Install, prepare the database, type check, run your tests, lint if there is a lint script, scan for secrets and known vulnerable dependencies. | Each check ends passed, failed, blocked or skipped, with a reason. | Check results; your test results become the regression baseline | W |
| 3 | Independent reviews Models from Anthropic, OpenAI and Google each read the same snapshot. None sees the others’ work or the check results. | Every observation cites a file and lines that exist in the snapshot. The rest are rejected and counted. | Observations; per model, the files it was sent | W |
| 4 | Consolidation Group observations by root cause. Nothing is dropped by vote. Up to five high-severity findings raised by one model go to the others for a second reading. | Every observation ends up in exactly one finding. No label above code supported. | Findings with their observations, support and dissent | R |
| 5a | Reproduce A model writes one focused test per finding. A model from another vendor checks what the test expects. The test runs on your original code. | Only the CLAIM assertion fails, for the reason claimed. Then the test is frozen by its hash. | Frozen test and sha256, original result, environment digest, sanitized log | H |
| 5b | Patch check A model proposes a diff. A diff that touches the frozen test, your tests, test config, package scripts or the lockfile is refused. | Frozen test passes on a patched copy; your tests and type check show no new failure. | Diff and digest, check results, patch state | H |
| 6 | Report Verdict, findings, coverage and cost, from the records above. | HOLD while a reproduced high or critical finding is open. | report.html, report.md, findings.json, developer bundle | R |
- H
- Hold point. The audit does not pass this step without the record.
- W
- Witness point. Logged as it happens; you can follow it in the event log.
- R
- Review point. The record is checked and goes into the report.
Limits of the standard audit
- Source
- up to 200,000 tokens
- Sandbox time
- 30 minutes
- Reproductions
- 5 findings
- Patch
- 1, at most 2 iterations
- Recheck
- 1, within 14 days
The token count is estimated from file sizes and checked before you pay. When an app needs more than this, the quote says so first.
05
Two doors
Exit codes: 0 when no reproduced high-severity finding is open, 1 when one is, 2 for a partial or failed audit, 3 for a usage or connection error.
Two doors, one job.
For the person who signs the handover
Upload a zip or choose a repository, tell us in a few lines what the app must never do, and confirm the quote. The report opens with the verdict and what was not checked, in words a client can follow. The developer bundle carries the frozen tests and the diffs for whoever fixes the code.
Both doors start the same job, with the same limits and the same price.
| report.html | One self-contained file you can forward to the client |
|---|---|
| report.md | The same report as text |
| findings.json | Every finding as structured data |
| bundle/ | Frozen tests, diffs, sanitized logs and a brief for a coding agent |
For the developer and the coding agent
$ shipreview quote ./app
packed 212 files, dropped .env and .env.local
eligible Next.js 15, TypeScript, Postgres
price $99 pilot quote q_7f2c91, valid 30 min
$ shipreview start --quote q_7f2c91
audit a_31d0e4 queued
$ shipreview status a_31d0e4 --watch
1 snapshot and inventory done
2 deterministic checks done
3 independent reviews done, 3 of 3 models
4 consolidation done
5 reproduce and repair done, 1 reproduced
6 report done
$ shipreview findings a_31d0e4
F-001 high reproduced invoice export
F-002 medium code_supported session cookie
…
$ echo $?
1
$ shipreview patch a_31d0e4:F-001 > F-001.diff
patch passed_specified_checks, not applied
$ git apply F-001.diff
$ shipreview recheck a_31d0e4 ./app
F-001 fixed_verified
verdict ship: no reproduced high-severity defect
open within the checked scope
$ echo $?
0
| Tool | Returns | Spends credit |
|---|---|---|
| audit_quote | Eligibility, scope, price, quote id | No |
| audit_start | Job id, at once | Yes |
| audit_status | Stage and coverage progress | No |
| audit_findings | Findings with labels | No |
| finding_patch | Diff and check results | Within the allowance |
| audit_recheck | Updated finding states | First one included |
No start without a quote. An agent cannot start a job until a quote with the price is on record and unexpired.
Keys have spend caps. Read and spend are separate scopes. A person sets each key’s daily and monthly cap.
Results are data. Anything that comes from your repository arrives in a field named untrusted, so a comment in the audited code cannot steer the agent that reads it.
We return diffs. We never apply them. The developer or the agent does, then asks for a recheck.
06
Exclusions
What it will not tell you.
- 6.1
Stack. The first version covers Next.js with TypeScript, npm, one service and one disposable Postgres database. For anything else the quote says no, and says why, before you pay.
- 6.2
Certification. An audit buys a stated amount of checking. It does not certify the app. A SHIP verdict is not a statement that the app has no bugs.
- 6.3
Tested conditions. A passing test shows behaviour under the conditions of that test: synthetic data, a fresh database, no network, the versions in your lockfile. Production can differ.
- 6.4
Defects nobody raised. We test what a model or a check raised. A defect that none of them noticed is not in the report and is not covered by the verdict.
- 6.5
Performance. We do not load-test. A performance remark without a measurement comes back as an improvement proposal.
- 6.6
The pilot. During the pilot one of us helps where the automation is not ready yet: setting up apps that do not start on their own, and reading each report before it goes out. The report names every step where a person helped.
Ship
No reproduced high-severity defect open within the checked scope.
07
Rate card
Prices, quoted before any work starts.
| Offer | Price | Includes | Status |
|---|---|---|---|
| Pilot audit | $99 | One audit of one app, with a person assisting where the report says so. Five places. | Available now |
| Standard audit | $129 | The limits above, the report and the developer bundle, one recheck within 14 days. | Planned |
| Changed-files recheck | $19or five for $95 | Frozen tests and your tests rerun on a new snapshot, when no more than 10% of the source changed. | Planned |
| Agency plan | $449a month | Four standard audits a month. | Planned |
USD, before tax. Planned prices may change after the pilot. If we cannot run an app we quoted as eligible, we refund or credit the audit. A clean result is not a refund case: you pay for the checking, not for findings.
08
Your code
Retention periods count from the day the report is delivered.
Who reads your code, where it runs, when it is deleted.
- Who reads it
-
Three model vendors receive the files that the coverage section of your report lists for each of them. Each keeps API data under its own terms. We link those terms rather than summarise them, because they change.
Model vendors and their terms Anthropic How long do you store my organization’s data? OpenAI Data controls in the OpenAI platform Google Gemini API Additional Terms of Service Before upload, the command-line packer drops .env and key files, honours .gitignore and shows you the file list.
- Where it runs
In a fresh isolated sandbox created for your audit and destroyed after it. The sandbox has network access while dependencies install and none after that. Model keys never enter it; every model call goes through our gateway outside the sandbox.
- When it is deleted
-
Deletion schedule Record Deleted Source snapshot 14 days after delivery Sanitized reports and evidence 30 days after delivery Billing records kept separately Fourteen days is the recheck window. A recheck after that needs a new upload.
- What we do not claim
We do not claim zero retention. The vendor terms above govern what each vendor receives, and we cannot shorten them. We do not publish your code or use it for training.
09
Questions
Questions, answered.
- Is this three models voting?
- No. Each model reads the code alone and none sees what the others wrote. We group their observations by root cause and keep every one, including a finding only one model raised. Agreement does not raise a label. A finding becomes reproduced only when a test fails on your code.
- We already use CodeRabbit.
- Keep it. It reviews pull requests continuously, and it documents its own repository scans and sandbox checks. We have not benchmarked against it and do not claim to beat it. This is a different purchase: one fixed-price gate on a whole app before handover, with a report written for your client.
- What if you find nothing?
- Then the report says so and lists what was checked and what was not. The verdict is SHIP in its narrow sense: no reproduced high-severity defect open within the checked scope. The price is the same.
- Can my agent run this unattended?
- Yes, within the limits you set. Give it a key with the spend scope and a cap. It asks for a quote, starts the job, polls the status, applies the diffs itself and asks for a recheck. The command exits 1 while a reproduced high-severity finding is open, so a CI job can stop on it.
- What stacks?
- Next.js with TypeScript, npm with a package-lock.json, one package.json at the root, Postgres as the only service, and a script that prepares the database. The quote checks all of this before you pay, and an app that does not fit costs nothing.
- Is a person involved?
- During the pilot, yes, and the report says at which steps. The standard audit is built to run without one. When a person does help, for example to get an app to start, the report will say so in the same way.
- How long does an audit take?
- We do not quote a time yet. Sandbox work is capped at 30 minutes; model reviews and, during the pilot, the assisted steps add to that. We will publish times once we have measured them.
10
Pilot
Book a pilot audit.
Five places at $99 each, for agencies with a Next.js app near handover. Write to us and we reply with a quote and what we need from you.
Email pilot@shipreview.techPut in the email
- The Next.js version
- Roughly how many source files the app has
- The handover date
- One thing the app must never do, for example show one client’s data to another
ShipReview is a working name. The service is in a paid pilot with five places. There are no customer logos on this page because there are no customers yet.