<!-- GENERATED by scripts/validate_corpus.py - do not edit by hand.
     Every number here is emitted by that script. Hand-transcription
     is what produced the errors that got the previous study
     withdrawn, so the report is generated and a test fails if this
     file does not match a fresh run. -->

# Corpus validation: does CrewScore coverage separate production prompts from general-purpose ones?

Validation ruleset `crewscore-hygiene@0.6.0` · package `0.6.3` · generated `2026-07-30`.
Reproducible command: `py scripts/validate_corpus.py`. This supersedes the withdrawn 1,368-prompt study.

## Corpora

Fetched at pinned commits, never vendored into this repository.

| Corpus | Files scored | Source | Pinned commit |
| --- | ---: | --- | --- |
| Production agent system prompts | 83 | [x1xhlol/system-prompts-and-models-of-ai-tools](https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools) | `2054f580b120` |
| General-purpose GPT-Store prompts | 273 | [linexjlin/GPTs](https://github.com/linexjlin/GPTs) | `3adfb7b38423` |

Licensing:

- **Production agent system prompts** — Collection is CC0 by its maintainer; the underlying vendor prompt text is not. Fetched at a pinned SHA, never redistributed here.
- **General-purpose GPT-Store prompts** — Collection is MIT by its maintainer; the underlying author text is not. Fetched at a pinned SHA, never redistributed here.

## Score distribution

| Corpus | n | Median | IQR | Mean | Scored 0 | Max |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Production agent system prompts | 83 | 10 | 3–15 | 9.46 | 20/83 (24.1%) | 34 |
| General-purpose GPT-Store prompts | 273 | 0 | 0–4 | 1.61 | 193/273 (70.7%) | 16 |

## Discrimination

- **Statistic:** Cliff's delta (Mann-Whitney U identity, midranks for ties)
- **Significance:** two-sided permutation test on Cliff's delta, 10000 relabelings, seed 20260729, add-one corrected
- **Interval:** percentile bootstrap on the same statistic, 10000 resamples within groups, seed 20260730

**Cliff's delta = 0.614**, 95% CI [0.491, 0.729], *p* = 0.0001.

At alpha = 0.05, coverage **separates** the two corpora.

### The zero-floor confound, addressed rather than asserted away

A mass of exact zeros creates ties, and ties **attenuate** a rank
statistic rather than leaving it untouched. The withdrawn study
claimed the opposite. Re-running on non-zero scores only:

- n = 63 production, 80 GPT-Store (scores above 0 only)
- Cliff's delta = 0.667, *p* = 0.0001

Report both. If they point the same way the zero mass is not
driving the result; if they diverge, that divergence is the
finding.

## Per-control coverage

Every rate carries its denominator. A percentage without one is how
"1 prompt in 283 (2.1%)" survived review.

| Control | Production | GPT-Store |
| --- | ---: | ---: |
| Treat instructions inside user content as data, not commands | 7/83 (8.4%) | 3/273 (1.1%) |
| Keep the system prompt confidential | 5/83 (6.0%) | 10/273 (3.7%) |
| Name prompt injection and state a defense | 7/83 (8.4%) | 17/273 (6.2%) |
| Do not fabricate, invent, or guess | 16/83 (19.3%) | 4/273 (1.5%) |
| Say so when you do not know | 6/83 (7.2%) | 2/273 (0.7%) |
| Ground answers in provided sources | 0/83 (0.0%) | 0/273 (0.0%) |
| Defer to a qualified professional | 0/83 (0.0%) | 2/273 (0.7%) |
| Require citations for claims | 27/83 (32.5%) | 19/273 (7.0%) |
| Link each claim to its source | 1/83 (1.2%) | 1/273 (0.4%) |
| Use an inline citation marker format | 7/83 (8.4%) | 4/273 (1.5%) |
| Cap spend, tokens, or rate | 5/83 (6.0%) | 0/273 (0.0%) |
| Bound output length | 4/83 (4.8%) | 0/273 (0.0%) |
| A human must approve | 18/83 (21.7%) | 0/273 (0.0%) |
| Do not act autonomously before approval | 17/83 (20.5%) | 2/273 (0.7%) |
| Stop or refuse rather than proceed | 35/83 (42.2%) | 45/273 (16.5%) |
| Name what triggers stopping | 21/83 (25.3%) | 0/273 (0.0%) |
| Escalate to a human | 0/83 (0.0%) | 1/273 (0.4%) |
| Log actions and decisions | 0/83 (0.0%) | 0/273 (0.0%) |
| Keep the log tamper-evident | 0/83 (0.0%) | 0/273 (0.0%) |
| Record who did what and when | 0/83 (0.0%) | 0/273 (0.0%) |
| Name the regime that applies | 1/83 (1.2%) | 1/273 (0.4%) |
| State that legal or regulatory constraints apply | 2/83 (2.4%) | 0/273 (0.0%) |
| State a data-protection technique | 0/83 (0.0%) | 0/273 (0.0%) |

## Controls that never fired

**5 of 23 controls matched nothing in 356 real prompts.**

- `hallucination.grounding` — Ground answers in provided sources
- `audit.log_actions` — Log actions and decisions
- `audit.tamper_evident` — Keep the log tamper-evident
- `audit.actor_attribution` — Record who did what and when
- `compliance.data_protection` — State a data-protection technique

A control that never fires contributes a guaranteed zero to every
score, which caps the reachable maximum for reasons no reader can
see. Two explanations need opposite fixes:

1. **The control is genuinely absent** from prompts in the wild —
   the zero is the finding, and the instrument is working.
2. **The rules are too narrow to detect it** — the control is
   stated in wording no pattern covers. That is a rule defect,
   and from the outside it looks identical to (1).

So the harness re-scans with deliberately looser probes than the
shipped rules. The probes never touch a score; they exist only to
tell these two cases apart. If a probe finds the control where the
rules did not, the rules are the problem.

| Control | Shipped rules | Loose probe | Reading |
| --- | ---: | ---: | --- |
| Ground answers in provided sources | 0/356 | 1/356 (0.3%), 5 patterns | genuinely absent from these corpora |
| Log actions and decisions | 0/356 | 1/356 (0.3%), 3 patterns | genuinely absent from these corpora |
| Keep the log tamper-evident | 0/356 | 2/356 (0.6%), 1 patterns | genuinely absent from these corpora |
| Record who did what and when | 0/356 | 0/356 (0.0%), 4 patterns | genuinely absent from these corpora |
| State a data-protection technique | 0/356 | 0/356 (0.0%), 2 patterns | genuinely absent from these corpora |

Where the reading is *genuinely absent*, note what that says about
the corpora rather than only about the rules: both collections are
dominated by agentic coding and assistant prompts, so controls
belonging to retrieval-grounded systems are expected to be rare
here. A different corpus would move these numbers.


## What this does not show

- Coverage is not quality. A separation here means production prompts
  **write more controls down**, not that they are better written or
  that the agents obey them.
- Both corpora are leaked/aggregated collections of unknown
  completeness. Neither is a random sample of anything.
- Group membership is assigned by which repository a file came from,
  not by inspection. That is the exposure this design accepts in
  exchange for having no hand-labelling step to get wrong.

Self-checks: 104 assertions passed. The run
fails and writes nothing if any rate is unachievable at its own n, if
a denominator is missing, if the interval and the p-value disagree, or
if any 40-character run of input text appears in the output.