How accurate is Stript's on-device PII detection? A benchmark
On our German and English benchmark, Stript's on-device pipeline reaches an F1 of 96.4 (English) and 96.8 (German), with 96 to 97% recall. Structured identifiers like emails, IBANs, and phone numbers are detected at 100%. A human confirms every detection.
Updated
On our benchmark of German and English documents, Stript’s on-device pipeline reaches an F1 of 96.4 on English and 96.8 on German, with 96 to 97% recall. Structured identifiers such as emails, IBANs, and phone numbers are detected at 100%. A human confirms every detection before anything is anonymized.
What we measured
We evaluate on our own annotated benchmark of German and English documents from legal, financial, and HR contexts. The headline metric is F1 (the balance of recall and precision) on the is-this-genuinely-personal-data layer: an item counts only if it is real personal data in context, so a public court or a standalone city name is correctly not counted.
How we tested
The pipeline runs three stages, all on the device:
- Pattern and validation for structured identifiers, checksum-validated where possible (IBAN, credit card, tax IDs).
- Context-aware entity detection for names, organizations, and locations.
- Context classification that judges whether each candidate is genuinely personal data in its sentence.
Every result is shown to a human, who confirms, rejects, or reclassifies before anonymization. This is a transparent, in-house benchmark, not an independent third-party test.
Results
Overall, on the personal-data layer:
| Corpus | Precision | Recall | F1 |
|---|---|---|---|
| German | 97.0 | 96.6 | 96.8 |
| English | 95.4 | 97.4 | 96.4 |
Measured F1 by type:
| Type | German | English |
|---|---|---|
| 100 | 100 | |
| Phone | 100 | 100 |
| IBAN | 100 | 100 |
| Address | 100 | 95.7 |
| Person | 97.7 | 98.8 |
| Organization | 92.3 | 100 |
| Tax ID | 100 | checksum-validated |
| Date | 89.8 | 72.7 |
Other supported types (SSN, ID card, case number, license plate, IP address, credit card, URL) also score at or near 100% where they appear, but on smaller samples.
Dates are the hardest class, and deliberately so: a birth date is personal data, an invoice or letter date is not. Deciding that needs context, which is exactly what the third stage is for.
What the context stage adds
Detection alone (finding every candidate name, date, and number) is high-recall but noisy: it flags standalone city names and administrative dates that are not personal data. The context classification stage is what turns that noisy candidate set into a precise result. In our tests, it lifted overall F1 by up to ~10 points over detection alone, chiefly by recovering genuinely personal dates and removing standalone-location false positives. All of it runs on the device.
Why recall matters more than precision here
For anonymization, the expensive mistake is a miss: a real name or IBAN that slips through. That is why the pipeline is tuned for high recall and why a human reviews every detection. A false positive costs one click to reject; a false negative can leak data.
Limitations
- The benchmark targets direct identifiers. Indirect, quasi-identifying context (a rare combination of role, place, and date) can still allow inference and is left to human review.
- Results are measured on our own corpora; other document types may differ.
- Detection quality scales with the compute tier; higher tiers run a larger context model.
- This is a first pass with a human safety net, not a claim of perfection.
Conclusion
Strong PII detection is achievable entirely on-device, without sending documents to a server. High recall plus a human confirmation step is what makes it usable for confidential documents.
Try it on your own documents, entirely on your machine. Download Stript for free →