1. Introduction

The literature on reproducibility in the digital humanities is largely prescriptive. It recommends version control, open data, documented pipelines, and independent verification. These recommendations are sound and widely endorsed. They are also, as a body of evidence, curiously one-sided: they describe systems that worked, or systems that ought to work, and very seldom report the specific ways in which a real project failed.

The reason is not mysterious. There is a professional cost to publishing one's own errors and no obvious reward. The consequence, however, is that practitioners inherit a literature rich in advice and poor in data. We are told to verify; we are not told what verification actually catches, what it reliably misses, or how many defects a competent project should expect to contain at the point of publication.

This paper contributes data of that kind. It reports every defect found during the construction of a computational study of Qurʾānic vocabulary — a corpus of 6,236 āyāt with 130,030 morphologically annotated segments, from which root, lemma and sense-level counts were derived and published. Twenty-one defects were found. Each is described with its consequence had it gone undetected, and with the mechanism that in fact detected it.

The paper makes no claim that this project was unusually error-prone. It makes the opposite claim: that a project with six independent audit layers and 305 passing checks still accumulated twenty-one defects is the ordinary condition of computational scholarship, and that the useful question is not whether defects exist but which mechanisms find which kinds.

2. The project

The material under study was an annotated Qurʾānic corpus assembled from two public sources: the Uthmanī text (6,236 āyāt) and the morphological annotation of the Quranic Arabic Corpus, comprising 130,030 segments tagged for root, lemma and part of speech. From these, 1,651 roots and 4,776 lemmas were indexed, and counting was performed at three levels — root, lemma, and sense — with outputs published as two reference documents.

The technical apparatus grew over the project's life to six audit layers comprising 305 automated checks: corpus integrity, morphological integrity, document structure, cross-document consistency, project-wide compliance, and a clean-room recount that shares no code with the counting engine. Every build refuses to write output if any check fails.

Crucially, most of that apparatus did not exist at the start. It was written in response to defects as they were discovered. This is itself a finding, examined in section 5.

3. The register

Table 1 lists all twenty-one defects in order of discovery, with the consequence had each gone undetected and the detector that found it. Defects are classified as machine-detected (an automated check reported it) or human-detected (a person reading output noticed it).

4. A silent failure mode specific to Arabic computing

Defect 3 warrants separate treatment because it is not a slip but a property of the writing system interacting with source-code serialisation, and because it recurred four times despite being known.

Arabic diacritics are combining marks. A regular-expression character class written with literal Arabic combining characters — to strip vowel marks, for example — is vulnerable to those characters being reordered when the file is written, read, or transferred between systems that apply Unicode normalisation. Reordering does not produce a syntax error. It produces a different but still valid character class. In the observed case, two ranges intended to cover annotation marks and vowel signs became a single range spanning U+0621–U+064A: the entire Arabic alphabet.

The consequence is that every string passed through normalisation returned empty. Every subsequent count returned zero, or matched everything, depending on the direction of the comparison. No exception was raised at any point. The code continued to look correct on inspection, because the characters are visually near-identical and their order is not perceptible in a rendered editor.

The remedy adopted was structural rather than cautionary. All Arabic handling was consolidated into a single module containing no Arabic characters at all: code points are written as hexadecimal integers, words are written in Buckwalter transliteration and converted at runtime, and normalisation is performed with translation tables built from integers rather than regular expressions over character ranges. The module executes a self-test at import and raises if any check fails, so that a corrupted build cannot start.

The generalisable point is that for a defect whose cause is a property of the characters rather than of the author's attention, care is not a remedy. Only removing the vulnerable construction from the codebase eliminates it.

5. Who found what

Table 2 summarises the detection split.

The zero is the substantive result. Two independent processes searched the same artefacts over the same period and their findings do not intersect at a single point.

The explanation is not that one detector is superior. It is that the two are sensitive to different classes of defect. An automated check can only test a condition someone thought to express. Every machine-detected defect in Table 1 corresponds to a check written because that kind of failure had been anticipated — usually because a similar one had already occurred. The human-detected defects share the opposite property: none could have been expressed as a check beforehand, because none had a definite shape until a reader encountered the output and found it unconvincing.

Defect 16 illustrates this precisely. A column of per-phase counts summed to 3,804 against a stated corpus total of 1,651. Every figure was correct: the column counted items present in each phase, and items recurring across phases are counted once per phase. No arithmetic check could fail, because no arithmetic was wrong. What was wrong was that the column was labelled in a way that invited addition. The defect existed only in the relationship between a correct table and a reader's reasonable expectation, and only a reader could locate it.

5.1 What the absence of overlap implies

In fields that must estimate an incompletely observed population, capture–recapture provides the standard approach: the overlap between two independent samples estimates the total. The Chapman estimator is

where n₁ and n₂ are the two sample sizes and m the overlap. With n₁=15, n₂=6 and m=0 this returns 111.

That figure should not be reported as an estimate and is not offered as one. Capture–recapture assumes both samples draw from the same population with independent and comparable detection probabilities. Section 5 argues precisely that this assumption fails here: the two detectors are sensitive to disjoint classes.

But the failure of the assumption is not a reason to discard the observation. It is the observation. When two searchers cover identical ground and share no findings, the supportable inference is not that the ground is now clear. It is that each is systematically blind to a class the other sees, with no information about how many classes are watched by neither. Nothing in the data supports a claim that the defect search in this project is near completion.

6. Discussion

6.1 Publishing in order of claim breadth

If undetected defects cannot be excluded, the practical response is to limit what any one of them can invalidate. A dataset description asserting that a particular annotation contains particular items survives a correction as a version increment. A paper asserting a substantive finding does not: an error in the measure removes the finding. This suggests sequencing publication by breadth of claim, with descriptive and methodological outputs preceding interpretive ones — not from timidity, but because it minimises the expected cost of the error one cannot rule out.

6.2 The cost of an audit that cries wolf

Defect 20 records three occasions on which an audit reported failures that did not exist — on one occasion 258 of them, caused by a check that stripped HTML tags but not character entities and therefore flagged every ellipsis as anomalous. On another, a corpus of 36,390 records stored as a nested structure was read as 7 unlabelled records.

False alarms are conventionally treated as a nuisance. They are better understood as defects in their own right, and among the more dangerous, because their effect is on the reader rather than the data: an audit that regularly reports non-existent failures trains its author to discount it, and the discounting is not selective. Every false alarm in this project was therefore logged, corrected, and given a regression test.

6.3 Against fully automated verification

Six of twenty-one defects were found by a person reading output, and no automated check in the six audit layers had any prospect of finding them, because none had been imagined when those layers were written. This is not an argument that automation is inadequate; fifteen defects say otherwise. It is an argument about complementarity. Automation is exhaustive within the space of anticipated failures and blind outside it. A reader is unreliable within that space and the only available instrument outside it.

A verification design that removes the human reader in the name of rigour therefore does not increase rigour. It eliminates the sole detector of the class of defect nobody predicted — which is, by construction, the class most likely to survive into publication.

7. Limitations

This is a single-project report, and its counts are not a sample from which rates can be generalised. The classification of defects as machine- or human-detected reflects which detector reported each first, and in a differently ordered project some might have been found by the other. The register was compiled by the author, who is also the source of every defect in it; a register kept by an independent party might differ in both content and emphasis. Finally, and consistently with the paper's own argument, the register is certainly incomplete.

8. Conclusion

Twenty-one defects were found in a project with six audit layers and 305 passing checks. Fifteen were found by machines and six by a person reading, with no overlap between them. One failure mode — the reordering of literal Arabic characters within regular-expression character classes — is specific to Arabic-language computing, fails silently, and recurred four times before a structural remedy removed it.

The absence of overlap between detectors is the paper's main quantitative result, and it does not support optimism. It indicates two processes each blind to what the other sees, and gives no basis for estimating what remains invisible to both.

The register is published open-ended. Its purpose is served if the twenty-second defect is reported by someone else.

References

1. Dukes, K. The Quranic Arabic Corpus. University of Leeds. Morphological annotation, released under the GNU General Public License.
2. Tanzil Project. Qurʾān text, Uthmanī orthography.
3. Chapman, D. G. (1951). Some properties of the hypergeometric distribution with applications to zoological sample censuses. University of California Publications in Statistics 1, 131– 160.
4. Clauset, A., Shalizi, C. R., & Newman, M. E. J. (2009). Power-law distributions in empirical data. SIAM Review 51(4), 661–703.
5. Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008). Fightin’ words: lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis 16(4), 372–403.
6. The Unicode Consortium. Unicode Standard Annex #15: Unicode Normalization Forms.

Appendix: what remains to be done before submission