Source-1: weights, loader and model card

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+images/source1_benchmark.png filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

AUTHORS

@@ -0,0 +1,8 @@
+# This is the list of Source-1's significant contributors.
+#
+# "The Source-1 Authors", as used in the copyright notices of this repository
+# (for example "Copyright 2026 The Source-1 Authors" in NOTICE), means the
+# people listed below. Each author is listed by their Hugging Face handle.
+
+The Source-1 Authors:
+  msmth

EVALUATION.md

@@ -0,0 +1,1042 @@
+# Source-1 evaluation
+
+Source-1 was compared with its teacher and 16 public quality scorers on three test sets. An independent proprietary
+LLM grader scored every chunk with Source-1's 13-field rubric. Each number says how closely a model's ranking agrees
+with the grader's, so it measures agreement with Source-1's own rubric, on home ground.
+
+## Summary
+
+Rank agreement (Spearman) with the grader's overall score. Each model is read on the chunks it scored.
+
+| test set | chunks | Source-1 (307M) | propella-1 4B (4.0B) | FineWeb-Edu classifier (English only) | teacher (open-weight, 27B) |
+|---|---|---|---|---|---|
+| held-out set, 53 languages (main result) | 495 | 0.900 | 0.756 | 0.529 | 0.912 |
+| English exam | 413 | 0.921 | 0.820 | 0.453 | 0.912 |
+| 12-language exam | 352 | 0.895 | 0.637 | - | 0.880 |
+
+propella-1 4B scored 493, 412 and 350 of these chunks. The FineWeb-Edu classifier is read on the 159 English held-out
+chunks, where Source-1 scores 0.864. The English exam has 414 chunks; the teacher scored 413, and comparisons use those.
+
+- Source-1 agrees with the grader more closely than each of the 16 public scorers, on every set where they were
+  compared. All 39 of these comparisons ([how they are counted](#every-public-scorer)) have 95% intervals clear of
+  zero.
+- It is level with its teacher within noise, with about 1/88 of the teacher's parameters. On the held-out set it is
+  0.012 lower (95% interval of the difference -0.03 to +0.004).
+- The closest public scorer, propella-1 4B, trails by 0.144 on the held-out set. A more generous reading of it,
+  chosen after the results were known, narrows the gap to 0.064-0.081 ([details](#how-propella-1-is-read)).
+- At its shipped drop line (the default `keep` flag), Source-1 catches 43 of the 64 chunks the grader drops on the
+  held-out set. The teacher, at its own keep flags, catches 46.
+
+The held-out set is the main result. The exams count less, because they helped choose the teacher
+([why](#independence-from-development)).
+
+## How we measured
+
+**The grader.** Its overall score and keep flag are computed from its 13 fields with the rubric's own formula, as for
+Source-1. Its drops are the chunks that hit the rubric's hard filters (spam, boilerplate or toxic text). Its labels
+were never trained on.
+
+**Home ground.** The held-out documents come from the same kinds of sources as the training data. The public scorers
+were built for their own definitions of quality, most for educational value, and are not wrong when they disagree with
+this rubric.
+
+**The three test sets.**
+
+| test set | chunks | languages | grader drops | what it is |
+|---|---|---|---|---|
+| held-out set (main result) | 495, one per document | 53 | 64 | documents from Source-1's held-out test split, never trained or calibrated on |
+| English exam | 414, from 332 documents (413 scored by the teacher) | English | 25 | 200 chunks drawn at random, plus 214 harder cases added on purpose |
+| 12-language exam | 352, one per document | 12 | 46 | web text drawn at random from FineWeb-2, about 30 chunks per language |
+
+**95% intervals.** Each difference between two models comes with a 95% interval from a paired bootstrap over
+documents. When the interval excludes zero, chance alone is an unlikely explanation for the difference.
+
+**How the public scorers were run.** Each public scorer ran from the pinned revision in
+[Appendix C](#appendix-c-public-scorer-repositories), following its model card and its own code or prompt where it
+publishes one, except as noted here. propella-1 and EAI-Distill were decoded greedily, although the propella-1 4B and
+EAI-Distill repositories default to sampling. propella-1 ran in plain transformers, without the SGLang server and JSON
+grammar its card recommends; the serving engine mainly affects speed, so we claim no speed comparison with it.
+
+<details>
+<summary>How the public scorers were run, in full</summary>
+
+### How the public scorers were run
+
+- Each public scorer ran from the repository and revision listed in
+  [Appendix C](#appendix-c-public-scorer-repositories), following its model card and its own code or prompt where it
+  publishes one.
+- The encoder classifiers ran in Hugging Face transformers as their model cards show, in the precision each card
+  states (bfloat16 where it says so, float32 otherwise).
+- The two fastText models ran in the fasttext library. They read the whole text with newlines turned into spaces, as
+  the DCLM code does.
+- EAI-Distill ran in transformers in float32 with greedy decoding (its repository's default settings sample).
+- propella-1 ran in plain transformers in bfloat16, with its repository's own prompt and chat template, greedy
+  decoding and no JSON grammar. Its card recommends serving with SGLang and a JSON grammar, and the 4B's default
+  settings sample at temperature 0.7. The serving engine mainly affects speed. Greedy decoding without a grammar gives
+  the same tokens as grammar-constrained greedy decoding, up to the first token the grammar would forbid, and it
+  avoids sampling noise. An answer that did not parse got one retry with more new tokens
+  ([details](#propella-1s-answers)). We claim no speed comparison with propella-1.
+
+</details>
+
+### Independence from development
+
+- The exam sets helped choose the teacher and its prompt setup, by agreement with the exam grader's labels, and
+  Source-1's backbone was kept over two other multilingual encoders in a comparison that looked at these sets.
+- Grades and reviews by proprietary LLMs, among them the grader's model family, informed the rubric, the drop-line
+  candidates and floor, and the data filters.
+- AI assistants helped draft the rubric text and write the project's code. The grader's model family includes one of
+  the assistants that helped draft the rubric text.
+- The held-out set was built after the teacher and its prompt were fixed. It was not used to choose the model. It was
+  not hidden, though: the candidate drop lines were drafted after its grades had been seen.
+- No grade or review by a proprietary LLM was ever used as a label, a training target or a training example.
+
+<details>
+<summary>Independence from development, in full</summary>
+
+What grades from the grader's model family did and did not influence:
+
+- **The exam sets helped choose the teacher.** The English and 12-language exams are the samples on which the teacher
+  was vetted and its prompt setup was chosen, by agreement with the exam grader's labels. The prompt setup covers the
+  answer format, reasoning on or off, and a rubric rule for pages stitched together from unrelated or scrambled text,
+  added after reviewing disagreements on these samples. The backbone was also kept over two other multilingual
+  encoders in a comparison that looked at these sets. So the exams are not independent of the grader: Source-1's
+  teacher was in part selected to agree with it there. They are reported, but they count less than the held-out set.
+- **The held-out set did not.** It was built after the teacher and its prompt had been fixed, and was not used to
+  choose either. It is called held-out because Source-1 never trained or calibrated on it. But it was not hidden
+  during development: Source-1's results on it were checked, and the candidate drop lines were drafted after its
+  grades had been seen. The released model was not chosen by its score on this set: it is the model trained on the
+  final, fully cleaned data.
+- **The rubric and the drop-line rule.** The choice between rubric revisions, the list of candidate drop lines and the
+  0.95 keep-agreement floor of the drop-line rule were decided using grades from proprietary LLM graders, among them
+  models of the grader's family, on samples of documents, some of them training chunks. The drop line itself was then
+  picked by that fixed rule on the teacher's validation labels. The grader's labels were not used to pick it.
+- **Training.** Every training label comes from the open-weight teacher. The grader's labels, and every other grade or
+  review by a proprietary LLM, were never used as labels, training targets or training examples. The learning rate
+  and the natural language mix came from a short sweep (two learning rates, two language mixes, half an epoch each)
+  on a smaller training set labeled by the same teacher, read on validation agreement with the teacher. The epoch
+  count came from the same validation curves. The checkpoint is the final step, which had the best validation
+  agreement with the teacher (Spearman 0.955). No graded test set was used for these choices.
+- **Data filtering.** Samples reviewed by a proprietary LLM measured how often the license and table-of-contents
+  filters missed or over-fired. This informed which collections were filtered out; the reviewed documents were then
+  kept out of training. AI assistants also reviewed source terms and document notices across the training data,
+  which informed the license rules. They also helped draft the rubric text and write the project's code.
+
+</details>
+
+<details>
+<summary>The test sets in detail: composition, near-duplicate check, safety filter</summary>
+
+### The test sets in detail
+
+- **Held-out set** (the main result): 495 chunks in 53 languages, one per document. All come from Source-1's held-out
+  test split, drawn from the four data stages (web 129, multilingual web 276, conversations/code/synthetic 48, open
+  books 42). The grader drops 64 of them. 159 chunks are English and 29 Chinese; most other languages have 1 to 12
+  chunks. Source-1 never trained or calibrated on these documents. They come from the same kinds of sources as the
+  training data. 42 of the 495 chunks (8.5%) come from sources that were later removed from training under the
+  license rules (19 from DCLM-baseline, 16 raw Common Crawl pages, and 7 from collections with unreliable or gated
+  license terms). The test split also keeps documents that the license filtering removed from training.
+- **English exam**: 414 chunks from 332 whole documents split into chunks. 200 chunks from 196 documents were drawn at
+  random from web, wiki, Common Pile, math and code sources (the **random-sample** chunks and documents). 214 chunks
+  from 136 documents were added on purpose to cover harder cases (long documents 57 chunks, academic 50, math 28,
+  code 27, spam 23, toxic 20, fiction 9). The grader drops 25 of the chunks (24 documents). The added chunks make the
+  set unlike a random draw. They move each model's numbers, up for some models and down for others (Source-1: rank
+  0.919 on the random-sample chunks alone and 0.921 on all 414; AUC 0.987 and 0.979). The teacher has no score for
+  one of the 414 chunks (a random-sample chunk). So the comparisons with the teacher and the public scorers' English
+  rows use the 413 chunks it scored (411 or 412 where a public scorer also lacks one). The 512-token comparison uses
+  all 414.
+- **12-language exam**: 352 chunks, one per document, all drawn at random from FineWeb-2 web text in 12 languages
+  (ar, bn, de, es, hi, ja, ko, ru, sw, th, vi, zh; about 30 each). It has no code or math: 351 of the 352 chunks are
+  plain text by the grader's label. The grader drops 46.
+
+Every exam document was checked against the training data (exact text, URL, title, long-line and shingle matching).
+No copies were found.
+
+#### Near-duplicate check
+
+The 495 held-out chunks were compared with every training and validation record. The check measures the share of a
+chunk's distinctive 40-character pieces that a single record contains. No chunk is covered 80% or more by any record.
+4 are covered 64% to 71%: three short files from code collections and one US government record, 98 to 345 tokens
+each, all short templated texts. 9 in all are covered 20% or more. These chunks stay in the reported set. Without the
+4, Source-1's rank agreement is 0.901 (491 chunks); without all 9, 0.900.
+
+#### Safety filter
+
+A safety filter fixed before any result was seen removed a small number of documents from every evaluation set and
+from Source-1's training, validation and test data. Documents that substantially copy removed text were removed from
+its data as well. Every model is compared on the same filtered chunks.
+
+</details>
+
+<details>
+<summary>The grader in detail, and how much its grades move when repeated</summary>
+
+### The grader in detail
+
+Strictly, these are two graders from one proprietary model family: one graded the exams, another the held-out set.
+This file calls either "the grader". That model family also includes one of the assistants that helped draft the
+rubric text. The exams were graded with an older revision of the rubric, from before the rubric settled how to score
+ads.
+
+The grader returns the 13 fields only. Its reference overall score and keep flag are computed from its scores in the
+same way as Source-1's: the overall formula of the rubric (see [README.md](README.md#what-it-returns)), and keep =
+false when the rubric's hard filters match (`toxicity >= 4`, `spam_seo >= 4` or `boilerplate >= 4.5`). "The grader's
+drops" in this file are those chunks: spam, boilerplate or toxic text under the hard filters.
+
+The same sets were scored by the teacher and by the 16 public quality scorers, open-weight models run as described
+under [How the public scorers were run](#how-the-public-scorers-were-run).
+
+#### Grader self-agreement
+
+The exam grader also graded part of each exam a second time. The two gradings agree at rank 0.904 on 97 English
+random-sample documents and 0.945 on 58 documents of the 12-language exam. This is a rough noise ceiling: it shows how
+much one grading moves when repeated. It is not a hard upper bound, because a model can agree with one grading more
+closely than two noisy gradings agree with each other.
+
+</details>
+
+### The one-line header
+
+<details>
+<summary>The one-line header that Source-1, the teacher and the grader saw</summary>
+
+
+Source-1, the teacher and the grader saw each chunk after a one-line header. It gives the chunk's source type
+("dataset record" for 478 of the 495 held-out chunks, and a code-file type such as "Python source file" for the other
+17), its part number when it is one part of a longer document and, for 121 of the 495 held-out chunks, a title. The
+public scorers read the text alone. On the 1,261 chunks of the held-out set and both exams, dropping the whole header
+moved Source-1's overall score by 0.03 on average (at most 0.655). Dropping a title moved it by 0.05 on average on the
+chunks that had one. Adding a URL, which no training input had, moved it by up to 0.7 and did not improve its rank
+agreement with the grader. So `source1.py` does not show a URL to the model by default.
+
+</details>
+
+<details>
+<summary>Metric definitions: rank, AUC, matched keep rate, intervals</summary>
+
+### Metric definitions
+
+Computed per chunk:
+
+- **Rank**: Spearman correlation between a model's overall score and the grader's overall score.
+- **AUC**: how well a model's score separates the grader's drops from its keeps (area under the ROC curve; a low score
+  means drop; ties count one half).
+- **Matched keep rate**: every model keeps the same share of chunks the grader keeps (87% on the held-out set), taking
+  its highest-scored chunks. **Keep agreement** is the share of chunks where the model's keep/drop matches the
+  grader's. **Drop recall** is the share of the grader's drops the model also drops. This puts scorers with different
+  scales on the same operating point. Chunks tied at the cut are kept fractionally (the expected value over every
+  order of the tied chunks), so a count of drops caught can be fractional; such counts are given as "about".
+- For Source-1 and the teacher, AUC and the matched keep rate use the overall score before it is clipped to 0, so
+  heavily penalized chunks are not tied at zero.
+- Intervals at a drop line are Wilson 95% intervals.
+- **Differences between two models** (Source-1 minus the other, on the chunks both scored) come with paired bootstrap
+  95% intervals: documents resampled with replacement, 2,000 resamples, seed 0, the same resamples for both models,
+  percentile intervals. Interval ends are given to two decimals (three when they are close to zero), because the
+  third decimal moves with the random seed.
+
+</details>
+
+## Results
+
+### Held-out set
+
+The table shows Source-1, the teacher and the three multilingual public scorers that agree best with the grader. Keep
+agreement and drop recall are read at a matched keep rate: each model keeps its top-scored 87% of chunks, the share
+the grader keeps ([Metric definitions](#metric-definitions)).
+
+| model | chunks | rank | AUC | keep agreement (87% kept) | drop recall (87% kept) |
+|---|---|---|---|---|---|
+| **Source-1** | 495 | 0.900 | 0.946 | 0.927 | 0.719 (46/64) |
+| Teacher (open-weight 27B LLM) | 495 | 0.912 | 0.948 | 0.919 | 0.688 (44/64) |
+| propella-1 4B | 493 | 0.756 | 0.862 | 0.886 | 0.562 (36/64) |
+| propella-1 1.7B | 495 | 0.736 | 0.855 | 0.896 | 0.599 (about 38/64) |
+| JQL-Edu (mean of 3 balanced heads) | 495 | 0.600 | 0.737 | 0.826 | 0.328 (21/64) |
+
+- At this matched rate Source-1 catches 46 of the grader's 64 drops and the teacher 44, a difference within noise. At
+  each model's own drop line (for the teacher, its own keep flags) they catch 43 and 46
+  ([The drop line](#the-drop-line)). Source-1 and the teacher are also level within noise on rank and AUC.
+- The lead over propella-1 4B, the closest public scorer, is at least as large outside English: 0.916 against 0.765
+  on the 335 non-English chunks.
+
+<details>
+<summary>Held-out set in detail: every interval, and results by data stage and for Chinese</summary>
+
+### Held-out set in detail
+
+- Every public scorer ranks the held-out set well below Source-1 on this rubric. The closest, propella-1 4B, reaches
+  0.756 on the 493 chunks it scored (Source-1 minus propella-1 4B: +0.144, 95% interval +0.11 to +0.18). It is also
+  further from the grader on the drops: AUC 0.862 against Source-1's 0.946 (+0.084, +0.05 to +0.12). At the matched
+  rate it catches 36 of the 64 drops to Source-1's 46 (drop recall +0.156, +0.06 to +0.25). Most public scorers do
+  not target spam, boilerplate or toxicity, which is what the grader drops. With propella-1's own ratings for them
+  added, its AUC gap is +0.049 (+0.02 to +0.07; see [How propella-1 is read](#how-propella-1-is-read)).
+- The lead over propella-1 4B is at least as large outside English: 0.916 against 0.765 on the 335 non-English chunks
+  in 52 languages (+0.151, +0.11 to +0.20; 34 of them are in languages propella-1's card does not list). It is +0.117
+  (+0.06 to +0.18) on the 158 English chunks it scored (of 159).
+- Source-1 ranks 0.012 below its teacher (0.900 vs 0.912; 95% interval of the difference -0.03 to +0.004), also
+  outside English (-0.012, -0.03 to +0.006). It is level with it on AUC (0.946 vs 0.948; -0.02 to +0.01). At the
+  matched rate it catches 46 of the grader's 64 drops and the teacher 44, a difference within noise (drop recall
+  +0.031, -0.04 to +0.10).
+
+By data stage, and for Chinese (same chunks; drops caught at each model's own drop line):
+
+| model | web (129) | multilingual web (276) | conversations, code, synthetic (48) | open books (42) | Chinese (29) |
+|---|---|---|---|---|---|
+| **Source-1**, rank | 0.891 | 0.905 | 0.728 | 0.677 | 0.934 |
+| Teacher, rank | 0.902 | 0.917 | 0.821 | 0.741 | 0.938 |
+| **Source-1**, drops caught | 14 of 17 | 28 of 44 | 1 of 2 | 0 of 1 | 11 of 14 |
+| Teacher, drops caught | 14 of 17 | 29 of 44 | 2 of 2 | 1 of 1 | 10 of 14 |
+
+The held-out set has 159 English chunks and 1 to 12 chunks for most other languages (29 for Chinese), so
+per-language results are noisy. Groups with fewer than 50 chunks are indicative only.
+
+</details>
+
+### Every public scorer
+
+Source-1 ranks the chunks closer to the grader than each of the 16 public scorers, on every set each was run on. All
+39 of these rank-agreement leads have 95% intervals clear of zero, also after a Bonferroni adjustment, while the
+teacher's intervals all include zero. The smallest lead in size is +0.101 (+0.07 to +0.14), over propella-1 4B on the
+English exam.
+
+<details>
+<summary>All 17 public-scorer rows with intervals, and how the 39 comparisons are counted</summary>
+
+We compared 16 public scorers. The tables have 17 public rows, because the Nemotron-CC 3-way ensemble is computed
+from three of the 16 (the two NeMo Curator classifiers and DCLM fastText). That gives 39 differences with intervals:
+17 on the held-out set, 17 on the English exam and 5 on the 12-language exam, where only the five multilingual
+scorers were run. Each cell is a rank agreement with the grader, and the lead is Source-1 minus that scorer on exactly
+the chunks it scored. The held-out set has 495 chunks unless noted; the English exam 411 to 413; the 12-language exam
+352 unless noted.
+
+Multilingual scorers, and the teacher:
+
+| model | held-out: rank | held-out: Source-1 lead (95% interval) | English exam: rank | English exam: lead | 12 languages: rank | 12 languages: lead |
+|---|---|---|---|---|---|---|
+| **Source-1** | 0.900 | - | 0.921 | - | 0.895 | - |
+| Teacher (open-weight 27B LLM) | 0.912 | -0.012 (-0.03 to +0.004) | 0.912 | +0.009 (-0.01 to +0.02) | 0.880 | +0.015 (-0.007 to +0.04) |
+| propella-1 4B | 0.756 (493 chunks) | +0.144 (+0.11 to +0.18) | 0.820 | +0.101 (+0.07 to +0.14) | 0.637 (350 chunks) | +0.258 (+0.19 to +0.33) |
+| propella-1 1.7B | 0.736 | +0.163 (+0.13 to +0.20) | 0.812 | +0.108 (+0.07 to +0.14) | 0.631 | +0.264 (+0.20 to +0.33) |
+| JQL-Edu | 0.600 | +0.299 (+0.25 to +0.36) | 0.541 | +0.380 (+0.30 to +0.46) | 0.379 | +0.516 (+0.42 to +0.61) |
+| FinePDFs-Edu | 0.476 | +0.424 (+0.36 to +0.49) | 0.787 | +0.134 (+0.10 to +0.18) | 0.332 | +0.563 (+0.47 to +0.66) |
+| FineWeb2-HQ (21 of the 53 languages) | 0.449 (342 chunks) | +0.456 (+0.38 to +0.54) | 0.448 | +0.473 (+0.39 to +0.56) | 0.496 (205 chunks) | +0.415 (+0.31 to +0.53) |
+
+English-only scorers (held-out set: its 159 English chunks):
+
+| model | held-out: rank | held-out: Source-1 lead (95% interval) | English exam: rank | English exam: lead |
+|---|---|---|---|---|
+| **Source-1** | 0.864 | - | 0.921 | - |
+| FineWeb-Edu classifier | 0.529 | +0.336 (+0.22 to +0.45) | 0.453 | +0.468 (+0.38 to +0.56) |
+| DCLM fastText (OH+ELI5) | 0.299 | +0.565 (+0.42 to +0.71) | 0.193 | +0.728 (+0.63 to +0.83) |
+| NeMo Curator edu (Nemotron-4 labels) | 0.412 | +0.452 (+0.32 to +0.59) | 0.260 | +0.661 (+0.55 to +0.78) |
+| NeMo Curator edu (Mixtral labels) | 0.498 | +0.366 (+0.24 to +0.50) | 0.422 | +0.499 (+0.41 to +0.59) |
+| Meta-rater reasoning | 0.580 | +0.284 (+0.19 to +0.39) | 0.710 | +0.211 (+0.15 to +0.27) |
+| Meta-rater readability | 0.574 | +0.291 (+0.19 to +0.40) | 0.619 | +0.302 (+0.24 to +0.37) |
+| Meta-rater cleanliness | 0.597 | +0.267 (+0.18 to +0.37) | 0.698 | +0.223 (+0.17 to +0.28) |
+| Meta-rater professionalism | 0.538 | +0.327 (+0.23 to +0.43) | 0.688 | +0.233 (+0.18 to +0.30) |
+| EAI-Distill 0.5B | 0.592 | +0.273 (+0.18 to +0.37) | 0.685 | +0.236 (+0.19 to +0.29) |
+| NVIDIA quality classifier (DeBERTa) | 0.326 | +0.539 (+0.39 to +0.70) | 0.181 | +0.740 (+0.59 to +0.90) |
+| Dolma 3 fastText quality | 0.341 | +0.524 (+0.39 to +0.66) | 0.386 | +0.535 (+0.46 to +0.62) |
+| Nemotron-CC 3-way ensemble (derived) | 0.353 | +0.511 (+0.37 to +0.67) | 0.295 | +0.626 (+0.53 to +0.72) |
+
+- Every one of the 39 public-scorer intervals excludes zero, and the leads stay clear of zero after a Bonferroni
+  adjustment for the 39 comparisons. The teacher's intervals all include zero.
+- This holds for rank agreement. The other measures are much noisier on the 159 English held-out chunks (see
+  [AUC and noise](#auc-and-noise)).
+
+</details>
+
+<details>
+<summary>AUC of every scorer, Source-1 on each scorer's chunks, and the noise behind the tables</summary>
+
+### AUC and noise
+
+AUC against the grader's drops, each scorer on the chunks it scored:
+
+| model | held-out: AUC | English exam: AUC | 12 languages: AUC |
+|---|---|---|---|
+| **Source-1** | 0.946 | 0.979 | 0.961 |
+| Teacher (open-weight 27B LLM) | 0.948 | 0.965 | 0.945 |
+| propella-1 4B | 0.862 | 0.924 | 0.836 |
+| propella-1 1.7B | 0.855 | 0.941 | 0.846 |
+| JQL-Edu | 0.737 | 0.789 | 0.645 |
+| FinePDFs-Edu | 0.731 | 0.855 | 0.659 |
+| FineWeb2-HQ | 0.780 | 0.795 | 0.768 |
+| FineWeb-Edu classifier | 0.812 | 0.754 | - |
+| DCLM fastText (OH+ELI5) | 0.741 | 0.576 | - |
+| NeMo Curator edu (Nemotron-4 labels) | 0.706 | 0.645 | - |
+| NeMo Curator edu (Mixtral labels) | 0.788 | 0.775 | - |
+| Meta-rater reasoning | 0.763 | 0.793 | - |
+| Meta-rater readability | 0.817 | 0.832 | - |
+| Meta-rater cleanliness | 0.873 | 0.880 | - |
+| Meta-rater professionalism | 0.735 | 0.777 | - |
+| EAI-Distill 0.5B | 0.786 | 0.835 | - |
+| NVIDIA quality classifier (DeBERTa) | 0.794 | 0.706 | - |
+| Dolma 3 fastText quality | 0.752 | 0.710 | - |
+| Nemotron-CC 3-way ensemble | 0.750 | 0.700 | - |
+
+Source-1's own rank / AUC on each scorer's chunks:
+
+- held-out set: 0.900 / 0.946 (on all 495 chunks and on propella-1 4B's 493), 0.864 / 0.941 on the 159 English chunks,
+  and 0.905 / 0.957 on FineWeb2-HQ's 342;
+- English exam: 0.921 / 0.979 (on each set of 411 to 413 chunks);
+- 12-language exam: 0.895 / 0.961 (on 352 chunks and on propella-1 4B's 350), and 0.910 / 0.962 on FineWeb2-HQ's 205.
+
+Noise:
+
+- None of the 2,000 resamples put any of the 39 differences at or below zero. Measured against its own noise, the
+  smallest lead is 5.3 bootstrap standard deviations above zero (Meta-rater cleanliness on the 159 English held-out
+  chunks). So the leads stay clear of zero after a Bonferroni adjustment for the 39 comparisons (normal
+  approximation).
+- On the 159 English held-out chunks, with 15 grader drops, the other measures are much noisier. The keep-agreement
+  or drop-recall interval at the matched rate reaches zero for 7 of the 12 English-only rows. Two AUC leads are only
+  just clear of zero: over the FineWeb-Edu classifier (+0.129, +0.004 to +0.275) and over Meta-rater cleanliness
+  (+0.069, +0.008 to +0.133).
+- The English-only scorers are compared on far fewer held-out chunks (159) than the multilingual ones. The exam sets
+  are not independent of the grader ([Independence from development](#independence-from-development)).
+- All of this is agreement with Source-1's own rubric. The public scorers were built for their own definitions of
+  quality (most for educational value) and are not wrong when they disagree with it.
+
+</details>
+
+<details>
+<summary>How each public scorer is read: inputs, main scores, EAI-Distill, the Nemotron-CC ensemble</summary>
+
+### How each public scorer is read
+
+Each public scorer is read through one main score, on the chunks it scored. The English-only scorers are read on
+English chunks and FineWeb2-HQ on its 21 languages. propella-1, JQL-Edu and FinePDFs-Edu (with its fallback model for
+five languages) are read on all 53, although propella-1's card does not list ten of them (az, fil, gu, kk, kn, ml, mr,
+ms, ta, te) and JQL-Edu's backbone covers 52. Of the 16, 11 are English-only and one covers 21 of the 53 languages;
+the other four were run on all 53. The public scorers read the text without Source-1's header.
+
+| public scorer | languages it was run on | input it reads | main score used here |
+|---|---|---|---|
+| propella-1 4B | all 53 (its card lists 43 of them) | the whole chunk, up to 50,000 characters | a weighted mean of four of its quality ratings, defined by us (see [How propella-1 is read](#how-propella-1-is-read)) |
+| propella-1 1.7B | all 53 (its card lists 43 of them) | the whole chunk, up to 50,000 characters | as propella-1 4B |
+| JQL-Edu | all 53 (its backbone covers 52) | the first 8,192 tokens | the mean of its three balanced educational-value heads |
+| FinePDFs-Edu | all 53 (a model per language; a fallback model for five) | about 2,000 tokens from the start, and from the end of long texts (the higher score counts) | its educational-value score |
+| FineWeb2-HQ (the per-language classifiers in epfml/FineWeb-HQ-Classifiers; on English text, its FineWeb-HQ classifier) | 21 of the 53 | the first 512 tokens | its probability of high quality |
+| FineWeb-Edu classifier | English | the first 512 tokens | its educational-value score |
+| DCLM fastText (OH+ELI5) | English | the whole text | its probability of the high-quality label |
+| NeMo Curator edu (Nemotron-4 labels) | English | the first 512 tokens | its educational-value score |
+| NeMo Curator edu (Mixtral labels) | English | the first 512 tokens | its educational-value score |
+| Meta-rater reasoning, readability, cleanliness, professionalism (four models) | English | the first 4,096 tokens | each model's expected rating |
+| EAI-Distill 0.5B | English | the whole text up to 30,000 characters (beyond that, the start, a middle part and the end, as its card says) | a 0-5 score defined by us from four of its labels (see [How EAI-Distill and the Nemotron-CC ensemble are read](#how-eai-distill-and-the-nemotron-cc-ensemble-are-read)) |
+| NVIDIA quality classifier (DeBERTa) | English | the first 1,024 tokens | its expected class (low 0, medium 1, high 2) |
+| Dolma 3 fastText quality | English | the whole text | its probability of the high-quality label |
+| Nemotron-CC 3-way ensemble (derived) | English | as its three parts (the two NeMo Curator classifiers and DCLM fastText) | the highest of the three parts' percentile buckets, with percentiles taken within each evaluation set |
+
+#### How EAI-Distill and the Nemotron-CC ensemble are read
+
+- **EAI-Distill**: reasoning depth and technical correctness are read as their position among the five ordered levels
+  (0 to 4) divided by 4. Extraction artifacts and missing content are 1 when the model reports any and 0 when it
+  reports none. Main score = 5 x mean(reasoning depth, technical correctness) - extraction artifacts - missing content,
+  clipped to 0-5. An indeterminate or abstaining answer is left out of the mean (no score when both are), and a missing
+  penalty counts 0. Its answer codes were read with the code tables of the Essential-AI/eai-taxonomy README at commit
+  `e8a934d5ca77a05f8daddc73466aedf8a9eb7a6c`.
+- **Nemotron-CC 3-way ensemble**: each part's score becomes a bucket, floor(20 x its percentile rank within the
+  evaluation set) (0 to 19), and the ensemble's score is the highest of the three buckets.
+
+</details>
+
+<details>
+<summary>Educational value alone: Source-1's educational_value field against every public scorer</summary>
+
+### Educational value alone
+
+Most public scorers were built to rate educational value. Read that way, Source-1's `educational_value` field ranks
+the held-out set closer to the grader's `educational_value` (0.879) than every public scorer's main score does. All
+17 intervals exclude zero, as they do on both exams. The closest is propella-1 4B's composite (0.823; Source-1 +0.056,
+95% interval +0.03 to +0.08). Among the dedicated educational-value classifiers, JQL-Edu trails by +0.174 (+0.14 to
++0.22) and, on the 159 English chunks, the FineWeb-Edu classifier by +0.238 (+0.16 to +0.33). On the English exam,
+Source-1's `educational_value` reaches 0.904 against 0.614 for the FineWeb-Edu classifier and 0.884 for propella-1 4B.
+The margin over propella-1 4B is small there: +0.020 (+0.002 to +0.040). The grader's `educational_value` follows
+Source-1's rubric anchors, not the annotation prompts these classifiers were trained on. propella-1's own
+educational-value rating agrees less with it (0.776 on the held-out set) than its composite does.
+
+</details>
+
+### How propella-1 is read
+
+<details>
+<summary>The composite we read it through, a more generous reading, and propella-1's answers</summary>
+
+propella-1 answers in words, not numbers. We map each of its ordered ratings to integers. We read it through a
+weighted mean of four quality ratings (educational value, reasoning, content quality, information density), weighted
+as in Source-1's quality formula. That leaves out its ratings for commercial bias, content ratio and integrity, and
+content safety, which are close to the grader's drop rules. It was also run on all 53 languages, ten of which its card
+does not list. As a check of a more generous reading, chosen after the results were known, we also restricted it to
+its 43 listed languages and subtracted rubric-style penalties for those ratings. Each was rescaled to 0-5, then 0.5,
+0.4 and 0.8 times the excess over 1 was subtracted for commercial bias, the larger of content ratio and integrity, and
+content safety, as in Source-1's overall score. Held-out set:
+
+| reading of propella-1 4B | chunks | propella-1 4B rank | Source-1 rank | Source-1 minus propella-1 4B: rank (95% interval) | AUC (95% interval) |
+|---|---|---|---|---|---|
+| main score, all 53 languages (the tables above) | 493 | 0.756 | 0.900 | +0.144 (+0.11 to +0.18) | +0.084 (+0.05 to +0.12) |
+| main score, its 43 listed languages | 459 | 0.766 | 0.899 | +0.133 (+0.10 to +0.17) | +0.080 (+0.05 to +0.12) |
+| with its own red-flag ratings as penalties, all 53 languages | 493 | 0.828 | 0.900 | +0.072 (+0.05 to +0.10) | +0.049 (+0.02 to +0.07) |
+| with its own red-flag ratings as penalties, its 43 listed languages | 459 | 0.835 | 0.899 | +0.064 (+0.04 to +0.09) | +0.048 (+0.03 to +0.07) |
+
+We tried four ways of weighting the penalties: the one above, boilerplate from content ratio alone, from the sum of
+content ratio and integrity, and no rescaling. Across them, on all 53 languages and on its 43 listed ones, the
+held-out rank gap ranges from 0.064 to 0.081. With the penalties as above, the exam gaps are +0.075 (+0.05 to +0.10)
+in English and +0.155 (+0.10 to +0.21) in the 12 languages. The lead holds under every reading we tried, but it
+roughly halves: the 0.144 in the main tables depends on reading propella-1 through its quality ratings alone.
+
+The exact mapping: each rating word is mapped to its position in the rating's scale, from 0: educational value (none,
+minimal, basic, moderate, high), reasoning (none, minimal, basic_reasoning, explanatory, analytical), content quality
+(unacceptable, poor, adequate, good, excellent), information density (empty, thin, moderate, adequate, dense). Main
+score = (0.30 x educational value + 0.20 x reasoning + 0.15 x content quality + 0.20 x information density) / 0.85, on
+that 0-4 scale. The penalized reading also maps commercial bias (none, minimal, moderate, heavy, pure_marketing),
+content ratio (complete_content, mostly_content, mixed_content, mostly_navigation, minimal_content) and content safety
+(safe, mild_concerns, nsfw, harmful, illegal) to 0-4 and content integrity (complete, mostly_complete, fragment,
+severely_degraded) to 0-3. It rescales each of them and the main score to 0-5, and subtracts 0.5 x max(0, commercial
+bias - 1) + 0.4 x max(0, max(content ratio, content integrity) - 1) + 0.8 x max(0, content safety - 1), clipped to
+0-5.
+
+#### propella-1's answers
+
+An answer that did not parse as JSON was run once more, with a limit of 1,536 new tokens instead of 512. On the
+evaluation chunks, the 4B was retried on 1 held-out chunk, and the 1.7B on 4 held-out chunks and 1 English exam chunk.
+After the retry every answer parsed, except that one 1.7B English exam answer: it ran to the token limit, and its
+ratings were read from the raw text. Seven answers used an educational-value word outside its scale (the 4B: 2
+held-out, 1 English exam and 2 12-language exam chunks; the 1.7B: 2 English exam chunks). Those chunks have no main
+score. This is why propella-1 4B is compared on 493 of the 495 held-out chunks and 350 of the 352 12-language chunks.
+
+</details>
+
+### Exam sets
+
+On both exams Source-1 and the teacher are level within noise (English +0.009, 95% interval -0.01 to +0.02;
+12 languages +0.015, -0.007 to +0.04).
+
+<details>
+<summary>Exam sets in detail: rank, AUC, whole documents and the drop line per document</summary>
+
+### Exam sets in detail
+
+| model | English: rank (413 chunks) | English: AUC | English: whole documents (rank) | 12 languages: rank (352 chunks) | 12 languages: AUC |
+|---|---|---|---|---|---|
+| **Source-1** | 0.921 | 0.979 | 0.920 | 0.895 | 0.961 |
+| Teacher | 0.912 | 0.965 | 0.901 | 0.880 | 0.945 |
+
+Whole documents: the token-weighted aggregation over each document's chunks, on the English exam's random-sample
+documents (196 for Source-1; 195 for the teacher, which has no score for one of them). Source-1 on the same 195 is also
+0.920.
+
+At the shipped drop line, counted per document (the teacher at its own keep flags):
+
+| exam documents | grader drops | Source-1: drops caught | Source-1: keep agreement | teacher: drops caught | teacher: keep agreement |
+|---|---|---|---|---|---|
+| English, all 332 | 24 | 14 (0.583) | 96.7% | 14 (0.583) | 96.1% |
+| English, the 196 random-sample documents | 13 | 9 (0.692) | 97.4% | 6 (0.462) | 94.9% |
+| 12 languages, all 352 (all random) | 46 | 31 (0.674) | 94.3% | 28 (0.609) | 93.5% |
+
+The exams were graded with an older revision of the rubric, before it settled how to score ads (`spam_seo` 3, kept).
+So part of the gap is rubric drift that affects the teacher and Source-1 alike.
+
+</details>
+
+### More results
+
+### The drop line
+
+<details>
+<summary>The drop line: how it was chosen, stricter lines, results on the held-out set</summary>
+
+
+`keep` is false when the scores cross the drop line stored in `calibration.json`:
+
+```
+drop if  toxicity >= 4  or  spam_seo >= 3.5  or  boilerplate >= 4.5
+```
+
+The rubric's own hard filters use `spam_seo >= 4`. Pages scored `spam_seo` 3, the level for ads and promotional pages,
+are kept by both lines. The line was chosen on the teacher's labels for the validation split (9,744 chunks) by a fixed
+rule: among five candidate lines, take the highest drop recall whose keep agreement with the teacher stays at or above
+0.95. On that split it agrees with the teacher's keep flags on 96.4% of chunks, catches 77.1% of the teacher's drops
+(803 of 1,042) and drops 9.4% of chunks. On the held-out test split (9,553 chunks, not used for the choice) it agrees
+on 96.7% and catches 80.2% (840 of 1,047). The candidate lines and the 0.95 floor were set with help from grades by
+the grader's model family ([Independence from development](#independence-from-development)).
+
+If you need to catch more low-quality text and can afford to lose more good text, pass a stricter line as
+`drop_line`. The candidates trade keep agreement for recall (validation split, against the teacher's keep flags):
+
+| drop line | keep agreement | teacher drops caught | wrong drops | share dropped |
+|---|---|---|---|---|
+| `toxicity >= 4 or spam_seo >= 4 or boilerplate >= 4.5` (rubric default) | 0.960 | 0.711 (741/1,042) | 89 | 8.5% |
+| **`toxicity >= 4 or spam_seo >= 3.5 or boilerplate >= 4.5` (shipped)** | **0.964** | **0.771 (803/1,042)** | **114** | **9.4%** |
+| `toxicity >= 4 or spam_seo >= 3 or boilerplate >= 4.5` | 0.929 | 0.830 (865/1,042) | 515 | 14.2% |
+| `toxicity >= 4 or spam_seo >= 3.5 or boilerplate >= 4` | 0.945 | 0.880 (917/1,042) | 408 | 13.6% |
+| `toxicity >= 4 or spam_seo >= 2.5 or boilerplate >= 4.5` | 0.856 | 0.872 (909/1,042) | 1,271 | 22.4% |
+
+`calibration.json` also stores one offset per quality score (mean teacher label minus mean model score on the
+validation split). They are small, between -0.016 and -0.003 points, and `source1.py` leaves them off unless you pass
+`apply_offsets=True`. The results in this file use the scores without offsets, as `source1.py` returns them by
+default, except the per-field bias table, which says where it applies them.
+
+On the held-out set, against the grader:
+
+| model | line | keep agreement | drop recall (caught / grader drops) | wrong drops | share dropped |
+|---|---|---|---|---|---|
+| **Source-1** | shipped line | 0.941 (0.917 to 0.959) | 0.672 (43/64; 0.550 to 0.774) | 8 | 10.3% |
+| Source-1 | rubric default line (`spam_seo >= 4`) | 0.931 | 0.594 (38/64) | 8 | 9.3% |
+| Teacher | its own keep flags | 0.943 | 0.719 (46/64) | 10 | 11.3% |
+
+17 of Source-1's 21 misses at the shipped line are also missed by the teacher at its own keep flags, and 16 of the 21
+are in multilingual web text. On the 29 Chinese chunks Source-1's line agrees with the grader on 89.7% and catches 11
+of 14 drops. The candidate lines were drafted after the held-out set's grades had been seen once, so treat these
+held-out drop-line numbers as slightly optimistic. The test-split numbers above do not have this problem.
+
+</details>
+
+<details>
+<summary>Per-field agreement and bias on the exam sets, and the red flags on the held-out set</summary>
+
+### Per-field agreement and bias
+
+Agreement per field on the exam sets. For the five quality scores: quadratic-weighted kappa on levels rounded to the
+nearest integer (English: the 200 random-sample chunks, 199 for the teacher; 12 languages: all 352 chunks). For the
+labels: unweighted Cohen's kappa (English: all 414 chunks, 413 for the teacher; 12 languages: all 352 chunks). A dash
+means the kappa is not meaningful. 351 of the 352 chunks in the 12-language exam are plain text by the grader's label,
+so content type has almost no variation there (Source-1 matches the grader on 350 of them, and its kappa is about 0).
+
+| field | English: Source-1 kappa | English: teacher kappa | 12 languages: Source-1 kappa | 12 languages: teacher kappa |
+|---|---|---|---|---|
+| educational_value | 0.831 | 0.816 | 0.788 | 0.803 |
+| reasoning_depth | 0.798 | 0.781 | 0.719 | 0.730 |
+| writing_quality | 0.769 | 0.811 | 0.785 | 0.786 |
+| information_density | 0.861 | 0.828 | 0.812 | 0.800 |
+| reliability | 0.739 | 0.790 | 0.735 | 0.743 |
+| format | 0.748 | 0.742 | 0.745 | 0.745 |
+| topic | 0.776 | 0.794 | 0.786 | 0.808 |
+| content_type | 0.833 | 0.864 | - | - |
+
+Bias is the mean of model minus grader on rounded levels, on the same chunks as the quality kappas. Source-1's values
+here include its optional calibration offsets (`apply_offsets=True`). Without them, as `source1.py` returns scores by
+default, they differ by at most 0.014 (English: `reasoning_depth` +0.390, `writing_quality` +0.325, `reliability`
++0.270). The teacher has no offsets.
+
+| field | English: Source-1 bias | English: teacher bias | 12 languages: Source-1 bias | 12 languages: teacher bias |
+|---|---|---|---|---|
+| educational_value | +0.280 | +0.307 | +0.111 | +0.142 |
+| reasoning_depth | +0.385 | +0.422 | +0.301 | +0.321 |
+| writing_quality | +0.320 | +0.281 | +0.131 | +0.153 |
+| information_density | -0.105 | -0.085 | -0.196 | -0.159 |
+| reliability | +0.260 | +0.261 | +0.196 | +0.196 |
+
+On the held-out set (quadratic-weighted kappa on rounded levels, all 495 chunks), the red flags reach 0.87 for
+`spam_seo`, 0.81 for `boilerplate` and 0.70 for `toxicity` (the teacher: 0.88, 0.81 and 0.73). The gated scores can
+only be compared on the few chunks where both the grader and the model give them: `math_quality` 0.64 on 8 chunks (the
+teacher 0.90) and `code_quality` 0.67 on 23 (the teacher 0.82 on 22). These numbers are very noisy.
+
+</details>
+
+### Reading the whole chunk
+
+<details>
+<summary>Reading the whole chunk: what the text past 512 tokens adds</summary>
+
+
+The same model was run with every input cut to its first 512 tokens at scoring time. It was trained on full chunks,
+so this measures what the text past 512 tokens adds, not how a model trained for 512 tokens would do. On the held-out
+set, 197 of the 495 chunks fit in 512 tokens and score identically both ways.
+
+| chunks | full chunk: rank | first 512 tokens: rank | difference (95% interval) |
+|---|---|---|---|
+| held-out set, all (495; 1,650 tokens on average) | 0.900 | 0.848 | +0.052 (+0.02 to +0.08) |
+| held-out set, 513 to 2,048 tokens (192) | 0.902 | 0.881 | +0.021 (-0.01 to +0.05) |
+| held-out set, over 2,048 tokens (106) | 0.875 | 0.753 | +0.122 (+0.05 to +0.21) |
+| English exam (414) | 0.921 | 0.845 | +0.075 (+0.05 to +0.11) |
+| 12-language exam (352) | 0.895 | 0.853 | +0.042 (+0.02 to +0.08) |
+
+The gain comes from the longer chunks. Finding the grader's drops barely changes (held-out AUC 0.946 against 0.943;
++0.003, -0.01 to +0.01). Only 74 held-out chunks are longer than 4,096 tokens, so this does not test the far end of
+the 8,192-token window.
+
+The 512-token limit of some public scorers does not explain their lower agreement on the held-out set. Cut to 512
+tokens, Source-1 still ranks it closer to the grader than propella-1 4B reading the whole chunk (0.848 against 0.756
+on its 493 chunks; +0.092, +0.05 to +0.14). It also ranks closer than each of the four scorers that read 512 tokens
+(leads +0.30 to +0.42, every interval clear of zero). On the English exam the cut model is only level with
+propella-1 4B (+0.026, -0.01 to +0.07).
+
+</details>
+
+<details>
+<summary>Agreement with the teacher on the test split: the job Source-1 was trained for</summary>
+
+### Agreement with the teacher on the test split
+
+How closely Source-1 reproduces the teacher's labels on 9,553 test chunks (8,473 documents, 53 languages) it never
+trained on: the job it was trained for. The grader results above instead measure agreement with an independent LLM
+grader applying the same rubric.
+
+| chunks | n | overall score rank agreement | keep/drop agreement (rubric default line) | label accuracy | quality score error (MAE, 0-5 scale) |
+|---|---|---|---|---|---|
+| all | 9,553 | 0.953 (0.950 to 0.955) | 0.963 | 0.907 | 0.254 |
+| web | 2,677 | 0.952 | 0.964 | 0.904 | 0.250 |
+| multilingual web | 5,344 | 0.955 | 0.960 | 0.912 | 0.245 |
+| conversations, code, synthetic | 950 | 0.895 | 0.964 | 0.890 | 0.325 |
+| open books | 582 | 0.844 | 0.985 | 0.901 | 0.248 |
+
+The keep/drop column applies the rubric's default hard filters to Source-1's scores. With the shipped drop line,
+agreement on this split is 0.967 (see [The drop line](#the-drop-line)). The interval on the overall rank agreement is
+a 95% bootstrap interval over the 8,473 documents (2,000 resamples, seed 0).
+
+Label accuracy is 0.877 for format, 0.853 for topic and 0.991 for content type. The gated scores agree least with the
+teacher: kappa 0.58 for `code_quality` (555 chunks where it applies) and 0.50 for `math_quality` (257 chunks).
+Agreement with the teacher is lowest for Gujarati (0.788), Georgian (0.867), Croatian (0.887), Malayalam (0.893),
+Bengali and Marathi (0.897) and Serbian (0.899).
+
+</details>
+
+<details>
+<summary>Speed, and how much scores move with precision and batching</summary>
+
+### Speed
+
+Measured with `source1.py` on one RTX 3090 in bf16 with the default batches, excluding load time, on the 495 held-out
+chunks and the 766 exam chunks. Repeated runs on the same GPU differed by up to about 10%. No speed comparison with
+the public scorers or the teacher was made under the same conditions, so none is claimed.
+
+| set | chunks | mean input tokens per chunk | chunks/s | input tokens/s | peak VRAM |
+|---|---|---|---|---|---|
+| held-out | 495 | 1,650 | 32.2 | 53,198 | 4.10 GB |
+| exam | 766 | 1,766 | 31.0 | 54,644 | 4.21 GB |
+
+Precision and batching: computing in bfloat16, both weight files give the same scores. On these 1,261 chunks, scoring
+each chunk alone instead of in the default batches moved `overall` by up to 0.04 and a single field by up to 0.10 (3
+labels changed, no keep decision). float32 differs from bfloat16 by a similar amount (up to 0.03 on `overall` and 0.09
+on a single field; 4 labels and 1 keep decision changed).
+
+</details>
+
+## Limitations
+
+- **Home ground.** It measures agreement with Source-1's rubric, as applied by graders from one proprietary model
+  family whose grades also steered development. It does not show how Source-1 does on text from other sources, or
+  that filtering with it trains better language models.
+- **The exams helped choose the teacher**, so they are not independent of the grader.
+- **It misses about a third of the grader's drops** at the shipped line: it catches 43 of 64 on the held-out set, the
+  teacher, at its own keep flags, 46.
+- **Like its teacher, it rates some qualities higher than the grader does.** On the English exam, `educational_value`
+  is 0.28 levels above the grader on average.
+- **Weaker on books and on conversations, code and synthetic text** (held-out rank 0.677 and 0.728, against about 0.90
+  for web text).
+- **One chunk of up to 8,192 tokens at a time.** Nothing outside a chunk is visible to it.
+- **It copies the teacher, biases included.** For example, it can score a thin affiliate page 3 (kept) where the
+  rubric says 4 (dropped).
+- **The gated scores are the least reliable fields, and `toxicity` is the weakest red flag.**
+- **Not a fact checker or a safety tool.**
+- **Less data for some languages.** The 16 smallest have 1,249 to 1,470 training chunks each.
+- **License screening has limits.** Notices the patterns miss, and opt-outs outside the text, were not caught.
+- **No reproduction kit.** The numbers cannot be recomputed from this repository alone
+  ([what is included](#reproducing-the-evaluation)).
+
+### Limitations in detail
+
+<details>
+<summary>The full text of each limitation</summary>
+
+- **Home-ground evaluation.** The benchmark measures agreement with Source-1's rubric as applied by independent LLM
+  graders from one proprietary model family (one graded the exams, another the held-out set). Grades from that family,
+  among other proprietary LLM graders, also steered the rubric revisions, the choice of the teacher and its prompt
+  setup, and the drop-line candidates and floor ([Independence from development](#independence-from-development)).
+  The held-out documents come from the same kinds of sources as the training data. The public scorers were built for
+  other definitions of quality, read the text without Source-1's header and are each read through one main score. A
+  more generous reading of propella-1 halves its gap to Source-1 ([How propella-1 is read](#how-propella-1-is-read)).
+  The comparison does not show how Source-1 does on text from other sources, or that filtering with Source-1 trains
+  better language models; neither has been tested.
+- **The exams are not independent of the grader.** They are the samples on which the teacher and its prompt setup were
+  chosen against the exam grader's labels. The held-out set, which played no part in that choice, is the main result.
+- **It misses about a third of the chunks the grader drops at the shipped line.** On the held-out set the shipped line
+  catches 67.2% of the grader's drops (43/64; interval 55.0% to 77.4%); the teacher catches 71.9% (46/64). 17 of
+  Source-1's 21 misses are also missed by the teacher, so most of what Source-1 misses its teacher misses too. Most
+  misses are in multilingual web text (28 of 44 caught there). If recall matters more than keeping good text, use a
+  stricter line ([The drop line](#the-drop-line)) or rank on `overall` and cut lower.
+- **Like its teacher, it rates some qualities higher than the grader does.** On the English exam's 200 random-sample
+  chunks, Source-1's `educational_value` is on average 0.28 levels above the grader's (the teacher 0.31). Its
+  `reasoning_depth`, `writing_quality` and `reliability` are 0.26 to 0.39 levels above (rounded levels, with or without
+  the optional calibration offsets; the teacher: 0.26 to 0.42). For `writing_quality` its bias is larger than the
+  teacher's (+0.32 to +0.325 against +0.28). In the 12 languages they are smaller (`reasoning_depth` +0.30,
+  `reliability` +0.20, `writing_quality` +0.13 to +0.14, `educational_value` +0.11; the teacher +0.32, +0.20, +0.15
+  and +0.14). See [Per-field agreement and bias](#per-field-agreement-and-bias).
+- **Weaker on books and on conversations, code and synthetic text.** Held-out rank is 0.677 for open books and 0.728
+  for conversations/code/synthetic (the teacher: 0.741 and 0.821; 42 and 48 chunks), against about 0.90 for web text.
+  Against the teacher on the test split it is 0.844 and 0.895. Book labels are nearly constant (long, formal, almost
+  always kept), which leaves little signal to learn from.
+- **8,192 tokens per chunk.** Longer documents are split and each chunk is judged on its own. Nothing outside a chunk
+  is visible to it, except the "Part i of n" header. Text in scripts that need many tokens per character fills the
+  window sooner: in training, 10.5% of Bengali chunks, 7.2% of Georgian, 6.3% of Arabic and 5.0% of Korean chunks
+  were longer than 8,192 tokens and were truncated. When you score with `source1.py`, a document longer than the
+  window is split into chunks rather than cut, so all of its text is read (the `truncated` field reports the rare
+  chunk that still had to be cut). `max_chunks` trades that for speed by scoring only some evenly spaced chunks.
+- **It copies the teacher, biases included.** The rubric puts ads, company pages and product pages at `spam_seo` 3,
+  which the shipped line keeps, and thin affiliate and doorway pages at 4, which it drops. The teacher does not always
+  follow the second rule and sometimes scores such pages 3, and Source-1 learned from those labels.
+- **The gated scores are the least reliable fields, and toxicity is the weakest red flag.** Against the teacher on
+  the test split, kappa is 0.58 for `code_quality` and 0.50 for `math_quality`. Against the grader on the held-out set
+  they can be compared only on 8 and 22 to 23 chunks ([Per-field agreement and bias](#per-field-agreement-and-bias)).
+  `toxicity` reaches 0.70 against the grader (the teacher 0.73).
+- **Not a fact checker or a safety tool.** `reliability` is a surface judgment of care and plausibility; the model does
+  not verify claims. Toxic text is rare in the training data. `toxicity` is meant as a data-filtering red flag, not a
+  moderation classifier.
+- **Languages with little data.** The 16 languages with the fewest training chunks (az, et, fil, gu, ka, kk, kn, lv,
+  ml, mr, ms, sq, sw, ta, te, ur) have 1,249 to 1,470 each, and Kannada had no books. Agreement with the teacher on the
+  test split is lowest for Gujarati (0.788), Georgian (0.867), Croatian (0.887), Malayalam (0.893), Bengali and Marathi
+  (0.897) and Serbian (0.899). The held-out set has 1 to 12 chunks for most non-English languages, so per-language
+  results there are noisy.
+- **License screening has limits.** Licenses come from each source's metadata. The training documents were also
+  screened by pattern matching on their own text: books for NonCommercial, NoDerivatives and all-rights-reserved
+  notices in their front and back matter, web pages for such terms and for the sites they come from, and every
+  document for text-and-data-mining and AI-training reservations. This screening did not catch notices worded in ways
+  the patterns miss, or reservations made outside the text itself (on the terms pages of sites the rules do not list,
+  or in machine-readable opt-out signals such as robots.txt). If you find such a document, tell us (see the contact
+  section of [README.md](README.md#contact-and-takedown)).
+- **No reproduction kit.** See [Reproducing the evaluation](#reproducing-the-evaluation).
+
+</details>
+
+## Reproducing and appendices
+
+<details>
+<summary>Reproducing the evaluation: what this repository does and does not include</summary>
+
+### Reproducing the evaluation
+
+What this repository gives you:
+
+- Source-1 itself: the weights, `source1.py` and `calibration.json`, which produce the Source-1 scores used here
+  (computed in bfloat16 on one RTX 3090 with the default batches; see [Speed](#speed) for how much scores move with
+  precision and batching).
+- The metric definitions and bootstrap settings ([Metric definitions](#metric-definitions)).
+- How each public scorer was run ([How the public scorers were run](#how-the-public-scorers-were-run)) and read
+  ([How each public scorer is read](#how-each-public-scorer-is-read)), with the exact definitions of the scores we
+  defined ourselves ([How propella-1 is read](#how-propella-1-is-read),
+  [How EAI-Distill and the Nemotron-CC ensemble are read](#how-eai-distill-and-the-nemotron-cc-ensemble-are-read)).
+- The Hugging Face repositories and revisions of the 16 public scorers
+  ([Appendix C](#appendix-c-public-scorer-repositories)).
+
+What it does not include: the ids and texts of the evaluation chunks, the grader's labels, any model's per-chunk
+scores, the script that computes the metrics, and the teacher's prompt and decoding settings. The numbers in this file
+cannot be recomputed from this repository alone.
+
+</details>
+
+### Appendix A: rubric anchors
+
+<details>
+<summary>Appendix A: rubric anchors, and the teacher's extra rules</summary>
+
+
+| field | 0 | 1 | 2 | 3 | 4 | 5 |
+|---|---|---|---|---|---|---|
+| educational_value | Teaches nothing: spam, ads, navigation, gibberish | Almost nothing to learn: a few incidental facts in promotional, personal or trivial text | Some useful information, but superficial, fragmentary, or mixed with irrelevant material | Useful and coherent; real knowledge or skills, without much depth or completeness | Clearly educational; explains concepts or methods well enough to learn from, minor gaps | Outstanding teaching material, comparable to an excellent textbook or expert tutorial |
+| reasoning_depth | No reasoning: fragments, lists, boilerplate | Bare assertions or opinions | Occasional explanation, mostly unsupported; steps skipped | Explains the why behind key points, some step-by-step structure | Consistent explicit reasoning: derivations, cause and effect, worked examples | Rigorous multi-step reasoning throughout: proofs, careful derivations, thorough analysis |
+| writing_quality | Unreadable: garbled, broken encoding, keyword soup | Very poor: frequent errors, incoherent | Below average: understandable but disorganized or repetitive | Adequate: clear and coherent, minor issues | Good: well organized, fluent, precise | Excellent: publication quality |
+| information_density | No real content | Mostly padding around a little content | Noticeable padding or digressions | Reasonable: mostly on point, some filler | Dense: most sentences carry information | Very dense yet readable |
+| reliability | Fabricated, nonsensical or deceptive | Largely unreliable: many errors, sensational claims | Questionable: some errors or unsupported claims | Generally plausible and consistent, informal or unverifiable | Careful and accurate; shows its work or cites sources | Authoritative: expert-level accuracy, well sourced |
+| spam_seo | None | Minor promotion: a call to action or a brief ad in otherwise genuine content | Noticeable promotion: repeated calls to action, affiliate links, marketing tone | Substantial: the text mainly exists to promote, sell or rank; visible keyword repetition | Mostly spam: keyword stuffing, clickbait, thin affiliate or doorway content | Pure spam: auto-generated SEO text, scams, keyword lists |
+| boilerplate | None: all real content | A little: a stray header, footer or copyright line | Noticeable: roughly 10-25% navigation, cookie or legal text | Substantial: roughly 25-50% templates, menus, link lists | Mostly boilerplate: over half templated or navigational | Entirely boilerplate: auto-generated pages, link lists, cookie banners, error or index pages |
+| toxicity | None | Mild: occasional profanity or rudeness | Moderate: insults, crude humor, mild sexual references | Significant: harassment, demeaning stereotypes, graphic violence, partly explicit | Severe: hate speech, targeted harassment, sexually explicit as the main content | Extreme: violent extremism, dehumanizing hate, incitement, sexual content involving minors |
+| code_quality | Not usable code: garbled, minified, obfuscated | Very poor: likely non-functional fragments, no structure, or auto-generated boilerplate | Poor: may work but messy | Acceptable: readable, plausibly correct, minimal docs | Good: clean, idiomatic, documented | Excellent: exemplary, production quality, instructive |
+| math_quality | Garbled math | Mostly wrong or incoherent | Some correct math, but errors or skipped steps | Generally correct, key steps shown | Correct, clean notation, complete steps | Rigorous and elegant, every step justified |
+
+The rubric anchors and the head layout are in `source1.json`. The teacher's prompt also had a few special rules that
+are not in `source1.json`, and Source-1 was trained on labels that follow them, as far as the teacher did:
+
+- Pages whose main purpose is to promote or sell a business, product or service (company "about us" pages, product
+  and landing pages, shop listings, brochures) are ads: format `product_page` and `spam_seo` 3, even when cleanly
+  written. Self-promotional press releases stay `news` with `spam_seo` 3. Selling alone is never a reason for
+  `spam_seo` 4 or 5; those levels are for keyword-stuffed text, doorway or thin affiliate pages made to rank, and
+  scams. Independent reviews, comparisons and news about products are not ads.
+- Pages stitched together from unrelated or scrambled text (often a keyword title over copied or shuffled
+  paragraphs) are `spam_seo` 4-5, with `reliability` and `writing_quality` 0-1.
+- Tag, category, archive and search-result pages, feeds, link directories and other index pages that mostly list
+  other pages are `boilerplate` 4, and 5 when the list is all they contain; empty auto-generated stub pages are 4-5.
+- Sexually explicit material as the main content is `toxicity` 4 in any language (5 if it involves minors).
+- General rules: judge only the text shown (a part of a longer document is not penalized for starting or ending
+  mid-thought); ignore personal-data placeholders such as `<EMAIL>`; judge every language by its own standards;
+  poor machine translation lowers `writing_quality`, and machine-translated filler written to rank is spam.
+
+</details>
+
+### Appendix B: training in detail
+
+<details>
+<summary>Appendix B: training in detail (model, data, filtering, labels, recipe)</summary>
+
+
+#### Model
+
+- Backbone: [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) (ModernBERT architecture, 22 layers, hidden
+  size 768, 8,192-token context; trained by its authors on 3T+ tokens across 1800+ languages). All backbone weights
+  were fine-tuned.
+- Heads: 13 linear heads on the mean-pooled final hidden states (about 68k parameters): one softmax head per label
+  (10, 15 and 4 classes) and one six-level softmax head per 0-5 field.
+- Total: 307M parameters. Trained with float32 weights in bfloat16 mixed precision; released in bfloat16 (default)
+  and float32.
+
+#### Data
+
+220,346 chunks were labeled. After license filtering, the safety filter and the held-out splits, and after setting
+aside a reserve that was never trained on (about 1% of training documents, picked by a hash of the document id and
+held back for label-quality checks, plus the training and validation documents reviewed during development; see
+[Independence from development](#independence-from-development)), 172,895 chunks (350M tokens, from 151,281 documents)
+were used for training, 9,744 for validation and 9,553 for testing. No label-quality result from the reserve is
+reported here. Documents were split 90/5/5 by a hash of the document id, so no document spans two splits.
+
+| stage | what it is | training chunks |
+|---|---|---|
+| Web | A stratified sample of filtered and unfiltered web text, PDFs, wikis, math pages, permissively licensed code, Common Pile sources and toxicity datasets, in 53 languages; low-quality pages included on purpose | 46,389 |
+| Multilingual web | A larger sample of the same kinds of sources, weighted toward languages other than English | 96,778 |
+| Conversations, code, synthetic | Chat and instruction data, permissively licensed code and commits, synthetic and machine-generated text, comments and other short or noisy text, and domain text (law, parliament proceedings, science articles, historical and OCR text) | 17,614 |
+| Open books | Books recorded as openly licensed or public domain, from Project Gutenberg, HAL, Wikibooks, Wikisource, OpenStax, the World Bank, EU and FAO publications, DOAB, OAPEN and others, after removing books whose own text states stricter terms (see below) | 12,114 |
+| **Total** | 53 languages; English is 38.9% of training chunks | **172,895** |
+
+Languages: ar, az, bg, bn, ca, cs, da, de, el, en, es, et, fa, fi, fil, fr, gu, he, hi, hr, hu, id, it, ja, ka, kk,
+kn, ko, lt, lv, ml, mr, ms, nl, no, pl, pt, ro, ru, sk, sl, sq, sr, sv, sw, ta, te, th, tr, uk, ur, vi, zh. Every
+non-English language has 1,249 to 3,437 training chunks; the 16 languages with the fewest are listed under
+[Limitations in detail](#limitations-in-detail).
+
+To replace documents that the license filtering below removed, 18,113 of the training and validation chunks were
+drawn by fixed sampling rules (no model chose documents) from the same kinds of open sources: FineWeb-2 (including its
+removed-documents part), the FineWeb-Edu annotations, C4, HPLT, FinePDFs, FineMath, permissively licensed code, Common
+Pile and Common Corpus documents, and open books. Every rule below was applied to them too.
+
+Filtering before training (training and validation splits; the held-out test split keeps every document the license
+filtering removed, for evaluation only):
+
+- License filtering removed 21,576 chunks from 19,720 documents:
+  - sources whose terms do not clearly cover this use: DCLM-baseline (its dataset card states that it is intended
+    for research use), raw Common Crawl WET pages (no dataset license), Stack v2 Edu (its upstream terms are gated),
+    Common Pile's YouTube transcripts (licenses asserted by uploaders over broadcasts), two collections with
+    unreliable license metadata, a corpus of third-party social media posts and a toxicity corpus whose texts are
+    not covered by its stated license, and French public data under the Licence Ouverte;
+  - code: copyleft licenses (GPL, AGPL, LGPL, MPL, EPL), code outside a permissive allow-list (MIT, Apache-2.0, BSD,
+    ISC, CC0, Unlicense), and code files whose own header states copyleft, proprietary or NonCommercial terms;
+  - books whose own front or back matter states stricter terms than the open license their platform recorded: a
+    scan of all 4,170 book documents found 137 that state NonCommercial or NoDerivatives terms or reserve all rights
+    with no open grant, and a wider pass over the same pages found 28 that state such terms in other wordings, forbid
+    sale or contradict their license record; also books deposited in HAL whose own text states no open license (84)
+    and library books from the Norwegian Colossal Corpus published after 1955 (14);
+  - other documents whose own text carries such notices (NonCommercial, NoDerivatives or all-rights-reserved
+    statements, publishers' copyright notices, text reprinted with permission): 141; web pages under NonCommercial or
+    NoDerivatives terms, or from sites whose terms put all their content under such terms (302); pages from sites
+    that re-host other people's documents, homework, shadow-library, pirated-novel, lyrics and subtitle sites (529);
+    and a few smaller groups (pages offering software cracks, open-education pages with no stated license,
+    GFDL-only pages, papers marked closed-access);
+  - documents whose own text reserves text-and-data-mining or AI-training rights: a scan of all 192,905 input
+    documents found 16.
+- Separately, a pre-specified safety filter removed documents from every split, and a rule fixed in advance also
+  removed documents that substantially copy text the safety filter removed (every split).
+- Book pages that are mostly a table of contents were left out of training (99 chunks).
+- Every exam document was checked against the training data (exact text, URL, title, long-line and shingle matching);
+  no copies were found.
+
+#### Labels
+
+- Every training label comes from one teacher: an open-weight 27B LLM scoring each chunk against the 13-field rubric,
+  self-hosted on our own and rented GPUs. No human labels were used.
+- No output of a proprietary model was used as a label or a training target. The evaluation grader's outputs were
+  never trained on. What grades from proprietary LLMs did inform is listed under
+  [Independence from development](#independence-from-development).
+- About 2% of training documents (3,387) come from public datasets of model-written text (synthetic textbooks, chat
+  logs, machine-generated-text detection sets, machine translations). They are there so the scorer learns to judge
+  such text; the teacher scored them like any other input.
+
+#### Recipe
+
+| setting | value |
+|---|---|
+| epochs | 2 (5,320 optimizer steps) |
+| tokens per step | 131,072 (about 65 chunks) |
+| max length | 8,192 tokens; 964 training chunks (0.6%) were longer and were truncated |
+| optimizer | AdamW, betas 0.9 / 0.98, eps 1e-6, weight decay 0.01, gradient clipping 1.0 |
+| learning rate | 5e-5 for the backbone, 10x for the heads; 5% warmup, cosine decay to 10% |
+| other | mean pooling, dropout 0.1, bf16 mixed precision, runs of spaces and tabs collapsed to one space before tokenizing (newlines kept), natural language mix (no reweighting), seed 0 |
+| checkpoint | the final step, which had the best validation overall Spearman against the teacher (0.955) |
+| how the settings were chosen | learning rate and language mix: a half-epoch sweep (two learning rates, two language mixes) on a smaller training set labeled by the same teacher, read on validation agreement with the teacher; epochs: the same validation curves (a third epoch added little while validation loss rose) |
+| hardware | one rented NVIDIA H100 NVL, about 1.9 hours |
+
+</details>
+
+<details>
+<summary>Appendix C: Hugging Face repositories and revisions of the public scorers</summary>
+
+### Appendix C: public scorer repositories
+
+| public scorer | Hugging Face repository | revision |
+|---|---|---|
+| propella-1 4B | `ellamind/propella-1-4b` | `bf607e62b6afa3e0e8d71c4d08d1429d9a09c82f` |
+| propella-1 1.7B | `ellamind/propella-1-1.7b` | `2cb58fd324fce70e1cb106df20bf4e1d79696021` |
+| JQL-Edu | `JQL-AI/JQL-Edu-Heads` (heads) and the embedding model its card names as the backbone (Snowflake's arctic-embed-m, version 2.0) | `5cb4a2d26c7961950b0facd1d8a390374027b7e4` (heads) and `95c2741480856aa9666782eb4afe11959938017f` (backbone) |
+| FinePDFs-Edu | `HuggingFaceFW/finepdfs_edu_classifier_<code>`, one model per language (list below) | per model |
+| FineWeb2-HQ | `epfml/FineWeb-HQ-Classifiers` (heads) and `FacebookAI/xlm-roberta-base` (backbone) | `1940ba2308cf2b12e530690c1eef183985dfcf29` and `e73636d4f797dec63c3081bb6ed5c7b0bb3f2089` |
+| FineWeb-Edu classifier | `HuggingFaceFW/fineweb-edu-classifier` | `284663cbb2dabf9bda30d8f8cc49601251ee1631` |
+| DCLM fastText (OH+ELI5) | `mlfoundations/fasttext-oh-eli5` | `cd8b714a90f2dbcd3b02cf5fc972e5d7c7f4f107` |
+| NeMo Curator edu (Nemotron-4 labels) | `nvidia/nemocurator-fineweb-nemotron-4-edu-classifier` | `842316292abe5bc78521758f5498d6a05adc0f8b` |
+| NeMo Curator edu (Mixtral labels) | `nvidia/nemocurator-fineweb-mixtral-edu-classifier` | `768fe255b7e7fbe222014e84cc6576a565516523` |
+| Meta-rater reasoning | `opendatalab/meta-rater-reasoning-rating` | `0072a9a83971eb4af6d689dfc64f8f203c45b398` |
+| Meta-rater readability | `opendatalab/meta-rater-readability-rating` | `5bfbee1110869ddcbf23447354a7311374784952` |
+| Meta-rater cleanliness | `opendatalab/meta-rater-cleanliness-rating` | `4403a9535d47cbc7cc99de26b25099335fe2d9b6` |
+| Meta-rater professionalism | `opendatalab/meta-rater-professionalism-rating` | `fc91d4be35fc91de3c65654bb59655ec533a1f61` |
+| EAI-Distill 0.5B | `EssentialAI/eai-distill-0.5b` | `39f51ea6e8f1e959961feea0403c69ecfcc8b342` |
+| NVIDIA quality classifier (DeBERTa) | `nvidia/quality-classifier-deberta` | `401824e175e89d3243bc376dc4ba262516615d81` |
+| Dolma 3 fastText quality | `allenai/dolma3-fasttext-quality-classifier` | `bb89085994fef638ca8dc2ca25169db328e314bb` |
+
+FinePDFs-Edu models used on the evaluation sets (`unknown` is its fallback model, used for fil, kn, ml, sw and te):
+
+| repository | revision |
+|---|---|
+| `HuggingFaceFW/finepdfs_edu_classifier_als_Latn` | `8f538c2701074964af4941048ec66077b1b6ca1f` |
+| `HuggingFaceFW/finepdfs_edu_classifier_arb_Arab` | `78462a34a522fbda15ad583ffa1cd98781571749` |
+| `HuggingFaceFW/finepdfs_edu_classifier_azj_Latn` | `6860d1ada3da270bce499e82a8177bfc895a87b0` |
+| `HuggingFaceFW/finepdfs_edu_classifier_ben_Beng` | `ca2a231ad78dc1948926cc5aa497240d95eeab40` |
+| `HuggingFaceFW/finepdfs_edu_classifier_bul_Cyrl` | `a23563de023ccabecf4c1e0d2210fe3588e1c381` |
+| `HuggingFaceFW/finepdfs_edu_classifier_cat_Latn` | `95c70a102e3862dc8708fe7b9e6bde361ed0643a` |
+| `HuggingFaceFW/finepdfs_edu_classifier_ces_Latn` | `43c57ff228771a55c4f496a1a680a1a7942463e7` |
+| `HuggingFaceFW/finepdfs_edu_classifier_cmn_Hani` | `b1157788380a284bac35fa96fb19654219f4f9b8` |
+| `HuggingFaceFW/finepdfs_edu_classifier_dan_Latn` | `c3746210c23a292dc10c74a338addadb80c11d3c` |
+| `HuggingFaceFW/finepdfs_edu_classifier_deu_Latn` | `eb2176fc3386be57b525a99fdee295b0580d1307` |
+| `HuggingFaceFW/finepdfs_edu_classifier_ekk_Latn` | `5b66ac177115e31f9b304c56408a18dbadde838c` |
+| `HuggingFaceFW/finepdfs_edu_classifier_ell_Grek` | `248951027a7d5e7853969863ef9f7628d379271b` |
+| `HuggingFaceFW/finepdfs_edu_classifier_fas_Arab` | `e3d91254e276f6fd6415c2aa19705441b064bf92` |
+| `HuggingFaceFW/finepdfs_edu_classifier_fin_Latn` | `b6925f941773d6a0716d8b3da09fed3130af16d6` |
+| `HuggingFaceFW/finepdfs_edu_classifier_fra_Latn` | `f5050a44f837329386ec89e9c8bb4380aa8765b4` |
+| `HuggingFaceFW/finepdfs_edu_classifier_guj_Gujr` | `72c1af11acd9085893fb1a6ae83c33ca49eaedaf` |
+| `HuggingFaceFW/finepdfs_edu_classifier_heb_Hebr` | `95d6c2d065d0f6943d607d5b7eb192ed7efb5bf3` |
+| `HuggingFaceFW/finepdfs_edu_classifier_hin_Deva` | `dfae45d02aa92adf72842e156d78a107e4f8a82d` |
+| `HuggingFaceFW/finepdfs_edu_classifier_hrv_Latn` | `ecd3cb19a72491f32f5ce628f1a3b8cf8b9c0be0` |
+| `HuggingFaceFW/finepdfs_edu_classifier_hun_Latn` | `078f8963005ccb83d24a444c8a87b0cf72443e74` |
+| `HuggingFaceFW/finepdfs_edu_classifier_ind_Latn` | `ee3e75ff7ddc4c20eea2bf524b6a77b1d224f786` |
+| `HuggingFaceFW/finepdfs_edu_classifier_ita_Latn` | `b67b3258ab616e68f2c1b61167e2d9b0673e95b1` |
+| `HuggingFaceFW/finepdfs_edu_classifier_jpn_Jpan` | `3478261183e28a6214b82b85bfa47adc2b4a503f` |
+| `HuggingFaceFW/finepdfs_edu_classifier_kat_Geor` | `5bf4a56cfa9249479098ce4a250fabb008c1278e` |
+| `HuggingFaceFW/finepdfs_edu_classifier_kaz_Cyrl` | `f88ddf006ec15795340263cdc1de07c4d8e1a7d6` |
+| `HuggingFaceFW/finepdfs_edu_classifier_kor_Hang` | `2baea20aa8f6640bd61ed879ba528292335f34ba` |
+| `HuggingFaceFW/finepdfs_edu_classifier_lit_Latn` | `d4b7281f2b258c2a8057e42c949ab7fbe196c242` |
+| `HuggingFaceFW/finepdfs_edu_classifier_lvs_Latn` | `0e1693b8ea51c6dfe76f8116604fc29ccf2119ce` |
+| `HuggingFaceFW/finepdfs_edu_classifier_mar_Deva` | `aae791cf53a4959b66ee67f26acc5479aa38d8e5` |
+| `HuggingFaceFW/finepdfs_edu_classifier_nld_Latn` | `b307a64f31a3c409d56c450f4f928aad596818e9` |
+| `HuggingFaceFW/finepdfs_edu_classifier_nob_Latn` | `34166e84a7fb08a905774c637b6ef2eb7b19cc1d` |
+| `HuggingFaceFW/finepdfs_edu_classifier_pol_Latn` | `58ff15760fa995fb7bea2c33a0761af1f9ce66a9` |
+| `HuggingFaceFW/finepdfs_edu_classifier_por_Latn` | `d11bd310217f4cdb27522eaadc36202d2df705d4` |
+| `HuggingFaceFW/finepdfs_edu_classifier_ron_Latn` | `92abfcae91841b726cc3ab71c3122cbfc77eb7b9` |
+| `HuggingFaceFW/finepdfs_edu_classifier_rus_Cyrl` | `23ba4c39b4565af85282c1f1d1bc8479fdaa482d` |
+| `HuggingFaceFW/finepdfs_edu_classifier_slk_Latn` | `7c1f7ec820a2d3f6eed0ede492d0417973dedb03` |
+| `HuggingFaceFW/finepdfs_edu_classifier_slv_Latn` | `8825bc094303a48239f3b0c40a023689ae5a6c14` |
+| `HuggingFaceFW/finepdfs_edu_classifier_spa_Latn` | `60eb17b37f8ea80fff614b50424359e974a43736` |
+| `HuggingFaceFW/finepdfs_edu_classifier_srp_Cyrl` | `393b63976a35e266b21b14d91aea89990b4cf8cc` |
+| `HuggingFaceFW/finepdfs_edu_classifier_swe_Latn` | `3da33c8970f10076e9da02649f51b10d359b2200` |
+| `HuggingFaceFW/finepdfs_edu_classifier_tam_Taml` | `68239bbdb85ab737aaed970d45d313af9f18f051` |
+| `HuggingFaceFW/finepdfs_edu_classifier_tha_Thai` | `db02cb1431acfb6baa956e8e09379f20ffd95980` |
+| `HuggingFaceFW/finepdfs_edu_classifier_tur_Latn` | `dcdccca95c802edf5f1454ad33620e7342261da2` |
+| `HuggingFaceFW/finepdfs_edu_classifier_ukr_Cyrl` | `1353ea90e4f65f8b33dce0570402f8692c982768` |
+| `HuggingFaceFW/finepdfs_edu_classifier_unknown` | `d61616d51ece5ce2159936d8ece8aa39a6ef68bc` |
+| `HuggingFaceFW/finepdfs_edu_classifier_urd_Arab` | `c2019778c7413f5a299d2ab4978acfadcdbe2030` |
+| `HuggingFaceFW/finepdfs_edu_classifier_v2_eng_Latn` | `90ddef285f67230389057c14b2f6bbfeb70d40ea` |
+| `HuggingFaceFW/finepdfs_edu_classifier_vie_Latn` | `870370fb168cc1c76549938b13f9cff953def4b7` |
+| `HuggingFaceFW/finepdfs_edu_classifier_zsm_Latn` | `5ad2ae90901c74585f0f921ab84fac0a52e3cbbe` |
+
+</details>

LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

NOTICE

@@ -0,0 +1,801 @@
+Source-1
+Copyright 2026 The Source-1 Authors
+
+Licensed under the Apache License, Version 2.0 (see the LICENSE file, or
+https://www.apache.org/licenses/LICENSE-2.0). The Source-1 Authors are listed in
+the AUTHORS file.
+
+This NOTICE has four parts and one appendix:
+  1. Base model: mmBERT-base (MIT License)
+  2. How the training scores were made
+  3. Data credits
+  4. Contact and removal requests
+  Appendix A. World Bank works (CC BY 3.0 IGO)
+
+
+================================================================================
+1. BASE MODEL: mmBERT-base (MIT License)
+================================================================================
+
+Source-1 is fine-tuned from mmBERT-base (https://huggingface.co/jhu-clsp/mmBERT-base),
+published by jhu-clsp (the mmBERT authors at Johns Hopkins University CLSP) under the
+MIT License. mmBERT is described in Marone et al., "mmBERT: A Modern Multilingual
+Encoder with Annealed Language Learning", arXiv:2509.06888. The upstream repository
+publishes no copyright line, so none is reproduced here.
+
+Changes made by the Source-1 Authors (2026-10-03): all encoder weights were further
+trained, and scoring heads (heads.safetensors) were added. The encoder weights are
+published in bfloat16 (model.safetensors, the default) and in float32
+(model.fp32.safetensors). The tokenizer is mmBERT's.
+
+mmBERT's tokenizer is based on the Gemma 2 tokenizer by Google. See the Gemma Terms of
+Use: https://ai.google.dev/gemma/terms
+
+MIT License (mmBERT-base):
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+================================================================================
+2. HOW THE TRAINING SCORES WERE MADE
+================================================================================
+
+Source-1 was fine-tuned on scores produced by an open-weight 27B LLM teacher scoring a
+13-field rubric (3 labels, 5 quality scores, 3 red flags and 2 gated scores). Every
+training target is a teacher score; no human labels were used. The teacher's weights
+are not part of this release.
+
+Source-1 was evaluated with an independent proprietary LLM grader, used for evaluation
+only: the grader's outputs were never used as training targets or training data, and
+the shipped drop line was chosen by a fixed rule on the teacher's labels. Grades and
+reviews by proprietary LLMs from the grader's model family did inform some design
+choices (the rubric revision, the choice of teacher and its prompt on the exam sets,
+the candidate drop lines and which collections were filtered out); see the
+development note in README.md and "Independence from development" in EVALUATION.md.
+
+
+================================================================================
+3. DATA CREDITS
+================================================================================
+
+This part covers the documents in Source-1's training and validation splits: 159,829
+documents (182,639 text chunks). The documents were inputs to be scored; their scores
+came from the teacher described in part 2. No training text is distributed with
+Source-1: the release contains model weights and code only.
+
+The credits below follow what each license or publisher asks for. Every work listed
+under a Creative Commons license was modified: it was used to train a classifier.
+Credit for each individual work belongs to the authors and publishers named in it.
+Naming a source does not suggest that its authors, publishers or licensors endorse
+Source-1 or any use of it.
+
+Each book is credited under the licence recorded for it by the platform it came from,
+except three works whose own text states an IGO licence (see 3.8). Before training,
+rules applied to each source's terms and to each document's own text kept 19,720
+documents out of the training and validation splits. Source-1 was not trained on them,
+and they are not credited here. They include: sources whose terms limit use to research
+or that carry no dataset licence (DCLM-baseline, raw Common Crawl WET pages); Stack v2
+Edu, whose upstream terms are gated; YouTube transcripts and a corpus of third-party
+social media posts; French public data under the Licence Ouverte; books and other
+documents whose own text states NonCommercial (NC) or NoDerivatives (ND) terms,
+reserves all rights or forbids reproduction without permission, or reserves
+text-and-data-mining or AI-training rights; books deposited in HAL whose own text
+states no open licence; web pages under NC or ND terms, and pages on sites that re-host
+other people's documents; and code whose licence or file header is copyleft,
+proprietary or not on a permissive allow-list.
+
+Every book that is not public domain or CC0 is also credited individually in the file
+CREDITS_BOOKS.tsv, which is part of this notice: title, authors, language, licence and
+source URL. Where a work's own text asks to be cited or attributed in a particular way,
+the file gives that citation or attribution: FAO's "Required citation", OpenStax's
+attribution request, the citations requested by Eurydice, JRC and other EU reports, and
+those of some other books and reports (university presses, Frontiers ebooks, research and
+project reports). The World Bank works are credited in Appendix A.
+
+License links used below:
+  ODC-By 1.0       https://opendatacommons.org/licenses/by/1-0/
+  CC BY 4.0        https://creativecommons.org/licenses/by/4.0/
+  CC BY 3.0        https://creativecommons.org/licenses/by/3.0/
+  CC BY 2.5        https://creativecommons.org/licenses/by/2.5/
+  CC BY 2.0        https://creativecommons.org/licenses/by/2.0/
+  CC BY 3.0 IGO    https://creativecommons.org/licenses/by/3.0/igo/
+  CC BY-SA 3.0 IGO https://creativecommons.org/licenses/by-sa/3.0/igo/
+  CC BY-SA 4.0     https://creativecommons.org/licenses/by-sa/4.0/
+  CC BY-SA 3.0     https://creativecommons.org/licenses/by-sa/3.0/
+  CC BY-SA 2.5     https://creativecommons.org/licenses/by-sa/2.5/
+  CC0 1.0          https://creativecommons.org/publicdomain/zero/1.0/
+  Apache-2.0       https://www.apache.org/licenses/LICENSE-2.0
+  MIT              https://opensource.org/license/mit
+  OPL v3.0         https://www.parliament.uk/site-information/copyright-parliament/open-parliament-licence/
+  Common Crawl Terms of Use
+                   https://commoncrawl.org/terms-of-use
+
+
+3.1 Open Data Commons Attribution License (ODC-By 1.0)
+--------------------------------------------------------------------------------
+
+Contains information from the following databases, which are made available under the
+ODC Attribution License (https://opendatacommons.org/licenses/by/1-0/):
+  - FineWeb                      https://huggingface.co/datasets/HuggingFaceFW/fineweb
+  - FineWeb-2 (including its removed-documents part)
+                                 https://huggingface.co/datasets/HuggingFaceFW/fineweb-2
+  - FinePDFs                     https://huggingface.co/datasets/HuggingFaceFW/finepdfs
+  - FineWeb-Edu Llama 3 annotations
+                                 https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-llama3-annotations
+  - FineTranslations             https://huggingface.co/datasets/HuggingFaceFW/finetranslations
+  - FineMath                     https://huggingface.co/datasets/HuggingFaceTB/finemath
+  - Cosmopedia v2 (SmolLM-Corpus)
+                                 https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus
+  - C4 (en.noclean)              https://huggingface.co/datasets/allenai/c4
+  - OpenWebMath                  https://huggingface.co/datasets/open-web-math/open-web-math
+  - WildChat-1M                  https://huggingface.co/datasets/allenai/WildChat-1M
+
+
+3.2 Common Crawl
+--------------------------------------------------------------------------------
+
+Much of the web text comes from Common Crawl (https://commoncrawl.org): through
+FineWeb, FineWeb-2, FinePDFs, FineMath, OpenWebMath, C4 and FineTranslations. Use was
+subject to the Common Crawl Terms of Use (https://commoncrawl.org/terms-of-use).
+The text of each page belongs to its owners.
+
+
+3.3 Creative Commons Attribution (CC BY)
+--------------------------------------------------------------------------------
+
+Modified: used to train a classifier.
+
+  - EUR-Lex resources, CC BY 4.0 (see also 3.7).
+    https://huggingface.co/datasets/joelniklaus/eurlex_resources
+  - Sangraha, synthetic part (AI4Bharat), CC BY 4.0. Machine translations of English
+    Wikipedia text (Wikipedia contributors, CC BY-SA).
+    https://huggingface.co/datasets/ai4bharat/sangraha
+  - PubMed Central Open Access Subset (U.S. National Library of Medicine), articles under
+    CC BY, credited to the authors named in each article. https://pmc.ncbi.nlm.nih.gov
+  - Wikinews (Wikinews contributors), CC BY 4.0; the German edition CC BY 2.5. The Arabic,
+    Persian, French and Swedish editions are listed under 3.4. https://www.wikinews.org
+  - Common Corpus (PleIAs), documents marked CC-By: open-access articles from OpenAlex and
+    arXiv, EUR-Lex and Eurovoc documents of the European Union (see 3.7), and German
+    political speeches, each under the license stated by its source.
+    https://huggingface.co/datasets/PleIAs/common_corpus
+  - Common Pile v0.1, documents under CC BY 2.5 / 3.0 / 4.0 (see 3.5).
+  - Open-access books, each credited to the authors and publishers named in it, under
+    the licence the platform records for it:
+      Language Science Press (CC BY 4.0)          https://langsci-press.org
+      OpenStax (Rice University), editions released under CC BY 4.0, credited as each
+      book asks, in most of them to "OpenStax and its content contributors" (the
+      wording for each book: CREDITS_BOOKS.tsv)   https://openstax.org
+      OAPEN Library (CC BY 2.0 / 3.0 / 4.0, as stated for each book)
+                                                  https://library.oapen.org
+      Directory of Open Access Books (CC BY 4.0)  https://directory.doabooks.org
+      HAL open archive (CC BY 4.0)                https://hal.science
+      Arabic E-Book Corpus, Hindawi Foundation, books under CC BY 4.0
+                                                  https://www.hindawi.org
+    EU Publications Office books: see 3.7. FAO books and the works under IGO licences:
+    see 3.8. Per-work credits for all of these books: CREDITS_BOOKS.tsv.
+
+
+3.4 Creative Commons Attribution-ShareAlike (CC BY-SA)
+--------------------------------------------------------------------------------
+
+Modified: used to train a classifier.
+
+  - Wikipedia (Wikipedia contributors), CC BY-SA 4.0 and CC BY-SA 3.0, also available
+    under the GNU Free Documentation License. With thanks to Wikipedia's volunteer editors.
+    Through https://huggingface.co/datasets/HuggingFaceFW/finewiki (August 2025 dumps) and
+    https://huggingface.co/datasets/wikimedia/wikipedia (English).
+  - Wikipedia talk-page comments (Wikipedia contributors), CC BY-SA 3.0, through
+    https://huggingface.co/datasets/OxAISH-AL-LLM/wiki_toxic
+  - Wikisource (Wikisource contributors), CC BY-SA 4.0; the dump used is published under
+    CC BY-SA 3.0 and the GFDL. The works themselves are public domain or freely licensed.
+    https://wikisource.org and https://huggingface.co/datasets/wikimedia/wikisource
+  - Wikibooks (Wikibooks contributors), CC BY-SA 4.0 (older revisions CC BY-SA 3.0).
+    https://www.wikibooks.org
+  - Wikinews, Arabic, Persian, French and Swedish editions (Wikinews contributors),
+    CC BY-SA 4.0. https://www.wikinews.org
+  - databricks-dolly-15k (Databricks), CC BY-SA 3.0.
+    https://huggingface.co/datasets/databricks/databricks-dolly-15k
+  - HC3, wiki_csai subset (Hello-SimpleAI), CC BY-SA 4.0.
+    https://huggingface.co/datasets/Hello-SimpleAI/HC3
+  - Stack Exchange (Stack Exchange contributors), CC BY-SA 2.5 / 3.0 / 4.0, through
+    Common Pile v0.1 (see 3.5).
+  - Kanripo (Kanseki Repository) transcriptions of premodern Chinese texts, CC BY-SA 4.0.
+    https://github.com/kanripo
+  - NDLA (Norwegian Digital Learning Arena) learning resources, CC BY-SA 4.0.
+    https://ndla.no
+  - Deutsches Textarchiv (DTA core corpus), CC BY-SA 4.0 as stated in each item's TEI
+    header. https://www.deutschestextarchiv.de
+  - Common Corpus (PleIAs), documents marked CC-By-SA.
+    https://huggingface.co/datasets/PleIAs/common_corpus
+  - Common Pile v0.1, documents under CC BY-SA 2.5 / 3.0 / 4.0 (see 3.5).
+  - Open-access books under CC BY-SA 3.0 / 4.0 from the OAPEN Library, the Directory of
+    Open Access Books and HAL, each credited to the authors and publishers named in it.
+    Per-work credits for these books and Wikibooks pages: CREDITS_BOOKS.tsv.
+
+
+3.5 Common Pile v0.1
+--------------------------------------------------------------------------------
+
+Includes documents from the Common Pile v0.1 (Kandpal et al., 2025, "The Common Pile
+v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text";
+https://huggingface.co/common-pile), each under the open license recorded in its
+metadata (public domain, CC0, CC BY, CC BY-SA, the Open Parliament Licence, or a
+permissive software license). Collections used: arXiv abstracts and papers,
+Biodiversity Heritage Library, Caselaw Access Project, Data Provenance Initiative,
+Directory of Open Access Books, Foodista, GitHub Archive, LibreTexts, Library of
+Congress, News, OERCommons, peS2o, pre-1929 books, Pressbooks, Project Gutenberg,
+Public Domain Review, Python Enhancement Proposals, Regulations.gov, Stack Exchange,
+Ubuntu IRC, UK Hansard, USGPO, USPTO and WikiTeam wikis.
+Modified (for the documents under a Creative Commons license): used to train a
+classifier.
+
+
+3.6 UK Parliament (Open Parliament Licence)
+--------------------------------------------------------------------------------
+
+Contains Parliamentary information licensed under the Open Parliament Licence v3.0.
+(UK Hansard, through Common Pile v0.1.)
+
+
+3.7 European Union
+--------------------------------------------------------------------------------
+
+  - EUR-Lex. Legal documents from EUR-Lex (https://eur-lex.europa.eu),
+    © European Union, 1998-2026, re-used under the Commission's document reuse policy,
+    Decision 2011/833/EU. Obtained through EUR-Lex resources
+    (https://huggingface.co/datasets/joelniklaus/eurlex_resources), and through Common
+    Corpus (collections "Eurlex" and "Eurovoc", recorded there as CC-By;
+    https://huggingface.co/datasets/PleIAs/common_corpus).
+  - European Parliament. Plenary debates from the Europarl corpus v10
+    (https://www.statmt.org/europarl/). © European Union - Source: European Parliament
+    (https://www.europarl.europa.eu).
+  - Publications Office of the European Union. Books from https://op.europa.eu,
+    © European Union. The reuse policy of European Commission documents is implemented by
+    Commission Decision 2011/833/EU; these books are re-used under CC BY 4.0. Modified:
+    used to train a classifier. Where a book asks to be cited in a particular way
+    (Eurydice, JRC and some other reports), CREDITS_BOOKS.tsv gives that citation.
+  - VoxPopuli European Parliament speech transcripts (CC0 1.0), through Common Corpus.
+
+
+3.8 International organizations
+--------------------------------------------------------------------------------
+
+World Bank (CC BY 3.0 IGO). Includes works of The World Bank, licensed under the Creative
+Commons Attribution 3.0 IGO license (https://creativecommons.org/licenses/by/3.0/igo/).
+Each work is credited in Appendix A as its copyright page asks. Modified: used to train a
+classifier. This credit does not suggest that The World Bank endorses Source-1 or its
+use. To the extent that Source-1 is regarded as an adaptation of these works: This is an
+adaptation of an original work by The World Bank. Views and opinions expressed in the
+adaptation are the sole responsibility of the author or authors of the adaptation and are
+not endorsed by The World Bank. (One work in Appendix A is a joint OECD and World Bank
+publication; its own adaptation notice is given there.)
+
+Food and Agriculture Organization of the United Nations (FAO), CC BY 4.0. Includes books
+from https://openknowledge.fao.org, copyright as stated in each book (mostly © FAO), under
+the Creative Commons Attribution 4.0 International licence. Modified: used to train a
+classifier. The citation FAO asks for ("Required citation") is given in CREDITS_BOOKS.tsv
+for every FAO book whose text carries one (114 of the 115). There is no suggestion that FAO
+endorses any specific organization, products or services, including Source-1. To the
+extent that Source-1 is regarded as an adaptation of these works: "This adaptation was not
+created by the Food and Agriculture Organization of the United Nations (FAO). FAO is not
+responsible for the content or accuracy of this adaptation. The original editions shall be
+the authoritative editions."
+
+Inter-American Development Bank (CC BY 3.0 IGO). Includes one book published by the
+Inter-American Development Bank (IDB) and deposited in HAL: Allen Blackman, Eduardo
+Cavallo, Bridget Hoffmann and Adrien Vogt-Schilb (eds.), "Peril and Promise: Tackling
+Climate Change in Latin America and the Caribbean", Inter-American Development Bank, 2025
+(ISBN 978-1-59782-574-0, digital edition; https://shs.hal.science/halshs-04977766v1),
+licensed under the Creative Commons Attribution 3.0 IGO license
+(https://creativecommons.org/licenses/by/3.0/igo/legalcode), as its own text states (HAL
+records CC BY 4.0). Also credited in CREDITS_BOOKS.tsv. Modified: used to train a
+classifier. The IDB's name is used here only to credit the IDB, and no IDB logo is used.
+This credit does not suggest that the IDB endorses Source-1 or its use.
+
+UNESCO (CC BY-SA 3.0 IGO). Includes two books published by or with UNESCO and deposited
+in HAL, credited in the form UNESCO's terms of use for its open access publications ask
+for:
+  - Cléo Lossouarn et al. / Water, Megacities and Global Change / ISBN
+    978-92-3-100161-1 – licensed under CC BY-SA 3.0 IGO. (© UNESCO / ARCEAU IdF 2016;
+    https://enpc.hal.science/hal-01449109v1)
+  - David C. Andolfatto and Thomas Schrom (eds.) / The Restoration of Mangal Bahudvara
+    Caitya. A Tashi Gomang Stupa / ISBN 978-9937-9301-6-1 – licensed under CC BY-SA 3.0
+    IGO. (© UNESCO 2021; https://hal.sorbonne-universite.fr/hal-03900987v1)
+Each book's own text states the Creative Commons Attribution-ShareAlike 3.0 IGO license
+(https://creativecommons.org/licenses/by-sa/3.0/igo/); HAL records CC BY-SA 4.0. Also
+credited in CREDITS_BOOKS.tsv, which lists the authors HAL records. Only the books' text
+was used. Modified: used to train a classifier. No UNESCO logo is used, and this credit
+does not suggest that UNESCO endorses Source-1 or its use. To the extent that Source-1 is
+regarded as an adaptation of these works: Source-1 is not an official UNESCO publication
+and shall not be considered as such.
+
+
+3.9 Licence Ouverte / Open Licence 2.0 (Etalab)
+--------------------------------------------------------------------------------
+
+No document under the Licence Ouverte is in Source-1's training or validation splits:
+the French public data in Common Corpus and the books deposited in HAL under this
+licence were removed before training.
+
+
+3.10 Other sources
+--------------------------------------------------------------------------------
+
+  - HPLT v2.0, cleaned (https://huggingface.co/datasets/HPLT/HPLT2.0_cleaned). HPLT does
+    not own the text from which these data were extracted; it licenses the packaging
+    under CC0 1.0 and runs a notice-and-takedown policy.
+  - Apache-2.0: OpenAssistant oasst2 (https://huggingface.co/datasets/OpenAssistant/oasst2);
+    Aya Dataset (https://huggingface.co/datasets/CohereLabs/aya_dataset); Cosmopedia
+    (https://huggingface.co/datasets/HuggingFaceTB/cosmopedia); Meta Kaggle Code notebooks
+    by Kaggle, through https://huggingface.co/datasets/HuggingFaceTB/issues-kaggle-notebooks
+  - MIT: RAID (https://huggingface.co/datasets/liamdugan/raid); hh-rlhf (Bai et al., 2022,
+    "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human
+    Feedback", arXiv:2204.05862).
+  - Source code under permissive licenses (MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause,
+    ISC, CC0-1.0, Unlicense), as recorded for each file's repository, from
+    https://huggingface.co/datasets/codeparrot/github-code-clean,
+    https://huggingface.co/datasets/codeparrot/github-code,
+    https://huggingface.co/datasets/bigcode/commitpackft,
+    https://huggingface.co/datasets/bigcode/commitpack, and the Common Pile v0.1
+    GitHub Archive collection. Copyright in each file stays with its authors.
+  - CC0 1.0 and public domain, with thanks: Civil Comments
+    (https://huggingface.co/datasets/google/civil_comments); PleIAs Post-OCR-Correction
+    (https://huggingface.co/datasets/PleIAs/Post-OCR-Correction); Project Gutenberg
+    (https://www.gutenberg.org); the public-domain and CC0 collections of Common Pile v0.1
+    and Common Corpus (court decisions, laws and parliamentary papers from Brazil, China,
+    Denmark, Germany, Korea, Russia and the United States; Brazilian Chamber of Deputies
+    speeches; VoxPopuli; open-access articles); CC0 articles from PubMed Central; and
+    public-domain or CC0 books from Wikisource, the National Diet Library of Japan
+    (https://lab.ndl.go.jp), DBNL (https://dbnl.org), Project Ben-Yehuda
+    (https://benyehuda.org), the Hindawi Foundation, the Norwegian Colossal Corpus
+    (National Library of Norway, through NbAiLab), PleIAs Ukrainian cultural-heritage
+    books (Internet Archive) and HAL.
+
+
+================================================================================
+4. CONTACT AND REMOVAL REQUESTS
+================================================================================
+
+Questions about these credits, corrections, and removal or opt-out requests:
+the Community tab of the model repository,
+https://huggingface.co/msmth/Source-1/discussions.
+
+
+================================================================================
+APPENDIX A. WORLD BANK WORKS (CC BY 3.0 IGO)
+================================================================================
+
+Each entry is the attribution the work's copyright page asks for, followed by its record in
+the World Bank Open Knowledge Repository. Modified: used to train a classifier. See 3.8 for
+the adaptation notice and https://creativecommons.org/licenses/by/3.0/igo/ for the license.
+
+A01. Alam, Muneeza Mehmood, and Lisa Bagnoli. 2024. Ten Thousand Steps in Her Shoes: The
+     Role of Public Transport in Women’s Economic Empowerment. Middle East and North
+     Africa Development Report Series. Washington, DC: World Bank.
+     doi:10.1596/978-1-46482091-5. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/41786
+
+A02. Alsharkas, Zeina, Jean Michel N. Marchat, Enrique Aldaz-Carroll, and Ana
+     Goicoechea. 2026. Competing in the Face of Climate Risks: Evidence from Firms and
+     Policy Priorities in MENAAP. Middle East, North Africa, Afghanistan, and Pakistan
+     Development Report. Washington, DC: World Bank. doi: 10.1596/978-1-4648-2335-0.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/45042
+
+A03. Arévalo-Sánchez, Inés, Janet Heisey, Sarang Chaudhary, Timothy Clay, Victoria
+     Strokova, Puja Vasudeva Dutta, and Colin Andrews. 2024. The State of Economic
+     Inclusion Report 2024: Pathways to Scale. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-2076-2. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/42408
+
+A04. Bachas, Pierre, Oyebola Okunogbe, Mahvish Shaukat, and Dario Tortarolo. 2026.
+     Raising Revenue Right: A Roadmap for Domestic Resource Mobilization. Policy
+     Research Report. World Bank. doi: 10.1596/978-1-4648-2034-2. License: Creative
+     Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/45448
+
+A05. Bandlien, Einar H., Sander De Kruijf, Eilen Arctander Vik, Ole Fredrik Ekern, Johan
+     Bernhard Siqueland Knudsen, Geoffrey Dyce, Silvana Tordo, and François Bertone.
+     2024. Water Management in Oil and Gas Operations: Industry Practice and Policy
+     Guidelines for Developing Countries. International Development in Focus.
+     Washington, DC: World Bank. doi:10.1596/978-1-4648-2047-2. License: Creative
+     Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/41055
+
+A06. Begazo, Tania, Moussa P. Blimpo, and Mark A. Dutz. 2023. Digital Africa:
+     Technological Transformation for Jobs. Washington, DC: World Bank. doi:
+     978-1-4648-1737-3. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/39491
+
+A07. Beylis, Guillermo, Roberto Fattal Jaef, Michael Morris, Ashwini Rekha Sebastian,
+     and Rishabh Sinha. 2020. Going Viral: COVID-19 and the Accelerated Transformation
+     of Jobs in Latin America and the Caribbean. World Bank Latin American and Caribbean
+     Studies. Washington, DC: World Bank. doi:10.1596/978-1-4648-1448-8. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/34413
+
+A08. Borgomeo, Edoardo, Claire Chase, Nicolas Salazar Godoy, and Victor Osei Kwadwo.
+     2023. Rising from the Depths: Water Security and Fragility in South Sudan.
+     International Development in Focus. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-1943-8. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/38379
+
+A09. Bossavie, Laurent, and Daniel Garrote-Sánchez. 2022. Safe and Productive Migration
+     from the Kyrgyz Republic: Lessons from the COVID-19 Pandemic. International
+     Development in Focus. Washington, DC: World Bank. doi:10.1596/978-1-4648-1905-6.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/38290
+
+A10. Chapman, Emily Weedon, and Margaux Vinez, eds. 2023. Working Today for a Better
+     Tomorrow in Ethiopia: Jobs for Poor and Vulnerable Households. International
+     Development in Focus. Washington, DC: World Bank. doi:10.1596/978-1-4648-2020-5.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/40801
+
+A11. Chatain, Pierre-Laurent, Emile van der Does de Willebois, and Maud Bökkerink. 2022.
+     Preventing Money Laundering and Terrorist Financing: A Practical Guide for Bank
+     Supervisors. Second edition. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-1851-6. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/37726
+
+A12. Chrimes, Tommy, Bram Gootjes, M. Ayhan Kose, and Collette Wheeler. 2024. The Great
+     Reversal: Prospects, Risks, and Policies in International Development Association
+     (IDA) Countries. Washington, DC: World Bank. doi: 10.1596/978-1-4648-2145-5.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/41403
+
+A13. Corsi, Anna, and Harris Selod. 2023. Land Matters: Can Better Governance and
+     Management of Scarcity Prevent a Looming Crisis in the Middle East and North
+     Africa? Washington, DC: World Bank. doi:10.1596/978-1-4648-1661-1. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/38384
+
+A14. Cruz, Marcio, ed. 2024. Digital Opportunities in African Businesses. Washington,
+     DC: World Bank. doi:10.1596/978-1-4648-2088-5. License: Creative Commons
+     Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/41447
+
+A15. Dalhuijsen, Emma, Eva Gutierrez, Tatsiana Kliatskova, Rachel Mok, and Martijn Gert
+     Jan Regelink. 2023. Greening National Development Financial Institutions: Trends,
+     Lessons Learned, and Ways Forward. International Development in Focus. Washington,
+     DC: World Bank. doi:10.1596/978-1-4648-2031-1. License: Creative Commons
+     Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/40432
+
+A16. Damania, Richard, Ebad Ebadi, Kentaro Mayr, Jason Russ, and Esha Zaveri. 2025.
+     Reboot Development: The Economics of a Livable Planet. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-2271-1. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/43522
+
+A17. de Nicola, Francesca, Aaditya Mattoo, and Jonathan Timmis. 2025. Firm Foundations
+     of Growth: Productivity and Technology in East Asia and Pacific. East Asia and
+     Pacific Development Studies. Washington, DC: World Bank. doi:
+     10.1596/978-1-4648-2200-1. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/43128
+
+A18. Didier, Tatiana, and Ana Paula Cusolito. 2024. Unleashing Productivity through Firm
+     Financing. Washington, DC: World Bank. doi:10.1596/978-1-4648-1939-1. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/42194
+
+A19. Fernandes, Ana Margarida, and Tristan Reed. 2026. Industrial Policy for
+     Development: Approaches in the 21st Century. Policy Research Reports. Washington,
+     DC: World Bank. doi:10.1596/978-1-4648-2276-6. License: Creative Commons
+     Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/44244
+
+A20. Hallegatte, Stéphane, Catrina Godinho, Jun Rentschler, Paolo Avner, Ira Irina
+     Dorband, Camilla Knudsen, Jana Lemke, and Penny Mealy. 2024. Within Reach:
+     Navigating the Political Economy of Decarbonization. Climate Change and Development
+     Series. Washington, DC: World Bank. doi:10.1596/978-1-4648-1953-7. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/40601
+
+A21. Hanusch, Marek, ed. 2023. A Balancing Act for Brazil’s Amazonian States: An Economic
+     Memorandum. International Development in Focus. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-1909-4. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/39778
+
+A22. Herrera Dappe, Matías, Mathilde Lebrand, and Aiga Stokenberga. 2024. Shrinking
+     Economic Distance: Understanding How Markets and Places Can Lower Transport Costs
+     in Developing Countries. Sustainable Infrastructure Series. Washington, DC: World
+     Bank. doi:10.1596/978-1-4648-2124-0. License: Creative Commons Attribution CC BY
+     3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/42061
+
+A23. Holla, Alaka, Norbert Schady, and Joana Silva, eds. 2026. Building Human Capital
+     Where It Matters: Homes, Neighborhoods, and Workplaces. World Bank. doi:
+     10.1596/978-1-4648-2277-3. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/44282
+
+A24. Hou, Xiaohui, Jigyasa Sharma, and Feng Zhao, eds. 2023. Silver Opportunity:
+     Building Integrated Services for Older Adults around Primary Health Care.
+     Washington, DC: World Bank. doi:10.1596/978-1-4648-1958-2. License: Creative
+     Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/39422
+
+A25. Iacovone, Leonardo, Henry Aviomoh, Matias Belacin, Laurent Bossavie, Ana Cusolito,
+     Rafael de Hoyos, Gianmarco Ottaviano, Fabian Scheifele, Iván Torre, and Yutaka
+     Yoshino. 2025. TIDES of Change: Igniting Productivity Growth in Europe and Central
+     Asia. Europe and Central Asia Studies. Washington, DC: World Bank. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/43788
+
+A26. Ianchovichina, Elena. 2024. The Evolving Geography of Productivity and Employment:
+     Ideas for Inclusive Growth through a Territorial Lens in Latin America and the
+     Caribbean. World Bank Latin American and Caribbean Studies. Washington, DC: World
+     Bank. doi:10.1596/978-1-4648-1959-9. License: Creative Commons Attribution CC BY
+     3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/40969
+
+A27. Iootty, Mariana, Asset Bizhan, and Paulo G. Correa. 2022. Boosting Productivity in
+     Kazakhstan with Micro-Level Tools: Analysis and Policy Lessons. International
+     Development in Focus. Washington, DC: World Bank. doi:10.1596/978-1-4648-1910-0.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/38460
+
+A28. Islamaj, Ergys, Aaditya Mattoo, Agustin Samano, and Matthew Wai-Poi. 2026. Small
+     Governments, Big Ambitions: Fiscal Policy in East Asia and Pacific. East Asia and
+     Pacific Development Studies. World Bank. doi:10.1596/978-1-4648-2318-3. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/45031
+
+A29. Junquera-Varela, Raúl Félix, and Cristian Óliver Lucas-Mas. 2024. Revenue
+     Administration Handbook. Washington, DC: World Bank. doi:10.1596/978-1-4648-2053-3.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/41090
+
+A30. Kassa, Woubet, Hiau Looi Kee, and Jean-Christophe Maur. 2026. Integrating Africa:
+     From Threads to Hubs. Africa Development Forum series. Washington, DC: World Bank.
+     doi: 10.1596/978-1-4648-2320-6. License: Creative Commons Attribution CC BY 3.0
+     IGO.
+     https://openknowledge.worldbank.org/handle/10986/44608
+
+A31. Kose, M. Ayhan, Peter Nagle, Franziska Ohnsorge, and Naotaka Sugawara. 2021. Global
+     Waves of Debt: Causes and Consequences. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-1544-7. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/32809
+
+A32. Lang, Megan, Jonah Rexer, Siddharth Sharma, and Margaret Triyana, eds. 2025. From
+     Risk to Resilience: Helping People and Firms Adapt in South Asia. South Asia
+     Development Matters. Washington, DC: World Bank. doi:10.1596/978-1-4648-2152-3.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/43230
+
+A33. Le, Sang Minh, Eric Hahn, Tu Anh Tran, Selin Mavituna, and Tam Minh Thi Ta. 2024.
+     Human Resources for Mental Health Service Delivery in Viet Nam: Toward Achieving
+     Universal Health Coverage. International Development in Focus. Washington, DC:
+     World Bank. doi:10.1596/978-1-4648-2122-6. License: Creative Commons Attribution CC
+     BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/41623
+
+A34. Li, Yue, and Martin Rama, eds. 2023. Private Cities: Outstanding Examples from
+     Developing Countries and Their Implications for Urban Policy. Urban Development
+     Series. Washington, DC: World Bank. doi:10.1596/978-1-4648-1833-2. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/39847
+
+A35. Mawejje, Joseph. 2025. Fiscal Vulnerabilities in Low-Income Countries: Evolution,
+     Drivers, and Policies. Washington, DC: World Bank. doi: 10.1596/978-1-4648-1968-1.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/42239
+
+A36. Mensah, Julia, Stephen Kisembe Kiirya, Elizabeth Asege Ekochu, Rogers Ayiko,
+     Brendan Michael Hayes, Collins Chansa, Grace Murindwa, Richard Crabbe, and Marc
+     DeFrancis, eds. 2024. Investing in Reproductive, Maternal, Newborn, Child, and
+     Adolescent Health in Uganda: What Have We Learned, and Where Do We Go from Here?
+     International Development in Focus. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-1993-3. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/41033
+
+A37. Mukim, Megha, and Mark Roberts, editors. 2023. Thriving: Making Cities Green,
+     Resilient, and Inclusive in a Changing Climate. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-1935-3. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/38295
+
+A38. OECD/The World Bank (2026), Compendium of Good Practices on Quality Infrastructure
+     2026: Rebuilding for the Future, OECD Publishing, Paris,
+     https://doi.org/10.1787/6981eda5-en. © OECD and The World Bank 2026. License:
+     Creative Commons Attribution 3.0 IGO (CC BY 3.0 IGO). To the extent that Source-1
+     is regarded as an adaptation of this work: This is an adaptation of an original
+     work by the OECD and the World Bank. The opinions expressed and arguments employed
+     in this adaptation should not be reported as representing the official views of the
+     OECD, its Development Centre or of their Member countries or of the World Bank, its
+     Board of Executive Directors or the governments they represent.
+     https://openknowledge.worldbank.org/handle/10986/44596
+
+A39. Peszko, Grzegorz, Markus Amann, Yewande Awe, Gary Kleiman, and Tamer Samah Rabie.
+     2022. Air Pollution and Climate Change: From Co-Benefits to Coherent Policies.
+     International Development in Focus. Washington, DC: World Bank. doi:
+     10.1596/978-1-4648-1835-6. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/38524
+
+A40. Riera-Crichton, Daniel, and Guillermo Vuletin. 2024. Public Spending Policies in
+     Latin America and the Caribbean: When Cyclicality Meets Rigidities. Latin American
+     Development Forum. Washington, DC: World Bank. doi: 10.1596/978-1-4648-2069-4.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/42022
+
+A41. Rocha, Nadia, and Michele Ruta. 2022. Deep Trade Agreements: Anchoring Global Value
+     Chains in Latin America and the Caribbean. Washington, DC: World Bank.
+     doi:10.1596/9781-4648-1824-0. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/37655
+
+A42. Rogger, Daniel, and Christian Schuster, eds. 2023. The Government Analytics
+     Handbook: Leveraging Data to Strengthen Public Administration. Washington, DC:
+     World Bank. doi:10.1596/978-1-4648-1957-5. License: Creative Commons Attribution CC
+     BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/39857
+
+A43. Schady, Norbert, Alaka Holla, Shwetlena Sabarwal, Joana Silva, and Andres Yi Chang.
+     2023. Collapse and Recovery: How the COVID-19 Pandemic Eroded Human Capital and
+     What to Do about It. Washington, DC: World Bank. doi:10.1596/978-1-4648-1901-8.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/39403
+
+A44. Sharma, Siddanth, Stefano M. Bertozzi, Victoria Y. Fan, Dean T. Jamison, Ole F.
+     Norheim, Hitoshi Oshitani, and Muhammad Ali Pate, eds. 2026. Investing in Pandemic
+     Prevention, Preparedness, and Response. Disease Control Priorities, fourth edition,
+     volume 2. Washington, DC: World Bank. doi:10.1596/978-1-4648-2213-1. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/43718
+
+A45. Shilpi, Forhad, Matthew E. Kahn, and Claudia Berg. 2025. Rethinking Resilience:
+     Adapting to a Changing Climate. Policy Research Report. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-2158-5. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/43919
+
+A46. Soh, Hoon Sahib, Youngsun Koh, and Anwar Aridi (eds.). 2023. Innovative Korea:
+     Leveraging Innovation and Technology for Development. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-1961-2. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/40234
+
+A47. Sosale, Shobhana, Seo Yeon Hong, Shalika Subasinghe, and Hiran Herat. 2023.
+     Enhancing Skills in Sri Lanka for Inclusion, Recovery, and Resilience.
+     International Development in Focus. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-2008-3. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/40663
+
+A48. Straub, Stéphane, He He, Yue Li, Xinxin Lyu, Jevgenijs Steinbuks, Estefanía Vergara
+     Cobos, Christopher Dann, Manuel García-Santana, and Harris Selod. 2026.
+     Infrastructure Foundations: From Current Assets to Future Growth. Sustainable
+     Infrastructure Series. Washington, DC: World Bank. doi:10.1596/978-1-4648-2311-4.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/44671
+
+A49. Sutton, William R., Alexander Lotsch, and Ashesh Prasann. 2024. Recipe for a
+     Livable Planet: Achieving Net Zero Emissions in the Agrifood System. Agriculture
+     and Food Series. Washington, DC: World Bank. doi:10.1596/978-1-4648-2093-9.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/41468
+
+A50. Tenenbaum, Bernard, Chris Greacen, and Ashish Shrestha. 2024. Mini Grid Solutions
+     for Underserved Customers: New Insights from Nigeria and India. International
+     Development in Focus. Washington, DC: World Bank. doi:10.1596/978-1-4648-2055-7.
+     License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/40923
+
+A51. Vasishtha, Garima, ed. 2026. Fiscal Policy in Commodity Exporters: A Balancing Act.
+     Advance Edition. Washington, DC: World Bank. License: Creative Commons Attribution
+     CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/45122
+
+A52. Vostroknutova, Ekaterina, James Sampi, Charl Jooste, and Jorge Thompson Araujo.
+     2025. Competition and Productivity Growth in Latin America and the Caribbean.
+     Washington, DC: World Bank. doi:10.1596/978-1-4648-2081-6. License: Creative
+     Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/42869
+
+A53. Vuletin, Guillermo. 2025. Rethinking Taxation for Growth in Latin America and the
+     Caribbean: Objectives, Behavioral Responses, and Technological Advances.
+     Washington, DC: World Bank. doi:10.1596/978-1-4648-2253-7. License: Creative
+     Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/43647
+
+A54. World Bank and the Development Research Center of the State Council, the People’s
+     Republic of China. 2022. Four Decades of Poverty Reduction in China: Drivers,
+     Insights for the World, and the Way Ahead. Washington, DC: World Bank.
+     doi:10.1596/978-14648-1877-6. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/37727
+
+A55. World Bank and World Trade Organization. 2020. Women and Trade: The Role of Trade
+     in Promoting Gender Equality. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-1541-6. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/34140
+
+A56. World Bank. 2023. Planning National Telemedicine and Health Hotline Services: A
+     Toolkit for Service Providers Working with Governments. International Development
+     in Practice. Washington, DC: World Bank. doi:10.1596/978-1-4648-1956-8. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/39465
+
+A57. World Bank. 2023. The Business of the State. Washington, DC: World Bank.
+     doi:10.1596/978-14648-1998-8. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/40343
+
+A58. World Bank. 2024. Digital Progress and Trends Report 2023. Washington, DC: World
+     Bank. doi:10.1596/978-1-4648-2049-6. License: Creative Commons Attribution CC BY
+     3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/40970
+
+A59. World Bank. 2024. Poverty, Prosperity, and Planet Report 2024: Pathways Out of the
+     Polycrisis. Washington, DC: World Bank. doi:10.1596/978-1-4648-2123-3. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/42211
+
+A60. World Bank. 2024. Services Unbound: Digital Technologies and Policy Reform in East
+     Asia and Pacific. East Asia and Pacific Development Studies. Washington, DC: World
+     Bank. doi:10.1596/978-1-4648-2082-3. License: Creative Commons Attribution CC BY
+     3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/42486
+
+A61. World Bank. 2024. The Path to 5G in the Developing World: Planning Ahead for a
+     Smooth Transition. Sustainable Infrastructure Series. Washington, DC: World Bank.
+     doi:10.1596/9781-4648-1604-8. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/41689
+
+A62. World Bank. 2026. From Prospective to Prepared Teacher: A Global Study of Initial
+     Teacher Education. Human Development Perspectives. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-2201-8. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/44498
+
+A63. World Bank. 2026. Institutions and Prosperity: Public Institutions for Enabling the
+     Private Sector. World Bank. DOI: 10.1596/978-1-4648-2189-9. License: Creative
+     Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/44937
+
+A64. World Bank. 2026. Women, Business and the Law 2026: Benchmarking Laws for Jobs and
+     Inclusive Growth. World Bank. doi:10.1596/978-1-4648-2196-7. License: Creative
+     Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/44315
+
+A65. World Bank. 2026. World Development Report 2026: The Promise of Artificial
+     Intelligence. World Bank. doi:10.1596/978-1-4648-2331-2. License: Creative Commons
+     Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/45296
+
+A66. World Bank. Poverty and Shared Prosperity 2022: Correcting Course. Washington, DC:
+     World Bank. doi:10.1596/978-1-4648-1893-6. License: Creative Commons Attribution CC
+     BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/37739
+
+A67. World Bank. Women, Business and the Law 2023. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-1944-5. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/39462
+
+A68. Zhang, Fan, Christian Borja-Vega, Hrishikesh Arvind Chandanpurkar, James
+     Famiglietti, Rick Hogeboom, Regassa Namara, Zarif Rasul, Pavel Luengas-Sierra, and
+     Deyu Rao. 2025. Continental Drying: A Threat to Our Common Future. Global Water
+     Monitoring. Washington, DC: World Bank. doi:10.1596/978-1-4648-2269-8. License:
+     Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/43683
+
+A69. Zhao, Feng, Rialda Kovacevic, David Bishai, and Jeff Weintraub, eds. 2024.
+     Strategic Investment for Health System Resilience: A Three-Layer Framework. Human
+     Development Perspectives. Washington, DC: World Bank.
+     doi:10.1596/978-1-4648-2116-5. License: Creative Commons Attribution CC BY 3.0 IGO.
+     https://openknowledge.worldbank.org/handle/10986/42085
+
+A70. Левитанская Екатерина, Мартин Мелецкий и Андрей Милютин. 2026. Дома и надежды:
+     мобилизация частного капитала для повышения доступности жилья и семейных инвестиций
+     в жильё в Казахстане. Международное развитие в центре внимания. Всемирный банк.
+     doi: 10.1596/978-1-4648-2346-6. Лицензия: Creative Commons Attribution CC BY 3.0
+     IGO. (English title: Homes and Hopes: Mobilizing Private Capital for Affordable
+     Housing and Family Investment in Kazakhstan.)
+     https://openknowledge.worldbank.org/handle/10986/44936

README.md

@@ -0,0 +1,409 @@
+---
+license: apache-2.0
+base_model: jhu-clsp/mmBERT-base
+base_model_relation: finetune
+# library_name is left out on purpose: Source-1 runs only through the included source1.py, and a
+# "library_name: transformers" entry would offer AutoModel and pipeline snippets that load the backbone without
+# the trained heads and return meaningless scores.
+pipeline_tag: text-classification
+language:
+- en
+- ar
+- az
+- bg
+- bn
+- ca
+- cs
+- da
+- de
+- el
+- es
+- et
+- fa
+- fi
+- fil
+- fr
+- gu
+- he
+- hi
+- hr
+- hu
+- id
+- it
+- ja
+- ka
+- kk
+- kn
+- ko
+- lt
+- lv
+- ml
+- mr
+- ms
+- nl
+- "no"
+- pl
+- pt
+- ro
+- ru
+- sk
+- sl
+- sq
+- sr
+- sv
+- sw
+- ta
+- te
+- th
+- tr
+- uk
+- ur
+- vi
+- zh
+tags:
+- data-filtering
+- pretraining-data
+- quality-classifier
+- data-quality
+- data-curation
+- text-quality
+- multilingual
+- modernbert
+- mmbert
+datasets:
+- HuggingFaceFW/fineweb-2
+- HuggingFaceFW/fineweb
+- HuggingFaceFW/finepdfs
+- HuggingFaceFW/finewiki
+- HPLT/HPLT2.0_cleaned
+- allenai/c4
+- wikimedia/wikipedia
+- wikimedia/wikisource
+- HuggingFaceFW/fineweb-edu-llama3-annotations
+- HuggingFaceTB/finemath
+- open-web-math/open-web-math
+- codeparrot/github-code-clean
+- bigcode/commitpackft
+- HuggingFaceTB/cosmopedia
+- google/civil_comments
+- PleIAs/common_corpus
+- OpenAssistant/oasst2
+- CohereLabs/aya_dataset
+- HuggingFaceFW/finetranslations
+- HuggingFaceTB/smollm-corpus
+- HuggingFaceTB/issues-kaggle-notebooks
+- allenai/WildChat-1M
+- codeparrot/github-code
+- bigcode/commitpack
+- OxAISH-AL-LLM/wiki_toxic
+- databricks/databricks-dolly-15k
+- Hello-SimpleAI/HC3
+- ai4bharat/sangraha
+- liamdugan/raid
+- PleIAs/Post-OCR-Correction
+- joelniklaus/eurlex_resources
+- common-pile/arxiv_abstracts_filtered
+- common-pile/arxiv_papers_filtered
+- common-pile/biodiversity_heritage_library
+- common-pile/biodiversity_heritage_library_filtered
+- common-pile/caselaw_access_project
+- common-pile/data_provenance_initiative_filtered
+- common-pile/doab_filtered
+- common-pile/foodista_filtered
+- common-pile/github_archive
+- common-pile/libretexts_filtered
+- common-pile/library_of_congress
+- common-pile/library_of_congress_filtered
+- common-pile/news_filtered
+- common-pile/oercommons_filtered
+- common-pile/peS2o_filtered
+- common-pile/pre_1929_books
+- common-pile/pre_1929_books_filtered
+- common-pile/pressbooks_filtered
+- common-pile/project_gutenberg_filtered
+- common-pile/public_domain_review_filtered
+- common-pile/python_enhancement_proposals_filtered
+- common-pile/regulations_filtered
+- common-pile/stackexchange
+- common-pile/ubuntu_irc_filtered
+- common-pile/uk_hansard_filtered
+- common-pile/usgpo
+- common-pile/usgpo_filtered
+- common-pile/uspto_filtered
+- common-pile/wikiteam
+- common-pile/wikiteam_filtered
+---
+
+# Source-1
+
+Source-1 scores text as pretraining data for language models. It reads up to 8,192 tokens at a time, in 53 languages.
+For each chunk it returns 13 fields (labels, quality scores and red flags), an overall 0-5 score and a keep/drop
+decision. 307M parameters, Apache-2.0.
+
+![Source-1 vs. public quality scorers](images/source1_benchmark.png)
+
+## Highlights
+
+- **Outperforms every public quality scorer we tested, on all three test sets**, measured as agreement with an
+  independent LLM grader using Source-1's rubric. On the held-out set: 0.90 against 0.76 for propella-1 4B; on its 159
+  English chunks, 0.86 against 0.53 for the FineWeb-Edu classifier.
+- **About 13x fewer parameters than propella-1 4B, and still closer to the grader.** Source-1 has 307M parameters,
+  propella-1 4B about 4.0B, and Source-1 agrees with the grader more closely on all three sets.
+- **Within 0.012 of its 27B teacher at about 1/88 the size.** On the held-out set Source-1 scores 0.900; the
+  open-weight 27B LLM teacher it learned from scores 0.912.
+- **Ahead on educational value alone, too.** On the English exam its `educational_value` reaches 0.90 rank agreement
+  with the grader's, against 0.61 for the FineWeb-Edu classifier and 0.88 for propella-1 4B.
+
+*Measured as agreement with an independent LLM grader applying Source-1's own rubric, on held-out data from the same
+kinds of sources; see [EVALUATION.md](EVALUATION.md) for methods, intervals and limits.*
+
+## Quick start
+
+Source-1 runs through the included `source1.py`, which needs only `torch`, `transformers`, `safetensors` and
+`tokenizers` (no `trust_remote_code`). Do not load it with transformers' `pipeline` or `AutoModel` classes: they load
+the backbone without the trained heads in `heads.safetensors` and return meaningless scores. The model is published on
+the Hugging Face Hub as `msmth/Source-1`.
+
+```bash
+pip install -U huggingface_hub            # provides the hf command
+hf download msmth/Source-1 --local-dir Source-1 --exclude "model.fp32.safetensors"
+cd Source-1
+pip install -r requirements.txt
+python source1.py --model . --input examples/sample.jsonl --output scores.jsonl --device cpu   # reproduces examples/expected_output.jsonl
+```
+
+```python
+from source1 import Source1
+
+model = Source1.from_pretrained(".")          # a local directory or a Hub repo id; bfloat16 weights by default
+
+doc = model.score(open("article.txt", encoding="utf-8").read(), title="Optional title")
+print(doc["overall"], doc["keep"])            # 0-5 score and the keep/drop decision
+print(doc["educational_value"], doc["spam_seo"], doc["format"])
+
+# Many documents at once: plain strings, or dicts with "text" and optionally "title".
+results = model.score_batch(["First document ...", {"text": "Second document ...", "title": "A title"}])
+
+# The full-precision copy of the weights (model.fp32.safetensors), computing in float32:
+model_fp32 = Source1.from_pretrained(".", precision="fp32", dtype="fp32")
+```
+
+```bash
+python source1.py --model . --input docs.jsonl --text-field text --output scores.jsonl --device cuda
+python source1.py --model . --input page.txt --device cpu --precision fp32 --dtype fp32
+```
+
+- **Output.** One flat dict per document: the 13 fields, `overall`, `keep`, `drop_reasons`, `parts` (chunks), the
+  length in tokens, whether a chunk was cut, and per-chunk results (`chunks`) for split documents.
+- **Precision.** `precision="bf16"` (default, `model.safetensors`, 0.6 GB) or `"fp32"` (`model.fp32.safetensors`,
+  1.2 GB, as trained; drop the `--exclude` above to get it, or load by Hub repo id, which downloads only the file you
+  ask for). It computes in bfloat16 on GPUs with native bfloat16 (NVIDIA Ampere or newer), as in the evaluation, and
+  in float32 elsewhere; float16 is refused. In bfloat16 scores move slightly with batch composition (up to about 0.04
+  on `overall`); use `dtype="fp32"` or `batch_tokens=1` to avoid it.
+- **Options.** `drop_line` (`"calibrated"` default, `"default"` for the rubric's hard filters, or an expression such
+  as `"toxicity >= 4 or spam_seo >= 3 or boilerplate >= 4.5"`), `apply_offsets`, `max_chunks`, `revision`;
+  `python source1.py --help` lists every command-line flag (JSONL, JSON or plain-text input).
+- **Examples and speed.** `examples/` holds six documents and their expected CPU output (a GPU can differ by a few
+  hundredths). Source-1 scores about 30 chunks (50,000 tokens) per second on one RTX 3090 in bfloat16.
+
+## What it returns
+
+| field | kind | scale or values |
+|---|---|---|
+| `format` | label | tutorial, reference, news, forum_qa, academic, fiction, code_file, product_page, blog_opinion, other |
+| `topic` | label | science, technology, programming, math, health, finance, history, politics_law, society, philosophy_religion, arts_entertainment, literature, sports, lifestyle, other |
+| `content_type` | label | plain_text, text_with_code, code_only, math_heavy |
+| `educational_value` | quality, 0-5, higher is better | Does it teach something useful? |
+| `reasoning_depth` | quality, 0-5 | Does it explain why and walk through steps, or just state facts? |
+| `writing_quality` | quality, 0-5 | Is it clear, well organized and coherent? (for code: naming, structure, comments) |
+| `information_density` | quality, 0-5 | How much real content per word, versus padding and filler? |
+| `reliability` | quality, 0-5 | Does it look careful and trustworthy, or sloppy and made up? (fiction is judged on care and consistency) |
+| `spam_seo` | red flag, 0-5, higher is worse | Text written to rank or sell rather than to inform: ads and pages that mainly promote score 3; keyword stuffing, clickbait and thin affiliate or doorway pages 4; auto-generated SEO text and scams 5 |
+| `boilerplate` | red flag, 0-5 | Templates, auto-generated pages, menus, link lists, cookie banners and other non-content text |
+| `toxicity` | red flag, 0-5 | Hate, harassment, explicit content (the text's own toxicity, not its subject) |
+| `code_quality` | gated, 0-5 or null | Is the code readable, correct-looking and documented, and written by a person rather than generated? Applies when `content_type` is text_with_code or code_only, or `format` is code_file |
+| `math_quality` | gated, 0-5 or null | Is the notation correct, are the steps shown, and do the solutions follow logically? Applies when `content_type` is math_heavy, or `topic` is math |
+
+Each 0-5 field is the expected level of a six-way head, a continuous number such as 2.73. Gated scores are null when
+the predicted labels say they do not apply. The anchors for every level are in `source1.json` and in
+[EVALUATION.md](EVALUATION.md#appendix-a-rubric-anchors), with the teacher's extra rules.
+
+```
+quality = 0.30*educational_value + 0.20*reasoning_depth + 0.15*writing_quality
+        + 0.20*information_density + 0.15*reliability
+if code_quality or math_quality applies:
+    quality = 0.8*quality + 0.2*mean(the gated scores that apply)
+penalty = 0.5*max(0, spam_seo - 1) + 0.4*max(0, boilerplate - 1) + 0.8*max(0, toxicity - 1)
+overall = clip(quality - penalty, 0, 5)
+
+drop if  toxicity >= 4  or  spam_seo >= 3.5  or  boilerplate >= 4.5        (keep = false)
+```
+
+The drop line in `calibration.json` is slightly stricter on spam than the rubric's own hard filters
+(`spam_seo >= 4`); both keep ads and promotional pages, which the rubric puts at `spam_seo` 3. It was picked on the
+teacher's validation labels by a fixed rule; stricter lines and their trade-offs are in
+[EVALUATION.md](EVALUATION.md#the-drop-line). You do not have to use `keep`: ranking on `overall`, or your own rules on
+single fields, may serve you better.
+
+A document longer than about 7,800 tokens is split into balanced chunks at natural boundaries. Each chunk is scored
+after a one-line header, as in training (source type, title if given, "Part i of n"). The document then takes the
+label that covers the most tokens and the token-weighted mean of each score (the maximum for `toxicity`; gated scores
+over the chunks where they apply), and `overall` and `keep` are recomputed from those.
+
+## Intended use
+
+- Filtering, ranking and weighting text for language-model pretraining in the 53 training languages.
+- Building data mixtures from the labels and the individual scores, not just one number.
+- Auditing a corpus: how much of it is boilerplate, spam, code, math, fiction, and so on.
+- Research on data quality and on distilling LLM judgments into small encoders.
+
+Out of scope:
+
+- Judging people, applications or student work. The scores describe text as training data.
+- Fact checking: `reliability` judges care and plausibility; it does not verify claims.
+- Content moderation or safety decisions: `toxicity` is a coarse flag for data filtering.
+- Deciding whether text is licensed, copyrighted or legal to use.
+- Languages outside the 53 listed (never trained or tested on), non-text inputs, and generating text.
+
+## Training
+
+- **Base model.** [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) (ModernBERT architecture, 8,192-token
+  context), all weights fine-tuned, with 13 linear heads on the mean-pooled hidden states: 307M parameters in total.
+- **Data.** 172,895 chunks (350M tokens, 151,281 documents) in 53 languages, 38.9% English: filtered and unfiltered
+  web text, PDFs, wikis, math, permissively licensed code, conversations, synthetic text and openly licensed books,
+  after license filtering and a safety filter. 9,744 chunks for validation and 9,553 for testing, split by document.
+- **Labels.** Every training label comes from an open-weight 27B LLM teacher scoring each chunk against the 13-field
+  rubric. No human labels were used, and no output of a proprietary model was used as a label or training target.
+- **Recipe.** 2 epochs (5,320 steps of 131,072 tokens), AdamW, learning rate 5e-5 (heads 10x), bfloat16 mixed
+  precision, about 1.9 hours on one H100 NVL. The learning rate and language mix came from a short sweep read on
+  validation agreement with the teacher; the checkpoint is the final step, which also had the best validation score.
+- **Development note.** AI assistants helped draft the rubric and write the code. Grades from proprietary LLM graders,
+  among them models of the evaluation grader's family, were used to choose between rubric revisions, to vet the
+  teacher and choose its prompt setup (on the samples that became the exam sets), and to set the candidate drop lines
+  and the 0.95 floor of the drop-line rule; LLM reviews also informed the license filtering. None of these grades or
+  reviews was used as a label or training example.
+
+Data sources, filtering, recipe and development details:
+[EVALUATION.md](EVALUATION.md#appendix-b-training-in-detail) and
+[Independence from development](EVALUATION.md#independence-from-development).
+
+## Evaluation
+
+Source-1 was compared with 16 public quality scorers on three test sets, each graded by an independent proprietary
+LLM grader applying Source-1's rubric; the grader's labels were never trained on. Each cell is the rank agreement
+(Spearman) between each model's main score and the grader's overall score, on the chunks that model scored.
+
+| test set | chunks | Source-1 | propella-1 4B | FineWeb-Edu classifier |
+|---|---|---|---|---|
+| held-out set, 53 languages (main result) | 495 | **0.900** | 0.756 | 0.529 (159 English chunks; Source-1 0.864) |
+| English exam | 412 to 413 | **0.921** | 0.820 | 0.453 |
+| 12-language exam (web text) | 350 to 352 | **0.895** | 0.637 | English only |
+
+The open-weight 27B LLM teacher scores 0.912, 0.912 and 0.880 on the same sets. All 39 public-scorer comparisons
+favor Source-1 with 95% intervals clear of zero. The held-out set is the main result because, unlike the exams, it
+played no part in choosing the teacher. Scoring each held-out chunk in full rather than its first 512 tokens raises
+agreement from 0.85 to 0.90 ([details](EVALUATION.md#reading-the-whole-chunk)).
+
+propella-1 has no single score: we read it through a composite of four of its quality ratings. Counting its own
+commercial-bias, content-ratio, integrity and safety ratings as well narrows its held-out gap from 0.14 to 0.06-0.08,
+depending on the weighting and languages, still in Source-1's favor
+([details](EVALUATION.md#how-propella-1-is-read)). The public scorers also read the text without Source-1's one-line
+header ([details](EVALUATION.md#the-one-line-header)).
+
+Every scorer, interval, drop-line result and caveat is in [EVALUATION.md](EVALUATION.md).
+
+## Limitations
+
+- **Home-ground benchmark.** The grader applies Source-1's own rubric, and grades from its model family steered rubric
+  revisions, the choice of teacher and the drop-line design. The public scorers were built for their own definitions
+  of quality. Whether filtering with Source-1 trains better language models has not been tested.
+- **The exams helped choose the teacher.** The two exam sets are the samples on which the teacher and its prompt setup
+  were chosen, against the exam grader's labels, so they are not independent of the grader.
+- **It rates some qualities higher than the grader.** On the English exam's random sample, `educational_value` is 0.28
+  levels above the grader on average, and `reasoning_depth`, `writing_quality` and `reliability` 0.26 to 0.39 (the
+  teacher: 0.31, and 0.26 to 0.42).
+- **It misses a third of the grader's drops at the shipped line.** It catches 43 of 64 (0.67) on the held-out set and
+  the teacher 46; 17 of Source-1's 21 misses are also missed by the teacher. A stricter `drop_line` catches more.
+- **Weaker on some kinds of text.** Held-out rank is 0.68 for books and 0.73 for conversations, code and synthetic text,
+  against about 0.90 for web text; `code_quality` and `math_quality` are the least reliable fields.
+- **Not a fact checker or a moderation tool.** `reliability` is a surface judgment; `toxicity` was trained on data where
+  toxic text is rare.
+- **One chunk at a time, and less data for some languages.** Nothing outside a chunk of up to 8,192 tokens is visible to
+  it; the 16 languages with the fewest training chunks have 1,249 to 1,470 each.
+- **License screening has limits.** Notices worded in ways the patterns miss, and opt-outs outside the text (such as
+  robots.txt), were not caught. If you find such a document, tell us (see below).
+- **No reproduction kit.** The evaluation chunks, the grader's labels, per-chunk scores, the metrics script and the
+  teacher's prompt are not included, so the numbers cannot be recomputed from this repository.
+
+More: [Limitations in detail](EVALUATION.md#limitations-in-detail).
+
+## Files
+
+| file | contents |
+|---|---|
+| `model.safetensors` | the fine-tuned mmBERT-base backbone in bfloat16 (default) |
+| `model.fp32.safetensors` | the same backbone in float32 (load with `precision="fp32"`) |
+| `config.json` | backbone configuration (ModernBERT) |
+| `heads.safetensors` | the 13 scoring heads |
+| `source1.json` | rubric, head layout, pooling, maximum length and text normalization |
+| `calibration.json` | the drop line and the quality-score offsets, with how they were chosen |
+| `tokenizer.json`, `tokenizer_config.json` | mmBERT's tokenizer (same vocabulary and merges, re-saved) |
+| `source1.py` | standalone loader, Python API and command line |
+| `requirements.txt` | `torch`, `transformers`, `safetensors`, `tokenizers` |
+| `examples/` | six sample inputs (`sample.jsonl`) and their expected command-line output (`expected_output.jsonl`) |
+| `EVALUATION.md` | the full evaluation, the rubric anchors and the training details |
+| `images/` | the benchmark chart above |
+| `LICENSE`, `NOTICE`, `AUTHORS` | license text, third-party notices and credits, authors |
+| `CREDITS_BOOKS.tsv` | per-work credits for the training books that are not public domain (part of `NOTICE`) |
+
+## License and credits
+
+Source-1 is released under the [Apache License 2.0](LICENSE). Copyright 2026 The Source-1 Authors (see `AUTHORS`).
+[`NOTICE`](NOTICE) holds the full third-party notices and data credits. In short:
+
+- **Base model.** Fine-tuned from [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) by the mmBERT authors at
+  Johns Hopkins University (Marone et al., 2025), MIT License; all encoder weights were further trained and 13 scoring
+  heads added. mmBERT's tokenizer is based on the Gemma 2 tokenizer by Google.
+- **Labels.** Produced by an open-weight 27B LLM released under Apache-2.0, self-hosted; no teacher weights are here.
+- **Training data.** Public sources under their own terms, credited in `NOTICE` and `CREDITS_BOOKS.tsv`: among them
+  data under the ODC Attribution License (FineWeb, FineWeb-2, FinePDFs, C4, FineMath and others; Common Crawl data
+  through them was subject to the Common Crawl Terms of Use), Wikipedia-family text (CC BY-SA, GFDL or CC BY; with
+  thanks to its volunteer editors), Common Pile v0.1, HPLT 2.0, EU publications, and Parliamentary information
+  licensed under the Open Parliament Licence v3.0. Books include World Bank publications under CC BY 3.0 IGO; the World
+  Bank and the other publishers do not endorse this model. Code is limited to permissive licenses.
+- **Share-alike text.** About 12.6% of training documents carry CC BY-SA or GFDL licenses. Source-1 is a classifier:
+  it outputs scores, not text, and no training text is distributed with it. The weights are released under Apache-2.0
+  with attribution, as comparable quality classifiers are, on the view that such a scorer is not an adaptation of the
+  text it was trained on. Copyleft code was removed anyway.
+- Upstream license metadata can be wrong. If you find a source that should not be here, please tell us (below).
+
+## Contact and takedown
+
+Questions, corrections and removal requests: the [Community tab](https://huggingface.co/msmth/Source-1/discussions). If you believe your content was used to train Source-1 and
+you want it excluded from future versions, tell us the URL or dataset and we will remove it from the training data of
+the next release.
+
+## Citation
+
+```bibtex
+@misc{source1_2026,
+  title        = {Source-1: a multilingual 13-field scorer for pretraining data},
+  author       = {{The Source-1 Authors}},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/msmth/Source-1}}
+}
+```
+
+Please also cite mmBERT:
+
+```bibtex
+@misc{marone2025mmbertmodernmultilingualencoder,
+  title         = {mmBERT: A Modern Multilingual Encoder with Annealed Language Learning},
+  author        = {Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
+  year          = {2025},
+  eprint        = {2509.06888},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.CL},
+  url           = {https://arxiv.org/abs/2509.06888}
+}
+```

calibration.json

@@ -0,0 +1,106 @@
+{
+  "version": 1,
+  "model": "Source-1",
+  "fitted_on": "a held-out validation split of 9,744 chunks (documents never trained on), against the labels of the teacher, an open-weight 27B LLM scoring the 13-field rubric; nothing here was fitted on evaluation labels",
+  "drop_line": {
+    "line": "toxicity >= 4 or spam_seo >= 3.5 or boilerplate >= 4.5",
+    "meaning": "keep is false when this expression is true on a chunk's or a document's scores. It is the schema's default hard filters (schema_default) with the spam_seo threshold lowered from 4 to 3.5. source1.py applies it by default (drop_line=\"default\" applies schema_default instead).",
+    "schema_default": "toxicity >= 4 or spam_seo >= 4 or boilerplate >= 4.5",
+    "rule": "the candidate with the highest drop recall among those whose keep agreement with the teacher's keep flags is at least 0.95",
+    "reference": "the teacher's keep flags: schema_default applied to the teacher's labels of each validation chunk",
+    "columns": {
+      "keep_agreement": "share of chunks where the line and the teacher make the same keep decision",
+      "drop_recall": "share of the teacher's drops the line also drops",
+      "teacher_drops": "chunks the teacher drops",
+      "caught": "of those, chunks the line drops too",
+      "wrong_drops": "chunks the line drops but the teacher keeps",
+      "line_drops": "chunks the line drops",
+      "drop_share": "share of all chunks the line drops"
+    },
+    "candidates": [
+      {
+        "line": "toxicity >= 4 or spam_seo >= 4 or boilerplate >= 4.5",
+        "keep_agreement": 0.96,
+        "drop_recall": 0.7111,
+        "teacher_drops": 1042,
+        "caught": 741,
+        "wrong_drops": 89,
+        "line_drops": 830,
+        "drop_share": 0.0852
+      },
+      {
+        "line": "toxicity >= 4 or spam_seo >= 3.5 or boilerplate >= 4.5",
+        "keep_agreement": 0.9638,
+        "drop_recall": 0.7706,
+        "teacher_drops": 1042,
+        "caught": 803,
+        "wrong_drops": 114,
+        "line_drops": 917,
+        "drop_share": 0.0941
+      },
+      {
+        "line": "toxicity >= 4 or spam_seo >= 3 or boilerplate >= 4.5",
+        "keep_agreement": 0.929,
+        "drop_recall": 0.8301,
+        "teacher_drops": 1042,
+        "caught": 865,
+        "wrong_drops": 515,
+        "line_drops": 1380,
+        "drop_share": 0.1416
+      },
+      {
+        "line": "toxicity >= 4 or spam_seo >= 2.5 or boilerplate >= 4.5",
+        "keep_agreement": 0.8559,
+        "drop_recall": 0.8724,
+        "teacher_drops": 1042,
+        "caught": 909,
+        "wrong_drops": 1271,
+        "line_drops": 2180,
+        "drop_share": 0.2237
+      },
+      {
+        "line": "toxicity >= 4 or spam_seo >= 3.5 or boilerplate >= 4",
+        "keep_agreement": 0.9453,
+        "drop_recall": 0.88,
+        "teacher_drops": 1042,
+        "caught": 917,
+        "wrong_drops": 408,
+        "line_drops": 1325,
+        "drop_share": 0.136
+      }
+    ]
+  },
+  "offsets_meaning": "per quality score: mean teacher label minus mean Source-1 score over the validation chunks; added to Source-1's score, an offset removes its average lean against the teacher. All are below 0.02 in absolute value, so source1.py reports the model's own scores unless apply_offsets=True.",
+  "offsets": {
+    "educational_value": {
+      "offset": -0.00264963054187195,
+      "chunks": 9744,
+      "mean_teacher": 1.9859,
+      "mean_source1": 1.9886
+    },
+    "reasoning_depth": {
+      "offset": -0.005093390804597586,
+      "chunks": 9744,
+      "mean_teacher": 1.5808,
+      "mean_source1": 1.5859
+    },
+    "writing_quality": {
+      "offset": -0.009194376026272266,
+      "chunks": 9744,
+      "mean_teacher": 2.9367,
+      "mean_source1": 2.9459
+    },
+    "information_density": {
+      "offset": -0.005678571428571644,
+      "chunks": 9744,
+      "mean_teacher": 2.485,
+      "mean_source1": 2.4907
+    },
+    "reliability": {
+      "offset": -0.01612294745484366,
+      "chunks": 9744,
+      "mean_teacher": 3.1512,
+      "mean_source1": 3.1673
+    }
+  }
+}

config.json

@@ -0,0 +1,80 @@
+{
+  "architectures": [
+    "ModernBertModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "bfloat16",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 1,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "local_rope_theta": 160000,
+  "mask_token_id": 4,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 0,
+  "position_embedding_type": "sans_pos",
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 1,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "vocab_size": 256000
+}

examples/expected_output.jsonl

@@ -0,0 +1,6 @@
+{"id": "tutorial-binary-search", "format": "tutorial", "topic": "programming", "content_type": "plain_text", "educational_value": 4.035, "reasoning_depth": 3.852, "writing_quality": 4.17, "information_density": 4.034, "reliability": 4.135, "spam_seo": 0.009, "boilerplate": 0.008, "toxicity": 0.004, "code_quality": null, "math_quality": null, "overall": 4.033, "keep": true, "drop_reasons": [], "parts": 1, "tokens": 367, "truncated": false}
+{"id": "seo-spam", "format": "product_page", "topic": "lifestyle", "content_type": "plain_text", "educational_value": 0.0, "reasoning_depth": 0.0, "writing_quality": 1.002, "information_density": 0.002, "reliability": 0.899, "spam_seo": 4.94, "boilerplate": 4.905, "toxicity": 0.002, "code_quality": null, "math_quality": null, "overall": 0.0, "keep": false, "drop_reasons": ["spam_seo >= 3.5", "boilerplate >= 4.5"], "parts": 1, "tokens": 115, "truncated": false}
+{"id": "python-code", "format": "code_file", "topic": "programming", "content_type": "code_only", "educational_value": 3.021, "reasoning_depth": 2.315, "writing_quality": 3.955, "information_density": 3.987, "reliability": 3.981, "spam_seo": 0.007, "boilerplate": 0.016, "toxicity": 0.009, "code_quality": 3.982, "math_quality": null, "overall": 3.482, "keep": true, "drop_reasons": [], "parts": 1, "tokens": 235, "truncated": false}
+{"id": "math-worked-example", "format": "tutorial", "topic": "math", "content_type": "plain_text", "educational_value": 4.165, "reasoning_depth": 4.093, "writing_quality": 3.869, "information_density": 4.119, "reliability": 4.101, "spam_seo": 0.483, "boilerplate": 0.105, "toxicity": 0.037, "code_quality": null, "math_quality": 4.109, "overall": 4.092, "keep": true, "drop_reasons": [], "parts": 1, "tokens": 194, "truncated": false}
+{"id": "spanish-water-cycle", "format": "reference", "topic": "science", "content_type": "plain_text", "educational_value": 3.109, "reasoning_depth": 2.381, "writing_quality": 4.006, "information_density": 3.98, "reliability": 3.996, "spam_seo": 0.001, "boilerplate": 0.001, "toxicity": 0.0, "code_quality": null, "math_quality": null, "overall": 3.405, "keep": true, "drop_reasons": [], "parts": 1, "tokens": 131, "truncated": false}
+{"id": "cookie-banner", "format": "other", "topic": "other", "content_type": "plain_text", "educational_value": 0.0, "reasoning_depth": 0.0, "writing_quality": 2.656, "information_density": 0.001, "reliability": 2.992, "spam_seo": 0.114, "boilerplate": 5.0, "toxicity": 0.001, "code_quality": null, "math_quality": null, "overall": 0.0, "keep": false, "drop_reasons": ["boilerplate >= 4.5"], "parts": 1, "tokens": 74, "truncated": false}

examples/sample.jsonl

@@ -0,0 +1,6 @@
+{"id": "tutorial-binary-search", "title": "Binary search, step by step", "text": "Binary search finds a value in a sorted list by halving the part of the list that can still contain it.\n\n1. Start with two markers: low at the first position and high at the last.\n2. Look at the middle element, at position (low + high) // 2.\n3. If it equals the target, you are done. If it is smaller, the target can only be to its right, so move low to the middle plus one. If it is larger, move high to the middle minus one.\n4. Repeat until low passes high; then the value is not in the list.\n\nWhy does this work? Because the list is sorted, one comparison with the middle element rules out half of the remaining candidates. A list of 1,000 items therefore needs at most 10 comparisons, since 2^10 = 1,024. A linear scan may need all 1,000.\n\nExample: search for 23 in [2, 5, 8, 12, 16, 23, 38, 56, 72, 91]. The middle element is 16, which is smaller, so we keep [23, 38, 56, 72, 91]. Its middle is 56, which is larger, so we keep [23, 38]. The middle of that is 23: found after three comparisons.\n\nA common bug is computing the middle as (low + high) / 2 in languages with fixed-size integers, where low + high can overflow; low + (high - low) // 2 avoids it."}
+{"id": "seo-spam", "text": "Best cheap running shoes 2026 - buy cheap running shoes online! Cheap running shoes sale, running shoes discount, best running shoes cheap price. Click here for the best cheap running shoes deals!!! Limited offer: cheap running shoes free shipping. Running shoes cheap, shoes running cheap, cheap shoes for running. Don't miss the best cheap running shoes - BUY NOW and save 70% on cheap running shoes. Top 10 cheap running shoes, cheap running shoes reviews, where to buy cheap running shoes. Visit our store today for cheap running shoes!"}
+{"id": "python-code", "code_language": "Python", "text": "def moving_average(values, window):\n    \"\"\"Return the simple moving averages of `values` over `window` consecutive items.\n\n    >>> moving_average([1, 2, 3, 4, 5], 2)\n    [1.5, 2.5, 3.5, 4.5]\n    \"\"\"\n    if window <= 0:\n        raise ValueError(\"window must be positive\")\n    if window > len(values):\n        return []\n    total = sum(values[:window])\n    out = [total / window]\n    for i in range(window, len(values)):\n        # slide the window: add the new item, drop the oldest one\n        total += values[i] - values[i - window]\n        out.append(total / window)\n    return out\n"}
+{"id": "math-worked-example", "title": "Solving a quadratic equation", "text": "Solve x^2 - 5x + 6 = 0.\n\nWe look for two numbers whose product is 6 and whose sum is 5: these are 2 and 3. Hence x^2 - 5x + 6 = (x - 2)(x - 3).\nA product is zero exactly when one of its factors is zero, so x - 2 = 0 or x - 3 = 0, which gives x = 2 or x = 3.\n\nCheck with the quadratic formula: x = (5 ± sqrt(25 - 24)) / 2 = (5 ± 1) / 2, so x = 3 or x = 2. Substituting x = 2 gives 4 - 10 + 6 = 0, and x = 3 gives 9 - 15 + 6 = 0, as required."}
+{"id": "spanish-water-cycle", "title": "El ciclo del agua", "text": "El ciclo del agua describe cómo el agua se mueve entre los océanos, la atmósfera y la tierra. El calor del sol evapora el agua de los mares y lagos; el vapor sube, se enfría y se condensa en pequeñas gotas que forman las nubes. Cuando las gotas crecen lo suficiente, caen como lluvia o nieve. Parte de esa agua corre por los ríos de vuelta al mar, otra parte se infiltra en el suelo y alimenta los acuíferos, y las plantas devuelven una fracción a la atmósfera por transpiración. Así, la misma agua puede recorrer el ciclo muchas veces a lo largo de los siglos."}
+{"id": "cookie-banner", "text": "Home | About us | Products | Contact | Login\n\nWe use cookies to improve your experience. By continuing to browse this site you agree to our use of cookies. Accept all | Reject | Settings\n\nShare on Facebook | Share on X | Share by email\n\nPrivacy policy | Terms of use | Sitemap | Copyright 2026 Example Store. All rights reserved."}

heads.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63f3a0a283d878dce4b172545b54d83e75c5aca91f2c944247b2413fee32c0f9
+size 275868

model.fp32.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1f6ff1712c30b9059e37d202817d57d7164678a2ac01292dd4f4904612a2454
+size 1227771776

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f33cf7774481046edc9fb1159ee9748c34aaa5532232fa8f5f2d6daf142e640
+size 613892480

requirements.txt

@@ -0,0 +1,8 @@
+# source1.py needs Python >= 3.10 and these packages. The minimum versions are the oldest this release was tested
+# with (torch 2.11 and 2.14, transformers 5.17), with both weight files: model.safetensors (bfloat16, the default)
+# and model.fp32.safetensors (float32). Older versions may work but were not tested. No GPU is needed: on a CPU, or a
+# GPU without native bfloat16, the bfloat16 weights are upcast to float32 when loaded.
+torch>=2.11
+transformers>=5.17
+safetensors>=0.8
+tokenizers>=0.23

source1.json

@@ -0,0 +1,302 @@
+{
+  "architecture": "Source1ForScoring",
+  "harness_version": "0.1.0",
+  "pooling": "mean",
+  "max_length": 8192,
+  "input_format": "classifier",
+  "layout": [
+    [
+      "format",
+      "label_single",
+      10
+    ],
+    [
+      "topic",
+      "label_single",
+      15
+    ],
+    [
+      "content_type",
+      "label_single",
+      4
+    ],
+    [
+      "educational_value",
+      "score",
+      6
+    ],
+    [
+      "reasoning_depth",
+      "score",
+      6
+    ],
+    [
+      "writing_quality",
+      "score",
+      6
+    ],
+    [
+      "information_density",
+      "score",
+      6
+    ],
+    [
+      "reliability",
+      "score",
+      6
+    ],
+    [
+      "spam_seo",
+      "score",
+      6
+    ],
+    [
+      "boilerplate",
+      "score",
+      6
+    ],
+    [
+      "toxicity",
+      "score",
+      6
+    ],
+    [
+      "code_quality",
+      "score",
+      6
+    ],
+    [
+      "math_quality",
+      "score",
+      6
+    ]
+  ],
+  "schema_fingerprint": "7c8ff0235cbc7416",
+  "layout_fingerprint": "3df647d7f4bd5ee0",
+  "schema": {
+    "name": "source-1",
+    "version": 1,
+    "labels": {
+      "format": {
+        "short": "fmt",
+        "kind": "single",
+        "description": "What kind of document this is.",
+        "values": {
+          "tutorial": "Step-by-step instruction, how-to guides, lessons, worked walkthroughs.",
+          "reference": "Documentation, manuals, API references, specifications, encyclopedia or wiki entries.",
+          "news": "News reporting, press releases, announcements of current events.",
+          "forum_qa": "Forum threads, Q&A pages, comment sections, mailing-list discussions.",
+          "academic": "Research papers, theses, textbooks, lecture notes, scholarly writing.",
+          "fiction": "Stories, novels, poetry, screenplays, fan fiction and other creative writing.",
+          "code_file": "A source-code file or notebook - code with or without comments.",
+          "product_page": "Product listings, shopping pages, marketing or landing pages, ads.",
+          "blog_opinion": "Personal blogs, essays, opinion pieces, newsletters, reviews.",
+          "other": "Anything else - legal text, transcripts, lists, records, fragments."
+        }
+      },
+      "topic": {
+        "short": "top",
+        "kind": "single",
+        "description": "The main subject of the document.",
+        "values": {
+          "science": "Natural sciences - physics, chemistry, biology, earth and space science.",
+          "technology": "Engineering, hardware, electronics, IT operations, applied technology.",
+          "programming": "Software development, computer science, code and developer tooling.",
+          "math": "Mathematics, statistics and formal logic.",
+          "health": "Medicine, health, nutrition, fitness and mental health.",
+          "finance": "Finance, economics, business, investing, accounting and careers.",
+          "history": "History, archaeology and historical biography.",
+          "politics_law": "Politics, government, law, public policy and current affairs.",
+          "society": "Social science, education, culture, relationships and psychology.",
+          "philosophy_religion": "Philosophy, ethics, religion and spirituality.",
+          "arts_entertainment": "Art, music, film, television, games, celebrities and pop culture.",
+          "literature": "Literature, language, linguistics, and writing about books and writing.",
+          "sports": "Sports and athletics.",
+          "lifestyle": "Food, travel, home, fashion, hobbies, parenting and pets.",
+          "other": "Anything else, or no clear subject."
+        }
+      },
+      "content_type": {
+        "short": "ctype",
+        "kind": "single",
+        "description": "What the text is made of.",
+        "values": {
+          "plain_text": "Prose or other natural-language text with no significant code or math.",
+          "text_with_code": "Natural-language text that includes code snippets or code blocks.",
+          "code_only": "Source code with, at most, comments and docstrings.",
+          "math_heavy": "Text dominated by equations, formulas, proofs or worked calculations."
+        }
+      }
+    },
+    "quality": {
+      "educational_value": {
+        "short": "edu",
+        "weight": 0.3,
+        "description": "Does it teach something useful?",
+        "rubric": {
+          "0": "Teaches nothing - spam, ads, navigation, gibberish, or text with no informational purpose.",
+          "1": "Almost nothing to learn - a few incidental facts buried in promotional, personal or trivial text.",
+          "2": "Some useful information, but superficial, fragmentary, or mixed with a lot of irrelevant material.",
+          "3": "Useful and coherent - conveys real knowledge or skills, though without much depth or completeness.",
+          "4": "Clearly educational - explains concepts, methods or knowledge well enough to learn from, with minor gaps.",
+          "5": "Outstanding teaching material - deep, clear and well structured, with examples or worked solutions; comparable to an excellent textbook or expert tutorial."
+        }
+      },
+      "reasoning_depth": {
+        "short": "rsn",
+        "weight": 0.2,
+        "description": "Does it explain why and walk through steps, or just state facts?",
+        "rubric": {
+          "0": "No reasoning at all - fragments, lists, boilerplate or non-content.",
+          "1": "Bare assertions or opinions with no explanation.",
+          "2": "Occasional explanation, but mostly unsupported statements; steps are skipped.",
+          "3": "Explains the why behind key points, with some step-by-step structure or argument.",
+          "4": "Consistent, explicit reasoning - derivations, cause and effect, justified arguments, worked examples.",
+          "5": "Rigorous multi-step reasoning throughout - proofs, careful derivations, or thorough analysis that motivates every step and weighs alternatives."
+        }
+      },
+      "writing_quality": {
+        "short": "wrt",
+        "weight": 0.15,
+        "description": "Is it clear, well organized and coherent? For code, judge naming, structure and comments.",
+        "rubric": {
+          "0": "Unreadable - garbled, machine-generated nonsense, broken encoding, keyword soup.",
+          "1": "Very poor - frequent errors, incoherent structure, fragments; hard to follow.",
+          "2": "Below average - understandable but disorganized, repetitive or error-prone.",
+          "3": "Adequate - clear and coherent with minor issues.",
+          "4": "Good - well organized, fluent, precise and sensibly structured.",
+          "5": "Excellent - exemplary, publication-quality writing."
+        }
+      },
+      "information_density": {
+        "short": "dns",
+        "weight": 0.2,
+        "description": "How much real content per word, versus padding and filler?",
+        "rubric": {
+          "0": "No real content - filler, repetition, boilerplate.",
+          "1": "Mostly padding - long intros, fluff, repeated phrases or SEO filler around a little content.",
+          "2": "Noticeable padding or digressions; it could be far shorter without losing anything.",
+          "3": "Reasonable - mostly on point, with some filler.",
+          "4": "Dense - little wasted text; most sentences carry information.",
+          "5": "Very dense yet readable - every sentence adds substance."
+        }
+      },
+      "reliability": {
+        "short": "rel",
+        "weight": 0.15,
+        "description": "Does it look careful and trustworthy, or sloppy and made up? For fiction and other creative writing, judge care and internal consistency; invented events are not a reliability problem.",
+        "rubric": {
+          "0": "Fabricated, nonsensical or deceptive - scams, fake news, conspiracy content, generated nonsense.",
+          "1": "Largely unreliable - many errors, sensational or unsupported claims, misleading framing.",
+          "2": "Questionable - some errors or unsupported claims; careless.",
+          "3": "Generally plausible and consistent; no obvious errors, but informal or unverifiable.",
+          "4": "Careful and accurate - precise, consistent claims; shows its work or cites sources.",
+          "5": "Authoritative - expert-level accuracy, well sourced, carefully qualified claims."
+        }
+      }
+    },
+    "red_flags": {
+      "spam_seo": {
+        "short": "spam",
+        "description": "Keyword stuffing, clickbait, affiliate filler, and other text written to rank or sell rather than to inform.",
+        "penalty": {
+          "threshold": 1,
+          "weight": 0.5
+        },
+        "rubric": {
+          "0": "None.",
+          "1": "Minor promotion - a call to action or a brief ad in otherwise genuine content.",
+          "2": "Noticeable promotion - repeated calls to action, affiliate links, marketing tone.",
+          "3": "Substantial - the text mainly exists to promote, sell or rank; visible keyword repetition.",
+          "4": "Mostly spam - keyword stuffing, clickbait, thin affiliate or doorway content.",
+          "5": "Pure spam - auto-generated SEO text, scams, keyword lists."
+        }
+      },
+      "boilerplate": {
+        "short": "boil",
+        "description": "Templates, auto-generated pages, lists of links, cookie banners, menus and other non-content text.",
+        "penalty": {
+          "threshold": 1,
+          "weight": 0.4
+        },
+        "rubric": {
+          "0": "None - all of the text is real content.",
+          "1": "A little - a stray header, footer or copyright line.",
+          "2": "Noticeable - navigation, cookie or legal text, or share-button text mixed in (roughly 10-25% of the text).",
+          "3": "Substantial - roughly 25-50% templates, menus, link lists or legal notices.",
+          "4": "Mostly boilerplate - over half of the text is templated or navigational.",
+          "5": "Entirely boilerplate - auto-generated pages, link lists, cookie banners, error or index pages."
+        }
+      },
+      "toxicity": {
+        "short": "tox",
+        "aggregate": "max",
+        "description": "Hate, harassment and explicit content. Rate the text's own toxicity, not its subject: scholarly, journalistic or educational discussion of hate, violence or sexuality is not toxic.",
+        "penalty": {
+          "threshold": 1,
+          "weight": 0.8
+        },
+        "rubric": {
+          "0": "None.",
+          "1": "Mild - occasional profanity or rudeness, not aimed at anyone.",
+          "2": "Moderate - insults, crude humor, mild sexual references, heated hostility.",
+          "3": "Significant - harassment, demeaning stereotypes, graphic violence, or partly explicit sexual content.",
+          "4": "Severe - hate speech, targeted harassment, or sexually explicit material as the main content.",
+          "5": "Extreme - violent extremism, dehumanizing hate, incitement, or sexual content involving minors."
+        }
+      }
+    },
+    "gated": {
+      "code_quality": {
+        "short": "code",
+        "description": "Is the code readable, correct-looking and documented, and written by a person rather than generated?",
+        "applies_when": {
+          "content_type": [
+            "text_with_code",
+            "code_only"
+          ],
+          "format": [
+            "code_file"
+          ]
+        },
+        "rubric": {
+          "0": "Not usable code - garbled, minified, obfuscated or broken beyond repair.",
+          "1": "Very poor - likely non-functional fragments, no structure, or auto-generated boilerplate.",
+          "2": "Poor - may work but messy; unclear names, no comments, hard-coded values.",
+          "3": "Acceptable - readable and plausibly correct, with some structure and minimal documentation.",
+          "4": "Good - clean, idiomatic, well named and documented; plausibly correct, with edge cases handled.",
+          "5": "Excellent - exemplary, production-quality, well-documented and instructive code."
+        }
+      },
+      "math_quality": {
+        "short": "math",
+        "description": "Is the notation correct, are the steps shown, and do the solutions follow logically?",
+        "applies_when": {
+          "content_type": [
+            "math_heavy"
+          ],
+          "topic": [
+            "math"
+          ]
+        },
+        "rubric": {
+          "0": "Garbled math - broken notation or nonsense.",
+          "1": "Mostly wrong or incoherent; results asserted without work.",
+          "2": "Some correct math, but errors, sloppy notation, or skipped steps that break the logic.",
+          "3": "Generally correct, with standard notation and the key steps shown.",
+          "4": "Correct, clean notation, complete steps and a clear logical flow.",
+          "5": "Rigorous and elegant - precise notation, every step justified, solutions easy to verify."
+        }
+      }
+    },
+    "composite": {
+      "gated_weight": 0.2,
+      "hard_filters": [
+        "toxicity >= 4",
+        "spam_seo >= 4",
+        "boilerplate >= 4.5"
+      ]
+    }
+  },
+  "text_normalize": "collapse_spaces"
+}

source1.py

@@ -0,0 +1,1011 @@
+#!/usr/bin/env python3
+# Copyright 2026 The Source-1 Authors
+# SPDX-License-Identifier: Apache-2.0
+"""Source-1: score text as language-model pretraining data.
+
+A standalone loader and scorer for Source-1, an mmBERT-base encoder fine-tuned with 13 scoring heads. It needs
+only torch, transformers, safetensors and tokenizers (no remote code, no other project code), and runs on a GPU or a
+CPU.
+
+    from source1 import Source1
+
+    model = Source1.from_pretrained("path/to/Source-1")     # a local directory or a Hugging Face repo id
+    model = Source1.from_pretrained("path/to/Source-1", precision="fp32")  # the full-precision weights instead
+    doc = model.score("Photosynthesis is how plants ...", title="Photosynthesis")
+    doc["overall"], doc["keep"], doc["educational_value"]
+    docs = model.score_batch(["first document", {"text": "second document", "title": "A title"}])
+
+    python source1.py --input docs.jsonl --output scores.jsonl     # one JSON object per line, "text" field
+    python source1.py --input page.txt                             # a .txt file is one document
+
+Output: one flat dict per document
+----------------------------------
+format, topic, content_type    labels: the most likely value (10, 15 and 4 values; see source1.json)
+educational_value, reasoning_depth, writing_quality, information_density, reliability
+                               quality scores 0-5, higher is better
+spam_seo, boilerplate, toxicity
+                               red flags 0-5, higher is worse
+code_quality, math_quality     gated scores 0-5, or None when the document is not code / not math
+overall                        composite 0-5: weighted quality, blended with the gated scores that apply, minus
+                               red-flag penalties (see ``composite``)
+keep                           False when the drop line of calibration.json matches (see ``DropLine``)
+drop_reasons                   the drop-line conditions that matched ([] when kept)
+parts, tokens, truncated       chunks the document was split into, its length in tokens, whether any chunk was
+                               longer than the model's 8,192 tokens and was cut
+chunks                         only when parts > 1: the same fields per chunk, with its part number, its character
+                               span in the cleaned text (``clean_text``) and its token count
+label_dist, ranges             only when parts > 1: each label value's token share; each score's [min, max]
+
+Scores are expected values (the six levels 0-5 weighted by their probabilities), so they are fractional, rounded
+to 3 decimals like every other number here. The scores are the model's own; ``apply_offsets=True`` adds the small
+calibration offsets of calibration.json to the quality scores (about 0.02 at most; off by default).
+
+How a document is scored (the same steps the model was trained and evaluated with)
+------------------------------------------------------------------------------------
+1. ``clean_text``: line endings to "\\n", control characters dropped, Unicode NFC, trailing spaces dropped, at most
+   two blank lines in a row.
+2. ``split_text``: a document longer than 7,808 tokens is split into balanced chunks, each ending at the most
+   natural boundary near its ideal end (headings, then paragraphs, lines, sentences, spaces; definitions in code).
+3. ``build_input``: each chunk gets a one-line header, a blank line, then the chunk text:
+       Source: dataset record | Title: <title> | Part 2 of 3 of a longer document
+   Every training input had a Source line, almost always "dataset record", so that is the default; code files had
+   "<language> source file" (pass ``code_language="Python"``). Title is added when given, Part when the document has
+   more than one chunk. ``url`` is accepted but not shown to the model unless ``show_url=True``: no training input had
+   one. On 1,261 held-out benchmark chunks (495 graded blind-set chunks from the test split and 766 exam chunks),
+   dropping the whole header moved overall by 0.03 on average (at most 0.655), dropping a title by 0.05 on the chunks
+   that had one; adding a URL moved it by up to 0.7 and did not improve its rank agreement with the independent
+   graders of the model card's evaluation.
+4. ``collapse_spaces``: runs of spaces and tabs become one space; at most two empty lines in a row.
+5. Tokenized with <bos> and <eos>, at most 8,192 tokens (a longer input is cut at its end).
+6. The final hidden states are mean-pooled over the tokens; one linear head per field.
+7. A document's chunks are combined by token-weighted vote (labels) and token-weighted mean (scores; toxicity takes
+   the maximum; gated scores average over the chunks where they apply). ``overall`` and ``keep`` are then computed
+   on the combined scores.
+
+Weights and precision
+---------------------
+Two copies of the backbone weights: ``model.safetensors`` in bfloat16 (the default, ``precision="bf16"``, half the
+size) and ``model.fp32.safetensors`` in float32 (``precision="fp32"``, the full-precision copy). ``precision`` picks
+the file; ``dtype`` picks what the model computes in. By default (``dtype="auto"``) it computes in bfloat16 on a GPU
+with native bfloat16 (NVIDIA Ampere and newer), as in the project's own evaluation, and in float32 on older GPUs and on
+a CPU, where bfloat16 weights are upcast to float32 (pass ``dtype="bf16"`` to compute in bfloat16 there too). Computing
+in bfloat16, both files give the same scores, because the bfloat16 file holds exactly the float32 weights rounded to
+bfloat16. Computing in float32, the bfloat16 weights move scores slightly away from the float32 weights' (on the same
+1,261 benchmark chunks: up to 0.03 on overall and 0.08 on a single field, 3 labels and 1 keep decision changed); use
+``precision="fp32", dtype="fp32"`` for full float32. float16 is refused: the
+mean pooling overflows its range on long inputs and gives NaN scores. In bfloat16 a chunk's scores depend slightly on
+which other inputs share its batch (scoring each benchmark chunk alone instead of in the default batches moved overall
+by up to 0.04 and a single field by up to 0.10; 3 labels changed, no keep decision), and float32 differs from bfloat16
+by a similar amount (up to 0.03 on overall and 0.09 on a single field; 4 labels and 1 keep decision changed); for
+scores that do not depend on the batch, compute in float32 (``dtype="fp32"``) or use ``batch_tokens=1`` (one input per
+forward pass).
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import json
+import math
+import operator
+import re
+import sys
+import time
+import unicodedata
+from bisect import bisect_left
+from collections.abc import Iterable, Iterator, Mapping
+from pathlib import Path
+from typing import Any
+
+import torch
+import torch.nn as nn
+
+__version__ = "1.0.0"
+
+MAX_LENGTH = 8192  # tokens per model input, <bos> and <eos> included
+CHUNK_TOKENS = 7808  # document tokens per chunk: 8,192 minus 384 kept for the header (as in training)
+DEFAULT_SOURCE = "dataset record"
+DEFAULT_BATCH_TOKENS = 65536  # padded tokens per forward pass
+LEVELS = (0, 1, 2, 3, 4, 5)
+# The backbone weights of each precision: bfloat16 (the default) and the full-precision float32 copy. The fp32 file
+# follows transformers' variant naming (model.<variant>.safetensors), so AutoModel loads it with variant="fp32".
+WEIGHTS = {"bf16": "model.safetensors", "fp32": "model.fp32.safetensors"}
+DEFAULT_PRECISION = "bf16"
+FILES = ("config.json", "heads.safetensors", "source1.json", "tokenizer.json")  # needed besides the weights
+CALIBRATION = "calibration.json"  # the calibrated drop line and offsets; required unless drop_line is given
+# What from_pretrained also downloads for a Hub repo id (with the weights of the chosen precision only): the
+# calibration and the license files.
+HUB_EXTRA = (CALIBRATION, "tokenizer_config.json", "LICENSE", "NOTICE", "AUTHORS", "CREDITS_BOOKS.tsv")
+
+
+def hub_files(precision: str = DEFAULT_PRECISION) -> tuple[str, ...]:
+    """The files from_pretrained downloads from a Hugging Face repo for ``precision``."""
+    return (WEIGHTS[resolve_precision(precision)], *FILES, *HUB_EXTRA)
+
+# --------------------------------------------------------------------------------------------- text
+
+
+_JUNK = re.compile("[\x00-\x08\x0b\x0e-\x1f\x7f\ud800-\udfff\ufeff\u200b\ufffe\uffff]")
+_TRAILING_WS = re.compile(r"[ \t]+\n")
+_BLANK_LINES = re.compile(r"\n{4,}")
+_SPACE_RUN = re.compile(r"[ \t]{2,}")
+_BLANK_RUN = re.compile(r"\n(?:[ \t]*\n){3,}")
+
+
+def clean_text(text: str) -> str:
+    """The document as the scorer sees it before chunking: "\\n" line endings (a form feed counts as a paragraph
+    break), control characters, zero-width spaces and byte-order marks dropped, Unicode NFC, no trailing spaces,
+    at most two blank lines in a row, no blank lines at either end."""
+    text = text.replace("\r\n", "\n").replace("\r", "\n").replace("\x0c", "\n\n")
+    text = _JUNK.sub("", text)
+    text = unicodedata.normalize("NFC", text)
+    text = _TRAILING_WS.sub("\n", text)
+    text = _BLANK_LINES.sub("\n\n\n", text)
+    return text.strip("\n").rstrip()
+
+
+def collapse_spaces(text: str) -> str:
+    """The model's input normalization: runs of 2+ spaces/tabs become one space, 3+ blank lines in a row (lines
+    holding only spaces or tabs) become two empty lines; newlines are kept."""
+    return _BLANK_RUN.sub("\n\n\n", _SPACE_RUN.sub(" ", text))
+
+
+_SURROGATES = re.compile("[\ud800-\udfff]")
+
+
+def _from_bytes(x: Any) -> Any:
+    """bytes and bytearray decoded as UTF-8 (invalid bytes become U+FFFD); anything else unchanged."""
+    return x.decode("utf-8", errors="replace") if isinstance(x, (bytes, bytearray)) else x
+
+
+def _one_line(s: Any, limit: int) -> str:
+    """A header value on one line: lone surrogates (which cannot be tokenized) dropped, every run of whitespace
+    (newlines included) made one space, cut to ``limit`` characters."""
+    s = re.sub(r"\s+", " ", _SURROGATES.sub("", str(_from_bytes(s)))).strip()
+    return s if len(s) <= limit else s[: limit - 1] + "\u2026"
+
+
+def source_description(source_type: str | None = None, code_language: str | None = None) -> str:
+    """The header's Source value: ``source_type`` when given ("" leaves the Source line out), else
+    "<code_language> source file" for code, else "dataset record"."""
+    if source_type is not None:
+        return str(source_type)
+    if code_language:
+        return f"{code_language} source file"
+    return DEFAULT_SOURCE
+
+
+def build_input(chunk: str, *, source_type: str | None = DEFAULT_SOURCE, title: str | None = None,
+                url: str | None = None, part: int = 1, parts: int = 1) -> str:
+    """One chunk as the model reads it (before ``collapse_spaces``): a header line, a blank line, the chunk.
+
+    >>> build_input("Text.", title="On  rivers", part=2, parts=3)
+    'Source: dataset record | Title: On rivers | Part 2 of 3 of a longer document\\n\\nText.'
+    """
+    head = []
+    if source_type:
+        head.append(f"Source: {_one_line(source_type, 200)}")
+    if title:
+        head.append(f"Title: {_one_line(title, 200)}")
+    if url:
+        head.append(f"URL: {_one_line(url, 300)}")
+    if parts > 1:
+        head.append(f"Part {part} of {parts} of a longer document")
+    line = " | ".join(head)
+    return f"{line}\n\n{chunk}" if line else chunk
+
+
+# --------------------------------------------------------------------------------------------- chunking
+
+# Cost of splitting at each boundary level; the distance from the ideal point (as a fraction of the target
+# chunk size) is added to it.
+_LEVEL_COST = (0.0, 0.12, 0.3, 0.5, 0.8)
+_HARD_CUT_COST = 2.0
+_HEADING = re.compile(
+    r"\n(?=#{1,6} |(?:chapter|CHAPTER|Chapter|PART|Part|BOOK|Book|SECTION|Section|ACT|Act"
+    r"|Kapitel|KAPITEL|Chapitre|CHAPITRE|Cap[ií]tulo|CAP[IÍ]TULO|Capitolo|CAPITOLO|Глава|ГЛАВА|Rozdział|ROZDZIAŁ)\b[^\n]{0,80}\n"
+    r"|第[一二三四五六七八九十百千〇零0-9]+[章节節回卷部篇][^\n]{0,80}\n"
+    r"|[=\-*_]{3,}[ \t]*\n|\x0c|\\(?:chapter|section|subsection)\b)"
+)
+_CODE_DEF = re.compile(
+    r"\n(?=(?:def |async def |class |function |func |fn |pub |impl |struct |enum |interface |trait |type |"
+    r"module |package |public |private |protected |internal |static |export |const |let |var |@|#include|# ?%%))"
+)
+_PARAGRAPH = re.compile(r"\n[ \t]*\n+")
+_LINE = re.compile(r"\n")
+_SENTENCE = re.compile(r"[.!?…।॥۔؟։።။។៕][\"'”’)\]»]*\s+|[。！？][」』”’）\]]*")
+_SPACE = re.compile(r"\s+")
+
+
+def _joins_previous(ch: str) -> bool:
+    """Characters that belong to the one before them: combining marks, ZWJ, variation selectors, skin tones."""
+    o = ord(ch)
+    return (unicodedata.category(ch) in ("Mn", "Mc", "Me") or o == 0x200D or 0xFE00 <= o <= 0xFE0F
+            or 0xE0100 <= o <= 0xE01EF or 0x1F3FB <= o <= 0x1F3FF)
+
+
+def split_text(text: str, offsets: list[int], max_tokens: int = CHUNK_TOKENS,
+               is_code: bool = False) -> list[tuple[int, int, int]]:
+    """Balanced chunks of ``text``: ``(start_char, end_char, tokens)`` each, ``offsets`` being the start character
+    of every token. A 9k-token document becomes two ~4.5k chunks, not 7.8k plus 1.2k; each chunk ends at the
+    cheapest boundary near its ideal end (headings or code definitions, paragraphs, lines, sentences, spaces), with
+    a hard cut only as a last resort, never inside a character's combining marks or an emoji sequence."""
+    n = len(offsets)
+    if not text:
+        return []
+    if n <= max_tokens:
+        return [(0, len(text), n)]
+    levels = [_CODE_DEF if is_code else _HEADING, _PARAGRAPH, _LINE, None if is_code else _SENTENCE, _SPACE]
+    spans: list[tuple[int, int, int]] = []
+    s_tok, s_char = 0, 0
+    while n - s_tok > max_tokens:
+        remaining = n - s_tok
+        k = math.ceil(remaining / max_tokens)
+        target = remaining / k
+        ideal = s_tok + target
+        hard = s_tok + max_tokens  # the chunk ends at or before token `hard`
+        lo_tok = max(s_tok + max(1, int(target * 0.5)), n - (k - 1) * max_tokens)
+        lo_char, hi_char = int(offsets[lo_tok]), int(offsets[hard])
+        best: tuple[float, int, int] | None = None
+        for level, pattern in enumerate(levels):
+            if best is not None and _LEVEL_COST[level] >= best[0]:
+                break  # nothing at this level or later can beat the current best
+            if pattern is None:
+                continue
+            for m in pattern.finditer(text, max(lo_char - 1, 0), min(len(text), hi_char + 256)):
+                pos = m.end()
+                if not lo_char <= pos <= hi_char:
+                    continue
+                tok = bisect_left(offsets, pos)
+                if not s_tok < tok <= hard:
+                    continue
+                cost = _LEVEL_COST[level] + abs(tok - ideal) / target
+                if best is None or cost < best[0]:
+                    best = (cost, pos, tok)
+        if best is None or best[0] >= _HARD_CUT_COST:
+            c, t = hi_char, hard
+            while c > s_char + 1 and c < len(text) and (_joins_previous(text[c]) or text[c - 1] == "\u200d"):
+                c -= 1
+            if c != hi_char:
+                t2 = bisect_left(offsets, c)
+                if t2 > s_tok:
+                    t = t2
+                else:
+                    c = hi_char
+            best = (_HARD_CUT_COST, c, t)
+        _, split_char, split_tok = best
+        spans.append((s_char, split_char, split_tok - s_tok))
+        s_tok, s_char = split_tok, split_char
+    spans.append((s_char, len(text), n - s_tok))
+    return spans
+
+
+def select_chunks(n: int, max_chunks: int) -> list[int]:
+    """Indices of at most ``max_chunks`` evenly spaced chunks out of ``n`` (0 keeps all)."""
+    if max_chunks <= 0 or n <= max_chunks:
+        return list(range(n))
+    if max_chunks == 1:
+        return [n // 2]
+    last = n - 1
+    return sorted({round(i * last / (max_chunks - 1)) for i in range(max_chunks)})
+
+
+# --------------------------------------------------------------------------------------------- scores
+
+_CMP = {ast.Eq: operator.eq, ast.NotEq: operator.ne, ast.Lt: operator.lt, ast.LtE: operator.le,
+        ast.Gt: operator.gt, ast.GtE: operator.ge}
+
+
+class DropLine:
+    """A drop line such as ``toxicity >= 4 or spam_seo >= 3.5 or boilerplate >= 4.5``: field names, numbers,
+    comparisons, ``and`` / ``or`` / ``not`` and parentheses. A comparison with a missing score (None) is false.
+    A chunk or document is kept when the line does not match."""
+
+    _NODES = (ast.Expression, ast.BoolOp, ast.And, ast.Or, ast.UnaryOp, ast.Not, ast.USub, ast.Compare, ast.Name,
+              ast.Load, ast.Constant, *_CMP)
+
+    def __init__(self, source: str, names: Iterable[str]):
+        self.source = source.strip()
+        tree = ast.parse(self.source, mode="eval")
+        for node in ast.walk(tree):
+            if not isinstance(node, self._NODES):
+                raise ValueError(f"{type(node).__name__} is not allowed in a drop line: {source!r}")
+            if isinstance(node, ast.Name) and node.id not in set(names):
+                raise ValueError(f"unknown name {node.id!r} in drop line {source!r}")
+        self.tree = tree.body
+        top_or = isinstance(self.tree, ast.BoolOp) and isinstance(self.tree.op, ast.Or)
+        self.terms = list(self.tree.values) if top_or else [self.tree]
+
+    def reasons(self, env: dict) -> list[str]:
+        """The terms of the line that match ``env`` (each ``or`` branch on its own); [] means keep."""
+        return [ast.unparse(t) for t in self.terms if self._eval(t, env)]
+
+    def _eval(self, node: ast.AST, env: dict) -> Any:
+        if isinstance(node, ast.Constant):
+            return node.value
+        if isinstance(node, ast.Name):
+            return env.get(node.id)
+        if isinstance(node, ast.BoolOp):
+            value: Any = isinstance(node.op, ast.And)
+            for v in node.values:
+                value = self._eval(v, env)
+                if bool(value) != isinstance(node.op, ast.And):
+                    return value
+            return value
+        if isinstance(node, ast.UnaryOp):
+            v = self._eval(node.operand, env)
+            if isinstance(node.op, ast.Not):
+                return not v
+            return None if v is None else -v
+        if isinstance(node, ast.Compare):
+            left = self._eval(node.left, env)
+            for op, comp in zip(node.ops, node.comparators):
+                right = self._eval(comp, env)
+                if not isinstance(op, (ast.Eq, ast.NotEq)) and (left is None or right is None):
+                    return False
+                try:
+                    if not _CMP[type(op)](left, right):
+                        return False
+                except TypeError:
+                    return False
+                left = right
+            return True
+        raise ValueError(f"unsupported drop-line node {type(node).__name__}")
+
+
+def _r3(x: float | None) -> float | None:
+    return None if x is None or (isinstance(x, float) and math.isnan(x)) else round(float(x), 3)
+
+
+def composite(schema: dict, scores: dict) -> float | None:
+    """The overall score (0-5) of a flat score dict, by the schema in source1.json: the weighted mean of the quality
+    scores; when gated scores apply, 80% of that plus 20% of their mean; minus, for each red flag, its penalty
+    weight times how far it is above its threshold (spam_seo 0.5, boilerplate 0.4, toxicity 0.8, each above 1);
+    clipped to 0-5."""
+    q_num = q_den = 0.0
+    for name, spec in schema["quality"].items():
+        v, w = scores.get(name), float(spec.get("weight", 1.0))
+        if v is not None and w > 0:
+            q_num += w * float(v)
+            q_den += w
+    if q_den <= 0:
+        return None
+    quality = q_num / q_den
+    gated = [float(scores[n]) for n in schema.get("gated", {}) if scores.get(n) is not None]
+    gw = float((schema.get("composite") or {}).get("gated_weight", 0.2))
+    if gated and gw > 0:
+        quality = (1 - gw) * quality + gw * (sum(gated) / len(gated))
+    penalty = 0.0
+    for name, spec in (schema.get("red_flags") or {}).items():
+        v, pen = scores.get(name), spec.get("penalty") or {}
+        if v is not None and float(pen.get("weight", 0.0)) > 0:
+            penalty += float(pen["weight"]) * max(0.0, float(v) - float(pen.get("threshold", 0.0)))
+    return round(min(5.0, max(0.0, quality - penalty)), 3)
+
+
+def gate_applies(schema: dict, name: str, labels: dict) -> bool:
+    """Whether gated score ``name`` applies, given the labels: any of its ``applies_when`` labels has one of the
+    listed values (code_quality: code content or a code file; math_quality: math-heavy content or the math topic)."""
+    for label, values in schema["gated"][name]["applies_when"].items():
+        values = [values] if isinstance(values, str) else values
+        if labels.get(label) in values:
+            return True
+    return False
+
+
+def aggregate(schema: dict, items: list[tuple[int, dict]]) -> dict:
+    """Document scores from ``(tokens, chunk_scores)`` pairs: labels by token-weighted vote, scores by token-weighted
+    mean (or max / min where the schema says so: toxicity uses max), gated scores over the chunks where they apply.
+    With several chunks also ``label_dist`` (token share of each label value) and ``ranges`` ([min, max] per score)."""
+    out: dict = {}
+    dist_out: dict = {}
+    ranges: dict = {}
+    total_w = sum(max(w, 1) for w, _ in items)
+    for name in schema["labels"]:
+        weights: dict[str, float] = {}
+        for w, sc in items:
+            v = sc.get(name)
+            if v is not None:
+                weights[v] = weights.get(v, 0.0) + max(w, 1)
+        if not weights:
+            out[name] = None
+            continue
+        dist = {k: round(v / total_w, 3) for k, v in sorted(weights.items(), key=lambda kv: -kv[1])}
+        out[name] = next(iter(dist))
+        if len(items) > 1:
+            dist_out[name] = dist
+    for group in ("quality", "red_flags", "gated"):
+        for name, spec in (schema.get(group) or {}).items():
+            vals = [(float(v), max(w, 1)) for w, sc in items if (v := sc.get(name)) is not None]
+            if not vals:
+                out[name] = None
+                continue
+            how = spec.get("aggregate", "mean")
+            if how == "max":
+                agg = max(v for v, _ in vals)
+            elif how == "min":
+                agg = min(v for v, _ in vals)
+            else:
+                agg = sum(v * w for v, w in vals) / sum(w for _, w in vals)
+            out[name] = _r3(agg)
+            if len(vals) > 1:
+                ranges[name] = [_r3(min(v for v, _ in vals)), _r3(max(v for v, _ in vals))]
+    if dist_out:
+        out["label_dist"] = dist_out
+    if ranges:
+        out["ranges"] = ranges
+    return out
+
+
+# --------------------------------------------------------------------------------------------- model
+
+_DTYPES = {"bfloat16": torch.bfloat16, "bf16": torch.bfloat16, "float32": torch.float32, "fp32": torch.float32}
+DTYPE_CHOICES = ("auto", "bf16", "bfloat16", "fp32", "float32")
+PRECISION_CHOICES = ("bf16", "fp32")
+_FP16 = ("float16 is not supported: summing the hidden states for mean pooling overflows float16's range on long "
+         "inputs and gives NaN scores. Use dtype='bf16' (GPUs from NVIDIA Ampere on) or dtype='fp32' (any device).")
+_FP16_WEIGHTS = ("there are no float16 weights, and float16 is not supported (summing the hidden states for mean "
+                 "pooling overflows its range on long inputs and gives NaN scores). Use precision='bf16' (the "
+                 "default, model.safetensors) or precision='fp32' (model.fp32.safetensors).")
+
+
+def resolve_precision(precision: Any = None) -> str:
+    """Which weights to load, "bf16" (model.safetensors, the default) or "fp32" (model.fp32.safetensors): None,
+    "bf16" / "bfloat16" / torch.bfloat16, or "fp32" / "float32" / torch.float32. float16 raises ValueError."""
+    if precision is None:
+        return DEFAULT_PRECISION
+    if isinstance(precision, str):
+        key = precision.lower().removeprefix("torch.")
+        if key in ("float16", "fp16", "half"):
+            raise ValueError(_FP16_WEIGHTS)
+        if key in ("bf16", "bfloat16"):
+            return "bf16"
+        if key in ("fp32", "float32"):
+            return "fp32"
+        raise ValueError(f"unknown precision {precision!r}: use 'bf16' (the default) or 'fp32'")
+    if precision == torch.float16:
+        raise ValueError(_FP16_WEIGHTS)
+    if precision == torch.bfloat16:
+        return "bf16"
+    if precision == torch.float32:
+        return "fp32"
+    raise ValueError(f"unknown precision {precision!r}: use 'bf16' (the default) or 'fp32'")
+
+
+def stored_dtype(path: str | Path) -> str | None:
+    """The dtype a safetensors file stores its tensors in ("BF16", "F32", ...; "mixed" when several), read from its
+    header; None when the header cannot be read."""
+    try:
+        with open(path, "rb") as f:
+            n = int.from_bytes(f.read(8), "little")
+            if not 0 < n <= 100_000_000:
+                return None
+            header = json.loads(f.read(n))
+    except (OSError, ValueError):
+        return None
+    kinds = {v.get("dtype") for k, v in header.items() if k != "__metadata__" and isinstance(v, dict)}
+    return None if not kinds else kinds.pop() if len(kinds) == 1 else "mixed"
+
+
+def _native_bf16(device: torch.device) -> bool:
+    """Whether ``device`` is a GPU that runs bfloat16 natively (not emulated)."""
+    if device.type != "cuda":
+        return False
+    try:
+        with torch.cuda.device(device):
+            return bool(torch.cuda.is_bf16_supported(including_emulation=False))
+    except TypeError:  # a torch without including_emulation
+        return torch.cuda.get_device_capability(device)[0] >= 8
+
+
+def resolve_dtype(dtype: Any, device: str | torch.device) -> torch.dtype:
+    """The dtype to run in: None or "auto" = bfloat16 on a GPU with native bfloat16, float32 everywhere else;
+    "bf16" / "bfloat16" / torch.bfloat16 or "fp32" / "float32" / torch.float32 as given. float16 raises ValueError."""
+    device = torch.device(device)
+    if dtype is None or (isinstance(dtype, str) and dtype.lower() == "auto"):
+        return torch.bfloat16 if _native_bf16(device) else torch.float32
+    if isinstance(dtype, str):
+        key = dtype.lower().removeprefix("torch.")
+        if key in ("float16", "fp16", "half"):
+            raise ValueError(_FP16)
+        if key not in _DTYPES:
+            raise ValueError(f"unknown dtype {dtype!r}: use 'auto', 'bf16' or 'fp32'")
+        return _DTYPES[key]
+    if dtype == torch.float16:
+        raise ValueError(_FP16)
+    if dtype not in (torch.bfloat16, torch.float32):
+        raise ValueError(f"unsupported dtype {dtype}: use torch.bfloat16 or torch.float32")
+    return dtype
+
+
+def _dtype_kwarg() -> str:
+    """transformers 4.56 renamed from_pretrained(torch_dtype=...) to dtype=..."""
+    import transformers
+
+    major, minor = (int("".join(c for c in p if c.isdigit()) or 0) for p in transformers.__version__.split(".")[:2])
+    return "dtype" if (major, minor) >= (4, 56) else "torch_dtype"
+
+
+_REPO_ID = re.compile(r"[A-Za-z0-9][\w.-]*/[\w.-]+", re.ASCII)
+
+
+def _resolve_dir(path_or_repo: str | Path, revision: str | None = None,
+                 precision: str = DEFAULT_PRECISION) -> Path:
+    """A local directory as is; a Hugging Face repo id ("owner/name") downloaded with huggingface_hub (only the files
+    source1.py needs, with the weights of ``precision`` only, at ``revision`` when given). Anything else raises
+    FileNotFoundError, never a download."""
+    s = str(path_or_repo)
+    path = Path(s).expanduser()
+    if path.is_dir():
+        if revision is not None:
+            raise ValueError("revision= applies to a Hugging Face repo id, not to a local directory")
+        return path
+    if path.exists():
+        raise FileNotFoundError(f"{s} is a file; pass the Source-1 directory that holds it")
+    is_repo_id = (isinstance(path_or_repo, str) and _REPO_ID.fullmatch(s) is not None
+                  and not s.startswith((".", "~", "/")) and not Path(s.split("/")[0]).exists())
+    if not is_repo_id:
+        raise FileNotFoundError(f"{s}: no such Source-1 directory")
+    from huggingface_hub import snapshot_download  # installed with transformers
+
+    return Path(snapshot_download(repo_id=s, revision=revision, allow_patterns=list(hub_files(precision))))
+
+
+def _load_calibration(path: Path, drop_line: str | None, apply_offsets: bool) -> dict | None:
+    """calibration.json of a Source-1 directory. It is required for the calibrated drop line (the default) and for
+    ``apply_offsets``; a missing or incomplete file then raises FileNotFoundError instead of silently falling back."""
+    cal_path = path / CALIBRATION
+    calibration = json.loads(cal_path.read_text(encoding="utf-8")) if cal_path.exists() else None
+    has_line = bool(((calibration or {}).get("drop_line") or {}).get("line"))
+    if drop_line in (None, "calibrated") and not has_line:
+        what = "has no drop line" if calibration is not None else "is missing"
+        raise FileNotFoundError(
+            f"{cal_path} {what}, and the default drop line comes from it: download it again, or pass "
+            "drop_line='default' (the rubric's own hard filters) or your own drop line to load without it")
+    if apply_offsets and not (calibration or {}).get("offsets"):
+        raise FileNotFoundError(f"{cal_path} is missing or has no offsets, which apply_offsets=True needs")
+    return calibration
+
+
+class Source1(nn.Module):
+    """Source-1: the mmBERT-base encoder, mean pooling and one linear head per field. Build it with
+    ``Source1.from_pretrained``; score documents with ``score`` / ``score_batch``."""
+
+    def __init__(self, backbone: nn.Module, tokenizer: Any, config: dict, calibration: dict | None = None,
+                 drop_line: str | None = None, apply_offsets: bool = False, show_url: bool = False):
+        super().__init__()
+        self.backbone = backbone
+        self.tok = tokenizer
+        self.config = config
+        self.schema = config["schema"]
+        self.layout = [tuple(x) for x in config["layout"]]
+        if any(kind not in ("label_single", "score") for _, kind, _ in self.layout):
+            raise ValueError("this scorer handles single-choice labels and 0-5 scores only")
+        if config.get("pooling", "mean") != "mean" or config.get("text_normalize") not in (None, "collapse_spaces"):
+            raise ValueError("unexpected source1.json: this scorer expects mean pooling and collapse_spaces")
+        self.normalize = collapse_spaces if config.get("text_normalize") else (lambda s: s)
+        hidden = int(backbone.config.hidden_size)
+        self.heads = nn.ModuleDict({name: nn.Linear(hidden, size) for name, _, size in self.layout})
+        self.max_length = int(config.get("max_length") or MAX_LENGTH)
+        self.labels = {name: list(spec["values"]) for name, spec in self.schema["labels"].items()}
+        self.fields = list(self.labels) + [n for g in ("quality", "red_flags", "gated") for n in self.schema.get(g, {})]
+        self.calibration = calibration or {}
+        default_line = " or ".join((self.schema.get("composite") or {}).get("hard_filters") or []) or "False"
+        if drop_line in (None, "calibrated"):
+            drop_line = (self.calibration.get("drop_line") or {}).get("line")
+            if not drop_line:
+                raise ValueError("no calibrated drop line (calibration.json missing or incomplete); pass "
+                                 "drop_line='default' or your own drop line")
+        elif drop_line == "default":
+            drop_line = default_line
+        self.drop_line = DropLine(drop_line, set(self.fields) | {"overall", "tokens", "parts"})
+        self.offsets = {k: float(v["offset"]) for k, v in (self.calibration.get("offsets") or {}).items()}
+        self.apply_offsets = apply_offsets
+        self.show_url = show_url
+        self.precision: str | None = None  # set by from_pretrained: "bf16" or "fp32", the weights it loaded
+        self.weights_file: str | None = None
+        self.weights_dtype: str | None = None  # how that file stores its tensors ("BF16", "F32")
+        ids = tokenizer.encode("", add_special_tokens=True).ids
+        if len(ids) != 2:
+            raise ValueError("expected the tokenizer to add exactly <bos> and <eos>")
+        self.bos_id, self.eos_id = ids
+        self.pad_id = int(tokenizer.token_to_id("<pad>") if tokenizer.token_to_id("<pad>") is not None else 0)
+
+    # ----------------------------------------------------------------------------------------- loading
+
+    @classmethod
+    def from_pretrained(cls, path_or_dir: str | Path, device: str | None = None, dtype: Any = None, *,
+                        precision: Any = DEFAULT_PRECISION, drop_line: str | None = None,
+                        apply_offsets: bool = False, show_url: bool = False, revision: str | None = None,
+                        **backbone_kwargs: Any) -> "Source1":
+        """Load Source-1 from a directory, or from a Hugging Face repo id ("owner/Source-1", downloaded with
+        huggingface_hub; ``revision`` pins a branch, tag or commit).
+
+        device: "cuda", "cuda:1", "cpu", ...; default the GPU when there is one, else the CPU.
+        precision: which weights to load: "bf16" (default) = model.safetensors, bfloat16; "fp32" =
+            model.fp32.safetensors, the full-precision float32 copy (a Hub repo id downloads only the chosen file).
+            float16 raises ValueError.
+        dtype: what the model computes in. None or "auto" (default) = bfloat16 on a GPU with native bfloat16
+            (Ampere and newer), float32 elsewhere (bfloat16 weights are then upcast to float32); or "bf16" / "fp32" /
+            torch.bfloat16 / torch.float32. float16 raises ValueError (it overflows).
+        drop_line: None or "calibrated" = calibration.json's line (the file is then required); "default" = the
+            schema's own hard filters (``toxicity >= 4 or spam_seo >= 4 or boilerplate >= 4.5``); or any expression
+            ``DropLine`` accepts.
+        apply_offsets: add calibration.json's offsets to the quality scores (tiny; off by default).
+        show_url: show ``url`` to the model in the header (it was never trained with one; off by default).
+        backbone_kwargs: passed to transformers' AutoModel.from_pretrained (e.g. attn_implementation="sdpa")."""
+        from safetensors.torch import load_file
+        from tokenizers import Tokenizer
+        from transformers import AutoModel
+
+        precision = resolve_precision(precision)
+        if device is None:
+            device = "cuda" if torch.cuda.is_available() else "cpu"
+        dtype = resolve_dtype(dtype, device)
+        if "variant" in backbone_kwargs:
+            raise TypeError("pass precision='bf16' or precision='fp32' instead of variant=")
+        path = _resolve_dir(path_or_dir, revision, precision)
+        weights = WEIGHTS[precision]
+        if not (path / weights).exists():
+            other = "bf16" if precision == "fp32" else "fp32"
+            have_other = (path / WEIGHTS[other]).exists()
+            raise FileNotFoundError(
+                f"{path} lacks {weights}, the {'float32' if precision == 'fp32' else 'bfloat16'} weights that "
+                f"precision={precision!r} loads" + (f"; precision={other!r} loads {WEIGHTS[other]}, which is there"
+                                                    if have_other else ""))
+        missing = [f for f in FILES if not (path / f).exists()]
+        if missing:
+            raise FileNotFoundError(f"{path} lacks {', '.join(missing)}")
+        config = json.loads((path / "source1.json").read_text(encoding="utf-8"))
+        calibration = _load_calibration(path, drop_line, apply_offsets)
+        tok = Tokenizer.from_file(str(path / "tokenizer.json"))
+        tok.no_truncation()
+        tok.no_padding()
+        variant = {"variant": "fp32"} if precision == "fp32" else {}
+        backbone = AutoModel.from_pretrained(str(path), **{_dtype_kwarg(): dtype}, **variant, **backbone_kwargs)
+        model = cls(backbone, tok, config, calibration, drop_line, apply_offsets, show_url)
+        model.heads.load_state_dict(load_file(str(path / "heads.safetensors")))
+        model.heads.to(dtype)
+        model.precision = precision
+        model.weights_file = weights
+        model.weights_dtype = stored_dtype(path / weights)
+        return model.to(device).eval()
+
+    @property
+    def device(self) -> torch.device:
+        return next(self.backbone.parameters()).device
+
+    # ----------------------------------------------------------------------------------------- inputs
+
+    def chunk(self, text: str, is_code: bool = False, max_tokens: int = CHUNK_TOKENS) -> list[tuple[int, int, int]]:
+        """``split_text`` with this model's tokenizer: (start_char, end_char, tokens) per chunk of a cleaned text."""
+        if not text:
+            return []
+        offsets = [s for s, _ in self.tok.encode(text, add_special_tokens=False).offsets]
+        return split_text(text, offsets, max_tokens, is_code)
+
+    def encode(self, texts: list[str]) -> list[tuple[list[int], bool]]:
+        """Token ids of model inputs (``collapse_spaces`` applied, <bos> ... <eos>, at most max_length tokens: a
+        longer input keeps its first max_length - 1 tokens and its <eos>), each with whether it had to be cut."""
+        out = []
+        for enc in self.tok.encode_batch([self.normalize(t) for t in texts], add_special_tokens=True):
+            ids = enc.ids
+            out.append((ids, False) if len(ids) <= self.max_length else (ids[: self.max_length - 1] + [self.eos_id], True))
+        return out
+
+    # ----------------------------------------------------------------------------------------- scoring
+
+    @torch.inference_mode()
+    def _logits(self, batch: list[list[int]]) -> dict[str, torch.Tensor]:
+        width = max(len(ids) for ids in batch)
+        x = torch.full((len(batch), width), self.pad_id, dtype=torch.long)
+        m = torch.zeros((len(batch), width), dtype=torch.long)
+        for row, ids in enumerate(batch):
+            x[row, : len(ids)] = torch.tensor(ids, dtype=torch.long)
+            m[row, : len(ids)] = 1
+        x, m = x.to(self.device), m.to(self.device)
+        out = self.backbone(input_ids=x, attention_mask=m)
+        hidden = out.last_hidden_state if hasattr(out, "last_hidden_state") else out[0]
+        mask = m.unsqueeze(-1).to(hidden.dtype)
+        pooled = (hidden * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1.0)  # mean over the real tokens
+        pooled = pooled.to(next(iter(self.heads.values())).weight.dtype)
+        return {name: head(pooled) for name, head in self.heads.items()}
+
+    @torch.inference_mode()
+    def _decode(self, logits: dict[str, torch.Tensor]) -> list[dict]:
+        n = next(iter(logits.values())).shape[0]
+        for name, x in logits.items():  # NaN would silently keep a document and break the JSON output
+            finite = torch.isfinite(x).all(dim=-1)
+            if not bool(finite.all()):
+                raise FloatingPointError(
+                    f"head {name!r} gave NaN or infinite outputs for {int((~finite).sum())} of {n} inputs; "
+                    "this should not happen in bfloat16 or float32")
+        rows: list[dict] = [{} for _ in range(n)]
+        levels = torch.tensor(LEVELS, dtype=torch.float32, device=next(iter(logits.values())).device)
+        for name, kind, _ in self.layout:
+            x = logits[name].float()
+            if kind == "label_single":
+                for row, i in enumerate(x.argmax(dim=-1).tolist()):
+                    rows[row][name] = self.labels[name][i]
+            else:
+                expected = (torch.softmax(x, dim=-1) * levels).sum(dim=-1).tolist()
+                for row, e in enumerate(expected):
+                    rows[row][name] = round(e, 3)
+        return rows
+
+    def _finish(self, scores: dict) -> dict:
+        """Gating, optional offsets, overall and keep for one chunk's raw head outputs (a flat dict)."""
+        labels = {k: scores[k] for k in self.labels}
+        out = {**labels}
+        for group in ("quality", "red_flags", "gated"):
+            for name in self.schema.get(group, {}):
+                v = scores[name]
+                if group == "gated" and not gate_applies(self.schema, name, labels):
+                    v = None
+                elif group == "quality" and self.apply_offsets and name in self.offsets:
+                    v = round(min(5.0, max(0.0, v + self.offsets[name])), 3)
+                out[name] = v
+        out["overall"] = composite(self.schema, out)
+        return out
+
+    def _judge(self, rec: dict) -> dict:
+        reasons = self.drop_line.reasons(rec)
+        rec["keep"] = not reasons
+        rec["drop_reasons"] = reasons
+        return rec
+
+    def score_inputs(self, inputs: list[str], batch_tokens: int = DEFAULT_BATCH_TOKENS) -> list[dict]:
+        """Score ready-made model inputs (``build_input`` output: header, blank line, chunk text), one dict per
+        input with the 13 fields, overall, keep, drop_reasons, input_tokens and truncated.
+
+        Inputs are sorted by length and batched with at most ``batch_tokens`` padded tokens per forward pass;
+        a batch that runs out of GPU memory is split in half and retried."""
+        encoded = self.encode(inputs)
+        ids = [e[0] for e in encoded]
+        results: list[dict | None] = [None] * len(ids)
+
+        def run(batch: list[int]) -> None:
+            try:
+                rows = self._decode(self._logits([ids[i] for i in batch]))
+            except torch.cuda.OutOfMemoryError:
+                torch.cuda.empty_cache()
+                if len(batch) == 1:
+                    raise
+                run(batch[: len(batch) // 2])
+                run(batch[len(batch) // 2:])
+                return
+            for i, raw in zip(batch, rows):
+                rec = self._judge(self._finish(raw))
+                rec["input_tokens"] = len(ids[i])
+                rec["truncated"] = encoded[i][1]
+                results[i] = rec
+
+        order = sorted(range(len(ids)), key=lambda i: -len(ids[i]))
+        start = 0
+        while start < len(order):
+            width = len(ids[order[start]])
+            batch = order[start: start + max(1, batch_tokens // max(width, 1))]
+            start += len(batch)
+            run(batch)
+        return results  # type: ignore[return-value]
+
+    def score_batch(self, docs: Iterable[str | bytes | dict], batch_tokens: int = DEFAULT_BATCH_TOKENS, *,
+                    text_field: str = "text", max_chunks: int = 0) -> list[dict]:
+        """Score a list of documents: strings (bytes are decoded as UTF-8), or dicts with the text under
+        ``text_field`` and optionally ``title``, ``url``, ``source_type`` and ``code_language`` (see ``score``).
+        A None text is scored as an empty document (keep False, drop_reasons ["empty text"]). Chunks of all
+        documents are batched together. ``max_chunks`` > 0 scores only that many evenly spaced chunks of a long
+        document (the Part numbers still count every chunk).
+
+        In bfloat16 a document's scores can shift slightly (up to about 0.04 on overall, 0.10 on a single field)
+        depending on which other documents share its batch, because the batch shape changes the kernels' rounding;
+        use dtype="fp32" or batch_tokens=1 when a document's scores must not depend on its neighbours."""
+        if isinstance(docs, (str, bytes, bytearray, Mapping)):
+            raise TypeError("score_batch takes a list of documents; use score() for one document")
+        plans: list[dict] = []
+        inputs: list[str] = []
+        for n, doc in enumerate(docs):
+            if isinstance(doc, Mapping):
+                if text_field not in doc:
+                    raise KeyError(f"document {n} has no {text_field!r} field")
+                d = {k: _from_bytes(v) for k, v in doc.items()}
+            else:
+                d = {text_field: _from_bytes(doc)}
+            raw = d[text_field]
+            if raw is not None and not isinstance(raw, str):
+                raise TypeError(f"document {n}: the text must be a str, bytes or None, not {type(raw).__name__}")
+            text = clean_text(raw or "")
+            code_language = d.get("code_language")
+            source = source_description(d.get("source_type"), code_language)
+            spans = self.chunk(text, is_code=bool(code_language))
+            chosen = select_chunks(len(spans), max_chunks)
+            first = len(inputs)
+            for i in chosen:
+                s, e, _ = spans[i]
+                inputs.append(build_input(text[s:e].strip(), source_type=source, title=d.get("title"),
+                                          url=d.get("url") if self.show_url else None, part=i + 1,
+                                          parts=len(spans)))
+            plans.append({"spans": spans, "chosen": chosen, "first": first})
+        scored = self.score_inputs(inputs, batch_tokens)
+        return [self._document(p, scored[p["first"]: p["first"] + len(p["chosen"])]) for p in plans]
+
+    def score(self, text: str | bytes, title: str | None = None, url: str | None = None, *,
+              source_type: str | None = None, code_language: str | None = None, max_chunks: int = 0,
+              batch_tokens: int = DEFAULT_BATCH_TOKENS) -> dict:
+        """Score one document (a str; bytes are decoded as UTF-8; anything else raises TypeError).
+
+        title: shown to the model in the header when given (as in training, where about a quarter of inputs had one).
+        url: kept out of the model's input unless the model was loaded with show_url=True.
+        source_type: the header's Source value; default "dataset record" ("" leaves the Source line out).
+        code_language: for source code, e.g. "Python": Source becomes "Python source file" and long files are split
+            at definitions."""
+        doc = {"text": text, "title": title, "url": url, "source_type": source_type, "code_language": code_language}
+        return self.score_batch([doc], batch_tokens, max_chunks=max_chunks)[0]
+
+    def _document(self, plan: dict, chunks: list[dict]) -> dict:
+        spans, chosen = plan["spans"], plan["chosen"]
+        if not spans:
+            out = {name: None for name in self.fields}
+            out.update(overall=None, keep=False, drop_reasons=["empty text"], parts=0, tokens=0, truncated=False)
+            return out
+        items = [(spans[i][2], c) for i, c in zip(chosen, chunks)]
+        agg = aggregate(self.schema, items)
+        out = {name: agg[name] for name in self.fields}
+        out["overall"] = composite(self.schema, out)
+        parts, tokens = len(spans), sum(s[2] for s in spans)
+        reasons = self.drop_line.reasons({**out, "parts": parts, "tokens": tokens})
+        out.update(keep=not reasons, drop_reasons=reasons, parts=parts, tokens=tokens,
+                   truncated=any(c["truncated"] for c in chunks))
+        if len(spans) > 1:
+            out["chunks"] = [{"part": i + 1, "start": spans[i][0], "end": spans[i][1], "tokens": spans[i][2], **c}
+                             for i, c in zip(chosen, chunks)]
+            for key in ("label_dist", "ranges"):
+                if key in agg:
+                    out[key] = agg[key]
+        return out
+
+
+# --------------------------------------------------------------------------------------------- CLI
+
+
+class BadInput(ValueError):
+    """A record of the input file that cannot be scored (the message starts with file:line)."""
+
+
+def _read_docs(path: Path, text_field: str, skip_bad: bool = False) -> Iterator[dict]:
+    """Documents of an input file: .jsonl / .ndjson (one JSON object per line), .json (a JSON array of objects, one
+    object, or JSON Lines), or any other file as one plain-text document. Invalid UTF-8 is replaced (with a warning
+    for JSON). A bad record raises BadInput, or with ``skip_bad`` is reported on stderr and skipped. A null text is
+    kept and scored as an empty document."""
+
+    def usable(rec: Any, where: str) -> bool:
+        if not isinstance(rec, dict):
+            problem = f"expected a JSON object, got {type(rec).__name__}"
+        elif text_field not in rec:
+            problem = f"no {text_field!r} field"
+        elif rec[text_field] is not None and not isinstance(rec[text_field], str):
+            problem = f"{text_field!r} is a {type(rec[text_field]).__name__}, not a string"
+        else:
+            return True
+        if not skip_bad:
+            raise BadInput(f"{where}: {problem}")
+        print(f"source1: skipped {where}: {problem}", file=sys.stderr)
+        return False
+
+    def decode(raw: bytes, where: str) -> str:
+        try:
+            return raw.decode("utf-8")
+        except UnicodeDecodeError as e:
+            print(f"source1: {where}: invalid UTF-8 at byte {e.start}; invalid bytes replaced with U+FFFD",
+                  file=sys.stderr)
+            return raw.decode("utf-8", errors="replace")
+
+    suffix = path.suffix.lower()
+    if suffix not in (".jsonl", ".ndjson", ".json"):
+        yield {text_field: path.read_text(encoding="utf-8", errors="replace"), "id": path.name}
+        return
+    if suffix == ".json":
+        try:
+            data = json.loads(decode(path.read_bytes(), str(path)).lstrip(""))
+        except json.JSONDecodeError:
+            data = None  # not a single JSON value: read the file as JSON Lines below
+        if data is not None:
+            for n, rec in enumerate(data if isinstance(data, list) else [data]):
+                if usable(rec, f"{path}[{n}]"):
+                    yield rec
+            return
+    with open(path, "rb") as f:
+        for n, raw in enumerate(f, 1):
+            where = f"{path}:{n}"
+            line = decode(raw, where)
+            if n == 1:
+                line = line.lstrip("")
+            if not line.strip():
+                continue
+            try:
+                rec = json.loads(line)
+            except json.JSONDecodeError as e:
+                if not skip_bad:
+                    raise BadInput(f"{where}: invalid JSON ({e.msg} at column {e.colno})") from None
+                print(f"source1: skipped {where}: invalid JSON ({e.msg} at column {e.colno})", file=sys.stderr)
+                continue
+            if usable(rec, where):
+                yield rec
+
+
+def main(argv: list[str] | None = None) -> int:
+    p = argparse.ArgumentParser(description="Score documents with Source-1 (one JSON object per document).")
+    p.add_argument("--model", default=str(Path(__file__).resolve().parent),
+                   help="Source-1 directory or Hugging Face repo id (default: this file's directory)")
+    p.add_argument("--input", required=True, help=".jsonl (one document per line), .json (an array of objects) or "
+                   "a text file (one document)")
+    p.add_argument("--text-field", default="text", help="JSON field holding the text (default: text); "
+                   "title, url, source_type and code_language fields are used when present")
+    p.add_argument("--output", help="output .jsonl (default: standard output)")
+    p.add_argument("--skip-bad", action="store_true", help="skip (and report on stderr) records that are not valid "
+                   "JSON objects with a string text, instead of stopping")
+    p.add_argument("--revision", help="branch, tag or commit, when --model is a Hugging Face repo id")
+    p.add_argument("--device", help="cpu, cuda, cuda:1, ... (default: cuda when available)")
+    p.add_argument("--precision", choices=PRECISION_CHOICES, default=DEFAULT_PRECISION, help="weights to load: "
+                   "bf16 (default, model.safetensors) or fp32 (model.fp32.safetensors, the full-precision copy)")
+    p.add_argument("--dtype", choices=DTYPE_CHOICES, default="auto", help="what to compute in; auto (default): "
+                   "bfloat16 on a GPU with native bfloat16, else float32 (bf16 weights upcast); float16 is not "
+                   "supported")
+    p.add_argument("--batch-tokens", type=int, default=DEFAULT_BATCH_TOKENS, help="padded tokens per forward pass")
+    p.add_argument("--max-chunks", type=int, default=0, help="score at most N evenly spaced chunks per document")
+    p.add_argument("--drop-line", help='"calibrated" (default), "default" (the schema\'s hard filters) or an '
+                   "expression such as 'toxicity >= 4 or spam_seo >= 3'")
+    p.add_argument("--apply-offsets", action="store_true", help="add the calibration offsets to the quality scores")
+    p.add_argument("--show-url", action="store_true", help="show the url field to the model (untrained)")
+    p.add_argument("--no-chunks", action="store_true", help="leave out the per-chunk list of split documents")
+    p.add_argument("--group", type=int, default=256, help="documents scored together")
+    args = p.parse_args(argv)
+    if not Path(args.input).is_file():
+        p.error(f"--input {args.input}: no such file")
+
+    t0 = time.time()
+    model = Source1.from_pretrained(args.model, device=args.device, dtype=args.dtype, precision=args.precision,
+                                    drop_line=args.drop_line, apply_offsets=args.apply_offsets,
+                                    show_url=args.show_url, revision=args.revision)
+    compute = str(next(model.parameters()).dtype).replace("torch.", "")
+    print(f"source1: loaded {model.weights_file} ({model.weights_dtype or '?'} weights) on {model.device}, computing "
+          f"in {compute}, in {time.time() - t0:.1f} s; drop line: {model.drop_line.source}", file=sys.stderr)
+    out = open(args.output, "w", encoding="utf-8") if args.output else sys.stdout
+    done = 0
+    t0 = time.time()
+
+    def flush(group: list[dict]) -> None:
+        nonlocal done
+        for rec, res in zip(group, model.score_batch(group, args.batch_tokens, text_field=args.text_field,
+                                                     max_chunks=args.max_chunks)):
+            if args.no_chunks:
+                res.pop("chunks", None)
+            if "id" in rec:
+                res = {"id": rec["id"], **res}
+            out.write(json.dumps(res, ensure_ascii=False, allow_nan=False) + "\n")
+        done += len(group)
+        print(f"source1: {done:,} documents, {done / max(time.time() - t0, 1e-9):.1f}/s", file=sys.stderr)
+
+    try:
+        group: list[dict] = []
+        try:
+            for rec in _read_docs(Path(args.input), args.text_field, args.skip_bad):
+                group.append(rec)
+                if len(group) >= args.group:
+                    flush(group)
+                    group = []
+        except BadInput as e:
+            if group:
+                flush(group)  # the documents read before the bad record are still scored and written
+            raise SystemExit(f"source1: {e}. The {done:,} documents before it were written; --skip-bad skips "
+                             "bad records") from None
+        if group:
+            flush(group)
+    finally:
+        if out is not sys.stdout:
+            out.close()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f
+size 34363188

tokenizer_config.json

@@ -0,0 +1,23 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<bos>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "mask_token": "<mask>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "sep_token": "<eos>",
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}