# Decisions

Append-only log. Format: date, title, 1-2 line rationale. Newest first.

## 2026-07-23 — Made Research Assistant and transliteration actually work properly (real bugs found by testing live), added real-time transliteration suggestions, then audited everything live via Playwright

User asked for Research Assistant and transliteration to genuinely work, not just exist - tested both live against the deployed model rather than assuming, and found two real bugs.

**Research Assistant relevance bug**: `search_entries()`'s scoring had no floor - any single query word matching anywhere was enough. Live-tested the UI's own default example query, "డెంగ్యూ వ్యాక్సిన్" (dengue vaccine) - it returned a completely unrelated Ebola article, solely because "vaccine" matched. Fixed by requiring a true majority of the query's distinct words (`len(query_words) // 2 + 1`) to appear in an article before it's even a scoring candidate - re-verified live: the broken case now returns nothing (honest), broad single/multi-word topical queries (Telangana, cricket, police, rain) still return solid matches. Also swapped the default example away from a disease name (inherently tied to today's news cycle, not evergreen) to "తెలంగాణ", and made the empty-result copy explain the live-feeds-only scope instead of reading as broken. **Known residual limitation, not fixed**: word-bag matching still can't tell "a Sharma who plays cricket" from "Rohit Sharma specifically" for a multi-word proper-noun query - true entity/phrase-awareness was judged over-engineering for this POC.

**Transliteration was actually fine on the backend** (live-tested directly via HTTP: `repu`→రేపు, `cheppadu`→చెప్పాడు, all correct, ~1.7s) **but unreachable from Compose**, the primary writing surface - only Quick Tools had it. Added a "Transliterate selection" button to Compose's Grammar-on-Roman-script result (reusing the already-fixed `input_script` detection from the previous session's badge fix), and a genuinely new capability: **real-time suggestions while typing** - a debounced check on the Latin word under the caret (only when there's no active selection, so it doesn't collide with the Grammarly-style floating toolbar) calls the same `/api/transliterate` endpoint for just that one word and shows up to 3 candidates in a small popover, reusing the floating toolbar's existing mirror-div caret-positioning code rather than inventing new positioning logic. Click or Tab accepts; typing on, moving the caret, or Escape dismisses; a request-sequence counter discards stale responses if the user kept typing before a round-trip returned. **Deliberately word-at-a-time on a pause, not a true per-keystroke phonetic IME** - real IME libraries are a much bigger, riskier build than this POC needs, and this gets "suggestions while you type" without it.

**Screenshot-audited all of it against live production via real Playwright browser sessions**, not just API calls - first hit a real blocker: the private Space's full browser navigation 404'd every time even though the same Bearer token worked fine for plain `requests` calls (an HF auth/routing quirk specific to `Sec-Fetch-Mode: navigate` requests on private Spaces, not something worth reverse-engineering further). Worked around it by testing against the local server first (confirms everything renders and degrades gracefully when transliteration is unavailable, as it correctly is on native Windows dev) - then the user independently switched the Space to public mid-session, which incidentally resolved the auth quirk entirely, letting the same Playwright script run directly against production. Confirmed live: Roman-script Grammar's honest badge, the new Transliterate-selection button (verified it actually replaced text, not just that the button appeared), Research Assistant's results, Submit-for-review's status pill, Editor Queue listing a submission with its travelled-along flags, Approve updating status, and the new real-time suggestion popover (typed "repu" → shows రేపు/రెపు/రేపూ → Tab accepts → re-triggers on the next word → a direct chip-click also works → typing an already-Telugu word correctly shows nothing). Zero console errors throughout. One real bug caught was in the *test script itself*, not the app - an early wait-condition matched a loading placeholder that happened to share a CSS class with the real result, so it passed instantly instead of waiting; fixed by waiting for a result-action button instead, which only appears once real data comes back.

## 2026-07-23 — Built a reporter→editor review-queue POC (closing the "no autonomous publishing" gap) + a full-product vision doc

Asked, given a free hand to redesign from a newsroom-writer/editor lens, what a proper editorial tool would need. The dock has always stated "no autonomous publishing... editor approval required" with nothing behind it - built the actual mechanism as a real, working POC, then separately documented the gap to a full product using the PRD's existing Reporter/Editor/Senior-Editor persona table (`Rachana_News_PRD.md` lines 44-49) as the lens. Planned via Plan mode first (multi-file, several real architecture choices) - plan file: `~/.claude/plans/recursive-tinkering-twilight.md`.

**Storage: an append-only JSONL event log (`data/processed/review_events.jsonl`), not a mutable JSON file.** Matches the project's only other server-side-persistence precedent (`usage_feedback.jsonl`) instead of inventing a new pattern. Current status per draft is derived by folding events in file order (latest wins) - this also eliminates any separate "dedup an open record" logic: resubmitting a draft after "changes requested" is just another `submitted` event, folding naturally makes it current.

**Identity: reused the frontend's existing client-generated `draft_id`** (`ui/rachana-desk.html`'s `newDraftId()`) as the only key end to end, rather than inventing a second server-side record id - one less ID-mapping layer for no real payoff at this scale.

**New `server/review.py`** (mirrors the `research.py`/`guardrails.py` "own module, imported into app.py" convention): `submit_draft`/`list_queue`/`get_status`/`approve`/`send_back`, a `threading.Lock` around read-fold-validate-append (approve/send-back need to check current status before acting - a stricter standard than `usage_feedback.jsonl`'s unlocked blind appends, justified because these are read-then-write, not blind appends). 5 new endpoints in `server/app.py`, none touching the model/GPU at all.

**UI: a third peer view, "Editor Queue," next to Compose/Quick Tools** - not a real login-gated role, since this project deliberately has no auth/accounts anywhere. Reused the existing `switchView()` toggle pattern (additive, didn't touch the existing two-view logic) and the existing sidebar-list/main-pane/dock triad exactly as-is (queue items in the sidebar list, read-only detail in the main pane, Approve/Send-back actions in the dock) rather than inventing a new layout convention. Compose mode gained a "Submit for review" button + a status pill (polls `/api/review/status/{draft_id}`). Guardrail signals already produced elsewhere (`needs_review`, `facts_preserved: false`, unsupported-script no-ops) now accumulate onto the active draft as flags (`pushDraftFlag()`, capped at 20) and ride along on submit, so the editor sees them without regenerating anything.

**Scope explicitly stops at human-in-the-loop** - submit/approve/send-back only. No real auth, no CMS write, no autonomous publishing - all flagged as full-product territory in the companion doc, not attempted here. This is not a re-litigation of the PRD's already-separately-scoped Phase 3 (agentic/CMS layer, PRD line 129) - deliberately stays on this side of it.

**Full-product vision doc, both formats per explicit request**: `docs/editorial-workflow-vision.md` (source of truth, versioned) and a published Claude Artifact (https://claude.ai/code/artifact/cdd2a6a1-2ab8-4c1f-af37-25393a57d934) from the same content - covers real RBAC, the CMS-write boundary, a real database, multi-user editing/locking, notifications, audit trail/versioning (and its convergence with the existing `/api/feedback` flywheel data), per-publication style config, ops analytics (distinct from but related to the PRD's existing training-flywheel-convergence metric, line 150), and an explicit note that this is a different system from Rachana Data Studio (PRD line 14/186 - that tool curates training data, this queue moves live drafts).

**Verification, and its real limit**: no GPU or browser access this session. Verified what's actually checkable that way - `server/review.py`'s logic end-to-end via a standalone scratch script (submit→queue→approve, and submit→queue→send_back→resubmit→queue, scratch event file so the real data file was never touched), `server.app` import-safety (confirms `load_model()` only fires on the FastAPI startup event, not at import time, so route registration is checkable without loading the 4B model), a Node syntax check of the updated `<script>` block, and a full DOM-id cross-reference (every `getElementById`/`querySelector` target confirmed present). **Not verified**: an actual browser click-through of Submit → Queue → Approve/Send-back - flagged as the next real thing to do, not claimed as done.

## 2026-07-23 — Cross-checked the newsroom-writer field-report artifact against live code, then fixed 2 of its 3 recommendations

Cross-check found the artifact fully accurate against current `server/*` and `ui/rachana-desk.html` (nothing had drifted - recent commits only touched the separate Gradio demo). User chose to act on the report's own recommendations rather than just re-verify the document. Implemented, without live GPU/browser testing (none available in this session - flagged as the one thing still worth a real click-through):

- **Roman-script "no changes" false reassurance (rec #3)**: `/api/grammar` already returned `input_script`, but the UI never consulted it after generation. Now `factBadgeHtml()` and the grammar diff-card's change-count label both check `input_script === "roman"/"mixed"` and show an explicit "not processed" message instead of a green "Facts preserved" badge / "మార్పులు లేవు" when the model was fed script it can't correct.
- **Headline-degrades-to-a-question (rec #1)**: added a question-mark check to `server/guardrails.py:check_headline_needs_review()` (checked before the existing completed-action check) - a headline ending in "?"/"？" is a reliable proxy for the exact live failure the report reproduced, and reuses the `needs_review`/`review_reason` plumbing already wired through both Quick Tools and Compose headline UI paths. This doesn't fix the model's tendency (that needs training-side work), it flags it for editor review, consistent with this project's existing heuristic-guardrail-not-retraining pattern.
- **Research's silent ~40s wait (rec #2)**: NOT converted to real backend progress (would mean restructuring `/api/research`'s response into a stream - a bigger, riskier change to a live-deployed endpoint than this session could verify without GPU access). Instead added a client-side elapsed-time-based stage ticker (`startResearchStageTicker()`, `RESEARCH_STAGES`) cycling through 4 plausible stage labels timed against the report's real measured ~40s - explicitly documented in-code as an estimate, not real backend state, matching the report's own framing ("without touching the underlying pipeline").

**Not done**: making headline generation itself reliably lead with the real news on multi-sentence input - that's a model quality/training issue, not fixable by a guardrail; the question-mark flag only catches after the fact.

## 2026-07-22 — HF billing set up, `rachana-news` Docker Space fixed and live on paid T4; scoped (not yet run) a larger-n rerun of three thin qualitative spot-checks

**Real production bug found+fixed on the live Docker Space**: `RUNTIME_ERROR` on boot, `libcudart.so.13: cannot open shared object file`. Root cause: `transformers==5.14.1` unconditionally imports `torchaudio` at module load (`modeling_utils.py` → `loss_utils.py` → `loss_rnnt.py`'s Parakeet ASR loss support) even though this project has zero audio use, and the Dockerfile's `fairseq` install step pulls in an *unpinned* transitive `torchaudio` that resolved a build targeting a newer CUDA runtime than the pinned `cu121` torch — raises an unhandled `OSError` (not `ImportError`, so transformers' usual optional-dep guards never catch it), crashing every boot. Fixed by pinning `torchaudio` to the same `cu121` index as `torch`, and re-asserting that exact pin with `--force-reinstall --no-deps` *after* the fairseq install (which is what reintroduces the mismatch). Verified two ways before pushing: local `docker build` + `docker run` got past the crash point into model loading; then confirmed live on the real Space (`hardware.current: "t4-small"`, real Gemma generation returned for a live headline request). See `errors.md`.

**User set up HF billing** (`canPay` flipped `False→True` mid-session) and requested `t4-small` for `KPrashanth/rachana-news` — the Space is now genuinely running on paid GPU, not just Docker-SDK-in-theory. Cost: $0.40/hr, `gcTimeout=1800` (30min idle sleep), so it doesn't bill while idle.

**Did a hands-on "newsroom writer" field test of the live paid product** (not a code read) — drove the real UI via authenticated Playwright (private Space, HF token as a Bearer header works for both API and page loads), walked through a realistic rough reporter's note across every tool. Found one real, reproducible quality issue live: headline generation is excellent on a clean single-fact lead but degrades to a *question* that buries the actual news when given a busier multi-sentence paragraph — matches the general "headline is the highest-variance capability" flag already in `progress.md`, now with a concrete live repro. Also reconfirmed the already-known Roman-script limitation live: the UI's warning banner shows correctly, but if generation proceeds anyway the result comes back labeled "No changes" with a *green* fact-preservation badge — a false-reassurance risk, not just a missed feature. Full report (with real screenshots/transcripts, not mocked): published as a Claude Artifact.

**Scoped, not yet run**: three of this project's earlier qualitative spot-checks were flagged as too-thin-n (`headline_hallucination_sample.jsonl` n=20, Roman-Telugu robustness n=6, Research Assistant queries n=2-3) — user asked to rerun at ~n=50 each. Methodology worked out so it doesn't need re-deriving next session:
- **Headline hallucination @ n=50**: sample fresh `headline_generation` rows from `data/processed/exp3_eval_resample2.jsonl` (400 available, distinct from the original 20), generate via the same local-load pattern as `scripts/eval_adapter.py`, auto-flag with the now-real `server/guardrails.py:check_headline_needs_review()` (didn't exist yet when the original n=20 was hand-reviewed), then manually review flagged + a random sample of clean ones.
- **Roman-Telugu robustness @ n=50**: no persisted Roman-Telugu prompt generator exists from the original n=6 (was ad hoc). Plan: take real Telugu eval sentences, romanize via local Ollama `gemma3:4b`, run through the real production prompts, auto-flag with `check_fact_preservation()` + `check_headline_needs_review()`, manually review flagged + sample.
- **Research Assistant @ n=50 queries**: `server/research.py`'s retrieval is a live-RSS keyword search with no archive/caching (only whatever's in the 4 feeds *right now*) — a naive random 50 topics would mostly return honest empty results and not actually test summarization/citation quality. Plan: fetch live entries once first to see what's actually in today's feeds, design the 50 queries around real current content (plus some deliberate known-mismatches to verify the honest-empty behavior still holds at scale).

Confirmed local RTX 5050 (cu128 torch stack) is ready to run these before pausing. **Paused here by explicit user request** — nothing expensive has run yet, this is a resume point, not a result.

## 2026-07-22 — Deployed to Hugging Face Spaces: git-init, Docker SDK product Space, and a separate ZeroGPU Gradio demo

Deployment goal reframed everything: HF Spaces run Linux containers, so the earlier WSL2/native-Windows venv-consolidation question was moot for this purpose - the Linux-flavored transliteration install already validated in WSL2 is exactly what a container needs.

**Repo git-initialized for the first time** (found a broken, empty `.git/info`-only stub - not real history, safely re-initialized). Real `.gitignore`/`.gitattributes` scope the repo to source + the exp3 adapter's final weights only (via LFS), excluding secrets/venvs/other checkpoints/generated caches - incidentally satisfies the earlier `AUDIT_REPORT.md`'s Finding 4.

**Main product → `KPrashanth/rachana-news` (Docker SDK, private, T4-small requested)**. Added `server/transliterate.py` (thin `XlitEngine` wrapper, imported defensively so local Windows dev keeps working without fairseq installed), a `GET /` route serving `ui/rachana-desk.html` directly, and made the UI's `API_BASE` self-detect `file://` vs served-origin. Built and smoke-tested the Docker image **locally** before ever pushing - caught and fixed three real issues this way: (1) unpinned `pip install torch` resolved bleeding-edge CUDA 13 wheels, newer than most Spaces GPU drivers likely support - pinned to the cu121 index instead; (2) fairseq's build-dependency step hung 5+ min in pip's isolated build env (same class of issue as flash-attn's build-time torch-import problem) - fixed with `--no-build-isolation` + pre-installed `cython`; (3) **the real one**: an unpinned `ai4bharat-transliteration` install resolved a newer `urduhack` (transitive dep, Urdu-only normalization, irrelevant to Telugu) demanding `transformers~=2.10` + `tensorflow~=2.2` (572MB) - caught mid-download locally, before it could downgrade the `transformers==5.14.1` this project needs for Gemma 3. Pinned `ai4bharat-transliteration==1.1.3` + `urduhack==0.1.4` explicitly to force the known-good resolution. GPU hardware request hit `402 Payment Required` - HF requires pre-paid billing credits before attaching paid GPU tiers; left for the user to set up.

**Billing blocker → led to testing ZeroGPU instead** (free/quota-based for HF Pro subscribers, confirmed active on this account). Real architectural mismatch identified before committing: ZeroGPU's `@spaces.GPU` decorator only works with the **Gradio SDK** (hooks into Gradio's own request/queue dispatch), not Docker SDK - incompatible with the main product's custom FastAPI+HTML architecture without a full rewrite.

**Validated via a throwaway spike (`KPrashanth/zerogpu-spike`) before committing to any rewrite** - exactly the right amount of caution, not too much: (1) a trivial `@spaces.GPU` matmul confirmed real GPU allocation works (`NVIDIA RTX PRO 6000 Blackwell ... MIG 2g.48gb`, one transient `uncorrectable ECC error` on an early call, retry succeeded - a hardware flake on HF's shared pool, not a real blocker); (2) the actual risk - whether `bitsandbytes` 4-bit + `PeftModel` loading (tied to a CUDA device at load time) is compatible with ZeroGPU's per-call ephemeral allocation model, and whether a persistent-load pattern survives across calls - **confirmed working**: load once into module-level state outside the decorator, cold call 9.48s (5.6s load + 1.7s gen), warm calls 2.45-2.51s (gen + allocation overhead only). This settled the real uncertainty before any UI porting work began.

**Built and deployed `gradio_demo/` → `KPrashanth/rachana-news-demo`** (Gradio SDK, ZeroGPU, private) covering headline/grammar/paraphrase/translate/research - all five tested live via `gradio_client` and confirmed producing correct real output (e.g. grammar correctly fixed agglutination: "జరుగుతుంది అని" → "జరుగుతుందని"; research returned a real, relevant, correctly-cited TV9 Telugu article for a rain query). **Transliteration deliberately excluded from this Space** - a real, structural constraint, not an oversight: Gradio SDK Spaces build via a single `pip install -r requirements.txt` pass with no ordering control, so there's no way to run the `pip<24.1` step the omegaconf fix needs before fairseq installs (confirmed by an actual failed build attempt, not assumed). `app.py`'s import stays defensive (same pattern as the Docker product) so the tab reports itself unavailable rather than breaking the demo. Helper modules (`guardrails.py`/`research.py`/`transliterate.py`) are deliberately duplicated into `gradio_demo/` rather than imported cross-repo, since this is a separate Space repo from the main product - a documented, accepted maintenance tradeoff for a demo, not an accident.

## 2026-07-22 — Unsloth + Flash-Attention-2 retested in WSL2 — still not viable, closed (not just deferred)

Followed up the IndicXlit win by retesting the other native-Windows rejection (`decisions.md` 2026-07-19: Unsloth was 2.4x *slower* than the proven stack because no FA2 build existed for native-Windows + Blackwell + cu128). Set up an isolated WSL2 venv (`~/venvs/rachana-unsloth-wsl`), same discipline as the Windows `.venv-unsloth`.

**Real mistake caught and fixed along the way**: first attempt installed torch 2.11.0 (this project's main pin) before Unsloth — Unsloth's resolver correctly detected the version conflict (`unsloth` requires `torch<2.11.0`) and started silently downloading a replacement torch 2.10.0 (915MB), exactly the failure mode `errors.md` already warned about. Killed it, reinstalled `torch==2.10.0+cu128` explicitly first (verified with a real GPU matmul, no Blackwell warning), *then* installed Unsloth — this time it correctly saw the pin already satisfied and left torch alone.

**Result: no real speedup available, confirmed via a different root cause than Windows.** No `flash-attn` wheel exists on PyPI for Python 3.10 + torch 2.10 + cu128 (source-only sdist). Its `setup.py` needs `torch` importable at build time (fixable with `--no-build-isolation`), but more fundamentally this machine has no CUDA toolkit (`nvcc`) installed — only pip's bundled CUDA runtime libraries, which isn't enough to compile flash-attn's kernels. Installing the full CUDA toolkit plus a 30min-2hr from-source compile, with no guarantee flash-attn even supports Blackwell/sm_120 yet, was explicitly declined by the user as not worth the time/risk — **the same "ecosystem gap, not a config mistake" verdict as native Windows, just reached by a different path.**

**Decision: Unsloth is closed for this project, not deferred.** Both the native-Windows and WSL2 paths independently converge on "no FA2 for this hardware generation yet." Training stays on the proven native-Windows stack (9.7s/step @ 4B) for any future v0.2 run. Revisit only if flash-attn or the wider ecosystem ships real Blackwell/sm_120 support later — not worth rechecking speculatively.

## 2026-07-21 — IndicXlit transliteration finally works, via WSL2 + Python 3.10 (reverses the native-Windows dead end)

Both native-Windows blockers were Windows/Python-3.12-shaped, not fundamental. Set up WSL2 Ubuntu-24.04 (already installed, GPU passthrough confirmed via real `nvidia-smi` + a real CUDA matmul — not just `torch.cuda.is_available()`) and worked through three distinct blockers in sequence, each a genuinely new finding:

1. `pip install fairseq @ git+...` on the default Ubuntu 24.04 Python (3.12) hit a **dependency-resolution error**, not the Windows symlink issue — modern pip (≥24.1) rejects `omegaconf`'s malformed wheel metadata (`PyYAML>=5.1.*` version specifier). Fixed with `pip install "pip<24.1"` first, exactly as pip's own error message suggested.
2. With pip downgraded, fairseq *installed* but **importing it crashed on Python 3.12**: `ValueError: mutable default ... not allowed: use default_factory` — Python 3.11+ dataclasses enforce a rule fairseq's old code violates. This is almost certainly why the user's referenced Colab example (Python 3.8) worked — **Python version, not OS, was the operative variable all along.** Fixed by installing Python 3.10 via the deadsnakes PPA and rebuilding the venv there (matching this project's existing Windows venv, also Python 3.10).
3. `pip install fairseq` via pip's own internal git clone (`--filter=blob:none`) failed once with `fatal: expected flush after ref listing` — a transient git-protocol hiccup, not a real blocker. Fixed by cloning manually first (`git clone` to a local dir) and `pip install -e <local-clone>` on the full checkout.

**Result — real test, not assumption**: loaded `XlitEngine('te')` and ran real Roman-Telugu words, including the two words that failed in the original 2026-07-21 stress test (`రైతులు`/farmers, `ముఖ్యమంత్రి`/chief minister) — both transliterated correctly this time, along with `namaste`→నమస్తే, `prabhutvam`→ప్రభుత్వం. No wrong-script/garbage output at all (unlike the community fork's dead end). Env lives at `~/venvs/rachana-translit-py310` inside WSL2 Ubuntu-24.04, fairseq source at `~/fairseq_src`.

**Integration is a separate open question, not yet decided**: the FastAPI server runs in the native-Windows venv, so this WSL2-only module can't be imported in-process. Likely path is a small HTTP call from `server/app.py` to a tiny service inside WSL2 (WSL2 localhost port-forwarding works both directions on modern Win11) — a new moving part (second long-running process) for what's currently a single-process POC, so this needs a real decision before building, not silent adoption.

## 2026-07-21 — Newsroom-writer-POV UI pass: headline length + word targets, Grammarly-style floating toolbar, scoped fact-preservation badge; IndicXlit retested and re-blocked

Asked to think from an actual newsroom writer's/editor's POV rather than just tool coverage. Identified real gaps (see session transcript for the full reporter/sub-editor/chief-editor breakdown) and built the three the user prioritized:

1. **Headline length badge + per-draft word-count target.** `graphemeLength()` uses `Intl.Segmenter` (grapheme clusters, not `.length`/UTF-16 code units) - matters specifically for Telugu, where vowel signs/conjuncts are multiple code points per visually-rendered character, so `.length` overcounts exactly the thing a headline char-limit cares about. Zones (<=60 good, 61-80 long, 81+ too long) are a generic default with no medium (print/web) selector - a real simplification, flagged as such. Word-count target is per-draft (stored in the same localStorage draft object), not global, matching how a real newsroom brief has its own column/word target.
2. **Grammarly-style floating inline toolbar.** Textareas have no native "pixel position of character N" API, so this uses the standard mirror-div technique (hidden div replicates the textarea's box model, a marker span at the target offset gives `offsetTop/offsetLeft`) - not a library, ~30 lines, and simple here because `canvas-area` has zero padding/border to replicate. Deliberately keeps the existing dock panel as the "always-present detail view" and the floating bubble as the "fast path" - both driven by the same `runSelectionAction()` call and the same response, not two separate code paths that could drift.
3. **Fact-preservation guardrail made general, but scoped correctly after a real false-positive was caught live.** `server/guardrails.py` gained `check_fact_preservation()` (numeric + capitalized-Latin-token preservation check) reused across grammar/paraphrase/translate/summarize. **Initially wired into `/api/headline` too - wrong, and caught by the Playwright loop, not by reasoning about it in advance**: a headline is a ~10-word compression of a full article, so dropping most numbers is normal, correct behavior, not a grounding failure - running the same near-lossless check there flagged nearly every real headline, which would train writers to ignore the badge (exactly the failure mode a trust signal must avoid). Removed from `/api/headline` - headline keeps only the existing `check_headline_needs_review` (completed-action assertion) check, which is the correct, low-false-positive signal for that task. en->te translation also skips the proper-noun half of the check (`check_proper_nouns=False`) - transliterating a Latin name into Telugu script is the task working correctly, not a dropped fact.

**Real bugs found via the live Playwright loop, not caught by static review** (full detail in `errors.md`): a floating-toolbar CSS `max-width` too narrow for its own unwrapped button row (buttons rendered outside their own background box), and an Escape-key dismiss that got immediately undone by the same keypress's `keyup` event re-triggering selection-status recompute - fixed with an explicit `dismissedRangeKey` tracker rather than a bare `state === "hidden"` check.

**IndicXlit/transliteration retested, real new information, still blocked.** User pointed at a Colab notebook (`IndicXlit_python_inference.ipynb`) showing `ai4bharat-transliteration` genuinely working - but in Colab (Linux, Python 3.8), not this project's Windows + Python 3.10 venv. Retested directly rather than trusting the stale 2026-07-18 finding: `pip install ai4bharat-transliteration` still fails, but with a **different, more specific error** than before - fairseq 0.12.1's PyPI sdist is missing `fairseq/version.txt` (a real upstream packaging defect, reproducible). Installing fairseq from GitHub source instead hits a second, different blocker: its `setup.py` creates a symlink (`fairseq/examples -> ../examples`) during build, which requires Windows Developer Mode (`AllowDevelopmentWithoutDevLicense` registry key) - confirmed not enabled on this machine. This is a system-wide Windows setting, not a project file, so left for the user to decide/enable rather than toggled directly. If enabled, worth retrying `pip install "fairseq @ git+https://github.com/facebookresearch/fairseq.git"` then `ai4bharat-transliteration` next session.

Explicit instruction this session: take full control of UI/UX, build a Notion/Grammarly-style writer workspace, use Playwright in a build→deploy→screenshot→fix loop, and prove the Gemma 3 exp3 checkpoint reacts correctly under real use. Previous Compose mode (2026-07-20 entries below) was a centered "article page" layout, functionally wired but never visually confirmed. Replaced the shell with a persistent 3-column app layout: a fixed navy+gold sidebar (brand identity, independent of the light/dark canvas theme - deliberately doesn't flip with the theme toggle, same reasoning real apps like Linear use a stable rail), a full-height main canvas, and a right-hand "సహాయకుడు · Writing Assistant" dock. Sidebar content is mode-dependent (drafts list in Compose, task list in Quick Tools) rather than a fourth column, keeping the shell to 3 panes in both modes.

**New real feature, not just a reskin**: client-side multi-document drafts (`localStorage`, key `rachana_drafts_v1`) - create/switch/delete/auto-save (title + body, 500ms debounce), seeded with one real demo article on first load. This was the single biggest missing "Notion-like" piece - the old UI had exactly one implicit document. No backend change needed; drafts are pure client state, matching the project's POC scope (no CMS/persistence layer built yet).

**Verification method**: installed Playwright into `.venv`, wrote a screenshot-driven loop (`scratchpad/shoot.py` pattern, not committed to the repo - lives in the assistant's scratch dir) that opens the HTML via `file://`, drives real interactions (select text, click every tool, run research), and waits on network completion rather than fixed sleeps. Ran the live server (`server/app.py`, exp3 adapter) against every screenshot round - every model output shown in this session's screenshots is real generation, not mocked. Found and fixed 3 real issues this way (see `errors.md` 2026-07-21 entries): a dropped Undo feature from the rebuild, a CSS media-query cascade-order bug causing the sidebar to visually overlap main content at narrow widths, and a UX gap where the backend's single-GPU generation lock (documented in `server/app.py`) left queued requests spinning with no explanation - added a global "busy" indicator (sidebar footer, `#busy-indicator`) driven by a `pendingRequests` counter wrapped around every fetch call site.

**What stayed unchanged**: all `/api/*` endpoints, request/response shapes, task definitions (headline/paraphrase/grammar/translate/summarize/research), the guardrail banners, the diff card, and the whole-document-headline-only vs. selection-scoped-everything-else split (still correct per 2026-07-20's training-scale finding below) - this was a shell/IA rebuild, not a re-litigation of what's already validated.

## 2026-07-20 — Session-closing note: server reliability issue found, Compose mode not yet visually verified

Closing out this session's work. Two things worth flagging clearly for next time: (1) the `uvicorn` dev server, run as an assistant background task, was killed unpredictably 3 separate times late in the session - including once seconds after a confirmed-healthy check - with nothing wrong in `server/app.py` itself each time it got the chance to run. Not root-caused; logged as a workaround in `errors.md` (run it in your own terminal for anything long-lived, don't rely on the assistant's background-task management to keep it alive). (2) The Compose-mode restructuring (previous entry) was built and reasoned through carefully (syntax-checked, every DOM ID cross-referenced against the HTML, same already-tested backend endpoints reused) but **has not actually been clicked through in a real browser** by either the assistant (no browser-automation tool available in this environment) or the user yet, given the server kept going down before a full end-to-end pass could happen. Treat it as "should work" based on static verification, not "confirmed working" - the first real thing to do next session is running the server from a real terminal and actually clicking through Compose end-to-end.

## 2026-07-20 — Restructured UI to a "Compose" unified article workspace; whole-document actions limited to headline only

Walking through a real writer's scenario (drafting a new article, needing to pull in research, get a headline, grammar-check parts of it - all against *the same* evolving piece) exposed that the task-picker POC didn't support this: each task had its own siloed input box, so a writer would have to re-paste the same draft into six separate tools. Restructured `ui/rachana-desk.html` toward a Google-Docs/Notion-AI-style model: one large auto-growing canvas (`<textarea>`, not contenteditable - simpler, reliable, and article output is plain text for a CMS anyway) with tools acting on it via two surfaces - a whole-document toolbar and a selection-based side panel (native `textarea.selectionStart/selectionEnd`, no rich-text editor library needed). The original per-task tool is kept unchanged as a "Quick Tools" mode (toggle in the masthead), not replaced - still useful for one-off work that doesn't need a full article context. Zero backend changes needed - every endpoint already accepted arbitrary text, so "whole document" and "selected text" are just different strings passed to the same `/api/*` routes.

**Real adjustment made during implementation, not assumed upfront**: the plan initially had Grammar/Translate/Summarize as whole-document toolbar actions alongside Headline. Checking what each task was actually trained on before building it: **only `headline_generation` was trained on full-article input** (TeClass articles → headline). GEC, paraphrase, translation, and sentence_compression were all trained on single sentences - running any of them against a whole multi-paragraph draft would risk the same kind of unreliability already found in zero-shot full-article summarization (2026-07-20, earlier entry). Moved all four to the **selection-tools panel** instead (act on a highlighted sentence/paragraph - their actual validated scale), keeping the whole-document toolbar to headline generation only. This teaches a clean, honest mental model: whole draft → headline; select text → everything else.

Verification note: the new Compose JS reuses the exact same `/api/headline`, `/api/grammar`, `/api/paraphrase`, `/api/translate`, `/api/research` calls already tested extensively earlier this session (same request/response shapes) - confirmed via a Node syntax check and a full cross-reference of every DOM ID the JS queries against the HTML (all present, no typos). Did not visually click through the new Compose UI in a real browser myself (no browser-automation tool available in this environment) - opened it for the user to check directly, same limitation already flagged earlier this session for the original POC build.

## 2026-07-20 — UI design-elevation pass: real loading states, motion, redesigned diff/research cards

After the initial POC build, did a dedicated visual/interaction polish pass on `ui/rachana-desk.html` (no backend changes). The most substantive fix wasn't decorative: clicking Generate again previously left the *previous* result visibly on screen while only the button label changed to "generating…" - a real "is this working or not" ambiguity in a live demo, not just an aesthetic gap. Replaced with an animated skeleton block in the output panel plus a spinner in the button, so it's unambiguous a new generation is in flight. Also added: fade-in motion on new results (`prefers-reduced-motion`-aware), a redesigned diff card for grammar output with an explicit change count, research-card source-initial badges reusing the wordmark's own visual language, and icon-in-circle banners for review/flag states instead of plain emoji-prefixed text. All additive CSS/markup - no JS logic or backend touched, so no server restart was needed to pick it up.

## 2026-07-20 — Pitch-ready POC scope finalized and built: 5 live tasks, controlled RAG, no notes→article/open search

Cofounder (also this project's AI engineer) wants the near-term goal reframed: build something that *feels and works* well enough on day 1 for the other cofounders to confidently pitch to prospective newsroom clients, with the understanding that real refinement happens once a client's newsroom starts a real beta and generates real usage data. This is a different bar than "production-correct for every capability" - it prioritizes credibility and a clean "improves with your data" narrative over completeness.

Finalized day-1 POC scope after discussion + real evidence-gathering (not assumption): **headline generation, grammar/GEC, paraphrase, translation (En↔Te)** - all validated by the two-independent-draw eval. **Summarization shipped sentence/short-passage only** - real testing (6 real articles, zero-shot) confirmed full-article summarization was never a trained/validated capability at any point in this project (even the very first baseline eval's "summarization" template was sentence-level), and the failure mode found (defaulting to copying the article's first sentence ~half the time) is systematic, not random. **Research Assistant (controlled RAG) added as new scope** - a curated allowlist of trusted Telugu news RSS feeds (verified live, not assumed - see next entry), query-driven only, always cited, explicitly NOT open web search or autonomous browsing. **Notes→article and press-release→news remain excluded** - unchanged from the standing PRD non-goal, no human-authored grounding data exists yet. **Transliteration elevated from "feature" to infrastructure** - today's Roman-Telugu robustness testing found three real quality problems (a "burst" vs. "scheduled repair" mis-framing, a dropped fact in GEC output, a garbled word from a spelling variant) that all trace to the LLM handling raw Roman-script input directly, so the intended fix is input normalization *before* the other tasks see it, not a standalone button.

Architecture locked in (plan file: `~/.claude/plans/polished-snacking-frog.md`): FastAPI backend (`server/`) loading the base model + Experiment 3 adapter once at startup (same pattern as `scripts/eval_adapter.py`), the existing `ui/rachana-desk.html` mockup wired to real endpoints rather than rebuilt in a framework, local-only deployment (no cloud hosting yet), a lightweight keyword heuristic (not an LLM self-check) for the completed-action guardrail. All four choices were explicit trade-offs the user picked for POC speed over completeness - revisit if/when this moves toward a real production deployment.

## 2026-07-20 — IndicXlit (official and community fork) both dead ends for this POC - real testing, not assumption

Investigated using AI4Bharat's IndicXlit as the Roman→Telugu input-normalization guardrail. Official `ai4bharat-transliteration` PyPI package depends on `fairseq`; a real `pip install --dry-run` attempt failed with a genuine build error (`fairseq`'s setup.py referencing a missing `version.txt` under pip's isolated build environment) - a known, current, unresolved packaging problem, not something specific to this machine. Found and reviewed a community fairseq-free alternative (`psidharth567/indic-xlit-270M`, a Gemma-3-270M-based char-level model in native `transformers`/safetensors format) - inspected its custom `modeling_transliteration.py` before running it (a small, safe embedding-wrapper fix, no red flags) via `trust_remote_code=True`. Built a minimal char-tokenizer + generation harness from its `vocab.json` (its own README references a `char_tokenizer.py` helper that isn't actually present in the repo) and tested real Roman→Telugu conversions (Kompally, Hyderabad, GHMC, etc.) - **output was garbage** (wrong-script symbols - Gurmukhi characters, math glyphs - unrelated to any real language), regardless of input, using the documented `[BOS][TARGET_LANG][source][SEP]` format from the README.

**Decision: no automatic Roman-Telugu normalization for this POC.** Both the real blocker (broken `fairseq` packaging) and the fallback (garbage output from the untested community port) are confirmed dead ends today, not just unvalidated - not worth more time before a pitch. Shipped the honest alternative instead: client-side script detection in the UI shows a visible notice ("type/paste Telugu script for best results") when input is Roman/mixed-script, for the two tasks (headline, grammar) where today's testing showed this actually matters. Revisit if the `fairseq` packaging gets fixed upstream, or a different working alternative turns up.

## 2026-07-20 — Real bug found in the completed-action guardrail: word-boundary matching required, not substring matching

Built a lightweight (no extra model call) heuristic to flag headlines that assert a completed/resolved action not stated as done in the source - directly targeting the real hallucination found in the headline spot-check (a headline claimed a theft gang had been "caught" - "పట్టు" - when the source only said police intended to catch them soon).

First version matched only *full* past-completed verb forms (e.g. "పట్టుకున్నారు") specifically to avoid false-positiving on the future-tense form sharing a prefix ("పట్టుకుంటామని" - will catch). **This was wrong**: re-testing the guardrail against the exact real article that produced the original hallucination showed `needs_review: false` - the model's actual output used the terser standalone headline form "పట్టు" alone (a common Telugu headline convention), which wasn't on the full-verb-form list at all. Fixed by switching to whitespace-tokenized word-boundary comparison (exact token equality between output and source, not substring search) - "పట్టు" as a standalone token is never equal to "పట్టుకుంటామని" as a single fused token, so this catches the real terse-form failure without reintroducing the future-tense false positive. Verified against the real failing case (now correctly flags) and three clean, accurate headlines (no false positives). Lesson for future guardrail work in this project: heuristic checks built from one example need to be re-verified against that *exact* real example after implementation, not just tested on synthetic hypotheticals - the first version looked reasonable on paper and still missed the real case.

## 2026-07-20 — Research Assistant's curated RSS allowlist: verified live, not assumed - 2 of 6 candidates rejected

Tested 6 candidate Telugu/Indian news RSS feed URLs directly (real HTTP requests + `feedparser`) before building anything around them. 4 worked with real current entries: BBC Telugu, Sakshi, TV9 Telugu, NTV Telugu. 2 did not: Eenadu returned HTTP 200 but zero entries (wrong URL path/feed format), Andhra Jyothy returned a plain 404. Shipped with only the 4 verified feeds. A two-step summarization prompt (extract 3-5 key facts, then combine into 2-3 sentences) was used for the Research Assistant's summarization step specifically to mitigate the first-sentence-extraction bug found in zero-shot article summarization testing earlier the same session - verified with real live queries against the real feeds (e.g. "డెంగ్యూ వ్యాక్సిన్" correctly retrieved and accurately summarized the matching dengue-vaccine article with a working citation link, not a copied first sentence). No-match queries correctly return an empty result rather than a fabricated one.

## 2026-07-19 — Proper-scale eval (n=100-200/task) confirms training-callback swings were mostly noise

Built `scripts/eval_adapter.py` (loads `google/gemma-3-4b-it` + the Experiment 3 adapter via PEFT, generates over a real eval file, writes `eval_results_*.jsonl` in the same schema as the baseline-eval harness) so `score_eval.py` could be reused unmodified. Ran it against the full protected eval set (`data/processed/exp3_eval.jsonl`, 100-200 rows/task, held out from training).

**Real numbers vs. the training callback's n=3 estimates — the callback was misleading**, not the model:
| task | callback (n=3) | proper eval (n=100-200) | Experiment 2 |
|---|---|---|---|
| headline_generation ROUGE-L | 0.460 | **0.388** | 0.157 |
| translation en→te chrF | 23.98 | **35.52** | 40.2 |
| translation te→en chrF | 56.54 | **39.92** | 36.5 |
| sentence_compression ROUGE-L | 0.190 | **0.222** | ~0.25 |
| GEC GLEU | 83.71 | **87.84** | 88.8 |

Headline generation's big win holds up at scale (still ~2.5x Experiment 2, the most trustworthy and important result). Everything else turns out much milder than the callback suggested: translation en→te dipped modestly (not crashed), te→en improved modestly (not doubled), GEC and sentence_compression are both essentially flat. Confirms the standing lesson (already logged for the baseline eval and the GEC/translation rechecks) that n=3-10 automated scores should never be trusted as a real capability signal, model-generation swings included.

**Second independent sample built and scored** (`scripts/prepare_eval_resample2.py`) to triangulate further: 400 fresh rows/task for headline_generation, sentence_compression, translation_en_to_te, translation_te_to_en (paraphrase has no automated metric, human-rated only) - drawn from real spare pool capacity (verified per-task: headline 1039, en_to_te 996, te_to_en 620 after re-streaming samanantar deeper, sentence_compression 13581), explicitly excluded against every input already used in train+eval (belt-and-suspenders on top of the streaming re-scan's natural non-overlap), fresh shuffle seed (123, not the project's usual 42) for genuine independence. **GEC deliberately excluded from this second check** - all 699 real BHASHA pairs are already fully spent across train+eval, zero spare; faking independence there by reusing seen rows would be worse than no check.

**The two independent draws agree closely** - this is the real, trustworthy signal:
| task | draw 1 (n=100-200) | draw 2 (n=400, independent) | Experiment 2 |
|---|---|---|---|
| headline_generation ROUGE-L | 0.388 | 0.353 | 0.157 |
| sentence_compression ROUGE-L | 0.222 | 0.214 | ~0.25 |
| translation en_to_te chrF | 35.52 | 35.30 | 40.2 |
| translation te_to_en chrF | 39.92 | 41.56 | 36.5 |

Conclusion, now confirmed rather than guessed: **headline generation's win is real and large** (~0.157→~0.37, the standout result of Experiment 3). Translation en→to→te has a **real, modest, reproducible dip** (40.2→~35.4) worth addressing in v0.2's data mixture - not the training-callback's alarming 23.98 crash. Translation te→en has a **real, modest improvement** (36.5→~40.7) - not the callback's inflated 56.5. Sentence_compression has a **small real dip** (~0.25→~0.22), minor. GEC (checked once, no spare data for a second draw - permanent scarcity limit, not a gap in rigor) is essentially flat (88.8→87.84). v0.1's per-task quality profile is now a settled, cross-validated picture, not an open question.

One incidental note: `score_eval.py`'s manual-rating CSV writer overwrites `manual_ratings_paraphrase.csv` each run rather than merging - the second (400-row) run's file replaced the first (150-row) run's file. No human ratings were filled in yet on either version (see progress.md "Waiting on human"), so nothing was lost - the file now just reflects the larger, more useful draw. Worth remembering if this script is ever run against multiple eval files expecting cumulative manual-rating output.

## 2026-07-19 — Experiment 3 completed; mixed per-task results, needs a proper eval before trusting the swings

Ran to completion on the proven native stack (23,735 examples, 6 tasks, 2 epochs) after the Unsloth/Accelerate/Lightning validation (see next entry) confirmed the plain stack was the right call. 2968/2968 steps, 9h23m wall-clock (train_runtime 33798s), no errors, adapter saved to `models/exp3_gemma3_4b_public_v0_1`. Full metrics pulled via the wandb API (run `9xikuyxh`, same method as Experiment 2 — the run URL itself needs browser login).

vs. Experiment 2 (4,335 examples, 1 epoch): headline_generation ROUGE-L 0.157→**0.460** (the capability flagged weakest after Exp2, biggest and most welcome swing), paraphrase ROUGE-L ~0.25→**0.413**, sentence_compression ROUGE-L ~0.25→0.190 (down), GEC GLEU 88.8→83.7 (down), translation en→te chrF 40.2→23.98 (down a lot), translation te→en chrF 36.5→**56.5** (up a lot). `mean_token_accuracy` ~0.69→0.681 (flat). Peak VRAM 7.17GB→7.167GB — confirms the earlier claim that peak VRAM tracks batch/seq_len/model size, not dataset size or epoch count, now checked across two different dataset sizes on the same config.

**Don't over-read the swings**: these `gen_eval/*` numbers come from the training callback's per-task spot-check, which only samples 3 rows/task by default (`--gen_eval_samples_per_task`) — the same "too-small-n" caveat already logged repeatedly for the baseline eval (translation, GEC) applies here too, and could plausibly explain why en→te and te→en moved in opposite directions by such a large margin. Headline generation's jump is the most trustworthy result here (large, in the expected direction, consistent with adding 6000 headline examples vs. however many Exp2 had proportionally) — the GEC/translation/sentence_compression dips need a real check with `score_eval.py` against a larger held-out sample before treating them as real regressions rather than noise. This is now the immediate next step, promoted above the other pending recheck items.

## 2026-07-19 — Unsloth validated and rejected for Experiment 3; Accelerate/PyTorch Lightning also ruled out

Followed up on the prior entry's open validation step. Tested all three candidates raised:

- **Accelerate**: no action, already using it transparently (TRL/`Trainer` dependency); its multi-GPU value doesn't apply on this single-GPU laptop.
- **PyTorch Lightning**: rejected without testing — it's a training-loop organization framework, not a speed/VRAM optimizer. `train_qlora.py` already sits on TRL's `SFTTrainer` (built on HF `Trainer`/Accelerate), the same category of thing Lightning provides. Adopting it would mean rewriting the training script for zero performance benefit.
- **Unsloth**: actually tested, in an isolated `.venv-unsloth` (not the main `.venv`) since a `pip install unsloth --dry-run` first revealed it pins `torch<2.11.0` and would have downgraded the main venv's proven `torch==2.11.0+cu128` Blackwell build. Installed `torch==2.10.0` explicitly from the `cu128` index in the isolated venv first, verified a real GPU matmul (no Blackwell warning) before installing Unsloth on top — it did not touch torch further. Two real bugs hit and fixed along the way (both specific to this Unsloth version/native-Windows combo, logged in `errors.md`): import order (Unsloth must be imported before `trl`/`transformers`/`peft`) and a missing `formatting_func` (Unsloth's compiled `SFTTrainer` doesn't auto-apply the chat template to a `"messages"`-column dataset the way vanilla TRL does).

**Real results, not estimates** (`scripts/train_unsloth_validate.py`, same LoRA config/seq_len/batch as the proven stack):
| scale | metric | proven stack | Unsloth |
|---|---|---|---|
| 1B (Exp 1 data, full epoch, 62 steps) | peak VRAM | 3.22 GB | **1.68 GB (-48%)** |
| 1B | wall-clock | 632s | 647s (~2% slower) |
| 1B | final train_loss | 3.753 | 3.913 (comparable) |
| 4B (Exp 1 data, 15-step probe) | peak VRAM | 7.17 GB (Exp 2, comparable config) | **5.20 GB (-27%)** |
| 4B | sec/step (steady state) | 9.7s (Exp 2 measured) | **23.3-23.8s (~2.4x slower)** |

Tried `lora_dropout=0` at 4B scale to rule out a documented Unsloth performance-hit path ("Unsloth will patch all other layers, except LoRA matrices, causing a performance hit" when dropout != 0) — only closed ~2% of the gap (23.3s vs 23.8s), not the real cause. The real cause is almost certainly `FA2 = False` in Unsloth's own startup banner: no Flash-Attention-2 build is available for this native-Windows + Blackwell (sm_120) + torch 2.10/cu128 combination, so Unsloth falls back to Xformers 0.0.35 attention, which is evidently much slower here than whatever path stock `transformers`+`bitsandbytes` uses on this same hardware. This directly contradicts Unsloth's "2x faster" framing for this specific setup.

**Decision: don't swap Experiment 3 to Unsloth.** VRAM savings are real (~27% at 4B scale) but Experiment 3's binding constraint was always runtime (~8hr estimate), not VRAM headroom for its current config - and a confirmed ~2.4x slowdown would push that to ~19+ hours for a VRAM cushion Experiment 3 doesn't need at its current seq_len/batch/target-module settings. Run Experiment 3 on the already-proven native stack as planned. Revisit Unsloth later only if/when Flash-Attention-2 gets real native-Windows Blackwell support - worth a periodic recheck, not a current blocker. `.venv-unsloth` left in place (not deleted) in case that recheck happens; `scripts/train_unsloth_validate.py` kept as the harness to rerun it cheaply.

## 2026-07-19 — Experiment 3 training run stopped mid-training, by explicit user request

The user had actually started the Experiment 3 command (found running as two live `train_qlora.py` processes, one holding ~2.5GB memory, consistent with active training) and separately asked to "close the background session." Given how destructive killing an in-progress ~8hr run would be, paused and confirmed exactly what was meant via a direct question before acting - the user confirmed they did want the training process killed. Both processes stopped by PID (`Stop-Process`), verified other unrelated python processes on the machine were untouched. Since `save_strategy="epoch"` only checkpoints at the end of a full epoch, and this was killed before completing epoch 1, **no adapter checkpoint was saved** - resuming Experiment 3 later means starting the run over, not resuming from a partial state. `exp3_train.jsonl`/`exp3_eval.jsonl` are unaffected and still valid for a future run.

## 2026-07-19 — Accelerate: no action needed; Unsloth: promising, verify before swapping Experiment 3 to it

Discussed both as levers for faster/lighter training given Experiment 2's tight VRAM (7.17GB/8GB peak) and Experiment 3's ~8hr estimate. Accelerate is already in use transparently - it's TRL/`transformers.Trainer`'s dependency for device placement and training-loop mechanics, already in `requirements.txt`. Its actual value (multi-GPU/distributed orchestration) doesn't apply on this single-GPU laptop, so there's no lever to pull there.

Unsloth is a real candidate, verified via its docs/GitHub rather than assumed: native Windows is a first-class target (dedicated PowerShell installer, not WSL-only), **Blackwell/RTX 50-series is explicitly supported ("no separate image needed")** - directly relevant since that's the exact GPU generation that caused the sm_120/cu124 torch mismatch earlier this project. Still uses `bitsandbytes` underneath (not a full stack replacement). Gemma 1-3 family is listed as supported, though not confirmed at the specific `gemma-3-4b-it` checkpoint level. One open gap: their install guide's example environment uses Python 3.12; this project's venv is 3.10 - not necessarily a blocker, but unconfirmed.

Decision: don't swap Experiment 3 (the closer-to-final v0.1 run) onto unvalidated new training internals. Validate first with a cheap test - rerun Experiment 1's tiny pipeline-check data (~10 min) through Unsloth and compare real speed/VRAM against the numbers already measured. Swap Experiment 3 over only if that's clean; otherwise Experiment 3 runs on the already-proven stack and Unsloth becomes a later optimization pass, not a blocker. Not yet run as of this entry.

## 2026-07-19 — Experiment 3 data built; deliberately excludes the unfiltered GEC synthetic batch

`scripts/prepare_experiment3_data.py` produces the full v0.1 mixture minus transliteration (same architecture decision as Experiment 2): 23,735 train rows (headline 6000, sentence_compression 5000, translation_en_to_te 5000, translation_te_to_en 3000, paraphrase 4000, gec 735) + 900 protected eval rows. Matches the ~23,700-example / ~8hr runtime estimate already logged from Experiment 2's measured throughput.

Explicitly does NOT merge in the new 1,165-pair GEC synthetic batch (874 correction + 291 identity, generated the same day) - that batch has a known ~1/3 quality-issue rate found on spot-check, and the follow-up filter pass to clean it up was explicitly deferred by the user. Training Experiment 3 - meant to be the closer-to-final v0.1 candidate - on knowingly-flawed synthetic data (including backwards-labeled pairs) would contradict the audit-before-training discipline used everywhere else in this project. GEC stays at the same 735 real+synthetic-identity examples Experiment 2 already validated. Command handed to the user to run in their own terminal; not yet executed as of this entry.

## 2026-07-18 — GEC synthetic corrupted-reference generator: built, run at 50% scope, real quality issues found on spot-check

Built `scripts/generate_gec_synthetic.py`: takes a clean Telugu sentence (from IndicParaphrase targets, deduped against real GEC sentences) and asks `gemma3:4b` (via Ollama, few-shot primed with 4 real BHASHA error examples) to introduce a similarly realistic writing mistake. Scoped to 50% of the original ~4,000-example GEC target per explicit instruction: ~874 synthetic correction pairs + ~291 synthetic identity pairs (identity side trivially generated, no LLM needed), maintaining the existing 75/25 correction:identity ratio.

Smoke-tested on 5 examples before the full run - caught a real bug in the first filter design: a word-level *ratio* similarity check (`difflib.SequenceMatcher`) is too length-sensitive, wrongly rejecting good short-sentence edits while still missing the actual failure mode (the model "fixing" a pre-existing typo in the mined IndicParaphrase source instead of introducing a new error, which flips the pair's label direction backwards). Replaced with an absolute word-diff-count filter (keep only if 1-2 words differ) - this doesn't penalize short sentences and better isolates "one localized change" from "a rewrite."

Full run: 874 correction + 291 identity = 1,165 pairs written, 66% yield after filtering (matching the smoke-test estimate of ~60-65%). **However, an at-scale spot-check of 15 sampled correction pairs found the word-diff-count filter still isn't sufficient**: roughly 1/3 of pairs have a real quality problem - ~13% are still backwards-labeled (multi-word diffs that happened to land at exactly 2, e.g. a genuine plural-form change alongside a spelling "fix"), and ~20% are meaning-changes rather than realistic grammar/spelling mistakes (statement→question, a dropped ability-modal, a headline fragment expanded into a full sentence) that the filter can't distinguish from genuine errors since they also present as small word-level diffs. Roughly 45-50% are genuinely good, matching the few-shot pattern well.

Decision: this is not a failed generator (50-65% genuine yield is a reasonable first-pass rate for an automated technique), but it does need a further quality pass before being trusted as training data - consistent with this project's standing rule that synthetic data needs audit before use. A second automated filter pass (LLM-based "is this a realistic error, and is the source actually clean" check) was proposed and **explicitly deferred by the user** to a later phase. The 1,165-pair batch is NOT merged into Experiment 3 or any other training run yet - see the following entry.

## 2026-07-18 — TranslateGemma teacher-model test: inconclusive, don't trust the numbers as-is

Tested `google/translategemma-4b-it` (real, Gemma-3-based, gated but access already granted - same as the other Gemma repos) on the exact same 10 translation_en_to_te/translation_te_to_en examples used for the original baseline eval, scored with the same chrF metric (`scripts/test_translategemma.py`). Two implementation notes for reuse: (1) the model's chat template needs a *list* containing one mapping as `content` (`[{"source_lang_code":..., "target_lang_code":..., "type":"text", "text":...}]`), not a plain string or bare dict - very different interface from base Gemma 3; (2) the ~8GB download can take 50+ minutes on this connection and got killed once by an internal execution limit before finishing weight-loading - safe to just rerun once the files are already cached (`~/.cache/huggingface/hub/models--google--translategemma-4b-it`), no need to re-download.

Raw scores: TranslateGemma 35.88 (en→te) / 49.28 (te→en) vs. baseline unmodified Gemma 3 4B's 36.13 / 55.29 - looks roughly even or slightly worse at face value. **Don't trust this as a real answer** - manual read-through of the actual outputs found the Samanantar references themselves are noisy (one reference translation was off by ~100,000x from the source's actual number - "10 thousand crores" rendered as "10 lakh"), and TranslateGemma's outputs read as more literal/faithful throughout, which chrF penalizes against a loosely-paraphrased or wrong reference. n=5/direction is also too thin to trust either way (same limitation already flagged for the original baseline eval). One minor real issue spotted: one output had a corrupted mixed-script fragment ("సమithi" instead of "సమితి") - isolated (1/10), not over-weighted, but logged.

Verdict: **inconclusive - PRD open question #3 (teacher-model decision) is not resolved by this test.** Would need a larger n (indicated by the same "5-10 examples is too small for model-selection calls" lesson already learned from the original baseline eval) and/or better-quality references (or human judgment on a sample) before making a real call on TranslateGemma as a teacher model.

## 2026-07-18 — Experiment 2 (Gemma 3 4B, small test) passed; VRAM headroom is tight, Experiment 3 runtime estimated

Trained `google/gemma-3-4b-it` QLoRA on 4,335 examples (6 tasks, transliteration dropped per the IndicXlit-module decision, GEC given all 735 real+synthetic-identity examples), seq_len 640, 1 epoch. Pulled full run data via the wandb API (`wandb.Api().run(...)`, since the run URL itself needs browser login - WebFetch can't see it) rather than relying on the truncated console summary.

**VRAM: peaked at 7.17GB allocated / 7.47GB reserved on the 8GB card** - close to the ceiling. Should still fit for Experiment 3 (peak VRAM scales with batch/seq_len/model size, not dataset size or epoch count), but there is very little margin left - don't increase seq_len, batch size, or add more LoRA target modules (the `gate/up/down_proj` fallback from the primer) without real risk of OOM.

**Runtime: 9.7 sec/step** (271 steps, 2,626s total). Experiment 3's full mixture minus transliteration (~23,700 examples) at 2 epochs is roughly 11x the steps here - **estimated ~8 hours**, not a guess. Plan around that before committing to an overnight/full-day run.

**Quality: clear improvement over Experiment 1's 1B run.** `mean_token_accuracy` reached ~0.69-0.70 (vs. 0.51 for the 1B/140-example run). Per-task generation quality (`gen_eval/*`, real generated text): GEC GLEU 88.8 (up from 75.0 in Experiment 1, despite GEC's data scarcity), translation chrF 40.2 (en→te) / 36.5 (te→en), paraphrase/sentence_compression ROUGE-L ~0.25 each. **Headline generation is the weakest at ROUGE-L 0.157** - consistent with the baseline eval's original finding that headline generation is the hardest capability; worth specific attention in Experiment 3, not assumed to fix itself with more data alone.

## 2026-07-18 — Gemma 3 4B already comprehends Roman-Telugu well (~90%, n=20); no dedicated training needed, but watch two failure modes

Built a hand-authored 20-example probe (`scripts/prepare_roman_telugu_eval.py`) testing reading comprehension (not production - that's IndicXlit's job, see the separate-transliteration-module discussion) of Roman-Telugu across 4 categories: pure-Roman comprehension, spelling-variant robustness, realistic mixed English/Roman/Telugu-script notes, and reasoning (arithmetic/combining facts) over romanized content. Scored via a strict substring-match scorer (`score_answer_match` in `scripts/metrics.py`) - raw automated score was 63% (12.5/20), but manual review of every "failure" found the scorer was too literal in 3 of 5 cases (accepted forms it didn't anticipate: spelled-out Telugu number words like "ఆరు" for "6", genitive-case inflection "కోట్ల" vs "కోట్లు", alternate valid transliteration "ఆరెంజ్" vs "ఆరంజ్" for "orange"). **Real corrected pass rate: 18/20 (90%)** - confirms the hypothesis that Gemma 3's pretraining already covers this, no dedicated fine-tuning needed for comprehension.

Two genuine failures worth remembering, not just noise: (1) one specific romanized spelling variant caused a full script derailment - the model answered in Hindi script with a Malayalam translation, not Telugu at all - a failure of language selection, not just content accuracy (n=1, don't over-generalize, but the failure mode exists). (2) A real reasoning error: when the same fact was restated in two different words/languages in one note ("delay" in English + the model's own "ఆలస్యం" for lateness), the model double-counted it as two separate facts (2+3+3=8 instead of 2+3=5 trains). This directly matters for the notes→article verification/grounding-check step designed earlier - it should specifically check for double-counted numbers from redundant/bilingual phrasing, not just missing or added facts.

Also extended `scripts/probe_tokenizer.py` with a Roman-Telugu vs Telugu-script vs English fragmentation comparison (same content, 3 scripts) - no dedicated finding beyond what the comprehension probe already showed, kept as a standing diagnostic.

Lesson logged for future eval design: strict substring/exact-match scoring under-counts true model correctness on open-ended multilingual answers unless the acceptable-answer list anticipates spelled-out numbers, case inflection, and loanword transliteration variants - cheap automated scores from a small literal matcher should get a manual spot-check on "failures" before being trusted, not taken at face value.

## 2026-07-18 — Comprehensive wandb logging added to train_qlora.py before Experiment 2

Default HF Trainer wandb logging (loss/grad_norm/lr/token accuracy, aggregate only) was judged insufficient for this project's needs: no per-task breakdown across the 7 mixed tasks, no qualitative generation tracking, no VRAM/throughput-over-time (explicitly needed per the original Experiment 2/3 staging plan - "measure throughput... peak VRAM... before committing to the full run"), no LoRA-config/data-manifest linkage, no run organization. Added: (1) `scripts/metrics.py` - the per-task scorer functions (chrF/CER+exact-match/ROUGE-L/GLEU) extracted out of `score_eval.py` into a shared module, so training-time and baseline-eval numbers are directly comparable; (2) a `GenerationEvalCallback` that runs real generation (not just teacher-forced loss) on a small per-task sample at every eval pass, logs per-task metrics as `gen_eval/<task>/<metric>`, and logs a `wandb.Table` of sample (input, reference, generated) triples so behavioral change is visible over training, not just the loss curve; (3) a `ResourceLogCallback` logging GPU memory (allocated/reserved/peak) and steps/sec at every logging step; (4) `nvidia-ml-py` installed so wandb's automatic system-metrics panel (GPU util/temp) populates; (5) explicit `wandb.init()` with a fixed project (`rachana-news`, not wandb's generic `huggingface` default), explicit run name/tags, and a config block combining CLI args + LoRA config + the data manifest JSON. Caught and fixed one real bug in the process: passing an explicit `step=` to the custom callbacks' `wandb.log()` calls raced with wandb's internal step counter (already advanced by HF's own eval logging) and silently dropped one log call per run - fixed by letting wandb auto-assign the step instead. Validated clean (no warnings, all metrics populated) before moving to Experiment 2.

## 2026-07-18 — Experiment 1 (Gemma 3 1B pipeline check) passed; proceeding to Experiment 2 without changes

`scripts/train_qlora.py` + `scripts/compare_base_vs_adapter.py` validated the full mechanical pipeline (data loading, QLoRA training, adapter save/reload, generation) on `google/gemma-3-1b-it` with 980 examples (140/task). Loss 5.38→3.1, adapter output differed from base on all 7 tasks, and the qualitative pattern was encouraging: the base model's dominant failure mode (answering with meta-commentary like "Here are a few options..." instead of a direct answer) was visibly suppressed post-adapter on most tasks. Transliteration output still came out in Telugu script rather than Roman, and paraphrase barely changed — both expected at this scale (1B model, 140 examples, 1 epoch) and not treated as a signal to change the recipe before Experiment 2. Per user instruction (2026-07-18), the GEC corrupted-reference generator and the notes/press-release reverse-generation pilot are deliberately deferred to run alongside/after the training experiments, not before.

## 2026-07-18 — Training environment: native Windows, not WSL2 (confirmed working, no fallback needed)

Tried native Windows first per the project's existing cheap-first-then-escalate pattern (same reasoning as the Ollama-over-transformers eval-harness decision) rather than setting up WSL2 Ubuntu-24.04 (already installed but unused) upfront. `torch`/`transformers`/`peft`/`bitsandbytes`/`accelerate`/`trl` all installed and verified working natively, including a real GPU matmul (not just `torch.cuda.is_available()`, which can return `True` even when the installed build can't actually run on the card — see `errors.md`). No need to fall back to WSL2 for Experiment 1/2. `HF_TOKEN` and `WANDB_API_KEY` are in a project-root `.env` (not committed to git — repo isn't git-initialized yet, but flag this if it ever is), loaded via `python-dotenv`. Gemma license accepted and verified against both `google/gemma-3-1b-it` and `google/gemma-3-4b-it` (gated repos, `gated=manual`, token has access to both).

## 2026-07-18 — v0.1 public-data checkpoint is disposable; v0.2 still restarts from base (not a reversal of the one-combined-run decision)

Adopted a staged approach for practical reasons — train `Rachana-News-Gemma3-4B-PublicData-v0.1` now on public data only, without waiting weeks for the intern-built editorial set. This looked like it contradicted the 2026-07-13 "one combined run, not stacked stages" decision, but it doesn't, provided v0.2 **restarts QLoRA from the base model** on a blended mixture (60% human-reviewed editorial / 30% high-quality public task data / 10% general instruction data) rather than continuing training on top of v0.1's adapter. v0.1 is a pipeline-validation and early-signal checkpoint only, not a foundation anything else builds on. If v0.2 is ever implemented as adapter-continuation instead of a fresh base-model run, that would reintroduce the catastrophic-forgetting risk the original decision exists to avoid — worth double-checking at implementation time.

## 2026-07-18 — GEC public data is far scarcer than assumed; supplement with corrupted-reference synthetic pairs

Audited actual BHASHA IndicGEC2025 Telugu data: only 699 real pairs total across train (599, incl. 47 identity/no-correction) + dev (100). No train split reaches anywhere near the ~4,000-example target floated for the v0.1 grammar/copyediting slice. Plan: keep all 699 real pairs as the trusted eval/training anchor, generate the remainder synthetically using the corrupted-reference technique the PRD already endorses for editing tasks (clean Telugu sentence → model or rule-based error injection → (corrupted, clean) pair), sourced from clean sentences in IndicCorpV2 or TeClass. Don't just shrink the target and call it done — the 75/25 correction/identity-pair ratio still needs to hold in the final mixture.

## 2026-07-18 — Public "summarization" data is sentence-level compression, not article-level newsroom summarization

Audited `ai4bharat/IndicSentenceSummarization` (te): 20,369 train rows, but median input is 123 characters (one sentence) and median output 46 characters. This is sentence compression, not condensing a full article into a brief — a materially different task than what "summarization" means for the product. Decision: still include it in the v0.1 mixture (it's real, available, and useful for that narrower skill), but tag it as `sentence_compression` in the training data, not `summarization`, so progress on it isn't mistaken for progress on real article-level summarization. Article-level summarization capability is deferred to v0.2's human-authored/reviewed data, same as notes→article and press-release→news.

## 2026-07-18 — Gemma tokenizer measured via Ollama's local API, not `transformers.AutoTokenizer`

Needed real token-length distributions (chars/token for Telugu vs English, chat-template overhead, real dataset token counts) to settle the v0.1 seq_len question. `google/gemma-3-4b-it` on HF is gated (needs login + ToS accept), and no HF token is configured in this environment (same constraint as BPCC/FLORES). Workaround: Ollama already has the GGUF locally (`gemma3:4b`), and its `/api/generate` endpoint with `"raw": true, "options": {"num_predict": 1}` returns `prompt_eval_count` — the exact token count of whatever raw string is sent, no chat template applied. `/api/chat` with a real system+user turn gives the template-inclusive count, so the difference isolates template overhead. No HF token or `transformers` install needed. Result: seq_len 512 is fine for the v0.1 mixture (p95 total sequence ≈539 tokens for the longest task, headline generation) — bumping to 640 for headroom, not because 512 was actually insufficient. Full numbers in `reports/tokenizer_probe.md`.

## 2026-07-18 — Notes→article and press-release→news: two-stage decomposition + reverse-generation from real articles

Design for the two no-public-data capabilities (PRD already mandates source-grounding for both, no bare "write about X" generation). Two decisions:
1. **Decompose into a structuring stage + a shared grounded-generation stage**: notes→fact-sheet and press-release→neutral-claims-extraction are the two upstream, task-specific skills; fact-sheet→article is one shared downstream skill reused by both, since both ultimately reduce to "write a grounded article from discrete verified facts."
2. **Generate training pairs backwards from real published articles** instead of authoring notes/press-releases from scratch: take a real Telugu news article (TeClass has 26,178 across train/dev/test, train+dev usable = 22,251 after excluding the test split already spent on baseline eval) and have the model reverse-generate plausible raw notes or promotional press-release framing that could have produced it. The real article becomes a free, human-quality training label; the reviewer's job narrows to checking the reverse-generated input doesn't *omit* any entity/number/quote/date that the target article states (an omission there would train the model to confabulate missing facts from thin input — the same hallucination failure mode already logged for headline generation and the PAN-card instruction test). This audit task is cheaper than authoring notes/articles cold, and may reduce the intern-hours estimate in `progress.md`'s pending staffing proposal — worth piloting on ~20-30 articles before revising headcount. Separately, real (press-release → resulting news coverage) pairs may exist by matching PR wire feeds against same-day coverage — worth a scoping check as an added public-data source, not yet committed to.

## 2026-07-16 — Gemma 4 exists; eval-only candidate, not adopted for v1

Verified via the official model card (`ai.google.dev/gemma/docs/core/model_card_4`, last updated 2026-07-16 — i.e. brand new) that Gemma 4 is a real, released model family: E2B (2.3B effective params), E4B (4.5B effective), 12B Unified, 26B A4B (MoE), 31B Dense. All variants are multimodal (text+image, audio on the smaller three); there is no text-only small variant the way Gemma 3 has 1B/270M. Given it's days old at time of writing, ecosystem tooling (PEFT/transformers/llama.cpp/Ollama support) is effectively unproven. Decision unchanged from the existing plan: train Rachana News v1 on Gemma 3 4B now; evaluate Gemma 4 E2B later as a baseline comparison only, once tooling matures. Also confirms the earlier correction that Gemma 3 4B (unlike 1B/270M) is multimodal — doesn't block text-only QLoRA fine-tuning (freeze vision tower, target decoder attention/MLP projections only), just means the checkpoint carries unused vision weights.

## 2026-07-13 — Gemma 3 license confirmed compatible with commercial fine-tuning, with pass-through obligations

Verified via Google's Gemma Terms of Use: commercial use and fine-tuning are permitted, no separate commercial license negotiation needed. But it's not a fully permissive open-source license — it's Google's own custom terms, and they carry forward to any Gemma-based fine-tune: a Prohibited Use Policy that must be contractually passed down to our own downstream users/customers, plus Google retains the right to unilaterally modify terms or terminate access. Doesn't block the current fine-tuning plan; means our own newsroom-customer ToS needs to flow down Gemma's Prohibited Use Policy, and terms should be periodically rechecked (e.g. at major Gemma version bumps). Logged in PRD §9 risk table too.

## 2026-07-13 — SFT sequencing: one combined run, not stacked stages

Plan is to build data for translation/transliteration/GEC/paraphrase-summarization (existing public data) and notes→article/press-release→news (generate + human audit) in parallel. But fine-tune once, from the base model, on the full combined mixture — not as two sequential fine-tune stages stacked on each other. Sequential stacking risks catastrophic forgetting (stage 2 overwrites what stage 1 taught), same risk the PRD already flags for the pretraining→SFT transition. An early checkpoint on just the 4 ready-now tasks is fine as a pipeline sanity-check, but shouldn't be the foundation the next round builds on.

## 2026-07-13 — Telugu instruction/procedural-writing: structure transfers, facts don't

Tested `gemma3:4b` (unmodified) on a "give step-by-step instructions in Telugu" prompt (PAN card application). Structure/fluency transferred cleanly from English — proper numbered steps, correct grammar. But it hallucinated specifics inside that fluent structure: an invented government body name, an invented acronym, and a fabricated URL. Same failure pattern as headline generation and the earlier ungrounded notes→article test. Conclusion: don't test structural instruction-following in Telugu again, that's confirmed fine — but any how-to/explainer content the product generates needs the same real-source-grounding rule as everything else; never trust generated facts without a source document behind them.

## 2026-07-13 — Set up persistent progress/decisions/errors docs

Sessions were relying on full conversation replay to reconstruct state and avoid repeat mistakes — expensive and lossy. `progress.md`/`decisions.md`/`errors.md`/`data-sources.md` plus `CLAUDE.md` (auto-loaded) replace that with cheap, structured reads.

## 2026-07-12 — Ollama (not transformers/WSL+bitsandbytes) for the baseline eval harness

The eval harness is a throwaway diagnostic, not part of the eventual SageMaker/`transformers` training pipeline. Optimized for speed of getting an eval running on native Windows over infra parity with training. Revisit with `transformers`+WSL if exact-checkpoint fidelity matters later.

## 2026-07-12 — Translation eval source switched from Helsinki-NLP/opus-100 to ai4bharat/samanantar mid-session

`opus-100` en-te turned out to be noisy, misaligned subtitle/song-lyric data on inspection (missing spaces, wrong-meaning pairs). `samanantar` is mined (not human-gold) but far more fluent and newsroom-relevant in spot checks. See `errors.md` and `data-sources.md`.

## 2026-07-12 — BPCC / FLORES-200 rejected as translation source (gated)

Both require an HF token + accepted dataset ToS (401 without). Didn't want to make that call on the user's behalf mid-session, so used an ungated alternative (`samanantar`) instead. Revisit with a logged-in HF token if BPCC-Human's gold quality is worth the setup step later.

## 2026-07-12 — GEC scored with BHASHA IndicGEC2025's own GLEU(1-4) formula

Reused the shared task's own `score.py` scoring logic instead of a generic edit-distance metric, so the number is roughly comparable to that leaderboard's scale.

## 2026-07-12 — ROUGE-L scored with a custom whitespace tokenizer, not the library default

`rouge_score`'s default tokenizer is ASCII-only and silently produces `rougeL=0.0` for Telugu script (see `errors.md`). Telugu is space-delimited, so whitespace splitting is the standard word-level unit here — not a fully "correct" fix for morphology, just the right default at this scale.

## 2026-07-12 — Notes→article eval prompts hand-authored, not dataset-sourced

PRD itself flags no good public source for notes→article; every other capability's eval prompts are dataset-sourced. This is the one deliberate exception.

## 2026-07-12 — Baseline eval scope: all 8 capabilities, shallow (~10 prompts each), not narrow-and-deep

Chosen over doing 1-2 capabilities deeply first, to get broad triage signal across the whole PRD §5 capability table before committing fine-tuning effort anywhere specific.