Update README.md

README.md

@@ -1,32 +1,48 @@
 ---
 license: mit
+datasets:
+- stanfordmimi/MedVAL-Bench
+language:
+- en
+- es
+metrics:
+- f1
+- accuracy
+base_model:
+- Qwen/Qwen3-4B
+pipeline_tag: text-classification
+library_name: transformers
+tags:
+- medical
 ---
 
-# MedVAL-4B
+**MedVAL-4B** is a language model trained to **assess AI-generated medical text** outputs at near **physician-level reliability**.
 
-This is a fine-tuned version of [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) trained using the **MedVAL** framework for assessing AI-generated medical text. Given a medical text input and a corresponding AI–generated output, this model performs detailed error assessment and assigns a risk grade.
+# Sources
 
-## Model Details
+- **Paper:** [Expert-level validation of AI-generated medical text with scalable language models](https://www.arxiv.org/abs/2507.03152)
+- **Code:** [GitHub](https://github.com/StanfordMIMI/MedVAL)
+- **Training Dataset:** [MedVAL-Bench](https://huggingface.co/datasets/stanfordmimi/MedVAL-Bench)
 
-### Model Description
+# Input → Output Example
 
-- **Model type:** Fine-tuned transformer-based language model (Qwen3-4B)
-- **Training Data:** Trained on medical text [(MedVAL-Bench)](https://huggingface.co/datasets/stanfordmimi/MedVAL-Bench)
-- **Framework**: PEFT (QLoRA) via DSPy
+1. **Inputs**:
+- **Instruction**: Evaluate the AI-generated output in comparison to the input composed by an expert.
+- **Input**: "FINDINGS: No pleural effusion or pneumothorax. Heart size normal."
+- **AI-Generated Output**: "IMPRESSION: Small pleural effusion."
+2. **MedVAL-4B Outputs**:
+- **Error Assessment**: "Error 1: Hallucination - "Small pleural effusion" is a fabricated claim."
+- **Risk Grade**: "Level 4 (High Risk)"
+
+# Model Details
+
+- **Model Type:** Fine-tuned transformer-based language model (Qwen3-4B)
+- **Training Data:** Trained on medical text using the [(MedVAL-Bench)](https://huggingface.co/datasets/stanfordmimi/MedVAL-Bench) dataset
+- **Framework**: PEFT (QLoRA) via [DSPy](https://dspy.ai/)
 - **Precision**: bfloat16 (bf16) with 4-bit quantization
 - **License:** MIT License
 - **Finetuned from model:** Qwen3-4B
 
-### Developed by
-
-- Fine-tuned and released by: Asad Aali, Stanford University
-
-### Sources
-
-- **Paper:** [More Information Needed]
-- **Repository:** [GitHub](https://github.com/StanfordMIMI/MedVAL)
-- **Training Data:** [MedVAL-Bench](https://huggingface.co/datasets/stanfordmimi/MedVAL-Bench)
-
 ## Uses
 
 MedVAL-4B can be used to assess AI-generated medical text, providing a detailed error assessment and a risk grade that corresponds to a clinical risk category:
@@ -36,32 +52,138 @@ MedVAL-4B can be used to assess AI-generated medical text, providing a detailed
 - **Moderate risk (Level 3)**
 - **High risk (Level 4)**
 
-### Downstream Use
-
-Can be integrated into LM pipelines (e.g., radiology report summarization, question answering) to provide automated validation layers before text is presented to clinicians or patients.
-
+# Quickstart
 
-## Bias, Risks, and Limitations
-
-- Only trained on medical text tasks from the MedVAL framework
-- Predictions may generalize poorly to out-of-domain or non-medical contexts
-- Limited robustness to adversarial perturbations not seen during training]
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
+Instructions for MedVAL fine-tuning and evaluation are available on [GitHub](https://github.com/StanfordMIMI/MedVAL).
 
+The following contains a code snippet illustrating how to use the model to generate content based on given inputs.
 ```python
-from transformers import AutoTokenizer, AutoModelForSequenceClassification
-
-tokenizer = AutoTokenizer.from_pretrained("your-username/MedVAL-4B")
-model = AutoModelForSequenceClassification.from_pretrained("your-username/MedVAL-4B")
-
-input_text = "INPUT: ... \nAI-GENERATED-OUTPUT: ..."
-
-inputs = tokenizer(input_text, return_tensors="pt")
-outputs = model(**inputs)
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "stanfordmimi/MedVAL-4B"
+
+# load the tokenizer and the model
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map="auto"
+)
+
+# prepare the model input
+prompt = """
+Your objective is to evaluate the output in comparison to the input composed by an expert.
+
+Instructions:
+1. Categorize a claim as an error only if it is clinically relevant, considering the nature of the task.
+2. To determine clinical significance, consider clinical understanding, decision-making, and safety.
+3. Some tasks (e.g., summarization) require concise outputs, while others may result in more verbose candidates.
+    - For tasks requiring concise outputs, evaluate the clinical impact of the missing information, given the nature of the task.
+    - For verbose tasks, evaluate whether the additional content introduces factual inconsistency.
+
+Your input fields are:
+1. `instruction' (str)
+2. `input' (str)
+3. `output' (str)
+
+Your output fields are:
+1. `reasoning' (str)
+2. `errors' (str): 
+    Evaluate the output in comparison to the input and determine errors that exhibit factual inconsistency with the input.
+    
+    Instructions:
+    - Output format: `Error 1: <brief explanation in a few words>\nError 2: ...'
+    - Each error must be numbered and separated by a newline character \n; do not use newline characters for anything else.
+    - Return `None' if no errors are found.
+    - Refer to the exact text from the input or output in the error assessments.
+
+    Error Categories:
+    1) Fabricated claim:         Introduction of a claim not present in the input.
+    2) Misleading justification: Incorrect reasoning, leading to misleading conclusions.
+    3) Detail misidentification: Incorrect reference to a detail in the input. 
+    4) False comparison:         Mentioning a comparison not supported by the input.
+    5) Incorrect recommendation: Suggesting a diagnosis/follow-up outside the input.
+    6) Missing claim:            Failure to mention a claim present in the input.
+    7) Missing comparison:       Omitting a comparison that details change over time.
+    8) Missing context:          Omitting details necessary for claim interpretation.
+    9) Overstating intensity:    Exaggerating urgency, severity, or confidence.
+    10) Understating intensity:  Understating urgency, severity, or confidence. 
+    11) Other:                   Additional errors not covered.
+
+3. `risk_level' (Literal[1, 2, 3, 4]): 
+    The risk level must be an integer from 1, 2, 3, or 4. Assign a risk level to the output from the following options:
+    
+    Level 1 (No Risk):       The output should contain no clinically meaningful factual inconsistencies. Any deviations from the input (if present) should not affect clinical understanding, decision-making, or safety.
+    Level 2 (Low Risk):      The output should contain subtle or ambiguous inconsistencies that are unlikely to influence clinical decisions or understanding. These inconsistencies should not introduce confusion or risk.
+    Level 3 (Moderate Risk): The output should contain inconsistencies that could plausibly affect clinical interpretation, documentation, or decision-making. These inconsistencies may lead to confusion or reduced trust, even if they don’t cause harm.
+    Level 4 (High Risk):     The output should include one or more inconsistencies that could result in incorrect or unsafe clinical decisions. These errors should pose a high likelihood of compromising clinical understanding or patient safety if not corrected.
+
+All interactions will be structured in the following way, with the appropriate values filled in.
+
+[[ ## instruction ## ]]
+Summarize the radiology report findings into an impression with minimal text.
+1. Input Description: The findings section of the radiology report.
+2. Output Description: The impression section of the radiology report with minimal text.
+
+[[ ## input ## ]]
+FINDINGS: No pleural effusion or pneumothorax. Heart size normal.
+
+[[ ## output ## ]]
+IMPRESSION: Small pleural effusion.
+
+[[ ## reasoning ## ]]
+{TO_BE_FILLED_BY_MODEL}
+
+[[ ## errors ## ]]
+{TO_BE_FILLED_BY_MODEL}
+
+[[ ## risk_level ## ]]
+{TO_BE_FILLED_BY_MODEL}
+
+[[ ## completed ## ]]
+"""
+
+messages = [
+    {"role": "user", "content": prompt}
+]
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+    enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
+)
+model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+
+# conduct text completion
+generated_ids = model.generate(
+    **model_inputs,
+    max_new_tokens=32768
+)
+output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 
+
+# parsing thinking content
+try:
+    # rindex finding 151668 (</think>)
+    index = len(output_ids) - output_ids[::-1].index(151668)
+except ValueError:
+    index = 0
+
+thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
+content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")
+
+print("thinking content:", thinking_content)
+print("content:", content)
+```
+
+# Citation
+
+If you use this model, please cite:
+
+```bibtex
+@article{aali2025expert,
+  title={Expert-level validation of AI-generated medical text with scalable language models},
+  author={Asad Aali and Vasiliki Bikia and Maya Varma and Nicole Chiou and Sophie Ostmeier and Arnav Singhvi and Magdalini Paschali and Ashwin Kumar and Andrew Johnston and Karimar Amador-Martinez and Eduardo Juan Perez Guerrero and Paola Naovi Cruz Rivera and Sergios Gatidis and Christian Bluethgen and Eduardo Pontes Reis and Eddy D. Zandee van Rilland and Poonam Laxmappa Hosamani and Kevin R Keet and Minjoung Go and Evelyn Ling and David B. Larson and Curtis Langlotz and Roxana Daneshjou and Jason Hom and Sanmi Koyejo and Emily Alsentzer and Akshay S. Chaudhari},
+  journal={arXiv preprint arXiv:2507.03152},
+  year={2025}
+}
+```