nielsr HF Staff commited on
Commit
8c8ec32
·
verified ·
1 Parent(s): f7e8e6f

Improve model card: Add library, abstract, code link, and usage example

Browse files

This PR enhances the model card by:
- Adding `library_name: transformers` to the metadata, which enables the "Use in Transformers" widget and improves discoverability.
- Including the paper abstract for a more comprehensive overview of the model's context.
- Providing a direct link to the associated GitHub repository for easier access to the code and training pipeline.
- Adding a Python code snippet for basic text generation inference using the `transformers` library and the appropriate chat template.
- Adding relevant tags (`science`, `scientific-reasoning`, `qwen`) for better categorization and searchability.

Files changed (1) hide show
  1. README.md +65 -10
README.md CHANGED
@@ -1,27 +1,43 @@
1
  ---
2
- license: apache-2.0
 
3
  datasets:
4
  - MegaScience/MegaScience
5
  language:
6
  - en
 
7
  metrics:
8
  - accuracy
9
- base_model:
10
- - Qwen/Qwen3-1.7B-Base
11
  pipeline_tag: text-generation
 
 
 
 
 
12
  ---
 
13
  # [MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning](https://arxiv.org/abs/2507.16812)
14
 
 
 
 
 
 
 
 
 
 
 
15
  ## Qwen3-1.7B-MegaScience
16
 
17
  ### Training Recipe
18
 
19
- - **LR**: 5e-6
20
- - **LR Schedule**: Cosine
21
- - **Batch Size**: 512
22
- - **Max Length**: 4,096
23
- - **Warm Up Ratio**: 0.05
24
- - **Epochs**: 3
25
 
26
  ### Evaluation Results
27
 
@@ -39,6 +55,45 @@ pipeline_tag: text-generation
39
  <img src="https://cdn-uploads.huggingface.co/production/uploads/616bfc2b40e2f69baa1c7add/VogIpBbjfNxXFP9DfVMms.png" alt="Data Pipeline" style="width:100%;">
40
  </div>
41
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
  ## Citation
43
 
44
  Check out our [paper](https://arxiv.org/abs/2507.16812) for more details. If you use our dataset or find our work useful, please cite
@@ -51,4 +106,4 @@ Check out our [paper](https://arxiv.org/abs/2507.16812) for more details. If you
51
  journal={arXiv preprint arXiv:2507.16812},
52
  url={https://arxiv.org/abs/2507.16812}
53
  }
54
- ```
 
1
  ---
2
+ base_model:
3
+ - Qwen/Qwen3-1.7B-Base
4
  datasets:
5
  - MegaScience/MegaScience
6
  language:
7
  - en
8
+ license: apache-2.0
9
  metrics:
10
  - accuracy
 
 
11
  pipeline_tag: text-generation
12
+ library_name: transformers
13
+ tags:
14
+ - science
15
+ - scientific-reasoning
16
+ - qwen
17
  ---
18
+
19
  # [MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning](https://arxiv.org/abs/2507.16812)
20
 
21
+ This repository contains the **Qwen3-1.7B-MegaScience** model, which is part of the research presented in the paper titled "[MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning](https://arxiv.org/abs/2507.16812)".
22
+
23
+ ### Abstract
24
+
25
+ Scientific reasoning is critical for developing AI scientists and supporting human researchers in advancing the frontiers of natural science discovery. However, the open-source community has primarily focused on mathematics and coding while neglecting the scientific domain, largely due to the absence of open, large-scale, high-quality, verifiable scientific reasoning datasets. To bridge this gap, we first present TextbookReasoning, an open dataset featuring truthful reference answers extracted from 12k university-level scientific textbooks, comprising 650k reasoning questions spanning 7 scientific disciplines. We further introduce MegaScience, a large-scale mixture of high-quality open-source datasets totaling 1.25 million instances, developed through systematic ablation studies that evaluate various data selection methodologies to identify the optimal subset for each publicly available scientific dataset. Meanwhile, we build a comprehensive evaluation system covering diverse subjects and question types across 15 benchmarks, incorporating comprehensive answer extraction strategies to ensure accurate evaluation metrics. Our experiments demonstrate that our datasets achieve superior performance and training efficiency with more concise response lengths compared to existing open-source scientific datasets. Furthermore, we train Llama3.1, Qwen2.5, and Qwen3 series base models on MegaScience, which significantly outperform the corresponding official instruct models in average performance. In addition, MegaScience exhibits greater effectiveness for larger and stronger models, suggesting a scaling benefit for scientific tuning. We release our data curation pipeline, evaluation system, datasets, and seven trained models to the community to advance scientific reasoning research.
26
+
27
+ ### Code / Project Page
28
+
29
+ The official code and data processing pipeline can be found at the [MegaScience GitHub repository](https://github.com/GAIR-NLP/lm-open-science-evaluation).
30
+
31
  ## Qwen3-1.7B-MegaScience
32
 
33
  ### Training Recipe
34
 
35
+ - **LR**: 5e-6
36
+ - **LR Schedule**: Cosine
37
+ - **Batch Size**: 512
38
+ - **Max Length**: 4,096
39
+ - **Warm Up Ratio**: 0.05
40
+ - **Epochs**: 3
41
 
42
  ### Evaluation Results
43
 
 
55
  <img src="https://cdn-uploads.huggingface.co/production/uploads/616bfc2b40e2f69baa1c7add/VogIpBbjfNxXFP9DfVMms.png" alt="Data Pipeline" style="width:100%;">
56
  </div>
57
 
58
+ ## Usage
59
+
60
+ This model can be loaded and used directly with the Hugging Face `transformers` library. Since this is a Qwen-based model, it supports the chat template for structured conversations.
61
+
62
+ ```python
63
+ from transformers import AutoModelForCausalLM, AutoTokenizer
64
+ import torch
65
+
66
+ model_id = "MegaScience/Qwen3-1.7B-MegaScience"
67
+
68
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
69
+ model = AutoModelForCausalLM.from_pretrained(
70
+ model_id,
71
+ torch_dtype=torch.bfloat16, # Use bfloat16 if your GPU supports it, otherwise float16
72
+ device_map="auto",
73
+ )
74
+
75
+ messages = [
76
+ {"role": "user", "content": "What is the primary function of mitochondria?\
77
+ Answer:"},
78
+ ]
79
+
80
+ # Apply chat template and generate response
81
+ text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
82
+ model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
83
+
84
+ generated_ids = model.generate(
85
+ model_inputs.input_ids,
86
+ max_new_tokens=100,
87
+ do_sample=True,
88
+ temperature=0.7,
89
+ top_p=0.9,
90
+ eos_token_id=tokenizer.eos_token_id,
91
+ )
92
+
93
+ generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
94
+ print(generated_text)
95
+ ```
96
+
97
  ## Citation
98
 
99
  Check out our [paper](https://arxiv.org/abs/2507.16812) for more details. If you use our dataset or find our work useful, please cite
 
106
  journal={arXiv preprint arXiv:2507.16812},
107
  url={https://arxiv.org/abs/2507.16812}
108
  }
109
+ ```