Hugging Face's logo

allenai
/
Llama-3.1-70B-Instruct-RM-RB2

Text Classification
Transformers
PyTorch
English
llama
text-generation-inference
Model card Files
xet
Community
1
Llama-3.1-70B-Instruct-RM-RB2 / README.md
saumyamalik's picture
saumyamalik
Update README.md
664f2a0 verified
preview code
|
raw
history blame
3.53 kB
metadata
license: llama3.1
language:
  - en
pipeline_tag: text-classification
datasets:
  - allenai/llama-3.1-tulu-3-70b-preference-mixture
  - Skywork/Skywork-Reward-Preference-80K-v0.2
base_model:
  - meta-llama/Llama-3.1-70B-Instruct
library_name: transformers

Model Card for {{MODEL_NAME_HERE}}

{{MODEL_NAME_HERE}} is one of 7 sets of reward models (RMs) released with Reward Bench 2. We have released a large set of 70 total reward model checkpoints that we used to develop the benchmark and correlate it with downstream PPO / Best-of-N performance.

Models | Code | Eval. Dataset v2 | Results v2 | Paper

Model Details

The model is a standard classifier, AutoModelForSequenceClassification within the HuggingFace ecosystem, trained on binary preference data. For each model in this batch the main revision is the best model we obtained for that base model, and we include all other training data and hyperparameter combinations in the revisions for further research.

To load a model from a revision, modify the following:

from transformers import AutoModelForSequenceClassification
rm = AutoModelForSequenceClassification("allenai/Llama-3.1-70B-Instruct-RM-RB2", revision="2")
Revision Training Data Learning Rate Num Epochs RewardBench 2 Score Factuality Precise IF Math Safety Focus Ties
main Combined 3e-6 1 76.1 81.3 41.9 69.9 88.4 86.5 88.3
1 Combined 3e-6 1 75.7 81.7 41.2 70.5 87.3 85.5 88.1
2 Combined 1e-6 1 73.1 74.7 37.5 69.4 86.2 80.6 89.9
  • Developed by: Allen Institute for AI
  • Training code: https://github.com/allenai/open-instruct
  • Language(s) (NLP): en
  • License: Llama 3.1 Community License Agreement
  • Finetuned from model [optional]: {{TODO_BASE_MODEL_HERE}}

License

All Llama 3.1 Tülu3 models are released under Meta's Llama 3.1 Community License Agreement. Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. Tülu3 is intended for research and educational use. For more information, please see our Responsible Use Guidelines.

The models have been fine-tuned using a dataset mix with outputs generated from third party models and are subject to additional terms: Gemma Terms of Use and Qwen License Agreement (models were improved using Qwen 2.5).

Citation 

@misc{RewardBench2,
    title={RewardBench 2: Advancing Reward Model Evaluation},
    author={Malik, Saumya and Pyatkin, Valentina and Land, Sander and Morrison, Jacob and Smith, Noah A. and Hajishirzi, Hannaneh and Lambert, Nathan},
    year={2025},
    howpublished={\url{https://huggingface.co/spaces/allenai/reward-bench}},
}

Model card contact: saumyam at allenai dot org