File size: 2,760 Bytes
fb55dfd b253413 fb55dfd b253413 fb55dfd ca13b48 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd ca13b48 b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 fb55dfd b253413 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
---
license: cc-by-nc-nd-4.0
language:
- en
library_name: transformers
tags:
- reward model
- RLHF
- medical
---
# JSL-MedMNX-7B-v4.0
[<img src="https://repository-images.githubusercontent.com/104670986/2e728700-ace4-11ea-9cfc-f3e060b25ddf">](http://www.johnsnowlabs.com)
This model is developed by [John Snow Labs](https://www.johnsnowlabs.com/).
Performance on biomedical benchmarks: [Open Medical LLM Leaderboard](https://huggingface.co/spaces/openlifescienceai/open_medical_llm_leaderboard).
This model is available under a [CC-BY-NC-ND](https://creativecommons.org/licenses/by-nc-nd/4.0/deed.en) license and must also conform to this [Acceptable Use Policy](https://huggingface.co/johnsnowlabs). If you need to license this model for commercial use, please contact us at [email protected].
## 💻 Usage
```python
!pip install -qU transformers accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "johnsnowlabs/JSL-MedMNX-7B-v4.0"
messages = [{"role": "user", "content": "What is a large language model?"}]
tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.float16,
device_map="auto",
)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```
## 🏆 Evaluation
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|-------------------------------|-------|------|-----:|--------|-----:|---|-----:|
|stem |N/A |none | 0|acc_norm|0.5783|± |0.0067|
| | |none | 0|acc |0.6177|± |0.0057|
| - medmcqa |Yaml |none | 0|acc |0.5668|± |0.0077|
| | |none | 0|acc_norm|0.5668|± |0.0077|
| - medqa_4options |Yaml |none | 0|acc |0.6159|± |0.0136|
| | |none | 0|acc_norm|0.6159|± |0.0136|
| - anatomy (mmlu) | 0|none | 0|acc |0.7111|± |0.0392|
| - clinical_knowledge (mmlu) | 0|none | 0|acc |0.7396|± |0.0270|
| - college_biology (mmlu) | 0|none | 0|acc |0.7778|± |0.0348|
| - college_medicine (mmlu) | 0|none | 0|acc |0.6647|± |0.0360|
| - medical_genetics (mmlu) | 0|none | 0|acc |0.7200|± |0.0451|
| - professional_medicine (mmlu)| 0|none | 0|acc |0.7868|± |0.0249|
| - pubmedqa | 1|none | 0|acc |0.7840|± |0.0184|
|