MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
Paper
•
2505.14963
•
Published
•
1
AIM-Harvard
university
AI & ML interests
Artificial Intelligence in Medicine (AIM) Program (NLP group/Bitterman lab: https://www.bittermanlab.org/)
Recent Activity
View all activity
AIM-Harvard's activity
shanchen
authored
3
papers
about 17 hours ago
shanchen
updated
2
datasets
14 days ago
clefourrier
posted
an
update
18 days ago
Post
595
Always surprised that so few people actually read the FineTasks blog, on
✨how to select training evals with the highest signal✨
If you're serious about training models without wasting compute on shitty runs, you absolutely should read it!!
An high signal eval actually tells you precisely, during training, how wel & what your model is learning, allowing you to discard the bad runs/bad samplings/...!
The blog covers in depth prompt choice, metrics, dataset, across languages/capabilities, and my fave section is "which properties should evals have"👌
(to know on your use case how to select the best evals for you)
Blog: HuggingFaceFW/blogpost-fine-tasks
✨how to select training evals with the highest signal✨
If you're serious about training models without wasting compute on shitty runs, you absolutely should read it!!
An high signal eval actually tells you precisely, during training, how wel & what your model is learning, allowing you to discard the bad runs/bad samplings/...!
The blog covers in depth prompt choice, metrics, dataset, across languages/capabilities, and my fave section is "which properties should evals have"👌
(to know on your use case how to select the best evals for you)
Blog: HuggingFaceFW/blogpost-fine-tasks
-
- 2 replies
shanchen
updated
a
collection
23 days ago
shanchen
published
a
dataset
23 days ago
shanchen
updated
a
dataset
23 days ago
shanchen
published
a
dataset
23 days ago
oskarvanderwal
authored
4
papers
about 2 months ago
Inseq: An Interpretability Toolkit for Sequence Generation Models
Paper
•
2302.13942
•
Published
•
1
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Paper
•
2304.01373
•
Published
•
9
Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model
Paper
•
2310.12611
•
Published
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Paper
•
2211.05100
•
Published
•
32
clefourrier
posted
an
update
3 months ago
Post
2485
Gemma3 family is out! Reading the tech report, and this section was really interesting to me from a methods/scientific fairness pov.
Instead of doing over-hyped comparisons, they clearly state that **results are reported in a setup which is advantageous to their models**.
(Which everybody does, but people usually don't say)
For a tech report, it makes a lot of sense to report model performance when used optimally!
On leaderboards on the other hand, comparison will be apples to apples, but in a potentially unoptimal way for a given model family (like some user interact sub-optimally with models)
Also contains a cool section (6) on training data memorization rate too! Important to see if your model will output the training data it has seen as such: always an issue for privacy/copyright/... but also very much for evaluation!
Because if your model knows its evals by heart, you're not testing for generalization.
Instead of doing over-hyped comparisons, they clearly state that **results are reported in a setup which is advantageous to their models**.
(Which everybody does, but people usually don't say)
For a tech report, it makes a lot of sense to report model performance when used optimally!
On leaderboards on the other hand, comparison will be apples to apples, but in a potentially unoptimal way for a given model family (like some user interact sub-optimally with models)
Also contains a cool section (6) on training data memorization rate too! Important to see if your model will output the training data it has seen as such: always an issue for privacy/copyright/... but also very much for evaluation!
Because if your model knows its evals by heart, you're not testing for generalization.
clefourrier
authored
a
paper
4 months ago
clefourrier
authored
a
paper
6 months ago
shanchen
authored
a
paper
7 months ago
daniellebitt
authored
a
paper
7 months ago