CFPB Assistant
🐨
CFPB Assistant using vectara-agentic
Vectara
company
AI & ML interests
retrieval augmented generation, grounded generation, large language models, LLMs, question answering, chatbot
Recent Activity
vectara's activity
ofermend
updated
4
Spaces
4 days ago
ofermend
updated
5
Spaces
5 days ago
17
Finance assistant
🐨
Finance chatbot using vectara-agentic
6
Hacker News chat
🐨
chatbot with HN data using vectara-agentic
2
Justice Harvard
🐨
Teacher Assistant for Justice Harvard using vectara-agentic
HMC Demo
🐨
Ask questions about Harvard Management
1
SuperMicro Demo
🐨
Ask questions about Supermicro documents
Post
333
Excited to share open-rag-eval (https://github.com/vectara/open-rag-eval) a new open source project to help scale RAG evaluation. The key benefit: it does not require golden answers so much more scalable.
Would love any thoughts or feedback (or even better - if you want to contribute a PR that would be great).
Would love any thoughts or feedback (or even better - if you want to contribute a PR that would be great).
stsui96
updated
a
dataset
9 days ago
joao-vectara
published
a
Space
14 days ago
Post
1587
Last year, I curated & generated a few multilingual SFT and DPO datasets by translating English SFT/DPO datasets into 9-10 languages using the
mistralai/Mistral-7B-Instruct-v0.2 model.
I hope it helps the community for pretraining/instruction tuning multilingual LLMs! I added a small diagram to briefly describe which datasets are added and their sources.
Happy to collaborate in either using these datasets for instruction FT, or wishes to extend translated versions of newer SFT/DPO english datasets!
nthakur/multilingual-sft-and-dpo-datasets-67eaf56fe3feca5a57cf7d74
I hope it helps the community for pretraining/instruction tuning multilingual LLMs! I added a small diagram to briefly describe which datasets are added and their sources.
Happy to collaborate in either using these datasets for instruction FT, or wishes to extend translated versions of newer SFT/DPO english datasets!
nthakur/multilingual-sft-and-dpo-datasets-67eaf56fe3feca5a57cf7d74
clefourrier
posted
an
update
about 2 months ago
Post
2371
Gemma3 family is out! Reading the tech report, and this section was really interesting to me from a methods/scientific fairness pov.
Instead of doing over-hyped comparisons, they clearly state that **results are reported in a setup which is advantageous to their models**.
(Which everybody does, but people usually don't say)
For a tech report, it makes a lot of sense to report model performance when used optimally!
On leaderboards on the other hand, comparison will be apples to apples, but in a potentially unoptimal way for a given model family (like some user interact sub-optimally with models)
Also contains a cool section (6) on training data memorization rate too! Important to see if your model will output the training data it has seen as such: always an issue for privacy/copyright/... but also very much for evaluation!
Because if your model knows its evals by heart, you're not testing for generalization.
Instead of doing over-hyped comparisons, they clearly state that **results are reported in a setup which is advantageous to their models**.
(Which everybody does, but people usually don't say)
For a tech report, it makes a lot of sense to report model performance when used optimally!
On leaderboards on the other hand, comparison will be apples to apples, but in a potentially unoptimal way for a given model family (like some user interact sub-optimally with models)
Also contains a cool section (6) on training data memorization rate too! Important to see if your model will output the training data it has seen as such: always an issue for privacy/copyright/... but also very much for evaluation!
Because if your model knows its evals by heart, you're not testing for generalization.
nthakur
authored
a
paper
2 months ago
clefourrier
authored
a
paper
3 months ago
clefourrier
authored
a
paper
5 months ago
nthakur
authored
a
paper
6 months ago
nthakur
authored
a
paper
9 months ago
Post
1834
I'm excited to share our updated hallucination evaluation model (called HHEM-2.1-Open) as well as the updated leaderboard that ranks LLM by the propensity to hallucinate.
vectara/Hallucination-evaluation-leaderboard
vectara/Hallucination-evaluation-leaderboard
- 1 reply