library_name: peft
license: apache-2.0
datasets:
- maastrichtlawtech/lleqa
language:
- fr
metrics:
- rouge
- meteor
pipeline_tag: text-generation
inference: false
tags:
- legal
wizardLM-7b-v1.0-lleqa
This is a wizardlm-7b-v1.0 model fine-tuned with QLoRA for long-form legal question answering in French.
Usage
[...]
Training
Data
We use the Long-form Legal Question Answering (LLeQA) dataset to fine-tune the model. LLeQA is a French native dataset for studying legal information retrieval and question answering. It consists of a knowledge corpus of 27,941 statutory articles collected from the Belgian legislation, and 1,868 legal questions posed by Belgian citizens and labeled by experienced jurists with a comprehensive answer rooted in relevant articles from the corpus.
Hyperparameters
We fine-tune the model through 4-bit QLoRA finetuning with an effective batch size of 8 for 10 epochs (i.e., 1.1K steps) using paged AdamW optimizer with default momentum parameters and constant learning rate schedule of 2e-4. We employ NormalFloat4 with double quantization for the base models and add LoRA adapters on all linear layers by setting r=16, alpha=32 while utilizing float16 as computation datatype. Additionally, we perform NTK-aware scaling of RoPE to extend the context window to 4096 tokens. Training takes around 7.5 hours to complete on a single Tesla V100 GPU with 32GBs of memory. More details can be found in this paper and repository.
Citation
@article{louis2023interpretable,
author = {Louis, Antoine and van Dijck, Gijs and Spanakis, Gerasimos},
title = {Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models},
journal = {CoRR},
volume = {abs/2309.17050},
year = {2023},
url = {https://arxiv.org/abs/2309.17050},
eprinttype = {arXiv},
eprint = {2309.17050},
}