|
|
--- |
|
|
license: llama3.1 |
|
|
datasets: |
|
|
- RUCKBReasoning/TableLLM-SFT |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- meta-llama/Llama-3.1-8B-Instruct |
|
|
tags: |
|
|
- table |
|
|
- QA |
|
|
- Code |
|
|
--- |
|
|
|
|
|
# TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios |
|
|
|
|
|
| **[Paper](https://arxiv.org/abs/2403.19318)** | **[Training set](https://huggingface.co/datasets/RUCKBReasoning/TableLLM-SFT)** | **[Github](https://github.com/RUCKBReasoning/TableLLM)** | **[Homepage](https://tablellm.github.io/)** | |
|
|
|
|
|
We present **TableLLM**, a powerful large language model designed to handle tabular data manipulation tasks efficiently, whether they are embedded in spreadsheets or documents, meeting the demands of real office scenarios. TableLLM is fine-tuned based on [Llama3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct). |
|
|
|
|
|
TableLLM generates either a code solution or a direct text answer to handle tabular data manipulation tasks based on different scenarios. Code generation is used for handling spreadsheet-embedded tabular data, which often involves the insert, delete, update, query, merge, and plot operations of tables. Text generation is used for handling document-embedded tabular data, which often involves the query operation of short tables. |
|
|
|
|
|
## Evaluation Results |
|
|
We evaluate the code solution generation ability of TableLLM on three benchmarks: WikiSQL, Spider and Self-created table operation benchmark. The text answer generation ability is tested on four benchmarks: WikiTableQuestion (WikiTQ), TAT-QA and FeTaQA. The evaluation result is shown below: |
|
|
|
|
|
| Model | WikiTQ | TAT-QA | FeTaQA | WikiSQL | Spider | Self-created | Average | |
|
|
| :------------------- | :----: | :----: | :----: | :-----: | :----: | :----------: | :-----: | |
|
|
| TaPEX | 38.6 | – | – | 83.9 | 15.0 | / | 45.8 | |
|
|
| TaPas | 31.6 | – | – | 74.2 | 23.1 | / | 43.0 | |
|
|
| TableLlama | 24.0 | 22.3 | 20.5 | 43.7 | - | / | 23.4 | |
|
|
| TableGPT2(7B) | 77.3 | 88.1 | 75.6 | 63.0 | 77.34 | 74.42 | 76.0 |
|
|
| Llama3.1 (8B) | 71.9 | 74.3 | 83.4 | 40.6 | 18.8 | 43.2 | 55.3 | |
|
|
| GPT3.5 | 58.5 | 72.1 | 71.2 | 81.7 | 67.4 | 77.1 | 69.8 | |
|
|
| GPT4o |**91.5**|**91.5**|**94.4**|<ins>84.0</ins>| 69.5 |<ins>77.8</ins>|<ins>84.8</ins>| |
|
|
| CodeLlama (13B) | 43.4 | 47.3 | 57.2 | 38.3 | 21.9 | 47.6 | 43.6 | |
|
|
| Deepseek-Coder (33B) | 6.5 | 11.0 | 7.1 | 72.5 | 58.4 | 73.9 | 33.8 | |
|
|
| StructGPT (GPT3.5) | 52.5 | 27.5 | 11.8 | 67.8 |**84.8**| / | 43.1 | |
|
|
| Binder (GPT3.5) | 61.6 | 12.8 | 6.9 | 78.6 | 52.6 | / | 36.3 | |
|
|
| DATER (GPT3.5) | 53.4 | 28.5 | 18.3 | 58.2 | 26.5 | / | 33.0 | |
|
|
| TableLLM-8B (Ours) |<ins>89.1</ins>|<ins>89.5</ins>|<ins>93.4</ins>|**89.6**|<ins>81.1</ins>|<ins>77.8</ins>|**86.7**| |
|
|
|
|
|
## Prompt Template |
|
|
The prompts we used for generating code solutions and text answers are introduced below. |
|
|
|
|
|
### Code Solution |
|
|
The prompt template for the insert, delete, update, query, and plot operations on a single table. |
|
|
``` |
|
|
[INST]Below are the first few lines of a CSV file. You need to write a Python program to solve the provided question. |
|
|
|
|
|
Header and first few lines of CSV file: |
|
|
{csv_data} |
|
|
|
|
|
Question: {question}[/INST] |
|
|
``` |
|
|
|
|
|
The prompt template for the merge operation on two tables. |
|
|
``` |
|
|
[INST]Below are the first few lines two CSV file. You need to write a Python program to solve the provided question. |
|
|
|
|
|
Header and first few lines of CSV file 1: |
|
|
{csv_data1} |
|
|
|
|
|
Header and first few lines of CSV file 2: |
|
|
{csv_data2} |
|
|
|
|
|
Question: {question}[/INST] |
|
|
``` |
|
|
|
|
|
The csv_data field is filled with the first few lines of your provided table file. Below is an example: |
|
|
``` |
|
|
Sex,Length,Diameter,Height,Whole weight,Shucked weight,Viscera weight,Shell weight,Rings |
|
|
M,0.455,0.365,0.095,0.514,0.2245,0.101,0.15,15 |
|
|
M,0.35,0.265,0.09,0.2255,0.0995,0.0485,0.07,7 |
|
|
F,0.53,0.42,0.135,0.677,0.2565,0.1415,0.21,9 |
|
|
M,0.44,0.365,0.125,0.516,0.2155,0.114,0.155,10 |
|
|
I,0.33,0.255,0.08,0.205,0.0895,0.0395,0.055,7 |
|
|
``` |
|
|
|
|
|
### Text Answer |
|
|
The prompt template for direct text answer generation on short tables. |
|
|
```` |
|
|
[INST]Offer a thorough and accurate solution that directly addresses the Question outlined in the [Question]. |
|
|
### [Table Text] |
|
|
{table_descriptions} |
|
|
|
|
|
### [Table] |
|
|
``` |
|
|
{table_in_csv} |
|
|
``` |
|
|
|
|
|
### [Question] |
|
|
{question} |
|
|
|
|
|
### [Solution][INST/] |
|
|
```` |
|
|
|
|
|
For more details about how to use TableLLM, please refer to our GitHub page: <https://github.com/TableLLM/TableLLM> |
