zjunlp/DataMind-Qwen2.5-7B

✨ DataMind

🔧 Installation

🔩Manual Environment Configuration

Conda virtual environments offer a light and flexible setup.

Prerequisites

• Anaconda Installation
• GPU support (recommended CUDA version: 12.4)

Configure Steps

1. Clone the repository:

git clone https://github.com/zjunlp/DataMind.git

2. Enter the working directory, and all subsequent commands should be executed in this directory.

cd DataMind/eval

3. Create a virtual environment using Anaconda.

conda create -n DataMind python=3.10
conda activate DataMind

4. Install all required Python packages.

pip install -r requirements.txt

🧐 Evaluation

Note:

• Ensure that your working directory is set to the eval folder in a virtual environment.
• If you have more questions, feel free to open an issue with us.
• If you need to use local model, you need to deploy it according to (Optional)local_model.sh.

Step 1: Prepare the parameter configuration

The evaluation datasets we used are in QRData and DiscoveryBench. The script expects data to be at data/QRData/benchmark/data/*.csv and data/DiscoveryBench/*.csv.

You can also download our sft models directly from Hugging Face: DataMind-Qwen2.5-7B ,DataMind-Qwen2.5-14B .

Here is the example: config.yaml

api_key: your_api_key # your API key for the model with API service. No need for open-source models.
data_root: /path/to/your/project/DataMind/eval/data # Root directory for data. (absolute path)

run_eval.sh

python do_generate.py \
  --model_name DataMind-Qwen2.5-7B \  # Model name to use.
  --check_model gpt-4o-mini \  # Check model to use.
  --output results \  # Output directory path.
  --dataset_name QRData \  # Dataset name to use, chosen from QRData, DiscoveryBench.
  --max_round 25 \  # Maximum number of steps.
  --api_port 8000 \  # API port number, it is necessary if the local model is used.
  --bidx 0 \  # Begin index (inclusive), `None` indicates that there is no restriction.
  --eidx None \  # End index (exclusive), `None` indicates that there is no restriction.
  --temperature 0.0 \  # Temperature for sampling.
  --top_p 1 \  # Top p for sampling.
  --add_random False \  # Whether to add random files.

(Optional)local_model.sh

CUDA_VISIBLE_DEVICES=$i python -m vllm.entrypoints.openai.api_server \
  --model $MODEL_PATH \ # Local model path.
  --served-model-name $MODEL_NAME \ # The model name specified by you.
  --tensor-parallel-size $i \ # Set the size of tensor parallel processing.
  --port $port # API port number, which is consistent with the `api_port` above.

Step 2: Run the shell script

(Optional) Deploy the local model if you need.

bash local_model.sh

Run the shell script to start the process.

bash run_eval.sh

✍️ Citation

If you find our work helpful, please use the following citations.

@article{zhu2025open,
  title={Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Study},
  author={Zhu, Yuqi and Zhong, Yi and Zhang, Jintian and Zhang, Ziheng and Qiao, Shuofei and Luo, Yujie and Du, Lun and Zheng, Da and Chen, Huajun and Zhang, Ningyu},
  journal={arXiv preprint arXiv:2506.19794},
  year={2025}
}