---
title: JARVIS Gaia Agent
emoji: 🦾
colorFrom: indigo
colorTo: green
sdk: gradio
pinned: false
license: mit
short_description: Enhanced JARVIS AI agent for GAIA benchmark
models:
  - meta-llama/Llama-3.2-1B-Instruct
  - sentence-transformers/all-MiniLM-L6-v2
datasets:
  - gaia-benchmark/GAIA
---

# Evolved JARVIS Gaia Agent

An advanced Python-based AI agent built with `langchain`, `langgraph`, SERPAPI, and OCR capabilities for web searches, file parsing, image analysis, and data retrieval. Deployed as a Hugging Face Space (`onisj/jarvis_gaia_agent`) for evaluating performance on the GAIA benchmark, targeting a score >30% (6/20 correct).

## Features

- **Web Search**: Integrates SERPAPI and DuckDuckGo for robust, multi-hop searches.
- **File Parsing**: Processes CSV, TXT, Excel, and PDF files for GAIA tasks.
- **Image Parsing**: Uses OCR (`easyocr`) to extract text from images.
- **Data Retrieval**: Includes a guest info retriever for structured queries.
- **External APIs**: Supports weather data (OpenWeatherMap) and Hugging Face Hub stats.
- **State Management**: Employs `langgraph` for multi-step reasoning workflows.
- **Exact-Match Answers**: Optimized for GAIA Level 1 questions with precise formatting (e.g., USD to two decimals, comma-separated lists).
- **Gradio Interface**: Provides a user-friendly UI for running evaluations and submitting answers.

## Directory Structure

```
jarvis_gaia_agent/
├── app.py                  # Main Gradio application with agent logic
├── state.py                # Defines JARVISState for LangGraph state management
├── search.py               # Web search tools (SERPAPI, multi-hop search)
├── tools/                  # Directory for all tools
│   ├── __init__.py         # Exports all tools
│   ├── file_parser.py      # Parses CSV, TXT, Excel, and PDF files
│   ├── image_parser.py     # OCR-based image parsing
│   ├── calculator.py       # Mathematical calculations
│   ├── document_retriever.py # PDF document retrieval
│   ├── duckduckgo_search.py # DuckDuckGo search integration
│   ├── weather_info.py     # Weather data via OpenWeatherMap
│   ├── hub_stats.py        # Hugging Face Hub statistics
│   ├── guest_info.py       # Guest information retrieval
├── requirements.txt        # Python dependencies
├── README.md               # Project documentation
├── .gitignore              # Excludes .env, temp/, etc.
├── temp/                   # Temporary directory for GAIA files (created at runtime)
```

## Models and Datasets

- **Models**:
  - `meta-llama/Llama-3.2-1B-Instruct`: Primary LLM for reasoning and tool selection (Hugging Face Inference API or local).
  - `sentence-transformers/all-MiniLM-L6-v2`: Embedding model for text similarity tasks.
  - Note: Together AI models (`meta-llama/Llama-3.3-70B-Instruct-Turbo-Free`, `deepseek-ai/DeepSeek-R1-Distill-Llama-70B-free`) are used via API but not hosted on Hugging Face, so they’re not listed in metadata.
- **Datasets**:
  - `gaia-benchmark/GAIA`: Benchmark dataset for evaluating agent performance.

## Prerequisites

- **Python**: 3.9 or higher.
- **Tesseract OCR**: Required for image parsing.
  - macOS: `brew install tesseract`
  - Ubuntu: `sudo apt-get install tesseract-ocr`
  - Windows: Install via [Tesseract Installer](https://github.com/UB-Mannheim/tesseract/wiki).
- **API Keys**: Set in `.env` (local) or Hugging Face Space Secrets (deployment):
  - `HUGGINGFACEHUB_API_TOKEN`: Hugging Face token for model access.
  - `TOGETHER_API_KEY`: Together AI API key for LLM inference.
  - `SERPAPI_API_KEY`: SERPAPI key for web searches.
  - `OPENWEATHERMAP_API_KEY`: OpenWeatherMap key for weather queries.
  - `SPACE_ID`: `onisj/jarvis_gaia_agent`.
- Install dependencies:
  ```bash
  pip install -r requirements.txt
  ```

## Setup and Local Testing

1. **Clone the Repository**:
   ```bash
   git clone https://huggingface.co/spaces/onisj/jarvis_gaia_agent
   cd jarvis_gaia_agent
   ```

2. **Create Virtual Environment**:
   ```bash
   python -m venv venv
   source venv/bin/activate  # Windows: venv\Scripts\activate
   ```

3. **Install Dependencies**:
   ```bash
   pip install -r requirements.txt
   ```

4. **Configure Environment Variables**:
   Create a `.env` file:
   ```text
   SPACE_ID=onisj/jarvis_gaia_agent
   HUGGINGFACEHUB_API_TOKEN=your_hf_token
   TOGETHER_API_KEY=your_together_api_key
   SERPAPI_API_KEY=your_serpapi_key
   OPENWEATHERMAP_API_KEY=your_openweather_key
   ```

5. **Test with Mock File** (optional):
   ```bash
   mkdir temp
   echo "Item,Type,Sales\nBurger,Food,1000\nCola,Drink,500" > temp/7bd855d8-463d-4ed5-93ca-5fe35145f733.xlsx
   ```

6. **Run Locally**:
   ```bash
   python app.py
   ```
   - Open `http://127.0.0.1:7860` (port may vary).
   - Log in with Hugging Face credentials.
   - Click “Run Evaluation & Submit All Answers” to test GAIA tasks.

## Deployment to Hugging Face Space

1. **Push Code**:
   ```bash
   git add .
   git commit -m "Update JARVIS Gaia Agent with README metadata"
   git push origin main
   ```

2. **Set Space Secrets**:
   - Go to `https://huggingface.co/spaces/onisj/jarvis_gaia_agent` > Settings > Repository Secrets.
   - Add:
     - `SPACE_ID`: `onisj/jarvis_gaia_agent`
     - `HUGGINGFACEHUB_API_TOKEN`
     - `TOGETHER_API_KEY`
     - `SERPAPI_API_KEY`
     - `OPENWEATHERMAP_API_KEY`

3. **Build and Run**:
   - Hugging Face auto-builds the Space after pushing.
   - Access the Gradio interface at `https://onisj-jarvis-gaia-agent.hf.space`.
   - Log in and click “Run Evaluation & Submit All Answers” to submit GAIA answers.

4. **Verify Submission**:
   - Check `status_output` for:
     ```
     Submission Successful!
     User: your_username
     Overall Score: XX% (Y/20 correct)
     Message: ...
     ```
   - Aim for >30% (6/20 correct).

## Troubleshooting

- **Model Access (404)**: Verify API keys; test `initialize_llm` locally.
- **SERPAPI Timeout**: Ensure `SERPAPI_API_KEY` is valid; check `search.py` logs.
- **GAIA File Access**: Confirm `temp/` directory permissions; test `download_file`.
- **Low GAIA Score**: Analyze `results_table` for errors; enhance `multi_hop_search_tool` or answer formatting.
- **Logs**: Check Space > Settings > Logs for build/run errors.

## License

MIT License. See [LICENSE](LICENSE) for details.

## Acknowledgements

- Built with `langchain`, `langgraph`, and Hugging Face tools.
- Evaluated on the GAIA benchmark (`gaia-benchmark/GAIA`).