README.md

---
library_name: transformers
tags:
- agent
- action
- vlm
base_model: Qwen/Qwen2.5-VL-3B-Instruct
license: cc-by-nc-4.0
---

# Model Card for Proxy Lite

<!-- Provide a quick summary of what the model is/does. -->

<div align="center">
  <img src="https://raw.githubusercontent.com/convergence-ai/proxy-lite/refs/heads/main/assets/proxy-lite.png" alt="Proxy Lite logo" width="600" height="auto" style="margin-bottom: 20px;" />
  <h2>
    A mini, open-weights, version of <a href="https://proxy.convergence.ai">Proxy</a>.
  </h2>
</div>

## Model Description

<!-- Provide a longer summary of what this model is. -->

- **Developed by:** Convergence AI
- **Model type:** 3B Vision-Language Model
- **Agent type**: Web-browsing Agent
- **License:** CC-BY-NC-4.0
- **Finetuned from model:** Qwen/Qwen2.5-VL-3B-Instruct
- [Running the agent](https://github.com/convergence-ai/proxy-lite)

## Running Proxy on the web

<!-- Provide the basic links for the model. -->

https://github.com/convergence-ai/proxy-lite to run Proxy lite on a browser

  ```
  git clone https://github.com/convergence-ai/proxy-lite.git
  make proxy
  proxy "Find some markets near Kings Cross and tell me their ratings."
  ```

<div align="center">
     <img src="https://raw.githubusercontent.com/convergence-ai/proxy-lite/refs/heads/main/assets/demo.gif" alt="Proxy Lite Demo" />
</div>

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Proxy Lite is designed and trained to complete automated tasks in a web browser.

Full code for running the model is available in the [github repository](https://github.com/convergence-ai/proxy-lite).

This includes a CLI tool for running the model, as well as a streamlit app.

You can use this [endpoint](https://huggingface.co/spaces/convergence-ai/demo-api) for small-scale testing.

---

#### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

We recommend hosting your own endpoint with vLLM, you can use the following command:

```bash
vllm serve convergence-ai/proxy-lite-3b \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --port 8008 \
```

The tool arguments are **very important** for parsing the tool calls from the model appropriately.

> **Important:** Qwen-2.5-VL Support in `transformers` is not yet available in the latest release so be sure to install from source.

#### Message History

When it comes to using and prompting Proxy Lite, please refer to the [repository](https://github.com/convergence-ai/proxy-lite) for more information, but the model expects a message history of the form:

```python
message_history = [
    {
        "role": "system", 
        "content": "You are Proxy Lite...", # Full system prompt in src/proxy_lite/agents/proxy_lite_agent.py
    }, # System prompt
    {
        "role": "user", 
        "content": "Find some markets near Kings Cross and tell me their ratings.",
    }, # Set the task
    {
        "role": "user", 
        "content": [
            {"type": "image_url", "image_url": {base64_encoded_screenshot} },
            {"type": "text", "text": "URL: https://www.google.com/ \n- [0] <a>About</a> \n- [1] <a>Store</a>...."}
        ] # This is the observation from the environment
    },
]
```

This would then build up the message history, alternating between the assistant (who takes the *action*) and the user (who provides the *observation*).

> **Context-Window Management:** When making calls to the model, all the observations other than the current one are discarded in order to reduce the large number of image tokens required. Since the model responses include reflection on the observations and are all included in the message history, the model is still aware of the entire history when planning new actions.

#### Tools

You should also pass the `Tools` that the model has access to, these will define the action space available to the model. You can do this with `transformers`:

```python
from qwen_vl_utils import process_vision_info
from transformers import AutoProcessor

from proxy_lite.tools import ReturnValueTool, BrowserTool
from proxy_lite.serializer import OpenAICompatableSerializer

processor = AutoProcessor.from_pretrained("convergence-ai/proxy-lite-3b")
tools = OpenAICompatableSerializer().serialize_tools([ReturnValueTool(), BrowserTool(session=None)])

templated_messages = processor.apply_chat_template(
    message_history, tokenize=False, add_generation_prompt=True, tools=tools
)

image_inputs, video_inputs = process_vision_info(message_history)

batch = processor(
    text=[templated_messages],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
```

Or you can send to the endpoint directly, which will handle the formatting:

```python
from openai import OpenAI

client = OpenAI(base_url="http://convergence-ai-demo-api.hf.space/v1")

response = client.chat.completions.create(
    model="convergence-ai/proxy-lite-3b",
    messages=message_history,
    tools=tools,
    tool_choice="auto",
)
```

---

## Evaluation

Proxy Lite scored 72.4% on the [WebVoyager](https://huggingface.co/datasets/convergence-ai/WebVoyager2025Valid) benchmark, placing it 1st out of all available open-weights models.

A breakdown of the results by website is shown below:

| web_name            | Success Rate (%) | Finish Rate (%) | Avg. Steps |
|---------------------|-----------------|-----------------|------------|
| Allrecipes         | 87.8            | 95.1            | 10.3       |
| Amazon            | 70.0            | 90.0            | 7.1        |
| Apple             | 82.1            | 89.7            | 10.7       |
| ArXiv             | 60.5            | 79.1            | 16.0       |
| BBC News          | 69.4            | 77.8            | 15.9       |
| Booking           | 70.0            | 85.0            | 24.8       |
| Cambridge Dict.   | 86.0            | 97.7            | 5.7        |
| Coursera          | 82.5            | 97.5            | 4.7        |
| ESPN              | 53.8            | 87.2            | 14.9       |
| GitHub            | 85.0            | 92.5            | 10.0       |
| Google Flights    | 38.5            | 51.3            | 34.8       |
| Google Map        | 78.9            | 94.7            | 9.6        |
| Google Search     | 71.4            | 92.9            | 6.0        |
| Huggingface       | 68.6            | 74.3            | 18.4       |
| Wolfram Alpha     | 78.3            | 93.5            | 6.1        |


---

## Out-of-Scope Use

Proxy Lite is specifically designed to automate routine tasks within a web browser environment. However, it should **not be used** for:

- **High-Stakes or Safety-Critical Applications:**  
  _Avoid using Proxy Lite for tasks such as financial transactions, healthcare operations, legal decision-making, or emergency responses, where any error could lead to serious harm or significant financial loss._

- **Unauthorized or Invasive Data Extraction:**  
  _Automated scraping or extraction of data from websites should only be performed with explicit permission. Proxy Lite should not be used to bypass websites' terms of service, copyright restrictions, or privacy policies._

- **Interactions with Malicious or Unverified Websites:**  
  _Using the model to navigate or interact with suspicious or untrusted websites may expose the system to security threats such as malware, phishing attacks, or other forms of cyber exploitation._

- **Compliance-Regulated or Legally Sensitive Actions:**  
  _Tasks that require adherence to strict legal or regulatory standards (e.g., processing personal data or sensitive information) should employ additional safeguards beyond what the model provides._

---

## Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

```bibtex
@article{proxy-lite,
  title={Proxy Lite - A Mini, Open-weights, Autonomous Assistant},
  author={Convergence AI},
  year={2025}
}
```