---
tags:
- multimodal
- NPU
- On-device
- Snapdragon PC
- Android
license: other
license_name: nexa-research
license_link: LICENSE
---
# **OmniNeural** — World’s First NPU-aware Multimodal Model
## **Overview**
**OmniNeural** is the first fully multimodal model designed specifically for Neural Processing Units (NPUs). It natively understands **text, images, and audio**, and runs across PCs, mobile devices, automobile, IoT, and robotics.
### **Demo running on Snapdragon NPU in Samsung S25 Ultra.**
A fully local, multimodal conversational AI assistant that hears you and sees what you see is finally possible. And it runs on NPU, keeping the battery life long.
---
## **Key Features**
- **Multimodal Intelligence** – Processes **text, image, and audio** in a unified model for richer reasoning and perception.
- **NPU-Optimized Architecture** – Uses ReLU ops, sparse tensors, convolutional layers, and static graph execution for maximum throughput — **20% faster than non-NPU-aware models** .
- **Hardware-Aware Attention** – Attention patterns tuned for NPU, lowering compute and memory demand .
- **Native Static Graph** – Supports variable-length multimodal inputs with stable, predictable latency .
- **Performance Gains** – **9× faster audio processing** and **3.5× faster image processing** on NPUs compared to baseline encoders .
- **Privacy-First Inference** – All computation stays local: private, offline-capable, and cost-efficient.
---
## **Performance / Benchmarks**
### Human Evaluation (vs baselines)
- **Vision**: Wins/ties in ~75% of prompts against Apple Foundation, Gemma-3n-E4B, Qwen2.5-Omni-3B.
- **Audio**: Clear lead over baselines, much better than Gemma3n and Apple foundation model.
- **Text**: Matches or outperforms leading multimodal baselines.
### Nexa Attention Speedups
- **9× faster** audio encoding (vs Whisper encoder).
- **3.5× faster** image encoding (vs SigLIP encoder).
---
## **Architecture Overview**
OmniNeural’s design is tightly coupled with NPU hardware:
- **NPU-friendly ops** (ReLU > GELU/SILU).
- **Sparse + small tensor multiplications** for efficiency.
- **Convolutional layers** favored over linear for better NPU parallelization.
- **Hardware-aware attention** patterns to cut compute cost.
- **Static graph execution** for predictable latency.
---
## **Production Use Cases**
- **PC & Mobile** – On-device AI agents combine **voice, vision, and text** for natural, accurate responses.
- Examples: Summarize slides into an email (PC)*, *extract action items from chat (mobile).
- Benefits: Private, offline, battery-efficient.
- **Automotive** – In-car assistants handle **voice control, cabin safety, and environment awareness**.
- Examples: Detects risks (child unbuckled, pet left, loose objects) and road conditions (fog, construction).
- Benefits: Decisions run locally in milliseconds.
- **IoT & Robotics** – Multimodal sensing for **factories, AR/VR, drones, and robots**.
- Examples: Defect detection, technician overlays, hazard spotting mid-flight, natural robot interaction.
- Benefits: Works without network connectivity.
---
## **How to use** //TODO
> ⚠️ Note: OmniNeural currently runs on Qualcomm NPUs (Snapdragon devices).
> Apple NPU support is planned for the next release.
**Install via Nexa-SDK:**
## **Links & Community** //TODO
## **License & Citation**
```bibtex
@misc{
title={OmniNeural: NPU-Optimized Multimodal Model for On-Device AI},
author={Nexa AI},
year={2025},
primaryClass={cs.CV},
url={https://huggingface.co/NexaAI/OmniNeural-4B},
}
```