README.md

metadata

title: fbaldassarri Repository Eval
emoji: 💻
colorFrom: indigo
colorTo: green
sdk: streamlit
sdk_version: 1.44.1
app_file: app.py
pinned: true
license: apache-2.0
short_description: Evaluation space for fbaldassarri Repository

fbaldassarri Model Comparison Tool

🌟 Features

• Model Discovery: Automatically finds all quantized models in your HuggingFace account
• Smart Filtering: Filter models by quantization method, base model, or custom text
• Detailed Metadata Analysis: Compare model architectures, sizes, and quantization details
• Repository Statistics: Track downloads, likes, and update frequency
• Interactive Visualizations: Multiple charts and graphs for performance analysis
• Selective Comparison: Use .modelsignore to exclude irrelevant models
• CPU-Optimized: Works perfectly on free tier HuggingFace Spaces without GPU
• Export Capabilities: Download comparison results as CSV for further analysis

📋 Usage Instructions

Basic Usage

1. Enter your HuggingFace username
2. The tool will automatically discover all your quantized models
3. Use the filters to select models by quantization method or base model
4. Click "Run Comparison" to analyze the selected models
5. View the results in the "Model Comparison", "Model Details", and "Visualizations" tabs

Advanced Features

Filtering Models

• Quantization Method: Select specific quantization methods (Intel AutoRound, AutoGPTQ, AutoAWQ)
• Base Model: Compare models derived from the same base architecture
• Text Filter: Find models containing specific text in their names

Using .modelsignore

Create a .modelsignore file in the same directory as app.py to exclude specific models from discovery:

# Comments start with #
fbaldassarri/llama-2-7b-*  # Ignores all llama-2-7b models
*mistral*                  # Ignores anything with "mistral" in the name
fbaldassarri/exact-model-name  # Ignores a specific model

Supports wildcards and exact matching patterns.

Comparison Methods

Two comparison methods are available:

1. Metadata Comparison Only: Fast analysis of model metadata and architecture
2. Metadata + Estimated Size: Additionally calculates the disk size of each model

📊 Visualization Types

The tool provides multiple visualization options:

1. Quantization Methods

• Distribution of quantization methods (pie chart)
• Distribution of precision formats (bar chart)
• Distribution of group sizes (bar chart)

2. Model Architecture

• Model size comparison (bar chart)
• Layer count comparison (bar chart)
• Hidden size comparison (bar chart)
• Attention heads comparison (bar chart)

3. Repository Statistics

• Downloads comparison (bar chart)
• Likes comparison (bar chart)
• Days since update comparison (bar chart)

🔧 Technical Details

Supported Quantization Methods

The tool is designed to detect and analyze:

• Intel AutoRound: Intel's quantization solution
• AutoGPTQ: Automatic GPTQ quantization
• AutoAWQ: Activation-aware weight quantization

Metadata Extraction

For each model, the tool extracts:

• Quantization method (based on name and tags)
• Precision (INT4, INT8, FP16, FP32)
• Group size (e.g., gs128)
• Base model architecture
• Hidden size and layer count
• Attention head count
• Maximum sequence length

Size Estimation

When the "Metadata + Estimated Size" option is selected, the tool:

1. Fetches the list of model files from the repository
2. Identifies .bin and .safetensors files
3. Calculates their total size in GB

This avoids downloading the actual model weights, making it CPU-friendly.

🔍 Example Use Cases

1. Comparing Quantization Methods: Determine which method (AutoGPTQ vs AutoAWQ) produces smaller models
2. Quantization Parameter Analysis: See how different group sizes affect model size
3. Popularity Tracking: Identify which of your quantized models has the most downloads
4. Architecture Analysis: Compare hidden sizes and layer counts across different quantized variants

⚠️ Troubleshooting

Common Issues

1. Slow Model Discovery
   • The tool may take time to discover models if you have many repositories
   • Use the .modelsignore file to exclude irrelevant models and speed up discovery
2. Memory Issues
   • The application is optimized for CPU usage, but comparing many models simultaneously may cause memory pressure
   • Use the "Maximum models to compare" slider to limit the number of models
3. Error: "Cannot subtract tz-naive and tz-aware datetime-like objects"
   • This error has been fixed in the latest version
   • Update to the latest version if you encounter it
4. Missing Model Information
   • Some information may be unavailable if models don't have standard config files
   • The tool will show "Unknown" for fields it couldn't extract

📝 Requirements

• Python 3.9+
• huggingface_hub
• streamlit
• pandas
• plotly
• numpy

See requirements.txt for a complete list of dependencies.

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgements

• HuggingFace for the Hub API and Spaces platform
• The developers of AutoGPTQ, AutoAWQ, and Intel AutoRound
• Streamlit for the interactive web framework

👤 Author

Created by Francesco Baldassarri

🤝 Contributing

Contributions, issues, and feature requests are welcome!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request