README.md

---
license: apache-2.0
tags:
- computer_vision
- pose_estimation
---

Copyright 2021-2023 by Mackenzie Mathis, Alexander Mathis, Shaokai Ye and contributors. All rights reserved.


- Please cite **Ye et al 2023** if you use this model in your work https://arxiv.org/abs/2203.07436v1 
- If this license is not suitable for your business or project
please contact EPFL-TTO (https://tto.epfl.ch/) for a full commercial license.

This software may not be used to harm any animal deliberately!


**MODEL CARD:**

This model was trained a dataset called "Quadrupred-40K." It was trained in PyTorch within a modifed [mmpose framework](https://github.com/open-mmlab/mmpose), available within the [DeepLabCut framework](www.deeplabcut.org). 
Full training details can be found in Ye et al. 2023, but in brief, this was trained with **HRNet**. We have another version available directly within the tensorflow version of DeepLabCut: https://huggingface.co/mwmathis/DeepLabCutModelZoo-SuperAnimal-Quadruped.


**Training Data:**

It consists of being trained together on the following datasets:

- **AwA-Pose** Quadruped dataset, see full details at (1).
- **AnimalPose** See full details at (2).
- **AcinoSet** See full details at (3).
- **Horse-30** Horse-30 dataset, benchmark task is called Horse-10; See full details at (4).
- **StanfordDogs** See full details at (5, 6).
- **AP-10K** See full details at (7).
- **iRodent** (https://zenodo.org/record/8250392) We utilized the iNaturalist API functions for scraping observations
with the taxon ID of Suborder Myomorpha (8). The functions allowed us to filter the large amount of observations down to the
ones with photos under the CC BY-NC creative license. The most common types of rodents from the collected observations are
Muskrat (Ondatra zibethicus), Brown Rat (Rattus norvegicus), House Mouse (Mus musculus), Black Rat (Rattus rattus), Hispid
Cotton Rat (Sigmodon hispidus), Meadow Vole (Microtus pennsylvanicus), Bank Vole (Clethrionomys glareolus), Deer Mouse
(Peromyscus maniculatus), White-footed Mouse (Peromyscus leucopus), Striped Field Mouse (Apodemus agrarius). We then
generated segmentation masks over target animals in the data by processing the media through an algorithm we designed that
uses a Mask Region Based Convolutional Neural Networks(Mask R-CNN) (9) model with a ResNet-50-FPN backbone (10),
pretrained on the COCO datasets (11). The processed 443 images were then manually labeled with both pose annotations and
segmentation masks.

Here is an image with the keypoint guide, the distribution of images per dataset, and examples from the datasets inferenced with a model trained with less data for benchmarking as in Ye et al 2023. 
Thereby note that performance of this model we are releasing has comporable or higher performance. 

Please note that each dataest was labeled by separate labs & seperate individuals, therefore while we map names
to a unified pose vocabulary, there will be annotator bias in keypoint placement (See Ye et al. 2023 for our Supplementary Note on annotator bias). 
You will also note the dataset is highly diverse across species, but collectively has more representation of domesticated animals like dogs, cats, horses, and cattle. 
We recommend if performance is not as good as you need it to be, first try video adaptation (see Ye et al. 2023), 
or fine-tune these weights with your own labeling. 

<p align="center">
<img src="https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1690988780004-AG00N6OU1R21MZ0AU9RE/modelcard-SAQ.png?format=1500w" width="95%">
</p>


1. Prianka Banik, Lin Li, and Xishuang Dong. A novel dataset for keypoint detection of quadruped animals from images. ArXiv, abs/2108.13958, 2021
2. Jinkun Cao, Hongyang Tang, Haoshu Fang, Xiaoyong Shen, Cewu Lu, and Yu-Wing Tai. Cross-domain adaptation for animal pose estimation.
2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9497–9506, 2019.
3. Daniel Joska, Liam Clark, Naoya Muramatsu, Ricardo Jericevich, Fred Nicolls, Alexander Mathis, Mackenzie W. Mathis, and Amir Patel. Acinoset:
A 3d pose estimation dataset and baseline models for cheetahs in the wild. 2021 IEEE International Conference on Robotics and Automation
(ICRA), pages 13901–13908, 2021.
4. Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yuksekgonul, Byron Rogers, Matthias Bethge, and Mackenzie W Mathis. Pretraining
boosts out-of-domain robustness for pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,
pages 1859–1868, 2021.
5. Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop
on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
6. Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla. Creatures great and smal: Recovering the shape and motion of
animals from video. In Asian Conference on Computer Vision, pages 3–19. Springer, 2018.
7. Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. Ap-10k: A benchmark for animal pose estimation in the wild. In Thirty-fifth
Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
8. iNaturalist. OGBIF Occurrence Download. https://doi.org/10.15468/dl.p7nbxt. iNaturalist, July 2020
9. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer
vision, pages 2961–2969, 2017.
10. Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection, 2016.
11. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’ar,
and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014