|
|
--- |
|
|
title: Optical Motion Capture AI |
|
|
sdk: docker |
|
|
app_port: 7860 |
|
|
app_file: app.py |
|
|
--- |
|
|
|
|
|
# mocap-ai |
|
|
Functionality to load FBX files, extract animation, process the animation and write it back to the file. |
|
|
|
|
|
# Classifier |
|
|
* Globals: file with hardcoded values like the marker names. |
|
|
* Utilities: |
|
|
* Visualizations |
|
|
* FBX Handler: |
|
|
* Load the `.fbx` file. |
|
|
* Go through each frame in the animation frame range and check if all skeleton nodes have a keyframe there. |
|
|
* If a keyframe is missing, remove that frame number from the valid frame numbers. |
|
|
* After finding all valid frames, go through all marker translation channels and store the global transform in a `pandas` DataFrame. |
|
|
* Add the actor numbers as categorical variables. |
|
|
* Save the DataFrame to a `.csv` file. |
|
|
* Inference file loader |
|
|
* Same as training file loader, but this one should process all frames regardless of keyframe presence. |
|
|
* Data augmentation: |
|
|
* Isolate a marker set. |
|
|
* Translate and rotate (optionally scale) with boundary check. |
|
|
* Model builder: |
|
|
* Instantiate a model with various hyperparameters. |
|
|
* Training loop: |
|
|
* Train given model with callbacks. |
|
|
* Test loop: |
|
|
* Validate model on validation/test data. |
|
|
* Development script: |
|
|
* Create new model, train it and test it. |
|
|
* Deployment script: |
|
|
* Deploys the model in a Docker image on HuggingFace. |
|
|
|
|
|
|
|
|
## References: |
|
|
1. PointNet: |
|
|
- Research paper: Qi, Charles R., et al. "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation." CVPR. 2017. [arXiv:1612.00593](https://arxiv.org/abs/1612.00593) |
|
|
- Official code repository (TensorFlow): https://github.com/charlesq34/pointnet |
|
|
- Official code repository (PyTorch): https://github.com/fxia22/pointnet.pytorch |
|
|
2. PointNet++: |
|
|
- Research paper: Qi, Charles R., et al. "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space." NeurIPS. 2017. [arXiv:1706.02413](https://arxiv.org/abs/1706.02413) |
|
|
- Official code repository (TensorFlow): https://github.com/charlesq34/pointnet2 |
|
|
- Official code repository (PyTorch): https://github.com/erikwijmans/Pointnet2_PyTorch |
|
|
3. DGCNN: |
|
|
- Research paper: Wang, Yue, et al. "Dynamic Graph CNN for Learning on Point Clouds." ACM Transactions on Graphics (TOG) 38.5 (2019): 1-12. [arXiv:1801.07829](https://arxiv.org/abs/1801.07829) |
|
|
- Official code repository (TensorFlow): https://github.com/WangYueFt/dgcnn |
|
|
- Official code repository (PyTorch): https://github.com/muhanzhang/DGCNN |
