README.md

metadata

title: Optical Motion Capture AI
sdk: docker
app_port: 7860
app_file: app.py

mocap-ai

Functionality to load FBX files, extract animation, process the animation and write it back to the file.

Classifier

• Globals: file with hardcoded values like the marker names.
• Utilities:
  • Split dataset into train/valid/test sets.
  • Visualizations
• Training file loader:
  • Load the .fbx file.
  • Go through each frame in the animation frame range and check if all skeleton nodes have a keyframe there.
  • If a keyframe is missing, remove that frame number from the valid frame numbers.
  • After finding all valid frames, go through all marker translation channels and store the global transform in a pandas DataFrame.
  • Add the actor numbers as categorical variables.
  • Save the DataFrame to a .csv file.
• Inference file loader
  • Same as training file loader, but this one should process all frames regardless of keyframe presence.
• Data augmentation:
  • Isolate a marker set.
  • Translate and rotate (optionally scale) with boundary check.
• Model builder:
  • Instantiate a model with various hyperparameters.
• Training loop:
  • Train given model with callbacks.
• Test loop:
  • Validate model on validation/test data.
• Development script:
  • Create new model, train it and test it.
• Deployment script:
  • Deploys the model in a Docker image on HuggingFace.

References:

1. PointNet:

• Research paper: Qi, Charles R., et al. "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation." CVPR. 2017. arXiv:1612.00593
• Official code repository (TensorFlow): https://github.com/charlesq34/pointnet
• Official code repository (PyTorch): https://github.com/fxia22/pointnet.pytorch

2. PointNet++:

• Research paper: Qi, Charles R., et al. "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space." NeurIPS. 2017. arXiv:1706.02413
• Official code repository (TensorFlow): https://github.com/charlesq34/pointnet2
• Official code repository (PyTorch): https://github.com/erikwijmans/Pointnet2_PyTorch

3. DGCNN:

• Research paper: Wang, Yue, et al. "Dynamic Graph CNN for Learning on Point Clouds." ACM Transactions on Graphics (TOG) 38.5 (2019): 1-12. arXiv:1801.07829
• Official code repository (TensorFlow): https://github.com/WangYueFt/dgcnn
• Official code repository (PyTorch): https://github.com/muhanzhang/DGCNN