Reconstruction Results
Hi, I wrote a script to evaluate your model on your dataset. Could you let me know if this matches your approach, or if I made any methodological errors?
https://colab.research.google.com/drive/1HPy-ta2q-7ULXhHLoRyjUJBbSlHrSaAG?usp=sharing
Hi @shmulc !
Thank you so much for trying our model.
I only took a very fast glance at your notebook but one thing that popped out was that you only sampled 50 examples, which is a very small sample. I didn't read anything else yet so there's a chance there are other minor things as well, curios to see the full banchmark results you recreated
Thanks!
It was intentional of course, it just takes a lot of time. To run the full benchmark you only need to comment it out
I didn't evaluated the full dataset, it will take around 2-3 hours on T4 in Collab, and I think that it will be much faster on stronger GPU
https://colab.research.google.com/drive/1HPy-ta2q-7ULXhHLoRyjUJBbSlHrSaAG?usp=sharing
Note:
If I follow your latency estimations it should take 0.09 seconds per sample so 450 seconds overall but it takes much longer.
I'm not sure if it's because there is other bottleneck or that your estimates are on stronger GPU.
Our measurements were on an L4 GPU, I'll check your notebook to see if everything runs on CUDA and if there's anything else