Spaces:
				
			
			
	
			
			
		Runtime error
		
	
	
	
			
			
	
	
	
	
		
		
		Runtime error
		
	
		Stanislaw Szymanowicz
		
	commited on
		
		
					Commit 
							
							·
						
						15f992e
	
1
								Parent(s):
							
							11ffd30
								
Add examples
Browse files- app.py +58 -12
- demo_examples/01_bigmac.png +0 -0
- demo_examples/02_hydrant.jpg +0 -0
- demo_examples/03_spyro.png +0 -0
- demo_examples/04_lysol.png +0 -0
- demo_examples/05_pinapple_bottle.png +0 -0
- demo_examples/06_unsplash_broccoli.png +0 -0
- demo_examples/07_objaverse_backpack.png +0 -0
- demo_examples/08_unsplash_chocolatecake.png +0 -0
- demo_examples/09_realfusion_cherry.png +0 -0
- demo_examples/10_triposr_teapot.png +0 -0
    	
        app.py
    CHANGED
    
    | @@ -95,22 +95,23 @@ def main(): | |
| 95 |  | 
| 96 | 
             
                    return ply_out_path
         | 
| 97 |  | 
| 98 | 
            -
                 | 
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
| 99 | 
             
                    gr.Markdown(
         | 
| 100 | 
             
                        """
         | 
|  | |
| 101 |  | 
| 102 | 
            -
                         | 
| 103 | 
            -
                        [Splatter Image](https://github.com/szymanowiczs/splatter-image) (CVPR 2024) is a fast, super cheap to train method for object 3D reconstruction from a single image. 
         | 
| 104 | 
             
                        The model used in the demo was trained on **Objaverse-LVIS on 2 A6000 GPUs for 3.5 days**.
         | 
| 105 | 
            -
                         | 
| 106 | 
            -
                        Upload an image of an object to see how the Splatter Image does.
         | 
| 107 | 
            -
                        
         | 
| 108 | 
            -
                        **Comments:**
         | 
| 109 | 
            -
                        1. The first example you upload should take about 4.5 seconds (with preprocessing, saving and overhead), the following take about 1.5s.
         | 
| 110 | 
            -
                        2. The model does not work well on photos of humans.
         | 
| 111 | 
            -
                        3. The 3D viewer shows a .ply mesh extracted from a mix of 3D Gaussians. Artefacts might show - see video for more faithful results.
         | 
| 112 | 
            -
                        4. Best results are achieved on the datasets described in the [repository](https://github.com/szymanowiczs/splatter-image) using that code. This demo is experimental.
         | 
| 113 | 
            -
                        5. Our model might not be better than some state-of-the-art methods, but it is of comparable quality and is **much** cheaper to train and run.
         | 
| 114 | 
             
                        """
         | 
| 115 | 
             
                        )
         | 
| 116 | 
             
                    with gr.Row(variant="panel"):
         | 
| @@ -131,6 +132,26 @@ def main(): | |
| 131 | 
             
                                    )
         | 
| 132 | 
             
                            with gr.Row():
         | 
| 133 | 
             
                                submit = gr.Button("Generate", elem_id="generate", variant="primary")
         | 
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
| 134 | 
             
                        with gr.Column():
         | 
| 135 | 
             
                            with gr.Row():
         | 
| 136 | 
             
                                with gr.Tab("Reconstruction"):
         | 
| @@ -140,6 +161,31 @@ def main(): | |
| 140 | 
             
                                        interactive=False
         | 
| 141 | 
             
                                    )
         | 
| 142 |  | 
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
| 143 | 
             
                    submit.click(fn=check_input_image, inputs=[input_image]).success(
         | 
| 144 | 
             
                        fn=preprocess,
         | 
| 145 | 
             
                        inputs=[input_image, preprocess_background],
         | 
|  | |
| 95 |  | 
| 96 | 
             
                    return ply_out_path
         | 
| 97 |  | 
| 98 | 
            +
                css = """
         | 
| 99 | 
            +
                    h1 {
         | 
| 100 | 
            +
                        text-align: center;
         | 
| 101 | 
            +
                        display:block;
         | 
| 102 | 
            +
                    }
         | 
| 103 | 
            +
                    """
         | 
| 104 | 
            +
             | 
| 105 | 
            +
                with gr.Blocks(css=css) as demo:
         | 
| 106 | 
             
                    gr.Markdown(
         | 
| 107 | 
             
                        """
         | 
| 108 | 
            +
                        # Splatter Image
         | 
| 109 |  | 
| 110 | 
            +
                        **Splatter Image (CVPR 2024)** [[code](https://github.com/szymanowiczs/splatter-image), [project page](https://szymanowiczs.github.io/splatter-image)] is a fast, super cheap-to-train method for object 3D reconstruction from a single image. 
         | 
|  | |
| 111 | 
             
                        The model used in the demo was trained on **Objaverse-LVIS on 2 A6000 GPUs for 3.5 days**.
         | 
| 112 | 
            +
                        Locally, on an NVIDIA V100 GPU, reconstruction (forward pass of the network) can be done at 38FPS and rendering (with Gaussian Splatting) at 588FPS.
         | 
| 113 | 
            +
                        Upload an image of an object or click on one of the provided examples to see how the Splatter Image does.
         | 
| 114 | 
            +
                        For best results clone the [main repository](https://github.com/szymanowiczs/splatter-image) and run the demo locally.
         | 
|  | |
|  | |
|  | |
|  | |
|  | |
|  | |
| 115 | 
             
                        """
         | 
| 116 | 
             
                        )
         | 
| 117 | 
             
                    with gr.Row(variant="panel"):
         | 
|  | |
| 132 | 
             
                                    )
         | 
| 133 | 
             
                            with gr.Row():
         | 
| 134 | 
             
                                submit = gr.Button("Generate", elem_id="generate", variant="primary")
         | 
| 135 | 
            +
             | 
| 136 | 
            +
                            with gr.Row(variant="panel"): 
         | 
| 137 | 
            +
                                gr.Examples(
         | 
| 138 | 
            +
                                    examples=[
         | 
| 139 | 
            +
                                        './demo_examples/01_bigmac.png',
         | 
| 140 | 
            +
                                        './demo_examples/02_hydrant.jpg',
         | 
| 141 | 
            +
                                        './demo_examples/03_spyro.png',
         | 
| 142 | 
            +
                                        './demo_examples/04_lysol.png',
         | 
| 143 | 
            +
                                        './demo_examples/05_pinapple_bottle.png',
         | 
| 144 | 
            +
                                        './demo_examples/06_unsplash_broccoli.png',
         | 
| 145 | 
            +
                                        './demo_examples/07_objaverse_backpack.png',
         | 
| 146 | 
            +
                                        './demo_examples/08_unsplash_chocolatecake.png',
         | 
| 147 | 
            +
                                        './demo_examples/09_realfusion_cherry.png',
         | 
| 148 | 
            +
                                        './demo_examples/10_triposr_teapot.png'
         | 
| 149 | 
            +
                                    ],
         | 
| 150 | 
            +
                                    inputs=[input_image],
         | 
| 151 | 
            +
                                    cache_examples=False,
         | 
| 152 | 
            +
                                    label="Examples",
         | 
| 153 | 
            +
                                    examples_per_page=20,
         | 
| 154 | 
            +
                                )
         | 
| 155 | 
             
                        with gr.Column():
         | 
| 156 | 
             
                            with gr.Row():
         | 
| 157 | 
             
                                with gr.Tab("Reconstruction"):
         | 
|  | |
| 161 | 
             
                                        interactive=False
         | 
| 162 | 
             
                                    )
         | 
| 163 |  | 
| 164 | 
            +
                    gr.Markdown(
         | 
| 165 | 
            +
                    """
         | 
| 166 | 
            +
                        ## Comments:
         | 
| 167 | 
            +
                        1. If you run the demo online, the first example you upload should take about 4.5 seconds (with preprocessing, saving and overhead), the following take about 1.5s.
         | 
| 168 | 
            +
                        2. The 3D viewer shows a .ply mesh extracted from a mix of 3D Gaussians. This is only an approximations and artefacts might show.
         | 
| 169 | 
            +
                        3. Known limitations include:
         | 
| 170 | 
            +
                        - sphere-like artefacts on the object and white halo around it: this is due to how the .ply mesh is extracted and limitations of the Gradio viewer
         | 
| 171 | 
            +
                        - see-through parts of objects, especially on the back: this is due to the model performing less well on more complicated shapes
         | 
| 172 | 
            +
                        - back of objects are blurry: this is a model limiation due to it being deterministic
         | 
| 173 | 
            +
                        4. Our model is of comparable quality to state-of-the-art methods, and is **much** cheaper to train and run.
         | 
| 174 | 
            +
             | 
| 175 | 
            +
                        ## How does it work?
         | 
| 176 | 
            +
             | 
| 177 | 
            +
                        Splatter Image formulates 3D reconstruction as an image-to-image translation task. It maps the input image to another image, 
         | 
| 178 | 
            +
                        in which every pixel represents one 3D Gaussian and the channels of the output represent parameters of these Gaussians, including their shapes, colours and locations.
         | 
| 179 | 
            +
                        The resulting image thus represents a set of Gaussians (almost like a point cloud) which reconstruct the shape and colour of the object.
         | 
| 180 | 
            +
                        The method is very cheap: the reconstruction amounts to a single forward pass of a neural network with only 2D operators (2D convolutions and attention).
         | 
| 181 | 
            +
                        The rendering is also very fast, due to using Gaussian Splatting.
         | 
| 182 | 
            +
                        Combined, this results in very cheap training and high-quality results.
         | 
| 183 | 
            +
                        For more results see the [project page](https://szymanowiczs.github.io/splatter-image) and the [CVPR article](https://arxiv.org/abs/2312.13150).
         | 
| 184 | 
            +
                        """
         | 
| 185 | 
            +
                    )
         | 
| 186 | 
            +
             | 
| 187 | 
            +
             | 
| 188 | 
            +
             | 
| 189 | 
             
                    submit.click(fn=check_input_image, inputs=[input_image]).success(
         | 
| 190 | 
             
                        fn=preprocess,
         | 
| 191 | 
             
                        inputs=[input_image, preprocess_background],
         | 
    	
        demo_examples/01_bigmac.png
    ADDED
    
    |   | 
    	
        demo_examples/02_hydrant.jpg
    ADDED
    
    |   | 
    	
        demo_examples/03_spyro.png
    ADDED
    
    |   | 
    	
        demo_examples/04_lysol.png
    ADDED
    
    |   | 
    	
        demo_examples/05_pinapple_bottle.png
    ADDED
    
    |   | 
    	
        demo_examples/06_unsplash_broccoli.png
    ADDED
    
    |   | 
    	
        demo_examples/07_objaverse_backpack.png
    ADDED
    
    |   | 
    	
        demo_examples/08_unsplash_chocolatecake.png
    ADDED
    
    |   | 
    	
        demo_examples/09_realfusion_cherry.png
    ADDED
    
    |   | 
    	
        demo_examples/10_triposr_teapot.png
    ADDED
    
    |   |