import gradio as gr readme_content = """## Stable Diffusion for Sound Generation This project applies stable diffusion[1] to sound generation. Inspired by the work of AUTOMATIC1111, 2022[2], we have implemented a preliminary version of text2sound, sound2sound, inpaint, as well as an additional interpolation feature, all accessible through a web UI. ### Neural Network Training Data: The neural network is trained using the filtered NSynth dataset[3], which is a large-scale and high-quality collection of annotated musical notes, comprising 305,979 musical notes. However, for this project, only samples with a pitch set to E3 were used, resulting in an actual training sample size of 4,096, making it a low-resource project. The training took place on an NVIDIA Tesla T4 GPU and spanned approximately 10 hours. ### Natural Language Guidance: Natural language guidance is derived from the multi-label annotations of the NSynth dataset. The labels included in the training are: - **Instrument Families**: bass, brass, flute, guitar, keyboard, mallet, organ, reed, string, synth lead, vocal. - **Instrument Sources**: acoustic, electronic, synthetic. - **Note Qualities**: bright, dark, distortion, fast decay, long release, multiphonic, nonlinear env, percussive, reverb, tempo-synced. ### Usage Hints: 1. **Prompt Format**: It's recommended to use the format “label1, label2, label3“, e.g., ”organ, dark, long release“. 2. **Unique Sounds**: If you keep generating the same sound, try setting a different seed! 3. **Sample Indexing**: Drag the "Sample index slider" to view other samples within the generated batch. 4. **Running on CPU**: Be cautious with the settings for 'batchsize' and 'sample_steps' when running on CPU to avoid timeouts. Recommended settings are batchsize ≤ 4 and sample_steps = 15. 5. **Editing Sounds**: Generated audio can be downloaded and then re-uploaded for further editing at the sound2sound/inpaint sections. 6. **Guidance Scale**: A higher 'guidance_scale' intensifies the influence of natural language conditioning on the generation[4]. It's recommended to set it between 3 and 10. 7. **Noising Strength**: A smaller 'noising_strength' value makes the generated sound closer to the input sound. References: [1] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695). [2] AUTOMATIC1111. (2022). Stable Diffusion Web UI [Computer software]. Retrieved from https://github.com/AUTOMATIC1111/stable-diffusion-webui [3] Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., & Norouzi, M. (2017). Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. [4] Ho, J., & Salimans, T. (2022). Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. """ def get_readme_module(): with gr.Tab("README"): # gr.Markdown("Use interpolation to generate a gradient sound sequence.") with gr.Column(scale=3): readme_textbox = gr.Textbox(label="readme", lines=40, value=readme_content, interactive=False)