webUI/natural_language_guided_4/README.py

import gradio as gr

readme_content = """## Stable Diffusion for Sound Generation

This project applies stable diffusion[1] to sound generation. Inspired by the work of AUTOMATIC1111, 2022[2], we have implemented a preliminary version of text2sound, sound2sound, inpaint, as well as an additional interpolation feature, all accessible through a web UI.

### Neural Network Training Data:
The neural network is trained using the filtered NSynth dataset[3], which is a large-scale and high-quality collection of annotated musical notes, comprising 305,979 musical notes. However, for this project, only samples with a pitch set to E3 were used, resulting in an actual training sample size of 4,096, making it a low-resource project.

The training took place on an NVIDIA Tesla T4 GPU and spanned approximately 10 hours.

### Natural Language Guidance:
Natural language guidance is derived from the multi-label annotations of the NSynth dataset. The labels included in the training are:

- **Instrument Families**: bass, brass, flute, guitar, keyboard, mallet, organ, reed, string, synth lead, vocal.

- **Instrument Sources**: acoustic, electronic, synthetic.

- **Note Qualities**: bright, dark, distortion, fast decay, long release, multiphonic, nonlinear env, percussive, reverb, tempo-synced.

### Usage Hints:

1. **Prompt Format**: It's recommended to use the format “label1, label2, label3“, e.g., ”organ, dark, long release“.

2. **Unique Sounds**: If you keep generating the same sound, try setting a different seed!

3. **Sample Indexing**: Drag the "Sample index slider" to view other samples within the generated batch.

4. **Running on CPU**: Be cautious with the settings for 'batchsize' and 'sample_steps' when running on CPU to avoid timeouts. Recommended settings are batchsize ≤ 4 and sample_steps = 15.

5. **Editing Sounds**: Generated audio can be downloaded and then re-uploaded for further editing at the sound2sound/inpaint sections.

6. **Guidance Scale**: A higher 'guidance_scale' intensifies the influence of natural language conditioning on the generation[4]. It's recommended to set it between 3 and 10.

7. **Noising Strength**: A smaller 'noising_strength' value makes the generated sound closer to the input sound.

References:

[1] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).

[2] AUTOMATIC1111. (2022). Stable Diffusion Web UI [Computer software]. Retrieved from https://github.com/AUTOMATIC1111/stable-diffusion-webui

[3] Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., & Norouzi, M. (2017). Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders.

[4] Ho, J., & Salimans, T. (2022). Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598.
"""

def get_readme_module():

    with gr.Tab("README"):
        # gr.Markdown("Use interpolation to generate a gradient sound sequence.")
        with gr.Column(scale=3):
            readme_textbox = gr.Textbox(label="readme", lines=40, value=readme_content, interactive=False)