Update webUI/natural_language_guided_4/README.py

webUI/natural_language_guided_4/README.py

@@ -1,53 +1,41 @@
-import gradio as gr
-
-readme_content = """## Stable Diffusion for Sound Generation
-
-This project applies stable diffusion[1] to sound generation. Inspired by the work of AUTOMATIC1111, 2022[2], we have implemented a preliminary version of text2sound, sound2sound, inpaint, as well as an additional interpolation feature, all accessible through a web UI.
-
-### Neural Network Training Data:
-The neural network is trained using the filtered NSynth dataset[3], which is a large-scale and high-quality collection of annotated musical notes, comprising 305,979 musical notes. However, for this project, only samples with a pitch set to E3 were used, resulting in an actual training sample size of 4,096, making it a low-resource project.
-
-The training took place on an NVIDIA Tesla T4 GPU and spanned approximately 10 hours.
-
-### Natural Language Guidance:
-Natural language guidance is derived from the multi-label annotations of the NSynth dataset. The labels included in the training are:
-
-- **Instrument Families**: bass, brass, flute, guitar, keyboard, mallet, organ, reed, string, synth lead, vocal.
-
-- **Instrument Sources**: acoustic, electronic, synthetic.
-
-- **Note Qualities**: bright, dark, distortion, fast decay, long release, multiphonic, nonlinear env, percussive, reverb, tempo-synced.
-
-### Usage Hints:
-
-1. **Prompt Format**: It's recommended to use the format “label1, label2, label3“, e.g., ”organ, dark, long release“.
-
-2. **Unique Sounds**: If you keep generating the same sound, try setting a different seed!
-
-3. **Sample Indexing**: Drag the "Sample index slider" to view other samples within the generated batch.
-
-4. **Running on CPU**: Be cautious with the settings for 'batchsize' and 'sample_steps' when running on CPU to avoid timeouts. Recommended settings are batchsize ≤ 4 and sample_steps = 15.
-
-5. **Editing Sounds**: Generated audio can be downloaded and then re-uploaded for further editing at the sound2sound/inpaint sections.
-
-6. **Guidance Scale**: A higher 'guidance_scale' intensifies the influence of natural language conditioning on the generation[4]. It's recommended to set it between 3 and 10.
-
-7. **Noising Strength**: A smaller 'noising_strength' value makes the generated sound closer to the input sound.
-
-References:
-
-[1] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).
-
-[2] AUTOMATIC1111. (2022). Stable Diffusion Web UI [Computer software]. Retrieved from https://github.com/AUTOMATIC1111/stable-diffusion-webui
-
-[3] Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., & Norouzi, M. (2017). Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders.
-
-[4] Ho, J., & Salimans, T. (2022). Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598.
-"""
-
-def get_readme_module():
-
-    with gr.Tab("README"):
-        # gr.Markdown("Use interpolation to generate a gradient sound sequence.")
-        with gr.Column(scale=3):
+import gradio as gr
+
+readme_content = """## Assisting Musicians with Generation of Musical Notes using a Text-Guided Diffusion Model
+
+
+### Training Data:
+The neural network is trained on the filtered NSynth dataset [3], which comes with the following labels:
+
+Instrument Families: bass, brass, flute, guitar, keyboard, mallet, organ, reed, string, synth lead, vocal.
+Instrument Sources: acoustic, electronic, synthetic.
+Note Qualities: bright, dark, distortion, fast decay, long release, multiphonic, nonlinear env, percussive, reverb, tempo-synced.
+
+YOU ARE NOT LIMITED TO THE ABOVE TERMS; THE MODEL CAN UNDERSTAND A WIDE RANGE OF VOCABULARY AND ACCEPTS NATURAL LANGUAGE INPUT!
+
+### Usage Hints:
+
+1. **Unique Sounds**: Start generating your unique sound in Text2Sound!
+
+2. **Sample Indexing**: Drag the "Sample index slider" to view other samples within the generated batch.
+
+3. **Editing Sounds**: Generated audio can be downloaded and re-uploaded for further editing in the Sound2Sound/Inpaint sections. YOU CAN ALSO UPLOAD OR RECORD AUDIO FROM OTHER SOURCES.
+
+4. **Arrangement** Once you have achieved a satisfactory timbre in the Text2Sound, Sound2Sound, or Inpaint module, you can name and save it in the bottom-right corner. Then, you can upload your MIDI file in the Arrangement module, assign the saved timbre to each track, and start playing!
+
+References:
+
+[1] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).
+
+[2] AUTOMATIC1111. (2022). Stable Diffusion Web UI [Computer software]. Retrieved from https://github.com/AUTOMATIC1111/stable-diffusion-webui
+
+[3] Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., & Norouzi, M. (2017). Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders.
+
+[4] Ho, J., & Salimans, T. (2022). Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598.
+"""
+
+def get_readme_module():
+
+    with gr.Tab("README"):
+        # gr.Markdown("Use interpolation to generate a gradient sound sequence.")
+        with gr.Column(scale=3):
             readme_textbox = gr.Textbox(label="readme", lines=40, value=readme_content, interactive=False)