Wonderful, could you please share something about SFT?
could you please share something about SFT? Like how to design the dataset, use which finetuning frame.
This model was trained with a low learning rate for 4 to 4.5 hours on 2× A100 SXM GPUs. It is an experimental successor to https://huggingface.co/prithivMLmods/Viper-Coder-v1.1
Regarding dataset preparation, it consists of modular combustion data entries from OpenR1's Code-Cot, Codeforces, Codeforces-Python, Code-Dataset-500K, CodeParrot, Python Core GitHub, Math-Code, and more.
For details on SFT (Supervised Fine-Tuning) and training, you can refer to the following Colab notebook, which I improved from scratch:
https://huggingface.co/datasets/prithivMLmods/Finetune-Colab-Nb/blob/main/SmolLM-FT/SmolLM-FT-360M.ipynb
This model was trained with a low learning rate for 4 to 4.5 hours on 2× A100 SXM GPUs. It is an experimental successor to https://huggingface.co/prithivMLmods/Viper-Coder-v1.1
Regarding dataset preparation, it consists of modular combustion data entries from OpenR1's Code-Cot, Codeforces, Codeforces-Python, Code-Dataset-500K, CodeParrot, Python Core GitHub, Math-Code, and more.
For details on SFT (Supervised Fine-Tuning) and training, you can refer to the following Colab notebook, which I improved from scratch:
https://huggingface.co/datasets/prithivMLmods/Finetune-Colab-Nb/blob/main/SmolLM-FT/SmolLM-FT-360M.ipynb
So nice for your reply! Actually, recently I'm faced with combining private code module(which is so hard to put into prompt) with code LLM（like generate " import a from my_module"）, so looking forward for any suggestions. Do you have any suggestions, with training or rag ,etc.
For that, you can follow different strategies in custom @Ken0102030405
You can start by indexing the module’s functions and methods, allowing the LLM to retrieve relevant parts dynamically. Another approach is using system prompts with detailed descriptions of the module. You can also store private module documentation or example code in a vector database and retrieve relevant snippets during generation.
Begin with the easiest method and move on based on your priority.