About training template

#1
by fdsfasfafa - opened

I am using LLaMA-Factory(https://github.com/hiyouga/LLaMA-Factory) for LoRA SFT, and during training, it uses the Chat Template. Do you also use the Chat Template? Here is one example:

<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|fim_prefix|>// Path: robotIp/index.vue\n//         },\r\n//         handleTouchEnd() {\n// Path: robotIp/index.vue\n//                 }\r\n//             }\n// Path: robotIp/index.vue\n//                 this.left = 0;//不让贴边 所以设置30没设置0\r\n//                 this.handleIconY()\n// robotIp/index.vue\n       },\r\n       tipsBottom:{\r\n         type: Number,\r\n         default: 0\r\n       }\r\n\t},\r\n\tdata() {\r\n\t\treturn{\r\n            iconUrl: require('@/assets/home/huazhixing.png'),\r\n            left: 0,\r\n            top: 0,\r\n            startToMove: false,\r\n            isShow: true,\r\n            timer: null,\r\n            currentTop: null,\r\n            clientW: document.documentElement.clientWidth,//视口宽\r\n            clientH: document.documentElement.clientHeight,//视口高\r\n        }\r\n\t},\r\n\tcreated() {\r\n        this.left = (this.clientW - this.itemWidth - this.right)\r\n        this.top = (this.clientH - this.bottom)\r\n        setTimeout(() => {\r\n        this.iconUrl = require('@/assets/home/huazhixing2.png')\r\n      }, 3000)\r\n\t},\r\n\tmounted() {\r\n\t\tthis.bindScrollEvent()\r\n\t},\r\n  beforeDestroy() {\r\n        // 记得销毁一些全局的的事件\r\n        this.removeScrollEvent()\r\n    },\r\n    methods: {\r\n        goCreatePage() {\r\n          this.$emit('openHuayan')\r\n        },\r\n        handleTouchStart() {\r\n            this.startToMove = true\r\n            this.$refs.dragIcon.style.transition = \"none\"\r\n            this.$refs.dragIcon.style.animation = \"none\"\r\n            this.iconUrl = require('@/assets/home/huazhixing.png')\r\n        },\r\n        handleTouchMove(e) {\r\n            const clientX = e.targetTouches[0].clientX;//手指相对视口的x\r\n            const clientY = e.targetTouches[0].clientY;//手指相对视口的y\r\n            const isInScreen = clientX <= this.clientW && clientX >= 0 && clientY <= this.clientH && clientY >=0\r\n            if(this.startToMove && e.targetTouches.length === 1) {\r\n                if(isInScreen) {\r\n                    this.left = clientX - this.itemWidth/2\r\n                    this.top = clientY - this.itemHeight/2\r\n                }\r\n                this.handleIconY()\r\n                this.isShow = true\r\n                this.timer && clearTimeout(this.timer)\r\n                <|fim_suffix|>\r\n            }\r\n        },\r\n        handleTouchEnd() {\r\n            if(this.left < (this.clientW / 2)) {\r\n                this.left = 0;//不让贴边 所以设置30没设置0\r\n                this.handleIconY()\r\n            } else {\r\n                this.left = this.clientW - this.itemWidth - 0;//不让贴边 所以减30\r\n                this.handleIconY()\r\n            }\r\n            this.$refs.dragIcon.style.transition = \"all .3s\"\r\n        },\r\n        handleIconY() {\r\n            if (this.top < 0) {\r\n                this.top = 5;//不上帖上边所以设置为30 没设置0\r\n            } else if(this.top + this.itemHeight > this.clientH) {\r\n                this.top = this.clientH - this.itemHeight - 5;//不让帖下边所以减30\r\n            }\r\n        },\r\n        bindScrollEvent() {\r\n            window.addEventListener('scroll', this.handleScrollStart)\r\n        },\r\n        removeScrollEvent() {\r\n            window.removeEventListener('scroll',this.handleScrollStart)\r\n        },\r\n        handleScrollStart() {\r\n            this.isShow = false\r\n            this.timer && clearTimeout(this.timer)\r\n            this.timer = setTimeout(() => {\r\n                this.handleScrollEnd()\r\n            },300)\r\n            this.currentTop = document.documentElement.scrollTop || document.body.scrollTop\r\n        },\r\n        handleScrollEnd() {\r<|fim_middle|><|im_end|>\n<|im_start|>assistant\n

model is Qwen-2.5-Coder-1.5b-Instruct

No i am using <|file_name|>{{{filename}}}<|fim_prefix|>{{{prefix}}}<|fim_suffix|>{{{suffix}}}<|fim_middle|>
no system message or user and assistant stuff this is also how Qwen-2.5-Coder handle codefim so its recommended to follow it (without <|file_name|> that i added it myself)
also i think training base model will be way better since base models are better in codefim than instruct

OK, thanks. How is the result after you fine-tune it? Can you provide the code of your fine-tuning?

fdsfasfafa changed discussion status to closed

Sign up or log in to comment