--- library_name: peft license: gemma base_model: google/gemma-3-1b-it tags: - llama-factory - ia3 - generated_from_trainer model-index: - name: train_rte_1744902654 results: [] --- # train_rte_1744902654 This model is a fine-tuned version of [google/gemma-3-1b-it](https://huggingface.co/google/gemma-3-1b-it) on the rte dataset. It achieves the following results on the evaluation set: - Loss: 0.0951 - Num Input Tokens Seen: 102120968 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-05 - train_batch_size: 4 - eval_batch_size: 4 - seed: 123 - gradient_accumulation_steps: 4 - total_train_batch_size: 16 - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - training_steps: 40000 ### Training results | Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen | |:-------------:|:--------:|:-----:|:---------------:|:-----------------:| | 0.1898 | 1.4207 | 200 | 0.2012 | 514544 | | 0.1042 | 2.8414 | 400 | 0.1239 | 1025888 | | 0.1043 | 4.2567 | 600 | 0.1146 | 1532968 | | 0.1134 | 5.6774 | 800 | 0.1085 | 2047232 | | 0.096 | 7.0927 | 1000 | 0.1066 | 2553328 | | 0.0885 | 8.5134 | 1200 | 0.1047 | 3064584 | | 0.107 | 9.9340 | 1400 | 0.1024 | 3578720 | | 0.1068 | 11.3494 | 1600 | 0.1018 | 4086872 | | 0.0856 | 12.7701 | 1800 | 0.1014 | 4597072 | | 0.1087 | 14.1854 | 2000 | 0.1011 | 5107408 | | 0.0908 | 15.6061 | 2200 | 0.0988 | 5618600 | | 0.0855 | 17.0214 | 2400 | 0.0978 | 6127088 | | 0.0963 | 18.4421 | 2600 | 0.0993 | 6640448 | | 0.0745 | 19.8627 | 2800 | 0.0979 | 7149392 | | 0.0842 | 21.2781 | 3000 | 0.0974 | 7655512 | | 0.0784 | 22.6988 | 3200 | 0.0975 | 8170864 | | 0.1001 | 24.1141 | 3400 | 0.0971 | 8679240 | | 0.077 | 25.5348 | 3600 | 0.0968 | 9186256 | | 0.0726 | 26.9554 | 3800 | 0.0965 | 9701496 | | 0.0733 | 28.3708 | 4000 | 0.0968 | 10208536 | | 0.0769 | 29.7914 | 4200 | 0.0951 | 10718920 | | 0.076 | 31.2068 | 4400 | 0.0967 | 11238024 | | 0.0867 | 32.6275 | 4600 | 0.0958 | 11745784 | | 0.0733 | 34.0428 | 4800 | 0.0957 | 12256384 | | 0.0601 | 35.4635 | 5000 | 0.0967 | 12764464 | | 0.0774 | 36.8841 | 5200 | 0.0965 | 13274464 | | 0.0876 | 38.2995 | 5400 | 0.0961 | 13783792 | | 0.0826 | 39.7201 | 5600 | 0.0959 | 14300048 | | 0.0713 | 41.1355 | 5800 | 0.0961 | 14802928 | | 0.0755 | 42.5561 | 6000 | 0.0956 | 15310736 | | 0.0748 | 43.9768 | 6200 | 0.0967 | 15826504 | | 0.0895 | 45.3922 | 6400 | 0.0959 | 16328920 | | 0.0761 | 46.8128 | 6600 | 0.0969 | 16846456 | | 0.0795 | 48.2282 | 6800 | 0.0969 | 17353504 | | 0.0678 | 49.6488 | 7000 | 0.0970 | 17866072 | | 0.0912 | 51.0642 | 7200 | 0.0972 | 18373808 | | 0.0715 | 52.4848 | 7400 | 0.0985 | 18884136 | | 0.0616 | 53.9055 | 7600 | 0.0978 | 19402904 | | 0.0724 | 55.3209 | 7800 | 0.0989 | 19913320 | | 0.0647 | 56.7415 | 8000 | 0.0987 | 20425320 | | 0.0797 | 58.1569 | 8200 | 0.0992 | 20932224 | | 0.0805 | 59.5775 | 8400 | 0.0992 | 21443632 | | 0.0844 | 60.9982 | 8600 | 0.0992 | 21958520 | | 0.0602 | 62.4135 | 8800 | 0.1000 | 22465568 | | 0.0566 | 63.8342 | 9000 | 0.1005 | 22978872 | | 0.0642 | 65.2496 | 9200 | 0.0990 | 23489416 | | 0.0583 | 66.6702 | 9400 | 0.1013 | 23998568 | | 0.0731 | 68.0856 | 9600 | 0.1013 | 24508040 | | 0.056 | 69.5062 | 9800 | 0.1012 | 25021816 | | 0.0578 | 70.9269 | 10000 | 0.1007 | 25536008 | | 0.0516 | 72.3422 | 10200 | 0.1020 | 26049360 | | 0.0648 | 73.7629 | 10400 | 0.1022 | 26563296 | | 0.0405 | 75.1783 | 10600 | 0.1033 | 27069112 | | 0.0563 | 76.5989 | 10800 | 0.1046 | 27583320 | | 0.0625 | 78.0143 | 11000 | 0.1013 | 28092696 | | 0.0595 | 79.4349 | 11200 | 0.1031 | 28604784 | | 0.0501 | 80.8556 | 11400 | 0.1043 | 29118496 | | 0.0726 | 82.2709 | 11600 | 0.1042 | 29628968 | | 0.0566 | 83.6916 | 11800 | 0.1029 | 30142648 | | 0.0538 | 85.1070 | 12000 | 0.1017 | 30650416 | | 0.0512 | 86.5276 | 12200 | 0.1067 | 31164200 | | 0.0589 | 87.9483 | 12400 | 0.1036 | 31680176 | | 0.0526 | 89.3636 | 12600 | 0.1056 | 32191784 | | 0.0449 | 90.7843 | 12800 | 0.1075 | 32704016 | | 0.0312 | 92.1996 | 13000 | 0.1090 | 33211832 | | 0.065 | 93.6203 | 13200 | 0.1094 | 33725272 | | 0.0463 | 95.0357 | 13400 | 0.1075 | 34239520 | | 0.063 | 96.4563 | 13600 | 0.1058 | 34749688 | | 0.0407 | 97.8770 | 13800 | 0.1091 | 35255888 | | 0.0497 | 99.2923 | 14000 | 0.1076 | 35764264 | | 0.037 | 100.7130 | 14200 | 0.1089 | 36272560 | | 0.0397 | 102.1283 | 14400 | 0.1124 | 36779392 | | 0.0427 | 103.5490 | 14600 | 0.1117 | 37288688 | | 0.0438 | 104.9697 | 14800 | 0.1146 | 37798808 | | 0.0433 | 106.3850 | 15000 | 0.1127 | 38306176 | | 0.0326 | 107.8057 | 15200 | 0.1142 | 38818056 | | 0.0338 | 109.2210 | 15400 | 0.1158 | 39327152 | | 0.0477 | 110.6417 | 15600 | 0.1136 | 39834472 | | 0.0473 | 112.0570 | 15800 | 0.1119 | 40347168 | | 0.0323 | 113.4777 | 16000 | 0.1155 | 40861968 | | 0.0418 | 114.8984 | 16200 | 0.1126 | 41373408 | | 0.0505 | 116.3137 | 16400 | 0.1181 | 41884568 | | 0.0467 | 117.7344 | 16600 | 0.1191 | 42393232 | | 0.0247 | 119.1497 | 16800 | 0.1172 | 42901968 | | 0.0405 | 120.5704 | 17000 | 0.1177 | 43418880 | | 0.0462 | 121.9911 | 17200 | 0.1214 | 43930128 | | 0.0461 | 123.4064 | 17400 | 0.1236 | 44439224 | | 0.0452 | 124.8271 | 17600 | 0.1203 | 44949200 | | 0.034 | 126.2424 | 17800 | 0.1279 | 45456488 | | 0.0374 | 127.6631 | 18000 | 0.1277 | 45966968 | | 0.0254 | 129.0784 | 18200 | 0.1254 | 46478752 | | 0.0537 | 130.4991 | 18400 | 0.1239 | 46990144 | | 0.0281 | 131.9198 | 18600 | 0.1228 | 47496384 | | 0.0332 | 133.3351 | 18800 | 0.1199 | 48002064 | | 0.033 | 134.7558 | 19000 | 0.1281 | 48514080 | | 0.0484 | 136.1711 | 19200 | 0.1304 | 49020824 | | 0.04 | 137.5918 | 19400 | 0.1247 | 49536472 | | 0.0251 | 139.0071 | 19600 | 0.1272 | 50047616 | | 0.0353 | 140.4278 | 19800 | 0.1256 | 50560928 | | 0.0299 | 141.8485 | 20000 | 0.1300 | 51077560 | | 0.0327 | 143.2638 | 20200 | 0.1311 | 51589728 | | 0.0277 | 144.6845 | 20400 | 0.1289 | 52091960 | | 0.0507 | 146.0998 | 20600 | 0.1286 | 52599968 | | 0.0341 | 147.5205 | 20800 | 0.1364 | 53105080 | | 0.0365 | 148.9412 | 21000 | 0.1392 | 53615088 | | 0.0213 | 150.3565 | 21200 | 0.1382 | 54126520 | | 0.0281 | 151.7772 | 21400 | 0.1352 | 54637568 | | 0.0163 | 153.1925 | 21600 | 0.1337 | 55145992 | | 0.0289 | 154.6132 | 21800 | 0.1359 | 55658768 | | 0.0309 | 156.0285 | 22000 | 0.1368 | 56165752 | | 0.0183 | 157.4492 | 22200 | 0.1384 | 56680344 | | 0.032 | 158.8699 | 22400 | 0.1418 | 57190184 | | 0.0219 | 160.2852 | 22600 | 0.1437 | 57701264 | | 0.0347 | 161.7059 | 22800 | 0.1421 | 58206800 | | 0.0175 | 163.1212 | 23000 | 0.1448 | 58714616 | | 0.0144 | 164.5419 | 23200 | 0.1444 | 59223288 | | 0.0166 | 165.9626 | 23400 | 0.1487 | 59731816 | | 0.0222 | 167.3779 | 23600 | 0.1439 | 60238984 | | 0.021 | 168.7986 | 23800 | 0.1526 | 60751824 | | 0.0262 | 170.2139 | 24000 | 0.1482 | 61264200 | | 0.0428 | 171.6346 | 24200 | 0.1494 | 61774128 | | 0.0264 | 173.0499 | 24400 | 0.1496 | 62287528 | | 0.0249 | 174.4706 | 24600 | 0.1520 | 62802568 | | 0.0155 | 175.8913 | 24800 | 0.1516 | 63313384 | | 0.0245 | 177.3066 | 25000 | 0.1509 | 63824360 | | 0.0282 | 178.7273 | 25200 | 0.1492 | 64334200 | | 0.0204 | 180.1426 | 25400 | 0.1512 | 64843720 | | 0.0286 | 181.5633 | 25600 | 0.1534 | 65355856 | | 0.0276 | 182.9840 | 25800 | 0.1583 | 65867080 | | 0.022 | 184.3993 | 26000 | 0.1538 | 66376432 | | 0.0233 | 185.8200 | 26200 | 0.1524 | 66891552 | | 0.0271 | 187.2353 | 26400 | 0.1514 | 67395432 | | 0.0234 | 188.6560 | 26600 | 0.1570 | 67911272 | | 0.0213 | 190.0713 | 26800 | 0.1551 | 68421624 | | 0.0429 | 191.4920 | 27000 | 0.1546 | 68928904 | | 0.0259 | 192.9127 | 27200 | 0.1597 | 69438872 | | 0.009 | 194.3280 | 27400 | 0.1635 | 69956344 | | 0.021 | 195.7487 | 27600 | 0.1613 | 70469232 | | 0.016 | 197.1640 | 27800 | 0.1608 | 70980552 | | 0.0275 | 198.5847 | 28000 | 0.1575 | 71494368 | | 0.0185 | 200.0 | 28200 | 0.1652 | 71999768 | | 0.0252 | 201.4207 | 28400 | 0.1622 | 72508480 | | 0.0222 | 202.8414 | 28600 | 0.1618 | 73019472 | | 0.0195 | 204.2567 | 28800 | 0.1617 | 73527632 | | 0.0186 | 205.6774 | 29000 | 0.1595 | 74041024 | | 0.0132 | 207.0927 | 29200 | 0.1639 | 74544840 | | 0.0153 | 208.5134 | 29400 | 0.1630 | 75056168 | | 0.0188 | 209.9340 | 29600 | 0.1643 | 75568152 | | 0.0178 | 211.3494 | 29800 | 0.1688 | 76079080 | | 0.0293 | 212.7701 | 30000 | 0.1658 | 76588464 | | 0.0235 | 214.1854 | 30200 | 0.1651 | 77091184 | | 0.0291 | 215.6061 | 30400 | 0.1702 | 77604592 | | 0.0127 | 217.0214 | 30600 | 0.1696 | 78117872 | | 0.0131 | 218.4421 | 30800 | 0.1662 | 78636456 | | 0.0127 | 219.8627 | 31000 | 0.1730 | 79145648 | | 0.0153 | 221.2781 | 31200 | 0.1726 | 79656680 | | 0.0296 | 222.6988 | 31400 | 0.1697 | 80170976 | | 0.0241 | 224.1141 | 31600 | 0.1696 | 80680568 | | 0.0186 | 225.5348 | 31800 | 0.1713 | 81190016 | | 0.0393 | 226.9554 | 32000 | 0.1698 | 81700080 | | 0.0215 | 228.3708 | 32200 | 0.1692 | 82211496 | | 0.0234 | 229.7914 | 32400 | 0.1699 | 82723360 | | 0.0241 | 231.2068 | 32600 | 0.1744 | 83233760 | | 0.0191 | 232.6275 | 32800 | 0.1713 | 83744152 | | 0.0164 | 234.0428 | 33000 | 0.1679 | 84252696 | | 0.0119 | 235.4635 | 33200 | 0.1694 | 84766920 | | 0.0143 | 236.8841 | 33400 | 0.1689 | 85270792 | | 0.0074 | 238.2995 | 33600 | 0.1702 | 85785816 | | 0.0126 | 239.7201 | 33800 | 0.1704 | 86297192 | | 0.0208 | 241.1355 | 34000 | 0.1740 | 86800112 | | 0.0254 | 242.5561 | 34200 | 0.1717 | 87308856 | | 0.0294 | 243.9768 | 34400 | 0.1661 | 87823904 | | 0.0242 | 245.3922 | 34600 | 0.1748 | 88328184 | | 0.0245 | 246.8128 | 34800 | 0.1724 | 88842368 | | 0.0227 | 248.2282 | 35000 | 0.1720 | 89352024 | | 0.0348 | 249.6488 | 35200 | 0.1734 | 89859880 | | 0.0206 | 251.0642 | 35400 | 0.1762 | 90371864 | | 0.0257 | 252.4848 | 35600 | 0.1751 | 90889904 | | 0.029 | 253.9055 | 35800 | 0.1770 | 91398024 | | 0.0244 | 255.3209 | 36000 | 0.1719 | 91909952 | | 0.0334 | 256.7415 | 36200 | 0.1736 | 92415808 | | 0.0205 | 258.1569 | 36400 | 0.1721 | 92924744 | | 0.0277 | 259.5775 | 36600 | 0.1740 | 93438136 | | 0.0268 | 260.9982 | 36800 | 0.1750 | 93945688 | | 0.026 | 262.4135 | 37000 | 0.1775 | 94456488 | | 0.0168 | 263.8342 | 37200 | 0.1777 | 94967816 | | 0.0111 | 265.2496 | 37400 | 0.1781 | 95480152 | | 0.0106 | 266.6702 | 37600 | 0.1813 | 95992952 | | 0.021 | 268.0856 | 37800 | 0.1764 | 96503784 | | 0.0197 | 269.5062 | 38000 | 0.1771 | 97017704 | | 0.0083 | 270.9269 | 38200 | 0.1765 | 97525848 | | 0.0072 | 272.3422 | 38400 | 0.1733 | 98034592 | | 0.0102 | 273.7629 | 38600 | 0.1757 | 98546400 | | 0.0293 | 275.1783 | 38800 | 0.1761 | 99055216 | | 0.0308 | 276.5989 | 39000 | 0.1703 | 99570208 | | 0.0165 | 278.0143 | 39200 | 0.1755 | 100077240 | | 0.0316 | 279.4349 | 39400 | 0.1736 | 100585296 | | 0.0158 | 280.8556 | 39600 | 0.1736 | 101096120 | | 0.017 | 282.2709 | 39800 | 0.1736 | 101609904 | | 0.016 | 283.6916 | 40000 | 0.1736 | 102120968 | ### Framework versions - PEFT 0.15.1 - Transformers 4.51.3 - Pytorch 2.6.0+cu124 - Datasets 3.5.0 - Tokenizers 0.21.1