README.md

metadata

license: apache-2.0
datasets:
  - TorpedoSoftware/the-luau-stack
  - TorpedoSoftware/roblox-info-dump
language:
  - en
  - fr
  - de
  - es
  - pt
  - it
base_model:
  - unsloth/Devstral-Small-2507-unsloth-bnb-4bit
tags:
  - roblox
  - luau
  - code
  - sft
  - transformers
  - trl
  - unsloth

Luau Devstral 24B Instruct v0.1

A Roblox Luau focused finetune of Devstral Small 2507.

Model Details

Model Description

Devstral Small 2507 is a powerful choice for local inference, achieving SOTA open source results at just 24B parameters. However, Roblox gamedev and Luau programming are generally not well represented in LLM training data. This model fine tunes Devstral on a corpus of permissively licensed Luau code and Roblox documentation, improving the model's Luau programming capabilities. Additionally, the jinja chat template contains a default system prompt that steers the model's Luau capabilities even further.

• Developed by: Zack Williams (boatbomber)
• Funded by: Torpedo Software LLC
• License: Apache 2.0
• Finetuned from model: unsloth/Devstral-Small-2507-unsloth-bnb-4bit

Model Sources

• Repository: https://huggingface.co/mistralai/Devstral-Small-2507
• Blog: https://mistral.ai/news/devstral-2507

Training Details

Training Data

1. https://huggingface.co/datasets/TorpedoSoftware/the-luau-stack
2. https://huggingface.co/datasets/TorpedoSoftware/roblox-info-dump

Preprocessing

Each datapoint from the training data was formatted as follows in order to provide the model with relevant context:

Repository: {repo_name}
Repository Description: {repo_description}

File Path: `{file_path}`
File Content:
```Lua
{file_content}
```\

Training Loss Curve

Imatrix Calibration

The imatrix for the GGUF quantizations was computed using 33.5MB of text containing a combination of wiki.train.raw and content from the-luau-stack & roblox-info-dump. This created an imatrix that is well suited to the specialized tasks this model is designed for while still maintaining broader intelligence as well. While we do provide several quantizations already, the imatrix.gguf is included in this repository should you want to create other quants yourself.

Environmental Impact

Carbon emissions estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: RTX 3090
• Hours used: 60
• Cloud Provider: My gaming PC
• Compute Region: Bay Area
• Carbon Emitted: 4.73 kg CO2eq (equivalent to 11.8 miles driven by an average ICE car)