JT-Coder-8B-Instruct

JT-Coder is a series of high-performance and energy-efficient code large language models (LLMs) developed by the JiuTian team. Our core philosophy is: high-quality data is more important than massive amounts of data. Through our innovative data-centric framework, JT-Coder, while pre-trained using only 1.6T tokens, comprehensively outperforms multiple models of similar scale trained on approximately 4x the data, providing a more efficient and reproducible path for the development of code LLMs.

Figure 1: Performance of JT-Coder-8B-Instruct on code generation benchmarks.

Core Features

• 🚀 State-of-the-Art Performance: JT-Coder achieves or surpasses the performance of existing top open-source models at both 1.5B and 8B scales across multiple code generation and comprehension benchmarks, including EvalPlus, BigCodeBench, LiveCodeBench, and FullstackBench.

• 🧠 Extreme Data Efficiency: We completed pre-training using only 1.6T high-quality tokens. Compared to similar models that typically use 5-6T of data, our data efficiency is improved by 4x, demonstrating the immense value of our advanced data processing pipeline.

• 💡 Innovative Data-Centric Framework:

  • Pre-training Phase: We meticulously cleaned open-source code data, filtering out low-quality and sensitive information. Simultaneously, we recovered and enriched high-value data such as Jupyter Notebooks, and synthesized large-scale, context-rich Q&A data and programming guides.

  • Instruction Tuning Phase: We pioneered the "Instruction Evolution" technique. This technique reverse-engineers the model's various effective outputs for simple instructions, transforming implicit characteristics within the code (e.g., algorithm selection, error handling) into explicit, complex instruction constraints, thereby significantly enriching the diversity and complexity of instruction data.

Model List

We have released the following pre-trained base models and instruction-tuned models:

Model Name	Type	Size
JT-Coder-8B-Instruct(You are here!)	Instruct	8B
JT-Coder-8B-Base	Base	8B
JT-Coder-1.5B-Instruct	Instruct	1.5B
JT-Coder-1.5B-Base	Base	1.5B

Quick Start: Inference with Transformers

You can easily run our models using the standard transformers library.

1. Install Dependencies

pip install torch transformers accelerate

2. Inference Code Example

Below is an example Python script for inference using the JT-Coder-8B-Instruct model.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# --- 1. Configure Model Path and Device ---
# Model ID on Hugging Face Hub
model_path = "JT-LM/JT-Coder-8B-Instruct" 
# Automatically select device (GPU preferred)
device = "cuda" if torch.cuda.is_available() else "cpu"

# --- 2. Load Tokenizer and Model ---
# trust_remote_code=True is necessary as we use a custom model structure
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    use_flash_attention_2=True
).to(device)
model.eval()

# --- 3. Construct Dialogue Input ---
# Use a list of dictionaries containing roles and content to represent dialogue history
messages = [
    {"role": "user", "content": "Please write a Python function to calculate the nth Fibonacci number, including detailed comments."},
]

# --- 4. Format and Encode with apply_chat_template ---
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(device)

# --- 5. Perform Inference ---
# Set generation parameters
generation_params = {
    "max_new_tokens": 2048,
    "do_sample": True,
    "temperature": 0.7,
    "top_p": 0.85,
    "top_k": 20,
}

# Generate response
with torch.no_grad():
    outputs = model.generate(inputs, **generation_params)

# --- 6. Decode and Print Results ---
# Decode only the newly generated part, skipping the original prompt
response = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)

print("--- User Query ---")
print(messages[0]['content'])
print("\n--- Model Response ---")
print(response)

License

The source code for this project is licensed under the Apache 2.0 license. The distribution and use of model weights adhere to their respective licensing agreements.

Acknowledgement

Our work is built upon the shoulders of giants in the open-source community, and we wish to express our profound gratitude.

We extend our sincere thanks to the Qwen Team. Adopting the Qwen2.5 tokenizer provided a robust vocabulary foundation that was crucial for our model's powerful multilingual and coding abilities.

Furthermore, we are deeply indebted to the creators and maintainers of pivotal open-source code datasets, including The Stack v2, Code-Matrix, etc. and the instruction data from projects like Opencoder. Their monumental efforts in collecting, curating, and sharing these vast resources provided the essential raw material for our data-centric framework. This project would not have been possible without their foundational contributions.

We hold the spirit of open collaboration in the highest regard and are proud to contribute back to the community that has enabled our research.

Disclaimer

JT-Coder is a large language model. While it has undergone rigorous data filtering and training, it may still generate inaccurate, biased, or harmful content. Users are advised to carefully evaluate the model's output and are responsible for any consequences arising from its use.