|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
- ja |
|
|
tags: |
|
|
- causal-lm |
|
|
- llama |
|
|
- accounting |
|
|
- tiktoken |
|
|
- business |
|
|
- multilingual |
|
|
library_name: transformers |
|
|
model-index: |
|
|
- name: soroban_3.8B_instruct_fp32 |
|
|
results: [] |
|
|
--- |
|
|
|
|
|
# 📎 Fast Accounting LLM (soroban3.8Bv0.0) |
|
|
|
|
|
**soroban_3.8B_instruct_fp32** is a 3.8B parameter LLaMA-style language model trained from scratch using multilingual corpora in **English and Japanese**, with a strong emphasis on **accounting**, **financial data**, and **business language**. It uses a custom tokenizer built on top of `tiktoken` (`o200k_base`), supports special tokens for dialogue, and is optimized for instructional and analytical text generation. The model is a checkpoint version of pretrained soroban which is further FTed on instruction datasets. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
- **Developed by:** FastAccounting Japan |
|
|
- **Model type:** Causal Language Model (LLaMA-style architecture) |
|
|
- **Language(s):** English and Japanese |
|
|
- **License:** MIT |
|
|
- **Finetuned from model:** Trained from scratch |
|
|
- **Tokenizer:** Custom BPE tokenizer (based on tiktoken `o200k_base`) |
|
|
|
|
|
### Model Sources |
|
|
- **Repository:** https://huggingface.co/FastAccounting/soroban_3.8B_instruct_fp32 |
|
|
|
|
|
## Uses (Coming Soon) |
|
|
|
|
|
### Direct Use |
|
|
- Chat-style generation for accounting and business tasks |
|
|
- Financial Q&A and report summarization |
|
|
- Instructional document parsing (in Japanese & English) |
|
|
|
|
|
### Downstream Use |
|
|
- Fine-tuning for audit compliance or domain-specific accounting QA |
|
|
|
|
|
### Out-of-Scope Use |
|
|
- Unfiltered open-domain conversation |
|
|
- Real-time decision-making in regulated financial environments |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
- **Domain Bias:** Model has strong exposure to accounting-style language; it may underperform in open-domain or creative tasks. |
|
|
- **Language Balance:** While trained on both Japanese and English, performance may vary between them depending on prompt structure. |
|
|
|
|
|
### Recommendations |
|
|
- Fine-tune or prompt carefully for non-accounting use cases. |
|
|
- Always validate financial output before applying in business or legal settings. |
|
|
|
|
|
## How to Get Started with the Model |
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("FastAccounting/soroban_3.8B_instruct_fp32") |
|
|
model = AutoModelForCausalLM.from_pretrained("FastAccounting/soroban_3.8B_instruct_fp32") |
|
|
|
|
|
prompt = "<|begin_of_text|>Explain cash flow in Japanese." |
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
inputs.pop("token_type_ids", None) # 💥 remove unused key |
|
|
outputs = model.generate(**inputs, max_new_tokens=100) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Data |
|
|
- Internal multilingual accounting corpus (~160B tokens) |
|
|
- Custom curated Japanese financial documents |
|
|
- Publicly available English accounting datasets |
|
|
|
|
|
### Training Procedure |
|
|
- **Precision:** fp32 precision |
|
|
- **Context length:** 4096 tokens |
|
|
- **Optimizer:** AdamW with weight decay |
|
|
- **Learning schedule:** Cosine decay with warmup |
|
|
|
|
|
## Evaluation |
|
|
- Ongoing evaluation with domain-specific metrics (financial text perplexity, accuracy on Japanese bookkeeping test questions) |
|
|
|
|
|
## Environmental Impact |
|
|
- **Hardware Type:** 24x NVIDIA H100 GPUs |
|
|
- **Hours used:** ~4560 GPU hours |
|
|
- **Cloud Provider:** On-prem H100 cluster |
|
|
- **Carbon Emitted:** Estimated 1250kg CO2eq |
|
|
|
|
|
## Technical Specifications |
|
|
|
|
|
### Model Architecture and Objective |
|
|
- 3.8B parameter LLaMA-style Transformer |
|
|
- 32 layers, 16 attention heads, RoPE, SwiGLU, FlashAttention2 |
|
|
- Untied input/output embeddings |
|
|
|
|
|
### Compute Infrastructure |
|
|
- **Framework:** PyTorch with Hugging Face Transformers |
|
|
- **Libraries:** DeepSpeed, Megatron-DS integration |
|
|
|
|
|
## Citation |
|
|
``` |
|
|
@misc{fa_llm_2025, |
|
|
author = {FastAccounting LLM}, |
|
|
title = {soroban_3.8B: A Multilingual Accounting Language Model}, |
|
|
year = {2025}, |
|
|
url = {https://huggingface.co/FastAccounting/fa_llm_1B}, |
|
|
} |
|
|
``` |
|
|
|
|
|
## Contact |
|
|
**Lead Developer:** Keshav Singh & Fujitake Masato |
|
|
|
|
|
**Organization:** FastAccounting Japan |
|
|
|
|
|
**Hugging Face Profile:** https://huggingface.co/FastAccounting |
|
|
|
|
|
|
