jinv2/opt125m-wikitext2-finetuned

OPT-125m Fine-tuned on WikiText-2 for Language Modeling

This repository contains the weights for the facebook/opt-125m model after fine-tuning on the wikitext-2-raw-v1 configuration of the WikiText dataset using a standard Causal Language Modeling (CLM) objective.

Model Description

This model is a version of the OPT (Open Pre-trained Transformer) architecture with 125 million parameters. It was fine-tuned solely on the WikiText-2 dataset. The primary goal of this fine-tuning process, as executed by the training script, was to adapt the model to the language patterns, vocabulary, and style present in the WikiText-2 corpus by learning to predict the next token in a sequence.

Note on Project Context: While this fine-tuning experiment was conducted as part of a larger initiative by "天算AI" / "SafeSky AI" aiming to develop AI Safety models, this specific model does not possess specialized AI safety capabilities. It was trained on a general-purpose dataset (WikiText-2) using a standard language modeling task, and should be considered a foundational experiment in the model fine-tuning process rather than a dedicated AI Safety model.

Intended Uses & Limitations

Intended Use:

• Text Generation: Generating English text that statistically resembles the style and content of the WikiText-2 dataset (e.g., encyclopedic, factual-style prose). It can be used for text continuation or completion based on a given prompt.
• Language Modeling Research: Serving as a benchmark or an example for standard causal language model fine-tuning on the WikiText-2 dataset.
• Educational Purposes: Understanding the fine-tuning process and the behavior of smaller OPT models.

Limitations:

• Lack of Safety Features: This model has not been trained on safety-specific data or with safety-oriented objectives. It cannot reliably identify or avoid generating harmful, biased, unethical, or factually incorrect ("hallucinatory") content beyond what might be implicitly learned (or avoided) from the relatively neutral WikiText-2 data. It is not robust against adversarial prompts.
• General Capabilities: As a 125m parameter model fine-tuned on a specific corpus, its general knowledge, reasoning abilities, and performance on tasks significantly different from WikiText-2's domain might be limited.
• Potential Biases: It may inherit biases present in the original OPT-125m model or the WikiText-2 dataset.

How to Use

You can use this model with the Hugging Face transformers library for text generation:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model_id = "jinv2/opt125m-wikitext2-finetuned"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

# Example prompt
prompt = "Artificial intelligence is transforming the world by"

# Tokenize prompt
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate text
output_sequences = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_length=60,
    num_return_sequences=1,
    do_sample=True,
    temperature=0.7,
    top_k=50
)

# Decode and print
generated_text = tokenizer.decode(output_sequences[0], skip_special_tokens=True)
print(generated_text)

Training Procedure

This model was fine-tuned from facebook/opt-125m on the wikitext-2-raw-v1 dataset using the Causal Language Modeling task with the Hugging Face Trainer. Key settings likely included a learning rate around 2e-5, standard text tokenization with padding and truncation, and training for a small number of epochs. (You can add more specific details here if you remember them)

---

This model represents an early experimental step within the broader AI Safety initiative of "天算AI" / "SafeSky AI". Future work will involve using safety-focused datasets and training methodologies. ```