base_model: EleutherAI/pythia-70m-deduped library_name: peft

Model Card for Fin-Interpret

This model is a fine-tuned version of EleutherAI's Pythia-70m-deduped, specifically adapted to understand and interpret text from financial documents. It leverages Parameter-Efficient Fine-Tuning (PEFT) to achieve this specialization efficiently.

Model Details

Model Description

A financial model trained to interprete financial documents. This model is built upon the EleutherAI/pythia-70m-deduped base model and fine-tuned using the PEFT library. The goal is to provide a language model that is adept at tasks related to financial text analysis, such as summarization, question answering, and information extraction from financial reports, filings, and news articles.

Training datasets: The model was trained on the Synthik Finance Narrative dataset, which contains a collection of financial documents. You can find more information about the dataset at https://www.synthik.io/datasets/36db6f48-0e63-43b5-a9d8-0252b8137835.
Funded by [Synthik]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Model type: Causal Language Model
Language(s) (NLP): English
License: Apache-2.0
Finetuned from model: EleutherAI/pythia-70m-deduped

Model Sources [optional]

Repository: [More Information Needed]
Paper [optional]: [More Information Needed]
Demo [optional]: [More Information Needed]

Uses

Direct Use

This model is intended for direct use in generating text-based responses to prompts related to financial documents. It can be used for tasks such as:

Answering questions about a company's financial performance based on an annual report.
Summarizing key points from a financial news article.
Extracting specific financial metrics from a given text.

Downstream Use [optional]

The model can be further fine-tuned on more specific financial NLP tasks, such as:

Sentiment analysis of financial news.
Named entity recognition for financial entities.
Classification of financial documents.

Out-of-Scope Use

This model is not a financial advisor and should not be used for making investment decisions. The outputs are generated based on patterns in the training data and may not be factually accurate. The model is not intended for use in high-stakes applications where incorrect information could lead to financial loss. Use in any of the following ways is out-of-scope:

Generating financial advice or recommendations.
Predicting stock market movements.
Automated trading.

Bias, Risks, and Limitations

The model's knowledge is limited to the information present in the Synthik Finance Narrative dataset and the underlying Pythia-70m-deduped model. Therefore, it may exhibit biases present in the training data. The model may generate plausible-sounding but incorrect or misleading information. It is crucial to verify any factual claims made by the model with reliable financial sources.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. It is recommended to have a human-in-the-loop to review the model's outputs before they are used in any decision-making process.

How to Get Started with the Model

Use the code below to get started with the model. Make sure you have the transformers and peft libraries installed.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load the base model and tokenizer
base_model_id = "EleutherAI/pythia-70m-deduped"
model = AutoModelForCausalLM.from_pretrained(base_model_id)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# Load the PEFT model
peft_model_id = "YOUR_MODEL_ID"  # Replace with the actual model ID
model = PeftModel.from_pretrained(model, peft_model_id)

# Prepare your input
prompt = "What were the key financial highlights from the attached report?"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate a response
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

The model was fine-tuned on the Synthik Finance Narrative dataset. This dataset is comprised of various financial documents and is designed to train models for financial text interpretation tasks.

Training Procedure

The model was fine-tuned using the Parameter-Efficient Fine-Tuning (PEFT) library, which significantly reduces the computational and storage costs associated with training large language models. This approach involves freezing the majority of the base model's parameters and only training a small number of additional parameters.

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

Training regime: [More Information Needed]

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

[More Information Needed]

Factors

[More Information Needed]

Metrics

[More Information Needed]

Results

[More Information Needed]

Summary

[More Information Needed]

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: [More Information Needed]
Hours used: [More Information Needed]
Cloud Provider: [More Information Needed]
Compute Region: [More Information Needed]
Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

The model uses the same architecture as the EleutherAI/pythia-70m-deduped model, which is a GPT-NeoX-based transformer model. The objective of the fine-tuning was to minimize the causal language modeling loss on the financial document dataset.

Compute Infrastructure

Hardware

[More Information Needed]

Software

The fine-tuning was performed using the Hugging Face transformers and peft libraries.

Glossary [optional]

PEFT (Parameter-Efficient Fine-Tuning): A set of techniques to adapt large pre-trained models to new tasks by fine-tuning only a small fraction of their parameters.
Causal Language Model: A type of language model that predicts the next token in a sequence given the preceding tokens.

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]

Framework versions

PEFT 0.7.1