alea-institute
/

kl3m-doc-pico-long-001

@@ -1,199 +1,208 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
 ## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+language:
+- en
 library_name: transformers
+license: cc-by-4.0
+tags:
+- kl3m
+- legal
+- financial
+- mlm
+- roberta
+- long-context
+pipeline_tag: feature-extraction
+widget:
+ - text: "<|cls|> This Credit Agreement is made and entered into as of [DATE], by and between [BORROWER NAME], a Delaware corporation with its principal place of business at [ADDRESS] (the \"<|mask|>\"), and [LENDER NAME], a national banking association (the \"Lender\"). <|sep|>"
+ - example: "<|cls|> Form 10-<|mask|> is a report filed with the Securities and Exchange Commission used by public companies to disclose financial results on a quarterly basis. It is submitted within 45 days of the end of each of the first three fiscal quarters of the company's fiscal year. <|sep|>"
+ - temperature: 0.7
+ - do_sample: true
+date: '2025-02-28T00:00:00.000Z'
 ---
+# kl3m-doc-pico-long-001
+`kl3m-doc-pico-long-001` is a domain-specific model based on the RoBERTa architecture, specifically designed for feature extraction in legal and financial document analysis with support for longer context windows. It shares the same basic architecture as the kl3m-doc-pico-001 model but has been trained with extended context capabilities of up to 4,096 tokens. While the model architecture supports masked language modeling (MLM), it is primarily optimized for feature extraction tasks.
 ## Model Details
+- **Architecture**: RoBERTa
+- **Size**: 41M parameters
+- **Hidden Size**: 256
+- **Layers**: 8
+- **Attention Heads**: 8
+- **Max Sequence Length**: 4,096
+- **Tokenizer**: [alea-institute/kl3m-004-128k-cased](https://huggingface.co/alea-institute/kl3m-004-128k-cased)
+## Use Cases
+This model is particularly useful for:
+- Document classification in legal and financial domains with lengthy documents
+- Analyzing relationships between distant parts of a document
+- Processing lengthy agreements, contracts, and regulatory filings
+- Feature extraction for downstream legal analysis tasks requiring longer context
+The extended context length makes this model especially suited for working with longer portions of legal documents, contracts, and financial statements that often exceed the context limits of standard models.
+## Usage
+The primary use case for this model is feature extraction for document embedding and downstream classification tasks. Here's how to use it for feature extraction:
+```python
+from transformers import AutoModel, AutoTokenizer
+import torch
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained("alea-institute/kl3m-doc-pico-long-001")
+model = AutoModel.from_pretrained("alea-institute/kl3m-doc-pico-long-001")
+# Example with legal document context
+text = "<|cls|> This Credit Agreement is made and entered into as of [DATE], by and between [BORROWER NAME], a Delaware corporation with its principal place of business at [ADDRESS] (the \"Borrower\"), and [LENDER NAME], a national banking association (the \"Lender\"). <|sep|>"
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+# Get the embeddings
+# The CLS token embedding is typically used for classification tasks
+cls_embedding = outputs.last_hidden_state[:, 0, :]
+print(f"CLS embedding shape: {cls_embedding.shape}")  # Should be [1, 256]
+# For document similarity, you can use the mean of all token embeddings
+mean_embedding = outputs.last_hidden_state.mean(dim=1)
+print(f"Mean embedding shape: {mean_embedding.shape}")  # Should be [1, 256]
+# You can also process multiple documents in a batch
+texts = [
+    "<|cls|> This Credit Agreement is made and entered into as of [DATE]... <|sep|>",
+    "<|cls|> Form 10-Q is a report filed with the Securities and Exchange Commission... <|sep|>"
+]
+batch_inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
+batch_outputs = model(**batch_inputs)
+# Get batch embeddings
+batch_embeddings = batch_outputs.last_hidden_state[:, 0, :]
+print(f"Batch embeddings shape: {batch_embeddings.shape}")  # Should be [2, 256]
+```
+While the model architecture supports masked language modeling (MLM), it was not specifically trained for this task. If you still want to experiment with it for MLM, you can use the following code, but be aware that performance may be limited:
+```python
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+import torch
+# Load model with MLM head (note: this part wasn't specifically trained)
+tokenizer = AutoTokenizer.from_pretrained("alea-institute/kl3m-doc-pico-long-001")
+mlm_model = AutoModelForMaskedLM.from_pretrained("alea-institute/kl3m-doc-pico-long-001")
+# Example with masked token
+text = "<|cls|> Form 10-<|mask|> is a report filed with the Securities and Exchange Commission... <|sep|>"
+inputs = tokenizer(text, return_tensors="pt")
+outputs = mlm_model(**inputs)
+# Get predictions for masked token
+masked_index = torch.where(inputs.input_ids[0] == tokenizer.mask_token_id)[0].item()
+probs = outputs.logits[0, masked_index].softmax(dim=0)
+top_5 = torch.topk(probs, 5)
+print("Top 5 predictions for masked token:")
+for i, (score, idx) in enumerate(zip(top_5.values, top_5.indices)):
+    token = tokenizer.decode(idx).strip()
+    print(f"{i+1}. {token} ({score.item():.3f})")
+```
+## Long Context Capabilities
+This model extends the context window from the standard 512 tokens to 4,096 tokens, enabling it to process much longer documents. The extended context window allows the model to:
+1. Process full legal agreements and contracts without truncation
+2. Maintain awareness of context from the beginning of a document when analyzing later sections
+3. Better handle documents with complex cross-references and definitions
+4. Reduce the need for document chunking in downstream applications
+## Training
+The model was trained on a diverse corpus of legal and financial documents, ensuring high-quality performance in these domains. This version has been specifically optimized for feature extraction with longer contexts, incorporating the following key aspects:
+1. Position embeddings extended to 4,096 tokens
+2. Training focused on dense document representation for retrieval and classification tasks
+3. Objectives optimized for feature extraction rather than token prediction
+4. Additional training on full-length documents to ensure contextual understanding
+It leverages the KL3M tokenizer which provides 9-17% more efficient tokenization for domain-specific content than general-purpose tokenizers.
+### Intended Usage
+This model was specifically designed and trained for:
+1. **Document embedding**: Generating fixed-length vector representations of documents for similarity comparison
+2. **Feature extraction**: Creating inputs for downstream classification and regression tasks
+3. **Semantic search**: Finding similar documents across large collections
+4. **Document clustering**: Discovering patterns across legal and financial document collections
+While the model architecture includes the capability for masked language modeling, the model weights were not specifically optimized for this task. For masked language modeling applications, consider using a model explicitly trained for that purpose.
+## Special Tokens
+This model uses custom special tokens which must be used explicitly:
+- CLS token: `<|cls|>` (ID: 5) - Should be added at the beginning of input text
+- MASK token: `<|mask|>` (ID: 6) - Used to mark tokens for prediction
+- SEP token: `<|sep|>` (ID: 4) - Should be added at the end of input text
+- PAD token: `<|pad|>` (ID: 2) - Used for padding sequences to a uniform length
+- BOS token: `<|start|>` (ID: 0) - Beginning of sequence
+- EOS token: `<|end|>` (ID: 1) - End of sequence
+- UNK token: `<|unk|>` (ID: 3) - Unknown token
+The model also includes additional special tokens for chat and instruction contexts:
+- `<|system|>` (ID: 7)
+- `</|system|>` (ID: 8)
+- `<|user|>` (ID: 9)
+- `</|user|>` (ID: 10)
+- `<|instruction|>` (ID: 11)
+- `</|instruction|>` (ID: 12)
+For best results, you should **explicitly add** the CLS and SEP tokens to your input text, as shown in the example above.
+## Limitations
+While providing extended context capabilities, this model has some limitations:
+- Smaller parameter count (41M) compared to larger language models
+- Primarily focused on English legal and financial texts
+- Best suited for domain-specific rather than general-purpose tasks
+- Requires domain expertise to interpret results effectively
+- May show decreased performance at the far edges of the context window
+## References
+- [KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications](https://arxiv.org/abs/2503.17247)
+- [The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models]() (Forthcoming)
+## Citation
+If you use this model in your research, please cite:
+```bibtex
+@misc{bommarito2025kl3m,
+  title={KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications},
+  author={Bommarito II, Michael J. and Katz, Daniel Martin and Bommarito, Jillian},
+  year={2025},
+  eprint={2503.17247},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL}
+}
+```
+## License
+This model is licensed under [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/).
+## Contact
+The KL3M model family is maintained by the [ALEA Institute](https://aleainstitute.ai). For technical support, collaboration opportunities, or general inquiries:
+- Email: [email protected]
+- Website: https://aleainstitute.ai
+- GitHub: https://github.com/alea-institute/kl3m-model-research
+![https://aleainstitute.ai](https://aleainstitute.ai/images/alea-logo-ascii-1x1.png)