README.md

---
language:
- en
library_name: transformers
license: cc-by-4.0
tags:
- kl3m
- legal
- financial
- mlm
- roberta
- embedding
- uncased
pipeline_tag: fill-mask
tags_extended:
- feature-extraction
widget:
 - text: "<|cls|> this credit agreement is made and entered into as of january 1, 2025, by and between acme corporation, a delaware <|mask|>, and first national bank, as lender. <|sep|>"
 - example: "<|cls|> the<|mask|> agreement contains the entire understanding between <|sep|>"
 - temperature: 0.7
 - do_sample: true
date: '2025-01-15T00:00:00.000Z'
---

# kl3m-doc-micro-uncased-001

`kl3m-doc-micro-uncased-001` is a domain-specific masked language model (MLM) based on the DeBERTa-v2 architecture, specifically designed for legal and financial document analysis. With approximately 118M parameters, it provides a specialized model for NLP tasks in both fill-mask prediction and feature extraction for document embeddings. This uncased variant is particularly useful for case-insensitive applications, maintaining strong performance while disregarding capitalization differences.

## Model Details

- **Architecture**: DeBERTa-v2
- **Size**: 118M parameters
- **Hidden Size**: 512
- **Layers**: 16
- **Attention Heads**: 16
- **Intermediate Size**: 2048
- **Max Position Embeddings**: 512
- **Tokenizer**: [alea-institute/kl3m-004-128k-uncased](https://huggingface.co/alea-institute/kl3m-004-128k-uncased)
- **Vector Dimension**: 512 (hidden_size)
- **Pooling Strategy**: CLS token or mean pooling

## Use Cases

This model is particularly useful for:

- Lightweight document classification in legal and financial domains
- Entity recognition for specialized terminology
- Understanding legal citations and references
- Filling in missing terms in legal documents
- Feature extraction for downstream legal analysis tasks
- Document similarity and retrieval tasks where capitalization is not significant
- Edge computing applications with limited resources

The uncased nature of this model makes it more efficient for scenarios where case distinctions are not important to the task, while its micro size makes it suitable for severely resource-constrained environments.

## Standard Test Examples

Using our standardized test examples for comparing embedding models:

### Fill-Mask Results

1. **Contract Clause Heading**:  
   `"<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"`

   Top 5 predictions:
   1. warranties (0.8665)
   2. warrants (0.0857)
   3. warrant (0.0108)
   4. covenants (0.0079)
   5. agreements (0.0057)

2. **Defined Term Example**:  
   `"<|cls|> \"effective<|mask|>\" means the date on which all conditions precedent set forth in article v are satisfied or waived by the administrative agent. <|sep|>"`

   Top 5 predictions:
   1. date (0.9979)
   2. time (0.0011)
   3. day (0.0004)
   4. dates (0.0002)
   5. deadline (0.0001)

3. **Regulation Example**:  
   `"<|cls|> all transactions shall comply with the requirements set forth in the truth in<|mask|> act and its implementing regulation z. <|sep|>"`

   Top 5 predictions:
   1. the (0.4650)
   2. this (0.4385)
   3. such (0.0104)
   4. any (0.0050)
   5. an (0.0045)

### Document Similarity Results

Using the standardized document examples for embeddings:

| Document Pair | Cosine Similarity (CLS token) | Cosine Similarity (Mean pooling) |
|---------------|-------------------------------|----------------------------------|
| Court Complaint vs. Consumer Terms | 0.607 | 0.723 |
| Court Complaint vs. Credit Agreement | 0.631 | 0.837 |
| Consumer Terms vs. Credit Agreement | 0.744 | 0.762 |

The micro-sized model shows strong document similarity performance despite its compact size, with mean pooling showing particularly good results for capturing similarity between related legal documents. The highest similarity with CLS tokens is between Consumer Terms and Credit Agreement (0.744), while with mean pooling, the highest similarity is between Court Complaint and Credit Agreement (0.837).

## Usage

### Masked Language Modeling

For masked language modeling tasks, you can use the simple pipeline API:

```python
from transformers import pipeline

# Load the fill-mask pipeline with the model
fill_mask = pipeline('fill-mask', model="alea-institute/kl3m-doc-micro-uncased-001")

# Example: Contract clause heading
# Note the mask token placement - directly adjacent to "and" without space
text = "<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"
results = fill_mask(text)

# Display predictions
print("Top predictions:")
for result in results:
    print(f"- {result['token_str']} (score: {result['score']:.4f})")

# Output:
# Top predictions:
# - warranties (score: 0.8665)
# - warrants (score: 0.0857)
# - warrant (score: 0.0108)
# - covenants (score: 0.0079)
# - agreements (score: 0.0057)
```

### Feature Extraction for Embeddings

For document embeddings and similarity calculations, you can also use the pipeline API:

```python
from transformers import pipeline
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# Load the feature-extraction pipeline
extractor = pipeline('feature-extraction', model="alea-institute/kl3m-doc-micro-uncased-001", return_tensors=True)

# Example legal documents
texts = [
    # Court Complaint
    "<|cls|> in the united states district court for the eastern district of pennsylvania\n\njohn doe,\nplaintiff,\n\nvs.\n\nacme corporation,\ndefendant. <|sep|>",
    
    # Consumer Terms
    "<|cls|> terms and conditions\n\nlast updated: april 10, 2025\n\nthese terms and conditions govern your access to and use of the service. <|sep|>",
    
    # Credit Agreement
    "<|cls|> credit agreement\n\ndated as of april 10, 2025\n\namong\n\nacme borrower inc.,\nas the borrower,\n\nand bank of finance,\nas administrative agent. <|sep|>"
]

# Strategy 1: CLS token embeddings
cls_embeddings = []
for text in texts:
    features = extractor(text)
    # Get the CLS token (first token) embedding
    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
    cls_embedding = features_array[0]
    cls_embeddings.append(cls_embedding)

# Calculate similarity between documents using CLS tokens
cls_similarity = cosine_similarity(np.vstack(cls_embeddings))
print("\nDocument similarity (CLS token):")
print(np.round(cls_similarity, 3))
# Output:
# [[1.    0.607 0.631]
#  [0.607 1.    0.744]
#  [0.631 0.744 1.   ]]

# Strategy 2: Mean pooling
mean_embeddings = []
for text in texts:
    features = extractor(text)
    # Average over all tokens
    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
    mean_embedding = np.mean(features_array, axis=0)
    mean_embeddings.append(mean_embedding)

# Calculate similarity using mean pooling
mean_similarity = cosine_similarity(np.vstack(mean_embeddings))
print("\nDocument similarity (Mean pooling):")
print(np.round(mean_similarity, 3))
# Output:
# [[1.    0.723 0.837]
#  [0.723 1.    0.762]
#  [0.837 0.762 1.   ]]

# Print pairwise similarities
doc_names = ["Court Complaint", "Consumer Terms", "Credit Agreement"]
print("\nPairwise similarities:")
for i in range(len(doc_names)):
    for j in range(i+1, len(doc_names)):
        print(f"{doc_names[i]} vs. {doc_names[j]}:")
        print(f"  - CLS token: {cls_similarity[i, j]:.4f}")
        print(f"  - Mean pooling: {mean_similarity[i, j]:.4f}")
# Output:
# Pairwise similarities:
# Court Complaint vs. Consumer Terms:
#   - CLS token: 0.6073
#   - Mean pooling: 0.7233
# Court Complaint vs. Credit Agreement:
#   - CLS token: 0.6314
#   - Mean pooling: 0.8370
# Consumer Terms vs. Credit Agreement:
#   - CLS token: 0.7435
#   - Mean pooling: 0.7620
```

## Training

The model was trained on a diverse corpus of legal and financial documents, ensuring high-quality performance in these domains. It leverages the KL3M tokenizer which provides 9-17% more efficient tokenization for domain-specific content than cl100k_base or the LLaMA/Mistral tokenizers.

Training included both masked language modeling (MLM) objectives and attention to dense document representation for retrieval and classification tasks. The model was trained on lowercase text to improve efficiency and reduce model size.

## Intended Usage

This model is intended for both:

1. **Masked Language Modeling**: Filling in missing words/terms in legal and financial documents
2. **Document Embedding**: Generating fixed-length vector representations for document similarity and classification

It is particularly well-suited for resource-constrained environments and applications where case-sensitivity is not important.

## Special Tokens

This model includes the following special tokens:

- CLS token: `<|cls|>` (ID: 5) - Used for the beginning of input text
- MASK token: `<|mask|>` (ID: 6) - Used to mark tokens for prediction
- SEP token: `<|sep|>` (ID: 4) - Used for the end of input text
- PAD token: `<|pad|>` (ID: 2) - Used for padding sequences to a uniform length
- BOS token: `<|start|>` (ID: 0) - Beginning of sequence
- EOS token: `<|end|>` (ID: 1) - End of sequence
- UNK token: `<|unk|>` (ID: 3) - Unknown token

Important usage notes:

When using the MASK token for predictions, be aware that this model uses a **space-prefixed BPE tokenizer**. The <|mask|> token should be placed IMMEDIATELY after the previous token with NO space, because most tokens in this tokenizer have an initial space encoded within them. For example: `"word<|mask|>"` rather than `"word <|mask|>"`.
   
This space-aware placement is crucial for getting accurate predictions.

## Limitations

While extremely compact, this model has significant limitations:

- Limited parameter count (118M) means it captures less nuance than larger language models
- Uncased nature means it cannot distinguish between different capitalizations
- Primarily focused on English legal and financial texts
- Best suited for domain-specific rather than general-purpose tasks
- Maximum sequence length of 512 tokens may require chunking for lengthy documents
- Requires domain expertise to interpret results effectively

## References

- [KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications](https://arxiv.org/abs/2503.17247)
- [The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models](https://arxiv.org/abs/2504.07854)

## Citation

If you use this model in your research, please cite:

```bibtex
@misc{kl3m-doc-micro-uncased-001,
  author = {ALEA Institute},
  title = {kl3m-doc-micro-uncased-001: A Domain-Specific Uncased Language Model for Legal and Financial Text Analysis},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/alea-institute/kl3m-doc-micro-uncased-001}}
}

@article{bommarito2025kl3m,
  title={KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications},
  author={Bommarito, Michael J and Katz, Daniel Martin and Bommarito, Jillian},
  journal={arXiv preprint arXiv:2503.17247},
  year={2025}
}

@misc{bommarito2025kl3mdata,
  title={The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models},
  author={Bommarito II, Michael J. and Bommarito, Jillian and Katz, Daniel Martin},
  year={2025},
  eprint={2504.07854},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}
```

## License

This model is licensed under [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/).

## Contact

The KL3M model family is maintained by the [ALEA Institute](https://aleainstitute.ai). For technical support, collaboration opportunities, or general inquiries:
 
- Email: [email protected]
- Website: https://aleainstitute.ai
- GitHub: https://github.com/alea-institute/kl3m-model-research

![logo](https://aleainstitute.ai/images/alea-logo-ascii-1x1.png)