Upload README.md with huggingface_hub

README.md

@@ -12,73 +12,135 @@ inference: false
 ---
 # ChronoGPT
 
-## Model Description
+## ChronoGPT Highlights
 
 ChronoGPT is a series **high-performance chronologically consistent large language models (LLMs)** designed to eliminate lookahead bias and training leakage while maintaining good language understanding in time-sensitive applications. The model is pretrained on **diverse, high-quality, open-source, and timestamped text** to maintain chronological consistency.
 
-All models in the series achieve **HellaSwag benchmark scores that surpass those of the GPT-2 124M model with the same parameter count.** This approach preserves the integrity of historical analysis and enables more reliable economic and financial modeling.
+All models in the series achieve **HellaSwag benchmark scores that surpass those of the GPT-2 124M model.** This approach preserves the integrity of historical analysis and enables more reliable economic and financial modeling.
 
 - **Developed by:** Songrun He, Linying Lv, Asaf Manela, Jimmy Wu
 - **Model type:** Transformer-based autoregressive decoder (Modified modded-NanoGPT architecture)
 - **Language(s) (NLP):** English
 - **License:** MIT License
 
-## Model Sources
+## Model Overview
 
-- **Paper:** "Chronologically Consistent Large Language Models" (He, Lv, Manela, Wu, 2025)
+**ChronoGPT** has the following features:
+- Type: Causal Language Models
+- Training Stage: Pretraining
+- Number of Parameters: ~124 Million
+- Encoder & Decoder Partitioning: 6 encoder and 6 decoder layers
+- Tokenizer: GPT2Tokenizer from HuggingFace
+- Context Length: 1,792
+- Embedding Dimension: 768
 
-## How to Get Started with the Model
+## 🚀 Quickstart
 
-The model is compatible with following requirements:
+You can try ChronoGPT directly in your browser via Google Colab:
 
-```sh
+<p align="left">
+  <a href="https://colab.research.google.com/github/LinyingLyu/ChronoGPT/blob/main/ChronoGPT_tutorial.ipynb" target="_blank">
+    <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"/>
+  </a>
+</p>
+
+Or run it locally with:
+
+```bash
 pip install -r requirements.txt
 ```
 
-Here is an example code of using the model:
+### Text Generation
+
+The following contains a code snippet illustrating how to use the model generate content based on given inputs. 
 
 ```python
-from modeling_chronogpt import ChronoGPT
-import tiktoken
 import torch
+import torch.nn.functional as F
+import tiktoken
+from huggingface_hub import HfApi, login
+from ChronoGPT_inference import *
 
-device = 'cuda:0'
-max_length = 1792
+# ----------------------------- Setup -----------------------------
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+cache_dir = 'cache'  # Update this path as needed
 
 tokenizer = tiktoken.get_encoding("gpt2")
-model = ChronoGPT.from_pretrained("manelalab/chrono-gpt-v1-19991231", trust_remote_code=True).to(device)
-
-text = "Obviously, the time continuum has been disrupted, creating a new temporal event sequence resulting in this alternate reality. -- Dr. Brown, Back to the Future Part II"
-
-inputs = torch.tensor(tokenizer.encode(text))[:max_length].reshape(1,-1).to(device)
-logits, emb = model(inputs)
+max_length = 50
+num_return_sequences = 5
+seed = 123
+
+# -------------------------- Load Model --------------------------
+model = ChronoGPT.from_pretrained(
+    "manelalab/chrono-gpt-v1-20241231",
+    trust_remote_code=True,
+    cache_dir=cache_dir
+).to(device)
+
+# ------------------------ Prepare Input -------------------------
+prompt = "Hello, I am a language model,"
+tokens = tokenizer.encode(prompt)
+tokens = torch.tensor(tokens, dtype=torch.long).unsqueeze(0)
+tokens = tokens.repeat(num_return_sequences, 1).to(device)
+
+# -------------------- Sampling Initialization -------------------
+xgen = tokens.clone()
+sample_rng = torch.Generator(device=device)
+sample_rng.manual_seed(seed)
+
+# ------------------------- Text Generation -----------------------
+while xgen.size(1) < max_length:
+    with torch.no_grad():
+        with torch.autocast(device_type='cuda', dtype=torch.bfloat16):
+            logits, _ = model(xgen)
+
+        logits = logits[:, -1, :]  # Last token logits
+        probs = F.softmax(logits, dim=-1)
+        topk_probs, topk_indices = torch.topk(probs, 50, dim=-1)
+
+        sampled_idx = torch.multinomial(topk_probs, 1, generator=sample_rng)
+        next_token = torch.gather(topk_indices, -1, sampled_idx)
+
+        xgen = torch.cat([xgen, next_token], dim=1)
+
+# ------------------------- Decode Output -------------------------
+for i in range(num_return_sequences):
+    decoded_tokens = xgen[i, :max_length].tolist()
+    decoded_text = tokenizer.decode(decoded_tokens)
+    print(f"Rank sample {i}:\n{decoded_text}\n")
 ```
 
-## Training Details
-
-### Training Data
+### Extract Embeddings 
 
-- **Pretraining corpus:** Our initial model chrono-gpt-v1-19991231 is pretrained on 21 billion tokens of pre-2000, diverse, high-quality, and open-source text data to ensure no leakage of data afterwards.
-- **Incremental updates:** Yearly updates from 2000 to 2024 with an additional 65 billion tokens of timestamped text.
+The following contains a code snippet illustrating how to use the model generate embeddings of all layers based on given inputs. 
 
-### Training Procedure
-
-- **Architecture:** modded NanoGPT-based model with the Muon optimizer, Skip connections, rotary embeddings and flex attention.
-- **Objective:** Autoregressive text generation.
-
-## Evaluation
+```python
+import torch
+import torch.nn.functional as F
+import tiktoken
+from huggingface_hub import HfApi, login
+from ChronoGPT_inference import *
 
-### Testing Data, Factors & Metrics
+# ----------------------------- Setup -----------------------------
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+cache_dir = 'cache'  # Update this path as needed
 
-- **Language understanding:** Evaluated on **HellaSwag benchmark** tasks.
-- **Financial forecasting:** Evaluated using **return prediction task** based on Dow Jones Newswire data.
-- **Comparison models:** ChronoGPT was benchmarked against **BERT, FinBERT, StoriesLM-v1-1963, and Llama 3.1**.
+tokenizer = tiktoken.get_encoding("gpt2")
 
-### Results
+# -------------------------- Load Model --------------------------
+model = ChronoGPT.from_pretrained(
+    "manelalab/chrono-gpt-v1-20241231",
+    trust_remote_code=True,
+    cache_dir=cache_dir
+).to(device)
 
-- **HellaSwag Score:** chrono-gpt-v1-19991231 and chrono-gpt-v1-20241231 achieved HellaSwag score of 0.295 and 0.324 respectively, outperforming GPT-2 (0.294).
-- **Stock return predictions:** During the sample from 2008-01 to 2023-07, chrono-gpt-v1-realtime achieves a long-short portfolio **Sharpe ratio of 4.50**, outperforming BERT, FinBERT, and StoriesLM-v1-1963, and comparable to **Llama 3.1 8B (4.90)**.
+# ----------------------- Embedding Generation ---------------------
+text = "Obviously, the time continuum has been disrupted, creating a new temporal event sequence resulting in this alternate reality."
 
+inputs = torch.tensor(tokenizer.encode(text))[:max_length].reshape(1,-1).to(device)
+logits, emb = model(inputs)
+print('Dimension of embeddings:', emb[0].shape)
+```
 
 ## Citation
 
@@ -91,10 +153,9 @@ logits, emb = model(inputs)
 }
 ```
 
-## Model Card Authors
+### Model Card Authors
 
 - Songrun He (Washington University in St. Louis, [email protected])
 - Linying Lv (Washington University in St. Louis, [email protected])
 - Asaf Manela (Washington University in St. Louis, [email protected])
-- Jimmy Wu (Washington University in St. Louis, [email protected])
-
+- Jimmy Wu (Washington University in St. Louis, [email protected])