John6666/alchemix-illustrious-v25-sdxl
Text-to-Image
•
Updated
John Smith PRO
John6666
AI & ML interests
None yet
Recent Activity
published
a model
about 1 hour ago
John6666/alchemix-illustrious-v25-sdxl
published
a model
about 1 hour ago
John6666/catpony-real-v31-sdxl
published
a model
about 1 hour ago
John6666/evalisennia-realistic-east-asian-female-v30-sdxl
Organizations
John6666's activity
published
8
models
about 1 hour ago
John6666/catpony-real-v31-sdxl
Text-to-Image
•
Updated
John6666/evalisennia-realistic-east-asian-female-v30-sdxl
Text-to-Image
•
Updated
John6666/femix-hassakuxl-v12-sdxl
Text-to-Image
•
Updated
John6666/ls-tiro-xl-vpred-v20-sdxl
Text-to-Image
•
Updated
John6666/lustify-sdxl-nsfw-checkpoint-olt-fixed-textures-sdxl
Text-to-Image
•
Updated
John6666/rrrreal-il-v20-sdxl
Text-to-Image
•
Updated
John6666/shiitake-mix-v200-sdxl
Text-to-Image
•
Updated
updated
2
models
about 1 hour ago
upvoted
an
article
about 1 hour ago
Article
The New and Fresh analytics in Inference Endpoints
•
13
updated
3
models
about 1 hour ago
updated
3
models
about 2 hours ago
reacted to
Kseniase's
post with 👀
about 2 hours ago
Post
507
8 types of RoPE
As we always use Transformers, it's helpful to understand RoPE—Rotary Position Embedding. Since token order matters, RoPE encodes it by rotating token embeddings based on their position, so the model knows how to interpret which token comes first, second, and so on.
Here are 8 types of RoPE that can be implemented in different cases:
1. Original RoPE -> RoFormer: Enhanced Transformer with Rotary Position Embedding (2104.09864)
Encodes token positions by rotating token embeddings in the complex plane via a position-based rotation matrix, thereby providing the self-attention mechanism with relative positional info.
2. LongRoPE -> LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (2402.13753)
Extends the context window of pre-trained LLMs to 2048k tokens, leveraging non-uniformities in positional interpolation with an efficient search.
3. LongRoPE2 -> LongRoPE2: Near-Lossless LLM Context Window Scaling (2502.20082)
Extends the effective context window of pre-trained LLMs to the target! length, rescaling RoPE guided by “needle-driven” perplexity.
4. Multimodal RoPE (MRoPE) -> Qwen2.5-VL Technical Report (2502.13923)
Decomposes positional embedding into 3 components: temporal, height and width, so that positional features are aligned across modalities: text, images and videos.
5. Directional RoPE (DRoPE) -> DRoPE: Directional Rotary Position Embedding for Efficient Agent Interaction Modeling (2503.15029)
Adds an identity scalar, improving how angles are handled without extra complexity. It helps balance accuracy, speed, and memory usage.
6. VideoRoPE -> VideoRoPE: What Makes for Good Video Rotary Position Embedding? (2502.05173)
Adapts RoPE for video, featuring 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable spacing.
7. VRoPE -> VRoPE: Rotary Position Embedding for Video Large Language Models (2502.11664)
An another RoPE for video, which restructures positional indices and balances encoding for uniform spatial focus.
8. XPos (Extrapolatable Position Embedding) -> https://huggingface.co/papers/2212.10
Introduces an exponential decay factor into the rotation matrix, improving stability on long sequences.
As we always use Transformers, it's helpful to understand RoPE—Rotary Position Embedding. Since token order matters, RoPE encodes it by rotating token embeddings based on their position, so the model knows how to interpret which token comes first, second, and so on.
Here are 8 types of RoPE that can be implemented in different cases:
1. Original RoPE -> RoFormer: Enhanced Transformer with Rotary Position Embedding (2104.09864)
Encodes token positions by rotating token embeddings in the complex plane via a position-based rotation matrix, thereby providing the self-attention mechanism with relative positional info.
2. LongRoPE -> LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (2402.13753)
Extends the context window of pre-trained LLMs to 2048k tokens, leveraging non-uniformities in positional interpolation with an efficient search.
3. LongRoPE2 -> LongRoPE2: Near-Lossless LLM Context Window Scaling (2502.20082)
Extends the effective context window of pre-trained LLMs to the target! length, rescaling RoPE guided by “needle-driven” perplexity.
4. Multimodal RoPE (MRoPE) -> Qwen2.5-VL Technical Report (2502.13923)
Decomposes positional embedding into 3 components: temporal, height and width, so that positional features are aligned across modalities: text, images and videos.
5. Directional RoPE (DRoPE) -> DRoPE: Directional Rotary Position Embedding for Efficient Agent Interaction Modeling (2503.15029)
Adds an identity scalar, improving how angles are handled without extra complexity. It helps balance accuracy, speed, and memory usage.
6. VideoRoPE -> VideoRoPE: What Makes for Good Video Rotary Position Embedding? (2502.05173)
Adapts RoPE for video, featuring 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable spacing.
7. VRoPE -> VRoPE: Rotary Position Embedding for Video Large Language Models (2502.11664)
An another RoPE for video, which restructures positional indices and balances encoding for uniform spatial focus.
8. XPos (Extrapolatable Position Embedding) -> https://huggingface.co/papers/2212.10
Introduces an exponential decay factor into the rotation matrix, improving stability on long sequences.
- 1 reply
reacted to
hanzla's
post with 🤗
about 2 hours ago
Post
188
👋 Hi all!
For any AI agent, internet search 🔎 is an important tool. However, with APIs like Tavily and Exa, it becomes really difficult to keep up with the cost. In some cases, these Internet APIs cost more than the LLM.
To solve, this, I am making a playwright wrapper API on top of publicly available searXNG instances. This will enable agent applications to fetch internet results for free.
Currently, I have set up a basic GitHub repo, and I will continue developing advanced search features, such as image search 🖼️
Github: https://github.com/HanzlaJavaid/Free-Search/tree/main
🚀 Try the deployed version: https://freesearch.replit.app/docs
If you find this useful, consider starring ⭐️ the GitHub repository to support further development!
For any AI agent, internet search 🔎 is an important tool. However, with APIs like Tavily and Exa, it becomes really difficult to keep up with the cost. In some cases, these Internet APIs cost more than the LLM.
To solve, this, I am making a playwright wrapper API on top of publicly available searXNG instances. This will enable agent applications to fetch internet results for free.
Currently, I have set up a basic GitHub repo, and I will continue developing advanced search features, such as image search 🖼️
Github: https://github.com/HanzlaJavaid/Free-Search/tree/main
🚀 Try the deployed version: https://freesearch.replit.app/docs
If you find this useful, consider starring ⭐️ the GitHub repository to support further development!
reacted to
louisbrulenaudet's
post with 👀
about 2 hours ago
Post
198
I’ve just released logfire-callback on PyPI, designed to facilitate monitoring of Hugging Face Transformer training loops using Pydantic Logfire 🤗
The callback will automatically log training start with configuration parameters, periodic metrics and training completion ⏱️
Install the package using pip:
First, ensure you have a Logfire API token and set it as an environment variable:
Then use the callback in your training code:
If you have any feedback, please reach out at @louisbrulenaudet
The callback will automatically log training start with configuration parameters, periodic metrics and training completion ⏱️
Install the package using pip:
pip install logfire-callbackFirst, ensure you have a Logfire API token and set it as an environment variable:
export LOGFIRE_TOKEN=your_logfire_tokenThen use the callback in your training code:
from transformers import Trainer, TrainingArguments
from logfire_callback import LogfireCallback
# Initialize your model, dataset, etc.
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3,
# ... other training arguments
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
callbacks=[LogfireCallback()] # Add the Logfire callback here
)
trainer.train()If you have any feedback, please reach out at @louisbrulenaudet