sayakpaul/butteflies_with_classes
Preview
•
Updated
•
222
Sayak Paul
sayakpaul
AI & ML interests
Diffusion models, representation learning
Recent Activity
updated
a dataset
8 days ago
sayakpaul/butteflies_with_classes
liked
a model
9 days ago
CompVis/stable-diffusion-v1-4
upvoted
an
article
10 days ago
How to train a new language model from scratch using Transformers and Tokenizers
Organizations
sayakpaul's activity
updated
a
dataset
8 days ago
liked
a
model
9 days ago
upvoted
an
article
10 days ago
Article
How to train a new language model from scratch using Transformers and Tokenizers
By
•
•
37
published
a
dataset
11 days ago
liked
a
model
12 days ago
ValueError: .to is not supported for 4-bit or 8-bit models
1
#2 opened 12 days ago
by
Boynn
Release Time of PipelineQuantizationConfig
1
#1 opened 12 days ago
by
Boynn
posted
an
update
13 days ago
Post
2334
Diffusers supports a good variety of quantization backends. It can be challenging to navigate through them, given the complex nature of diffusion pipelines in general.
So, @derekl35 set out to write a comprehensive guide that puts users in the front seat. Explore the different backends we support, learn the trade-offs they offer, and finally, check out the cool space we built that lets you compare quantization results.
Give it a go here:
https://lnkd.in/gf8Pi4-2
So, @derekl35 set out to write a comprehensive guide that puts users in the front seat. Explore the different backends we support, learn the trade-offs they offer, and finally, check out the cool space we built that lets you compare quantization results.
Give it a go here:
https://lnkd.in/gf8Pi4-2
upvoted
an
article
14 days ago
Article
Exploring Quantization Backends in Diffusers
By
and 2 others
•
•
31
updated
a
model
14 days ago
published
an
article
14 days ago
Article
Exploring Quantization Backends in Diffusers
By
and 2 others
•
•
31
updated
a
dataset
14 days ago
posted
an
update
15 days ago
Post
1672
Despite the emergence of combining LLM and DiT architectures for T2I synthesis, its design remains severely understudied.
This was done long ago and got into CVPR25 -- super excited to finally share it now, along with the data and code ♥️
We explore several architectural choices that affect this design. We provide an open & reproducible training recipe that works at scale.
Works like Playground v3 have already explored a deep fusion between an LLM and a DiT, sharing their representations through layerwise attention. They exhibit excellent performance on T2I.
Despite its compelling results and other performance virtues, it remains unexplored, which is what we want to improve in our work. Specifically, we take a pre-trained LLM (Gemma-2B) and trainable DiT, and set out to explore what makes a "good deep fusion" between the two for T2I.
We explore several key questions in the work, such as:
Q1: How should we do attention? We considered several alternatives. PixArt-Alpha like attention (cross-attention) is very promising.
Q2: Should we incorporate additional text modulation?
Q3: Can we eliminate timestep conditioning?
Q4: How do we do positional encodings?
Q5: Do instruction-tuned LLMs help deep fusion?
Q6: Would using a decoder LLM from a multimodal model be helpful?
Q7: Does using a better variant of Gemma help?
Based on the above findings, we arrive at FuseDiT with the following components on top of the base architecture from the findings of our experiments.
* No AdaLN-Zero modules
* 1D + 2D-RoPE
* Gemma 2 2B, adjusting DiT configurations accordingly
We trained FuseDiT on a mixture from CC12M, JourneyDB, & SA (~26M image-text pairs) for 800 steps. While not the best model, it's encouraging to develop something in a guided manner using open datasets.
To know more (code, models, all are available), please check out the paper:
https://lnkd.in/gg6qyqZX.
This was done long ago and got into CVPR25 -- super excited to finally share it now, along with the data and code ♥️
We explore several architectural choices that affect this design. We provide an open & reproducible training recipe that works at scale.
Works like Playground v3 have already explored a deep fusion between an LLM and a DiT, sharing their representations through layerwise attention. They exhibit excellent performance on T2I.
Despite its compelling results and other performance virtues, it remains unexplored, which is what we want to improve in our work. Specifically, we take a pre-trained LLM (Gemma-2B) and trainable DiT, and set out to explore what makes a "good deep fusion" between the two for T2I.
We explore several key questions in the work, such as:
Q1: How should we do attention? We considered several alternatives. PixArt-Alpha like attention (cross-attention) is very promising.
Q2: Should we incorporate additional text modulation?
Q3: Can we eliminate timestep conditioning?
Q4: How do we do positional encodings?
Q5: Do instruction-tuned LLMs help deep fusion?
Q6: Would using a decoder LLM from a multimodal model be helpful?
Q7: Does using a better variant of Gemma help?
Based on the above findings, we arrive at FuseDiT with the following components on top of the base architecture from the findings of our experiments.
* No AdaLN-Zero modules
* 1D + 2D-RoPE
* Gemma 2 2B, adjusting DiT configurations accordingly
We trained FuseDiT on a mixture from CC12M, JourneyDB, & SA (~26M image-text pairs) for 800 steps. While not the best model, it's encouraging to develop something in a guided manner using open datasets.
To know more (code, models, all are available), please check out the paper:
https://lnkd.in/gg6qyqZX.
updated
a
dataset
15 days ago
Update README.md to include some samples
#3 opened 16 days ago
by
sayakpaul
authored
a
paper
19 days ago
Update README.md to include paper name
#2 opened 19 days ago
by
sayakpaul
commented
a
paper
19 days ago
updated
a
collection
21 days ago
updated
a
collection
22 days ago