README
π
BangumiBase
community
AI & ML interests
Character Database of Bangumis (If you need character LoRAs, see: https://huggingface.co/CyberHarem)
Recent Activity
narugo1992Β
updated
a
Space
4 days ago
AbstractPhilΒ
posted
an
update
about 1 month ago
Post
411
With flan-t5-base and clip models as teachers; I have produced and successfully trained a dual-shunt cross-attention adapter archetype. This is NOT a lora.
This adapter is currently tasked with taking the T5-flan-base to guide the outputs of VIT-L-14 and/or VIT-bigG-14, and the opposite is equally usable and utilizable within the archetype. Meaning the CLIP_G can also guide the T5-FLAN-base.
These checkpoints were trained with 20 million synthetic human-templated captions, and they can be heavily improved by multiple languages, additional depiction context, and any sort of finetune task desired of the user that can be applied to the T5-flan-base with little to no training due to the adapter's functionality and accuracy.
VIT-L-14 adapters only took a couple hours on a colab a100 and the VIT-bigG-14 took about 4 hours. So you can rapidly adapt many of these in short periods of time with almost no additional overhead beyond the single t5-flan-base required. Each can be compiled, loaded, and offloaded.
This is a cross-attention system meant to shape encoded text after the output is received from the clip models and is very fast to inference - the t5-flan-base on the other hand isn't the fastest.
It's trained on a form of cooperative association with a series of complex losses designed specifically for this associative process.
This adapter has individual gating for tokenization context with a multitude of safeguards to prevent overfitting during rapid learning and can be paired with any number of additional other adapters.
I'm currently formatting the comfyui nodes that will allow easy conditioning shift to showcase the full power of this cooperative system's capability.
The comfyui nodes will be available here shortly, I just need to write them.
https://github.com/AbstractEyes/comfy-clip-shunts
This adapter is currently tasked with taking the T5-flan-base to guide the outputs of VIT-L-14 and/or VIT-bigG-14, and the opposite is equally usable and utilizable within the archetype. Meaning the CLIP_G can also guide the T5-FLAN-base.
These checkpoints were trained with 20 million synthetic human-templated captions, and they can be heavily improved by multiple languages, additional depiction context, and any sort of finetune task desired of the user that can be applied to the T5-flan-base with little to no training due to the adapter's functionality and accuracy.
VIT-L-14 adapters only took a couple hours on a colab a100 and the VIT-bigG-14 took about 4 hours. So you can rapidly adapt many of these in short periods of time with almost no additional overhead beyond the single t5-flan-base required. Each can be compiled, loaded, and offloaded.
This is a cross-attention system meant to shape encoded text after the output is received from the clip models and is very fast to inference - the t5-flan-base on the other hand isn't the fastest.
It's trained on a form of cooperative association with a series of complex losses designed specifically for this associative process.
This adapter has individual gating for tokenization context with a multitude of safeguards to prevent overfitting during rapid learning and can be paired with any number of additional other adapters.
I'm currently formatting the comfyui nodes that will allow easy conditioning shift to showcase the full power of this cooperative system's capability.
The comfyui nodes will be available here shortly, I just need to write them.
https://github.com/AbstractEyes/comfy-clip-shunts
-
- 1 reply
AbstractPhilΒ
posted
an
update
about 1 month ago
Post
371
The T5-small + VIT-L-14 guidance shunt adapter is ready for toy use.
AbstractPhil/t5-vit-14-v1
Included is a simple drop-in for sdxl experimentation using colab.
The outcome is okay but not great - diffusers is a headache so I spent more time trying to disjoint that machine than I did actually messing with this adapter.
I trained two variations of the baseline adapter;
t5-small vanilla and t5-small-human-associated-try2-pass3.
The vanilla was more accurate to adding context while the human associated stays locked onto human topics like a bloodhound... badly. Both ended up being substandard, even with a robust adapter like this.
Finetunes with specific goals can complete at runtime if desired due to the t5-small's tiny size, clip_l's inference speed, and the adapter's size. The adapter is very small and has safeguards for overfitting that can be disabled, so runtime freezing and adaptive shifts can be a viable methodology to immediate task pipeline adaptation.
The t5-small lacks the behavioral complexity of a model more built for such a task such as the base, large, or xxl - or even the Flan T5-small. However, this doesn't slow the little brain slug down. It guides and it's wrappers have many rapid generation potentials, whether it's trained the way I trained it or not.
The proof of concept is there, and the outcomes are present. Judge yourself.
The next variation will be more dims, more catches, higher conv, and additional safeguards to prevent overfitting - as well as including considerably more laion flavors so the T5-flan-base doesn't overwhelm or vise-versa.
AbstractPhil/t5-vit-14-v1
Included is a simple drop-in for sdxl experimentation using colab.
The outcome is okay but not great - diffusers is a headache so I spent more time trying to disjoint that machine than I did actually messing with this adapter.
I trained two variations of the baseline adapter;
t5-small vanilla and t5-small-human-associated-try2-pass3.
The vanilla was more accurate to adding context while the human associated stays locked onto human topics like a bloodhound... badly. Both ended up being substandard, even with a robust adapter like this.
Finetunes with specific goals can complete at runtime if desired due to the t5-small's tiny size, clip_l's inference speed, and the adapter's size. The adapter is very small and has safeguards for overfitting that can be disabled, so runtime freezing and adaptive shifts can be a viable methodology to immediate task pipeline adaptation.
The t5-small lacks the behavioral complexity of a model more built for such a task such as the base, large, or xxl - or even the Flan T5-small. However, this doesn't slow the little brain slug down. It guides and it's wrappers have many rapid generation potentials, whether it's trained the way I trained it or not.
The proof of concept is there, and the outcomes are present. Judge yourself.
The next variation will be more dims, more catches, higher conv, and additional safeguards to prevent overfitting - as well as including considerably more laion flavors so the T5-flan-base doesn't overwhelm or vise-versa.
-
- 1 reply
AbstractPhilΒ
posted
an
update
about 1 month ago
Post
596
Forcefeeding masked T5-Small 1 billion human-association captions to fry it's brain. I really don't know how long it'll take until I start nor do I know the logistic challenges I'll face when moving data from A to B, but the outcome should completely fry it and make it only fixate on human and diffusion responses. Should be a fun experiment that can just kind of run on automation.
The experiment's captions are available... mostly on my hf, I've had some rate limit problems that caused them to halt and I think I need to autogen another 100 million complex captions.
This WILL form heavy bias and burn-points. Random words will be peppered in the mix to allow the T5-Small to retain at least some semblance of what it was before I lobotomize it.
Likely I'll completely freeze half and burn the other half for a couple million as a test point. See how it takes or if it dies before 50k or something and need a refined process.
Oh great, even better. It didn't include the longer prompt variations. This won't start today.
Alright training began. I'm introducing a high degree variant of noise and chatter for the t5 to learn to bypass - while simultaneously increasing additional information output from the t5 in the process.
So far the outcome has been a degree of introduction for new information in the output. while simultaneously introducing rule of 3 parameterization into the T5 small.
I have high hopes.
The experiment's captions are available... mostly on my hf, I've had some rate limit problems that caused them to halt and I think I need to autogen another 100 million complex captions.
This WILL form heavy bias and burn-points. Random words will be peppered in the mix to allow the T5-Small to retain at least some semblance of what it was before I lobotomize it.
Likely I'll completely freeze half and burn the other half for a couple million as a test point. See how it takes or if it dies before 50k or something and need a refined process.
Oh great, even better. It didn't include the longer prompt variations. This won't start today.
Alright training began. I'm introducing a high degree variant of noise and chatter for the t5 to learn to bypass - while simultaneously increasing additional information output from the t5 in the process.
So far the outcome has been a degree of introduction for new information in the output. while simultaneously introducing rule of 3 parameterization into the T5 small.
I have high hopes.
-
- 3 replies
AbstractPhilΒ
posted
an
update
about 2 months ago
Post
441
My indev Surge training methodology and paradigm is powerful. The preliminary tests will be available for debugging soon using a customized sd-scripts and a series of full finetunes using sdxl as a catalyst to the training paradigm.
https://civitai.com/articles/14195/the-methodology-of-surge-training-loss-math
The datasets I'm sourcing are going to be catalysts and tests for the power of Surge to teach very sticky or difficult to understand elements; such as text, positioning, offset, controlnet poses, and more directly into the very stubborn SDXL infrastructure without additional tools.
Should be noted that my current running finetunes based on BeatriXL are not Surge trained - so you won't gain knowledge on Surge from them.
GPT and I have prototyped a new version of SD15 that operates on additional attention heads to match the Surge formula, the Omega-VIT-L reformed, a zeroed unet, and the Flux 16 channel AE.
I'll call it SD-SURGE - as it's not sd15 anymore.
The first surge trainings are already under way.
https://civitai.com/articles/14195/the-methodology-of-surge-training-loss-math
The datasets I'm sourcing are going to be catalysts and tests for the power of Surge to teach very sticky or difficult to understand elements; such as text, positioning, offset, controlnet poses, and more directly into the very stubborn SDXL infrastructure without additional tools.
Should be noted that my current running finetunes based on BeatriXL are not Surge trained - so you won't gain knowledge on Surge from them.
GPT and I have prototyped a new version of SD15 that operates on additional attention heads to match the Surge formula, the Omega-VIT-L reformed, a zeroed unet, and the Flux 16 channel AE.
I'll call it SD-SURGE - as it's not sd15 anymore.
The first surge trainings are already under way.
-
- 1 reply
Post
3650
πAraClip is now fully integrated with Hugging Face π€
AraClip is a specialized CLIP model that was created by @pain and optimized for Arabic text-image retrieval tasksπ₯
π Try it out π
π€ model: Arabic-Clip/araclip
𧩠Gradio demo: Arabic-Clip/Araclip-Simplified
π website: https://arabic-clip.github.io/Arabic-CLIP/
AraClip is a specialized CLIP model that was created by @pain and optimized for Arabic text-image retrieval tasksπ₯
π Try it out π
π€ model: Arabic-Clip/araclip
𧩠Gradio demo: Arabic-Clip/Araclip-Simplified
π website: https://arabic-clip.github.io/Arabic-CLIP/
- 2 replies
Post
4502
I have just released a new blogpost about kv caching and its role in inference speedup π
π https://huggingface.co/blog/not-lain/kv-caching/
some takeaways :
π https://huggingface.co/blog/not-lain/kv-caching/
some takeaways :
-
- 4 replies
Post
1777
we now have more than 2000 public AI models using ModelHubMixinπ€
Post
4124
Published a new blogpost π
In this blogpost I have gone through the transformers' architecture emphasizing how shapes propagate throughout each layer.
π https://huggingface.co/blog/not-lain/tensor-dims
some interesting takeaways :
In this blogpost I have gone through the transformers' architecture emphasizing how shapes propagate throughout each layer.
π https://huggingface.co/blog/not-lain/tensor-dims
some interesting takeaways :
Post
2194
Welcome back,
Small Language Models Enthusiasts and GPU Poor oss enjoyers lets connect.
Just created an organization which main target is to have fun with smaller models tuneable on consumer range GPUs, feel free to join and lets have some fun, much love ;3
SmolTuners
Small Language Models Enthusiasts and GPU Poor oss enjoyers lets connect.
Just created an organization which main target is to have fun with smaller models tuneable on consumer range GPUs, feel free to join and lets have some fun, much love ;3
SmolTuners
- 3 replies
Post
2338
great blogpost! π₯@wolfram
https://huggingface.co/blog/wolfram/llm-comparison-test-2024-12-04
https://huggingface.co/blog/wolfram/llm-comparison-test-2024-12-04
Post
2422
ever wondered how you can make an API call to a visual-question-answering model without sending an image url π
you can do that by converting your local image to base64 and sending it to the API.
recently I made some changes to my library "loadimg" that allows you to make converting images to base64 a breeze.
π https://github.com/not-lain/loadimg
API request example π οΈ:
you can do that by converting your local image to base64 and sending it to the API.
recently I made some changes to my library "loadimg" that allows you to make converting images to base64 a breeze.
π https://github.com/not-lain/loadimg
API request example π οΈ:
from loadimg import load_img
from huggingface_hub import InferenceClient
# or load a local image
my_b64_img = load_img(imgPath_url_pillow_or_numpy ,output_type="base64" )
client = InferenceClient(api_key="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
messages = [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image in one sentence."
},
{
"type": "image_url",
"image_url": {
"url": my_b64_img # base64 allows using images without uploading them to the web
}
}
]
}
]
stream = client.chat.completions.create(
model="meta-llama/Llama-3.2-11B-Vision-Instruct",
messages=messages,
max_tokens=500,
stream=True
)
for chunk in stream:
print(chunk.choices[0].delta.content, end="")Post
1814
@Blane187
could you please modify the title of your blogpost? content is cool, title could be nicer imo https://huggingface.co/blog/Blane187/wtf-is-rvc
- 3 replies
Post
6914
π₯ New state of the art model for background removal is out
π€ You can try the model at ZhengPeng7/BiRefNet
π model shows impressive results outperforming briaai/RMBG-1.4
π you can try out the model in: ZhengPeng7/BiRefNet_demo
πpaper: Bilateral Reference for High-Resolution Dichotomous Image Segmentation (2401.03407)
π€ You can try the model at ZhengPeng7/BiRefNet
π model shows impressive results outperforming briaai/RMBG-1.4
π you can try out the model in: ZhengPeng7/BiRefNet_demo
πpaper: Bilateral Reference for High-Resolution Dichotomous Image Segmentation (2401.03407)
- 1 reply
Post
1969
Cool things this week from
@huggingface
!
πAI math olympiad winner NuminaMath is here!
π€Announcing New Hugging Face and Keras NLP integration
β¨UI overhaul to HF tokens!
π§ Embed our dataset viewer on any webpage!
https://huggingface.co/blog/winning-aimo-progress-prize
https://huggingface.co/blog/keras-nlp-integration
https://huggingface.co/settings/tokens
https://x.com/julien_c/status/1812099420726456457
Check out the full list on our discord! π
https://discord.com/invite/JfAtkvEtRb
πAI math olympiad winner NuminaMath is here!
π€Announcing New Hugging Face and Keras NLP integration
β¨UI overhaul to HF tokens!
π§ Embed our dataset viewer on any webpage!
https://huggingface.co/blog/winning-aimo-progress-prize
https://huggingface.co/blog/keras-nlp-integration
https://huggingface.co/settings/tokens
https://x.com/julien_c/status/1812099420726456457
Check out the full list on our discord! π
https://discord.com/invite/JfAtkvEtRb
Post
7805
I am now a huggingface fellow π₯³
-
- 15 replies
Post
2714
I have finished writing a blogpost about building an image-based retrieval system, This is one of the first-ever approaches to building such a pipeline using only open-source models/libraries π€
You can checkout the blogpost in https://huggingface.co/blog/not-lain/image-retriever and the associated space at not-lain/image-retriever .
β¨ If you want to request another blog post consider letting me know down below or you can reach out to me through any of my social media
π Happy reading !
You can checkout the blogpost in https://huggingface.co/blog/not-lain/image-retriever and the associated space at not-lain/image-retriever .
β¨ If you want to request another blog post consider letting me know down below or you can reach out to me through any of my social media
π Happy reading !
Post
1482
Hello beautiful people.
I wanted to thank everyone that read my blogpost and I am glad to share that we have achieved 11000 readers π₯³
I couldn't have done this without you, so once again thanks a lot everyone for the support π
If you haven't already you can read my blog post at: https://huggingface.co/blog/not-lain/rag-chatbot-using-llama3
I wanted to thank everyone that read my blogpost and I am glad to share that we have achieved 11000 readers π₯³
I couldn't have done this without you, so once again thanks a lot everyone for the support π
If you haven't already you can read my blog post at: https://huggingface.co/blog/not-lain/rag-chatbot-using-llama3
Post
2136
It is with great pleasure I inform you that huggingface's ModelHubMixin reached 200+ models on the hub π₯³
ModelHubMixin is a class developed by HF to integrate AI models with the hub with ease and it comes with 3 methods :
* save_pretrained
* from_pretrained
* push_to_hub
Shoutout to @nielsr , @Wauplin and everyone else on HF for their awesome work π€
If you are not familiar with ModelHubMixin and you are looking for extra resources you might consider :
* docs: https://huggingface.co/docs/huggingface_hub/main/en/package_reference/mixins
πblog about training models with the trainer API and using ModelHubMixin: https://huggingface.co/blog/not-lain/trainer-api-and-mixin-classes
πGitHub repo with pip integration: https://github.com/not-lain/PyTorchModelHubMixin-template
πbasic guide: https://huggingface.co/posts/not-lain/884273241241808
ModelHubMixin is a class developed by HF to integrate AI models with the hub with ease and it comes with 3 methods :
* save_pretrained
* from_pretrained
* push_to_hub
Shoutout to @nielsr , @Wauplin and everyone else on HF for their awesome work π€
If you are not familiar with ModelHubMixin and you are looking for extra resources you might consider :
* docs: https://huggingface.co/docs/huggingface_hub/main/en/package_reference/mixins
πblog about training models with the trainer API and using ModelHubMixin: https://huggingface.co/blog/not-lain/trainer-api-and-mixin-classes
πGitHub repo with pip integration: https://github.com/not-lain/PyTorchModelHubMixin-template
πbasic guide: https://huggingface.co/posts/not-lain/884273241241808
Post
2376
By popular demand, HF activity tracker v1.0 is here! π let's build it together!π€
Lots of things to improve, feel free to open PRs in the community tab!
good PR ideas:
- track more types of actions that include date+time
- bigger plot
- track discord activity too π€―
- link github? β‘
https://huggingface.co/spaces/huggingface-projects/LevelBot
Lots of things to improve, feel free to open PRs in the community tab!
good PR ideas:
- track more types of actions that include date+time
- bigger plot
- track discord activity too π€―
- link github? β‘
https://huggingface.co/spaces/huggingface-projects/LevelBot
- 2 replies