Francesco Scheffczyk

fscheffczyk

https://www.francesco-scheffczyk.de/

francesco-sch

AI & ML interests

None yet

Recent Activity

liked a Space 27 days ago

Trudy/gemini-image-to-code

updated a Space 4 months ago

fscheffczyk/uncanny-type-gradio

reacted to akhaliq's post with ❤️ 11 months ago

Ferret-UI Grounded Mobile UI Understanding with Multimodal LLMs https://huggingface.co/papers/2404.05719 Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest (e.g., icons, texts) than natural images, we incorporate "any resolution" on top of Ferret to magnify details and leverage enhanced visual features. Specifically, each screen is divided into 2 sub-images based on the original aspect ratio (i.e., horizontal division for portrait screens and vertical division for landscape screens). Both sub-images are encoded separately before being sent to LLMs. We meticulously gather training samples from an extensive range of elementary UI tasks, such as icon recognition, find text, and widget listing. These samples are formatted for instruction-following with region annotations to facilitate precise referring and grounding. To augment the model's reasoning ability, we further compile a dataset for advanced tasks, including detailed description, perception/interaction conversations, and function inference. After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions. For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks. Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.

View all activity

Organizations

fscheffczyk's activity

liked a Space 27 days ago

Gemini Image to Code

🌠

liked 2 Spaces about 1 year ago

12.6k

Open LLM Leaderboard

🏆

Track, rank and evaluate open LLMs and chatbots

Real-Time Latent Consistency Model ControlNet-Lora-SD1.5

🖼

liked a model over 1 year ago

jinaai/jina-embeddings-v2-base-en

Feature Extraction • Updated Jan 6 • 277k • • 712

liked a dataset over 1 year ago

cerebras/SlimPajama-627B

Preview • Updated Jul 7, 2023 • 70.9k • 453

liked a model almost 2 years ago

prompthero/openjourney

Text-to-Image • Updated May 15, 2023 • 22.3k • • 3.12k

liked 4 datasets almost 2 years ago

liked 3 datasets about 2 years ago

fka/awesome-chatgpt-prompts

Viewer • Updated Jan 6 • 203 • 11.7k • 7.55k

abisee/cnn_dailymail

Viewer • Updated Jan 18, 2024 • 936k • 67.5k • 241

defunct-datasets/the_pile_openwebtext2

Updated Nov 27, 2023 • 135 • 15

liked a model about 2 years ago

facebook/bart-large-cnn

Summarization • Updated Feb 13, 2024 • 4.45M • • 1.29k

liked a model almost 3 years ago

gsurma/ai_dreamer

Text-to-Image • Updated Feb 12, 2022 • 15 • 4

liked 2 Spaces about 3 years ago

178

VQGAN CLIP

💩

Emotion classification in tweets

🤡

liked a model about 3 years ago

joeddav/distilbert-base-uncased-go-emotions-student

Text Classification • Updated Feb 19, 2021 • 12k • • 73