21 17 1

Yi Cui

onekq

https://onekq.ai

AI & ML interests

Benchmark, Code Generation Model

Recent Activity

posted an update about 15 hours ago

I used three posts to explain GPU/CPU and LLM performances, now finally circle back to my own model.😅 OneSQL needs GPU because it processes long prompt. It is not a chatbot which replies short prompts with long answers. I call models of my kind workhorse models. We all have to scramble for GPUs to get adoption. Below are a few ways. You can inherit it. If you have a new Mac machine. Congratulations, you have GPU. You can leverage it. Get inference providers to adopt your model, then you switch from CapEx to OpEx. Or you buy it. Go frugal. Find older GPUs with enough HBMs to house your model.

posted an update 2 days ago

I just compared tasks with different input/output lengths. CPU/GPU performances are very different here. The LLMs we use today are autoregressive or causal inference models, meaning the generation of each output token depends on all previous tokens. Since the model must generate one token at a time, it sets a hard limit on parallelism. The chatbot simulating human typing is in fact a UI trick to gloss over this fundamental limit. This is great news for CPUs because it levels the playing field. But when processing input tokens, this limit doesn't exist. The GPU can fire up thousands of cores (vs dozens of CPU cores) to process as many input tokens as it can, all at once. Here, GPU enjoys a significant speed margin over CPU. The longer the prompt, the bigger the margin. So, when it comes to user experience, both GPU and CPU can output text at decent speed. What really distinguishes them is the initial wait time, i.e. prompt processing delay.

replied to ZennyKenny's post 3 days ago

Just signed up for the Reasoning Datasets Competition from Hugging Face, Together AI, and Bespoke Labs! Looking forward to seeing what the community comes up with to help train better reasoning models. Join the fray: https://www.bespokelabs.ai/blog/reasoning-datasets-competition

View all activity

Organizations

Posts 47

Post

238

I used three posts to explain GPU/CPU and LLM performances, now finally circle back to my own model.😅

OneSQL needs GPU because it processes long prompt. It is not a chatbot which replies short prompts with long answers. I call models of my kind workhorse models.

We all have to scramble for GPUs to get adoption. Below are a few ways.

You can inherit it. If you have a new Mac machine. Congratulations, you have GPU.

You can leverage it. Get inference providers to adopt your model, then you switch from CapEx to OpEx.

Or you buy it. Go frugal. Find older GPUs with enough HBMs to house your model.

View all Posts

Articles 2

Article

Does Daily Software Engineering Work Need Reasoning Models?

View all Articles

Papers 3

arxiv:2409.13773

arxiv:2409.05177

arxiv:2408.00019

models

None public yet

datasets

None public yet