@sdianahu: 1/ fast AI inference is about to replay the history lesson from search engines on why low latency is so important

1/ fast AI inference is about to replay the history lesson from search engines on why low latency is so important

2/ in 2009 google ran an experiment by slowing search down by 400ms, and searches per user dropped 0.59%, and the damage persisted for weeks after they removed the delay

bing saw the same thing , a 2 sec delay cost 1.8% of queries and 4.3% of revenue per user

amazon found every 100ms cost ~1% of sales

3/ now look at time-to-first-token (TTFT): reasoning models ~4s!! frontier labs at 820ms-1.2s! cerebras ~120-160ms, groq ~150ms

you don’t patch your way across that via just clever software to fix it; it’s a bet you make in silicon years ahead

4/ (a side note the theoretical latency limit is in physics

light in fiber moves at ~2/3 the speed of light nyc <-> london is already ~56ms before any compute happens)

5/ search engines can hide compute by copying/caching the index everywhere. AI inference is harder to cache, so the options are: replicate models at the edge, run them on device, custom silicon

6/ here’s where the search analogy breaks though, quality still dominates right now labs compete almost entirely on quality, and users put up with slowness because the answer is worth the wait (deliberation feels like intelligence)

7/ but OSS models are not that far behind we’ll end up with clusters of models that are basically equally good

the moment the answers are interchangeable, the only thing left to compete on is speed

perhaps via custom HW

https://services.google.com/fh/files/blogs/google_delayexp.pdf…

https://niels-ole.com/amazon/performance/2018/10/27/100ms-latency-1percent-revenue.html…

https://cambridge.org/core/books/abs/trustworthy-online-controlled-experiments/speed-matters/35D34F276D4AF1AEE4B98D3E09B9E168…

Hey @sdianahu - just saw this - building something of raw intuition Will get back soon!