Infra and AI

I don’t really want to use the term “AI infra” – it sounds like just repackaging compute, storage, and networking with a fresh coat of paint. But recently, using Fable 5 to pedal cocoon/sandbox really made me feel, man, times have changed.

I’ve wanted to write about my experiences developing cocoonstack for a long time – how to coordinate with Opus 4.6~4.8 and now Fable 5, how to keep the project moving forward while maintaining a certain level of taste (without ending up as a pile of shit like Openclaw or some Toutiao system), and how to keep the human in the loop. But of course, I’m lazy, so I kept putting it off until now. I’ve been using cocoon to build a new project called sandbox, pedaled Fable 5 for a few days, and figured it’s a good time to share my views on infra and AI.

First, let me tell a story. In 2012, Python 2.7 was still mainstream. I was at Douban’s platform team working on DAE under the guidance of Professor. Douban did something interesting: bypassing the infamous Python GIL using Gevent’s monkey patch and our own greenify – both at the Python level and C level (I know you want to complain about the pros and cons of implicit async, but that’s not the point). At that time, we had to make a decision about Redis connections: should we make it asynchronous like MySQL connections or keep it synchronous (patch or no patch)?

The final conclusion was: no patch. The reason is simple: maintaining epoll itself has a cost. Don’t overthink it – no matter which layer has a loop polling file descriptors, there’s a loop, right? MySQL deserves it because DB operations are expensive, IO wait times are long, and a single request could block for tens to hundreds of milliseconds, leaving the business layer waiting. Redis/Memcached are lightning fast, and the overhead of your black magic might be greater than just letting the GIL handle synchronous requests – a total negative optimization. So at that time, if you looked at Douban’s business systems, concurrency and async (bypassing the GIL) were everywhere, but the cache remained on the traditional synchronous stack.

Later, when interviewing people, I often asked: you have several heavy-IO file descriptors, tell me when to choose select vs epoll under different circumstances. But sadly, after years of interviewing, almost no one could answer well. You can’t say select is useless, right?

So, does AI’s increasing power impact infrastructure engineers? A little, but not much. AI’s impact on infra isn’t that “people who can’t code can now write infra.” It’s that “the execution power of people who can design gets amplified.”

The essence of an infra “iron rice bowl” is trade-offs and taking the blame – exactly what AI isn’t good at. Some on Twitter say trade-offs can be expressed with objective functions, but the problem is: who defines the objective function? Latency, cost, stability, maintainability, migration risk, incident responsibility, team capability – these weights are never a pure math problem. They’re an organizational problem, a matter of human dynamics. So in the end, it has to be human in the loop. The current models from the top two companies have clearly done massive optimization on parameter scale and post-training, but this brings a problem: the data overfits. Simply put, token prediction is like a greedy algorithm. In most cases, predicting the next token by picking the highest probability is fine. But when you hit scenarios requiring strategy and trade-offs, greedy algorithms being too greedy isn’t the best choice – because everything has a cost, my friend.

A simple example: look at e2b.dev or fly.io – they use UFFD and FUSE lazy loading for fast process startup. Similarly, open-source sandboxes on GitHub use similar techniques for “startup speed.” So what’s the cost? When I was building cocoon/sandbox, I found that using UFFD for cloud-hypervisor restore caused Windows resume from freeze to usable state to become very slow. It looks like the process boots up in a flash, device simulation is blazing fast, but the system isn’t usable. So what’s the point of that speed, right?

It seems CH’s restore dropped from ~150ms to 70-80ms, but the system resume time went from 1s straight to 7s. Depending on your OS, Linux behavior differs from Windows, and large-memory machines differ from small ones. That means when designing and implementing clone/restore, you have to make trade-offs. Even as strong as Fable, it doesn’t handle this well – it still needs human input.

After all, how much “shit” has a model actually eaten? Hahaha. Us old racks have eaten way more.

But on the other hand – execution and point analysis – I have to say the evolution of models is visibly improving. Used well, they can greatly expand your capability range, quickly implementing many things you only thought about but were too lazy to do.

Opus 4.6 ~ Opus 4.8 is roughly at the level of Toutiao 1-2 to barely 2-2. Currently, the domestic Qwen 27B is also around a strong 1-2. The later parts of cocoon were mostly patched up with Opus – a bit tiring. But this Fable 5, with xhigh enabled, a 3-1 executor truly comes online. For a project like sandbox, I’m familiar with Go/Python. It extracted my coding habits, and in every commit iteration, it self-checks and writes quite cleanly. I’m not saying it’s 100% perfect, but at least it’s readable, functional, and fully tested. For Rust, I don’t have much experience, but a quick glance shows it’s stable and steady. Honestly, if I had to do it myself, it would take at least ten days or two weeks. And I only contributed detailed design, my own hard-earned experience, and my coding habits…

This brings me back to what I said in my previous post: veterans are really partying, but they will eventually wither. The iron rice bowl may still be iron, but I’m afraid there will only be bowls left, no people. What’s there to say? Time to pedal Fable and fulfill childhood dreams, haha.