Now the bottleneck is compression / memory (and a little on input filtering)

Long rambling post ahead, 100% typed by hand for those who care, not even reformatted. A few weeks back I said vision was one of the most major bottlenecks, shortly after Astra dropped and blew many of us away, I'm biased but imo it's no coincidence that it's far above 5.6 models in it's visual intelligence. Computers are "better" at, short, medium, long term memory than humans. Well no, in almost every way they're better at short term memory, but for medium and long term they're not, in terms of accuracy yes but not "cost". "Compaction" is a crutch as are 258k token context windows. Humans won't often remember a new phone number they're told 10 seconds ago but can remember in a 10 million token sized codebase they've worked on for 10 years that the niche error for android users on specifically major versions 9-10 inclusive is down to the approximate 20% of the file. Human compression / input filtering / memory is considerably less "taxing" than for an LLM and still manages to be "better" in practice. If you're relaxing watching TV, zoning out, barely paying attention, then forced to rewatch the same episode, it'll feel boring because of how familiar it is, despite putting zero work in you are still managing to absorb so much for so cheap, I'm convinced for text this issue is very solvable and will unlock extremely long horizon cheap quality work as Astra if it could "hold". We don't store our entire memory up front, but we're also far more efficient at recalling it. Imo the analogy best suited to what LLMs should do imo to better mimic the good parts of a human whilst retaining the upsides of an LLM: Treat context as short term memory only, still 100x larger than a human's but not hundreds if not thousands of times larger. imo 10k context window is more than sufficient for ASI, and yes I did pull that number out of my arse (kinda, humans are NGI, and imo our equivalent of a context window is an equivalent of 1k tokens if being VERY generous). Instead rely more on cheap "tool" calls for 1. compression 2. input filtering 3. memory Analogies of current LLM parts and support, biggest problem areas (imo) in bold: context window ~= human uncompressed + compressed short term memory, e.g. a new phone number, the last 5 minutes of a movie, etc. maybe I'll be wrong but LLMs can handle so much more uncompressed "context window" than us I don't believe it's necessary to compress yet, it's just too cheap to worry and the later we compress the more relevant the compression should be in theory, it's a balancing act and since context is ahead and compression is behind I think it's fine that their balance is totally different from humans, the idea isn't to attempt to copy humans 1:1 just use them as inspiration to overcome limitations. LLM CoT ~= human CoT, no they're not identical but sufficiently close not to bloat and already long post. LLM output ~= human output, same deal, they're both just output tokens really, much like us whether we speak or think, for most people (most people have an inner monologue) at least. Conversation compression != compaction, compaction is a a crutch, inferior to what should be used Tool calls, grep, fuzzy searches, semantic searches, around, awk, etc ~= skimming, humans can cheaply skim files and these tools aren't terrible substitutes but could do meaningfully better with more research imo source files != long term memory, just reading their names and looking around them is a crutch code-map.md != long term memory, a hacky work around I believe if done correctly, a latent compression of your entire project, all your project rules, all your global rules, a billion skills, etc. can be "in" context (rather than actually all IN context at once, it's more like the lookup is there, and it's easy to "pluck" the necessary compressed info out which can then either be directly useful or point you at the right place to look instantly). Not another Jev post but I believe tools like Jev can actually help close a gap. Cheaply deciding what's "tags" are important for a problem and then tools can automatically update context as they're needed and flush them once they stop being needed. Using "tags" (maybe in latent space) and automatic tool based context population when necessary I genuinely believe you could cheaply get extreme competency with a small context window (including the currently loaded latent knowledge). I didn't discuss input filtering much, but we know uncached input reads aren't free, imo with compression/memory "solved" this is comparatively much easier, rather than reading a whole logs file a semantic search for "fail|error|issue" using a cheap tool and not including anything not needed in context going forward is already on the cusp of being good enough, it's just occasionally very bad (had cursor do some commands and idk what it was smoking but despite doing almost nothing it read 14 million tokens in one chat and blew $30 in credits, clearly at least cursor is not properly guarding LLMs from console output!) so I brought it up less as a true bottleneck but a common pitfall where even the industry leaders have room to improve. Basically it boils down to: SQL is dumb but cheap and fast, LLMs are expensive and slow but smart, try and leverage more towards SQL like things (or literally sometimes SQL) for when they're suitable. But rather than plugins, frontier labs should do it. imo they should genuinely challenge themselves to have a relatively tiny context window to now and make it work just as well as now. Jev sits above SQL ofc, smarter but as cheap and fast as it is, extremely slow and expensive compared to SQL for an SQL appropriate task. Humans are living proof that it's possible to acquire and recall long term memories cheaply and relevantly, you can actively try not remember something and be unable to not remember it (I'd successfully gone a few months without thinking about a video I saw of someone dying, and writing that made me think of it again, a sad but relevant example my brain "plucked" from itself "effortlessly"). And once it's cracked, I believe they will easily surpass humans because our fleshy brains are incredibly unreliable compared to bits. Imo it's borderline a solution to continual learning because if done correctly then it's possible to store every skill ever made "on" every worker, doesn't mean it's constantly adding expense to context, but when it needs it the skill will automatically enter context, meaning instead of reinventing the wheel like it does now, it applies a solution it previously found. Privacy is a concern, as is jailbreaking, so sadly probably can't just allow skills from users to reach other users all willy nilly, but conceptually you could and it'd mean that the LLM would seemingly get smarter all the time, instead of figuring out how to do X again when the 1000th person asks it to do it, it'd cheaply, quickly, and accurately recall a skill "made" for this exact purpose. Not quite continual learning though, in the sense that it can't truly get more "intelligent" but imo a human can (barely, and extremely slowly). But replacing 50% of white collar jobs with just Astra level intelligence if unlimited cheap fast "lookup" is solved seems feasible. And I mean in a way where it's basically automating the replacement as well, where as currently if you wanted to replace all those workers you'd need people to do the work of harnesses and teaching the LLMs about the company, etc. Instead it could just be installed on all computers at the company, soak up what it needs, and automatically detect when it can do the same thing at least as well but ofc faster.