@FakeMaidenMaker: MIT just open-sourced an inference library that lets large models read tens of millions of tokens at once—RLM from MIT CSAIL's OASYS lab, with involvement from DSPy and ColBERT author Omar Khattab, even VentureBeat covered it…

MIT just open-sourced an inference library that lets large models read tens of millions of tokens at once—RLM From MIT CSAIL's OASYS lab, with involvement from DSPy and ColBERT author Omar Khattab, even VentureBeat covered it. It quickly reached 5k stars soon after open-sourcing. Ordinary models have a fatal flaw when processing ultra-long texts: A book, an entire codebase, or hundreds of pages of reports either can't fit into the context window, or the model forgets earlier parts when reading later ones, ultimately giving you a blurry answer. RLM takes a completely different approach. Instead of expanding the window, it lets the model treat the entire context as variables at hand, writing its own code to chunk, grep, and recursively call sub-models to look up information—much like a programmer debugging in a REPL, rather than memorizing all the code at once. Paper data shows it can process inputs one or two orders of magnitude beyond the model's native context window, and even in short-context scenarios, its quality significantly surpasses that of ordinary cutting-edge models. Integration is simple: replace llm.completion() with rlm.completion() and install via pip install rlms. The limit of the context window has never been the model's boundary, but the tool's edge.