Rebuilding my custom agentic AI because the original architecture was too heavy

I’ve decided to rebuild a custom agentic AI I’ve been working on. The previous architecture became too heavy for the kind of system I actually want to build. Instead of solving that by throwing more compute at it, I decided to rethink the architecture from the ground up. One of the biggest changes is that I removed the call to a pre-made LLM. I’m now working on building my own intelligence layer instead of relying on an external LLM API as the core of the agent. The goal is to have a lightweight agent that can develop capabilities such as: Perception Memory and knowledge Decision-making Learning from data Behavioral adaptation Interaction with its environment Taking actions rather than simply generating text I’m not trying to claim that I’ve built a replacement for today’s large LLMs. This is still a work in progress, and rebuilding the architecture means I’m essentially experimenting with the fundamentals again. The interesting challenge for me is seeing how much agentic behavior can be achieved with a much smaller and more efficient architecture, rather than simply increasing model size and compute. I’m curious what others here think: when building an agent from the ground up, how much of the intelligence actually needs to come from an LLM?