I spent the last few weeks building a voice assistant that actually runs my Windows PC. Here's what surprised me.

I wanted something closer to the movie version of an AI assistant: say "Hey Milo, open my resume" or "set a 10 minute pasta timer" and have it just happen, without me opening a chat window. So I built one in Python (Gemini for the thinking, pywebview for the interface, a folder of plugins for everything it can do). A few things I didn't expect: Opening apps is the hardest "easy" feature. "Open word" sounds trivial until you see how many shapes Windows apps come in: Store apps, desktop programs, shortcuts, OneDrive moved folders. I ended up building an index from Windows' own Start menu list and hazed matching against it. Being interruptible matters more than sounding good. The first version talked over me. Letting the mic listen while it speaks, just for "stop" or its name, made it feel 10x more like a real assistant. Startup time decides whether you keep using it. Loading everything before showing the window felt broken. I moved all the loading to a background thread and show a loading screen with a real progress checklist instead, and it feels way faster even though it isn't. Free API tiers are tiny once an assistant is "always on." Every reminder, search and follow-up is a request, and the free limit went fast. Anything that touches money needs hard limits the AI can't talk its way around. It's hooked into my trading platform, so there's a max size it can never exceed, anything above a smaller size waits for me to say yes, and once I hit a daily loss limit it can't open anything new until tomorrow. For people who've built (or tried to build) something like this: what's the one feature that made you actually keep using it day to day? And what would you never let an assistant do on its own?