Are we getting way too comfortable with AI remembering everything we tell it?

I understand why AI memory is useful having an assistant remember your projects, preferences and previous conversations makes it significantly better than starting from zero every time. But I was thinking about how quickly that can escalate. Give it access to years of chats, then your email, calendar, documents, maybe eventually messages from other apps and now you've built something that potentially knows more about your life than any individual person does. And what I think is really weird is that I don't think most people would ever sit down and intentionally create a database containing all of that information about themselves. We just slowly give AI access to it because each individual permission is convenient. I'm not anti AI I use it constantly but I'm just wondering where everyone personally draws the line between having useful context and giving one system way too much context. Do you actually want your AI assistant to eventually remember basically everything or is there certain stuff you'll always keep separate?