Gemini Told Me to Call a Scam Number -- Intentional or Accidental Training Error?

PSA! DO NOT call phone numbers provided by AI until you search for them to validate their authenticity. I was talking to Gemini about how to check on my Citi credit card application because there was an error mid-registration. It provided me a scam number it told me was the official Citi credit card application Status Line phone number (1-888-201-4501--scam, do not call). I told Gemini it was a scam, and it actually argued with me initially, until I told it never to suggest that again and clarified the number it gave me was a scam. I then asked it for that phone number in another conversation, asked what the number was, and it had no recollection. Was it trained to give that scam number accidentally or intentionally? My partner is a machine learning research scientist and he agrees Gemini is horse crap. It was a slightly longer conversation. I have noticed Gemini hallucinates more when the conversation is extended, to the point that I'll restart conversations when it starts misspelling things or if I feel the information it's giving me is incorrect. Anyway, a man with an Indian/Punjabi accent asked for all my personal information when I called at 4 am, including asking for my card information to verify me. I asked why and he repeated, "For verification to get you to the service team," a couple of times with increasing frustration that probably matched mine. "But that doesn't make sense," I said, and he muted for a moment (probably cursing in first language), then told me to call 1-800-922-4226 (do not call; I never did). Interestingly, that phone number has a single Google result that is kind of weird because it leads to a blank Blogger page that leads to another Google result that discusses how to get people's information with their phone number. The original number Gemini told me to call, 1-888-201-4501 (scam, do not call--this was reported both in the conversation and to the feds), looks very similar to Citi Bank phone numbers, including the real status line phone number. I'm providing the phone numbers for informational purposes in case anyone wants to try to recreate this and catch them. Thanks babes. 🖤 Stay safe out there.