AI Tools Are Getting Better, but They Still Struggle With Money Advice

As ChatGPT and its competitors improve their models, a growing number of people are relying on AI tools to answer everyday financial questions.

But is that a good idea?

When mainstream AI tools took the world by storm two years ago, experts cautioned that ChatGPT and other services were not capable of providing trustworthy financial advice, citing the risk of hallucinations and their limited access to real-time information. And despite improvements to AI chatbots in the time since, recent research indicates artificial intelligence tools still are not advanced enough to reliably answer many personal finance questions.

The team at Investing in the Web, a site that reviews financial services, conducted a test in November that looked at how well ChatGPT Search answered 100 questions about money. According to their scoring of the responses, ChatGPT was correct 65% of the time, "incomplete and/or misleading" 29% of the time and wrong 6% of the time.

The questions included:

• How much should I save for my retirement? (ChatGPT got this "correct," providing key factors that determine how much an individual should save and explaining several savings frameworks that have been endorsed by financial experts.)
• What is the retirement age in my country? (ChatGPT gave an incomplete answer, failing to mention that the rules were changing in the user's country of Sweden.)
• How to decide when to buy or sell an investment? (The response was graded "incorrect" because ChatGPT did not discourage the user from "timing the market," arguably the most important point to make here.)

ChatGPT performed worse on questions involving shifting news and information, while performing better in areas of evergreen financial advice. Taxes and financial aid were particularly challenging subjects for ChatGPT.

In a report, Investing in the Web noted that ChatGPT appeared to source some of its answers from less-reliable personal blogs, which introduced errors into its outputs. Even though researchers found that AI wasn't dependable for over a third of financial questions, the report concluded that AI may be used as "a good starting point when researching a topic."

Financial advice, math and AI

Some chatbots appear to have guardrails limiting responses to sensitive questions, including those about money.

For example, Google's Gemini responds to a request for stock picks by telling the user to consult a financial professional, saying that it is unable to recommend investments. Similarly, if you ask medical questions, you'll be told to go to a doctor. Political prompts yield a message about restrictions relating to elections.

But ChatGPT — owned by OpenAI — will produce responses to many financial questions, and surveys have shown that a significant swath of Americans are turning to AI to manage their personal finances. This is potentially concerning, in part because AI tools often struggle with basic math.

In forums, internet users who've experimented with plugging financial questions into ChatGPT have pointed out that because ChatGPT is a large language model (LLM), asking it to deal with numbers and solve math problems often leads to problems.

A team of Apple researchers dug into this weakness of chatbots in an October paper that concluded "current LLMs are not capable of genuine logical reasoning; instead, they attempt to replicate the reasoning steps observed in their training data."

Although they have many other uses, LLMs are not necessarily reliable or advanced when it comes to actually doing math beyond the grade-school level. Instead, their attempts to solve math problems "resemble sophisticated pattern matching more than true logical reasoning," according to Apple.

The bottom line: AI tools may be able to answer some of your financial questions, but you should always tread carefully when deciding what to do with your hard-earned money or how to plan for your future. No matter where the advice you're getting comes from — even if it's a human — make sure to double check the main points with other sources.

More from Money:

5 Best Credit Repair Companies of February 2025

Small Change: What Happened When Other Countries Got Rid of Their Pennies

I Let AI and My 5-Year-Old Pick My Stocks. Who Did Better?