GPT and other AI models can't analyze an SEC filing, researchers find

Patronus AI co-founders Anand Kannappan and Rebecca Qian

Patronus AI

Large language models, similar to the one at the heart of ChatGPT, frequently fail to answer questions derived from Securities and Exchange Commission filings, researchers from a startup called Patronus AI found.

Even the best-performing artificial intelligence model configuration they tested, OpenAI's GPT-4-Turbo, when armed with the ability to read nearly an entire filing alongside the question, only got 79% of answers right on Patronus AI's new test, the company's founders told CNBC.

Oftentimes, the so-called large language models would refuse to answer, or would "hallucinate" figures and facts that weren't in the SEC filings.

"That type of performance rate is just absolutely unacceptable," Patronus AI co-founder Anand Kannappan said. "It has to be much much higher for it to really work in an automated and production-ready way."

The findings highlight some of the challenges facing AI models as big companies, especially in regulated industries like finance, seek to incorporate cutting-edge technology into their operations, whether for customer service or research.

The ability to extract important numbers quickly and perform analysis on financial narratives has been seen as one of the most promising applications for chatbots since ChatGPT was released late last year. SEC filings are filled with important data, and if a bot could accurately summarize them or quickly answer questions about what's in them, it could give the user a leg up in the competitive financial industry.

In the past year, Bloomberg LP developed its own AI model for financial data, business school professors researched whether ChatGPT can parse financial headlines, and JPMorgan is working on an AI-powered automated investing tool, CNBC previously reported. Generative AI could boost the banking industry by trillions of dollars per year, a recent McKinsey forecast said.

But GPT's entry into the industry hasn't been smooth. When Microsoft first launched its Bing Chat using OpenAI's GPT, one of its primary examples was using the chatbot to quickly summarize an earnings press release. Observers quickly realized that the numbers in Microsoft's example were off, and some numbers were entirely made up.

Part of the challenge when incorporating LLMs into actual products, say the Patronus AI co-founders, is that LLMs are nondeterministic — they're not guaranteed to produce the same output every time for the same input. That means that companies will need to do more rigorous testing to make sure they're operating correctly, not going off-topic, and providing reliable results.

The founders met at Facebook parent company Meta, where they worked on AI problems related to understanding how models come up with their answers and making them more "responsible." They founded Patronus AI, which has received seed funding from Lightspeed Venture Partners, to automate LLM testing with software, so companies can feel comfortable that their AI bots won't surprise customers or workers with off-topic or wrong answers.

"Right now evaluation is largely manual. It feels like just testing by inspection," Patronus AI co-founder Rebecca Qian said. "One company told us it was 'vibe checks.'"

Patronus AI worked to write a set of more than 10,000 questions and answers drawn from SEC filings from major publicly traded companies, which it calls FinanceBench. The dataset includes the correct answers, and also where exactly in any given filing to find them. Not all of the answers can be pulled directly from the text, and some questions require light math or reasoning.

Qian and Kannappan say it's a test that gives a "minimum performance standard" for language AI in the financial sector.

Here's some examples of questions in the dataset, provided by Patronus AI:

Patronus AI tested four language models: OpenAI's GPT-4 and GPT-4-Turbo, Anthropic's Claude 2 and Meta's Llama 2, using a subset of 150 of the questions it had produced.

It also tested different configurations and prompts, such as one setting where the OpenAI models were given the exact relevant source text in the question, which it called "Oracle" mode. In other tests, the models were told where the underlying SEC documents would be stored, or given "long context," which meant including nearly an entire SEC filing alongside the question in the prompt.

GPT-4-Turbo failed at the startup's "closed book" test, where it wasn't given access to any SEC source document. It failed to answer 88% of the 150 questions it was asked, and only produced a correct answer 14 times.

It was able to improve significantly when given access to the underlying filings. In "Oracle" mode, where it was pointed to the exact text for the answer, GPT-4-Turbo answered the question correctly 85% of the time, but still produced an incorrect answer 15% of the time.

But that's an unrealistic test because it requires human input to find the exact pertinent place in the filing — the exact task that many hope that language models can address.

Llama 2, an open-source AI model developed by Meta, had some of the worst "hallucinations," producing wrong answers as much as 70% of the time, and correct answers only 19% of the time, when given access to an array of underlying documents.

Anthropic's Claude 2 performed well when given "long context," where nearly the entire relevant SEC filing was included along with the question. It could answer 75% of the questions it was posed, gave the wrong answer for 21%, and failed to answer only 3%. GPT-4-Turbo also did well with long context, answering 79% of the questions correctly, and giving the wrong answer for 17% of them.

After running the tests, the co-founders were surprised about how poorly the models did — even when they were pointed to where the answers were.

"One surprising thing was just how often models refused to answer," said Qian. "The refusal rate is really high, even when the answer is within the context and a human would be able to answer it."

Even when the models performed well, though, they just weren't good enough, Patronus AI found.

"There just is no margin for error that's acceptable, because, especially in regulated industries, even if the model gets the answer wrong 1 out of 20 times, that's still not high enough accuracy," Qian said.

But the Patronus AI co-founders believe there's huge potential for language models like GPT to help people in the finance industry — whether that's analysts, or investors — if AI continues to improve.

"We definitely think that the results can be pretty promising," said Kannappan. "Models will continue to get better over time. We're very hopeful that in the long term, a lot of this can be automated. But today, you will definitely need to have at least a human in the loop to help support and guide whatever workflow you have."

An OpenAI representative pointed to the company's usage guidelines, which prohibit offering tailored financial advice using an OpenAI model without a qualified person reviewing the information, and require anyone using an OpenAI model in the financial industry to provide a disclaimer informing them that AI is being used and its limitations. OpenAI's usage policies also say that OpenAI's models are not fine-tuned to provide financial advice.

Meta did not immediately return a request for comment, and Anthropic didn't immediately have a comment.

Don't miss these stories from CNBC PRO:

News Outlets Question Fisker's (FSR) Liquidity, Pending Class Action Suit Alleges Fisker Obscured Costs - Hagens Berman

Hagens Berman, National Trial Attorneys, Encourages FSR Investors with Substantial Losses to Contact Firm's Attorneys

San Francisco, California--(Newsfile Corp. - January 3, 2024) - Hagens Berman urges Fisker Inc. (NYSE: FSR) investors who suffered substantial losses to submit your losses now.

Class Period: Aug. 4, 2023 - Nov. 20, 2023Lead Plaintiff Deadline: Jan. 26, 2024Visit: www.hbsslaw.com/investor-fraud/FSRContact An Attorney Now: FSR@hbsslaw.com844-916-0895

Fisker Inc. (NYSE: FSR) Securities Fraud Class Action:

Fisker's recent decision to dial back its full-year production target has caused news outlets to question whether the company will survive the next year.

On Dec. 7, 2023, MotleyFool observed, in response to the production cut, that the company "prioritized liquidity[,]" "[p]ut another way, Fisker requires cash for working capital needs and is therefore scaling back production[,]" and "[t]hat's a pretty dire situation to be in for any company."

Then, on Dec. 19, The Wall Street Journal noted the "cash crunch" several EV makers are facing, including Fisker. According to WSJ, Fisker recently reported as having only 187 days of cash on hand, meaning it could run out of cash before the middle of next year.

The recent reports questioning Fisker's solvency come in the wake of the filing of a securities class action suit, alleging that earlier this year Fisker tried to obscure its costs to make it appear more viable.

"The litigation challenges Fisker's cost accounting and internal controls over financial reporting," said Reed Kathrein, the Hagens Berman partner leading the investigation.

Specifically, the complaint alleges that Defendants misrepresented and concealed: (1) that Fisker had a material weakness in its internal control over financial reporting; (2) that Fisker had incorrectly accounted for certain costs; (3) that as a result the Company was likely to delay filing its quarterly report; and (4) that Fisker's infrastructure was limiting its ability to deliver its production.

Investors began to learn the truth on Nov. 8, 2023, when Fisker announced that it was delaying its Q3 2023 earnings release until after markets close on Nov. 13, 2023. Fisker blamed the delay on the unexplained departure of its, now former, Chief Accounting Officer John Finnucan on Oct. 27, 2023 and the appointment of a replacement effective Nov. 6, 2023.

Then, on Nov. 13, 2023, Fisker released dismal Q3 2023 results. It further revealed that material weaknesses in its internal control over financial reporting prevented it from timely filing its quarterly report with the SEC.

On Nov. 20, 2023, WSJ reported that Fisker has lost another Chief Accounting Officer. According to the WSJ, Florus Belting, who was named to the post in early Nov., resigned from the company the day after Fisker reported Q3 results.

Finally, on Nov. 22, 2023, the Company filed its Q3 2023 quarterly report on Form 10-Q, which disclosed that the Company had "identified approximately $20 million of expenses" which were "incorrectly recorded primarily as selling, general and administrative expenses in our preliminary earnings results, but were later determined to be associated with production set-up activities" and that "other inventory adjustments were recorded resulting in a $4.0 million increase in net loss subsequent to the preliminary earnings results."

If you invested in Fisker and have substantial losses, or have knowledge that may assist the firm's investigation, submit your losses now »

If you'd like more information and answers to frequently asked questions about the Fisker case and our investigation, read more »

Whistleblowers: Persons with non-public information regarding Fisker should consider their options to help in the investigation or take advantage of the SEC Whistleblower program. Under the new program, whistleblowers who provide original information may receive rewards totaling up to 30 percent of any successful recovery made by the SEC. For more information, call Reed Kathrein at 844-916-0895 or email FSR@hbsslaw.com.

# # #

About Hagens BermanHagens Berman is a global plaintiffs' rights complex litigation law firm focusing on corporate accountability through class-action law. The firm is home to a robust securities litigation practice and represents investors as well as whistleblowers, workers, consumers and others in cases achieving real results for those harmed by corporate negligence and fraud. More about the firm and its successes can be found at hbsslaw.com. Follow the firm for updates and news at @ClassActionLaw.

Attorney advertising.Past results do not guarantee future outcomes.Services may be performed by attorneys in any of our offices.

Contact:Reed Kathrein, 844-916-0895

To view the source version of this press release, please visit https://www.newsfilecorp.com/release/193101