Stocks in the US AI technology company fell in value at the end of yesterday’s trading, with the NASDAQ Composite Index down 1.4%. Some who lost their value fell 9.4%, while ARM Holdings fell 5%. According to the Financial Times (Paywall), the market saw its biggest day drop from the beginning of August on Tuesday.
Some traders fell over by a report (PDF) released by AI company Nanda. This focused on the high failure rates of many generation AI projects in commercial organizations. Project Nanda, which is published at the Massachusetts Technology Media Lab, describes it as an organization that builds the “agent web.” This paper has been located behind the research wall since its publication, but can be downloaded from this site.
The research authors state that only 5% of Gen AI Pilots reach production, creating measurable monetary value, with the majority of the projects having little impact on profit and loss indicators. The study conducted by Nanda consisted of content from 52 structured interviews with enterprise decision makers, an analysis of researchers of over 300 public AI initiatives and presentations, and a research questionnaire completed by 153 company leaders. After the Gen AI project left pilot status, it measured return on investment over six months.
While many organizations deploy AI in front office or customer-facing business capabilities, successful projects tend to be found in back office workflows, the paper says. It is mainly in the back office mundane tasks where savings occur, mainly due to the low needs of third-party institutions and BPOs. The survey found that AI projects had little impact on the overall internal staff level.
Although 90% of staff said they personally benefit from using commonly available AI, these subjective benefits are not translated at the institutional level, usually in the form of large-scale language models like ChatGPT. Approximately 40% of the companies surveyed paid for a subscription to LLMS.
Many owners of failed projects cited the lack of contextual awareness demonstrated by generative AI models, namely adapting to the situation, changing over time, and reminiscing of previous enquiries. Nanda says that providing such a system and forming partnerships with organizations that allow them to adapt to the specific circumstances of the organization is a key component of success. The paper highlights some quotes “derived from interviews” including 60%-70%, including “(AI system) not learning from feedback”, “(AI system) not learning”, and “many manual contexts needed each time”.
The most positively impacted verticals of Gen AI were Media & Telecom, followed by professional services, Healthcare & Pharma, Consumer & Retail and Financial Services. The launch rates for generative AI projects in the Energy & Materials sector are currently negligible, the paper states. From a business unit perspective, sales and marketing are, or are, at, or are, at, based, where most projects are based. Finance and procurement are the least popular places where AI projects can be launched.
The typical organizational area where generation AI is most deployed is in sales and marketing, while finance and sourcing are the least popular sites. Additionally, complex tasks are the least likely to be expected to be completed by AI. Managers assign projects such as client management to AI, but tasks like summarizing reports and writing emails are sent to humans in 70% of cases.
The lack of language and academic rigor of published reports suggests that their source and purpose are more similar to marketing than intellectual and technical discussions. The authors of the paper seek to develop strategic partnerships with knowledgeable vendors to increase the chances of success in generating AI projects. “There is an unprecedented opportunity for vendors who can provide a deeply integrated learning-ready AI system,” the paper concludes.
The Nanda Report headlines are read calmly among decision makers responsible for implementing generative AI, but the underlying message of the paper is undermined by the intent behind the publication. While stock prices this week could be influenced by partisan investigations from authors with obvious skins in the game, Nanda’s publications seem likely to simply reflect trading floor concerns regarding the practical effectiveness of generation AI as a business tool.
(Image Source: “Arthur Daily” by Tim Dennell is licensed under CC BY-NC-ND 2.0.)
