Since generative AI went mainstream a year ago, it has inspired an equal measure of hype and fear. Boosters of tools such as ChatGPT and DALL-E predict that they will transform our economy, while skeptics worry about their potential to produce inaccurate or harmful results and ultimately replace workers. But until recently, no one had tested what really happens when companies unleash generative AI at scale in real workplaces.
Customer service agents who an AI coach aided saw their performance improve. Image credit: Sarah McKinney
The first such study, released as a National Bureau of Economic Research working paper earlier this year, found the best-case scenario: Providing workers with a generative AI tool similar to ChatGPT can lead to more productive workers, happier customers, and higher employee retention.
Researchers studied nearly 5,200 customer support agents at a Fortune 500 software firm who gained access to a generative AI-based assistant in a phased rollout between November 2020 and February 2021. During support chats, the generative AI tool shared real-time recommendations with operators, suggesting how to respond to customers and supplying links to internal documents about technical issues.
Compared to a group of workers operating without the tool, those who had help from the chatbot were 14% more productive, on average, based on the number of issues they resolved per hour. The AI-supported agents ended conversations faster, handled more chats per hour, and were slightly more successful in resolving problems. Notably, the effect was largest for the least skilled and least experienced workers, who saw productivity gains of up to 35%.
Big Gains With Fewer Pains
“These are huge numbers,” says Erik Brynjolfsson, a professor (by courtesy) at Stanford Graduate School of Business and a senior fellow at the Stanford Institute for Human-Centered Artificial Intelligence. “I’ve done lots of work on the introduction of new information technology over the years, and often companies are happy to get 1% or 2% productivity gains.”
Brynjolfsson was also surprised that productivity rose so quickly. “Often with new technologies, there’s a bit of a decline before it takes off, because it’s difficult and costly to implement changes, retrain workers, and change business processes,” he says. “In this case, we did not see a lull, and performance took off over just a few months.”
The reason for the boost, according to Brynjolfsson and coauthors Danielle Liopen in new window, an associate professor at the MIT Sloan School of Management, and Lindsey Raymondopen in new window, a PhD student at MIT Sloan, was that the bot learned what the most successful agents were doing right by digesting millions of transcripts of service interactions. It then disseminated these lessons — often tacit knowledge that’s hard to distill into employee trainings — to the wider workforce. With help from AI, agents who’d been on the job for two months performed as well as unsupported agents with six months of experience.
“For most of the past 30 years, computers and digital technologies have helped higher-skilled workers more than less-skilled workers, which has led to a growing gap in wages and income inequality,” Brynjolfsson says. “It was fascinating to see that this technology goes the other way around — it’s a good sign.”
The intervention also led to happier customers, as measured by both customer surveys and a textual analysis of their language in conversations. “People like it when you solve their problems, and the system seems to be doing a better job of that,” Brynjolfsson says. The team didn’t find much evidence that customers were more satisfied simply because agents with the AI assistant used more positive language — the research found that agents tended to use a peppy tone even before the system was introduced.
Far from resenting an AI coach, agents with access to the tool were less likely to quit, a significant finding in an industry with chronically high turnover. “We don’t know for sure why this occurred, but I would guess that it’s more enjoyable to be in a job where the customers like you and you can solve customer problems faster,” Brynjolfsson says.
AI vs. Inequality
Overall, the researchers concluded that generative AI was a win for the company’s employees, customers, and stockholders alike. “It wasn’t squeezing one group for the benefit of the others — all three were better off,” Brynjolfsson says. In 2021, he co-founded a startup called Workhelix to help other companies apply generative AI to boost productivity.
Taking a broader view, Brynjolfsson says the study signals that generative AI will turbocharge productivity in the U.S. economy over the coming decade. He was so confident in his prediction that he placed a $400 bet on Longbets.org that non-agricultural productivity growth would average over 1.8% annually through 2029 (compared to the Congressional Budget Office estimate of less than 1.5%).
“That would mean we could raise living standards and address a lot of problems, like the budget deficit, healthcare, and the environment,” he says. “And if it’s helping less-skilled workers more, that can help reduce inequality.”
Brynjolfsson cautions that the study doesn’t illuminate how generative AI will reshape the broader labor market. Yet he argues that employers who interpret the research as an excuse to lay off higher-skilled workers are missing the point. “The lesson is that, more often than not, you’ll benefit by augmenting workers rather than trying to replace them,” he says. “A smart company is going to make sure they compensate and retain the high-skilled workers, so the system can continue to learn from them.”
Source: Stanford University