Generative AI (GenAI)-enabled software development will improve productivity and work efficiency – the question is, how much? Most market research on this topic shows considerable gains in productivity. Research from Harvard found that specialists, depending on the task and seniority, saw a 43% increase in productivity. Likewise, a report from Goldman Sachs suggests that productivity could rise by 1.5 percentage points with GenAI after ten years of broad adoption, equating to almost double the pace of US productivity growth. While insightful, most of these findings come from controlled settings that don’t necessarily reflect the nuances of real-life use cases.
To better answer how much GenAI can enhance productivity in software development, a leading digital transformation services and product engineering company decided to record its practical findings and insights from a recent large-scale GenAI implementation project with one of its clients. This client wanted to adopt GenAI into the work processes of 10 development teams across three workstreams, entailing over 100 specialists. These real-life findings reveal the various challenges businesses will encounter along the journey; moreover, they underscore the necessity of a company-wide roadmap for scaling GenAI adoption.
Addressing Specialists’ Negative Attitudes and Expectations
Many challenges can delay the success of a GenAI project, such as legal and regulatory concerns, a lack of processing capacity, security and privacy, etc. However, the most significant roadblock encountered during this large-scale implementation was the specialists’ attitudes and expectations around the technologies. During the implementation, the engineering company observed that the client’s specialists had certain expectations about GenAI and how it would augment their work. When these initial expectations didn’t align with the outcomes regarding quality or execution time, they would develop negative attitudes toward the technologies. In particular, when the GenAI didn’t, in their words, “Do the work for me,” they would respond with comments like: “I expected better and don’t want to waste my time anymore.”
Businesses must shift perceptions and transition to a new working culture that prevents these negative attitudes from manifesting and hampering adoption and accurate measuring. Surveys and assessments are an efficient means of mapping and categorizing the attitudes and perceived engagement of one’s specialists. From there, companies should group specialists based on their feelings toward GenAI. Then, businesses can create tailored change management approaches for each group to promote successful AI integration; for example, the most skeptical specialists will receive more attention and care than neutral specialists.
Accounting for The Complexities of Real-world Projects
The second most obstructive challenge was accurately measuring the impact of GenAI on productivity while accounting for the complexities of real-world project conditions. In controlled environments, it is easier to gauge the influence of GenAI – however, as mentioned earlier, such tests do not consider certain variables and inconsistencies. Projects aren’t stagnant. They evolve constantly. An organization may have a situation where they have rotating specialists due to vacation schedules and sick days or sudden changes in priorities. Specialists are also not always working on specific project activities where GenAI impact can be the most beneficial because they have meetings to attend, emails to answer and other tasks outside the sprint scope that often get overlooked in productivity measurements. These inconsistencies and variables must be accounted for when objectively measuring the impact of GenAI on software development.
Other best practices include integrating task management tools into workflows to see how long tasks stay in each status to determine non-technical specialists’ productivity and efficiency. Likewise, business intelligence solutions can automatically gather data points, reducing errors and saving time. Additionally, organizations can mitigate the complexities of real-world project conditions and ensure a more accurate evaluation of GenAI’s impact on productivity by employing thorough data cleanup practices.
Company-Wide Roadmap: Measuring Accurately
This large-scale GenAI implementation also highlighted the value of a company-wide roadmap that marks the beginning and end of the integration. Businesses should note that a crucial element of this roadmap is defining the metrics they will use for the baseline and final reporting stages. Dozens of different metrics can help assess GenAI’s impact on productivity, including, but not limited to, velocity in time, throughput, average rework and code review time, code review failure and acceptance rates, time spent on bug fixing, etc.
After defining these metrics, companies should classify them into objective and subjective categories. Businesses can also use data from task-tracking tools like Jira for objective metrics. Likewise, they must maintain and adhere to quality flows, timely task updates and thorough stage completion. Recall that subjective metrics, like specialist and pilot surveys, will help businesses understand adoption levels and correlations with objective measurements. From a frequency perspective, measurements should be routine and scheduled, not sparse and random. Furthermore, the project’s findings emphasize the usefulness of metrics such as average daily impact, perceived proficiency, performance changes, work coverage, AI tools usage and uninterrupted workflow to measure adoption progression.
Company-Wide Roadmap Continued: Learning and Culture Development at Scale
In addition to effectively measuring the impact of GenAI, another vital component of a successful roadmap is that it drives continuous learning and AI fluency through different training and coaching strategies. These initiatives will ultimately foster a company-wide learning culture, enabling AI adoption at scale across the enterprise. Various strategies include creating working groups that focus on where and how the company can leverage GenAI as well as encouraging individuals to share what is and is not working. Also, it is helpful to set up growth and development priorities accompanied by learning paths at the individual and team levels.
Another way companies can build a culture that readily adopts new GenAI technologies is by highlighting quick-win use cases. These will demonstrate the power of GenAI to the larger organization and reluctant skeptics. Businesses should also establish security guidelines and rules of engagement with AI to empower teams to experiment and explore new approaches without exposing the company to risk. Likewise, organizations must enforce adherence to industry standards and other best practices while addressing change management among individuals and teams at the task and tool levels.
Keeping People at the Center
The two most important takeaways from this real-world implementation are: firstly, GenAI can lead to substantial productivity gains within the confines of a proper strategy and roadmap; secondly, such an integration has an undeniable human element that companies must address accordingly. GenAI will forever change how these specialists perform daily tasks. It is also likely that GenAI may make some specialists feel threatened by the technology which may cause resistance to adoption. Ultimately, the key to a successful GenAI implementation remains distinctly human. It’s crucial for businesses to grasp the depth of this, as it is humans that operationalize the technology, unlocking its practical value.
