By integrating AI into code review workflows, engineering leaders can detect risks in systems that often evade human detection at scale.
For engineering leaders managing distributed systems, the trade-off between speed of deployment and operational stability often determines the success of a platform. At Datadog, the company responsible for observability of complex infrastructure around the world, we operate under intense pressure to maintain this balance.
When a client’s system fails, they rely on Datadog’s platform to diagnose the root cause. This means reliability must be established well before the software is deployed into production.
Extending this reliability is an operational challenge. Code reviews have traditionally served as the primary gatekeeper, a high-stakes phase where senior engineers try to find errors. However, as teams grow, relying on human reviewers to maintain deep contextual knowledge of the entire codebase becomes unsustainable.
To address this bottleneck, Datadog’s AI Development Experience (AI DevX) team integrated OpenAI’s Codex to automate the detection of risks that human reviewers often miss.
Why static analysis is not enough
The enterprise market has long relied on automated tools to assist with code reviews, but their effectiveness has historically been limited.
Early iterations of AI code review tools often operated like “advanced linters” that could identify superficial syntax issues but failed to capture the broader system architecture. Because these tools lacked the ability to understand context, Datadog engineers often dismissed their suggestions as noise.
The central problem was not to detect errors in isolation, but to understand how a particular change would ripple through interconnected systems. Datadog needed a solution that could infer about a codebase and its dependencies, rather than simply scanning for style violations.
The team integrated the new agent directly into the workflow of one of its most active repositories, allowing it to automatically review all pull requests. Unlike static analysis tools, this system compares developer intent with actual code submissions and runs tests to verify behavior.
For CTOs and CIOs, the challenge in implementing generative AI often lies in proving its value beyond theoretical efficiency. Datadog bypassed standard productivity metrics by creating an “incident replay harness” that tests tools against past outages.
Rather than relying on hypothetical test cases, the team reconstructed past pull requests known to have caused incidents. We then ran an AI agent on these specific changes to determine whether they flagged issues that humans missed in code reviews.
The result was concrete data points for risk mitigation. Agents identified more than 10 cases (approximately 22% of the incidents investigated) where feedback could have prevented the error. These were pull requests that had already bypassed human review, and we found that the AI surfaced risks that were invisible to engineers at the time.
This validation changed the internal conversation about the tool’s utility. Brad Carter, who leads the AI DevX team, said that while he welcomes the efficiency gains, “at our scale, preventing incidents is much more compelling.”
How AI code review is changing engineering culture
By deploying this technology to over 1,000 engineers, we impacted the code review culture within our organization. Rather than replacing the human element, AI acts as a partner that handles the cognitive load of interactions between services.
Engineers reported that the system consistently flagged issues that were not obvious from direct differences in the code. It identified missing test coverage in the area of service-to-service coupling and pointed out interactions with modules that developers did not directly touch.
This depth of analysis has changed the way engineering staff interacts with automated feedback.
“For me, Codex comments feel like the smartest engineers I’ve ever worked with, who have endless hours to find bugs. I see connections that my brain doesn’t hold all at once,” Carter explains.
The ability of AI code review systems to understand the status of changes allows human reviewers to shift their focus from finding bugs to evaluating architecture and design.
From bug hunting to reliability
For enterprise leaders, the Datadog case study illustrates the evolution of how code reviews are defined. It is no longer viewed simply as an error detection checkpoint or cycle time metric, but as a core reliability system.
By surfacing risks beyond individual circumstances, this technology supports strategies where confidence in shipping codes grows with your team. This aligns with Datadog’s leadership priorities, which believe that reliability is a fundamental component of customer trust.
“We are the platform that businesses turn to when everything else breaks,” Carter says. “Preventing incidents strengthens the trust our customers place in us.”
The successful integration of AI into code review pipelines suggests that the technology’s greatest value to enterprises may lie in its ability to enforce complex quality standards that protect the bottom line.
See also: Agentic AI scaling requires new memory architecture
Want to learn more about AI and big data from industry leaders? Check out the AI & Big Data Expos in Amsterdam, California, and London. This comprehensive event is part of TechEx and co-located with other major technology events. Click here for more information.
AI News is brought to you by TechForge Media. Learn about other upcoming enterprise technology events and webinars.
