Building Prism: Our In-House AI Code Review System

AI Made Implementation Faster, but Not Development

Our AI Platform Team actively uses Claude and Codex to develop the AIVOps platform. As a result, detailed implementation has become roughly two to three times faster than before we started using AI models.

However, faster implementation did not necessarily mean faster development overall. The amount of code that developers had to review increased significantly, and it was difficult for people to quickly assess the logic generated by AI. As a result, the review process actually became slower.

We ended up spending more time on code reviews than on implementation, and overall development speed remained almost unchanged.

Figure 1. Time spent on development before and after adopting AI tools — Implementation became 2–3 times faster, but code review became the bottleneck, leaving overall development speed largely unchanged.

This led us to ask: “Could we use AI to make code reviews faster as well?”

Even without a developer reviewing the code first, an agent could review it from various software engineering perspectives. This would give developers another opportunity to check the implementation, while also providing reviewers with useful insights into the code and helping them conduct reviews more quickly and easily.

There Were Already Various Code Review Solutions

Since this seemed like a common problem that developers everywhere would encounter, we assumed there must already be solutions available. As expected, there were.

Some repository platforms offered built-in review features, while third-party services could be integrated with repositories to provide automated reviews. There were also tools that reviewed code directly within a developer’s local codebase.

Since we did not know which approach would work best for our team, we tried them one by one using each service’s free tier and gathered feedback from our developers.

After using the different services, one conclusion became clear: code review services were generally helpful for development, and the developers wanted to continue using them.

In some cases, they caught errors that developers had missed. They also provided useful insights, such as code summaries, which made the review process easier.

Figure 2. Three types of code review solutions available on the market — We evaluated platform-native, third-party integrated, and local tools one by one through their free tiers.

Still, We Decided to Build Our Own System

Despite their usefulness, there were several reasons why we could not simply adopt these solutions as they were.

The first challenge was detailed configuration. Some services did not clearly expose their configuration options, requiring us to search for hidden settings one by one. Others offered very little room for customization, making it impossible to change review triggers or even forcing PR approval in ways that did not fit our workflow.

The biggest concern, however, was cost.

Our team generates numerous PRs every day, making services that charge per PR difficult to use in practice. Even subscription-based services typically cost more than around $20 per user per month.

Security was another concern. If we sent internal company code to an external SaaS platform for review, it would be difficult to completely eliminate the risk of code exposure.

Considering the limited customization, significant cost, and security concerns, we concluded that building our own system would be a better option than adopting an existing solution.

That is how we started developing our in-house code review system, Prism.

The name reflects the idea behind the system: just as a prism separates light into different colors, Prism analyzes code from multiple perspectives, including security, performance, readability, and software design.

Figure 3. Three reasons we chose not to adopt existing solutions — Limited customization, cost, and security concerns led us to build our own system.

It Started Simple, but Gradually Evolved

The initial version of Prism had a very simple architecture.

When a PR event was received from a repository, the system put the PR information into a prompt and returned an AI-generated code review.

Even this simple approach produced meaningful results. However, because we could not provide sufficient repository context, the reviews could not take into account the conventions and design patterns of the existing codebase.

Since passing the entire repository as context was not practical, we initially included the complete contents of files changed in the PR, along with files organically connected to those files, as context for the review.

This created another problem.

Because PR-related content was added to the context in a uniform way, the amount of tokens being passed to the model became too large. In many cases, there was more surrounding code than changed code, which actually reduced the quality of the reviews.

So instead of mechanically adding more context, we changed the architecture so that the model could retrieve the information it needed on its own.

We gave the model tools to read files and git logs, and allowed it to gather information directly until it determined that it had enough context to conduct the review.

This approach allowed us to obtain high-quality reviews with fewer tokens while still providing sufficient context.

Figure 4. A shift in how context is collected — We moved from uniformly injecting context to an agent-driven approach where the model gathers only the information it needs.

It Is Now Used Across the Company

Today, we are running code reviews at a very low cost of approximately KRW 100 per PR.

Despite the low cost, Prism catches more issues than we initially expected, making it a useful part of our development workflow. It has also evolved beyond a tool for our own team into a company-wide internal project, helping us identify potential errors and problems at the code review stage before they make their way further into development.

Figure 5. An actual code review left by Prism on a PR — Along with a summary, it identifies issues by severity down to the file and line level.

The points that matter most can differ from one project to another. To address this, we added simple repository-level settings that allow teams to specify additional perspectives for each project.

We also made it possible to include conversations and questions exchanged during the PR process as context, and have continued to add features to improve the quality of the reviews.

Figure 6. Prism repository settings — Multiple internal project repositories can be registered, with repository-specific AI instructions incorporated into the review process.

We are now taking this one step further.

Rather than simply displaying review results, Prism can use those results to have an agent automatically create improvement commits or open pull requests. In this way, we are working toward automating the development loop itself.

We are also experimenting with scheduled reviews for entire repositories, going beyond individual PRs. The goal is to regularly assess repository-wide stability and continuously reduce technical debt.

Figure 7. Automating the development loop with Prism — Beyond review comments, Prism can automatically generate improvement PRs and perform scheduled repository-wide reviews.

Looking ahead, we are also considering introducing local LLMs to eliminate model usage costs altogether.

We have tested models such as Llama, and cloud-based models currently deliver better results when response time and resource requirements are taken into account. However, the availability and performance of local LLMs are improving rapidly, so we believe this is a direction worth pursuing in the near future.

What Matters Is Platform Stability and Software Quality

No matter how quickly AI can generate code, quality will eventually suffer if it is not backed by proper validation.

Prism is another part of our ongoing effort to maintain the stability of our platform and the quality of our software.

Our team will continue to think about what else is needed to build and maintain a stable platform and high-quality software.

*Related post from AI Platform Group:
How to Interpret Training Results Without a Model Expert