Are You Still Opening Folders on the Inspection PC to Check False Positives and False Negatives?

We Were Lucky Enough to Find It, But That’s Where We Got Stuck

The sequence of events on the day a claim comes in is usually similar. Once the lot number or product number and an approximate production time are provided, someone sits down in front of the inspection PC on the production floor, opens a path such as E:\Image\202609\02\2000026090213020949, and starts going through images from that time period, using the timestamps in the filenames as clues. A single line can generate tens of thousands of images a day, and File Explorer starts to struggle once the preview count goes into the hundreds.

That day, we were still relatively lucky. The lot in question was from ten days earlier, so the images had not yet been deleted. It took four hours to find that one image. But that was where we got stuck. The inspection result was still recorded in the local database, but the criteria values that had been applied at the time and the score produced by the model had been overwritten by subsequent recipe changes, so they could no longer be verified. In the end, the conclusion reached in the meeting was, “Let’s try increasing the sensitivity a little.” With no evidence to support the decision, it was not an adjustment based on data, but a guess.

The reason guessing is dangerous is simple. Over-inspection means classifying a good product as defective, while under-inspection means letting a defective product pass; the two are essentially connected like opposite ends of a single dial being pulled in different directions. If you increase sensitivity to reduce under-inspection, over-inspection rises with it. So turning the dial without evidence is closer to shifting losses from one side to the other than making an improvement. For it to be an improvement, you need to be able to compare the numbers before and after the adjustment.

And if that lot had been from two months earlier, there would not even have been an image to find in the first place.

It’s Not That the Records Don’t Exist. They Just Don’t Stay for Long

Let’s clear up a common misconception first. These days, most inspection systems store images together with inspection results. Image folders accumulate on the inspection PC, while inspection results are stored in a local database. The problem is not that there are no records, but that those records are stored separately within the narrow confines of the inspection PC. The folders contain only filenames and timestamps, while the inspection history sits in the database, so connecting the two means someone has to manually match the timestamps and visually track them down every time.

A single high-resolution line can generate hundreds of gigabytes of data a day. The inspection PC’s disk cannot hold more than a few weeks’ worth, so automatic circular deletion, starting with the oldest data, is typically enabled by default. The local database is no different: records are cleaned up after a certain period, and even while they remain, they can only be accessed by sitting in front of that particular PC. On top of that, even within the same factory, different equipment manufacturers use different schemas and folder structures, making it impossible from the outset to view the entire line as a single dataset.

In short, the records on an inspection PC are records designed to keep the line running today, not records designed to look back several months and improve the process. This is not anyone’s fault or a matter of negligence. It is simply the natural result of equipment designed to prioritize inspection speed above all else. But improvement cannot begin on top of that structure.

The Inspection PC Cannot Answer Four Questions

“How high was the over-inspection rate last month?” The data may already have been deleted, or even if it remains, it is scattered across individual lines with no practical way to aggregate it. “Why was this product classified as NG?” The inspection result may still be there, but the criteria values, model score, and recipe version at that exact moment were not preserved together. No matter how much you zoom in on pixels stripped of their context, they cannot tell you why the inspection made that decision.

“Is this phenomenon concentrated around a particular piece of equipment or time period?” To cross-reference these conditions, someone has to go around to multiple PCs, collect the data, and build an Excel file. That Excel file is then used once on the person’s PC and disappears. “Can we build a retraining dataset from these cases?” By the time someone tries to collect them, a significant portion has already been deleted, and even what remains has become disconnected from the inspection history, meaning the labels have to be recreated from scratch. This is why most retraining efforts end up being planned and then postponed.

Data Quietly Changes as It Passes Through Human Hands

To answer these questions, the data somehow has to be taken outside the inspection PC. But there is a deeper problem here than the fact that it takes four hours to find an image: Can you actually trust the data once you have taken it out? Since the only way to get data out of an inspection PC is ultimately through people, human intervention occurs every time the data moves. Each action is performed for a reasonable reason, but with every intervention, the data gradually moves away from its original state. When the disk fills up, old folders are moved to an external drive. Someone copies subfolders to their own PC to analyze them. Images are moved into OK and NG folders for labeling.

The most common problem is mixing. If an image that was confirmed as a good product during reinspection is moved into an OK folder, the fact that the product was originally flagged as NG by the inspection system disappears at that moment. An entire over-inspection case that should have served as evidence for improvement effectively vanishes. Data loss happens without warning. The exact period in which a problematic case occurred may be deleted during disk cleanup, and no one is notified if a large file transfer is interrupted halfway through. Transformation is the hardest to notice. The moment an image is resized or recompressed for sharing, an image that looks identical to the human eye can become a different image to an algorithm, and a scratch visible in the original can be buried in compression artifacts. Creating standardized copies for viewing according to defined rules is completely different from resizing images on the fly to send them by email. In the latter case, no one records what information was lost.

Table 1. Data contamination caused by points of human intervention

The worst part is that this kind of contamination leaves no error message. The copy succeeds, and the file opens normally. So the over-inspection rate calculated from the resulting data is reported as is, and the model trained on that data is deployed on the line as is. When the model behaves strangely, the algorithm is questioned again, but the real cause is that the training data had already been mixed. This is not a question of whether the person in charge is diligent. Any process that relies on data passing through human hands will eventually end up this way, no matter how diligent the people involved are.

Inspection Data Should Be Records, Not Files

At this point, the nature of the problem starts to look different. “Why was this image classified as NG?” is an individual lookup, so as we saw earlier, if you spend enough time, you can somehow get to the answer. But the question actually needed for improvement is an aggregate question: “How are the inspection results distributed?” That question cannot be answered on top of a system where data disappears after a few weeks.

That led us to a simple conclusion: Instead of treating each inspection as a single image file, we need to treat it as a record in which the inspection result, the conditions at the time, and the pixels themselves are bound together, and that record needs to be stored outside the inspection PC for the long term. Once the unit and lifespan of the data change, the questions above become simple queries. The over-inspection rate can be aggregated, the reason for a decision becomes a matter of reading the condition values stored alongside it, equipment-specific concentration can be analyzed through grouping, and a retraining dataset becomes a matter of selecting records that meet specific conditions.

Table 2. The difference in how data is structured and how long it is retained

Over-Inspection Can Be Counted, but Under-Inspection Can Only Be Narrowed Down

Figure 1. Moving the decision threshold causes over-inspection and under-inspection to move in opposite directions. The OK cases that barely pass just before the threshold are the next candidates for under-inspection, and this figure can only be drawn when inspection scores are preserved as records.

However, over-inspection and under-inspection cannot be handled in exactly the same way. Even when the same records are collected, the two types of misclassification provide different kinds of information. Over-inspection involves cases that the inspection system has already marked as NG, so as long as the records are preserved, they can be counted immediately. Divide the number of cases overturned as good products during reinspection, and you have the over-inspection rate. Breaking it down by equipment, product, or time period is also just one query away. This is the point where “there seem to be more of them lately” turns into a number.

Under-inspection is different. The moment the inspection system classifies a product as normal, that case leaves no signal. No matter how diligently you collect the data, you cannot know how many under-inspection cases are hidden within it unless you actually examine the images. Ultimately, the under-inspection cases the production floor knows about are only the ones someone has reported, and simply having a storage system does not change this situation by itself.

So there are two things you can do about under-inspection. The first is to trace it back. When a report comes in from a downstream process or a customer, immediately retrieve the relevant lot using its conditions and check not only the result but also the inspection criteria and score applied at the time. This is exactly where the folder-based approach breaks down, as we saw earlier. It can take several hours to find the data, and if the issue happened several months ago, the data may not even exist anymore.

The other is to narrow it down. You cannot have a person review every case that received a normal classification, but if the inspection score is recorded, you can select only the cases that barely passed. An OK case whose confidence score is close to the decision boundary is a candidate for the next under-inspection case. Reviewing a sample from that range produces different results from reviewing a random sample, even when the same amount of time is spent. Under-inspection is not something you simply find; it is something you narrow down.

From Inspection to Retraining, There Should Be No Point Where People Move Files

This is the workflow we configure on the production floor. Once an inspection is complete, the inspection result is immediately sent to the server next to the inspection PC and recorded in the database, while the image initially remains on the inspection PC. This is to avoid placing additional load on the inspection path. The result and JPG image are then paired by FOV and transferred to long-term storage. From this point on, the result is no longer dependent on the lifespan of the inspection PC, and yield is calculated here. However, because lossless originals require too much storage capacity, they remain on the inspection PC, and this difference becomes a problem again later.

Review also takes place in the same environment. Because the image is stored together with information about what defect was missed, what inspection criteria were applied at the time, and what confidence score the model produced, the answer to “Why was this classified this way?” becomes a matter of querying the record rather than making a guess. Areas that a human reviewer identifies as misclassifications are then passed directly to the training platform, where they become the starting point for labeling. This is where review and labeling become a single operation rather than two separate tasks.

Table 3. The workflow from inspection to retraining

There is not a single step in this table where a person has to copy or move a file. The mixing, loss, and transformation we discussed earlier all occur at those points, so eliminating those points is itself a way of preserving data integrity. And because the inspection results generated after the new model is deployed on the line are accumulated through the same process, this workflow does not run once and end. It continuously completes another cycle as the line operates.

Even So, the Original Images Do Not Last More Than a Month

There is also a limitation we should acknowledge honestly. All JPG images paired with inspection results are sent to long-term storage and retained for about a year. This is sufficient for aggregating yield and reviewing misclassifications. However, when the images are needed for training, uncompressed originals are often required, and BMP originals are too large to move to long-term storage, so they remain on the inspection PC. And on the inspection PC, they disappear within a month. In other words, the results live for a year, while the originals live for a month.

This eleven-month gap leads to an uncomfortable conclusion. Even if you dig through long-term storage and find a problem from two or three months ago, the original image that the inspection system saw at that time has already been deleted from the inspection PC. You have identified the cause, but you have no material left with which to fix it. Ultimately, the upper limit of improvement is determined less by how long you retain data than by how quickly you discover problems. “Let’s review it when we have some time” is not simply a decision to postpone the schedule; it is closer to deciding to throw away the training data from that period.

So what we recommend is not simply extending the retention period, but aligning two things. First, bring the review cycle within the retention period of the originals. Second, let the inspection results determine which images should be retained as originals. NG cases, the surrounding images, and the OK cases near the inspection decision boundary mentioned earlier are cases that the next model will inevitably need to review again, so retain them as originals. For normal cases that passed comfortably, retain only samples. It is better to select what needs to be retained and keep it for the required period than to try to retain everything and lose everything after a month.

Figure 2. Inspection results and JPG images live for one year, while lossless originals live for one month. The later a problem is discovered, the more likely it is to enter a period where there is no material left to fix it.

One more point: Because all of this storage and querying takes place on servers within the factory, no traffic needs to cross the network boundary even in network-separated environments. There is also no reason to change every line at once. Starting with a single line where problems occur most frequently—or even simply moving inspection results and condition values outside the inspection PC first, without moving the images—can immediately answer half of the questions discussed above.

If You Were to Check Your Current System

We recommend checking a few things against the inspection system you are currently operating. If a problem is discovered in a lot from three months ago, is the original image still available to run training on? Can you state last month’s over-inspection and under-inspection rates as numbers? Can you find an image of a defect under a specific set of conditions within three minutes, and see the inspection criteria and score from that moment alongside it? And when you find that image, can you be certain that it is exactly the same image the inspection system actually saw at the time?

If the answer to even one of these questions is “No,” the reason improvement is slow may not be the algorithm. The inspection system is already generating tens of thousands of inspection records every day. As long as those records are deleted from the inspection PC within a few weeks, we will keep coming back to the same situation: spending four hours finding a single image, only to be unable to fully trust that one image and turning the dial based on guesswork.

*Related articles from the Vision Platform Group:
Why AIVEX Chose Hybrid Computing