[AI Inspection: Challenges in the Field (5)] What to Consider When Generating Defect Data for Model Training

In Part 4, we wrote that foundation models can reduce the amount of training data required. However, even foundation models cannot solve the problem when there are no defect samples to begin with. In practice, the data we receive often consists of just 4–10 images per defect type.

At this point, there are two options: wait several months for defects to occur naturally, or create defects and use them as training data.

First, Why Simply Cutting and Pasting Doesn’t Work

The first method that comes to mind is to cut out a defect region and paste it onto an image of a good part. This is also a long-established approach in the Copy-Paste family of methods.

Training on this type of data may improve the metrics on the validation set to some extent. But once the model is deployed on the production line, false positives increase significantly. Pasted defects do not follow the lighting direction at that location, ignore the curvature of the surface, and leave unnatural edges around the boundary. Because the model learns the artifacts of the synthesis rather than the defect itself, it responds to irrelevant areas on the actual production line.

Similar problems arise when using diffusion-based inpainting as-is. Because the model learns the defect and background in an entangled manner, it may generate something inside the mask, but the result may not blend naturally with its surroundings or may become overfitted to the shape of the mask.

Let the Influence Flow Only from Background to Defect

What we need is not to simply draw a defect into an image, but to generate a defect that could plausibly have occurred in that specific location. To do that, the model first needs to understand the lighting, texture, and curvature of the target surface.

That is why we enforced an asymmetric rule during training.

[Figure 1] One-way influence between background and defect

We allow the model to use background information as a reference when generating a defect, while blocking the influence in the opposite direction. As a result, the defect is generated according to the physical conditions of that location, while the original texture and patterns of the background are preserved. We published this algorithm in an archived paper titled Background-Aware Defect Generation for Robust Industrial Anomaly Detection. It ranked first among comparable models on the average of four standard metrics on MVTec AD.

This asymmetry also has an additional benefit: the original image cannot be reconstructed backward from the generated result. In processes such as semiconductor, display, and secondary battery manufacturing, where taking data outside the facility itself may require approval, this provides a basis for generating augmented data within an internal closed network using only the characteristics of a small number of samples, without taking actual defective samples outside the facility.

How Do You Train a Generative Model with Just Four Defect Images?

This naturally raises a question: if there are only four defect images, how can you train a generative model with them? Generative models generally require more data than discriminative models.

The answer is to divide what the model needs to learn into two parts. Reproducing a plausible background and generating the shape of a defect are different problems, and they require different amounts of data.

There is plenty of data for the background. As we have repeatedly discussed in Parts 1 and 4, good-part images accumulate simply by running the production line. The surface texture of the target and the shading created by its lighting can be learned sufficiently from good-part images alone. Defect samples are needed only for learning the characteristics of the defect itself, so we adapt only that part using a small number of samples. We do not retrain the entire model.

This follows the same idea we used in Part 4, where a thin layer was designed to absorb differences between product variants. We separate what can be learned from large amounts of data from what requires only a small amount of data, and keep the latter as lightweight as possible.

That is why the separation between background and defect discussed above is not simply a matter of generation quality. If the two capabilities are entangled, four defect images would also have to influence the model’s ability to reproduce the background, making few-shot learning itself difficult to achieve. Separating the two capabilities is what makes few-shot learning possible.

There is another benefit. When a new defect type is added for the same target, the background training can be reused and only the defect component needs to be adapted again. There is no need to start from scratch every time a new defect type is added.

The Mask Is Both the Input and the Output

A generative model needs to know where to create the defect. That input is the mask.

In practice, how the mask is obtained makes a difference. Engineers may modify masks from actual defects they have collected, or the system can generate masks within a predefined region where defects are likely to occur. The important point is not to generate defects in locations where they cannot physically occur. If a defect that only occurs at a weld is generated on a flat surface, the detection model learns to monitor the flat surface for that defect. In effect, you are increasing the data while reducing performance.

There is another part of the output that makes a significant practical difference. Because the model knows which pixels contain the generated defect, the ground-truth mask is generated automatically as well. Even if the amount of data is increased 100-fold, there is no additional labeling workload.

This mask can be directly used to train a segmentation model. Manually drawn masks can vary because different operators may use different criteria when defining boundaries. Generated masks do not have this kind of operator-to-operator variation.

Generating 1,000 Images of the Same Defect Is Pointless

Simply increasing the quantity does not make the data more useful. As we wrote in Part 1, rotation and flipping do not increase the diversity of defect shapes. The same trap exists in generation. If the prompt and mask are fixed and only the seed is changed, similar defects will be generated repeatedly.

So we vary the generation conditions themselves. We randomly change the location, size, orientation, and intensity of the defect within predefined ranges. The basis for defining those ranges comes from the actual manufacturing process. The defect-prone regions discussed earlier are used again here.

The Strategy Depends on the Process

Even when we use the same generation engine, we divide how the input is processed according to the characteristics of the manufacturing process.

In large-area scanning processes, the background is relatively simple and defects are small. If the image is downscaled, the fine textures that need to be inspected become blurred. So the system divides the image into patches and generates defects at the original resolution. Fine textures on metal surfaces need to be preserved at the pixel level to maintain the precision of the inspection model.

In complex-structure imaging, the resolution is lower, but the context is more complex and defects are larger. Global context, such as the alignment between components, is more important. So engineers define regions of interest (ROIs) where defects may occur and restrict generation to those areas.

Specify the Defect Shape with Text

When the generated result is not what we want, the conventional approach is to have an engineer adjust parameters or redraw the mask. This requires specialized knowledge and often involves many iterations. More importantly, there was no straightforward way to directly specify the direction, size, or depth of a defect.

So we changed the input to natural language.

[Figure 2] Processing flow from natural-language input to generation

The LLM converts the user’s input into a prompt format that the generative model can understand, and those instructions guide the shape of the defect. If the result is not what the user wants, they can simply edit the text and generate it again.

Below are results generated using the same model and the same mask, with only the prompt changed.

[Figure 3] Comparison of results generated by changing only the prompt (MVTec AD hazelnut)

The direction, size, and presence or absence of shading are differentiated according to the prompt.

The system is not yet perfect. In the current generation architecture, the mask’s prior information tends to overpower the text condition, limiting how much the text can control the generated defect beyond the mask shape. We are working toward dynamically generating masks by separating the text-prompt encoding for the background and foreground.

The People Who Know the Inspection Criteria Should Be Able to Create Them

The reason we changed the input to text is not simply convenience.

In inspection, the difficult samples are not obvious defects but those near the boundary between pass and fail. And the people who know that boundary best are not AI engineers but quality personnel on the production floor. If a quality specialist reviews a generated result and says, “It doesn’t look like that,” or “At this level of intensity, this should pass,” that judgment can be reflected in the next generation. By generating more borderline samples and adding them to the training data, the model’s decision boundary can be brought closer to the criteria used on the production floor.

In Part 1, we wrote that it is difficult to know borderline samples in advance. Generation reverses that process. Instead of looking for borderline samples, we create them, show them to the quality personnel, and confirm where the boundary between pass and fail should be.

We are currently working on translating these empirical inspection criteria into numerical parameters such as clarity, size, and depth.

Whether It Helps Is Verified with the Detection Model

A generated image looking plausible and improving inspection performance are two different things. Judging only by whether the image looks natural to the human eye can lead to the wrong conclusion.

That is why we validate the results on the detection-model side. We compare a model trained only on original data with one trained using both original and generated data, evaluating both on the same set of real defects. Generated images are never included in the evaluation set. If a model trained on a synthetic distribution is evaluated on that same synthetic distribution, it will naturally perform well, but those numbers have little to do with real-world performance.

The synthetic-artifact learning described earlier also becomes apparent in this comparison. If the validation metrics improve but false positives increase on the real-world defect set, the model has learned the synthesis method rather than the defects themselves.

We also determine the ratio of original to generated data through experimentation. If the proportion of generated data is increased indefinitely, the model is pulled toward the synthetic distribution. So we stop before performance on the real-world defect set begins to decline.

Summary

With just a few samples per defect type, we can create training sets ranging from hundreds to 1,000 images. Because the masks are generated along with the images, there is no additional labeling workload, and because the original image cannot be reconstructed from the generated result, the entire process can be completed within a closed network.

The resulting data feeds into the hybrid architecture described in Part 3. When there are not enough samples of a new defect identified as Unknown by the unsupervised model, the generated data is used to supplement the samples and incorporate the defect into the supervised model.

[AI Inspection: Challenges in the Field] Series

Part 1. Starting AI Inspection Without Defective Products
Part 2. What Actually Goes Wrong When Detecting Anomalies Using Only Normal Samples
Part 3. Why AIVEX Combines Supervised and Unsupervised Learning in a Single Model
Part 4. How We Processed Thousands of Product Variants with a Single Model
Part 6. The Problem of Manually Aligning the Optical System Whenever the Product Variant Changes