[AI Inspection: Challenges in the Field (6)] The Problem of Manually Aligning the Optical System Whenever the Product Variant Changes

The process of adapting inspection equipment to a new product variant is called teaching. It involves setting the coordinates for where and at what distance the camera and profiler should view the product, and in most manufacturing environments, this is done by a person. The operator watches the screen, moves the axes little by little to align them, and saves the settings.

When there are around ten product variants, this is manageable. The problem comes after that. Every time a new variant is added, an engineer has to get involved, the results can vary depending on the operator and the date, and the line has to stop during teaching. When the number of variants reaches thousands, this approach no longer works. What is needed is not a faster way to configure each variant, but a framework that does not depend on individual product variants.

We Finish It Before the Inspection Zone

The first thing we changed was where teaching is performed.

If the optical system is aligned inside the inspection zone, the line has to stop for the duration of the alignment. So we created a separate registration zone in front of the inspection line. As the product passes through, the system estimates its shape and calculates the position of the inspection optics in advance. Since the inspection process starts with the optics already aligned, teaching time is not deducted from the inspection takt time.

We Sequentially Overlap Three Types of Sensors

In the registration zone, we use three sensors with different measurement dimensions in sequence. There is a reason for this order: the output from each stage determines the accuracy of the next.

In the 1D stage, a displacement sensor scans the cross-section to obtain a one-dimensional profile, from which the system calculates feature points in the height direction. At the same time, the system performs an initial verification by comparing these values against the reference specifications. If the shape deviates significantly from the specifications, the system filters it out at this stage and skips the subsequent alignment and inspection altogether. This prevents us from spending optical alignment time on obvious defects.

In the 2D stage, a camera captures the product from above, detects its outer boundary, and fills in the planar coordinates. This is where the 1D results are used again. Because the camera is mounted at a fixed height, a taller product has a shorter working distance, which changes the actual distance represented by each pixel. In other words, even if the same number of pixels is measured, the conversion to millimeters can differ depending on the product height. So the system first corrects the resolution using the height obtained from the 1D stage, and then calculates the planar dimensions. Changing this order would make this correction impossible.

In the 3D stage, the system calculates the profiler’s initial position based on the results from the previous two stages and moves it into position. It then makes several measurements while fine-tuning the position to determine whether the target is properly within the measurement range.

All three stages share one common premise: the system must first define what constitutes a feature point. This definition is newly established for each product family. A curved, symmetric component and a stepped machined part require different points to be observed, with different numbers and names. But the procedure for finding them is the same.

[Figure 1] The procedure for finding feature points is similar even when the product cross-sections differ

For any target, the system first finds the vertices, then scans outward and inward from them to identify points where the slope changes, classifying them as shoulder lines and curvature inflection points, and calculates the width from those coordinates. When the product family changes, a new definition file is created, but the algorithm itself is reused.

Where to Use Rule-Based Methods and Where to Use Deep Learning

Even within the same system, we use different methods at different stages.

The 1D stage is rule-based. The cross-sectional profile consists of clearly defined physical quantities, and what matters most here is reproducibility rather than accuracy. If feature points are identified using deep learning, it can be difficult to trace why a particular position was produced when the value is wrong. Without being able to identify the cause in the field, it is difficult to respond. With rules defined by conditions and thresholds, the logs show exactly which condition caused the result.

The 2D stage uses deep learning. Top-view images contain conveyor rollers, reflected light, and variations in illumination, resulting in a complex background. Extracting the outer boundary using brightness thresholds can easily fail when the lighting changes even slightly. For detecting complex boundaries in particular, the segmentation model proved much more robust.

However, we do not use the deep learning output as-is. Since the model can sometimes mistake part of the product for the background, the system applies morphological operations to fill holes and clean up the boundaries before calculating dimensions. The system also does not measure from just one direction. It draws measurement lines at multiple angles, removes the upper and lower extremes, and averages the remaining values. This prevents a single detection error from affecting the final dimension.

Finally, the system compares the resulting measurement values with the reference specifications for the product variant. Depending on the difference between the two values, it determines whether to use the measured value as-is or pull it toward the reference value. This prevents a failed measurement from being directly used for alignment.

Two Key Decisions in Signal Processing

Earlier, we said that the 1D stage is rule-based. But the results differed significantly depending on how those rules were defined.

The raw profile cannot be used as-is. There is slight tilt in the conveyor, while surface texture, dust, and light flare appear as spikes.

[Figure 2] Profiles Before and After Preprocessing

The first decision concerned floor-surface processing. In the past, we fixed the floor value as a constant, but this would become inaccurate whenever the specifications changed or the conveyor developed a tilt. So the algorithm now estimates the floor-surface tilt through regression, converts it into an angle, and rotates the entire coordinate system to level it before detecting feature points. This produces consistent results even when the product is placed at an angle.

The second decision concerned the criterion for identifying feature points. A common approach is to use a relative threshold—for example, setting the threshold as a certain percentage of the maximum slope observed in a given sample. But when product size and tilt vary from sample to sample, the criterion itself changes every time. Even with the same algorithm, the results can differ from left to right, and the values can fluctuate when the sample changes.

So we switched to a physical absolute criterion. The algorithm identifies a feature point only when the instantaneous slope exceeds a certain value and the drop is also greater than a certain value. We also considered using the second derivative to detect inflection points, but excluded it because the results varied depending on the sensor’s profile shape.

Moving One Axis Moves Two Axes on the Screen

What is more difficult than identifying the shape is moving the optical system to the desired position.

If the sensor is not installed perfectly orthogonally or the optical characteristics of the lens come into play, moving only the mechanical X-axis motor causes the profile on the sensor screen to move not only horizontally but also vertically. The screen coordinates and mechanical coordinates do not correspond one-to-one and interfere with each other.

When a person performs the alignment, they compensate for this interference by feel. They move X, pull Z back to compensate for the resulting offset, and then X becomes misaligned again. Even experienced operators take time to do this, and the final position varies from person to person. This approach cannot be used for automation.

Instead, the system quantifies this interference once.

[Figure 3] Procedure for Calculating the Sensitivity Matrix

The system makes several scans while moving each axis by small increments from a reference position, and calculates the actual displacement on the sensor screen through profile registration. The relationship between mechanical movement and sensor displacement is represented by a sensitivity matrix. Multiplying its inverse matrix by the target displacement vector produces the motor control values that cancel out the interference in a single operation.

A single matrix, however, has its limitations. Since sensitivity changes with the FOV and the curvature of the target, it must be calculated separately for each sensor and each segment.

The main benefit is not absolute accuracy, but consistency. The same input produces the same result every time, there is no room for human judgment to intervene, and the same calculation can be applied to new product variants.

The System Must Know Whether Alignment Has Been Achieved

There is an easily overlooked aspect of automated alignment. Moving the position is only half the job; verifying that the resulting position is correct is equally important.

If inspection begins without proper alignment, the inspection result itself cannot be trusted. There is no way to tell whether an OK result means there is no defect or the defect is simply outside the camera’s field of view. This is why the system performs multiple measurements while adjusting the position in the 3D stage described earlier. It determines on its own whether the target is within the measurement range, and if it is not, it recalibrates the axes using the inverse matrix.

If the target fails to enter the measurement range within the predefined number of attempts, the system marks the case as an alignment failure. Separating it without producing an inspection result is better than allowing an unreliable judgment to pass as normal.

We Do Not Perform Movements That Are Not Necessary

Knowing the target position does not mean we move the target to the exact center of the screen. Large axis movements increase motor load and vibration, and also take more time.

So the engineer first defines the zone in which the feature points need to be located. If the target is already within that zone, the system sets the movement distance to zero. If it is outside, the system moves only the minimum distance required to bring it inside the boundary.

The same principle applies to top-and-bottom inspection. After inspecting the top and flipping the product to inspect the bottom, the system does not perform the bottom alignment from scratch. Instead, it reuses the alignment result calculated for the top. This applies only to product variants for which the assumption of top-bottom symmetry holds, but it reduces takt time accordingly.

What We Confirmed in the First Application

Our first mass-production application was a multi-variant tire line. From the perspective of automated alignment, it was a challenging environment and therefore a suitable validation case. There were more than thousands of specifications, shape variations between individual tires even within the same specification before curing, and significant reflection and texture noise from the rubber surface.

What we focused on in validation was reproducibility, not absolute accuracy. We rotated the same sample by 36 degrees and measured it five times under each of ten conditions, for a total of 50 measurements, to verify whether the same feature points consistently produced the same values. The purpose of alignment is to place the inspection optics in the same position every time, so a small absolute bias is not a problem as long as the variation is small. The repeatability variation of the key feature points was within the alignment tolerance required by the customer.

However, the variation in width was relatively large. A single cross-sectional scan cannot adequately represent the entire shape. Therefore, we use the 2D result, which observes the entire cross-section, as the final width value, while the 1D width is used only as a reference.

What Is Reused and What Must Be Redefined

The principle is simple when moving to a different target: redefine the parts that depend on values specific to the product, and reuse the parts that determine how those values are obtained.

Three things need to be redefined: the type and number of feature points, the reference specifications for each product variant, and the training data for the 2D detection model. The first two come from drawings and specification sheets, while the last is trained using good-part images from the relevant line.

Everything else can be reused: the separation of the registration and inspection zones; the sequential use of 1D, 2D, and 3D; floor-surface rotation correction and absolute-criterion feature-point detection; the method of canceling interference between axes using the inverse matrix; and the entry decision and minimum-movement principle. None of these components needs to know what the product is.

That means the same framework can be applied to multi-variant, precision-alignment inspections such as machined parts with different cross-sectional shapes, stacked components that need to be aligned based on thickness or steps, and rotationally symmetric parts with multiple specifications.

Closing the Six-Part Series

In Part 1, we identified five barriers: defect data accumulates slowly, defect types are imbalanced, labeling errors are introduced, defects unseen during training emerge, and the number of product variants keeps growing.

Across the five parts, we described how we approached each of these challenges. But the common thread was not the solutions themselves; it was how we look at the problems. All five are not exceptions in the field—they are the default. Data is inherently scarce, labels are inherently imperfect, product variants naturally increase, and products are naturally positioned slightly differently each time.

A system designed with these conditions treated as exceptions may work well in a PoC, only to stop working in mass production. If they are treated as the default and the system is designed accordingly, it becomes a system that improves the longer the line runs. That is the story we wanted to tell across these six parts.

[AI Inspection: Challenges in the Field] Series

Part 1. Starting AI Inspection Without Defective Products
Part 2. What Actually Goes Wrong When Detecting Anomalies Using Only Normal Samples
Part 3. Why AIVEX Combines Supervised and Unsupervised Learning in a Single Model
Part 4. How We Processed Thousands of Product Variants with a Single Model
Part 5. What to Consider When Generating Defect Data for Model Training