Model Performance Is Decided by Labels: Annotation Guidelines and Quality Control for Vision AI Data Labeling

Two People, One Image, Two Different Labels

On a visual inspection line, one inspector rejects a 0.3 mm scratch while another passes it. That disagreement flows straight into your labels. If 10% of your evaluation labels are wrong, even a perfect model may plateau at a measured accuracy of around 90%. Label inconsistency sets the ceiling on model performance.

This also tells you when to fix labels before tuning the model. If architecture changes move accuracy by less than a percentage point, and most of the misclassified samples turn out to be images that humans also disagree on, the problem is probably in the labels.

Writing Annotation Guidelines

Class Definitions and Edge Cases

Write each class criterion in measurable terms (e.g., scratches 0.5 mm or longer are defects)

Collect borderline cases separately and state explicit rules for them

Give unjudgeable images, such as out-of-focus or glare-washed shots, their own "cannot judge" class instead of forcing a pass/fail call

Example Images

Show correct examples next to commonly mislabeled ones for every class. One side-by-side pair reduces annotator variance more than a paragraph of text.

Aligning with the Quality Standard

Label criteria must match the plant's quality standard, including its limit samples. Have the quality team review and sign off on the draft guideline. This heads off later disputes that "the model judges differently from our standard."

Measuring Agreement and Controlling Quality

Cross-labeling: Have two or more annotators label 5–10% of the data independently

Agreement metrics: Use Cohen's kappa for two annotators, Fleiss' kappa for three or more, and IoU for boxes and segmentation masks. Raw percent agreement is misleading: when 95% of parts are good, random guessing already agrees about 90% of the time

Gold data: Mix samples with confirmed answers into the queue and track each annotator's accuracy

Revision loop: Gather disagreements weekly, settle them in a review meeting, and publish the decisions in the next guideline version

Operating the Labeling Work

| Model | Strength | Weakness |

|-------|----------|----------|

| In-house plant staff | Deep domain knowledge | Competes with their day jobs, limited throughput |

| External vendor | Throughput and cost efficiency | Weak on edge cases |

| Hybrid | Vendor labels first, in-house staff review edge cases | Needs deliberate process design |

Model pre-labeling speeds things up a lot, but it invites confirmation bias as annotators approve predictions without really checking them. Track approval rates against gold-data accuracy. Active learning sends the samples the model is least sure about to annotators first, aiming for more improvement from the same budget.

Dataset Versioning and Lineage

Link guideline version → dataset version → model version in one traceable history

Split train, validation, and test sets by lot, product, or capture date. Consecutive video frames spread across splits will inflate your test scores

Mask sensitive content such as worker faces, license plates, and drawings before labeling, and set out in the contract what may leave your premises

Pre-Project Checklist

Have class definitions and edge-case rules been agreed with the quality team?

Are an agreement target (e.g., kappa ≥ 0.8) and a review ratio defined?

Has a pilot measured labeling time per image, so schedule and cost can be estimated?

Are split criteria and security handling documented?

POLYGLOTSOFT supports the full vision AI data pipeline, from building models to writing annotation guidelines, managing agreement, and versioning datasets. If you are preparing to automate inspection, we can design it with you starting from the data assessment stage. Reach out through our contact page or KakaoTalk channel.