The Same Customer Is in Your ERP Five Times: AI-Based Master Data Deduplication and Cleansing

The Quiet Losses Caused by Duplicate Master Data

Run an ERP for a few years and the same customer ends up registered under several codes: "Hankook Precision Co., Ltd.", "Hankook Precision", and a Korean-script variant of the same name. No error message appears, so nobody treats it as a problem, yet the numbers drift quietly out of line.

Distorted sales and inventory totals: A customer worth ₩500 million a year, split across five codes, looks like five ₩100 million accounts and drops out of the key-account ranking. Credit limits and receivables are tracked separately per code as well.

Duplicate purchasing: When the same part exists under two item codes, a purchase order goes out on one code while stock sits unused on the other, and safety stock is held separately for each.

Stalled AI and BI projects: Before you can forecast demand or build a dashboard, someone has to answer "do these codes refer to the same thing?" Many projects spend their schedule on this cleansing step, not on modeling.

Why Rules Alone Fall Short

The first instinct is a set of replacement rules: strip "Co., Ltd." and "Inc.", remove spaces, and group whatever matches exactly. It helps, but it runs out of road quickly.

Spelling variants: Romanized names, abbreviations, "Precision" versus "Precision Industries", and plain typos cannot all be enumerated.

Free-text item names: "Hex bolt M8x25 SUS304" and "BOLT HEX SUS M8*25L" are the same item, yet the strings barely overlap, because specification, material, and unit are all packed into a single name field.

The threshold dilemma: With one string-similarity score, a cutoff of 0.9 misses real duplicates, while 0.7 merges unrelated companies such as "Daehan Electric" and "Daehan Electronics".

Designing an Entity Matching Pipeline

Normalization and Blocking

Start by standardizing legal-entity suffixes, whitespace, special characters, and full-width versus half-width characters. Then cut down the comparisons. Comparing 20,000 customers against each other produces roughly 200 million pairs, so you compare only records that share a block, such as the first characters of the name, the last digits of the phone number, or the district in the address. That brings the candidates down to a few hundred thousand pairs. Use several blocking keys together so a typo in one field does not hide a match.

Combining Match Signals

Combine several signals instead of relying on one metric.

String similarity: Edit distance and character n-grams catch typos and spacing differences.

Embedding similarity: Catches pairs that mean the same thing but share few characters, such as Korean versus English names or reordered item descriptions.

A classifier: Takes those scores plus address, representative, and phone matches as inputs and outputs the probability that two records are the same entity. A few hundred pairs labeled by your own staff are enough to get started.

Separating Confirmed Keys from Inferred Matches

A matching business registration number is a confirmed match, not an inference. Group those records first, without sending them through the model. Keep in mind that one registration number can cover several sites and delivery addresses, so merge the customer while keeping ship-to locations as child attributes. Only records with a missing or mistyped number go on to inferred matching.

Using an LLM as a Judgment Aid

LLMs are good at recognizing that "M8x25" and "M8*25L" describe the same specification. Calling one on every candidate pair, however, is too slow and too expensive, and the same pair can receive different answers on different runs. It is safer to apply the LLM only to the ambiguous score band, store each verdict and its reasoning for reuse, and constrain the output to "same / different / cannot determine".

What People Still Have to Decide

Three Threshold Bands

Divide the score into three bands instead of drawing one line. For example, 0.92 and above merges automatically, 0.70 to 0.92 goes to a review queue, and anything below is left alone. Set the boundaries by inspecting samples yourself, and keep the auto-merge band narrower for customers than for items, since a wrong customer merge costs more.

The Surviving Code and Historical Transactions

You need a stated principle for which code survives. A common choice is the code with recent activity, the most transactions, and the most complete mandatory fields. Do not overwrite codes on historical documents. Keep an old-code-to-surviving-code mapping table and translate at query and aggregation time, which is far easier to defend in an audit.

Undo and Change History

Merges can be wrong. Deactivate old codes instead of deleting them, and record who merged what, when, and on what basis, so that any single merge can be reversed.

Keeping the Data Clean Afterward

Cleansing is not a one-time job. If the entry points stay the same, duplicates pile up again.

Real-time similar-record warnings: On the new-record screen, show existing entries that resemble the name being typed, and require a reason if the user registers anyway.

Permissions and approval: Reduce the number of people who can create master records, and route requests from business users through an approver.

Named data owners: Assign each master to a department, such as sales administration for customers and purchasing or production engineering for items, and track turnaround time on the review queue as an operating metric.

Getting Started with POLYGLOTSOFT

POLYGLOTSOFT supports the whole flow: diagnosing duplicates in customer and item masters, building the matching pipeline, developing the review screens, integrating code mapping with your ERP, and adding similar-record warnings at registration. The work integrates with your existing ERP, with no replacement required, and our subscription development service lets you start small with a diagnosis and a pilot cleansing run. If master data is blocking your reporting and analytics, contact POLYGLOTSOFT.