Why Rule-Based Scorecards Go Stale So Quickly
Most B2B companies start with a rule-based scorecard: 10 points for a whitepaper download, 5 for a pricing page visit, minus 5 for a free email domain. It's easy to set up, but the cracks show fast. The weights reflect a marketer's intuition, and the scorecard stays frozen while products and channels change. After a few "hot" leads go nowhere, sales stops looking at the score, and every inquiry ends up getting the same treatment.
B2B sales cycles often run for months, so even defining a "good lead" is hard. That's why a machine learning project should start not with picking an algorithm, but with deciding exactly what you're predicting: the label.
Preparing Training Data: What Counts as a Conversion?
Choosing the target event
Signed contracts are the most accurate label, but they're rare and slow to arrive. In practice, many teams start with an intermediate event that sits close to revenue and occurs often enough, such as a booked sales meeting or a quote request, and move to a contract-based model once they have more data. Put a time window on the label too, for example "meeting booked within 90 days of the first inquiry."
Combining features and removing leakage
Model Choice and Explainability
Logistic regression stays stable on small datasets, and you can read the direction of each feature's effect straight from the model. Gradient boosting (LightGBM, XGBoost) captures feature interactions and usually wins once you have thousands of records. Compare both on the same time-based split; if the gap is small, the easier-to-explain model will usually get adopted more readily.
Whichever you choose, show reps the reasons behind each score. Turning SHAP values into a short line listing the top three drivers, such as "Manufacturing, 300+ employees; 3 pricing-page visits in the last 7 days; implementation timeline stated in inquiry," does a lot to get reps to trust and use the score.
Putting Scores into the Sales Process
Grade-based SLAs and routing
Grades are easier to run with than raw scores. For example, the top 20% become Grade A and get a call from a senior rep within one business hour. Grade B gets a response within one business day, and Grade C goes into a nurturing track built on newsletters and case studies.
CRM integration and rescoring
Add `lead_score`, `lead_grade`, `score_reasons`, and `scored_at` fields to the CRM, and rescore every active lead in a nightly batch. When a lead climbs from C to A as visits pile up, an automatic alert to the owner keeps that re-engagement signal from slipping through.
Measuring Impact and Retraining
The key metric is conversion lift in the top grade. Say you receive 600 inquiries a year with a 5% overall meeting rate, or 30 meetings. If 18 of the 120 Grade A leads convert, Grade A converts at 15%, three times the average. Part of that lift may simply come from reps working A leads harder, though. To isolate the model's real effect, run an A/B assignment for a period in which a share of leads is routed without regard to score.
Model performance decays as markets and offerings change. Review grade-level conversion rates and key feature distributions every quarter, and consider retraining right after a product launch or pricing change, whatever the schedule says.
A Realistic Starting Point for Smaller Datasets
A few hundred inquiries is enough to begin. With around 60 conversions, a logistic regression with roughly six features is a realistic choice. Run it alongside your existing rule-based score and compare which one better separates the leads that actually convert.
POLYGLOTSOFT supports the full lead scoring lifecycle: CRM data assessment and label definition, model development, CRM field integration, nightly rescoring batches, and performance dashboards. If you need a scoring system your sales team will actually trust, get in touch with us today.