Methods
What ERBOT does at each stage, and the two design principles that define it: missing-value-aware everything, and truth that never leaks.
The ten stages
er_load()Reads CSVs and parses ground truth into a canonical record table. Text is normalized once, upstream of everything — so every downstream stage sees the same strings.
er_diagnose()Profiles each field: type, missingness, cardinality. Recommends blocking keys and warns about fields that will never discriminate (e.g. 98% missing).
er_block()Generates candidate pairs without the O(n²) blowup: standard blocking, prefix blocking, or sorted-neighborhood. Missing keys become singletons — never silently paired.
er_similarity()missing-value-aware per-field scores — Jaro-Winkler, Levenshtein, token Jaccard, bag-of-words, categorical, numeric. If either side is missing, the field contributes NA, never an imputed guess.
er_combine()Fuses per-field similarities into one pair score with missing-value-aware weighting: fields that are missing simply don't vote. No leakage from imputation.
er_weights()Learns field weights three ways: uniform, Fellegi–Sunter EM on the unlabeled pairs, or ARI-guided search on a labeled fit set. The default never touches truth.
er_classify()Pairwise match / non-match decisions: converts the fused similarity matrix S into a binary match graph M (thresholding or a 2-component GMM via EM). The eleven supervised classifiers below learn P(match | per-field similarities) from labeled pairs. Classification is pair-level and supervised — clustering comes next, as a separate stage.
er_cluster() / er_cluster_all()Nine unsupervised methods (threshold connected-components, Louvain, Leiden, HDBSCAN, …) group records into entities from the match graph. Clustering is record-level and unsupervised: it consumes the classifier's pair decisions, it does not replace them. Structural inapplicability returns valid=FALSE with a reason — never a silent garbage label.
er_merge()Post-processing: consensus merging across methods, small-cluster absorption. The default merge is consensus — the old truth-picking 'best' now warns loudly.
er_evaluate()ARI, B-cubed, AMI and friends. Predictions are aligned to ground truth by record id, by name. A single mismatched id is a loud error, not a wrong number in a table.
Honest evaluation
Entity resolution has a reproducibility problem: the ground truth used to score a method also leaks into choosing it — tuning the threshold on the test set, picking the best of twelve methods by test ARI, splitting records instead of entities so near-duplicates appear in both train and test.
ERBOT's defaults address all three:
- Truth is evaluation-only. Ground truth drives metrics, never clustering, classifier training, or method selection — unless you explicitly set
supervised_selection = TRUE. - Entity-disjoint splits.
er_split()ander_stratified_three_way_split()partition by entity, so no duplicate family appears on both sides of a split. Tuning uses Fit/Validation; Test is touched once. - Protocol B is a real held-out. Tune with entity-disjoint CV on the Selection set, then cluster and evaluate the Test set exactly once.
Eleven supervised classifiers
Each classifier learns a pairwise match probability from per-field similarities — not from embedding differences. Every one carries a validity discipline: when the data makes a method structurally inapplicable (rank-deficient design, separation, missing packages), it returns valid = FALSE with a human-readable reason, and the pipeline falls back to Louvain with a warning. Inapplicability is reported as inapplicability — never as a bad score.
Missing-value-aware by construction
Real-world records are messy: missing phones, empty addresses, blank names. ERBOT never imputes a similarity it didn't observe. A missing field yields NA for that pair-field, and the combiner simply re-weights the fields that are present. Blocking treats missing keys as singletons rather than pairing every null with every null.