Development has started on the 1.1 "Subword" theme: a pluggable tokenizer
protocol and a corpus-fitted subword tokenizer backed by SentencePiece
(sentencepiece in Suggests). See notes/v1.1/00_index.md for the build queue.
First stable release. The token core, the DuckDB backend, embedding matching, diagnostics, and calibration are feature-complete and the public API is stable. This release adds the documentation that makes the package usable end to end.
workshop_register, workshop_listings, workshop_panel, match_labels_example: synthetic woodworking-workshop data with planted difficulty tiers and ground-truth links, used throughout the articles. Each tier (containment, movers, phonetic twins, hub tokens) has a minority that measurably benefits from the feature it exercises.Staged entity resolution, region-free linking across blocks, and an always-on cost guard, plus embedding reuse and faster preparers.
Run strategies in order, carry residuals forward, resolve entities once at the end.
multi_stage_dedup() and multi_stage_search(): run an ordered list of strategies as successive passes. multi_stage_dedup() finds duplicates within one table; multi_stage_search() links records across tables, or across years of one pooled table with self = TRUE. Both take a mix of exact, fuzzy, and embedding strategies; multi_stage_search() supports collapse-and-continue so a slowly drifting name links one step at a time. Renames multi_stage_match().exact_strategy(): identical-token-set matching as a strategy, for a cheap first pass. Runs through detect_duplicates() and search_candidates() like any strategy. Optional containment matches a subset rather than an exact set, with a per-column min_containment_tokens floor.resolve_entities(): group an edge list into entities (connected components) and pick a representative per group.materialize_records(): fetch the original rows for a set of ids, the complement of extract_unmatched().plan_strategy(): compare blocking keys before matching. Reports each candidate's block sizes, comparison cost, and how many true twins stay co-blocked, without computing any scores.rarity_distribution(): report a column's token frequency and rarity before matching, with a suggested min_rarity.find_stopwords(): list a column's high-frequency, low-information tokens for filter_stopwords().duckdb_control(): one object for DuckDB execution tuning (batch size, scoring chunk key, per-chunk failure policy, progress), passed as control =. Replaces the loose batch arguments.Follow an entity across geographic blocks (movers, name drift, year to year) without giving up block-based cost control.
block_on_tokens(): block on a record's own rare name tokens instead of a fixed key, so two records sharing any rare token are compared wherever they sit. Mix it with plain column names in block_by.rarity_scope = "global": measure rarity across the whole corpus, so a distinctive name reads as strong evidence in any block and a common one stays weak.max_fanout / on_fanout: an automatic ceiling on comparison cost. When a hot or boilerplate token would fan a dense block into a near-quadratic join, joinery drops the offending tokens with a warning ("cap", the default) or stops ("abort"). On by default. Replaces max_comparisons.joinery.embedding_cache_dir to persist across sessions, or joinery.embedding_reuse = FALSE to opt out.clear_embedding_cache(): empty the cache, optionally on disk too.score_embeddings() scores all pairs in a block as one matrix product, dropping a few hundred thousand pairs from seconds to a fraction of a second.drop_short_tokens(): drop tokens below a length, useful after phonetic encoding.as_cologne(), as_soundex(), as_metaphone(), and as_nysiis() now encode token lists as well as raw strings, so you can encode after tokenizing.normalize_street() gains drop_house_numbers and drop_stopwords to strip address noise.word_tokens(), filter_stopwords(), drop_numeric_tokens(), token_shapes(), and extract_initials() now run group-wise over token tables.search_candidates() rejects overlapping id spaces and prepare_search_data() rejects duplicate ids, both of which corrupt results silently otherwise.resolve_entities() no longer drops singletons when ids mix integer and double forms.summarise_matches() (DuckDB) no longer produces an out-of-range histogram bin for scores just above 1.0.drop_joinery_temp_tables() is now exported.Internal consolidation after the calibration work, plus fixes surfaced by a full-scale Yellow-Pages panel build. Output schemas unchanged.
[0, sum(weights)].tbl |> filter(...)) are accepted everywhere.summarise_matches(entity_cols =): count duplicate groups whose listed columns are single-valued, separating real stopword clusters from cardinality artefacts.cli::cli_abort() with rlang argument checks across exported verbs.R/ reorganised under an eight-prefix naming scheme (see CLAUDE.md).An optional post-match filter that learns to drop false positives from a small labelled sample. The same verb works on token and embedding strategies.
match_features(): build a one-row-per-pair feature table from a match result, with token-overlap counts, auxiliary-side informativeness (aIP, after Doherr 2023), and string similarities.fit_filter() / apply_filter(): fit a logistic false-positive filter and apply it, choosing a threshold by Youden's J unless you set one.calibrate_matches(): one verb composing features, fit, and apply.calibrate(): evaluate a fitted filter on a labelled set; returns reliability, Brier score, log-loss, confusion matrix, and a threshold sweep.sample_matches() stratification, plus export_for_labelling() / import_labels() for a CSV round-trip.fit_filter() via joinery_recipe(). All tidymodels packages are optional; the glm path needs none.Verbs to answer four questions about a strategy and its results: will it work, did it work, why this pair, and where to look.
audit_strategy(): grade a strategy before matching.summarise_matches(): overview of a dedup or candidate result, unified across backends.explain_match(): per-token attribution of a single pair's score.sample_matches(): draw pairs by mode (high, low, borderline, ambiguous, top-gap, random).compare_stages(): per-stage coverage for multi-stage workflows.tinyplot functions, one per view.recommendations().Optional semantic matching that complements rather than replaces the token core. Use embeddings for fields where word-overlap fails (paraphrases, multilingual variants, fuzzy free-text descriptions) and combine them with token strategies via multi_stage_match().
embedding_strategy(): declarative strategy for embedding-based linkage, mirroring the ergonomics of search_strategy(). Specify one or more embedding columns, an optional block_by, an optional threshold, and an optional weights vector across embedding columns.normalize flag for users who want to keep raw magnitudes.detect_duplicates(), search_candidates(), and extract_unmatched() all accept an Embedding_Strategy and return the standard joinery output schemas (duplicate_group / match_id, score, rank, original columns).multi_stage_match() accepts a sequence of mixed Search_Strategy and Embedding_Strategy objects, threading residuals between stages and stopping early when either side is exhausted. Useful pattern: cheap token stage first, then embedding stage on the residual.block_by support for embeddings so cosine search runs within blocks (e.g. country, year bucket) instead of across the whole table.tidyllm (optional Suggests dependency): provider-agnostic helpers for Ollama, OpenAI, and other tidyllm-supported backends, so users can move from raw text to a matchable embedding column without leaving R.block_by SQL bug fixed.A maintenance release with no new user-facing features. The goal was to harden the test suite and close coverage gaps before resuming feature work on embeddings and diagnostics.
methods_duckdb.R coverage raised from 34% to 90%; full behavioural parity with the data.table backend now exercised by tests.embedding_methods_* coverage raised to 95%+ on both data.table and DuckDB backends.batch_duckdb brittleness diagnosed and fixed (see notes/batch_duckdb_brittleness.md). User-facing impact: small inputs no longer hit pathological batching behaviour.local_tests/.batch_duckdb small-table brittleness.This release implements advanced matching heuristics that significantly improve accuracy and robustness.
rIP Smoothing: Four smoothing methods for token weights:
smoothing(method = "log"): Log transformationsmoothing(method = "softmax", temperature = 1.0): Softmax with temperaturesmoothing(method = "offset", alpha = 0.1): Additive smoothingsmoothing(method = "none"): No smoothing (default)Containment: Control maximum matches per record:
max_candidates parameter limits top-N matchesFeedback Weighting: Penalize low token overlap:
feedback_strength parameter (0-1) controls intensity.score_pairs_sql() helper consolidates scoring logicdetect_duplicates() and search_candidates()