The 1.1 "Subword" theme matches names that share no whole word. German
compounds and scanner noise keep records apart that a reader sees at once as
the same workshop, because Holzbaugesellschaft and Holzbau are two
different words to a whole-word tokenizer. joinery can now learn a vocabulary
of word pieces from your own text and match on those pieces instead.
find_subwords() learns a subword vocabulary from a character vector or from
columns of a table, on any backend. It returns a fitted model that carries
the trained vocabulary with it, so you can save it with saveRDS() and get
the same splits in a later session. Both algorithms SentencePiece offers are
available, byte-pair encoding and unigram.
subword_tokens(sw) puts that model in a strategy formula in place of
word_tokens(). The pieces are ordinary tokens, so rarity weighting,
blocking, thresholds and explain_match() behave as they did. The package's
acceptance test measures what this buys on twenty workshops written as one
compound in a register and split apart in a directory: word tokens recover
four pairs in ten, a fitted vocabulary recovers all of them, and neither
joins two different workshops.
custom_tokens() takes any tokenizer you like, either a plain function that
returns a list of token vectors or a fitted object with a tokenize()
method. subword_tokens() is a named shortcut for the second form.
Subword matching uses the sentencepiece package, a suggested dependency.
Install it when you want subword tokens. The rest of joinery works without
it.
New article on the website, "Subword tokenizers", covering the fit-and-apply loop, the two settings that decide whether the result is any good, and how to bring your own tokenizer.
match_features() gains identity_by, which names the column saying
which entity each candidate belongs to. Supply it when the side you search
has already been resolved, such as a search over one representative per
entity, and the features gain an ident_cnt column counting how many
distinct entities a candidate set spans. The existing icnt counts
records, which is the same number until several records stand for one
entity. Nothing changes when you leave the new argument out.
An empty result from detect_duplicates() now carries your original
columns on the in-memory backend, matching what the DuckDB backend always
returned. Code that reads a column off a dedup result no longer needs a
special case for the empty run.
sysctl on macOS, /proc/meminfo on Linux). It now reads the memory
budget straight from the connection's own memory_limit setting, which
defaults to 80% of system RAM and reflects any user override. This removes
the shell call that broke on systems without sysctl on the PATH (the
CRAN M1mac check failures, where batch planning errored with "argument is
of length zero"), and makes the budget detection work identically on all
platforms, including Windows.First stable release. The token core, the DuckDB backend, embedding matching, diagnostics, and calibration are feature-complete and the public API is stable. This release adds the documentation that makes the package usable end to end.
workshop_register, workshop_listings, workshop_panel, match_labels_example: synthetic woodworking-workshop data with planted difficulty tiers and ground-truth links, used throughout the articles. Each tier (containment, movers, phonetic twins, hub tokens) has a minority that measurably benefits from the feature it exercises.Staged entity resolution, region-free linking across blocks, and an always-on cost guard, plus embedding reuse and faster preparers.
Run strategies in order, carry residuals forward, resolve entities once at the end.
multi_stage_dedup() and multi_stage_search(): run an ordered list of strategies as successive passes. multi_stage_dedup() finds duplicates within one table; multi_stage_search() links records across tables, or across years of one pooled table with self = TRUE. Both take a mix of exact, fuzzy, and embedding strategies; multi_stage_search() supports collapse-and-continue so a slowly drifting name links one step at a time. Renames multi_stage_match().exact_strategy(): identical-token-set matching as a strategy, for a cheap first pass. Runs through detect_duplicates() and search_candidates() like any strategy. Optional containment matches a subset rather than an exact set, with a per-column min_containment_tokens floor.resolve_entities(): group an edge list into entities (connected components) and pick a representative per group.materialize_records(): fetch the original rows for a set of ids, the complement of extract_unmatched().plan_strategy(): compare blocking keys before matching. Reports each candidate's block sizes, comparison cost, and how many true twins stay co-blocked, without computing any scores.rarity_distribution(): report a column's token frequency and rarity before matching, with a suggested min_rarity.find_stopwords(): list a column's high-frequency, low-information tokens for filter_stopwords().duckdb_control(): one object for DuckDB execution tuning (batch size, scoring chunk key, per-chunk failure policy, progress), passed as control =. Replaces the loose batch arguments.Follow an entity across geographic blocks (movers, name drift, year to year) without giving up block-based cost control.
block_on_tokens(): block on a record's own rare name tokens instead of a fixed key, so two records sharing any rare token are compared wherever they sit. Mix it with plain column names in block_by.rarity_scope = "global": measure rarity across the whole corpus, so a distinctive name reads as strong evidence in any block and a common one stays weak.max_fanout / on_fanout: an automatic ceiling on comparison cost. When a hot or boilerplate token would fan a dense block into a near-quadratic join, joinery drops the offending tokens with a warning ("cap", the default) or stops ("abort"). On by default. Replaces max_comparisons.joinery.embedding_cache_dir to persist across sessions, or joinery.embedding_reuse = FALSE to opt out.clear_embedding_cache(): empty the cache, optionally on disk too.score_embeddings() scores all pairs in a block as one matrix product, dropping a few hundred thousand pairs from seconds to a fraction of a second.drop_short_tokens(): drop tokens below a length, useful after phonetic encoding.as_cologne(), as_soundex(), as_metaphone(), and as_nysiis() now encode token lists as well as raw strings, so you can encode after tokenizing.normalize_street() gains drop_house_numbers and drop_stopwords to strip address noise.word_tokens(), filter_stopwords(), drop_numeric_tokens(), token_shapes(), and extract_initials() now run group-wise over token tables.search_candidates() rejects overlapping id spaces and prepare_search_data() rejects duplicate ids, both of which corrupt results silently otherwise.resolve_entities() no longer drops singletons when ids mix integer and double forms.summarise_matches() (DuckDB) no longer produces an out-of-range histogram bin for scores just above 1.0.drop_joinery_temp_tables() is now exported.Internal consolidation after the calibration work, plus fixes surfaced by a full-scale Yellow-Pages panel build. Output schemas unchanged.
[0, sum(weights)].tbl |> filter(...)) are accepted everywhere.summarise_matches(entity_cols =): count duplicate groups whose listed columns are single-valued, separating real stopword clusters from cardinality artefacts.cli::cli_abort() with rlang argument checks across exported verbs.R/ reorganised under an eight-prefix naming scheme (see CLAUDE.md).An optional post-match filter that learns to drop false positives from a small labelled sample. The same verb works on token and embedding strategies.
match_features(): build a one-row-per-pair feature table from a match result, with token-overlap counts, auxiliary-side informativeness (aIP, after Doherr 2023), and string similarities.fit_filter() / apply_filter(): fit a logistic false-positive filter and apply it, choosing a threshold by Youden's J unless you set one.calibrate_matches(): one verb composing features, fit, and apply.calibrate(): evaluate a fitted filter on a labelled set; returns reliability, Brier score, log-loss, confusion matrix, and a threshold sweep.sample_matches() stratification, plus export_for_labelling() / import_labels() for a CSV round-trip.fit_filter() via joinery_recipe(). All tidymodels packages are optional; the glm path needs none.Verbs to answer four questions about a strategy and its results: will it work, did it work, why this pair, and where to look.
audit_strategy(): grade a strategy before matching.summarise_matches(): overview of a dedup or candidate result, unified across backends.explain_match(): per-token attribution of a single pair's score.sample_matches(): draw pairs by mode (high, low, borderline, ambiguous, top-gap, random).compare_stages(): per-stage coverage for multi-stage workflows.tinyplot functions, one per view.recommendations().Optional semantic matching that complements rather than replaces the token core. Use embeddings for fields where word-overlap fails (paraphrases, multilingual variants, fuzzy free-text descriptions) and combine them with token strategies via multi_stage_match().
embedding_strategy(): declarative strategy for embedding-based linkage, mirroring the ergonomics of search_strategy(). Specify one or more embedding columns, an optional block_by, an optional threshold, and an optional weights vector across embedding columns.normalize flag for users who want to keep raw magnitudes.detect_duplicates(), search_candidates(), and extract_unmatched() all accept an Embedding_Strategy and return the standard joinery output schemas (duplicate_group / match_id, score, rank, original columns).multi_stage_match() accepts a sequence of mixed Search_Strategy and Embedding_Strategy objects, threading residuals between stages and stopping early when either side is exhausted. Useful pattern: cheap token stage first, then embedding stage on the residual.block_by support for embeddings so cosine search runs within blocks (e.g. country, year bucket) instead of across the whole table.tidyllm (optional Suggests dependency): provider-agnostic helpers for Ollama, OpenAI, and other tidyllm-supported backends, so users can move from raw text to a matchable embedding column without leaving R.block_by SQL bug fixed.A maintenance release with no new user-facing features. The goal was to harden the test suite and close coverage gaps before resuming feature work on embeddings and diagnostics.
methods_duckdb.R coverage raised from 34% to 90%; full behavioural parity with the data.table backend now exercised by tests.embedding_methods_* coverage raised to 95%+ on both data.table and DuckDB backends.batch_duckdb brittleness diagnosed and fixed. User-facing impact: small inputs no longer hit pathological batching behaviour.local_tests/.batch_duckdb small-table brittleness.This release implements advanced matching heuristics that significantly improve accuracy and robustness.
rIP Smoothing: Four smoothing methods for token weights:
smoothing(method = "log"): Log transformationsmoothing(method = "softmax", temperature = 1.0): Softmax with temperaturesmoothing(method = "offset", alpha = 0.1): Additive smoothingsmoothing(method = "none"): No smoothing (default)Containment: Control maximum matches per record:
max_candidates parameter limits top-N matchesFeedback Weighting: Penalize low token overlap:
feedback_strength parameter (0-1) controls intensity.score_pairs_sql() helper consolidates scoring logicdetect_duplicates() and search_candidates()