Which strings count as the gold answer?

Three rules for deciding that an answer string is the gold, measured on the 5,255 questions of the Campaign-4 HotPotQA frozen test set: exact match (the current rule: equality after normalization), token F1 ≥ .5 (the HotPotQA F1 on normalized tokens, with the digit guard) and Zipf-weighted F1 ≥ .5 (every token counted with the weight 9 − Zipf(token), the word's surprisal in decades of general English, with the digit guard). The candidate space of a question is the prior enumeration of Qwen3-8B (every string with prior ≥ 10⁻⁴ at temperature 2) plus the gold and its casing, slash and "or" variants. A rule's gold class is the gold plus every candidate it accepts; the class mass is the summed prior of the class, which is what the back-off prior of the reward becomes under that rule. Normalization (lower case, articles and punctuation removed) is applied under every rule; the digit guard requires equal multisets of digit groups in the raw strings ("1.5" never equals "15"). Study code: artifacts/hotpotqa-matching-20261027/ in thinking-experiments at 8d92fce.

The prior under each rule

Distribution of the gold-class mass over the 5,255 questions. Today's prior is the exact-match row; ε = 5 × 10⁻⁵ is the reward's floor, below which the Other bucket earns no gradient.

rule	median	2% / 98% quantiles	mean	share below ε
exact match	4.7e-04	1.1e-20 / .9989	0.159	0.411
token F1 ≥ .5 + digit guard	.0106	5.2e-19 / .9992	0.217	0.217
Zipf-weighted F1 ≥ .5 + digit guard	7.1e-03	4.5e-19 / .9991	0.207	0.246

Both F1 rules lift the median by a factor 15 to 22 and halve the share of questions below ε. The 2% quantile stays twenty decades below ε under every rule: at the bottom of the distribution the prior does not know the gold entity at all, and no matcher helps there.

Mass placed on the wrong side of the class, LM-judged

For every rule and direction, 200 draws proportional to prior mass, with replacement, from (a) the class members that are not the gold surface and (b) the candidates outside the class; 874 distinct pairs in all, each labelled once by a Claude judge given the question, the gold and the candidate (not the paragraphs). Criterion: name, format and casing variants and more specific forms of the gold are "same"; less specific forms, other referents sharing words, changed numbers, units, dates or polarity are "different". The share of draws labelled wrong estimates the share of the pool's mass that is on the wrong side; multiplied by the pool mass per question it is the wrongly placed mass per question (bold). Brackets: 95% binomial interval on the share, in percent.

rule	class mass per question that is not the gold (false positive) = share of the non-gold class mass	correct mass per question left outside the class (false negative) = share of the outside mass
exact match	0 by construction	0.073 = 13.5% of 0.543 [8.8, 18.2]
token F1 ≥ .5 + digit guard	0.021 = 36.8% of 0.058 [30.1, 43.4]	0.022 = 4.5% of 0.486 [1.6, 7.4]
Zipf-weighted F1 ≥ .5 + digit guard	0.013 = 26.5% of 0.047 [20.4, 32.6]	0.022 = 4.5% of 0.496 [1.6, 7.4]

Exact match leaves .073 of correct mass per question outside the gold class, five decades more than its median class mass; both F1 rules recover two thirds of it at the price of .013 to .021 of wrong mass admitted. The Zipf weights remove a third of the plain F1's wrong admissions (the shared-generic-token merges) at no measurable cost in misses. What remains wrongly admitted under the Zipf weights is mostly less specific forms and two-way choices, which no token weighting can separate from the gold; what remains missed is abbreviations and nicknames with no shared token.

Caveats. The judge is a language model without the paragraphs; on category golds it was lenient ("skyscraper" for "building", "singers" for "musician" labelled same), so the false-negative shares are upper bounds. One pair was labelled unsure and counts one half. The 200 draws of the two F1 false-negative pools overlap in only 5 pairs; their equal shares are a coincidence within the intervals.

Examples: the heaviest judged pairs (prior candidates)

Per rule and direction, the pairs with the largest prior mass that the judge put on the other side of the rule. "prior" is the candidate's enumerated prior probability; "F1" and "Zipf F1" are the plain and the Zipf-weighted token F1 of the pair (both rules accept at .5 when the digit guard also passes; a pair with F1 ≥ .5 outside the F1 class failed the digit guard).

exact match: outside the class, judged same (false negative)

prior	F1	Zipf F1	gold	candidate	judge	question
.0290	0.67	0.70	Ferdinand	Rio Ferdinand	full name of Ferdinand	A celebrity magazine that has featured Snoop Dogg and footballers such as Riyad Mahrez and the creator's team-mate Juan Mata was launched by which former Manchester United player?
3.2e-03	0.00	0.00	Greenwich Village, New York City	NYC	question asks the city; NYC is it	The director of the romantic comedy "Big Stone Gap" is based in what New York city?
1.2e-03	0.50	0.39	Mexico City	México City	accent variant	Mundo Perdido was influenced by an ancient Mesoamerican city located in a sub-valley of the Valley of Mexico, that is 40 km northeast of what modern day city?
2.2e-04	0.15	0.18	Modern thinkers associated with classical realism are Carl von Clausewitz	human nature realism	human nature realism = classical realism	The Prussian General Carl von Clausewitz is associated with what type of realism?
3.4e-05	0.00	0.00	Australian	Australia	country form of Australian	In what country was the actor Chris Hemsworth, one of the stars in the motion picture Thor: The Dark World, born?
1.9e-05	0.50	0.47	film director	film directors	plural form	What was the profession of Cate Shortland and Sidney Franklin?
7.3e-06	0.40	0.41	Anglican Church	Church of England	Church of England is the Anglican Church here	Elizabeth Wordsworth's father was bishop in what church?
7.1e-06	0.67	0.79	Yanzhou	Yanzhou District	District suffix added, same place	Which city is under Jining, Kaiyuan, Liaoning or Yanzhou District?

token F1 ≥ .5 + digit guard: in the class, judged different (false positive)

prior	F1	Zipf F1	gold	candidate	judge	question
.2639	0.73	0.70	Academy Award for Best Art Direction	Academy Award for Best Director	other award	The 2008 American romantic drama film directed by David Fincher won what directing award?
.1711	0.50	0.44	Secretary of Defense	United States Secretary of State	other office	The winner of the 1984 United States Senate election in Maine took on what political role from 1997-2001?
.1370	0.67	0.51	Battle of Guam	Battle of Manila	opposite choice	Which battle occurred first, the Battle of Manila or the Battle of Guam?
.0976	0.50	0.37	Berea College	Carleton College	different college	James Fieser received his B.A. from what liberal arts work college that was founded in 1855?
.0856	0.50	0.49	George II	George III	different monarch	Thomas Major was the engraver to the estranged son of which monarch?
.0777	0.50	0.60	the summer solstice	the winter solstice	winter vs summer solstice	This annual event held in several major cities in Finland is one of many all-night arts festival whose name is given to areas of high latitude in the weeks around what?
.0509	0.67	0.53	University of Vienna	University of Virginia	Vienna vs Virginia	An American who is a German literature translator and did a thesis on an Austrian prodigy born in 1874 taught at which university?
.0414	0.50	0.53	Lake County, Illinois	Illinois	county dropped, less specific	Marine Air Control Group 48 is stationed at Naval Station Great Lakes which located where?

token F1 ≥ .5 + digit guard: outside the class, judged same (false negative)

prior	F1	Zipf F1	gold	candidate	judge	question
4.1e-05	0.00	0.00	magazine	Jewish magazines	more specific form	Tikkun and Moment are both what?
1.8e-05	0.67	0.71	712 5th Avenue	712 Fifth Avenue	5th/Fifth formatting	Which tower is taller, 712 Fifth Avenue or Manhattan Life Insurance Building?
4.3e-06	0.00	0.00	Pacific Lutheran University (PLU), founded by Norwegian Lutheran pioneers in 1890	after	gold implies PLU earlier, so after	Was California State University, Dominguez Hills founded before or after Pacific Lutheran University?
3.1e-06	0.12	0.17	Lostprophets disbanded in 2013 after Watkins was charged with sexual offences in late 2012.	Ian Watkins	gold sentence identifies Watkins	Josey Scott and Ian Watkins were both promising musicians. Which of these talented men was incarcerated, impacting his career with a rock band?
1.2e-06	0.00	0.00	the Nazi Party	National Socialism	National Socialism is the Nazi movement	Friedrich Ratzel coined the phrase Lebensraum; which political movement made it notorious by taking it to extremes?
1.1e-06	0.40	0.45	Alphonse Gabriel Capone	Al Capone	nickname of same person	Dean O'Banion was the main rival of which mobster, known as Scarface?
1.1e-06	0.00	0.00	fourteen members	14	numeral form of fourteen	How many members does the Tennessee Volunteers football team's Conference have?
1.0e-06	0.00	0.00	opera	operatic	adjective form	What genre of composers were both Claudio Monteverdi and Georges Bizet?

Zipf-weighted F1 ≥ .5 + digit guard: in the class, judged different (false positive)

prior	F1	Zipf F1	gold	candidate	judge	question
.1370	0.67	0.51	Battle of Guam	Battle of Manila	opposite choice	Which battle occurred first, the Battle of Manila or the Battle of Guam?
.1000	0.50	0.55	the Nazi Party	Nazi Germany	party vs state, different referent	Friedrich Ratzel coined the phrase Lebensraum; which political movement made it notorious by taking it to extremes?
1.0e-02	0.80	0.74	WBO lightweight title	Lightweight Title	less specific, missing WBO	Jimmy Garcia lost by unanimous decision to a professional boxer that challenged for what title in 1995?
3.8e-03	0.67	0.68	Robert Michael John Fox	Michael Fox	Michael Fox not a used name for Robert Fox	Which son of Robin Fox MC, is an English theater and film producer
1.4e-03	0.67	0.68	Club	The Club International	names the UK original, not the spinoff	Which one of Magna Publishing Group's tiles is a spinoff of the UK's Club International?
5.6e-04	0.67	0.53	Latin American culture	Afro-Latin American culture	Afro-Latin narrows referent	The Carnaval del Pueblo is Eurpoe's largest celebration of what expression of the people of Latin America?
2.5e-04	0.67	0.62	Blue	Blue Spider	other act	What act for Innocent Records achieved Platinum sales and shares its name with a primary color in the RGB color model?
2.6e-05	0.50	0.52	American rocket engineer	engineer	less specific	What was the occupation of the man who played a leading role in developing a man-portable recoilless anti-tank rocket launcher weapon?

Zipf-weighted F1 ≥ .5 + digit guard: outside the class, judged same (false negative)

prior	F1	Zipf F1	gold	candidate	judge	question
.2282	0.40	0.36	Clive Staples Lewis	C.S. Lewis	C.S. Lewis = Clive Staples Lewis	What British novelist born in November 1898 wrote Surprised by Joy: The Shape of My Early Life?
1.9e-05	0.00	0.00	musician	singers	singers are musicians; both are singers	Tom Araya and Marcus Mumford were both what?
8.9e-06	0.29	0.33	Makoto, born July 15, 1980	Makoto Koshinaka	full name of Makoto	Who is younger, Dan Bejar and Makoto Koshinaka?
4.5e-06	0.50	0.48	Sven Magnus	Magnus Carlsen	Sven Magnus Carlsen	which Norwegian chess grandmaster was Espen Agdestein his manager
3.1e-06	0.12	0.17	Lostprophets disbanded in 2013 after Watkins was charged with sexual offences in late 2012.	Ian Watkins	gold sentence identifies Watkins	Josey Scott and Ian Watkins were both promising musicians. Which of these talented men was incarcerated, impacting his career with a rock band?
2.1e-06	0.40	0.43	Emperor of Japan	Emperor Hirohito	Hirohito was that Emperor; more specific	Who was the head of the Imperial Family that was making an broadcast announcement of Japan's surrender in 1945?
1.6e-06	0.50	0.49	rock band	alternative rock	more specific rock subgenre, both fit	What kind of music do The Stone Roses and The Lemonheads play?
1.5e-06	0.00	0.00	building	skyscraper	skyscraper is a more specific building	What type of structure does 277 Park Avenue and 126 Madison Avenue have in common?

The rules on the model's own hypotheses (eps 5e-5 cell, test rollouts)

Same three rules applied to every hypothesis listed by the final checkpoint of the ε = 5e-5 cell on the frozen test set (63,060 attempts, 62,841 valid, 4 samples per question and stratum). Coverage is the share of valid attempts with at least one accepted hypothesis; p(gold) is the mean stated mass on accepted hypotheses.

rule	stratum 0 coverage	p(gold)	stratum 1 coverage	p(gold)	stratum 2 coverage	p(gold)
exact match	0.787	0.558	0.617	0.370	0.357	0.200
token F1 ≥ .5 + digit guard	0.926	0.724	0.775	0.494	0.499	0.275
Zipf-weighted F1 ≥ .5 + digit guard	0.925	0.719	0.765	0.486	0.474	0.263

Wrongly placed hypotheses, LM-judged as above but with draws uniform over hypothesis instances (a pair listed by many attempts is drawn proportionally more often; "attempts" in the example tables is that multiplicity): 200 draws per pool, 975 distinct pairs. The share of wrong draws times the pool size (number of hypothesis instances in the pool) is the estimated number of wrongly placed hypothesis instances (bold); brackets are the 95% binomial interval on the share, in percent.

rule	accepted hypotheses that are not the gold (false positives) = share of the accepted non-identical instances	rejected hypotheses that are the gold (false negatives) = share of the rejected instances
exact match	0 by construction	29,533 = 15.5% of 190,537 [10.5, 20.5]
token F1 ≥ .5 + digit guard	12,168 = 41.2% of 29,497 [34.4, 48.1]	8,052 = 5.0% of 161,042 [2.0, 8.0]
Zipf-weighted F1 ≥ .5 + digit guard	9,269 = 36.8% of 25,223 [30.1, 43.4]	14,465 = 8.8% of 165,316 [4.8, 12.7]

Error rates. Over the 227,410 hypothesis instances (36,873 identical to the gold after normalization): FPR = wrongly accepted / all wrong instances, FNR = wrongly rejected / all right instances. The denominators are estimated from the same judged shares, so the intervals carry the judge's sampling error (each rate varied over its own share's interval).

rule	FPR	FNR
exact match	0.000 [0.000, 0.000]	0.445 [0.351, 0.515]
token F1 ≥ .5 + digit guard	0.074 [0.062, 0.085]	0.129 [0.056, 0.192]
Zipf-weighted F1 ≥ .5 + digit guard	0.058 [0.048, 0.068]	0.215 [0.131, 0.284]

Variants of the Zipf rule, re-estimated from the same labelled draws (a variant can only flip pairs the baseline's pools contain, so these are exact for the baseline's pools and ignore pairs outside them): the digit guard with thousands separators normalized and Roman numerals I/V/X read as numbers; WordNet synonym tokens inside the F1 (a candidate token is shared with a gold token when the two share a synset); a weighted-recall floor of .9 (the candidate must cover 90% of the gold's token weight, which rejects less specific forms); and the intersection with the ProtoQA WordNet matcher at τ = 1, or that matcher alone. The first two rows are the exact match of the training reward and the plain token F1, re-estimated on the same draws so that all rows share one denominator. Total error = (FP + FN) / all 227,410 hypothesis instances, i.e. one minus accuracy, with equal cost for the two error types. The intervals are 10,000 paired bootstrap resamples of the labelled draws within their pools (Claude's labels, Codex's independent aggregation of 27 October 2026); the last column spans the point estimates under the labels of three independent same-model judges.

rule	FPR	FNR	total error	95% bootstrap interval (Claude's labels)	total error, range over three independent judges
exact match (the training reward's matcher, scoring.answer_match_single)	0.000	0.445	0.132	[0.106, 0.164]	0.153–0.155
token F1 >= .5 + original digit guard (no Zipf weights)	0.077	0.169	0.104	[0.078, 0.137]	0.117–0.123
Zipf F1 >= .5 + original digit guard (baseline)	0.058	0.209	0.103	[0.076, 0.134]	0.116–0.123
Zipf F1 >= .5 + fixed digit guard	0.057	0.197	0.099	[0.073, 0.129]	0.112–0.119
Zipf F1 >= .6 + fixed digit guard	0.034	0.248	0.097	–	–
Zipf F1 >= .5 + fixed guard, WordNet synonym tokens inside the F1	0.063	0.184	0.099	[0.073, 0.129]	0.112–0.119
Zipf F1 >= .6 + fixed guard, WordNet synonym tokens inside the F1	0.034	0.230	0.092	[0.067, 0.123]	0.106–0.112
character Levenshtein similarity >= .8 + fixed guard	0.002	0.420	0.125	–	–
word Levenshtein similarity >= .5 + fixed guard (unit costs, over the longer token count)	0.050	0.241	0.107	–	–
word Levenshtein similarity >= .6 + fixed guard	0.011	0.349	0.111	–	–
Zipf-weighted token edit similarity >= .5 + fixed guard	0.057	0.197	0.099	–	–
Zipf-weighted token edit similarity >= .6 + fixed guard	0.033	0.252	0.098	–	–
Zipf F1 >= .5 + fixed guard, fuzzy tokens (character similarity >= .8)	0.057	0.197	0.099	–	–
Zipf F1 >= .5 + fixed guard + weighted recall >= .9	0.010	0.334	0.106	[0.080, 0.139]	0.130–0.133
Zipf F1 >= .5 + original guard AND ProtoQA WordNet matcher at tau = 1	0.001	0.424	0.126	[0.115, 0.173]	0.164–0.166
ProtoQA WordNet matcher at tau = 1 alone (no F1)	0.001	0.412	0.122	[0.112, 0.169]	0.161–0.163

Definitions. With c and g the token lists of the candidate and the gold after normalization and w(t) = 9 − Zipf(t): shared weight S = Σ_t w(t) · min(count of t in c, count of t in g); precision P = S / Σ_{t in c} w(t); recall R = S / Σ_{t in g} w(t); F1 = 2PR / (P + R), 0 when S = 0 or a yes/no answer meets a different one. In the WordNet variants a candidate token is also paired with a gold token when the two share a synset; tokens are paired one to one by the optimal (Hungarian) assignment and a pair contributes the smaller of the two weights. The recall variant adds R ≥ .9. Raising the threshold from .5 to .6 rejects about two wrong acceptances for every right one lost; the sum FPR + FNR is .264 against .247, so .6 is the better point only if a false positive in the reward costs more than a false negative.

exact match (the training reward's matcher, scoring.answer_match_single): pairs flipped against the baseline

fp->rejected (197 pairs)

attempts	judge	gold	candidate
2	different	French and English	French, English, and Shoshone
2	same	Jango Fett	Jango
2	different	natural-ingredients-only personal care products	personal care products
1	same	"Queen City"	Queen City of Maine
1	same	"personal earnings" (such as salary and wages), "business income" and "capital gains"	personal earnings, business income, capital gains
1	same	100 million	100 million copies
1	different	110	more than 110
1	same	16th	16th largest
1	same	1730	about 1730
1	same	1968	March and April 1968
1	same	200,000	200,000 passengers
1	same	2000	2000, in France
1	same	2005 season	2005
1	same	70 m and 90 m	70 m and 90 m events
1	same	728,000 ft²	728,000 sq ft
1	same	75 mi	about 75 mi
1	same	79 AD	AD 79
1	same	7pm	Monday at 7pm
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	Adam Levine	Adam
1	same	Alan David Sokal	Alan Sokal
1	different	Alice's Adventures in Wonderland	Alice's Adventures in Wonderland and Through the Looking-Glass
1	different	Amazing Stories	Wonder Stories
1	different	American	American comedian
1	different	American writer	writer
1	different	An agricultural cooperative	cooperative association
1	same	Anglican Church	Anglican
1	different	Anne	Anne & Gilbert
1	same	Aqua	Aqua (band)
1	same	Argentine Americans	Argentine American
1	same	Asiana Town building	Asiana Town
1	different	BBC Formula One	BBC
1	different	BBC Formula One	Formula One
1	same	Barton Lee Hazlewood	Lee Hazlewood
1	different	Bath, Maine	Bath
1	different	Battle of Singapore	Battle of the Bulge
1	same	Beatrice Tinsley	Beatrice Muriel Hill Tinsley
1	same	Berber	Berber languages
1	same	Berea College	Berea College, Berea, Kentucky
1	same	Biscayne National Park to the east and Everglades National Park to the west	Biscayne National Park and Everglades National Park
1	different	Blackheart Records	Blackheart Records and Gallo Record Company
1	same	Bosco Wong Chung-chak	Bosco Wong
1	unsure	British Columbia, Canada	Hedley, British Columbia
1	same	British Labour Party	Labour Party
1	same	Brooklyn, New York	Brooklyn
1	same	Béla Viktor János Bartók	Béla Bartók
1	same	Canadian professional ice hockey centre	ice hockey centre
1	same	Carl Zeiss AG	Carl Zeiss
1	different	Charles Carroll of Carrollton	Charles Carroll the Settler
1	different	Corendon Airlines	Corendon Group
1	same	Cosmopolitan of Las Vegas	The Cosmopolitan in Las Vegas
1	same	DI Humphrey Goodman	Detective Inspector Humphrey Goodman
1	same	Delphine Software International	Delphine Software
1	same	Denver, Colorado	southwest Denver, Colorado near Bear Creek
1	same	Dominican	Dominican Republic
1	same	Don S. Davis	Don Sinclair Davis
1	same	Doña Ana	Doña Ana County
1	same	Earvin "Magic" Johnson Jr.	Magic Johnson
1	same	East Lancashire Railway	Heritage East Lancashire Railway
1	different	Eielson Air Force Base	Alaska Air Force Base
1	same	Ellie Kemper	Ellie Kemper in the title role
1	same	Felipe de Jesús Calderón Hinojosa	Felipe Calderón Hinojosa
1	same	Ferdinand	Rio Ferdinand
1	same	Firth of Clyde	eastern shore of the Firth of Clyde
1	different	Flatbush section of Brooklyn, New York City	New York City
1	different	Franklin, Indiana	Franklin
1	different	German Type IXC U-boat	Type VIIC U-boat
1	different	Griffon Bruxellois	Griffon
1	same	Hakim Abdullah Jamal	Hakim Jamal
1	same	Harvard University	Harvard University and Harvard
1	same	Heather Graham	Heather Graham Pozzessere
1	same	Hershey, Pennsylvania	Hershey
1	same	Hiroshima and Nagasaki	Atomic bombings of Hiroshima and Nagasaki
1	same	I. helicon	Ionolyce helicon
1	same	Ireland	Republic of Ireland
1	same	J35	General Electric/Allison J35
1	same	Jack Elam	William Scott Elam
1	same	James G. Kiernan	Kiernan, James G.
1	same	Jeffersontown, Kentucky	Jeffersontown
1	different	Joan Feynman	Richard Feynman
1	same	Joe McCoy and Memphis Minnie	Kansas Joe McCoy and Memphis Minnie
1	different	Kansas	Kansas City
1	same	Karnataka, India	Karnataka
1	different	Kentucky Music Hall of Fame	Hall of Fame
1	different	Kentucky, Virginia, and Tennessee	Virginia, Tennessee, and North Carolina
1	same	King Edward II	King Edward II of England
1	different	Kings Point, New York	Kings Point
1	same	Kurt Julian Weill	Kurt Weill
1	same	Las Vegas	Las Vegas, Nevada
1	same	Lieutenant Colonel Iceal E. "Gene" Hambleton	Iceal E. "Gene" Hambleton
1	same	Linda Maria Ronstadt	Linda Ronstadt
1	same	London Coliseum	Coliseum Theatre
1	same	Louis "Louie" Zamperini	Louis Silvie "Louie" Zamperini
1	same	Mark August Helfrich	Mark Helfrich
1	different	Marshal of France	Marshal
1	different	Marshal of France	Marshal General of France
1	same	Marvel	Marvel Comics
1	different	Marvel Comics	DC Comics
1	same	Memphis	Memphis, TN
1	different	Menges	Division Menges
1	different	Mercury Seven	Mercury program
1	same	Michael Tippett	Michael Kemp Tippett
1	same	Mississippi University for Women	Mississippi University for Women v. Hogan
1	same	Naomi Elaine Campbell	Naomi Campbell
1	different	New York Republican State Committee	New York Republican State Committee and Cox Enterprises
1	different	New Zealand national team	Australia national team
1	same	Nobel Prize	Nobel Award
1	same	Norse	Norse mythological
1	same	Norwood, Massachusetts	Norwood
1	same	Oedipus Rex	Oedipus Tyrannus
1	different	Ogallala Aquifer	aquifer
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	same	Philip Morris Glass	Philip Glass
1	same	Pierre Nlend Womé	Pierre Womé
1	same	Princess Elizabeth	Queen Elizabeth
1	same	Queen Mathilde	Mathilde
1	same	Reinhard Heydrich	Nazi leader Reinhard Heydrich
1	same	Richard Allen Street	Richard Street
1	same	Richard Dafydd Vivian Llewellyn	Richard Llewellyn
1	different	Richardson, Texas	Richardson Square
1	same	Romance language	Romance
1	same	Ryan Guno Babel	Ryan Babel
1	different	Sam Walton	Sally Walton
1	same	Samuel Francis Du Pont.	Rear Admiral Samuel Francis Du Pont
1	same	Sculptor Gutzon Borglum	Gutzon Borglum
1	different	Silvia Navarro	Silvia Navarro and Sergio Basañez
1	different	Smithfield, Rhode Island	Rhode Island
1	different	South Korean horror film	horror film
1	same	Spyker F1	Spyker F1 Team
1	different	St. Louis Cardinals	St. Louis
1	same	Stephen Roger "Steve" Bruce	Steve Bruce
1	same	Steve McGarrett	Lieutenant Commander Steve McGarrett
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Syracuse University	Syracuse University campus
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Crowned Prince of the Philadelphia Mob	The Crowned Prince
1	same	The Danny Kaye Show	CBS' The Danny Kaye Show
1	same	The Dragon School in Oxford	The Dragon School
1	different	The Lancia Rally	Lancia Delta Group A
1	different	The Stereophonics and The Fabulous Counts	The Fabulous Counts
1	same	Theo James Walcott	Theo Walcott
1	same	Thomas Warburton	Tom Warburton
1	same	Thriller	Thriller film
1	same	Thriller	suspense thriller
1	different	Tony Ducks	Ducks
1	different	Tony Ducks	Tony
1	different	Tsavo East National Park	Tsavo West National Park
1	different	Tudor music and English folk-song	English folk-song
1	different	USS Essex	USS Franklin
1	different	United States Auto Club	Auto Club Speedway
1	same	United States presidential election of 2016	2016 United States presidential election
1	different	Valdosta	Valdosta State University
1	same	Vice President George	Vice President George Mifflin Dallas
1	same	Vladimir Vladimirovich Nabokov	Vladimir Nabokov
1	same	West Virginia Mountaineers	West Virginia University
1	same	Wildhorn, Bricusse and Cuden	Frank Wildhorn, Leslie Bricusse, and Steve Cuden
1	different	Windermere, Cumbria	Windermere Hotel
1	different	World War II	World War I
1	different	Yoo Seung-ho	Seung-ho
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	a syllabic abbreviation from "Triangle Below Canal Street	Triangle Below Canal Street
1	same	against the feather of truth	heart against the feather of truth
1	same	alleged paranormal phenomena	alleged paranormal activities
1	same	an adventure novel	Scottish adventure
1	same	approximately 14,000	14,000
1	same	as loud and dirty as possible	loud and dirty
1	different	bat and ball	ball
1	same	dinosaur	green dinosaur
1	different	end of the 18th century	18th century
1	same	family	grass family
1	same	film lyricist	film lyricist and poet
1	same	flowering plants	flowering plant
1	different	great man	great
1	same	high school diploma or GED	high school diploma
1	same	ice hockey	professional ice hockey
1	same	in the late 1960s	late 1960s
1	same	mathematician	German mathematician
1	same	middleweight	middleweight division
1	same	modern-day New York City	New York City
1	same	more than 70 countries	more than 70
1	same	much better fuel economy	attractive fuel economy
1	different	nearly 80 years	80 years
1	different	northern mockingbird	Southern Mockingbird
1	same	originated in the United Kingdom.	United Kingdom
1	different	political satire black comedy film	black comedy
1	different	research university	university
1	same	secondary school study	secondary school
1	same	seventh generation	seventh
1	different	short story writer	writer
1	different	singer and songwriter	songwriter
1	different	singer and songwriter	singer
1	different	small forward	power forward
1	different	the "First Family of Competitive Eating"	International Federation of Competitive Eating
1	different	the State House in Augusta	Augusta
1	different	underground hip hop	hip hop
1	different	video game	game
1	same	yen	Japanese yen

token F1 >= .5 + original digit guard (no Zipf weights): pairs flipped against the baseline

fp->rejected (7 pairs)

attempts	judge	gold	candidate
1	same	1968	March and April 1968
1	same	728,000 ft²	728,000 sq ft
1	same	Jack Elam	William Scott Elam
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Lancia Rally	Lancia Delta Group A
1	different	the State House in Augusta	Augusta

fn->accepted (8 pairs)

attempts	judge	gold	candidate
1	same	20	20 principal operations
1	same	British	British racing driver
1	different	Kona coast	Hawaiian coast
1	same	New York	Midtown Manhattan, New York City
1	different	Phoenician alphabet	Greek alphabet
1	different	dream pop influence	pop
1	same	five	five separate bills
1	different	the University College of North Staffordshire	University of Maryland, College Park

Zipf F1 >= .5 + fixed digit guard: pairs flipped against the baseline

fp->rejected (1 pairs)

attempts	judge	gold	candidate
1	different	World War II	World War I

fn->accepted (1 pairs)

attempts	judge	gold	candidate
1	same	1800000 sqft	1,800,000 square feet

Zipf F1 >= .6 + fixed digit guard: pairs flipped against the baseline

fp->rejected (52 pairs)

attempts	judge	gold	candidate
1	same	1968	March and April 1968
1	same	728,000 ft²	728,000 sq ft
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	Amazing Stories	Wonder Stories
1	different	American	American comedian
1	different	An agricultural cooperative	cooperative association
1	same	Argentine Americans	Argentine American
1	different	BBC Formula One	BBC
1	different	Battle of Singapore	Battle of the Bulge
1	same	Brooklyn, New York	Brooklyn
1	different	Charles Carroll of Carrollton	Charles Carroll the Settler
1	same	Denver, Colorado	southwest Denver, Colorado near Bear Creek
1	same	Earvin "Magic" Johnson Jr.	Magic Johnson
1	different	Flatbush section of Brooklyn, New York City	New York City
1	different	German Type IXC U-boat	Type VIIC U-boat
1	different	Griffon Bruxellois	Griffon
1	same	I. helicon	Ionolyce helicon
1	same	Ireland	Republic of Ireland
1	same	J35	General Electric/Allison J35
1	same	Jack Elam	William Scott Elam
1	different	Joan Feynman	Richard Feynman
1	same	London Coliseum	Coliseum Theatre
1	different	Marvel Comics	DC Comics
1	different	Mercury Seven	Mercury program
1	different	New Zealand national team	Australia national team
1	same	Nobel Prize	Nobel Award
1	same	Oedipus Rex	Oedipus Tyrannus
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	same	Princess Elizabeth	Queen Elizabeth
1	different	Richardson, Texas	Richardson Square
1	different	Sam Walton	Sally Walton
1	different	Silvia Navarro	Silvia Navarro and Sergio Basañez
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Lancia Rally	Lancia Delta Group A
1	same	Thomas Warburton	Tom Warburton
1	different	USS Essex	USS Franklin
1	different	United States Auto Club	Auto Club Speedway
1	different	Windermere, Cumbria	Windermere Hotel
1	different	World War II	World War I
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	an adventure novel	Scottish adventure
1	different	bat and ball	ball
1	same	family	grass family
1	same	flowering plants	flowering plant
1	different	northern mockingbird	Southern Mockingbird
1	different	political satire black comedy film	black comedy
1	different	short story writer	writer
1	different	singer and songwriter	singer
1	different	small forward	power forward
1	different	the "First Family of Competitive Eating"	International Federation of Competitive Eating
1	different	the State House in Augusta	Augusta

Zipf F1 >= .5 + fixed guard, WordNet synonym tokens inside the F1: pairs flipped against the baseline

fp->rejected (1 pairs)

attempts	judge	gold	candidate
1	different	World War II	World War I

fn->accepted (3 pairs)

attempts	judge	gold	candidate
1	same	1800000 sqft	1,800,000 square feet
1	same	Argentinian	Argentine
1	different	short story writer	author

Zipf F1 >= .6 + fixed guard, WordNet synonym tokens inside the F1: pairs flipped against the baseline

fp->rejected (49 pairs)

attempts	judge	gold	candidate
1	same	1968	March and April 1968
1	same	728,000 ft²	728,000 sq ft
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	Amazing Stories	Wonder Stories
1	different	American	American comedian
1	different	An agricultural cooperative	cooperative association
1	different	BBC Formula One	BBC
1	different	Battle of Singapore	Battle of the Bulge
1	same	Brooklyn, New York	Brooklyn
1	different	Charles Carroll of Carrollton	Charles Carroll the Settler
1	same	Denver, Colorado	southwest Denver, Colorado near Bear Creek
1	same	Earvin "Magic" Johnson Jr.	Magic Johnson
1	different	Flatbush section of Brooklyn, New York City	New York City
1	different	German Type IXC U-boat	Type VIIC U-boat
1	different	Griffon Bruxellois	Griffon
1	same	I. helicon	Ionolyce helicon
1	same	Ireland	Republic of Ireland
1	same	J35	General Electric/Allison J35
1	same	Jack Elam	William Scott Elam
1	different	Joan Feynman	Richard Feynman
1	same	London Coliseum	Coliseum Theatre
1	different	Marvel Comics	DC Comics
1	different	Mercury Seven	Mercury program
1	different	New Zealand national team	Australia national team
1	same	Oedipus Rex	Oedipus Tyrannus
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	same	Princess Elizabeth	Queen Elizabeth
1	different	Richardson, Texas	Richardson Square
1	different	Sam Walton	Sally Walton
1	different	Silvia Navarro	Silvia Navarro and Sergio Basañez
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Lancia Rally	Lancia Delta Group A
1	same	Thomas Warburton	Tom Warburton
1	different	USS Essex	USS Franklin
1	different	United States Auto Club	Auto Club Speedway
1	different	Windermere, Cumbria	Windermere Hotel
1	different	World War II	World War I
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	an adventure novel	Scottish adventure
1	different	bat and ball	ball
1	same	family	grass family
1	different	northern mockingbird	Southern Mockingbird
1	different	political satire black comedy film	black comedy
1	different	short story writer	writer
1	different	singer and songwriter	singer
1	different	small forward	power forward
1	different	the "First Family of Competitive Eating"	International Federation of Competitive Eating
1	different	the State House in Augusta	Augusta

fn->accepted (1 pairs)

attempts	judge	gold	candidate
1	same	Argentinian	Argentine

character Levenshtein similarity >= .8 + fixed guard: pairs flipped against the baseline

fp->rejected (188 pairs)

attempts	judge	gold	candidate
2	different	French and English	French, English, and Shoshone
2	same	Jango Fett	Jango
2	different	natural-ingredients-only personal care products	personal care products
1	same	"Queen City"	Queen City of Maine
1	same	"personal earnings" (such as salary and wages), "business income" and "capital gains"	personal earnings, business income, capital gains
1	same	100 million	100 million copies
1	different	110	more than 110
1	same	16th	16th largest
1	same	1730	about 1730
1	same	1968	March and April 1968
1	same	200,000	200,000 passengers
1	same	2000	2000, in France
1	same	2005 season	2005
1	same	70 m and 90 m	70 m and 90 m events
1	same	728,000 ft²	728,000 sq ft
1	same	75 mi	about 75 mi
1	same	79 AD	AD 79
1	same	7pm	Monday at 7pm
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	Adam Levine	Adam
1	same	Alan David Sokal	Alan Sokal
1	different	Alice's Adventures in Wonderland	Alice's Adventures in Wonderland and Through the Looking-Glass
1	different	Amazing Stories	Wonder Stories
1	different	American	American comedian
1	different	American writer	writer
1	different	An agricultural cooperative	cooperative association
1	same	Anglican Church	Anglican
1	different	Anne	Anne & Gilbert
1	same	Aqua	Aqua (band)
1	same	Asiana Town building	Asiana Town
1	different	BBC Formula One	BBC
1	different	BBC Formula One	Formula One
1	same	Barton Lee Hazlewood	Lee Hazlewood
1	different	Bath, Maine	Bath
1	different	Battle of Singapore	Battle of the Bulge
1	same	Beatrice Tinsley	Beatrice Muriel Hill Tinsley
1	same	Berber	Berber languages
1	same	Berea College	Berea College, Berea, Kentucky
1	same	Biscayne National Park to the east and Everglades National Park to the west	Biscayne National Park and Everglades National Park
1	different	Blackheart Records	Blackheart Records and Gallo Record Company
1	same	Bosco Wong Chung-chak	Bosco Wong
1	unsure	British Columbia, Canada	Hedley, British Columbia
1	same	British Labour Party	Labour Party
1	same	Brooklyn, New York	Brooklyn
1	same	Béla Viktor János Bartók	Béla Bartók
1	same	Canadian professional ice hockey centre	ice hockey centre
1	same	Carl Zeiss AG	Carl Zeiss
1	different	Charles Carroll of Carrollton	Charles Carroll the Settler
1	different	Corendon Airlines	Corendon Group
1	same	DI Humphrey Goodman	Detective Inspector Humphrey Goodman
1	same	Delphine Software International	Delphine Software
1	same	Denver, Colorado	southwest Denver, Colorado near Bear Creek
1	same	Dominican	Dominican Republic
1	same	Don S. Davis	Don Sinclair Davis
1	same	Doña Ana	Doña Ana County
1	same	Earvin "Magic" Johnson Jr.	Magic Johnson
1	same	East Lancashire Railway	Heritage East Lancashire Railway
1	different	Eielson Air Force Base	Alaska Air Force Base
1	same	Ellie Kemper	Ellie Kemper in the title role
1	same	Felipe de Jesús Calderón Hinojosa	Felipe Calderón Hinojosa
1	same	Ferdinand	Rio Ferdinand
1	same	Firth of Clyde	eastern shore of the Firth of Clyde
1	different	Flatbush section of Brooklyn, New York City	New York City
1	different	Franklin, Indiana	Franklin
1	different	German Type IXC U-boat	Type VIIC U-boat
1	different	Griffon Bruxellois	Griffon
1	same	Hakim Abdullah Jamal	Hakim Jamal
1	same	Harvard University	Harvard University and Harvard
1	same	Heather Graham	Heather Graham Pozzessere
1	same	Hershey, Pennsylvania	Hershey
1	same	Hiroshima and Nagasaki	Atomic bombings of Hiroshima and Nagasaki
1	same	I. helicon	Ionolyce helicon
1	same	Ireland	Republic of Ireland
1	same	J35	General Electric/Allison J35
1	same	Jack Elam	William Scott Elam
1	same	James G. Kiernan	Kiernan, James G.
1	same	Jeffersontown, Kentucky	Jeffersontown
1	different	Joan Feynman	Richard Feynman
1	different	Kansas	Kansas City
1	same	Karnataka, India	Karnataka
1	different	Kentucky Music Hall of Fame	Hall of Fame
1	different	Kentucky, Virginia, and Tennessee	Virginia, Tennessee, and North Carolina
1	same	King Edward II	King Edward II of England
1	different	Kings Point, New York	Kings Point
1	same	Kurt Julian Weill	Kurt Weill
1	same	Las Vegas	Las Vegas, Nevada
1	same	Lieutenant Colonel Iceal E. "Gene" Hambleton	Iceal E. "Gene" Hambleton
1	same	Linda Maria Ronstadt	Linda Ronstadt
1	same	London Coliseum	Coliseum Theatre
1	same	Louis "Louie" Zamperini	Louis Silvie "Louie" Zamperini
1	same	Mark August Helfrich	Mark Helfrich
1	different	Marshal of France	Marshal
1	different	Marshal of France	Marshal General of France
1	same	Marvel	Marvel Comics
1	different	Marvel Comics	DC Comics
1	same	Memphis	Memphis, TN
1	different	Menges	Division Menges
1	different	Mercury Seven	Mercury program
1	same	Michael Tippett	Michael Kemp Tippett
1	same	Naomi Elaine Campbell	Naomi Campbell
1	different	New York Republican State Committee	New York Republican State Committee and Cox Enterprises
1	different	New Zealand national team	Australia national team
1	same	Nobel Prize	Nobel Award
1	same	Norse	Norse mythological
1	same	Norwood, Massachusetts	Norwood
1	same	Oedipus Rex	Oedipus Tyrannus
1	different	Ogallala Aquifer	aquifer
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	same	Philip Morris Glass	Philip Glass
1	same	Pierre Nlend Womé	Pierre Womé
1	same	Princess Elizabeth	Queen Elizabeth
1	same	Queen Mathilde	Mathilde
1	same	Reinhard Heydrich	Nazi leader Reinhard Heydrich
1	same	Richard Allen Street	Richard Street
1	same	Richard Dafydd Vivian Llewellyn	Richard Llewellyn
1	different	Richardson, Texas	Richardson Square
1	same	Romance language	Romance
1	same	Ryan Guno Babel	Ryan Babel
1	different	Sam Walton	Sally Walton
1	same	Samuel Francis Du Pont.	Rear Admiral Samuel Francis Du Pont
1	same	Sculptor Gutzon Borglum	Gutzon Borglum
1	different	Silvia Navarro	Silvia Navarro and Sergio Basañez
1	different	Smithfield, Rhode Island	Rhode Island
1	different	South Korean horror film	horror film
1	same	Spyker F1	Spyker F1 Team
1	different	St. Louis Cardinals	St. Louis
1	same	Stephen Roger "Steve" Bruce	Steve Bruce
1	same	Steve McGarrett	Lieutenant Commander Steve McGarrett
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Syracuse University	Syracuse University campus
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Crowned Prince of the Philadelphia Mob	The Crowned Prince
1	same	The Danny Kaye Show	CBS' The Danny Kaye Show
1	same	The Dragon School in Oxford	The Dragon School
1	different	The Lancia Rally	Lancia Delta Group A
1	different	The Stereophonics and The Fabulous Counts	The Fabulous Counts
1	same	Theo James Walcott	Theo Walcott
1	same	Thriller	Thriller film
1	same	Thriller	suspense thriller
1	different	Tony Ducks	Ducks
1	different	Tony Ducks	Tony
1	different	Tudor music and English folk-song	English folk-song
1	different	USS Essex	USS Franklin
1	different	United States Auto Club	Auto Club Speedway
1	same	United States presidential election of 2016	2016 United States presidential election
1	different	Valdosta	Valdosta State University
1	same	Vice President George	Vice President George Mifflin Dallas
1	same	Vladimir Vladimirovich Nabokov	Vladimir Nabokov
1	same	West Virginia Mountaineers	West Virginia University
1	same	Wildhorn, Bricusse and Cuden	Frank Wildhorn, Leslie Bricusse, and Steve Cuden
1	different	Windermere, Cumbria	Windermere Hotel
1	different	World War II	World War I
1	different	Yoo Seung-ho	Seung-ho
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	a syllabic abbreviation from "Triangle Below Canal Street	Triangle Below Canal Street
1	same	alleged paranormal phenomena	alleged paranormal activities
1	same	an adventure novel	Scottish adventure
1	same	approximately 14,000	14,000
1	same	as loud and dirty as possible	loud and dirty
1	different	bat and ball	ball
1	same	dinosaur	green dinosaur
1	different	end of the 18th century	18th century
1	same	family	grass family
1	same	film lyricist	film lyricist and poet
1	different	great man	great
1	same	high school diploma or GED	high school diploma
1	same	ice hockey	professional ice hockey
1	same	in the late 1960s	late 1960s
1	same	mathematician	German mathematician
1	same	middleweight	middleweight division
1	same	modern-day New York City	New York City
1	same	more than 70 countries	more than 70
1	same	much better fuel economy	attractive fuel economy
1	different	nearly 80 years	80 years
1	same	originated in the United Kingdom.	United Kingdom
1	different	political satire black comedy film	black comedy
1	different	research university	university
1	same	secondary school study	secondary school
1	same	seventh generation	seventh
1	different	short story writer	writer
1	different	singer and songwriter	songwriter
1	different	singer and songwriter	singer
1	different	small forward	power forward
1	different	the "First Family of Competitive Eating"	International Federation of Competitive Eating
1	different	the State House in Augusta	Augusta
1	different	underground hip hop	hip hop
1	different	video game	game
1	same	yen	Japanese yen

fn->accepted (1 pairs)

attempts	judge	gold	candidate
1	same	Ulanhot	Ulan Hot

word Levenshtein similarity >= .5 + fixed guard (unit costs, over the longer token count): pairs flipped against the baseline

fp->rejected (40 pairs)

attempts	judge	gold	candidate
1	different	110	more than 110
1	same	1968	March and April 1968
1	same	2000	2000, in France
1	same	728,000 ft²	728,000 sq ft
1	same	79 AD	AD 79
1	same	7pm	Monday at 7pm
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	An agricultural cooperative	cooperative association
1	different	BBC Formula One	BBC
1	different	Blackheart Records	Blackheart Records and Gallo Record Company
1	unsure	British Columbia, Canada	Hedley, British Columbia
1	same	Brooklyn, New York	Brooklyn
1	same	Denver, Colorado	southwest Denver, Colorado near Bear Creek
1	same	Ellie Kemper	Ellie Kemper in the title role
1	different	Flatbush section of Brooklyn, New York City	New York City
1	same	Ireland	Republic of Ireland
1	same	J35	General Electric/Allison J35
1	same	Jack Elam	William Scott Elam
1	same	James G. Kiernan	Kiernan, James G.
1	different	Kentucky, Virginia, and Tennessee	Virginia, Tennessee, and North Carolina
1	same	London Coliseum	Coliseum Theatre
1	different	Marshal of France	Marshal
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	different	Silvia Navarro	Silvia Navarro and Sergio Basañez
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Crowned Prince of the Philadelphia Mob	The Crowned Prince
1	different	The Lancia Rally	Lancia Delta Group A
1	different	Tudor music and English folk-song	English folk-song
1	different	United States Auto Club	Auto Club Speedway
1	different	Valdosta	Valdosta State University
1	different	World War II	World War I
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	an adventure novel	Scottish adventure
1	different	bat and ball	ball
1	different	political satire black comedy film	black comedy
1	different	short story writer	writer
1	different	singer and songwriter	songwriter
1	different	singer and songwriter	singer
1	different	the State House in Augusta	Augusta

fn->accepted (2 pairs)

attempts	judge	gold	candidate
1	different	Kona coast	Hawaiian coast
1	different	Phoenician alphabet	Greek alphabet

word Levenshtein similarity >= .6 + fixed guard: pairs flipped against the baseline

fp->rejected (133 pairs)

attempts	judge	gold	candidate
2	different	French and English	French, English, and Shoshone
2	same	Jango Fett	Jango
1	same	"Queen City"	Queen City of Maine
1	same	"personal earnings" (such as salary and wages), "business income" and "capital gains"	personal earnings, business income, capital gains
1	different	110	more than 110
1	same	16th	16th largest
1	same	1730	about 1730
1	same	1968	March and April 1968
1	same	200,000	200,000 passengers
1	same	2000	2000, in France
1	same	2005 season	2005
1	same	728,000 ft²	728,000 sq ft
1	same	79 AD	AD 79
1	same	7pm	Monday at 7pm
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	Adam Levine	Adam
1	different	Alice's Adventures in Wonderland	Alice's Adventures in Wonderland and Through the Looking-Glass
1	different	Amazing Stories	Wonder Stories
1	different	American	American comedian
1	different	American writer	writer
1	different	An agricultural cooperative	cooperative association
1	same	Anglican Church	Anglican
1	different	Anne	Anne & Gilbert
1	same	Aqua	Aqua (band)
1	same	Argentine Americans	Argentine American
1	different	BBC Formula One	BBC
1	different	Bath, Maine	Bath
1	same	Beatrice Tinsley	Beatrice Muriel Hill Tinsley
1	same	Berber	Berber languages
1	same	Berea College	Berea College, Berea, Kentucky
1	different	Blackheart Records	Blackheart Records and Gallo Record Company
1	unsure	British Columbia, Canada	Hedley, British Columbia
1	same	Brooklyn, New York	Brooklyn
1	same	Béla Viktor János Bartók	Béla Bartók
1	different	Charles Carroll of Carrollton	Charles Carroll the Settler
1	different	Corendon Airlines	Corendon Group
1	same	DI Humphrey Goodman	Detective Inspector Humphrey Goodman
1	same	Denver, Colorado	southwest Denver, Colorado near Bear Creek
1	same	Dominican	Dominican Republic
1	same	Earvin "Magic" Johnson Jr.	Magic Johnson
1	same	Ellie Kemper	Ellie Kemper in the title role
1	same	Ferdinand	Rio Ferdinand
1	same	Firth of Clyde	eastern shore of the Firth of Clyde
1	different	Flatbush section of Brooklyn, New York City	New York City
1	different	Franklin, Indiana	Franklin
1	different	German Type IXC U-boat	Type VIIC U-boat
1	different	Griffon Bruxellois	Griffon
1	same	Harvard University	Harvard University and Harvard
1	same	Hershey, Pennsylvania	Hershey
1	same	Hiroshima and Nagasaki	Atomic bombings of Hiroshima and Nagasaki
1	same	I. helicon	Ionolyce helicon
1	same	Ireland	Republic of Ireland
1	same	J35	General Electric/Allison J35
1	same	Jack Elam	William Scott Elam
1	same	James G. Kiernan	Kiernan, James G.
1	same	Jeffersontown, Kentucky	Jeffersontown
1	different	Joan Feynman	Richard Feynman
1	different	Kansas	Kansas City
1	same	Karnataka, India	Karnataka
1	different	Kentucky, Virginia, and Tennessee	Virginia, Tennessee, and North Carolina
1	different	Kings Point, New York	Kings Point
1	same	London Coliseum	Coliseum Theatre
1	different	Marshal of France	Marshal
1	same	Marvel	Marvel Comics
1	different	Marvel Comics	DC Comics
1	same	Memphis	Memphis, TN
1	different	Menges	Division Menges
1	different	Mercury Seven	Mercury program
1	different	New Zealand national team	Australia national team
1	same	Nobel Prize	Nobel Award
1	same	Norse	Norse mythological
1	same	Norwood, Massachusetts	Norwood
1	same	Oedipus Rex	Oedipus Tyrannus
1	different	Ogallala Aquifer	aquifer
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	same	Princess Elizabeth	Queen Elizabeth
1	same	Queen Mathilde	Mathilde
1	same	Reinhard Heydrich	Nazi leader Reinhard Heydrich
1	same	Richard Dafydd Vivian Llewellyn	Richard Llewellyn
1	different	Richardson, Texas	Richardson Square
1	same	Romance language	Romance
1	different	Sam Walton	Sally Walton
1	different	Silvia Navarro	Silvia Navarro and Sergio Basañez
1	different	South Korean horror film	horror film
1	same	Stephen Roger "Steve" Bruce	Steve Bruce
1	same	Steve McGarrett	Lieutenant Commander Steve McGarrett
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Crowned Prince of the Philadelphia Mob	The Crowned Prince
1	same	The Dragon School in Oxford	The Dragon School
1	different	The Lancia Rally	Lancia Delta Group A
1	different	The Stereophonics and The Fabulous Counts	The Fabulous Counts
1	same	Thomas Warburton	Tom Warburton
1	same	Thriller	Thriller film
1	same	Thriller	suspense thriller
1	different	Tony Ducks	Ducks
1	different	Tony Ducks	Tony
1	different	Tudor music and English folk-song	English folk-song
1	different	USS Essex	USS Franklin
1	different	United States Auto Club	Auto Club Speedway
1	same	United States presidential election of 2016	2016 United States presidential election
1	different	Valdosta	Valdosta State University
1	same	Wildhorn, Bricusse and Cuden	Frank Wildhorn, Leslie Bricusse, and Steve Cuden
1	different	Windermere, Cumbria	Windermere Hotel
1	different	World War II	World War I
1	different	Yoo Seung-ho	Seung-ho
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	a syllabic abbreviation from "Triangle Below Canal Street	Triangle Below Canal Street
1	same	an adventure novel	Scottish adventure
1	same	approximately 14,000	14,000
1	same	as loud and dirty as possible	loud and dirty
1	different	bat and ball	ball
1	same	dinosaur	green dinosaur
1	different	end of the 18th century	18th century
1	same	family	grass family
1	same	film lyricist	film lyricist and poet
1	same	flowering plants	flowering plant
1	different	great man	great
1	same	mathematician	German mathematician
1	same	middleweight	middleweight division
1	same	much better fuel economy	attractive fuel economy
1	different	northern mockingbird	Southern Mockingbird
1	same	originated in the United Kingdom.	United Kingdom
1	different	political satire black comedy film	black comedy
1	different	research university	university
1	same	seventh generation	seventh
1	different	short story writer	writer
1	different	singer and songwriter	songwriter
1	different	singer and songwriter	singer
1	different	small forward	power forward
1	different	the State House in Augusta	Augusta
1	different	video game	game
1	same	yen	Japanese yen

Zipf-weighted token edit similarity >= .5 + fixed guard: pairs flipped against the baseline

fp->rejected (1 pairs)

attempts	judge	gold	candidate
1	different	World War II	World War I

fn->accepted (1 pairs)

attempts	judge	gold	candidate
1	same	1800000 sqft	1,800,000 square feet

Zipf-weighted token edit similarity >= .6 + fixed guard: pairs flipped against the baseline

fp->rejected (55 pairs)

attempts	judge	gold	candidate
1	same	1968	March and April 1968
1	same	728,000 ft²	728,000 sq ft
1	same	79 AD	AD 79
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	Amazing Stories	Wonder Stories
1	different	American	American comedian
1	different	An agricultural cooperative	cooperative association
1	same	Argentine Americans	Argentine American
1	different	BBC Formula One	BBC
1	different	Battle of Singapore	Battle of the Bulge
1	same	Brooklyn, New York	Brooklyn
1	different	Charles Carroll of Carrollton	Charles Carroll the Settler
1	same	Denver, Colorado	southwest Denver, Colorado near Bear Creek
1	same	Earvin "Magic" Johnson Jr.	Magic Johnson
1	different	Flatbush section of Brooklyn, New York City	New York City
1	different	German Type IXC U-boat	Type VIIC U-boat
1	different	Griffon Bruxellois	Griffon
1	same	I. helicon	Ionolyce helicon
1	same	Ireland	Republic of Ireland
1	same	J35	General Electric/Allison J35
1	same	Jack Elam	William Scott Elam
1	same	James G. Kiernan	Kiernan, James G.
1	different	Joan Feynman	Richard Feynman
1	different	Kentucky, Virginia, and Tennessee	Virginia, Tennessee, and North Carolina
1	same	London Coliseum	Coliseum Theatre
1	different	Marvel Comics	DC Comics
1	different	Mercury Seven	Mercury program
1	different	New Zealand national team	Australia national team
1	same	Nobel Prize	Nobel Award
1	same	Oedipus Rex	Oedipus Tyrannus
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	same	Princess Elizabeth	Queen Elizabeth
1	different	Richardson, Texas	Richardson Square
1	different	Sam Walton	Sally Walton
1	different	Silvia Navarro	Silvia Navarro and Sergio Basañez
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Lancia Rally	Lancia Delta Group A
1	same	Thomas Warburton	Tom Warburton
1	different	USS Essex	USS Franklin
1	different	United States Auto Club	Auto Club Speedway
1	different	Windermere, Cumbria	Windermere Hotel
1	different	World War II	World War I
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	an adventure novel	Scottish adventure
1	different	bat and ball	ball
1	same	family	grass family
1	same	flowering plants	flowering plant
1	different	northern mockingbird	Southern Mockingbird
1	different	political satire black comedy film	black comedy
1	different	short story writer	writer
1	different	singer and songwriter	singer
1	different	small forward	power forward
1	different	the "First Family of Competitive Eating"	International Federation of Competitive Eating
1	different	the State House in Augusta	Augusta

Zipf F1 >= .5 + fixed guard, fuzzy tokens (character similarity >= .8): pairs flipped against the baseline

fp->rejected (1 pairs)

attempts	judge	gold	candidate
1	different	World War II	World War I

fn->accepted (1 pairs)

attempts	judge	gold	candidate
1	same	1800000 sqft	1,800,000 square feet

Zipf F1 >= .5 + fixed guard + weighted recall >= .9: pairs flipped against the baseline

fp->rejected (126 pairs)

attempts	judge	gold	candidate
2	same	Jango Fett	Jango
2	different	natural-ingredients-only personal care products	personal care products
1	same	"personal earnings" (such as salary and wages), "business income" and "capital gains"	personal earnings, business income, capital gains
1	same	2005 season	2005
1	same	728,000 ft²	728,000 sq ft
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	Adam Levine	Adam
1	same	Alan David Sokal	Alan Sokal
1	different	Amazing Stories	Wonder Stories
1	different	American writer	writer
1	different	An agricultural cooperative	cooperative association
1	same	Anglican Church	Anglican
1	same	Argentine Americans	Argentine American
1	same	Asiana Town building	Asiana Town
1	different	BBC Formula One	BBC
1	different	BBC Formula One	Formula One
1	same	Barton Lee Hazlewood	Lee Hazlewood
1	different	Bath, Maine	Bath
1	different	Battle of Singapore	Battle of the Bulge
1	same	Biscayne National Park to the east and Everglades National Park to the west	Biscayne National Park and Everglades National Park
1	same	Bosco Wong Chung-chak	Bosco Wong
1	unsure	British Columbia, Canada	Hedley, British Columbia
1	same	British Labour Party	Labour Party
1	same	Brooklyn, New York	Brooklyn
1	same	Béla Viktor János Bartók	Béla Bartók
1	same	Canadian professional ice hockey centre	ice hockey centre
1	same	Carl Zeiss AG	Carl Zeiss
1	different	Charles Carroll of Carrollton	Charles Carroll the Settler
1	different	Corendon Airlines	Corendon Group
1	same	DI Humphrey Goodman	Detective Inspector Humphrey Goodman
1	same	Delphine Software International	Delphine Software
1	same	Don S. Davis	Don Sinclair Davis
1	same	Earvin "Magic" Johnson Jr.	Magic Johnson
1	different	Eielson Air Force Base	Alaska Air Force Base
1	same	Felipe de Jesús Calderón Hinojosa	Felipe Calderón Hinojosa
1	different	Flatbush section of Brooklyn, New York City	New York City
1	different	Franklin, Indiana	Franklin
1	different	German Type IXC U-boat	Type VIIC U-boat
1	different	Griffon Bruxellois	Griffon
1	same	Hakim Abdullah Jamal	Hakim Jamal
1	same	Hershey, Pennsylvania	Hershey
1	same	I. helicon	Ionolyce helicon
1	same	Jack Elam	William Scott Elam
1	same	Jeffersontown, Kentucky	Jeffersontown
1	different	Joan Feynman	Richard Feynman
1	same	Karnataka, India	Karnataka
1	different	Kentucky Music Hall of Fame	Hall of Fame
1	different	Kentucky, Virginia, and Tennessee	Virginia, Tennessee, and North Carolina
1	different	Kings Point, New York	Kings Point
1	same	Kurt Julian Weill	Kurt Weill
1	same	Lieutenant Colonel Iceal E. "Gene" Hambleton	Iceal E. "Gene" Hambleton
1	same	Linda Maria Ronstadt	Linda Ronstadt
1	same	London Coliseum	Coliseum Theatre
1	same	Mark August Helfrich	Mark Helfrich
1	different	Marshal of France	Marshal
1	different	Marvel Comics	DC Comics
1	different	Mercury Seven	Mercury program
1	same	Naomi Elaine Campbell	Naomi Campbell
1	different	New Zealand national team	Australia national team
1	same	Nobel Prize	Nobel Award
1	same	Norwood, Massachusetts	Norwood
1	same	Oedipus Rex	Oedipus Tyrannus
1	different	Ogallala Aquifer	aquifer
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	same	Philip Morris Glass	Philip Glass
1	same	Pierre Nlend Womé	Pierre Womé
1	same	Princess Elizabeth	Queen Elizabeth
1	same	Queen Mathilde	Mathilde
1	same	Richard Allen Street	Richard Street
1	same	Richard Dafydd Vivian Llewellyn	Richard Llewellyn
1	different	Richardson, Texas	Richardson Square
1	same	Romance language	Romance
1	same	Ryan Guno Babel	Ryan Babel
1	different	Sam Walton	Sally Walton
1	same	Sculptor Gutzon Borglum	Gutzon Borglum
1	different	Smithfield, Rhode Island	Rhode Island
1	different	South Korean horror film	horror film
1	different	St. Louis Cardinals	St. Louis
1	same	Stephen Roger "Steve" Bruce	Steve Bruce
1	different	The Crowned Prince of the Philadelphia Mob	The Crowned Prince
1	same	The Dragon School in Oxford	The Dragon School
1	different	The Lancia Rally	Lancia Delta Group A
1	different	The Stereophonics and The Fabulous Counts	The Fabulous Counts
1	same	Theo James Walcott	Theo Walcott
1	same	Thomas Warburton	Tom Warburton
1	different	Tony Ducks	Ducks
1	different	Tony Ducks	Tony
1	different	Tsavo East National Park	Tsavo West National Park
1	different	Tudor music and English folk-song	English folk-song
1	different	USS Essex	USS Franklin
1	different	United States Auto Club	Auto Club Speedway
1	same	Vladimir Vladimirovich Nabokov	Vladimir Nabokov
1	same	West Virginia Mountaineers	West Virginia University
1	different	Windermere, Cumbria	Windermere Hotel
1	different	World War II	World War I
1	different	Yoo Seung-ho	Seung-ho
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	a syllabic abbreviation from "Triangle Below Canal Street	Triangle Below Canal Street
1	same	alleged paranormal phenomena	alleged paranormal activities
1	same	an adventure novel	Scottish adventure
1	same	approximately 14,000	14,000
1	same	as loud and dirty as possible	loud and dirty
1	different	bat and ball	ball
1	different	end of the 18th century	18th century
1	same	flowering plants	flowering plant
1	different	great man	great
1	same	high school diploma or GED	high school diploma
1	same	in the late 1960s	late 1960s
1	same	modern-day New York City	New York City
1	same	more than 70 countries	more than 70
1	same	much better fuel economy	attractive fuel economy
1	different	nearly 80 years	80 years
1	different	northern mockingbird	Southern Mockingbird
1	same	originated in the United Kingdom.	United Kingdom
1	different	political satire black comedy film	black comedy
1	different	research university	university
1	same	secondary school study	secondary school
1	same	seventh generation	seventh
1	different	short story writer	writer
1	different	singer and songwriter	songwriter
1	different	singer and songwriter	singer
1	different	small forward	power forward
1	different	the "First Family of Competitive Eating"	International Federation of Competitive Eating
1	different	the State House in Augusta	Augusta
1	different	underground hip hop	hip hop
1	different	video game	game

Zipf F1 >= .5 + original guard AND ProtoQA WordNet matcher at tau = 1: pairs flipped against the baseline

fp->rejected (185 pairs)

attempts	judge	gold	candidate
2	different	French and English	French, English, and Shoshone
2	same	Jango Fett	Jango
2	different	natural-ingredients-only personal care products	personal care products
1	same	"Queen City"	Queen City of Maine
1	same	"personal earnings" (such as salary and wages), "business income" and "capital gains"	personal earnings, business income, capital gains
1	same	100 million	100 million copies
1	same	16th	16th largest
1	same	1968	March and April 1968
1	same	200,000	200,000 passengers
1	same	2000	2000, in France
1	same	2005 season	2005
1	same	70 m and 90 m	70 m and 90 m events
1	same	728,000 ft²	728,000 sq ft
1	same	7pm	Monday at 7pm
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	Adam Levine	Adam
1	same	Alan David Sokal	Alan Sokal
1	different	Alice's Adventures in Wonderland	Alice's Adventures in Wonderland and Through the Looking-Glass
1	different	Amazing Stories	Wonder Stories
1	different	American	American comedian
1	different	American writer	writer
1	different	An agricultural cooperative	cooperative association
1	same	Anglican Church	Anglican
1	different	Anne	Anne & Gilbert
1	same	Aqua	Aqua (band)
1	same	Asiana Town building	Asiana Town
1	different	BBC Formula One	BBC
1	different	BBC Formula One	Formula One
1	same	Barton Lee Hazlewood	Lee Hazlewood
1	different	Bath, Maine	Bath
1	different	Battle of Singapore	Battle of the Bulge
1	same	Beatrice Tinsley	Beatrice Muriel Hill Tinsley
1	same	Berber	Berber languages
1	same	Berea College	Berea College, Berea, Kentucky
1	same	Biscayne National Park to the east and Everglades National Park to the west	Biscayne National Park and Everglades National Park
1	different	Blackheart Records	Blackheart Records and Gallo Record Company
1	same	Bosco Wong Chung-chak	Bosco Wong
1	unsure	British Columbia, Canada	Hedley, British Columbia
1	same	Brooklyn, New York	Brooklyn
1	same	Béla Viktor János Bartók	Béla Bartók
1	same	Canadian professional ice hockey centre	ice hockey centre
1	same	Carl Zeiss AG	Carl Zeiss
1	different	Charles Carroll of Carrollton	Charles Carroll the Settler
1	different	Corendon Airlines	Corendon Group
1	same	Cosmopolitan of Las Vegas	The Cosmopolitan in Las Vegas
1	same	DI Humphrey Goodman	Detective Inspector Humphrey Goodman
1	same	Delphine Software International	Delphine Software
1	same	Denver, Colorado	southwest Denver, Colorado near Bear Creek
1	same	Dominican	Dominican Republic
1	same	Don S. Davis	Don Sinclair Davis
1	same	Doña Ana	Doña Ana County
1	same	Earvin "Magic" Johnson Jr.	Magic Johnson
1	same	East Lancashire Railway	Heritage East Lancashire Railway
1	different	Eielson Air Force Base	Alaska Air Force Base
1	same	Ellie Kemper	Ellie Kemper in the title role
1	same	Felipe de Jesús Calderón Hinojosa	Felipe Calderón Hinojosa
1	same	Ferdinand	Rio Ferdinand
1	same	Firth of Clyde	eastern shore of the Firth of Clyde
1	different	Flatbush section of Brooklyn, New York City	New York City
1	different	Franklin, Indiana	Franklin
1	different	German Type IXC U-boat	Type VIIC U-boat
1	different	Griffon Bruxellois	Griffon
1	same	Hakim Abdullah Jamal	Hakim Jamal
1	same	Harvard University	Harvard University and Harvard
1	same	Heather Graham	Heather Graham Pozzessere
1	same	Hershey, Pennsylvania	Hershey
1	same	Hiroshima and Nagasaki	Atomic bombings of Hiroshima and Nagasaki
1	same	I. helicon	Ionolyce helicon
1	same	Ireland	Republic of Ireland
1	same	J35	General Electric/Allison J35
1	same	Jack Elam	William Scott Elam
1	same	James G. Kiernan	Kiernan, James G.
1	same	Jeffersontown, Kentucky	Jeffersontown
1	different	Joan Feynman	Richard Feynman
1	same	Joe McCoy and Memphis Minnie	Kansas Joe McCoy and Memphis Minnie
1	different	Kansas	Kansas City
1	same	Karnataka, India	Karnataka
1	different	Kentucky Music Hall of Fame	Hall of Fame
1	different	Kentucky, Virginia, and Tennessee	Virginia, Tennessee, and North Carolina
1	same	King Edward II	King Edward II of England
1	different	Kings Point, New York	Kings Point
1	same	Kurt Julian Weill	Kurt Weill
1	same	Las Vegas	Las Vegas, Nevada
1	same	Lieutenant Colonel Iceal E. "Gene" Hambleton	Iceal E. "Gene" Hambleton
1	same	Linda Maria Ronstadt	Linda Ronstadt
1	same	London Coliseum	Coliseum Theatre
1	same	Louis "Louie" Zamperini	Louis Silvie "Louie" Zamperini
1	same	Mark August Helfrich	Mark Helfrich
1	different	Marshal of France	Marshal
1	different	Marshal of France	Marshal General of France
1	same	Marvel	Marvel Comics
1	different	Marvel Comics	DC Comics
1	same	Memphis	Memphis, TN
1	different	Menges	Division Menges
1	different	Mercury Seven	Mercury program
1	same	Michael Tippett	Michael Kemp Tippett
1	same	Mississippi University for Women	Mississippi University for Women v. Hogan
1	same	Naomi Elaine Campbell	Naomi Campbell
1	different	New York Republican State Committee	New York Republican State Committee and Cox Enterprises
1	different	New Zealand national team	Australia national team
1	same	Norse	Norse mythological
1	same	Norwood, Massachusetts	Norwood
1	same	Oedipus Rex	Oedipus Tyrannus
1	different	Ogallala Aquifer	aquifer
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	same	Philip Morris Glass	Philip Glass
1	same	Pierre Nlend Womé	Pierre Womé
1	same	Princess Elizabeth	Queen Elizabeth
1	same	Queen Mathilde	Mathilde
1	same	Reinhard Heydrich	Nazi leader Reinhard Heydrich
1	same	Richard Allen Street	Richard Street
1	same	Richard Dafydd Vivian Llewellyn	Richard Llewellyn
1	different	Richardson, Texas	Richardson Square
1	same	Ryan Guno Babel	Ryan Babel
1	different	Sam Walton	Sally Walton
1	same	Samuel Francis Du Pont.	Rear Admiral Samuel Francis Du Pont
1	same	Sculptor Gutzon Borglum	Gutzon Borglum
1	different	Silvia Navarro	Silvia Navarro and Sergio Basañez
1	different	Smithfield, Rhode Island	Rhode Island
1	different	South Korean horror film	horror film
1	same	Spyker F1	Spyker F1 Team
1	different	St. Louis Cardinals	St. Louis
1	same	Stephen Roger "Steve" Bruce	Steve Bruce
1	same	Steve McGarrett	Lieutenant Commander Steve McGarrett
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Syracuse University	Syracuse University campus
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Crowned Prince of the Philadelphia Mob	The Crowned Prince
1	same	The Danny Kaye Show	CBS' The Danny Kaye Show
1	same	The Dragon School in Oxford	The Dragon School
1	different	The Lancia Rally	Lancia Delta Group A
1	different	The Stereophonics and The Fabulous Counts	The Fabulous Counts
1	same	Theo James Walcott	Theo Walcott
1	same	Thomas Warburton	Tom Warburton
1	same	Thriller	Thriller film
1	same	Thriller	suspense thriller
1	different	Tony Ducks	Ducks
1	different	Tony Ducks	Tony
1	different	Tsavo East National Park	Tsavo West National Park
1	different	Tudor music and English folk-song	English folk-song
1	different	USS Essex	USS Franklin
1	different	United States Auto Club	Auto Club Speedway
1	different	Valdosta	Valdosta State University
1	same	Vice President George	Vice President George Mifflin Dallas
1	same	West Virginia Mountaineers	West Virginia University
1	same	Wildhorn, Bricusse and Cuden	Frank Wildhorn, Leslie Bricusse, and Steve Cuden
1	different	Windermere, Cumbria	Windermere Hotel
1	different	World War II	World War I
1	different	Yoo Seung-ho	Seung-ho
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	a syllabic abbreviation from "Triangle Below Canal Street	Triangle Below Canal Street
1	same	against the feather of truth	heart against the feather of truth
1	same	alleged paranormal phenomena	alleged paranormal activities
1	same	an adventure novel	Scottish adventure
1	same	approximately 14,000	14,000
1	same	as loud and dirty as possible	loud and dirty
1	different	bat and ball	ball
1	same	dinosaur	green dinosaur
1	different	end of the 18th century	18th century
1	same	family	grass family
1	same	film lyricist	film lyricist and poet
1	different	great man	great
1	same	high school diploma or GED	high school diploma
1	same	ice hockey	professional ice hockey
1	same	mathematician	German mathematician
1	same	middleweight	middleweight division
1	same	modern-day New York City	New York City
1	same	more than 70 countries	more than 70
1	same	much better fuel economy	attractive fuel economy
1	different	nearly 80 years	80 years
1	different	northern mockingbird	Southern Mockingbird
1	same	originated in the United Kingdom.	United Kingdom
1	different	political satire black comedy film	black comedy
1	different	research university	university
1	same	secondary school study	secondary school
1	same	seventh generation	seventh
1	different	short story writer	writer
1	different	singer and songwriter	songwriter
1	different	singer and songwriter	singer
1	different	small forward	power forward
1	different	the "First Family of Competitive Eating"	International Federation of Competitive Eating
1	different	the State House in Augusta	Augusta
1	different	underground hip hop	hip hop
1	different	video game	game
1	same	yen	Japanese yen

ProtoQA WordNet matcher at tau = 1 alone (no F1): pairs flipped against the baseline

fp->rejected (185 pairs)

attempts	judge	gold	candidate
2	different	French and English	French, English, and Shoshone
2	same	Jango Fett	Jango
2	different	natural-ingredients-only personal care products	personal care products
1	same	"Queen City"	Queen City of Maine
1	same	"personal earnings" (such as salary and wages), "business income" and "capital gains"	personal earnings, business income, capital gains
1	same	100 million	100 million copies
1	same	16th	16th largest
1	same	1968	March and April 1968
1	same	200,000	200,000 passengers
1	same	2000	2000, in France
1	same	2005 season	2005
1	same	70 m and 90 m	70 m and 90 m events
1	same	728,000 ft²	728,000 sq ft
1	same	7pm	Monday at 7pm
1	different	AT&T Center in San Antonio	San Antonio Arena
1	different	Adam Levine	Adam
1	same	Alan David Sokal	Alan Sokal
1	different	Alice's Adventures in Wonderland	Alice's Adventures in Wonderland and Through the Looking-Glass
1	different	Amazing Stories	Wonder Stories
1	different	American	American comedian
1	different	American writer	writer
1	different	An agricultural cooperative	cooperative association
1	same	Anglican Church	Anglican
1	different	Anne	Anne & Gilbert
1	same	Aqua	Aqua (band)
1	same	Asiana Town building	Asiana Town
1	different	BBC Formula One	BBC
1	different	BBC Formula One	Formula One
1	same	Barton Lee Hazlewood	Lee Hazlewood
1	different	Bath, Maine	Bath
1	different	Battle of Singapore	Battle of the Bulge
1	same	Beatrice Tinsley	Beatrice Muriel Hill Tinsley
1	same	Berber	Berber languages
1	same	Berea College	Berea College, Berea, Kentucky
1	same	Biscayne National Park to the east and Everglades National Park to the west	Biscayne National Park and Everglades National Park
1	different	Blackheart Records	Blackheart Records and Gallo Record Company
1	same	Bosco Wong Chung-chak	Bosco Wong
1	unsure	British Columbia, Canada	Hedley, British Columbia
1	same	Brooklyn, New York	Brooklyn
1	same	Béla Viktor János Bartók	Béla Bartók
1	same	Canadian professional ice hockey centre	ice hockey centre
1	same	Carl Zeiss AG	Carl Zeiss
1	different	Charles Carroll of Carrollton	Charles Carroll the Settler
1	different	Corendon Airlines	Corendon Group
1	same	Cosmopolitan of Las Vegas	The Cosmopolitan in Las Vegas
1	same	DI Humphrey Goodman	Detective Inspector Humphrey Goodman
1	same	Delphine Software International	Delphine Software
1	same	Denver, Colorado	southwest Denver, Colorado near Bear Creek
1	same	Dominican	Dominican Republic
1	same	Don S. Davis	Don Sinclair Davis
1	same	Doña Ana	Doña Ana County
1	same	Earvin "Magic" Johnson Jr.	Magic Johnson
1	same	East Lancashire Railway	Heritage East Lancashire Railway
1	different	Eielson Air Force Base	Alaska Air Force Base
1	same	Ellie Kemper	Ellie Kemper in the title role
1	same	Felipe de Jesús Calderón Hinojosa	Felipe Calderón Hinojosa
1	same	Ferdinand	Rio Ferdinand
1	same	Firth of Clyde	eastern shore of the Firth of Clyde
1	different	Flatbush section of Brooklyn, New York City	New York City
1	different	Franklin, Indiana	Franklin
1	different	German Type IXC U-boat	Type VIIC U-boat
1	different	Griffon Bruxellois	Griffon
1	same	Hakim Abdullah Jamal	Hakim Jamal
1	same	Harvard University	Harvard University and Harvard
1	same	Heather Graham	Heather Graham Pozzessere
1	same	Hershey, Pennsylvania	Hershey
1	same	Hiroshima and Nagasaki	Atomic bombings of Hiroshima and Nagasaki
1	same	I. helicon	Ionolyce helicon
1	same	Ireland	Republic of Ireland
1	same	J35	General Electric/Allison J35
1	same	Jack Elam	William Scott Elam
1	same	James G. Kiernan	Kiernan, James G.
1	same	Jeffersontown, Kentucky	Jeffersontown
1	different	Joan Feynman	Richard Feynman
1	same	Joe McCoy and Memphis Minnie	Kansas Joe McCoy and Memphis Minnie
1	different	Kansas	Kansas City
1	same	Karnataka, India	Karnataka
1	different	Kentucky Music Hall of Fame	Hall of Fame
1	different	Kentucky, Virginia, and Tennessee	Virginia, Tennessee, and North Carolina
1	same	King Edward II	King Edward II of England
1	different	Kings Point, New York	Kings Point
1	same	Kurt Julian Weill	Kurt Weill
1	same	Las Vegas	Las Vegas, Nevada
1	same	Lieutenant Colonel Iceal E. "Gene" Hambleton	Iceal E. "Gene" Hambleton
1	same	Linda Maria Ronstadt	Linda Ronstadt
1	same	London Coliseum	Coliseum Theatre
1	same	Louis "Louie" Zamperini	Louis Silvie "Louie" Zamperini
1	same	Mark August Helfrich	Mark Helfrich
1	different	Marshal of France	Marshal
1	different	Marshal of France	Marshal General of France
1	same	Marvel	Marvel Comics
1	different	Marvel Comics	DC Comics
1	same	Memphis	Memphis, TN
1	different	Menges	Division Menges
1	different	Mercury Seven	Mercury program
1	same	Michael Tippett	Michael Kemp Tippett
1	same	Mississippi University for Women	Mississippi University for Women v. Hogan
1	same	Naomi Elaine Campbell	Naomi Campbell
1	different	New York Republican State Committee	New York Republican State Committee and Cox Enterprises
1	different	New Zealand national team	Australia national team
1	same	Norse	Norse mythological
1	same	Norwood, Massachusetts	Norwood
1	same	Oedipus Rex	Oedipus Tyrannus
1	different	Ogallala Aquifer	aquifer
1	same	Petrus Josephus Hubertus (Pierre) Cuypers	Pierre Cuypers
1	same	Philip Morris Glass	Philip Glass
1	same	Pierre Nlend Womé	Pierre Womé
1	same	Princess Elizabeth	Queen Elizabeth
1	same	Queen Mathilde	Mathilde
1	same	Reinhard Heydrich	Nazi leader Reinhard Heydrich
1	same	Richard Allen Street	Richard Street
1	same	Richard Dafydd Vivian Llewellyn	Richard Llewellyn
1	different	Richardson, Texas	Richardson Square
1	same	Ryan Guno Babel	Ryan Babel
1	different	Sam Walton	Sally Walton
1	same	Samuel Francis Du Pont.	Rear Admiral Samuel Francis Du Pont
1	same	Sculptor Gutzon Borglum	Gutzon Borglum
1	different	Silvia Navarro	Silvia Navarro and Sergio Basañez
1	different	Smithfield, Rhode Island	Rhode Island
1	different	South Korean horror film	horror film
1	same	Spyker F1	Spyker F1 Team
1	different	St. Louis Cardinals	St. Louis
1	same	Stephen Roger "Steve" Bruce	Steve Bruce
1	same	Steve McGarrett	Lieutenant Commander Steve McGarrett
1	same	Styal Mill	Quarry Bank Mill (also known as Styal Mill)
1	same	Syracuse University	Syracuse University campus
1	same	Tamaulipas	Mexican state of Tamaulipas
1	different	The Crowned Prince of the Philadelphia Mob	The Crowned Prince
1	same	The Danny Kaye Show	CBS' The Danny Kaye Show
1	same	The Dragon School in Oxford	The Dragon School
1	different	The Lancia Rally	Lancia Delta Group A
1	different	The Stereophonics and The Fabulous Counts	The Fabulous Counts
1	same	Theo James Walcott	Theo Walcott
1	same	Thomas Warburton	Tom Warburton
1	same	Thriller	Thriller film
1	same	Thriller	suspense thriller
1	different	Tony Ducks	Ducks
1	different	Tony Ducks	Tony
1	different	Tsavo East National Park	Tsavo West National Park
1	different	Tudor music and English folk-song	English folk-song
1	different	USS Essex	USS Franklin
1	different	United States Auto Club	Auto Club Speedway
1	different	Valdosta	Valdosta State University
1	same	Vice President George	Vice President George Mifflin Dallas
1	same	West Virginia Mountaineers	West Virginia University
1	same	Wildhorn, Bricusse and Cuden	Frank Wildhorn, Leslie Bricusse, and Steve Cuden
1	different	Windermere, Cumbria	Windermere Hotel
1	different	World War II	World War I
1	different	Yoo Seung-ho	Seung-ho
1	different	Yubin, Yeeun	Sunmi, Yubin
1	same	a syllabic abbreviation from "Triangle Below Canal Street	Triangle Below Canal Street
1	same	against the feather of truth	heart against the feather of truth
1	same	alleged paranormal phenomena	alleged paranormal activities
1	same	an adventure novel	Scottish adventure
1	same	approximately 14,000	14,000
1	same	as loud and dirty as possible	loud and dirty
1	different	bat and ball	ball
1	same	dinosaur	green dinosaur
1	different	end of the 18th century	18th century
1	same	family	grass family
1	same	film lyricist	film lyricist and poet
1	different	great man	great
1	same	high school diploma or GED	high school diploma
1	same	ice hockey	professional ice hockey
1	same	mathematician	German mathematician
1	same	middleweight	middleweight division
1	same	modern-day New York City	New York City
1	same	more than 70 countries	more than 70
1	same	much better fuel economy	attractive fuel economy
1	different	nearly 80 years	80 years
1	different	northern mockingbird	Southern Mockingbird
1	same	originated in the United Kingdom.	United Kingdom
1	different	political satire black comedy film	black comedy
1	different	research university	university
1	same	secondary school study	secondary school
1	same	seventh generation	seventh
1	different	short story writer	writer
1	different	singer and songwriter	songwriter
1	different	singer and songwriter	singer
1	different	small forward	power forward
1	different	the "First Family of Competitive Eating"	International Federation of Competitive Eating
1	different	the State House in Augusta	Augusta
1	different	underground hip hop	hip hop
1	different	video game	game
1	same	yen	Japanese yen

fn->accepted (1 pairs)

attempts	judge	gold	candidate
1	same	Argentinian	Argentine

Examples: the most listed judged pairs

exact match: outside the class, judged same (false negative)

attempts	F1	Zipf F1	gold	candidate	judge	question
12.0000	0.00	0.00	cocktails	cocktail	Singular/plural variant	The Polar Bear and Daiquiri are both examples of what?
12.0000	0.80	0.70	Division I	NCAA Division I	formatting variant	In what NCAA division is the team Michael Beasley played basketball for in college?
12.0000	0.80	0.73	Blake Tollison Shelton	Blake Shelton	drops middle name	Who's fifth album and debut single are Startin' Fires and Austin respectively?
12.0000	0.67	0.72	Marine Corps	United States Marine Corps	full name of branch	In which branch of the U.S. military did the man who Robert Conrad portrayed in the television series "Baa Baa Black Sheep" serve?
12.0000	0.67	0.68	Henry Madison Morris	Henry M. Morris	initial for Madison	The Genesis Flood is a book by John C. Whitcomb and what founder of the Creation Research Society and the Institute for Creation Research?
12.0000	0.86	0.79	Thomas Penson De Quincey	Thomas De Quincey	name without middle name	On the Knocking at the Gate in Macbeth is an essay by what English essayist best known for "Confessions of an English Opium-Eater"?
11.0000	0.67	0.64	World War II	Second World War	Synonymous war name	The 59th Infantry Division covered the deployment of the 35th Infantry during the Allied operation during what war?
9.0000	0.67	0.61	20 assists	20	same number, unit implied	How many assists did the Bosnian football player account for during the 2008–09 VfL Wolfsburg season?

token F1 ≥ .5 + digit guard: in the class, judged different (false positive)

attempts	F1	Zipf F1	gold	candidate	judge	question
12.0000	0.50	0.45	India's first dedicated multi-wavelength space observatory.	space observatory	less specific than gold description	Gaia is a space observatory of the European Space Agency, what is Astrosat?
12.0000	0.57	0.62	Port Moresby, Papua New Guinea	Port Moresby	drops the nation asked for	What city and nation was the location of the 2015 Pacific Games where the Kiribati national basketball team finished with a 0-4 record?
12.0000	0.67	0.46	University of Delaware	University of Debrecen	wrong university	Which college is located in the United States, University of Delaware of University of Debrecen?
12.0000	0.50	0.51	Fort Valley, Georgia	Georgia	state only, city dropped	Where is the head quarter of the Blue Bird Body Company?
12.0000	0.50	0.51	Noel Gallagher	Liam Gallagher	wrong brother	Who sang lead vocals on the Oasis hit single which had an acoustic debut in drummer Tony McCarroll's last concert ?
12.0000	0.67	0.70	ice hockey	hockey	less specific than ice hockey	What sport does 200-foot game and Jonathan Toews have in common?
12.0000	0.50	0.64	Vilnius Old Town	Vilnius	drops Old Town	In which of the largest surviving medieval old towns in Europe is the Vilnius Cathedral located?
12.0000	0.67	0.44	Battle of Prome	Battle of Yiwu	other battle	Chiang Chungcheng was involved involved in what battle in 1942?

token F1 ≥ .5 + digit guard: outside the class, judged same (false negative)

attempts	F1	Zipf F1	gold	candidate	judge	question
10.0000	0.33	0.35	2016 Oklahoma Sooners football team	Oklahoma	Oklahoma names the Sooners team; year implied	Which Bob Stoops coached team beat the 2016 Auburn Tigers football team?
8.0000	0.00	0.00	The sponsorship scandal	AdScam	AdScam nickname for sponsorship scandal	The Government of Canada, which is the federal administration of Canada, is involved in what scandal in the province of Quebec?
4.0000	0.00	0.00	The National Basketball Association's Rookie of the Year	Eddie Gottlieb Trophy	Eddie Gottlieb Trophy is the NBA Rookie of the Year trophy	Kevin Wayne Durant was named what ?(An Award initiated following 1952-53 NBA Season?
4.0000	0.67	0.67	Bill Boyd	Bill Boyd (born 1931)	same person, birth year added	Who is the chairman of the company that operates Suncoast Hotel and Casino?
4.0000	0.40	0.42	Harvard University	Harvard Psychological Clinic	clinic is part of Harvard University	What university employed the psychologist that inspired David McClelland's thinking on power??
4.0000	0.67	0.68	575 acres (2.08 km²)	575 acres	metric conversion dropped	What is the are of the university at which Dick Biddle served as head football coach?
3.0000	0.33	0.39	FLDS Church	Fundamentalist Church of Jesus Christ of Latter-Day Saints (FLDS Church)	full name of FLDS Church	What organization is known for having a president who was convicted of felony charges that was harbored by his brother, another high ranking member?
3.0000	0.00	0.00	band	musical group	band is a musical group	What kind of group does At the Drive-In and Muse have in common?

Zipf-weighted F1 ≥ .5 + digit guard: in the class, judged different (false positive)

attempts	F1	Zipf F1	gold	candidate	judge	question
12.0000	0.75	0.79	Tsavo East National Park	Tsavo West National Park	West vs East park	Tsavo Man-Eaters were found at what Kenyan reserve?
12.0000	0.80	0.75	Smithfield, Rhode Island	Rhode Island	State only, less specific	In what town is the university which the Bryant Bulldogs represent?
12.0000	0.67	0.69	World War II	World War I	WWI vs WWII	The 34th Pursuit Squadron of the United States Air Force was among the 60,000-80,000 prisoners of war who were involved the Bataan Death March during what war?
12.0000	0.67	0.67	research university	university	drops research	University of Wisconsin–Madison and University of Notre Dame are both what?
12.0000	0.67	0.73	Kansas	Kansas City	Kansas City is not Kansas	What city does Saint Paul School of Theology and United Methodist Church of the Resurrection have in common?
12.0000	0.50	0.54	short story writer	writer	Less specific profession	What profession was both John Updike and Bret Easton Ellis ?
12.0000	0.67	0.71	American writer	writer	drops American	What was both John Cheever and Laura Ingalls Wilder occupations?
12.0000	0.50	0.51	Marvel Comics	DC Comics	other company	Tao Okamoto made her debut as the lead in the 2013 film featuring the character from what Comic company's line?

Zipf-weighted F1 ≥ .5 + digit guard: outside the class, judged same (false negative)

attempts	F1	Zipf F1	gold	candidate	judge	question
12.0000	0.57	0.48	New York	Midtown Manhattan, New York City	more specific location in New York	Where is the headquarters for the advertising agency which is recognized for its work on a campaign for the founder of "Complex" magazine?
12.0000	0.00	0.00	Ulanhot	Ulan Hot	Spelling variant	Is Xingcheng or Ulan Hot located in the Inner Mongolia region of China?
12.0000	0.00	0.00	the Troubles	Northern Ireland conflict	alternative name for the Troubles	Which conflict was this person whose assassination led to Mark Fulton taking over command the Loyalist Volunteer Force involved in?
10.0000	0.00	0.00	Argentinian	Argentine	Argentine equals Argentinian	Michaël Llodra of France, called "the best volleyer on tour", defeated Juan Martín del Potro a professional of what nationality?
9.0000	0.50	0.45	five	five separate bills	same number with noun	California joined the Union due to the passage of a package of how many separate bills?
8.0000	0.29	0.33	Makoto, born July 15, 1980	Makoto Koshinaka	Makoto Koshinaka is the person meant	Who is younger, Dan Bejar and Makoto Koshinaka?
8.0000	0.40	0.49	University of Missouri	Missouri Tigers	Missouri Tigers is that university's team	Robin Pingeton replaced Cindy Stein as head coach of which university women's basketball team?
8.0000	0.40	0.55	1800000 sqft	1,800,000 square feet	formatting variant	What is the gross leasable area of the shopping mall served by the Merrick Boulevard buses?

Stronger Zipf weighting: fewer missed correct answers

30 October 2026 · Controlled comparison

With w(t) = (9 − Zipf(t))γ, only γ changes from 1 to 1.5. The acceptance threshold stays at .6, with the same WordNet token matching and fixed digit guard. Three independent judges using the same model labelled the same 396 pairs (400 draws). Both settings use the same sampling weights; unsure judgments count as half correct.

γ	False-positive rate	False-negative rate	Total error
1.0	2.80–3.28%	27.32–28.41%	10.57–11.24%
1.5	2.88–3.29%	25.65–26.66%	10.10–10.71%

Ranges span the three judges, not confidence intervals. Rates are estimated over listed hypothesis instances pooled across valid rollouts; problems are not averaged first. Total error counts false positives and false negatives equally. The false-negative rate falls for every judge: more correct hypotheses are accepted. False-positive rates stay similar or rise slightly.

One recovered match and one new false match

Each score below is compared with the same .6 cutoff. All three judges and Claude agree on the correctness of these two pairs.

Gold	Candidate	Score, γ = 1	Score, γ = 1.5	Effect
Thomas Warburton	Tom Warburton	0.594	0.639	recovered correct match
Joan Feynman	Richard Feynman	0.579	0.616	new false match

Emphasizing a shared surname can recover a valid name variant, but can also hide a different first name.

Exploratory result: 15 sampled pairs change decision. Most of the estimated gain comes from one draw, “1,800,000 square feet” versus “1800000 sqft”, in the larger sampling pool. Paired 95% confidence intervals include zero improvement for every judge; fresh validation is needed.

All 1849 labelled pairs

Browse the labelled pairs (one source and one pool at a time; "any" shows all)

source prior candidatesrollout hypothesesany pool exact match, fntoken F1 ≥ .5 + digit guard, fptoken F1 ≥ .5 + digit guard, fnZipf-weighted F1 ≥ .5 + digit guard, fpZipf-weighted F1 ≥ .5 + digit guard, fnany label anysamedifferentunsure search

source	prior / attempts	F1	Zipf F1	gold	candidate	label	judge	pools (draws)	question