Skip to content

Less mixing, more matching

Thursday, April 23, 2026

Mix’n’match is one of my more popular tools, and with over 262 million entries one of the largest. It also has been around for awhile (November 2013), almost as old as Wikidata. It started out as a PHP script, which then grew into a collection of classes, but eventually the size of the data, as well as changes on Toolforge, resulted in me writing a server in Rust, to process data in the background (scraping websites, matching entries, maintenance functions). This resulted in a massive duplication of code, as the Rust back-end needs to deal with the same database and data structures as the PHP front-end; plus, all the (now legacy) PHP back-end code.

Recent changes in a catalog. The user links show recent edits by the user.

I finally got around to implementing the missing part, which is a Rust rewrite of the PHP web API. This leverages the existing Rust data structures, database access etc., but also re-uses the HTML/JS for the front-end. One reason that kept me from doing this was me letting users add small PHP scripts to perform specific functions, e.g. extract birth and death dates from entry descriptions for a specific catalog. As you can see in my recent blog entry, I am now offering Lua instead, but keep the legacy PHP code around, for comparison mostly.

A migration of this magnitude (the web API alone was >3600 lines of PHP; over 15K lines of PHP across all PHP classes) can not be expected to go perfectly smooth, when you are a team of 1 (well, more 0.2, as I have a day job, and one of these pesky “life” things that are all the rage now), but it’s up and running now, with surprisingly little fallout.

Besides the consolidation of the two code bases into the Rust one, and the Rust web server (axum) being a bit more snappy than the lighttp/PHP one, I have also put some effort into improving the interface. For example, you can now see more details about a catalog and how it behaves, and (if you created the catalog, or are a “catalog admin”) you can also change it; that includes setting up specific auto-matchers to specify what kind of items you are looking to match the catalog against.

Adding and removing catalogs from a group or search filter is now much easier.

Many pages have an improved UI, e.g. Recent Changes and User Changes (which is actually a new page type). Importing files as a catalog is now a wizard-style component, and the scraper page has more information feedback about just why that regexp doesn’t work (and is now linked from every catalog that has a scraper in the “Action” drop-down).

As you might have already seen, the landing page has changed as well. Top groups has been revamped to be more usable. There are also small improvements all over, e.g. proper pagination for issues or jobs.

Please leave all bug reports, improvement suggestions etc. in the issue tracker of the Rust repo. Otherwise, happy matching!

Filed in Rust, Wikidata | | Comments (2)

From PHP to Lua

Thursday, April 16, 2026

A minor tech announcement for Mix’n’match:

Code fragments for Mix’n’match catalogs are switching from PHP to Lua. User-supplied PHP is not secure enough, and will not scale for future development. I have converted many code fragments from PHP to Lua already, and new code fragments have to be Lua.

Lua is not exactly my favorite language, but it has a sweet spot between the number of people who know it or can easily learn it, and safety/resource usage etc. for letting user-written code run within Mix’n’match/Toolforge.

PHP code that was not “translated” yet will be supported for a while, but not forever. Existing PHP code will be shown as a “hint” for Lua, for the time being.

Example: https://mix-n-match.toolforge.org/#/code/7098

Filed in Wikimedia | | Comments (1)

Of Mixing and Matching

Friday, March 27, 2026

Mix’n’match is one of my more popular tools. It has grown over the years in both functionality and data size. It currently contains >260M entries in over 6300 catalogs. The code base has been growing organically with the data size. I felt it was time to modernize some things.

First, the speed. Some pages, most notably the landing page, would load details on all active catalogs in the database. On every load. This was not sustainable. The updated version has some layout simplification, but shows the same information as before, and reduces the (uncached) loading time from ~35 to 1.8 seconds (YMMV).

The interface was updated to bootstrap5, and many interfaces have ever so slightly changed, for the better I think.

I have also taken pains to support mobile better. This includes moving the “mobile game” from its own, separate site to integrate into Mix’n’match.

Finally, under the hood, this is my first tool where I moved from my home-grown Vue module loading system to proper ES6 components. While this will make little difference to users, it is an important step towards maintainability. It also allowed me to write JavaScript unit tests.

Please give the new version a spin, and let me know about issues and suggestions in the issue tracker.

Filed in Uncategorized | | Comments (1)

APIs and Descriptions

Thursday, March 12, 2026

AutoDesc has been a longstanding tool to generate descriptions for Wikidata items. Originally a spin-off from Reasonator, it started as my only JavaScript-based server application, generating both short and (for biographical items, and two languages) long descriptions. When the usual bitrot set in, I decided to move it over to Python, hoping to attract more developers; that did, however, not pan out. Also, I only managed to port the short description code, dropping support for the long descriptions in the process.

Now, I have moved it to trusty Rust, and also ported the long description mode from the original JavaScript. I threw in some caching, so descriptions are now much faster – often <1sec for short descriptions.

In other news, my MediaWiki Rust API has now added all of the MediaWiki Action API functionality as proper Rust elements. Generators are also supported, as is continue. Results that are based on page_info results can be automatically parsed into an appropriate struct. I even threw in a prelude::* module to get everything you need in one line. Due to the massive addition of functionality, the version has also been bumped to 0.5.0.

Filed in Rust, Wikidata, Wikimedia | | Comments Off on APIs and Descriptions

geohack rust

Wednesday, November 19, 2025

One of my oldest (first commit: 2013-06-24) tools is geohack, which was majorly improved by Egil Kvaleberg, who also co-maintained it. The tool is not only old but popular; many Wikipedia languages, including English and German, link to it from basically every page that uses coordinates.

Its popularity recently became a problem; it exceeded the maximum resources a tool on Toolforge can summon, hitting 60 requests/sec. using four separate copies of itself (4 CPUs).

Memory usage PHP -> Rust

So I obeyed the mantra of current technology, and rewrote it in Rust. Running its own webserver, the Rust version caches the various templates it uses, and does not require Toolforge disk or database access. First indications are that resource usage is massively down; a single copy/CPU can answer the incoming request deluge. Also, memory usage has dropped from 1.5GB to ~0.1GB.

The rewrite of the tool was done in very “literal” fashion, to preserve all corner case functionality where possible; indeed, the Rust version generates the exact output (not counting irrelevant whitespace and floating point rounding issues) as the PHP version. Eighteen examples from the PHP version, including locations on other planets and asteroids, now serve as Rust unit tests.

 

All this came at the cost of sometimes awkward Rust style. I am working on refactoring the code to make it more maintainable. Pull requests are always welcome, as are git issues and feature requests.

Filed in Rust, Wikimedia | | Comments Off on geohack rust

Inference, continued

Tuesday, April 1, 2025

A while ago I wrote about WD-infernal, an API to infer some information about a Wikidata item, that needs to be checked by a user (somehow). The idea was to offer standardized inference to multiple tools and Wikidata user scripts.

I have now added two new functionalities:

1. referee, which follows extrnal IDs and external links from associated Wikipedia pages from an item, to try and find potential references. The checks are quite suble, for example, trying multiple data formats to find a reference for a date, and infering the language of an external web page to use the appropriate label of a statement (eg for gender:male, use the German label for “male” on a German page). I had written backend code for a little script many years ago, but now I am bringing this functionality to the async/multithread/Rust age.

2. cross_categories, a functionality that I had also writen before, in a (now defunct) tool called CrossCats. For a given Wikidata item about a Wikipedia category, it will find all articles in that category on all associated wikis, aggregate and count the items for these articles, and return the ones that are in a given language Wikipedia, but not yet in that category tree.

As with the previous functionality, these are API endpoints returning JSON. They do not have a UI yet. I will eventually change the aforementioned referee script to use this endpoint. I might also work on a CrossCats interface, unless someone else beats me to it.

Filed in Rust, Wikidata | | Comments Off on Inference, continued

REST in Rust

Friday, March 14, 2025

The new Wikibase REST API brings standardized and simplified querying and editing of items, properties, statements etc. to Wikibase installations, first and foremost Wikidata. Last year, Wikimedia Sverige was entertaining the idea of a grant application to Wikimedia Deutschland. Part of the proposal was for me to write a Rust crate (i.e. library) for easier access to the new REST API. The project eventually didn’t go ahead, but I had started writing the code anyway, out of personal interest. To make it more chellenging, I tried to apply industry-level coding standards to the project; most of my tools are not written with this in mind, as usually I am the only one working on them.

grcov code coverage (“functions” does not work properly with the testing mode used)

Now, I have an (almost) feature-complete result:

What about those coding standards? Well:

  • 248 unit tests in the codebase, running in 0.2sec (on my machine), that are also run on every github push
  • many tests mock a HTTP server to allow local testing, including simulated editing
  • grcov reports >97% code coverage
  • tarpaulin reports ~90% code coverage (but doesn’t see some traits being used when they are)
  • cyclomatic complexity mean is 2.1 for the entire codebase, <7 everywhere (according to lizard)
  • function line length is <40 lines for the entire codebase
  • maintainability index median is 126.3 (original) or 73.9 (VS Code)
  • scripts and instructions for calculating the code metrics are part of the repo, as it an analysis.tab file
  • passes miri for all of my code (tests using some external code are deactivated for miri)

I will certainly make it a point to use the crate in both new and existing projects. I hope some of you will join me, and please feed back comments and issues on the issue tracker. Pull requests welcome!

Filed in Rust, Wikidata | | Comments Off on REST in Rust

Wednesday, December 11, 2024

So many Wikidata items have a “described at URL” (P973) statement, where we do not have a property to use an ID, or the source does not use IDs. I was wondering if some URL domains have accumulated in larger numbers for P973, which would make them candidates for properties. So I listed and counted the (unique) URLs from all such statements, and counted the domain names from that. Results:

Domain name Occurrences
www.sciencedirect.com 499,443
link.springer.com 151,935
digicol.dpm.org.cn 127,400
www.tandfonline.com 102,998
journals.sagepub.com 75,685
clevelandart.org 67,668
onlinelibrary.wiley.com 59,064
ieeexplore.ieee.org 48,151
iopscience.iop.org 47,917
www.mdpi.com 44,406
www.cambridge.org 36,292
linkinghub.elsevier.com 34,884
academic.oup.com 34,047
www.ssrn.com 28,785
www.nber.org 28,776
www.jstage.jst.go.jp 24,556
link.aps.org 23,120
parismuseescollections.paris.fr 19,194
journals.lww.com 18,481
data.collectienederland.nl 17,959
www.kci.go.kr 17,668
www.dbpia.co.kr 17,616
pubs.acs.org 17,389
dati.beniculturali.it 17,034
journals.ashs.org 16,875
muse.jhu.edu 15,841
www.emerald.com 14,996
pubs.aip.org 13,749
www.vanbommelvandam.nl 13,653
www.degruyter.com 12,675
xlink.rsc.org 12,022
www.inderscience.com 11,460
www.digiporta.net 11,442
datos-geodesia.ign.es 11,246
www.wga.hu 11,173
www.victorianresearch.org 10,610
www.indianjournals.com 10,364
kokoelmat.fng.fi 10,150

These are the domain names with >10K occurences. Full list here. I hope this list will lead to some new property propositions!

Filed in Wikidata | | Comments (1)

Infer-nal

Wednesday, November 20, 2024

After I recently wrote a small on-Wiki tool that can suggest statements to add to a Wikidata item, I thought that something like this might be useful in other tools as well. So, using the same concept and technology (Rust/axum) from my Authority Control API, I wrote WD-Infernal, an API that takes data, such as location coordinates or people’s names, and tries to infer Wikidata statements from that.

The idea of statement inference is obviously not novel, but AFAIK it is only implemented in individual codebases such as bots. Implemented as an API, any tool, bot etc. can use the inferred statements. I hope that feedback will improve the existing functions, and add new ones to this API.

In the greater scheme of things, my hope is to create more APIs to offer functionality around Wikidata, and to inspire others to do the same. An ecosystem of APIs would both simplify and standardize recurring tasks of (semi-)automatic Wikidata editing.

Filed in Rust, Wikidata | | Comments Off on Infer-nal

Using AI to add to Wikidata

Monday, October 28, 2024

AI and the WikiVerse have a complicated an developing relationship. Here, I investigate possible uses of AI to assist with imports of unstructured data from Mix’n’Match into Wikidata.

Approach

At the time of writing, Mix’n’match contains ~162M entries. Many of them have a more-or-less helpful short description, usually taken from the respective third-party source. I do use some old-fashioned but reliable parsing to extract information like birth and death dates from descriptions such as this: Born/died: 1869, Arenys de Mar (Espagne) - 1939, Paris (France).

Years are fine, but more complex dates (including localized names of months, and days like “January 1st, 1901”) already require more involved parsers. And we know which Paris is meant in the above description, but how can a parser be sure it’s not Paris (Texas)?

So I took the unmatched entries from catalog 6410 (MnM entry IDs, names, and descriptions only) and asked an AI to generate me some Wikidata statments (here I use Claude Sonnet 3.5, but CharGPT yields similar results): The following is a list of entries with IDs, names, and descriptions. Please create appropriate Wikidata statements per entry. Try to infer nationality and gender where possible. Use Wikidata item IDs instead of text where possible.

Results

This yields a list of statements, grouped by entry (input data here: Born/died: 1945, Newark (New Jersey, États-Unis)):

For Barbara Kruger (ID: 168457784):
CREATE
LAST|Len|"Barbara Kruger"
LAST|P31|Q5
LAST|P21|Q6581072
LAST|P27|Q30
LAST|P569|+1945-00-00T00:00:00Z/9
LAST|P19|Q18127

Analysis

  • I find it insanely funny and interesting that Claude gives me QuickStatement format
  • Claude will only parse about a dozen or so entries before stopping, and I need to manually say “continue”
  • P31 (human) and P21 (female) were inferred by Claude, probably from the name
  • P27 was correctly identified as United States, even though it’s in French
  • P569 (birth date) is correct Wikidata-internal format
  • P19 (place of birth), however, is not Newark, New Jersey, but record label

Some modification of the prompt gives me the parsed values in both Wikidata items and plain text. I wrote a simple parser for that output and loaded it into a database table:

More investigation shows that Claude parses the descriptions correctly, and identifies the relevant properties (eg place of birth: Newark, New Jersey). But, with the exception of basic items auch as Q5 (human), it often fails to identify the correct item, and silently hallucinates a different one.

Conclusion

At this point in time, AI appears to be quite useful to extract information from Mix’n’Match-style descriptions and present them as Wikidata statement parts, namely property and a standardized text value (eg country of citizenship: United States). However, finding the matching Wikidata item for this does often elude the AI, and yields faulty item IDs.

This can still be quite useful. Just telling apart humans and eg companies in the same MnM catalog would be a help. Also, because the output of the AI is standardized, I can easily find the correct item for citizenship myself. Also, finding the text for the birthplace helps, and the reformatted birthplace (“Newark, New Jersey”) is easier to find. Claude even annotates that it determined the gender from the name, which could easily be translated into a Wikidata qualifier.

In summary, AI can yield useful analysis of MnM entry descriptions. Given the cost and throughput limitations of commercial AI models, it would be helpful if the WMF would set up a free AI model for Toolforge-internal use. Even if these would be less powerful, getting a structured key-value list from an unstructured MnM entry would be immensly helpful, as it would mitigate the need for bespoke code per catalog, and potentially yield more information per entry. Also, a standardized “AI-generated” reference for such AI-based statements would help.

Filed in Wikidata | | Comments (1)