The Whelming

Mix’n’match is one of my more popular tools, and with over 262 million entries one of the largest. It also has been around for awhile (November 2013), almost as old as Wikidata. It started out as a PHP script, which then grew into a collection of classes, but eventually the size of the data, as well as changes on Toolforge, resulted in me writing a server in Rust, to process data in the background (scraping websites, matching entries, maintenance functions). This resulted in a massive duplication of code, as the Rust back-end needs to deal with the same database and data structures as the PHP front-end; plus, all the (now legacy) PHP back-end code.

Recent changes in a catalog. The user links show recent edits by the user.

I finally got around to implementing the missing part, which is a Rust rewrite of the PHP web API. This leverages the existing Rust data structures, database access etc., but also re-uses the HTML/JS for the front-end. One reason that kept me from doing this was me letting users add small PHP scripts to perform specific functions, e.g. extract birth and death dates from entry descriptions for a specific catalog. As you can see in my recent blog entry, I am now offering Lua instead, but keep the legacy PHP code around, for comparison mostly.

A migration of this magnitude (the web API alone was >3600 lines of PHP; over 15K lines of PHP across all PHP classes) can not be expected to go perfectly smooth, when you are a team of 1 (well, more 0.2, as I have a day job, and one of these pesky “life” things that are all the rage now), but it’s up and running now, with surprisingly little fallout.

Besides the consolidation of the two code bases into the Rust one, and the Rust web server (axum) being a bit more snappy than the lighttp/PHP one, I have also put some effort into improving the interface. For example, you can now see more details about a catalog and how it behaves, and (if you created the catalog, or are a “catalog admin”) you can also change it; that includes setting up specific auto-matchers to specify what kind of items you are looking to match the catalog against.

Adding and removing catalogs from a group or search filter is now much easier.

Many pages have an improved UI, e.g. Recent Changes and User Changes (which is actually a new page type). Importing files as a catalog is now a wizard-style component, and the scraper page has more information feedback about just why that regexp doesn’t work (and is now linked from every catalog that has a scraper in the “Action” drop-down).

As you might have already seen, the landing page has changed as well. Top groups has been revamped to be more usable. There are also small improvements all over, e.g. proper pagination for issues or jobs.

Please leave all bug reports, improvement suggestions etc. in the issue tracker of the Rust repo. Otherwise, happy matching!