The Whelming

With over 13 million items, Wikidata is a vast repository of human- and machine-readable data, and with over 70 million edits in the last 11 month, its value as a data repository is growing fast. Much of that value is stored in statements about items, and the connections between items.

42 might take a while

Surprisingly, ways to query this data behemoth are quite limited. There is text search, which will find you items with certain labels, and there is an API which can get information about a specific item and its properties. And yet, a way to ask more complex questions is nowhere in sight. Getting a list of people born in New York is currently impossible, without resorting to rather slow and tedious third-party tools (I should know; I wrote some of them). “Simple” queries are being worked on, but due to personnel priorities, as well as the nature of the underlying data storage, it might be quite a while to get answers to compound queries from Wikidata itself.

Worse, even if statements were present as “proper” database tables (they currently exists only as JSON blobs, if you really want to know), developers would be faced with a variety of the “category tree” problem that we all know and love on Wikipedia: Trees are not quickly and easily searched in databases, turning category tree intersections and related operations into horror stories to scare other programmers around a virtual campfire. Several variations of “CatScan” exist, but cannot really solve these issues. The same kind of challenges will hit Wikidata for “subclass of” and “parent taxon” chains that need following and branching.

So, a while ago, I asked the Wikidata section of Wikimedia Labs for a server to try out an idea of mine: What if I could keep the entirety of Wikidata in memory, as one giant, interlinked data structure? Is that possible? Would complex searches be quick enough? To make a long story short: Yes, and yes 🙂 The machine, with the software I dub “Wikidata Query” (WDQ for short), has been online for a while, and some people know about it, but with some recent updates to the code, interface, and documentation, I felt it’s time to write it up in a blog post.

The bridges of Germany

The query designer with the “bridges in Germany” example query.

Why does this need to be so complicated? What is the problem I am trying to solve? Let’s look at an example: I want to get a list of all the bridges in Germany. This sounds simple enough at first, but the devil is in the detail. What is a bridge, at least in Wikidata? Well, obviously anything that is an “instance of: bridge”, right? Right. And wrong, because that would be incomplete. This is a bridge in Germany, but it is an “instance of: railway bridge”, with “railway bridge” being a “subclass of: bridge”. This is a “direct” subclass, but since there is no “chain length limit” in Wikidata, this might go over several “subclass steps”. Thus, a bridge in Wikidata is an “instance of” anything that is a “subclass of” bridge, or “subclass of subclass of” bridge, etc. Basically, we need to construct a “reverse subclass tree” (reverse because we follow “incoming” links from the root) of everything that has “bridge” at its root, and get all the instances of all these bridge subclasses.

Now we have a list of all bridges. ALL bridges, worldwide. But we want only bridges in Germany, remember? So, how do we know that something (bridge or otherwise) is in Germany? As you might have guessed, “country: Germany” is not going to cut it. This time, we need all items that are leafs of the “is in administrative unit” reverse tree that can be traced back to the root “Germany”. For the previously mentioned bridge, that’s five hops.

“All” we need to do now is to find the items that are in both the “bridge” and the “Germany” lists of items, and we’re done! This sounds like a lot of work, and it is! However, thanks to the in-memory implementation (currently taking an easy 9GB of RAM, including multilingual item labels), this query takes only 0.8 seconds, which is quite reasonable IMO.

Designer code and raw query power

While this works nicely, and could be the basis for many other tools, constructing the query (which in this case, is obviously “claim[31:(TREE[12280][][279])] AND tree[183][150][17,131]” ) requires some basic programming skills, and a lot of time to find the item and property IDs. Also, the JSON output, while perfectly machine-readable, is not exactly easy on the eye. Therefore, I also wrote a query designer that should cater to a larger audience. It offers combinations of searched for claims, negative claims (“does not have a claim/property”), perfect matches for strings (e.g., “has this VIAF ID”), and the aforementioned chains or trees.

Due to lack of manpower, the query designer does not (yet) cover all possibilities the API offers. Manually constructed queries can search for date ranges per property (as in “born between dates X and Y”), or for locations around a point. All of these can be combined with AND/OR, grouped, and (in some cases) use nested subqueries, which makes the query language quite powerful.

There are a few minor drawback in all of this: WDQ is based on the Wikidata dumps, static snapshots that are made at more-or-less regular intervals. I do update WDQ with the daily “differential” dumps, so it should never be more than 24h out of date, but in the ever-changing world of Wikidata that can feel like an eternity. Also, neither qualifiers nor sources are supported yet.

If you would like to build your own tools on top of WDQ (which I would welcome very much!), have a look at the API documentation, with details on how to construct a query, and advanced/experimental features such as properties and labels. I would also welcome help with maintaining, improving, and extending the code (hereby GPL 2+, code on Bitbucket).

Wikidata is booming. A huge number of editors, many of them new contributors to the Wonderful World of Wikimedia projects, are adding, fixing, and sourcing claims to the database. A ridiculously high amount of language links have been moved from Wikipedia to Wikidata. Part of the promise of this new project is to unify factoids across the different language editions of Wikipedia; another is to make the key facts of Wikipedia (and other sources) accessible in a machine-readable form for third-party users. For those users, which will probably outgrow the use of claims on Wikipedia in the long run, the amount of (reasonably reliable) data is paramount. Wikipedia really took off once it reached a critical mass of useful knowledge; the same is due to happen to Wikidata.

Or is it? When it comes to claims and statements, a huge amount of heavy lifting has been, and is being, done by bots. The idea is simple; there is a lot of simple, factual data that can be extracted from a Wikipedia article and inserted into the corresponding Wikidata item, using categories, templates, or specific matches in the article text itself. For proper provenance, bots usually add “imported from en.wikipedia” as a “source” to the imported claim.

But this well-working system is about to end, sabotaged by people obsessing with “proper sources”. Now, don’t get me wrong; I do think all statements in Wikidata should be sourced properly! It is a tremendous opportunity to link the world’s knowledge up in the most fine-grained form imaginable. It is, undoubtedly, the right thing to do. But the drive to enforce this now, for all bots, is premature. Mainly because it is impossible.

I know. I created a list of birth and death dates from Wikipedia articles. Over 300.000 Wikidata items about people could have that information, in a matter of hours. But this has been verboten by the bureaucracy, as the dates do not have proper sources. Non-technical people may ask, “so why not just add sources”? I’d love to; except, it might be somewhat hard to have a bot scan the sources in the Wikipedia article, go to a library, get all the books, read them, find the mentioned dates, return the books, find or create the book items in Wikidata, and link them up accordingly. A bot couldn’t source web resources either (even if it would somehow know which one to use), since there is currently no URI type on Wikidata.

So all these dates will have to be entered and sourced manually, by a human. And this is why it’s the wrong approach; it doesn’t scale, even in the Wikimedia world, which is not exactly lacking volunteers. It could be done eventually, but the flame of hope that is Wikidata now will have faded by then. Facts on Wikipedia have been checked to a high degree, as was demonstrated time and again. The important thing right now is to get these facts into Wikidata, to keep the momentum, to get others interested in using, and working with, Wikidata as a machine-readable knowledge repository. Sources will, and should, be added. In due time.