Since a few years, I run the WikiData Query tool (WDQ) to provide a query functionality to Wikidata. Nowadays, the (confusingly similarly named) SPARQL-based WDQS is the “official” way to query Wikidata. WDQS has been improving a lot, and while some of my tools still support WDQ, I deliberately left that option out of new tools like PetScan. But before I shut down WDQ, and the tools that use it, for good, I wanted to know if it is still used, and if SPARQL could take over.
I therefore added a query logger to Autolist1 and Autolist2. The logs contain all WDQ queries run through those tools. I will monitor the results for a while, but here is what I saw so far. I will comment on translating the query to SPARQL using WDQ2SPARQL, the general ability for such queries, and the performance of WDQS. “OK” means the query could be converted automatically to SPARQL, runs, and produces a similar (as in, equal or more up-to-date) result.
| WDQ | Comment |
|---|---|
| CLAIM[279:13219666] | OK |
| BETWEEN[569,1016-1,1016-12] | Update: This has been implemented by smalyshev no, runs OK! |
| (CLAIM[1435:10387684] OR CLAIM[1435:10387575]) AND NOCLAIM[380] AND NOCLAIM[481] | OK |
| BETWEEN[569,1359-1,1359-12] | Update: This has been implemented by smalyshev no, runs OK! |
| CLAIM[31:5] | All humans ~3.2M humans on Wikidata. Not really a useful query in these tools. |
| Q22686 | Single item. Doesn’t really need a query? |
| Q22686 | Single item. Doesn’t really need a query? |
| CLAIM[106:170790] AND CLAIM[27:35] | OK |
| CLAIM[195:842858] | OK |
| Gustav III | What the hell? |
| claim[17] | All items with “country”. Not really a useful query in these tools. |
| claim[31] | All items with “instance of”. Not really a useful query in these tools. |
| claim[106:82955] and claim[509:(tree[12078][][279])] | OK |
| claim[31:5] | All humans ~3.2M humans on Wikidata. Not really a useful query in these tools. |
| claim[31:5] | All humans ~3.2M humans on Wikidata. Not really a useful query in these tools. |
| claim[21] | All items with gender. Not really a useful query in these tools. |
| LINK[lvwiki] AND CLAIM[31:5] | OK |
| LINK[lvwiki] AND CLAIM[31:5] | OK |
| claim[27] and noclaim[21] | OK |
| LINK[lvwiki] AND CLAIM[31:56061] | OK |
| LINK[lvwiki] AND tree[56061][150][17,279] | OK |
| claim[31:(tree[16521][][279])] | OK |
As far as I can tell, SPARQL could take over for WDQ immediately.
11 Comments
The last 12 hours or so, WDQ is down.
Did you close it already?
My python script(s) use WDQ extensively. If it’s down for good I need to investigate how to rewrite those to use SparQL instead!
Tech issues:
https://phabricator.wikimedia.org/T141606
Should be back soon. Nevertheless, you should switch to SPARQL soon. There is a tool that will rewrite most WDQ queries into SPARQL:
https://tools.wmflabs.org/wdq2sparql/
If you use PHP, you might want to look at the code I use to get SPARQL results as WDQ item lists:
https://bitbucket.org/magnusmanske/magnustools/src/241df94aa20fcfd7766da47f11b2a8b48e33d760/public_html/php/common.php?at=master&fileviewer=file-view-default#common.php-614
WDQ is back now, but struggling.
I had a problem with Listeria where the template code would fail because the SPARQL query contained a pipe character and Mediawiki interpreted this as a template parameter separator. After having tried various escape characters without success, I had to fall back to state the query in WDQ.
Anders, feel free to reach out to me to discuss what’s missing from Wikidata Query Service! I’d love to try to get any bugs that you find with it fixed. Here’s my contact info: https://meta.wikimedia.org/wiki/User:Deskana_(WMF)
BETWEEN actually should work, but needs Y-m-d date. Y-m not supported, but I can implement it.
OK, BETWEEN examples should work now too.
Dan, thanks, but I actually have not had any problems with WQS itself other than the occasional cache hiccups. My problem above is with the Listeria bot using an input format that does not allow one to validly express all queries in SPARQL that one could in WDQ.
I have a problem with SPARQL queries that work one day and timeout the next day. Example:
#Bridges in USA w/o crosses
SELECT ?s WHERE {
?s wdt:P17 wd:Q30 .
?s wdt:P31/wdt:P279* wd:Q12280 .
OPTIONAL {
?s wdt:P177 ?dummy
}
FILTER ( !bound(?dummy) ) .
}
So I fall back to the WDQ equivalent:
CLAIM[17:30] AND CLAIM[31:(TREE[12280][][279])] AND NOCLAIM [177]
This is probably not the place for discussing query optimisation, but your SPARQL runs in 0.4 seconds. How would that timeout?
Sorry, evidently wdqs.wmflabs.org, which I was using, is different from query.wikidata.org.