Skip to content

WDQ, obsolete?

Since a few years, I run the WikiData Query tool (WDQ) to provide a query functionality to Wikidata. Nowadays, the (confusingly similarly named) SPARQL-based WDQS is the “official” way to query Wikidata. WDQS has been improving a lot, and while some of my tools still support WDQ, I deliberately left that option out of new tools like PetScan. But before I shut down WDQ, and the tools that use it, for good, I wanted to know if it is still used, and if SPARQL could take over.

I therefore added a query logger to Autolist1 and Autolist2. The logs contain all WDQ queries run through those tools. I will monitor the results for a while, but here is what I saw so far. I will comment on translating the query to SPARQL using WDQ2SPARQL, the general ability for such queries, and the performance of WDQS. “OK” means the query could be converted automatically to SPARQL, runs, and produces a similar (as in, equal or more up-to-date) result.

WDQ Comment
CLAIM[279:13219666]  OK
BETWEEN[569,1016-1,1016-12] BETWEEN not implemented in WDQS, but manual translation feasible

Update: This has been implemented by smalyshev no, runs OK!

(CLAIM[1435:10387684] OR CLAIM[1435:10387575]) AND NOCLAIM[380] AND NOCLAIM[481]  OK
BETWEEN[569,1359-1,1359-12] BETWEEN not implemented in WDQS, but manual translation feasible

Update: This has been implemented by smalyshev no, runs OK!

CLAIM[31:5]  All humans ~3.2M humans on Wikidata. Not really a useful query in these tools.
Q22686  Single item. Doesn’t really need a query?
Q22686  Single item. Doesn’t really need a query?
CLAIM[106:170790] AND CLAIM[27:35]  OK
CLAIM[195:842858]  OK
Gustav III  What the hell?
claim[17]  All items with “country”. Not really a useful query in these tools.
claim[31]  All items with “instance of”. Not really a useful query in these tools.
claim[106:82955] and claim[509:(tree[12078][][279])]  OK
claim[31:5]   All humans ~3.2M humans on Wikidata. Not really a useful query in these tools.
claim[31:5]   All humans ~3.2M humans on Wikidata. Not really a useful query in these tools.
claim[21]  All items with gender. Not really a useful query in these tools.
LINK[lvwiki] AND CLAIM[31:5]  OK
LINK[lvwiki] AND CLAIM[31:5]  OK
claim[27] and noclaim[21]  OK
LINK[lvwiki] AND CLAIM[31:56061]  OK
LINK[lvwiki] AND tree[56061][150][17,279]  OK
claim[31:(tree[16521][][279])]  OK

As far as I can tell, SPARQL could take over for WDQ immediately.

11 Comments

  1. Edo de Roo wrote:

    The last 12 hours or so, WDQ is down.

    Did you close it already?

    My python script(s) use WDQ extensively. If it’s down for good I need to investigate how to rewrite those to use SparQL instead!

    Friday, July 29, 2016 at 09:30 | Permalink
  2. Magnus wrote:

    Tech issues:
    https://phabricator.wikimedia.org/T141606

    Should be back soon. Nevertheless, you should switch to SPARQL soon. There is a tool that will rewrite most WDQ queries into SPARQL:
    https://tools.wmflabs.org/wdq2sparql/

    If you use PHP, you might want to look at the code I use to get SPARQL results as WDQ item lists:
    https://bitbucket.org/magnusmanske/magnustools/src/241df94aa20fcfd7766da47f11b2a8b48e33d760/public_html/php/common.php?at=master&fileviewer=file-view-default#common.php-614

    Friday, July 29, 2016 at 09:43 | Permalink
  3. Magnus wrote:

    WDQ is back now, but struggling.

    Friday, July 29, 2016 at 10:11 | Permalink
  4. Anders Feder wrote:

    I had a problem with Listeria where the template code would fail because the SPARQL query contained a pipe character and Mediawiki interpreted this as a template parameter separator. After having tried various escape characters without success, I had to fall back to state the query in WDQ.

    Monday, August 1, 2016 at 23:03 | Permalink
  5. Dan Garry wrote:

    Anders, feel free to reach out to me to discuss what’s missing from Wikidata Query Service! I’d love to try to get any bugs that you find with it fixed. Here’s my contact info: https://meta.wikimedia.org/wiki/User:Deskana_(WMF)

    Tuesday, August 2, 2016 at 01:38 | Permalink
  6. Stanislav Malyshev wrote:

    BETWEEN actually should work, but needs Y-m-d date. Y-m not supported, but I can implement it.

    Tuesday, August 2, 2016 at 02:07 | Permalink
  7. Stanislav Malyshev wrote:

    OK, BETWEEN examples should work now too.

    Tuesday, August 2, 2016 at 23:24 | Permalink
  8. Anders Feder wrote:

    Dan, thanks, but I actually have not had any problems with WQS itself other than the occasional cache hiccups. My problem above is with the Listeria bot using an input format that does not allow one to validly express all queries in SPARQL that one could in WDQ.

    Wednesday, August 3, 2016 at 17:34 | Permalink
  9. Bruce Myers wrote:

    I have a problem with SPARQL queries that work one day and timeout the next day. Example:
    #Bridges in USA w/o crosses
    SELECT ?s WHERE {
    ?s wdt:P17 wd:Q30 .
    ?s wdt:P31/wdt:P279* wd:Q12280 .
    OPTIONAL {
    ?s wdt:P177 ?dummy
    }
    FILTER ( !bound(?dummy) ) .
    }

    So I fall back to the WDQ equivalent:
    CLAIM[17:30] AND CLAIM[31:(TREE[12280][][279])] AND NOCLAIM [177]

    Friday, August 12, 2016 at 21:32 | Permalink
  10. Magnus wrote:

    This is probably not the place for discussing query optimisation, but your SPARQL runs in 0.4 seconds. How would that timeout?

    Saturday, August 13, 2016 at 15:55 | Permalink
  11. Bruce Myers wrote:

    Sorry, evidently wdqs.wmflabs.org, which I was using, is different from query.wikidata.org.

    Saturday, August 13, 2016 at 18:18 | Permalink