{"id":707,"date":"2023-07-07T15:53:11","date_gmt":"2023-07-07T14:53:11","guid":{"rendered":"http:\/\/blog.magnusmanske.de\/?p=707"},"modified":"2023-07-07T15:53:11","modified_gmt":"2023-07-07T14:53:11","slug":"mixnmatch-background-sync","status":"publish","type":"post","link":"http:\/\/magnusmanske.de\/wordpress\/archives\/707","title":{"rendered":"Mix&#8217;n&#8217;match background sync"},"content":{"rendered":"<p>My <a href=\"http:\/\/mix-n-match.toolforge.org\">Mix&#8217;n&#8217;match tool<\/a> helps matching third-party catalogs to Wikidata items. Now, things happen on Mix&#8217;n&#8217;match and Wikidata in parallel, amongst them:<\/p>\n<ul>\n<li>Wikidata items are deleted<\/li>\n<li>Wikidata items are merged, leving one to redirect to the other<\/li>\n<li>External IDs are added to Wikidata<\/li>\n<\/ul>\n<p>This leads to the states of Mix&#8217;n&#8217;match and Wikidata diverging over time. I already had some automated measures in place to keep them in sync, and there is a &#8220;manual sync&#8221; function on each catalog that has a Wikidata property, but it is not ideal, especially since deleted\/redirect items can show up as mismatches in various places.<\/p>\n<p>I previously blogged about another tool of mine, <a href=\"http:\/\/magnusmanske.de\/wordpress\/?p=665\">Wikidata Recent Changes<\/a> (WDRC), which records fine-grained changes of Wikidata items, and offers an API to check on them. The other day, I added recording of item creations, deletions, and redirect creations. So it was finally time to use WDRC myself. Every 15 minutes, Mix&#8217;n&#8217;match now<\/p>\n<ul>\n<li>un-matches all deleted items<\/li>\n<li>points matches to redirected items to the redirect target<\/li>\n<li>checks all catalogs with a Wikidata property for new external IDs on Wikidata, and sets them as matches (if the Mix&#8217;n&#8217;match entry is not already set).<\/li>\n<\/ul>\n<p>Please note that this applies only to edits <em>from now on<\/em>; there may be many Mix&#8217;n&#8217;match entries that are still matched to deleted\/redirected items on Wikidata. But, one thing at a time.<\/p>\n<p>Technical remark: I am using the JSONL output format of WDRC, for several reasons:<\/p>\n<ul>\n<li>On the WDRS side, the lines are just printed out, so no need to cache some giant result set when generating it<\/li>\n<li>On the consumer side (Mix&#8217;n&#8217;match), I can stream the API result into a temporary file, which consumes no memory. Then, I read the file line-by-line, using only memory for a single entry<\/li>\n<\/ul>\n<p>This way, I can request (almost) unlimited output from the API, and process it reliably, with very little resources (which are at a premium these days on Toolforge).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>My Mix&#8217;n&#8217;match tool helps matching third-party catalogs to Wikidata items. Now, things happen on Mix&#8217;n&#8217;match and Wikidata in parallel, amongst them: Wikidata items are deleted Wikidata items are merged, leving one to redirect to the other External IDs are added to Wikidata This leads to the states of Mix&#8217;n&#8217;match and Wikidata diverging over time. I [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-707","post","type-post","status-publish","format-standard","hentry","category-wikidata"],"_links":{"self":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/707","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/comments?post=707"}],"version-history":[{"count":1,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/707\/revisions"}],"predecessor-version":[{"id":708,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/707\/revisions\/708"}],"wp:attachment":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/media?parent=707"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/categories?post=707"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/tags?post=707"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}