{"id":471,"date":"2017-10-11T18:02:17","date_gmt":"2017-10-11T17:02:17","guid":{"rendered":"http:\/\/blog.magnusmanske.de\/?p=471"},"modified":"2017-10-11T21:35:05","modified_gmt":"2017-10-11T20:35:05","slug":"the-big-ones","status":"publish","type":"post","link":"http:\/\/magnusmanske.de\/wordpress\/archives\/471","title":{"rendered":"The Big Ones"},"content":{"rendered":"<p style=\"padding-left: 30px;\"><em>Update:<\/em> After fixing an import error, and cross-matching of BNF-supplied VIAF data, 18% of BNF people are matched in Wikidata. This has been corrected in the text.<\/p>\n<p>My <a href=\"https:\/\/tools.wmflabs.org\/mix-n-match\/\">mix&#8217;n&#8217;match tool<\/a> holds a lot of entries from third-party catalogs &#8211;\u00a021,795,323 at the time of writing. That&#8217;s a lot, but it doesn&#8217;t cover &#8220;the big ones&#8221; &#8211; VIAF, BNF, etc., which hold many millions of entries each. I could &#8220;just&#8221; (not so easy) import those, but:<\/p>\n<ul>\n<li>Mix&#8217;n&#8217;match is designed for small and medium-sized entry lists, a few hundred thousand at best. It does not scale well to larger catalog sizes<\/li>\n<li>Mix&#8217;n&#8217;match is designed to work with many different catalogs, so the database structure represents the least common denominator &#8211; ID, title, short description. Catalog-specific metadata gets lost, or is not easily accessible after import<\/li>\n<li>The sheer number of entries might require different interface solutions, as well as automated matching tools<\/li>\n<\/ul>\n<p><a href=\"http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-11-at-17.54.39.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright size-medium wp-image-472\" src=\"http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-11-at-17.54.39-300x214.png\" alt=\"\" width=\"300\" height=\"214\" srcset=\"http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-11-at-17.54.39-300x214.png 300w, http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-11-at-17.54.39-768x547.png 768w, http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-11-at-17.54.39-1024x729.png 1024w, http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2017\/10\/Screen-Shot-2017-10-11-at-17.54.39.png 1791w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a>To at least get a grasp of how many entries we are dealing with in these catalogs, and inspired by the <a href=\"https:\/\/meta.wikimedia.org\/wiki\/Grants:Project\/Hjfocs\/soweego\">Project soweego<\/a> proposal, I have used a <a href=\"http:\/\/www.bnf.fr\/fr\/acc\/x.accueil.html\">BNF<\/a> data dump to extract 1,637,195 entries (less than I expected) into a new database, one that hopefully will keep other large catalogs in the future. There is much to do; currently, only <s>102,115<\/s> 295,763 entries (~<s>6<\/s>18%) exist on Wikidata, according to the SPARQL query service.<\/p>\n<p>As one can glimpse from the screenshot, I have also extracted some metadata into a &#8220;proper&#8221; database table. All this is preliminary; I might have missed entries or good metadata, or gotten things wrong. For me, the important thing is that (a) there is some query-able data on <s>Labs<\/s> <a href=\"https:\/\/tools.wmflabs.org\/\">Toolforge<\/a>, and that (re-)import and matching of the data is fully automated, so it can be re-run is something turns out to be problematic.<\/p>\n<p>I shall see where I go from here. Obvious candidates include auto-matching (via names and dates) to Wikidata, and adding BNF references to relevant statements. If you have a Toolforge user account, you can access the new database (read-only) as <tt>s51434__mixnmatch_large_catalogs_p<\/tt>. Feel free to run some queries or build some tools around it!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Update: After fixing an import error, and cross-matching of BNF-supplied VIAF data, 18% of BNF people are matched in Wikidata. This has been corrected in the text. My mix&#8217;n&#8217;match tool holds a lot of entries from third-party catalogs &#8211;\u00a021,795,323 at the time of writing. That&#8217;s a lot, but it doesn&#8217;t cover &#8220;the big ones&#8221; &#8211; [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-471","post","type-post","status-publish","format-standard","hentry","category-wikidata"],"_links":{"self":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/471","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/comments?post=471"}],"version-history":[{"count":4,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/471\/revisions"}],"predecessor-version":[{"id":476,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/471\/revisions\/476"}],"wp:attachment":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/media?parent=471"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/categories?post=471"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/tags?post=471"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}