{"id":416,"date":"2016-08-03T22:11:34","date_gmt":"2016-08-03T21:11:34","guid":{"rendered":"http:\/\/blog.magnusmanske.de\/?p=416"},"modified":"2016-08-03T22:11:35","modified_gmt":"2016-08-03T21:11:35","slug":"livin-on-the-edge","status":"publish","type":"post","link":"http:\/\/magnusmanske.de\/wordpress\/archives\/416","title":{"rendered":"Livin’ on the edge"},"content":{"rendered":"

A few days ago, Lydia posted<\/a>\u00a0about the first prototype of the new structured data system for Commons<\/a>, based on Wikidata<\/a> technology. While this is just\u00a0a first step, structured data for Commons seems\u00a0finally within reach.<\/p>\n

And that brings home the reality of over 32 million files on Commons, all having unstructured<\/em> data about them, in the shape of the file description pages. It would be an enormous task do manually transcribe\u00a0all these descriptions, licenses, etc. to the appropriate data structures. And while\u00a0we will have to do just that for many of the files, the ones that can<\/em> be\u00a0transcribed by a machine, should be.<\/p>\n

So I went ahead and re-wrote a prototype tool I had build for just this occasion a while ago. I call it CommonsEdge<\/b><\/a>\u00a0(a play on Common sedge<\/a>). It is both an API, and an interface to that API. It will parse a file description page on Commons, and return a JSON object with the data elements corresponding to the description page.\u00a0An important detail is that this parser\u00a0does not just pick some elements it understands, and ignore the rest; internally, it tries to “explain” all elements of the description (templates, links, categories, etc.) as data, and fails if it\u00a0can not explain one. That’s right, the API call will fail<\/em>\u00a0with an error, unless 100% of the page would be\u00a0represented in the JSON object returned. This prevents “half-parsed” pages; a file description page that is successfully pared by the API can safely be replaced in its entirety by the resulting structured data. In case of failure, the\u00a0error message is usually quite specific and detailed about the cause; this allows for incremental improvements of the parser.<\/p>\n

\"Screen<\/a>At the moment of writing, I find\u00a0that ~50-60% of\u00a0file descriptions (based on sets of 1000 random files) produce a JSON object, that is, can be completely understood by the parser, and completely represented in the result. That’s 16-19 million files descriptions that can be converted to structured data automatically, today. Most of the failures appear to be due to bespoke templates; the more common ones can be added over time.<\/p>\n

A word about\u00a0the output: Since the structured data setup, including properties and foreign keys, is\u00a0still in flux, I opted for a simple output format. It is not<\/em> Wikibase format, but similar; most elements (except categories and coordinates, I think) are just lists of type-and-value tuples (example<\/a>).\u00a0I try to use URLs as much as possible, for example, when referencing\u00a0users on Commons (or other Wikimedia projects) or flickr. Licenses are currently links to the Wikidata element corresponding to the used template (ideally, I would like to resolve that\u00a0through Wikidata properties pointing to the appropriate license).<\/p>\n

Source code is available<\/a>. Pull requests are welcome.<\/p>\n","protected":false},"excerpt":{"rendered":"

A few days ago, Lydia posted\u00a0about the first prototype of the new structured data system for Commons, based on Wikidata technology. While this is just\u00a0a first step, structured data for Commons seems\u00a0finally within reach. And that brings home the reality of over 32 million files on Commons, all having unstructured data about them, in the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13,9,7],"tags":[],"class_list":["post-416","post","type-post","status-publish","format-standard","hentry","category-commons-2","category-wikidata","category-wikimedia-2"],"_links":{"self":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/416","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/comments?post=416"}],"version-history":[{"count":2,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/416\/revisions"}],"predecessor-version":[{"id":419,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/416\/revisions\/419"}],"wp:attachment":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/media?parent=416"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/categories?post=416"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/tags?post=416"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}