{"id":638,"date":"2020-03-09T13:44:51","date_gmt":"2020-03-09T12:44:51","guid":{"rendered":"http:\/\/blog.magnusmanske.de\/?p=638"},"modified":"2020-03-09T13:44:51","modified_gmt":"2020-03-09T12:44:51","slug":"eval-not-evil","status":"publish","type":"post","link":"http:\/\/magnusmanske.de\/wordpress\/archives\/638","title":{"rendered":"Eval, not evil"},"content":{"rendered":"<p>My <a href=\"https:\/\/tools.wmflabs.org\/mix-n-match\/#\/\">Mix-n-match tool<\/a> deals with third-party catalogs, and helps matching their entries to Wikidata. This involves, as a necessity, importing minimal information about those entries into the Mix&#8217;n&#8217;match database, but ideally also imports additional metadata, such as (in case of biographical entries) gender, birth\/death dates, VIAF etc., which are invaluable in automatically matching entries to Wikidata items, and thus greatly reduce volunteer workload.<\/p>\n<p>However, virtually none of the (currently) ~2600 catalogs in Mix&#8217;n&#8217;match offers a standardized format to retrieve either basic or meta-data. Some catalogs are imported by volunteers from <a href=\"https:\/\/tools.wmflabs.org\/mix-n-match\/import.php\">tabbed files<\/a>, but most are &#8220;scraped&#8221;, that is, automatically read and parsed, from the source website.<\/p>\n<p>Some source websites are set up in a way that allows a standardized scraping tool to run there, and I offer a <a href=\"https:\/\/tools.wmflabs.org\/mix-n-match\/#\/scraper\/new\">web form<\/a> to create new scrapers; over 1400 of these scrapers have run successfully, and ~750 of them can automatically run again on a regular basis.<\/p>\n<p>But the autoscraper does not handle metadata, such as birth\/death dates, and many catalogs need bespoke import code even for the basic information. Until recently, I had hundreds of scripts, some of them consisting of <a href=\"https:\/\/bitbucket.org\/magnusmanske\/mixnmatch\/src\/139dd44122a1076a7b1f5637c7d376be93cf6c74\/scripts\/person_dates\/update_person_dates.php?at=master\">thousands of lines of code<\/a>, running data retrieval and parsing:<\/p>\n<ul>\n<li>Basic (ID, name, URL) information retrieval from source site<\/li>\n<li>Creating or amending entry descriptions from source site<\/li>\n<li>Importing auxiliary data (other IDs, such as VIAF, coordinates, etc.) from source site<\/li>\n<li>extraction of birth\/death dates from descriptions, taking care not to use estimates, &#8220;flourit&#8221; etc<\/li>\n<li>extraction of auxiliary data from descriptions<\/li>\n<li>linking of two related catalogs (e.g. one for painters, one for paintings) to improve matching (e.g. artworks only from that artist on Wikidata)<\/li>\n<\/ul>\n<p>and many others.<\/p>\n<p>Over time, all this has become unwieldy, unstructured, repetitive; I have written bespoke scrapers only to find that I already had one somewhere else etc.<\/p>\n<p><a href=\"http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2020\/03\/Screenshot-2020-03-09-at-11.27.55.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright size-medium wp-image-639\" src=\"http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2020\/03\/Screenshot-2020-03-09-at-11.27.55-300x267.png\" alt=\"\" width=\"300\" height=\"267\" srcset=\"http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2020\/03\/Screenshot-2020-03-09-at-11.27.55-300x267.png 300w, http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2020\/03\/Screenshot-2020-03-09-at-11.27.55-768x683.png 768w, http:\/\/magnusmanske.de\/wordpress\/wp-content\/uploads\/2020\/03\/Screenshot-2020-03-09-at-11.27.55.png 979w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a>So I went to radically redesign all these processes. My approach is that since only some small piece of code performs the actual scraping\/parsing logic, these code fragments are now stored in the Mix&#8217;n&#8217;match database, associated with the respective catalog. I imported many code fragments from the &#8220;old&#8221; scripts into this table. I also wrote function-specific wrapper code that can load and execute a code fragment (via the <a href=\"https:\/\/www.php.net\/manual\/en\/function.eval.php\">eval<\/a> function, which is often considered &#8220;evil&#8221;, hence the blog post title) on its associated catalog. An example of such code fragments for a catalog can be seen <strong><a href=\"https:\/\/tools.wmflabs.org\/mix-n-match\/#\/code\/2440\">here<\/a><\/strong>.<\/p>\n<p>I can now use that web interface to retrieve, create, test, and save code, without having to touch the command line at all.<\/p>\n<p>In an ideal world, I would let everyone add and edit code here; however, since the framework executes PHP code, this would open the way for all kinds of malicious attacks. I can not think of a way to safeguard against (deliberate or accidental) destructive code, though I have put some mitigations in place, in case I make a mistake. So, for now, you can look, but you can&#8217;t touch. If you want to contribute code (new or patches), please give it to me, and I&#8217;ll be happy to add it!<\/p>\n<p>This code migration is just in its infancy; so far, I support four functions, with a total of 591 code fragments. Many more to come, over time.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>My Mix-n-match tool deals with third-party catalogs, and helps matching their entries to Wikidata. This involves, as a necessity, importing minimal information about those entries into the Mix&#8217;n&#8217;match database, but ideally also imports additional metadata, such as (in case of biographical entries) gender, birth\/death dates, VIAF etc., which are invaluable in automatically matching entries to [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-638","post","type-post","status-publish","format-standard","hentry","category-wikidata"],"_links":{"self":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/638","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/comments?post=638"}],"version-history":[{"count":2,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/638\/revisions"}],"predecessor-version":[{"id":641,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/638\/revisions\/641"}],"wp:attachment":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/media?parent=638"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/categories?post=638"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/tags?post=638"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}