{"id":629,"date":"2019-06-05T11:47:59","date_gmt":"2019-06-05T10:47:59","guid":{"rendered":"http:\/\/blog.magnusmanske.de\/?p=629"},"modified":"2019-06-05T11:47:59","modified_gmt":"2019-06-05T10:47:59","slug":"batches-of-rust","status":"publish","type":"post","link":"http:\/\/magnusmanske.de\/wordpress\/archives\/629","title":{"rendered":"Batches of Rust"},"content":{"rendered":"<p><a href=\"https:\/\/tools.wmflabs.org\/quickstatements\">QuickStatments<\/a> is a workhorse for Wikidata, but it had <a href=\"http:\/\/magnusmanske.de\/wordpress\/?p=627\">a few problems<\/a> of late.<\/p>\n<p>One of those is bad performance with batches. Users can submit a batch of commands to the tool, and these commands are then run on the Labs server. This mechanism has been bogged down for several reasons:<\/p>\n<ul>\n<li>Batch processing written in PHP<\/li>\n<li>Each batch running in a separate process<\/li>\n<li>Limitation of 10 database connection per tool (web interface, batch processes, testing etc. <em>together<\/em>) on Labs<\/li>\n<li>Limitation of (16? observed but not validated) simultaneous processes per tool on Labs cloud<\/li>\n<li>No good way to auto-start a batch process when it is submitted (currently, auto-starting a PHP process every 5 minutes, and exit if there is nothing to do)<\/li>\n<li>Large backlog developing<\/li>\n<\/ul>\n<p>Amongst continued bombardment on Wiki talk pages, Twitter, Telegram etc. that &#8220;my batch is not running (fast enough)&#8221;, I went to mitigate the issue. My approach is to do all the batches in a <a href=\"https:\/\/github.com\/magnusmanske\/quickstatements_rs\">new processing engine<\/a>, written in Rust. This has several advantages:<\/p>\n<ul>\n<li>Faster and easier on the resources than PHP<\/li>\n<li>A single process running on Labs cloud<\/li>\n<li>Each batch is a thread within that process<\/li>\n<li>Checking for a batch to start every second (if you submit a new batch, it should start almost immediately)<\/li>\n<li>Use of a database connection pool (the individual thread might have to wait a few milliseconds to get a connection, but the system never runs out)<\/li>\n<li>Limiting simultaneous batch processing for batches from the same user (currently: 2 batches max) to avoid the MediaWiki API &#8220;you-edit-too-fast&#8221; error<\/li>\n<li>Automatic handling of maxlag, bot\/OAuth login etc. by using my <a href=\"http:\/\/magnusmanske.de\/wordpress\/?p=608\">mediawiki crate<\/a><\/li>\n<\/ul>\n<p>This is now running on Labs, processing all (~40 at the moment) open batches simultaneously. <a href=\"https:\/\/grafana.wikimedia.org\/d\/000000170\/wikidata-edits?refresh=1m&amp;orgId=1\">Grafana<\/a> shows the spikes in edits, but no increased lag so far. The process is given 4GB of RAM, but could probably do with a lot less (for comparison, each individual PHP process used 2GB).<\/p>\n<p>A few caveats:<\/p>\n<ul>\n<li>This is a &#8220;first attempt&#8221;. It might break in new, fun, unpredicted ways<\/li>\n<li>It will currently <em>not<\/em> process batches that deal with Lexemes. This is mostly a limitation of the <a href=\"https:\/\/crates.io\/crates\/wikibase\">wikibase<\/a> crate I use, and will likely get solved soon. In the meantime, please run Lexeme batches only within the browser!<\/li>\n<li>I am aware that I have now code duplication (the PHP and the Rust processing). For me, the solution will be to implement QuickStatements command parsing in Rust as well, and replace PHP completely. I am aware that this will impact third-party use of QuickStatements (e.g. the <a href=\"https:\/\/github.com\/wmde\/wikibase-docker\">WikiBase docker container<\/a>), but the PHP and Rust sources are independent, so there will be no breakage; of course, the Rust code will likely evolve away from PHP in the long run, possibly causing incompatabilities<\/li>\n<\/ul>\n<p>So far, it seems to be <a href=\"https:\/\/tools.wmflabs.org\/quickstatements\/#\/batches\">running fine<\/a>. Please let me know if you encounter any issues (unusual errors in your batch, weird edits etc.)!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>QuickStatments is a workhorse for Wikidata, but it had a few problems of late. One of those is bad performance with batches. Users can submit a batch of commands to the tool, and these commands are then run on the Labs server. This mechanism has been bogged down for several reasons: Batch processing written in [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15,9],"tags":[],"class_list":["post-629","post","type-post","status-publish","format-standard","hentry","category-rust","category-wikidata"],"_links":{"self":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/629","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/comments?post=629"}],"version-history":[{"count":1,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/629\/revisions"}],"predecessor-version":[{"id":630,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/posts\/629\/revisions\/630"}],"wp:attachment":[{"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/media?parent=629"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/categories?post=629"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/magnusmanske.de\/wordpress\/wp-json\/wp\/v2\/tags?post=629"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}