{"id":710,"date":"2023-09-25T17:00:33","date_gmt":"2023-09-25T16:00:33","guid":{"rendered":"http:\/\/blog.magnusmanske.de\/?p=710"},"modified":"2023-09-25T17:00:33","modified_gmt":"2023-09-25T16:00:33","slug":"get-into-the-flow","status":"publish","type":"post","link":"http:\/\/magnusmanske.de\/wordpress\/archives\/710","title":{"rendered":"Get into the Flow"},"content":{"rendered":"

Unix philosophy<\/a> contains the notion that each program should perform an single function (and perform that function exceptionally well), and then be used together with other single-function programs to form a powerful “toolbox”, with tools connected via the geek-famous pipe (“|”).<\/p>\n

\"\"<\/a>

A ToolFlow workflow, filtering and re-joining data rows<\/p><\/div>\n

The Wikimedia ecosystem has lots of tools that perform a specific function, but there is little in terms of interconnection between them. Some tool prorammers have added other tools as (optional) inputs (eg PetScan) or outputs (eg PagePile), but that is the extend of it. There is PAWS<\/a>, a Wikimedia-centric Jupyter notebook, but it does require a certain amount of coding, which excludes many volunteers.<\/p>\n

So I finally got around to implementing the “missing pipe system” for Wikimedia tools, which I call ToolFlow<\/a><\/strong>. I also started a help page<\/a> explaining some concepts. The basic idea is that a user can create a new workflow (or fork an existing one). A workflow would usually start with one or more adapters<\/em> that represent the output of some tool (eg PetScan<\/a>, SPARQL<\/a> query, Quarry<\/a>) for a specific query. The adapter queries the tool, and represents the tool output in a standardized internal format (JSONL) that is written into a file on the server. These files can then be filtered and combined to form sub- and supersets. Intermediate files will be cleaned up automatically, but the output of steps (ie nodes) that have no further steps after them is kept, and can only be cleared by re-running the entire workflow run. Output can also be exported via a generator<\/em>; at the moment, the only generator is a Wikipage editor, which will create a Wikitext table on a Wiki page of your choice from an output file.<\/p>\n

Only the owner (=creator) of a workflow can edit or run it, but the owner can also set a scheduler (think UNIX cronjob<\/em>) for the workflow to be run regularly every day, week, or month. ToolFlow remembers your OAuth details, so it can edit wiki pages regularly, updating a wikitext table with the current results of your workflow.<\/p>\n

I have created some demo workflows:<\/p>\n