I have ~350 tools running on Toolforge (and on-wiki scripts), so some years ago, I wrote a tool to rule manage them all. That included core to scour my talk pages (and my tools talk pages) on various wikis, github, bitbucket, codeberg, etc. for issue reports, keyword-guess (on wikis) which tool they belong to, and unify the issue reports. As it’s aggregating bug reports, I called it buggregator.
I recently thought about the various long-running Rust services I have running on Toolforge, including web pages with compute (e.g. PetScan), background jobs (e.g. Listeria), web APIs (e.g. wd-infernal), and both (Mix’n’match). These are different from the usual PHP-based webservices, which just spawn a script on demand, and forget about once it has run its course.
To keep an eye on the Rust services, I added a dashboard to buggregator. It tracks all 8 (9 if you count Mix’n’match webservice and background jobs as two separate ones) long-running Rust services, records minutely metrics, and logs restarts and their causes (manually/planned, out-of-memory, crash, killed by the Toolforge watchdog, etc.). It also shows a quick summary of the state of the services.
This is primarily for myself, but if you ever find a service of mine unresponsive, feel free to have a look.
Post a Comment