๐ŸŽฏ Coordinator

Loadingโ€ฆ
Run progress
โ€“
no active run
Throughput
โ€“
URLs / sec
ETA
โ€“
est. remaining
Jobs
โ€“
active / queued
Clients
โ€“
active / total
Proxies
โ€“
active / total
Loading statisticsโ€ฆ

๐Ÿš€ Add a machine

Run this on a clean Debian/Ubuntu box as root. It installs the proxy and/or crawler client, auto-registers with this coordinator (client ID from the box's reverse DNS), and reinstalls itself on reboot via systemd. Safe to re-run.

TOKEN is the coordinator's REGISTER_TOKEN. Each proxy is issued its own auth token automatically and saved to /srv/fhscrape/proxy.env on the box.