Last Week
Last week, I was working through tablet screenshots, keeping the product videos moving, and monitoring a database upgrade in integration. This week, an unexpected infrastructure problem took over: my self-hosted GitHub runners stopped getting through their jobs.
Shokken is available in the US on Android and iOS. I am also preparing a reduced introductory Pro offer for the first group of users, but the subscription infrastructure for that offer is not ready yet. Subscribing normally still uses the existing pricing. If you are interested in the introductory offer, get in touch so I can arrange the license manually.
That was the work I wanted to move forward. Instead, I spent the week getting my development pipeline working again.
Why I Run the Tests at Home
Shokken has an automated pipeline for builds, tests, and verification. GitHub coordinates those jobs, while my own hardware does the work.
Some of the heavier tests run the app inside an emulator. The pipeline has to build the app, start the simulated device, install the app, run through the scenarios, and report the results. That needs enough processing power and memory to finish in a useful amount of time.
Self-hosting lets me put hardware I already own to work. I have a runner pool on a virtual machine managed by Proxmox at home, with enough capacity for those heavier jobs. In my experience, the full pipeline generally finishes in about twenty to thirty minutes.
That has been a useful balance of cost and performance. It also means that when something underneath the runner breaks, fixing it is my job.
Everything Worked Except the Downloads
I have dealt with runner maintenance before. At one point, I had pinned the runner software to a version that became too old to keep accepting jobs. That was my responsibility, and I now use Ansible to keep the runners updated.
When jobs started failing this week, I checked that first. The updates were working. The failures were happening during downloads.
I deliberately fetch fresh files for these jobs because I have had enough trouble with stale caches in the past. Normally, those downloads are quick. Suddenly, some were crawling along at kilobytes per second until they timed out.
I worked through possible local causes, including disk space and operating-system problems, before investigating the network. My home connection was still fast in ordinary speed tests. Most websites worked normally. The problem seemed specific to the route used to reach GitHub resources.
One packet-loss test reported roughly 96 percent loss, with some packets arriving out of order. That result, together with the stalled downloads, pointed toward a network-path problem. It did not tell me which operator was responsible or exactly where the fault was.
That is an awkward problem to explain: the internet is up, the usual sites load, and the speed test looks good, but the service I need to validate my app is effectively unusable.
What Does It Mean in English?
A fast internet connection does not guarantee a fast trip to every destination. The route matters too.
Imagine that the road out of your neighborhood is clear, and you can drive to most places without trouble. One particular destination requires a stretch of road where traffic barely moves. Measuring how quickly you can reach a nearby shop will not reveal that problem.
The runner still had a working connection. It needed a different way to reach the resources its jobs depended on.
A VPN Got the Pipeline Moving Again
Moving everything to GitHub-hosted runners was not a quick switch for my setup. I would need to change and validate roughly twenty workflows, especially the ones that run emulator tests, and reassess the capacity and cost.
Instead, I tried changing the network route using a VPN tunnel I already had available.
I configured a routing policy on my UniFi firewall so the runner virtual machine used that tunnel, with an endpoint in Los Angeles. As soon as it was in place, the jobs started passing again. I was seeing downloads in the range of fifty to one hundred megabits per second, which was what I expected from that connection.
The tunnel gave me a working path. It did not repair the original route, which was still having trouble at the time of recording.
The workaround was straightforward once I understood what was happening. Diagnosing the failure, finding the workaround, and validating it took the week.
The Cost That Does Not Appear on the Bill
I still get useful performance from my own hardware, and self-hosting helps keep runner spending down. But the maintenance cost is real, even when it arrives as lost development time instead of an invoice.
I was already trying to fix a problem in the app when the pipeline failed. I needed the tests to validate that work. Suddenly, I had to stop developing Shokken and repair the infrastructure that made development possible.
As a hobby project, tracking down a strange network issue could be interesting. When the same infrastructure is blocking the product I am trying to build, it feels very different.
For now, I am keeping the self-hosted runners. They are working again, and changing providers would take its own effort. But I am giving more thought to paying someone else to take on part of that responsibility.
If Shokken starts bringing in paying users, this may be one of the first places where spending more money makes sense. The question is how much development time I can afford to lose keeping the cheaper setup running.