I built a routine that publishes posts for me. Then I spent longer on the thing that checks the routine than I did on the routine itself, which felt daft at the time and doesn't now.
The Step That Was Bothering Me
Publishing a post here used to go like this. Write the markdown. Hand-edit a JSON file with the title, the excerpt, the tags, a reading time I mostly guessed at. Run the build. Commit, push, wait a bit. Then open the page and have a look.
It's that last step I'd started to have a problem with.
"Have a look" isn't a check. It's a bloke glancing at a screen and deciding it seems about right. It catches a blank page. It does not catch a stale title, or a post that never made it into the feed, or a card missing from the index because the grid didn't rebuild properly.
I know it doesn't catch those things, because a while back it didn't.
The Bug That Started All This
Earlier in the year I had a post go live with an "Article Not Found" message sitting underneath it. Content all present and correct, and then this apologetic little error block hanging off the bottom like a receipt nobody asked for.
The page returned a 200. Title was right. I'd looked at it. I'd looked at the top of it, anyway, which it turns out is not the same thing.
So when I came to automate the checking, the obvious move was: fetch the live page, search the HTML for "Article Not Found", fail if it's there.
Except that check is completely worthless, and it took me an embarrassingly long time to notice why.
The error block is part of the page template. It's on every post, all the time, sitting there hidden. So searching for that text always finds it. A check built that way would have failed every single post I've ever published, including the thirteen perfectly good ones. Flip the logic to make it pass and it then passes everything, including the broken one.
The thing that actually matters is whether the block is still hidden. Present and invisible is correct. Present and visible is the bug.
Two checks, near enough identical to look at. One of them tells you precisely nothing. That's the bit I keep coming back to.
What It Checks Now
The routine ends by fetching the live site and asserting eleven things. Some are boring - did it return a 200, is the title right. The ones that earn their keep are less obvious.
Does the page say it was prerendered? Every generated page carries a little attribute naming its own slug. No attribute means the build didn't produce this post and the page has quietly fallen back to rendering in the browser instead, which is exactly the situation the original bug lived in. This is the check that goes after the cause rather than the symptom.
Is there actually any text? A page can return a 200, have a perfect title, and contain nothing whatsoever. So it counts the words. Crude, but it works.
Is the post in the sitemap, the feed, and the card grid? Three separate files, generated by three different bits of the build. A post can render beautifully and still be effectively invisible because one of those didn't get rebuilt.
If anything fails, it reverts the merge, pushes, and confirms the post has actually gone. Then it puts the ticket back in review with the number of the check that failed and what the page really contained, so next time I start from a fact rather than a hunch.
Then I Tested the Tester
Here's the part that justifies the whole faff.
A kill switch you've never fired isn't a kill switch, it's a comfort blanket. So before I trusted it with anything, I wrote something that stands up a copy of the site, breaks the page seven different ways, and checks the verifier spots each one and points at the right thing.
All seven caught the right failure. And all seven reported the wrong result.
Turns out that on Windows, if you end a process while it's still got network connections open, you don't get your exit code back. You get a crash code instead. So the checker was printing a very pleased-with-itself "11 out of 11 passed" and then exiting with a number that means the process fell over.
Which matters, because the exit code is the whole decision. That's what picks between "leave it up" and "rip it back out". Every successful check would have been read as a failure. It would have cheerfully reverted good posts all day long while its own report insisted everything was fine.
The checks were right. The bit reporting on the checks was lying. I'd never have found that by reading the code, and I say that as someone who read the code.
Yes, This Was Disproportionate
I should be straight about the effort here, because the ratio looks mad written down.
The publishing bit took an afternoon. Read a ticket, write a file, run a build, push a commit. Not hard, and honestly the genuinely tricky problems were solved by the build script months ago.
Everything else took days. The tone-of-voice checker, the eleven assertions, the seven ways of breaking it on purpose, and then fixing the bug that the breaking-it-on-purpose turned up.
And the payoff isn't speed. I can publish a post by hand in ten minutes. I've done it thirteen times. The payoff is that I don't have to hold the checklist in my head any more, and if something does go wrong it takes itself back down without me finding out about it on a Tuesday three weeks later.
That only pays off if the checks are real ones. Otherwise I've just built a more elaborate way of hoping.
The Bit I Actually Care About
This is the same argument I end up having with people about anything they've built quickly.
Writing the thing is now genuinely easy. It's never been easier. What most people skip is everything after: the maintenance, the support, the patching, all the fun stuff nobody demos. Verification belongs on that list, and it's usually the first thing off it.
So the question I'd ask of anything you've automated is: what would have to be true for this to have worked, and which of those does it actually check? Most of the time the honest answer is "it didn't throw an error", which tells you something about your code and nothing at all about the world.
The gap between those two is where things go wrong quietly, for weeks, while everyone assumes it's fine.
I might be overthinking it. I've been told before that I do. But every automation I've seen go properly wrong went wrong in that gap, and it was never the running of it that failed. It was that nobody was checking.
Got something you've automated but don't fully trust? That's the gap our business process automation work lives in. Give me a shout and we can talk about what proving it would take.