We released the WLAN Pros Toolbox, 170+ free Wi-Fi tools, a week ago. A couple days later, we went hunting for our own mistakes.
We found them.
Ping Sweep scans a subnet and tells you which devices are alive. It was calling dead ones alive. You cross it off your list, skip the investigation, and go looking somewhere else while the thing you needed sits offline. (The mistake was showing the time-out value next to every single IP Address)
The corrections are live on iPhone, Mac, and Android. They are not on Windows. That specific build machine was left back in my home office and the build key along with it, so Windows users are on the pre-audit build right now, dead hosts called alive and all. I would rather tell you than let you find out. (It will be a quick fix once I return to the office)
We are opening the source under AGPL-3.0, so you will be able to check us. Then, the night before this post was meant to go out, I found a worse one on my own phone.
I was on cellular. No Wi-Fi at all. I opened Test My Connection, the front door of the app, and it showed me a Wi-Fi signal card with a green LIVE badge, a Wi-Fi data rate of 29 Mbps, and this verdict: "Your internet can carry more than your Wi-Fi link is passing. Boost the Wi-Fi signal to raise the ceiling.
The app just told me, with no Wi-Fi to go improve his Wi-Fi. The 29 Mbps was a stale reading from the last time that phone had been on a network. The app kept showing the last number it had ever seen, and it labeled that number Live.
The bug was not new. It shipped in earlier releases, which means it was live on our own front page the whole time we spent auditing ourselves and feeling good about it. Our audit swept reference data, the tables and formulas behind the numbers. This one was a state bug: what the app does when the thing it is measuring is not there. We asked whether our numbers were right. We never asked whether the app was honest when it had no number at all.
We fixed it, and I held this post until the corrected build was live. Publishing "we audited ourselves and fixed it" on top of a live bug on our own front page would have been the exact thing this post argues against.
We say we care about accuracy and about this community.
This is what that looks like when nobody is making us.
When a tool hands you a result, how do you know you can trust it?