What we got wrong measuring 209 websites

We built the Estate Agent Website Index by pointing a script at 209 firms and scoring what came back. The first version of that script was wrong about 23 of them, and it was wrong in a way that would have been published.

Since we are asking the trade to take our measurements seriously, here is the part that did not work.

Twenty-three sites that were not broken

The first run produced a neat cluster at the bottom of the table. Sixteen firms scored exactly 33 out of 100. A second run produced seven more, all on exactly 45.

Identical scores across unrelated firms is the tell. Real websites do not fail in the same way to the same decimal. Something in our own script was producing that number, not the sites.

Both groups had refused us. The sixteen returned an HTTP 403 — their server looked at an automated request and declined to answer it, which is a fact about their firewall and says nothing whatever about the website a person sees. The seven were behind a bot check: a 166-byte holding page that redirects a real browser to a captcha and hands a script a blank. Our scanner read that holding page as their homepage, found no enquiry form, no listings and no mobile layout, and marked them down for all three.

Both groups are now excluded from the Index and the reason is published on the page rather than buried. They are not scored badly. They are not scored.

Three smaller ones, same shape

A firm was marked down for an invalid security certificate because its bare domain mismatched. Its www address, the one it actually redirects to, was perfectly valid, and our checker gave up before trying it.

And the single worst-scoring firm in an early run turned out to be a network timeout on our end. Their site answered normally a few minutes later. "Did not load" is the most damaging line a scorecard can carry, so it now gets a second attempt before we write it.

Why we are telling you

Partly because the Index is only worth anything if the method can be checked, and a method nobody can audit is an opinion with numbers on it.

Mostly because of what those errors would have cost. Twenty-three named local firms, told in writing that their website was failing, when the truth was that their security was working properly and ours was not clever enough to notice. Getting that letter would be irritating. Getting it from someone trying to sell you a website would be worse.

The rule that came out of it: a fault we invent costs more than a fault we miss. Every ambiguous case now resolves to "cannot assess" rather than a low score, which is why 17 of the original 209 are excluded from the table and the exclusions are counted in public.

What this means for your scorecard

It means the findings on it are ones we are prepared to defend. If anything on yours looks wrong, it may well be — tell us and we will check it by hand and correct the record. Two of the fixes above came from doing exactly that against sites we could verify ourselves.

Ask for yours. It is free, and it is a rank out of 192 with the working shown.