Consequences (and potential consequences) of AI web crawlers

Feedback Website
1

Recently I’ve found that the main website (at www.mentalhealthlaw.co.uk) is sometimes too slow to be usable, and occasionally just displays an error message. This is because of crawlers – likely AI crawlers – from all over the world.

I’ve started adding a password temporarily so that only I can access the website when working on it. I try only to do that out of office hours, but it’s not a good long-term solution.

Next I plan to try a service called Cloudflare which is meant to help, but will mean having to click a “Verify you are human” box occasionally.

If the problem gets worse rather then better then it might be necessary to make the website a subscription service. I don’t know how that would work, but maybe it could be tied in with the CPD scheme.

Here’s an article which I found tonight but explains the situation well: Steven J Vaughan-Nichols, ‘AI web crawlers are destroying websites in their never-ending hunger for any and all content: But the cure may ruin the web…’ (The Register, 29 August 2025).

Any ideas?

2

I installed Cloudflare this afternoon and added some custom rules about when the human verification box appears. You should rarely see it. It blocked over 200,000 requests from all sorts of countries in the first few hours, and also caches many files so that you download them from their servers rather than mine.

The website seems to be lightning fast again! So far so good.

1 Like
3

Great news Jonathan. I was meaning to reply to this post but hadn’t found the time yet. I was going to strongly suggest the site should be kept free to access as - although I use it religiously and would be very happy to pay for it as a resource - it’s a unique forum that does a lot of good for people who would be unable to pay for a variety of reasons.

Fingers crossed Cloudflare continues to work it’s magic.

1 Like