Rate-limited websites

Sometimes when Insites tests a website, the website becomes inaccessible to Insites. This is usually due to one of the following:

  • The website cannot handle many visitors
  • The website has protection designed to prevent attacks by bots
  • The website was experiencing temporary downtime

If this happens, Insites may report that links were broken which appear to work for you, or that a page was an error message that you can’t see.

The website cannot handle many visitors

This is more common on smaller websites. If a website is large and well funded, it is more likely to be the website has protection against attacks by bots.

By default, Insites will attempt to download up to six webpages from a website at once, which is equivalent to six highly active visitors browsing the site. For some websites, this is more than they can handle, and the website slows down or may fail to respond entirely.

You have two options to fix this:

For the latter, we recommend checking the desktop speed test to see if specific pages are slow – e.g. if a badly programmed piece of code is overwhelmed – and if so, focus on fixing those. If a website is using very low grade hosting without caching, that is likely the cause.

The website has protection designed to prevent attacks by bots

A common attack on websites is a Distributed Denial of Service (DDoS) attack, where a large number of computers automatically attempt to access a website at once. The idea is to overwhelm the website with so much traffic that it cannot be accessed by other people.

Some websites have systems designed to protect against this. They detect what they believe too many requests, and block those requests.

If this happens to Insites, then Insites will be unable to see the website. There are two things you can try:

The website was experiencing temporary downtime

For example, sometimes a website may be unavailable while a new version is made live, or while it is backed up.

This is very rare, but easy to test. If the website is working for you now, re-test it in Insites. If you still have problems, this is not the cause.