But companies are crawling everything like mad - I’ve noticed a 400% upturn this year alone in bot traffic on a low traffic web forum and a few sites I host, so much so that I’m having to do some fairly heavy filtering upstream to keep them out. (They don’t resepect robots.txt, obviously)
When bot traffic outnumbers legitimate traffic at least 10x, it makes you wonder why you’re paying to host stuff.
“We’re shocked” - nobody.
But companies are crawling everything like mad - I’ve noticed a 400% upturn this year alone in bot traffic on a low traffic web forum and a few sites I host, so much so that I’m having to do some fairly heavy filtering upstream to keep them out. (They don’t resepect robots.txt, obviously)
When bot traffic outnumbers legitimate traffic at least 10x, it makes you wonder why you’re paying to host stuff.