@fossilesque@mander.xyzM to Science Memes@mander.xyzEnglish • 14 days agoBlack Mirror AImander.xyzimagemessage-square211fedilinkarrow-up11.59Karrow-down115
arrow-up11.58Karrow-down1imageBlack Mirror AImander.xyz@fossilesque@mander.xyzM to Science Memes@mander.xyzEnglish • 14 days agomessage-square211fedilink
minus-square@jaschen@lemm.eelinkfedilinkEnglish4•14 days agoIt’s naive to assume that google crawlers respect robot.txt.
minus-square@rosco385@lemm.eelinkfedilinkEnglish3•14 days agoIt’d be more naive to have a robot.txt file on your webserver and be surprised when webcrawlers don’t stay away. 😂
It’s naive to assume that google crawlers respect robot.txt.
It’d be more naive to have a robot.txt file on your webserver and be surprised when webcrawlers don’t stay away. 😂