alyaza [they/she]@beehaw.orgM to Technology@beehaw.orgEnglish · 1 year agoThe rise and fall of robots.txt: As unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart.www.theverge.comexternal-linkmessage-square4linkfedilinkarrow-up138arrow-down10cross-posted to: technology@lemmy.world
arrow-up138arrow-down1external-linkThe rise and fall of robots.txt: As unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart.www.theverge.comalyaza [they/she]@beehaw.orgM to Technology@beehaw.orgEnglish · 1 year agomessage-square4linkfedilinkcross-posted to: technology@lemmy.world
minus-squaregayhitler420@lemm.eelinkfedilinkarrow-up14·1 year agorobots.txt isn’t a basic social contract, it’s a file intended to save web crawlers precious resources.
robots.txt isn’t a basic social contract, it’s a file intended to save web crawlers precious resources.