The world of artificial intelligence moves fast. While court cases attempt to decide whether using copyrighted text, images, and video to train AI models is “fair use”, as tech companies argue, those same firms are already running out of new data to harvest.An MIT research group led the study which looked at 14,000 web domains that are included in three major AI training data sets.
Robot.txt, a decades-old method for website owners to stop automated bots from crawling their pages, is increasingly being deployed to block tech companies from collecting data., some AI executives worry about hitting the “data wall”. Essentially, data owners, such as photographers, have become distrustful toward the AI industry and are making things difficult.
OpenAI, which operates DALL-E and ChatGPT, says it respects robots.txt. So do major search engines and Anthropic. However, other players have been accused of ignoring them.