Thread: Bots
View Single Post
  #4  
Old 18 Apr 2012, 12:44
kh99 kh99 is offline
 
Join Date: Aug 2009
Real name: Kevin
I think you can create a file called robots.txt and upload it. Here's a link to a site about that: http://www.robotstxt.org/robotstxt.html . To be honest it's been so long since I've dealt with that stuff myself that I don't know how well it works. But like it says on that site, it depends on bots voluntarily obeying your rules. If they choose not to, then I don't see that you have any choice other than to try to ban them by user agent, or maybe ip (although I'm guessing that doesn't work all that well).
Reply With Quote