Register Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
  #1  
Old 18 Apr 2012, 12:06
Irishnotsane Irishnotsane is online now
 
Join Date: Apr 2012
Bots

How i prevent bots from crawling on my website/forum?
Reply With Quote
  #2  
Old 18 Apr 2012, 12:19
kh99 kh99 is offline
 
Join Date: Aug 2009
Real name: Kevin
There's a bunch of mods having to do with spiders, just search for spider or spiders. There's this one for instance: www.vbulletin.org/forum/showthread.php?t=268208 which I think allows you to ban only the "bad" ones.
Reply With Quote
  #3  
Old 18 Apr 2012, 12:34
Irishnotsane Irishnotsane is online now
 
Join Date: Apr 2012
Thanks, but is there a way to prevent them fullstop, as opposed to banning only the bad ones?
Reply With Quote
  #4  
Old 18 Apr 2012, 12:44
kh99 kh99 is offline
 
Join Date: Aug 2009
Real name: Kevin
I think you can create a file called robots.txt and upload it. Here's a link to a site about that: http://www.robotstxt.org/robotstxt.html . To be honest it's been so long since I've dealt with that stuff myself that I don't know how well it works. But like it says on that site, it depends on bots voluntarily obeying your rules. If they choose not to, then I don't see that you have any choice other than to try to ban them by user agent, or maybe ip (although I'm guessing that doesn't work all that well).
Reply With Quote
  #5  
Old 18 Apr 2012, 12:48
Irishnotsane Irishnotsane is online now
 
Join Date: Apr 2012
Ah right, so by adding

User-agent: *
Disallow: /
In my robots.txt file in my ROOT directory, it'll disallow all bots (unless of course they choose to ignore it)

So by having this, plus that product, it should get rid of 90% (or so) of bots?
Reply With Quote
  #6  
Old 18 Apr 2012, 12:52
kh99 kh99 is offline
 
Join Date: Aug 2009
Real name: Kevin
Originally Posted by Irishnotsane View Post
So by having this, plus that product, it should get rid of 90% (or so) of bots?
I'm not sure about that, but I'm pretty sure lots of other people around here will be able to comment....
Reply With Quote
  #7  
Old 21 Apr 2012, 00:33
Videx's Avatar
Videx Videx is offline
 
Join Date: Feb 2007
These days what you're talking about are generally called "spiders", with 'bots' being spam bots. You rarely want to ban spiders - they are your friends.

Spiders Display on Forumhome and Who's Online for vb4
Reply With Quote
  #8  
Old 21 Apr 2012, 01:56
Irishnotsane Irishnotsane is online now
 
Join Date: Apr 2012
What do these "Spiders" even do?

I don't really want them on my forum regardless of if they're my friends or not, I want it to be just me and my members so I'd rather stay off Google and other search engines.. I own a small community based forum where users are invited in, so pointless being on Google anyway.
Reply With Quote
  #9  
Old 21 Apr 2012, 02:01
ForceHSS's Avatar
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
the link in post 2 is the best to stop bots the bad ones that is
Reply With Quote
Reply

Similar Threads
Thread Thread Starter Forum Replies Last Post
Member Bots 1.0 - Allow bots to act as members trilljester vBulletin 3.5 Add-ons 220 13 Jul 2013 01:00
Member Bots 1.01 - Bots are treated as registered members trilljester vBulletin 3.0 Full Releases 9 09 Nov 2008 01:23



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 21:21.

Layout Options | Width: Wide Color: