vBulletin Mods

The Official vBulletin Modifications Site

by Irishnotsane
18 Apr 2012 12:06

How i prevent bots from crawling on my website/forum?

kh99 18 Apr 2012 12:19

There's a bunch of mods having to do with spiders, just search for spider or spiders. There's this one for instance: www.vbulletin.org/forum/showthread.php?t=268208 which I think allows you to ban only the "bad" ones.

Irishnotsane 18 Apr 2012 12:34

Thanks, but is there a way to prevent them fullstop, as opposed to banning only the bad ones?

kh99 18 Apr 2012 12:44

I think you can create a file called robots.txt and upload it. Here's a link to a site about that: http://www.robotstxt.org/robotstxt.html . To be honest it's been so long since I've dealt with that stuff myself that I don't know how well it works. But like it says on that site, it depends on bots voluntarily obeying your rules. If they choose not to, then I don't see that you have any choice other than to try to ban them by user agent, or maybe ip (although I'm guessing that doesn't work all that well).

Irishnotsane 18 Apr 2012 12:48

Ah right, so by adding


User-agent: *
Disallow: /
In my robots.txt file in my ROOT directory, it'll disallow all bots (unless of course they choose to ignore it)

So by having this, plus that product, it should get rid of 90% (or so) of bots?

kh99 18 Apr 2012 12:52


Originally Posted by Irishnotsane (Post 2321246)
So by having this, plus that product, it should get rid of 90% (or so) of bots?

I'm not sure about that, but I'm pretty sure lots of other people around here will be able to comment....

Videx 21 Apr 2012 00:33

These days what you're talking about are generally called "spiders", with 'bots' being spam bots. You rarely want to ban spiders - they are your friends.

Spiders Display on Forumhome and Who's Online for vb4

Irishnotsane 21 Apr 2012 01:56

What do these "Spiders" even do?

I don't really want them on my forum regardless of if they're my friends or not, I want it to be just me and my members so I'd rather stay off Google and other search engines.. I own a small community based forum where users are invited in, so pointless being on Google anyway.

ForceHSS 21 Apr 2012 02:01

the link in post 2 is the best to stop bots the bad ones that is

All times are GMT. The time now is 23:19.

Powered by vBulletin® Version 3.8.14
Copyright © 2020, MH Sub I, LLC dba vBulletin. All Rights Reserved. vBulletin® is a registered trademark of MH Sub I, LLC
Copyright ©2001 - , vbulletin.org. All rights reserved.