Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
Ban Spiders by User Agent Details »
Ban Spiders by User Agent
Mod Version: 3.1.2, by Simon Lloyd (Coder) Simon Lloyd is offline
Developer Last Online: Jan 2020 I like it Show Printable Version Email this Page

vB Version: 4.x.x Rating: (64 votes - 4.86 average) Installs: 485
Released: 09 Aug 2011 Last Update: 18 Dec 2014 Downloads: 1924
Supported Uses Plugins  

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
http://www.vbulletin.org/forum/showp...&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
http://www.vbulletin.org/forum/showp...&postcount=137

How does it work?
http://www.vbulletin.org/forum/showp...&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
http://www.vbulletin.org/forum/showp...&postcount=318
http://www.vbulletin.org/forum/showp...7&postcount=51

Where's output.txt located?
http://www.vbulletin.org/forum/showp...&postcount=216

Bad bot lists
http://www.vbulletin.org/forum/showp...&postcount=259
http://www.vbulletin.org/forum/showp...&postcount=224
http://www.vbulletin.org/forum/showp...&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

____________________________________________________________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers

ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity
  1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
  2. Added clickable link to visited thread
22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

Download Now

Only licensed members can download files, Click Here for more information.

Supporters / CoAuthors

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • If you like this modification support the author by donating.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
Similar Mod
Mod Developer Type Replies Last Post
Miscellaneous Hacks Ban Spiders by User Agent Simon Lloyd vBulletin 3.8 Add-ons 188 20 Jul 2015 13:34

  #751  
Old 19 Dec 2014, 11:53
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
Originally Posted by ozzy47 View Post
Yes, I know Simon, I thought about doing it that way also, but the payoff, is not worth the trouble. It would bloat the mod, and become a resource hog in no time.
Thats what i'm saying, if all they want to do with their forums is watch bots/spiders (and thats not what the mod is about) then they'll have hours and hours of fun but no members as the forum would slow down to a snails pace.

People, just rest assured that the mod is doing it's job and giving you back valuable resources please dont waste them again by trying to build in fancyful functionality thats really a whim and not a must have
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #752  
Old 19 Dec 2014, 11:56
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Aug 2009
Real name: Chris
Exactly, which is why I only made it a count of total blocked, it's quick to write, and quick to read and display the count.

I used the analogy, KISS, keep it simple stupid.
__________________
You can get access to my 180 mods for vB 3.6 - 4.x at The Admin Zone as well as the professional support you are used to. New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders! ​ OzzModz down. Site has had a data breach, checking how the intrusion happened. Change your PW if you use the same one on my site and others.
Reply With Quote
  #753  
Old 19 Dec 2014, 11:58
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
Originally Posted by ozzy47 View Post
Exactly, which is why I only made it a count of total blocked, it's quick to write, and quick to read and display the count.

I used the analogy, KISS, keep it simple stupid.
who you calling stupid????? Lol!
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #754  
Old 19 Dec 2014, 12:01
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Aug 2009
Real name: Chris
I call myself stupid all the time, when I am writing code.
__________________
You can get access to my 180 mods for vB 3.6 - 4.x at The Admin Zone as well as the professional support you are used to. New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders! ​ OzzModz down. Site has had a data breach, checking how the intrusion happened. Change your PW if you use the same one on my site and others.
Reply With Quote
  #755  
Old 19 Dec 2014, 12:25
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
Originally Posted by ozzy47 View Post
I call myself stupid all the time, when I am writing code.
Haha!, i hear ya!
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #756  
Old 19 Dec 2014, 13:35
Black Snow Black Snow is offline
 
Join Date: Jul 2012
Originally Posted by ozzy47 View Post
Say what? This is already recording the blocked bots. In four different possible ways, in a log file, in a thread, in a email and the bots blocked count.

Now writing each bot individually to the DB, is not a good idea, as the table would grow too big, and be a giant PITA when you query it to get the info. Think about it, in just five days, only blocking baidu, I have a count of 9062.

Now lets take that and round it to 10,000 for seven days. Lets say I am going to block the 281 bots I have in my list. 281 x 10,000 x 52 = 146,120,000 entries a year.

Sure each bot is not going to hit me 10,000 times a week, but you can see, this can really grow fast.

And my site is not that busy, now imaginge on a busy site, the entries to the table in the DB could be 250,000,000 plus a year. Now run a query to retrieve the info from that table, and it would hurt the site every time it is ran.

IMO, the best way to go is how it is now, just increase the count every time a bot is blocked, and get that number and display it.
I understand where you are coming from but that is not my intention completely. I will log, for example, Baidu into the database and each time it is blocked, the count will go up by one. I just want to be able to query the DB to see a list of all blocked bots and how many times it was blocked. The log, output.txt, is messy IMHO and I want to be able to query the bots list.
Reply With Quote
  #757  
Old 19 Dec 2014, 13:43
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Aug 2009
Real name: Chris
I can see that, but as described, that is still going to be something that is totally un necessary, and still slow things down, on a busy site.

But it is cool in theory, I started to do just that, but decided against it as I was going along, and found the best thing was to have a running count total. People don't care how many times not X was blocked, just as long as it was blocked.
__________________
You can get access to my 180 mods for vB 3.6 - 4.x at The Admin Zone as well as the professional support you are used to. New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders! ​ OzzModz down. Site has had a data breach, checking how the intrusion happened. Change your PW if you use the same one on my site and others.
Reply With Quote
  #758  
Old 19 Dec 2014, 22:52
too_cool_3's Avatar
too_cool_3 too_cool_3 is offline
 
Join Date: Sep 2007
Real name: Marc
I have one question regarding Ban Spiders by User Agent. Since many of the User Agent's in the lists contain words like 'bot', 'spider' and 'crawl', will enabling the option Ban Spiders in List ban even good spiders from the forum???

e.g.
Google Spider (crawl-66-249-75-154.googlebot.com)
BingBot Spider (msnbot-157-55-39-186.search.msn.com)
__________________
TBMOTORING - Thunder Bay Automotive Enthusiast Club
http://tbmotoring.com
Reply With Quote
  #759  
Old 19 Dec 2014, 22:53
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Aug 2009
Real name: Chris
It will only ban spiders you have listed, not all spiders.
__________________
You can get access to my 180 mods for vB 3.6 - 4.x at The Admin Zone as well as the professional support you are used to. New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders! ​ OzzModz down. Site has had a data breach, checking how the intrusion happened. Change your PW if you use the same one on my site and others.
Reply With Quote
  #760  
Old 20 Dec 2014, 07:39
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
Originally Posted by too_cool_3 View Post
I have one question regarding Ban Spiders by User Agent. Since many of the User Agent's in the lists contain words like 'bot', 'spider' and 'crawl', will enabling the option Ban Spiders in List ban even good spiders from the forum???

e.g.
Google Spider (crawl-66-249-75-154.googlebot.com)
BingBot Spider (msnbot-157-55-39-186.search.msn.com)
I've explained this many many times throughout all the Ban Spiders By UserAgent threads, take a look here for the latest explanation http://www.vbulletin.org/forum/showp...&postcount=652
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #761  
Old 05 Jan 2015, 03:00
mradlin's Avatar
mradlin mradlin is offline
 
Join Date: Nov 2008
Installed! Works great!
Reply With Quote
  #762  
Old 05 Jan 2015, 11:57
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Aug 2009
Real name: Chris
Glad to hear.
__________________
You can get access to my 180 mods for vB 3.6 - 4.x at The Admin Zone as well as the professional support you are used to. New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders! ​ OzzModz down. Site has had a data breach, checking how the intrusion happened. Change your PW if you use the same one on my site and others.
Reply With Quote
  #763  
Old 06 Jan 2015, 19:29
ikorolis's Avatar
ikorolis ikorolis is offline
 
Join Date: Sep 2006
Real name: Ilias
Don't say nothing for me is the best add on (plugin) for VB

Installed => Updated => Works fine VB 4.2.2 PL3 and finally stop bad bots/spiders.
__________________
The Internet World
http://www.theinternetworld.org
Reply With Quote
  #764  
Old 06 Jan 2015, 19:58
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
Glad you like it and its doing its job
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #765  
Old 08 Jan 2015, 19:07
too_cool_3's Avatar
too_cool_3 too_cool_3 is offline
 
Join Date: Sep 2007
Real name: Marc
I have been using this mod in combination with 'Spambot Stopper - Prevent Spambots from Registering' and 'Hostname or Useragent Registration Ban' for the last 3 weeks and have had only one spammer get through and register on my forum. Previously, 100's were getting through.

I have attached my 'Ultimate Spider List' to be used with this mod. It's a combination of the 3 lists that are linked in the 1st post of this thread. I imported them into excel, removed the duplicates and sorted alphabetically. Simon Lloyd, feel free to add a link to the file in your first post.
Attached Files
File Type: txt too_cool_3's Ultimate Spider List.txt (3.5 KB, 17 views)
__________________
TBMOTORING - Thunder Bay Automotive Enthusiast Club
http://tbmotoring.com

Last edited by too_cool_3; 12 Jan 2015 at 18:52. Reason: updated txt file
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 22:36.

Layout Options | Width: Wide Color: