Register Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
  #1  
Old 09 Jul 2014, 09:30
dacho's Avatar
dacho dacho is offline
 
Join Date: Dec 2004
Real name: David
How can I stop Unknown robot, spider, crawler

Hello mate

How can we stop the Unknown robot, spider, crawler ?
They probably skipped and not refer to robot.txt
In robot.txt I made some bots are enabled and blocking all others bots, and still without permission they scan me
Reply With Quote
  #2  
Old 09 Jul 2014, 09:36
Dave Dave is offline
 
Join Date: Jun 2010
Real name: Dave
I believe it usually takes a few days up to a week before those robots "look" at the robots.txt file and update it on their side.
By the way you named it robots.txt right? With the s after robot?
Reply With Quote
  #3  
Old 09 Jul 2014, 15:08
dacho's Avatar
dacho dacho is offline
 
Join Date: Dec 2004
Real name: David
Yes my friends, it is clear that file name robots.txt and sits on Root.
It's not about takes a few days up to a week before those robots "look" at the robots.txt file and update it on their side

It probably really are a lot of bots, spiders and crawlers pass through the robots.txt and go inside without identified

The Solution probably need to combine the robots.txt and .htaccess

See the following guide :
http://michael.langley.id.au/blog/posts/28
Reply With Quote
  #4  
Old 09 Jul 2014, 15:14
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Robots.txt isn't a law. Bots aren't required to follow it or even index it. Only the legitimate SE bots will. Bad bots simply ignore it. It's just a sign basically, politely saying "please don't crawl these areas of my site." It's alot like gun laws in the US - criminals simply ignore them.

For the others, you want the "Ban Spiders by User Agent" Mod.
Reply With Quote
  #5  
Old 09 Jul 2014, 15:15
ForceHSS's Avatar
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
http://www.vbulletin.org/forum/showthread.php?t=268208
This will stop them as soon as you install it
Reply With Quote
  #6  
Old 09 Jul 2014, 23:17
Lee G Lee G is offline
 
Join Date: Jun 2006
Real name: Lee
This mod by Paul M will help you track them
http://www.vbulletin.org/forum/showthread.php?t=232182

There used to be an updated spider list for vbulletin on the net. No idea what or if that's still shared

You can then track the heavy hitters and block them
Reply With Quote
  #7  
Old 09 Jul 2014, 23:28
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Originally Posted by Lee G View Post
There used to be an updated spider list for vbulletin on the net. No idea what or if that's still shared[/B]
It is:

Updated vBulletin spiders list


But that is NOT the block list for the ban spiders mod, there's never been a "official" list for that as the Mod is blank when first installed..
Reply With Quote
  #8  
Old 09 Jul 2014, 23:57
ForceHSS's Avatar
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
When was the last time that list was updated Max
Reply With Quote
  #9  
Old 10 Jul 2014, 00:00
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Jul 2009
Real name: Chris
It was updated in January of this year. It had over 200 spiders added.
__________________
You can get access to my 180 mods for vB 3.6 - 4.x at The Admin Zone as well as the professional support you are used to. New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders! ​ OzzModz down. Site has had a data breach, checking how the intrusion happened. Change your PW if you use the same one on my site and others.
Reply With Quote
  #10  
Old 10 Jul 2014, 00:03
ForceHSS's Avatar
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
So the last few in the last few posts have not been added? Can they be added
Reply With Quote
  #11  
Old 10 Jul 2014, 00:10
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Jul 2009
Real name: Chris
They will once I get a few more, actually this weekend I can push out a update, I think I have about 30 more added since the last release.
__________________
You can get access to my 180 mods for vB 3.6 - 4.x at The Admin Zone as well as the professional support you are used to. New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders! ​ OzzModz down. Site has had a data breach, checking how the intrusion happened. Change your PW if you use the same one on my site and others.
Reply With Quote
  #12  
Old 10 Jul 2014, 00:14
ForceHSS's Avatar
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
Good can you tell me when its updated so I can download the new one
Reply With Quote
  #13  
Old 10 Jul 2014, 00:18
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Jul 2009
Real name: Chris
If you subscribe to the thread on my site, you will know when I push the update. Then you can unsubscribe after that.
__________________
You can get access to my 180 mods for vB 3.6 - 4.x at The Admin Zone as well as the professional support you are used to. New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders! ​ OzzModz down. Site has had a data breach, checking how the intrusion happened. Change your PW if you use the same one on my site and others.
Reply With Quote
  #14  
Old 10 Jul 2014, 00:20
ForceHSS's Avatar
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
Will do
Reply With Quote
  #15  
Old 10 Jul 2014, 01:20
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Originally Posted by ozzy47 View Post
If you subscribe to the thread on my site, you will know when I push the update. Then you can unsubscribe after that.
Actually it would be helpful Force, to stay subscribed and help us add new spiders by posting them in the thread.
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 09:23.

Layout Options | Width: Wide Color: