Register Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
  #1  
Old 11 Feb 2010, 01:34
Glenn_E Glenn_E is offline
 
Join Date: Dec 2009
How do I limit the # of google and yahoo crawler bots?

I'm suddenly getting 10-15 google / yahoo bots at a time. How do I limit the number of bot connections?
Reply With Quote
  #2  
Old 14 Feb 2010, 02:30
Andy Andy is offline
 
Join Date: Oct 2003
Normally you only see one or two Google spiders at a time. However with Yahoo you can see dozens or even hundreds of spiders. Here is an example of a robots.txt which only allows Google and Bing to index your forum.

User-agent: Googlebot
Disallow:

User-agent: MSNBot
Disallow:

User-agent: *
Disallow: /
Reply With Quote
  #3  
Old 15 Feb 2010, 05:29
gulfcoasttundra's Avatar
gulfcoasttundra gulfcoasttundra is offline
 
Join Date: Sep 2009
Real name: Ben
in reverse of this "issue/Problem" how do you make your site more visible to these bots/spiders to improve searchability?


Thanks
Ben
Reply With Quote
  #4  
Old 15 Feb 2010, 06:15
final kaoss final kaoss is offline
 
Join Date: Apr 2006
glenn, go to google.com/webmasters and then after you've configured your site for it, click your site and then

*Site configuration
*Settings
*Crawl Rate, you can set the crawl rate to either be faster or slower
Reply With Quote
  #5  
Old 21 Feb 2010, 04:41
Glenn_E Glenn_E is offline
 
Join Date: Dec 2009
Thanks guys... I'm still bringing my site up to speed and am not ready for the swarm of crawlers. Nice to know that my site is hitting the top of the search list on the keywords I use
Reply With Quote
  #6  
Old 22 Feb 2010, 04:22
Videx's Avatar
Videx Videx is offline
 
Join Date: Feb 2007
I wouldn't completely disallow a spider. After all, you want them to index things. I used this instead:
User-agent: Slurp
Crawl-delay: 1500
Slurp was the only one I saw with an inordinate number of connections (in the hundreds).
Reply With Quote
  #7  
Old 23 Feb 2010, 12:12
toibs toibs is offline
 
Join Date: Dec 2001
Real name: Paul
Is there any way to disallow google from indexing one specific forum using robots.txt, rather than just enable/disabling it??

Thx
Paul
Reply With Quote
  #8  
Old 23 Feb 2010, 16:21
Videx's Avatar
Videx Videx is offline
 
Join Date: Feb 2007
Originally Posted by toibs View Post
Is there any way to disallow google from indexing one specific forum
I've always understood that the bots all browse the forum with Guest permissions (I hope!), so you can just disallow Guests from seeing that forum.
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 15:04.

Layout Options | Width: Wide Color: