Register Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
  #1  
Old 05 Apr 2018, 10:36
kai_sey kai_sey is offline
 
Join Date: Nov 2007
How to block crawl bots without blocking users?

I need to block google & co from searching particular forums on my site, whilst still allowing access to guests & members.

I know this can be done via the robots.txt file - I'm just wondering what to put there.

Lets say the particular forum I want to block is: www.domain.com/forums/audi-versus-bmw/ (Note: I need to disallow all threads/posts in that particular forum).

So, is the following correct?

User-agent: *
Disallow: /forums/audi-versus-bmw/

Would the above disallow also include blocking of the actual threads & posts in that forum? Or would it only block the actual url www.domain.com/forums/audi-versus-bmw?

Thanks.
Reply With Quote
  #2  
Old 05 Apr 2018, 16:49
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Originally Posted by kai_sey View Post

Would the above disallow also include blocking of the actual threads & posts in that forum?.
Robots.txt doesn't disallow or block anything. Think of it as a sign on the door saying "please don't look in the drawers in the rooms." The search engine spiders don't get blocked, and don't have to comply in any way. It's just a way to ask the nice ones "please don't index this."

If you want human guests to see something while not logged in, then google and every other spider can see it as well.
Reply With Quote
  #3  
Old 05 Apr 2018, 17:10
kai_sey kai_sey is offline
 
Join Date: Nov 2007
Originally Posted by Max Taxable View Post
Robots.txt doesn't disallow or block anything. Think of it as a sign on the door saying "please don't look in the drawers in the rooms." The search engine spiders don't get blocked, and don't have to comply in any way. It's just a way to ask the nice ones "please don't index this."

If you want human guests to see something while not logged in, then google and every other spider can see it as well.
Alright thanks.

Then is it possible to do this via .htaccess?
Reply With Quote
  #4  
Old 05 Apr 2018, 21:50
snakes1100 snakes1100 is offline
 
Join Date: Dec 2001
Real name: Anthony
Yes, you can use this list for example.

http://tab-studio.com/en/blocking-robots-on-your-page/
Reply With Quote
Reply

Similar Threads
Thread Thread Starter Forum Replies Last Post
Member Bots 1.0 - Allow bots to act as members trilljester vBulletin 3.5 Add-ons 220 13 Jul 2013 01:00
Member Bots 1.01 - Bots are treated as registered members trilljester vBulletin 3.0 Full Releases 9 09 Nov 2008 01:23



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 20:21.

Layout Options | Width: Wide Color: