vBulletin Mods

The Official vBulletin Modifications Site
https://www.vbulletin.org/forum/showthread.php?t=324269

Search Engine Friendly Archives and store_sitemap directory
by Snowhog
23 Jan 2017 15:47

How and where does vBulletin create and store Search Engine Friendly Archives?

We have the store_sitemap directory in our robots.txt file as a disallow:

disallow: /store_sitemap/

I'm not sure this should be a disallow. Should the files in this directory be available to spiders/crawlers?

bzcomputers 26 Jan 2017 23:30

Quote:

Originally Posted by Snowhog (Post 2581237)
How and where does vBulletin create and store Search Engine Friendly Archives?

We have the store_sitemap directory in our robots.txt file as a disallow:

disallow: /store_sitemap/

I'm not sure this should be a disallow. Should the files in this directory be available to spiders/crawlers?

You should "allow: /store-sitemap/"; without allowing this directory to be scanned/indexed will stop Google and every other search engine from being able to download and see the site as you wish it to be seen.

Sample Page:
Quote:

<url>
<loc>http://www.domain.com /</loc>
<lastmod>2017-01-01</lastmod>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>

Snowhog 27 Jan 2017 00:05

Thank you. I had already made up my mind that it shouldn't be a Disallow and removed it (as an entry) from our robots.txt file.


All times are GMT. The time now is 21:12.

Powered by vBulletin® Version 3.8.14
Copyright © 2020, MH Sub I, LLC dba vBulletin. All Rights Reserved. vBulletin® is a registered trademark of MH Sub I, LLC
Copyright ©2001 - , vbulletin.org. All rights reserved.