TECH6.0

David · #1 04-08-2008, 02:34 AM

This robots.txt will help prevent duplicate content for your vBulletin robots.txt. It basically tells spiders not to crawl all the crap and extra junk that vBulletin produces and will give you better indexed pages to serve on the SERP's

Code:

User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/customprofilepics/
Disallow: /forum/images/
Disallow: /forum/modcp/
Disallow: /forum/ajax.php
Disallow: /forum/album.php
Disallow: /forum/attachment.php

Remember the robots.txt has to go into your web root (not the forums folder) and it must point to the files/folders you don't want crawled. (change the /forum/ part to whatever your location is of these files)seo vb

webwizzy · #2 04-08-2008, 02:37 AM

thanks for this.

I hope it does not have any side effects !

David · #3 04-08-2008, 02:40 AM

wow.. thanks for this, rep+

I hope it does not have any side effects !

As Said by webwizzy

Only positive ones!

And its been reported that it helps lost hair start to regrow*

*highly unconfirmed at this point

justin · #4 04-08-2008, 02:57 AM

So my forum is just in public public_html do I just put this in there? And thanks for this. I had been wondering how you stopped these bots crawling the wrong things. Rep added. Thank you.

David · #5 04-08-2008, 03:02 AM

You just put it in your root ex: http://ctforums.org/robots.txt

Then remove the /forum/ from all the paths ex:

Disallow: showthread.php
Disallow: tags.php

and so on.

justin · #6 04-08-2008, 03:04 AM

Excellent thank you David. Much appreciated.

webwizzy · #7 09-01-2009, 08:51 AM

Here's an upgraded robots.txt for vBulletin 3.8.x, assuming you have your forums installed at site root. Also note a sitemap referance at the end.

Note: We should never give out your admincp directory in robots.txt for security purposes.

Code:

User-agent: *
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /signaturepics/
Disallow: /customgroupicons/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /archive/
Disallow: /ajax.php
Disallow: /announcement.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /converse.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /global.php
Disallow: /group_inlinemod.php
Disallow: /groupsubscription.php
Disallow: /infraction.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member_inlinemod.php
Disallow: /misc.php
Disallow: /moderation.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /picture_inlinemod.php
Disallow: /posthistory.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

Sitemap: http://site.com/sitemap_index.xml.gz

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

TECH6.0

vBulletin robots.txt to stop duplicate content