| |||||||
This is a discussion on Do you keep robots.txt file? within the Search Engine Optimisation section, part of the Web Design/Development and SEO category; I have never kept robots.txt file in main directory of my website. Is it necessary to keep robots.txt file. Whats ...
 |
|
| | LinkBack | Thread Tools | Display Modes |
|
#2
19-02-2009, 05:27 PM
| ||||
| ||||
Here's a related thread: http://tech6.com/f51/vbulletin-robot...-content-t203/ Yes, I consider it important, it tells the search engine spiders which areas of a site to crawl and which not. You can also referance your sitemap file in robots.txt. It always resides in root of a site so you can view any site's robots.txt (ofcourse those who have) by visiting http://site.com/robots.txt Check out google's lol http://www.google.com/robots.txt
__________________ Would you like to Link To Us | Support TECH6 by going Premium Know more about me at Vinayaks.com | Follow TECH6 at Twitter   |
|
#3
11-10-2010, 05:09 PM
| ||||
| ||||
|
Robots.txt is a simple text file that tells the bots like googlebot essentially where to go or not. If you connect to your site via FTP or our file manager, you should see a file named robots.txt in your main directory. You can simply download this file and open with notepad or another text editor and make changes.
|
|
#4
13-01-2011, 12:24 AM
| ||||
| ||||
|
I always keet robots.txt file in root directory of my website........
|
|
#5
09-02-2011, 11:31 AM
| ||||
| ||||
|
If you connect to your site via FTP or our file manager, you should see a file named robots.txt in your main directory. You can simply download this file and open with notepad or another text editor and make changes.
|
|
#6
21-04-2011, 03:45 PM
| ||||
| ||||
|
no its not compulsory but i use it to block some spam IP's
__________________ Web Design Delhi |
|
#7
26-04-2011, 02:16 PM
| ||||
| ||||
| RE:
If you want to disallow some page to crawl by search engine spider then it will help you lot.
|
|
#8
03-11-2011, 02:17 PM
| ||||
| ||||
|
Well I use robots.txt, even a basic one, to stop those annoying logs from appearing. Seems to be that the spiders will consistently look for the robots.txt file first and if they don't, an absurdity ensues. And I have a lot of spiders sniffing at my sites.
|
|
#9
10-11-2011, 03:22 PM
| ||||
| ||||
| Index Google site Index Help!? |
|
#10
26-11-2011, 06:58 PM
| ||||
| ||||
|
Its not necessary that you use robots.txt. It is use for tell to Google that which pages you want to Crawl and which are not. Some times you use irrelevant page s in yous website, which are harmful for your website. then you can tell to Google with the help of robots.txt that you don't want to crawl them. So I think you should use the robots.txt. |
|
#11
24-02-2012, 02:25 PM
| ||||
| ||||
| Re:
Yes, I have added robots in my website, it will helpful to quick crawling and you can also hide pages from index by using robots.txt.
__________________ Dedicatednow |
|
| Bookmarks |
| Thread Tools | |
| Display Modes | |
| |
Linear Mode
