Community Discussion

A question about robots.txt file

October 22, 2012
Tommy Matalino Author Level 1

Write the reason you're deleting this FAQ

Do we really need to manually create the Robots.txt file?
3

Comments 3

January 19, 2013
Beverly
It's possible, you could use a premade robots.txt file, such as a template used by another site and change any relevant details to what your site needs. That would almost be automatic but would allow you to have input.

Enter a reason for deleting this comment

January 29, 2013
Linksdepth
It's possible, you could use a premade robots.txt file, such as a template used by another site and change any relevant details to what your site needs. That would almost be automatic but would allow you to have input.


Can U suggest me giving some example please........

Enter a reason for deleting this comment

December 28, 2018
Corzhens
I guess what you mean about the robot.txt file is if you need that for the search engines to crawl on your site. From what I understand about the robot.txt file it is a guide for the search engine crawlers for the direction which parts of the web page that it should see and which parts that the crawlers should not see. But without the robot.txt file the crawler will check on everything that is on the web page which is the standard. In other words you don’t need the robot.txt file.

Enter a reason for deleting this comment

Please login or sign up to leave a commentJoin