Community Discussion

What type of Robots.txt file allow all bot to visit my site, my blog and make index it?

October 31, 2012
citadelsystems Author Level 1

Write the reason you're deleting this FAQ

Hello everyone...
I want to know that what extension helps to allow all bot to index our sites and crawl them.
thanks
2

Comments 2

November 1, 2012
rannt
Put this into your robot.txt file:

User-agent: *
Disallow:

Thank you
Gašper

Enter a reason for deleting this comment

December 30, 2018
Corzhens
From what I understand with the robot crawlers of search engines, they take on regular sites with no problem. Regular site means you have not robot.tx in your site because that file is an instruction to the robot crawlers on what to do. If there is no robot.txt file then that means there is no instruction so the robot crawlers will do the default which is to crawl on your web pages.

Enter a reason for deleting this comment

Please login or sign up to leave a commentJoin