You require a robots.txt file only if you don’t want searchbots to crawl and therefore index certain pages. i.e you want to block particular pages of your website from coming in search results of search engines. If you want your entire site to be indexed, you don’t need robots.txt file in the first place.

Robots.txt is simply an ASCII text file placed at the root of your domain.

For example, http://www.domainname.com/robots.txt. Continue Reading