Must i have a robot.txt file? Cause i have no idea how to set this up, even after reading tutorials and such...Not all that great at codding things...I just post I usually tend to have a Google Bot on my site just about all the time, give and take some time off here and there...Yahoo Bot comes by once in awhile too...I have seen 2 Google Bots on at 1 time, that was the most so far...Also noticed a technorati bot on there from time to time...And Alex stops by too Is there anyway to get more bots to visit my site? Cause i know theres plenty more out there..
The only use of the robots.txt file is to "prevent" pages or directories from being crawled by web robots\crawlers.
No problem, below is a basic robots.txt code to allow everything to be indexed. Just make a text file in notepad and call it robots.txt and insert the below code, save it and upload it to your top diretory.
ill do that, thanks... Just for reference though...Does it matter if i have that or not...Will any bot still visit my site if theres no robot.txt file at all or is it just something like a + to have it anyway...
It is not required, like I said the robots.txt file main purpose is to tell web robots what content NOT to crawl or index.
When we want Search Engine didn’t come direct on this web page then we insert that web page in our robots.txt file.