I'm starting up a new site that will be updated with new pages and content often, for SEO purposes is it worth taking out the robots.txt file that negates crawlers searching my /articles/ folder as it will be updated with 2 articles a week, growing to 10 articles every month each one will have a standard navigational link back to my homepage. What do you think? Would I benefit from removing the robots.txt file?
I dont have a robots.txt and my site is indexed great I think you only really need it if there are pages you dont want in google.
The Robots.txt file is used to stop bots from indexing your site. I banned one bot from my one of my sites called ia_archive. This bot snoops around and takes pics of your site and then archives it. I believe this is an invasion of privacy and also copyright infringement. It's more of a constitutional thing with me. If you don't have a robots.txt file - nothing will hurt it. It's to keep out the bots you don't want on your site.
You're better off using a sitemap and telling the crawlers which pages to index, rather than using a robots.txt to exclude them.
Or if you focus on Google, just put a nofollow attribute on these links. they will be indexed but will not transmit juice to those pages. No real need for a robots.txt here