Hey guys, I currently own a website which sells online advertising. My robots file is extremely simple: User-agent: * Allow: / Allow: /robots.txt Is there anything I should do to improve it/get google to crawl faster, more efficiently etc? Regards
Adding a sitemap would be the best thing you could do there. Go to http://www.xml-sitemaps.com/ or a similar site to generate a sitemap.xml file, upload the file to the public directory of your web hosting account, and add:- Sitemap: /sitemap.xml to your robots.txt file. Then go to Google's Webmaster tools site and submit a link to the sitemap there.
It'd be nice to see your website to know if you're using robots the best. There's also no need to do Allow: / or Allow: robots.txt. I wrote up a short guide on writing a robots.txt: http://www.webmasterindex.net/seo-how-to-write-a-robots-txt-file/
Hmm. 3 year old domain, not indexed in Google yet? I hope it wasn't banned! First thing, make a Google Webmasters account and do what the person above there said, create and submit a sitemap. You can also submit the URL: http://www.google.com/addurl/. Not sure you've done that already. If after a week your site is still not indexed - I would submit a Reconsideration Request, because your site was most likely banned. https://www.google.com/webmasters/tools/reconsideration
Hey there! No, the domain is little over 3months old, and this is the 4th day that our website has been live! How did you find 3 years? Maybe that was a previous owner who got rid of it?