Hi, Why google is not indexing all of my pages. Only 16 pages are indexed from my site www.sportscarsfans.com and my site has more than 250 pages The last time google crawled my home page 19th Oct I submitted sitemap 2 days ago and it was download and its status is "OK" can some one tell me solution? check my source of homepage if something is wrong there or check robots.txt @ moderators: I have posted this topic in sitemap but got no reply so now I am posting on main... Thanks ! kutchi
Can't locate your robots.txt... where is it? Anyway, just put this in your robots.txt: User-agent: * Disallow: Code (markup): That'll allow all spiders to crawl all your pages.
they check to see if you have a robots.txt.if it is blank or not there, they just go right in. other wise you wouldn't have 16 pages indexed. when a bot visits,it only scans a certain amount of pages.a sitemap just tells the bot all the pages you have. put this in the robots.txt.it tells other bots that you have a sitemap. Sitemap: http://www.domain.com/sitemap.xml PHP: