Hello Everyone, I have auto-generated robots.txt. But i am facing much issues like 1. in google search results shows: "A description for this result is not available because of this site's robots.txt – learn more." 2. Many other important Urls not indexed by google. Can anyone help me to find the issues? Is this robots file ok or not? My Robots.txt shown as below: # Google Image Crawler Setup User-agent: Googlebot-Image Disallow: # Crawlers Setup User-agent: * # Directories Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /downloader/ Disallow: /errors/ Disallow: /includes/ #Disallow: /js/ #Disallow: /lib/ Disallow: /magento/ #Disallow: /media/ Disallow: /pkginfo/ Disallow: /report/ Disallow: /scripts/ Disallow: /shell/ Disallow: /skin/ Disallow: /stats/ Disallow: /var/ # Paths (clean URLs) Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Disallow: /catalogsearch/ #Disallow: /checkout/ Disallow: /control/ Disallow: /customize/ Disallow: /newsletter/ Disallow: /poll/ Disallow: /review/ Disallow: /sendfriend/ Disallow: /tag/ Disallow: /catalog/product/gallery/ Disallow: /faqs/ Disallow: /checkout/ Disallow: /disclaimer-policy/ Disallow: /return-policy/ # Files Disallow: /cron.php Disallow: /cron.sh Disallow: /error_log Disallow: /install.php Disallow: /LICENSE.html Disallow: /LICENSE.txt Disallow: /LICENSE_AFL.txt Disallow: /STATUS.txt # Paths (no clean URLs) #Disallow: /*.js$ #Disallow: /*.css$ Disallow: /*.php$ Disallow: /*?SID=
Not to worry about. You have Disallow the link, that’s why it is showing in SERP. You can check the whole discussion here https://productforums.google.com/fo...asters/crawling-indexing--ranking/0BiNHmODzeg