Hi Everyone, I have a question regarding a problem with e-commerce URL. When a user enters a search query on the e-commerce website a custom URL is generated, this URL is been crawled by Google and shows in Google Webmaster tool as 'URL Not Found' or 'Unreachable'. How can we prevent this URL's generated by search to reflect in webmaster tool as 'Not Found'. I can't keep on removing it every time and forever. I want a permanent solution that can prevent these URL from Google. Looking forward for your answers!
you can use robots.txt for that.with robots you can actually tell search engines which pages are to be indexed or not to be crawled.
I haven't blocked any URL so far. The URL extension for search is this ('www.myblog.com/search?updated-max=2009-08-04T00%3A03%3A00-07%3A00&max-results=5'). Now seeing the URL extension what URL should I command to be blocked in search engine using Robots.txt?