Yahoo! Search: New robots.txt header tags

minstrel · #1 Dec 6th 2007, 4:41 am

Yahoo! Search Support for X-Robots-Tag Directive to Simplify Webmaster's Control
Wed, Dec 5 2007
by Sharad Verma, Yahoo! Search

Quote:

Today we're announcing support for tags that give webmasters even more flexibility over which pages and documents are crawled and indexed by Yahoo! Search. Specifically, we're extending our support of page level exclusion tags -- NOINDEX, NOARCHIVE, NOSNIPPET, NOFOLLOW -- to provide additional control for archiving and summarization of ANY file type. Previously, these page level tags could only be expressed within html pages through the META directive (for e.g. <META NAME="Slurp" CONTENT="NOARCHIVE">), but based on feedback from our webmasters, Yahoo! now enables these tags to be expressed through X-Robots-Tag directive in the http header, giving webmasters the flexibility to achieve exclusions on PDF, Word documents, PowerPoint, video, and other file types, including html files, and increasing their coverage through a simplified process. Additionally, webmasters no longer need access to html templates in order to express exclusions for html files. To take advantage of this feature, simply add the following page level tags to the X-Robots-Tag directive in the HTTP Header. Here are a few examples:

X-Robots-Tag: NOINDEX -- If you don't want to show the URL in the Yahoo! Search results. Note: We'll still need to crawl the page to see and apply the tag, so if you don't wish to have the page crawled, use robots disallow on robots.txt.
X-Robots-Tag: NOARCHIVE -- If you don't want to display cache link in the search results page.
X-Robots-Tag: NOSNIPPET -- If you don't want to display summary in the search results page.
X-Robots-Tag: NOFOLLOW -- If you don't want Yahoo! to crawl links in the page.

ForgottenCreature · #2 Dec 6th 2007, 4:43 am

Wow, that's a good move for yahoo.

Boulder · #3 Dec 6th 2007, 5:03 am

Wow! thanks for the news minstrel.

So how would we use those tags in a page? Just simply put them in the header as they are shown, above the <html> tag or do they need to be within the doc type tag? <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0">

It also mentions this on that link minstrel provided:

Quote:

Originally Posted by ysearchblog.com

Along with this change, we'll be rolling out additional changes to our crawling, indexing and ranking algorithms over the next few days.

So this in effect is kind of like Yahoo's PR update maybe.. LOL

Boulder

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Yahoo & robots.txt	glorybeme	Yahoo	3	Apr 28th 2007 12:39 pm
Does Yahoo Slurp really support wildcards in robots.txt?	kmike	robots.txt	4	Mar 10th 2007 1:51 pm
Yahoo - Supporting wildcards in robots.txt	dhruv37	Yahoo	1	Nov 14th 2006 9:45 pm
Yahoo Ignoring robots.txt	Notorious	robots.txt	3	Aug 22nd 2006 7:58 am
Yahoo ignoring robots.txt	jeff123	Yahoo	4	Aug 18th 2005 8:50 pm