Yahoo! Search: New robots.txt header tags

minstrel Illustrious Member

Messages:: 15,082

Likes Received:: 1,243

Best Answers:: 0

Trophy Points:: 480

#1

Yahoo! Search Support for X-Robots-Tag Directive to Simplify Webmaster's Control
Wed, Dec 5 2007
by Sharad Verma, Yahoo! Search

Today we're announcing support for tags that give webmasters even more flexibility over which pages and documents are crawled and indexed by Yahoo! Search. Specifically, we're extending our support of page level exclusion tags -- NOINDEX, NOARCHIVE, NOSNIPPET, NOFOLLOW -- to provide additional control for archiving and summarization of ANY file type. Previously, these page level tags could only be expressed within html pages through the META directive (for e.g. <META NAME="Slurp" CONTENT="NOARCHIVE">), but based on feedback from our webmasters, Yahoo! now enables these tags to be expressed through X-Robots-Tag directive in the http header, giving webmasters the flexibility to achieve exclusions on PDF, Word documents, PowerPoint, video, and other file types, including html files, and increasing their coverage through a simplified process. Additionally, webmasters no longer need access to html templates in order to express exclusions for html files. To take advantage of this feature, simply add the following page level tags to the X-Robots-Tag directive in the HTTP Header. Here are a few examples:

X-Robots-Tag: NOINDEX -- If you don't want to show the URL in the Yahoo! Search results. Note: We'll still need to crawl the page to see and apply the tag, so if you don't wish to have the page crawled, use robots disallow on robots.txt.

X-Robots-Tag: NOARCHIVE -- If you don't want to display cache link in the search results page.

X-Robots-Tag: NOSNIPPET -- If you don't want to display summary in the search results page.

X-Robots-Tag: NOFOLLOW -- If you don't want Yahoo! to crawl links in the page.

Click to expand...

minstrel, Dec 6, 2007 IP

ForgottenCreature Notable Member

Messages:: 7,448

Likes Received:: 169

Best Answers:: 0

Trophy Points:: 260

#2

Wow, that's a good move for yahoo.

ForgottenCreature, Dec 6, 2007 IP

Boulder Well-Known Member

Messages:: 806

Likes Received:: 46

Best Answers:: 0

Trophy Points:: 118

#3

Wow! thanks for the news minstrel.

So how would we use those tags in a page? Just simply put them in the header as they are shown, above the <html> tag or do they need to be within the doc type tag? <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0">

It also mentions this on that link minstrel provided:

ysearchblog.com said:

Along with this change, we'll be rolling out additional changes to our crawling, indexing and ranking algorithms over the next few days.
Click to expand...

So this in effect is kind of like Yahoo's PR update maybe.. LOL

Boulder

Boulder, Dec 6, 2007 IP

Log in or Sign up

Yahoo! Search: New robots.txt header tags

minstrel Illustrious Member

ForgottenCreature Notable Member

Boulder Well-Known Member

Useful Searches