Google Spider able to Crawl more now!

[usasportstraining]

Google's GoogleBot is now able to crawl forms and they say they can now "scan" Flash and Javascript.

That's big news!!

All those links could be crawlable now that were previously not found due to javascript.

Crawling through HTML forms
Friday, April 11, 2008 at 10:50 AM
Written by Jayant Madhavan and Alon Halevy, Crawling and Indexing Team

Google is constantly trying new ideas to improve our coverage of the web. We already do some pretty smart things like scanning JavaScript and Flash to discover links to new web pages, and today, we would like to talk about another new technology we've started experimenting with recently.

In the past few months we have been exploring some HTML forms to try to discover new web pages and URLs that we otherwise couldn't find and index for users who search on Google. Specifically, when we encounter a <FORM> element on a high-quality site, we might choose to do a small number of queries using the form. For text boxes, our computers automatically choose words from the site that has the form; for select menus, check boxes, and radio buttons on the form, we choose from among the values of the HTML. Having chosen the values for each input, we generate and then try to crawl URLs that correspond to a possible query a user may have made. If we ascertain that the web page resulting from our query is valid, interesting, and includes content not in our index, we may include it in our index much as we would include any other web page.

Needless to say, this experiment follows good Internet citizenry practices. Only a small number of particularly useful sites receive this treatment, and our crawl agent, the ever-friendly Googlebot, always adheres to robots.txt, nofollow, and noindex directives. That means that if a search form is forbidden in robots.txt, we won't crawl any of the URLs that a form would generate. Similarly, we only retrieve GET forms and avoid forms that require any kind of user information. For example, we omit any forms that have a password input or that use terms commonly associated with personal information such as logins, userids, contacts, etc. We are also mindful of the impact we can have on web sites and limit ourselves to a very small number of fetches for a given site.

The web pages we discover in our enhanced crawl do not come at the expense of regular web pages that are already part of the crawl, so this change doesn't reduce PageRank for your other pages. As such it should only increase the exposure of your site in Google. This change also does not affect the crawling, ranking, or selection of other web pages in any significant way.

This experiment is part of Google's broader effort to increase its coverage of the web. In fact, HTML forms have long been thought to be the gateway to large volumes of data beyond the normal scope of search engines. The terms Deep Web, Hidden Web, or Invisible Web have been used collectively to refer to such content that has so far been invisible to search engine users. By crawling using HTML forms (and abiding by robots.txt), we are able to lead search engine users to documents that would otherwise not be easily found in search engines, and provide webmasters and users alike with a better and more comprehensive search experience.

[kodut]

big news if its real , a generation of search will begin

[usasportstraining]

Originally Posted by kodut

big news if its real , a generation of search will begin

It's straight from Google's blog. I'd say it's as real as it gets.

[TechEvangelist]

I guess that explains why I have been seeing search URLs showing up in Google's index for some of my sites. It looked like someone was posting the URLs from the search results pages in the sites, but I could never find the source of the links.

This is a double-edged sword. Search results pages typically are not optimized. I don't see any other type of form that they would care to crawl.

What is the benefit of this? I have not seen any of these pages show up on Google's search pages and I don't think users want to land on internal search results pages. Am I missing something here?

[rainborick]

Google has been scanning JavaScript and Flash files for URLs for 3-4 years now. They look for complete URLs in the form of "http://www.example.com/". They added the ability to extract text from Flash files over 2 years ago. What is truly new is that they are experimenting with following <form>s to see if they resolve to crawlable pages, such as a site's custom search form. Note that the <form> has to use the "GET" method so that there is a URL to put in the index.

[kks_krishna]

good news. They need more links

[usasportstraining]

Originally Posted by rainborick

Google has been scanning JavaScript and Flash files for URLs for 3-4 years now. They look for complete URLs in the form of "http://www.example.com/". They added the ability to extract text from Flash files over 2 years ago. What is truly new is that they are experimenting with following <form>s to see if they resolve to crawlable pages, such as a site's custom search form. Note that the <form> has to use the "GET" method so that there is a URL to put in the index.

That explains a few things about what I've been seeing for reported backlinks.

Are you sure they've been scanning javascript for 3 or 4 years though? I've been hearing that they could not, up until now. It seems to me it's a fairly new development, not as new as reading forms, but new just the same.

[wilhb81]

WoW, this will be a great news for us, especially for those site that full of flash..

[Divisive Cottonwood]

Why is this great news? I don't understand. Can somebody give me an example where this will make a difference to a website.

[Legendary11]

more scanning, page indexing more pages on the net more chance of getting visitors?

[usasportstraining]

Originally Posted by Divisive Cottonwood

Why is this great news? I don't understand. Can somebody give me an example where this will make a difference to a website.

More ways for Google to crawl and gather data means more ways for us to benefit.

If they can find links in javascript and flash, then it gives us more avenues to have backlinks.

There may be some benefits for them crawling forms as well.

[mhmdkhamis]

good news

thanks man

[rupertValentino]

thanks for spreading the news.ciao

[godsofchaos]

Thats huge news! Now widgets like Criteo and blogsphere will get a run for their money....

[Proximity]

great news lets hope it is benefitial

[ancientcity]

wow, this is interesting

[pioneer1]

Google is going to to deep scan web pages. Forms, buttons, check boxes etc.

[elias_sorensen]

Cool news ;-)

[usasportstraining]

Originally Posted by godsofchaos

Thats huge news! Now widgets like Criteo and blogsphere will get a run for their money....

Exactly!

That's my hope.

[arjun316]

great news i hope they crawl my website more and let it be on top of search engine