You know, it doesn't take that much computing power to index every page on the net. Majestic12 are a company doing the same as Google but achieve it through a distributed computing model. Basically you can download a program which will crawl the web and deliver that data back to Majestic. They index over 1 trillion links. That's more than Google. In fact, their software checks a page once every day or two, arguably more than Google does. Here's the kicker. They are doing all that with about 1000 installs of the software. That's 1000 computers running some software in the background. They said in a blog post that just 8000 computers is enough to index the whole known world wide web once per DAY! To give you a comparison, it is believed Google have 900'000 servers running their services. So, how can Google do it? Because it's just not that hard anymore.
I think 1 ask you question if you can handle that many. The web already has truly is the web pages. When there are jillians more web pages in the years ahead, Google still continue to be the best index for all information? It might certain other web sites or services start to have A foothold in certain areas?
Hi Google is the company which is working for data only. So google made crawler or bot or spider who works every second. the main work of the spider is just to search the new thins on the web and then store in database. So whatever change you are doing in your post that will also save when spider will reach over it. So this is the method to save all the links in database of every search engine.
Google keep all the data in his database like we are keeping books in shelf and whenever we need book we just open the shelf and take the book.
Google has made automated softwares with algorithms in it. These algorithms make run the spiders.also help in storing the webpages in the database of google.