If get links from another site to yours, and the page that they are linking to is removed and a 404 error comes up instead. In terms of SE spiders, what happens? If your 404 page looks like any other site on your page and just says "page not found" or something - but contains your typical navigation on top or bottom or whatever. Will that stop the bot and make it leave, or will it continue to follow links on that page and go thru your site like it would as if that page had existed?
Will it bounce to domain's home page and continue or vacate entirely? Or how can I tell if a 404 header is being returned?
Interesting. Thanks. Seems that one shouldn't delete pages then, just empty their contents or make something else take its place on that page.
If you don't remove the page and either set up a 301 redirect or custom 404, webmasters who have linked to you may never notice that their link is old or incorrect.
By the book, pages that are gone forever and don't have a new location are supposed to return the status 410 (Gone). 404 has somewhat of a temporary meaning. J.D.
It's hard to tell - it may differ from crawler to crawler. This is what RFC 2616 (HTTP) says about 410: The big difference between 404 and 410 is that 410 (Gone) is cacheable. If a proxy server, crawler or a browser is capable of using caching, then after the first hit that returns 410, they will not make a request to your website for as long as the 410 response is valid in the cache (this time depends on many factors). This, in turn, may reduce unnecessary traffic through your website. J.D.