Make 404 your friend - don't waste it

[expat]

Being a cold gray day in the UK I have nothing better to do than surf and find link partners and check sites out.

Boy - checked about 30 sites and half had 404 come up one way or the other when I looked deep.

Why? Even the cheapest hosting provides you with a custom 404 page and just a bit of php or ssi let's you send an e-mail to yourself or your development and you can spruce the page up a bit so your visitors don't get annoyed.

I make mistakes but 404 - mail keeps me on track. I only need one beta tester (visitor) to find a dodgy link and I can easy find and repair it.

(Oh and I do hate the sites that don't admit it but send you to the home page)

I also don't run a robots file - so each time G Y I and the lot request it I get a 404 - just scanning through them I know who visits when and how often.

So why not more people make 404 their friend is slightly beyond me.

M

[hans]

i fully aggree with your post about 404 - its amazing how many people have death links.

i tested and used many link-checkers .. including offline ones or such as from w3c.org and other free services ..

yet by FAR the very best has proven to be webcheck (http://mired.org:8080/webcheck/) - it allows
file:///
ftp:///
and http://
mode
hence i use it in two versions
offline in my linux box ( with NO external links except for nameresolution )
ONLINE installed on the server of my domain and called by a small shell script upon request if i make many changes OR at least by crontab once a week.

webcheck gives to singe most comprehensive ouitput and detailed report i ever saw ..

written in python and free ..

what you want better ??

i decided recently to make a separate online installation with its own configuration file to save my online time ( my site has some 3000+ files ) and many thousand links - on my usual 28 or so kB dial up - it would take sonme 3 hrs or so :-)
after my crontab ( or upone request ) has started the webcheck script - i get the full report emailed to me by my script a few minutes later as soon as finished.

with the online version its a matter of minutes
while my offline version does the same job in file:/// mode in less than 30 seconds for the full site.

its just another way of preventing 404 in a professional and efficient way.

and if you have a non-profit or humanitarian web site - i may even run my online installation free on our site via crontab once a week or once a month within my free resources available !

may be you like to try it yourself if you have an active site or many links

have fun

[Owlcroft]

A php 404 page is invaluable. It is especially critical when there have been changes to a site's organization. Not so long ago, for example, I changed virtually all my pages from html to shtml, so I could stop changing hundreds of "site directory" lists every time I did something; meanwhile, though, there were scads of links out there to the old URLs. A simple php script in the 404 page now deduces the new wanted page URL and sends back a 301 with the correct location (I used to use a redirect page, but that's silly).

Professional sites may not have this issue, but it is common for little guys like me: you start off with a small site off some general domain you have, something like--

http://mydomain.com/myhobbyhorse/

Then, your hobbyhorse gets interesting, and larger, so maybe you go for a vanity "domain" and run--

http://myhobbyhorse.mydomain.com

Finally, you stop being cheap and spring for the whole thing, and are--

http://myhobbyhorse.com

Well, as you've grown, you've acquired in-links, but just try to get any good fraction of them to update those links. But an only moderately smart 404 page can handle the whole problem, dispensing 301s left and right.

I don't know--and would love to, if anyone does know--whether Google and Inktomi "look through" redirects to determine the true link, or if they discount all redirected linkage. But I reckon it's best to send them a 301 anyway.

[rfuess]

Thanks for the idea. I haven't thought of interpreting renamed pages, although I do have a 404 page that has an appology and a site map.

I'll have to have a data store to track these as I go along. . .

Always learning . . .

Robert Fuess
Spiderweb Logic (http://www.spiderweblogic.com)

Cool HTML color chart (http://www.spiderweblogic.com/HTML-color-chart.aspx)

[TwisterMc]

I have a 404 page that just has an error message and links back into my site. No email script though. However the company i work for doesn't do custom 404 pages. I tell they NEED to and they don't do anything about it. :rolleyes:

[leeds1]

My 404 links back to the home page - is that really cheesy ?

[TwisterMc]

My 404 links back to the home page - is that really cheesy ?

Does it link back or refresh? It's not a bad idea but can be really annoying for the developer if it refreshes. Also, if it refreshes, you'll need to make sure the visitor knows what's going on and doesn't get more confused. :)

At least you have one going somewhere! That's good.

[leeds1]

it's just a redirect so any not found page goes to the index.

I've found yahoo "makes up" pages in my site that I don't have or ever had - with #1 rankings - really strange

[younghistorians]

What type of php code do you guys use on your error pages? I'm interested to see it and how it works.

[leeds1]

I just pop it into my .htaccess file

[expat]

simple e-mail code


<?php
$count = 0;
// don't send mail if we don't know the referring page
// it's probably a searchengine or infected windows system requesting known NT server exploits
// may reinstate but wana know if works
//if ($HTTP_REFERER == "")
//{
//$count++;
//}
// debug comment, you can remove this line if you want:
echo "<!-- count: $count -->\n";
if ($count == 0)
{
$today = date("j F Y, G:i:s");
$message = "Date and Time: $today\nRequest URL: http://$SERVER_NAME$REQUEST_URI\nReferring (http://$SERVER_NAME$REQUEST_URInReferring) page: $HTTP_REFERER\n\nClient: $HTTP_USER_AGENT\nRemote IP: $REMOTE_ADDR\n\n";
$message .= "This is an automated message.\n\nHave a nice day.";
mail("xxxxxxx@xxxxx.com (xxxxxxx@xxxxx.com)", "Error 404", $message, "From: page404_dep@xxxxx.com\nReply-To (page404_dep@xxxxx.comnReply-To): page404_dep@xxxx.com");
}
?>
Nothing fancy send an e-mail to my development mail account.
The one above also sends me all the robot requests going wrong as I don't use robot file and I see all the crap requests from mail exploits etc.
M

[Help Desk]

Do custom 404 error pages get cached by search engines?

[expat]

no "normal" standard 40x, 50x, etc get cached.

but if you link to your custom 404 it can be cached......

Here is a famous one...

http://www.searchlores.org/404.htm

M

[schlottke]

That is a pretty funny 404, Im going to have to make one of those- even though I don't have much of a problem lossed pages.

[LucidNet]

cheers expat for the info on 404, im getting mine sorted, never knew how useful they were till now!

Just a few questions,

How come the 404 page has to be bigger than 10k?

Also, should i design the page to look like part of the site, with an explanation of what has happened and maybe a sitemap, or...

...should it be more clear that there has been an error by using a different layout, and maybe black text on white background, instead of my sites black background and white text.

Also should i set up 400, 401, 403 and 500 pages too just so they match, or will it not really matter at the end of the day

Thanks

[expat]

Hi LucidNet,

the 10k question I can't answer as I don't care about size if a message takes 40k so be it.

Regarding page design it depends on your visitors. The page will be seen by your visitors so you have to make a decission what you can ask them to do.

You can be flippant or include scripts that try to find a likely page and show a selection.... it's your visitors that you want to give it another try......

Depending on sites e.g. if they are script / pgm heavy and/or include authorisation request sections I do generate the other likely pages sometimes with a delayed redirect to the homepage as I find it anoying clicking on a link and landing on the homepage without any explanation.

Hope it helps
M

[donn]

I know I'm adding this to an old page, but it's relevent. Does anyone try to monetize thier 404 pages? Maybe dump an adsense search on it, or something.

[brian394]

I know I'm adding this to an old page, but it's relevent. Does anyone try to monetize thier 404 pages? Maybe dump an adsense search on it, or something.
Short answer is, you can't do this, because it's not allowed by Google's TOS (https://www.google.com/adsense/localized-terms)...


You shall not, and shall not authorize or encourage any third party to:
...
(v) display any Ad(s), Link(s), or Referral Button(s) on any error page, on any registration or "thank you" page (e.g., a page that thanks a user after he/she has registered with the applicable Web site), on any chat page, in any email, or on any Web page or any Web site that contains any pornographic, hate-related, violent, or illegal content;

Long answer is, yes you can do this, but only if you go to great lengths to make your 404 page not look like or behave as a typical 404 error page would (in other words, you make it into a content page). You must basically transform the entire page, so that's it's not actually an error, but rather a dynamically generated content page.

I see a lot of sites which do this, and it can be done successfully, however, there is one very important thing you have to remember (actually a few things). First of all, you absolutely must include a lot of other substantial content on the page. If you remember from the Google Adsense Program Policies it states ...

No Google ad may be placed on any non-content-based pages.

A simple "File Not Found" message is not going to cut it. Another important thing to remember is that you should not include the number 404 anywhere in the title or on the page itself, nor should your web server return a response code of 404. Take a look at this for example...

http://www.vincentrich.com/test

That's an example of a monetized 404 page. You can still clearly see the site's logo, navigation structure, newsletter box, category list, search box, links to archived posts, and so on. There's still plenty of content on the page for Google to derive ads from. Another thing you might want to include are "close matches" to what the user typed in. For example, if the user typed in

http://www.xyz.com/gogle.html

Your 404 page might say...

The file "gogle.html" was not found. However, the following similar pages were found...

google.html

I see shareware sites which do this a lot. If you try to type in a page which doesn't exist, for example...

www.example-shareware-site.com/business.html

It may return back to you a list of all matching programs in the "Business" category, but it will not say "Error" or "File Not Found" anywhere.

[goldensea80]

:)) WTF!!
Oops! Sorry the file you are requesting has been here done that and is GONE!
From: http://haleokala.com/laprice.htm

[goldensea80]

And what about this: http://www.tinsanity.net/404.shtml

[tycoonjo]

adsense use can make a nice profit on these

[goldensea80]

adsense use can make a nice profit on these
No, you cannot!

By the way, here is my new 404 page, is that cool ;): http://imagesh.net/404.php

[Toolz]

I like to redirect to a site map when I get time to do maintenance.

Then users can find their way again and you don't loose them.

[AdamSee]

http://fourshapes.com/dpfduifdufisaui/

I'm going to have to add some humour to it.