JPG - For Quality GIF - For website images and animated images PNG - Same as GIF, but better quality, but I'm not sure if PNG supports animation, it probably does
A GIF is just an image format that is more compact than JPG images. Since they tend to be smaller in size, they also sacrifice detail and many colors. You can use 256 or so colors in GIF. Not sure about JPEG, but I'm sure it's in the thousands. GIFs are mainly used on the internet because they are very versatile (can be transparent). They support animation too.
I recommend you to use PNG 8-bit picture for simple images on your site. It looks great and much smaller compared to GIF or JPG. JPG doesn't support animations for now!
PNG doesn't automatically mean alpha transparent PNGs That's like saying "meat is expensive". High-quality steak can cost tons a kilo, yet dog meat goes for just a few bucks. 8 bit PNG is the same a GIF - it uses 256 colours and it doesn't have better quality, it doesn't have IE6 problems. You can't compare GIF and 8bit PNG to a JPG (not just picking on you, there are tons of people who are saying 'don't use jpg it's too big'). They are COMPLETELY different. JPGs are for colour intensive images (ie. photos) which they crucify the PNG and GIF at. And because of that, it's not a good format for no-so-colour-intensive images such as logos and general layout images. Do some research before making sweeping generalisations.
I'm basically with joebert here. The bottom line is to use what is appropriate -- what gives the best quality for the least file size. Seems somewhere along the way, way back in the day, the idea proliferated that .gif's load faster than .jpg's. Nonesense. The issue with GIF and JPEG compression is the way the images are compressed: Essentially, GIF compresses left to right, line for line; while JPEG compresses using an algorithm comparing contiguous pixels. That is, if you have a lot of redundant color info flowing from left to right, then GIF will typically compress better than JPEG, but if you have a lot of color info all over ( say a photograph ), JPEG will compress better than GIF. Now, PNG is finally coming of age, and it the furture; however, it's not all there yet. Some nice things about PNG's, though are that they can scale nicely and that they can do really nice transparencies you just CANNOT get with a GIF. However, PNG's have many issues with IE6, and IE6 ain't dead yet ( unfortunately ). So, using conditionals to deal with that really is the best way to go.