My site design is different in IE and Firefox

No, they are not totally different. Internet Explorer 6 and 7 both use the Trident rendering engine. IE 7 uses an updated version of the engine, but it still uses the same engine as IE 6.

So yes, while there are four major browsers you should test on, they are (in alphabetical order), Firefox, Internet Explorer, Opera (you forgot this one), and Safari.