That question totally depends on what you need/want out of a car and your personal finances.
If you are looking for something very fast and extremely sporty, then a used M3 is perfect.
You have to consider the upkeep on an M3, and depending on the year and mileage the M3 might be out of factory warranty. Even if the car is certified pre-owned, you will be responsible for the regualr maintenance...brakes, rotors, oil changes, etc... The new 2004 will come with 4 years/50,000 warranty and FREE maintenance. If you are considering a 325 vs. an M3...why not consider a 2004, or maybe even 2003 330ci???
Good Luck!!
When I got my 04 330cic, I chose it over a 2004 M3 coupe (silver with Red leather). The 330 was more comfortable, more practical, less to insure and I like the Steptronic a lot more than the SMG, and the 6 speed manual. I am so glad I did not get the M3!!!