A car with a wreck on car fax is worth 10-20% less than a comparable car without an accident history.
I can afford the car, but a $10-$20k hit in value isn't something I want to assume when the accident wasn't my fault. The car isn't fine in my opinion. It is likely "fine" in the respect that it can be fixed, but it isn't "fine" in the respect that is was an un-wrecked, unpainted car, worth more a couple of days ago than it was today.
The fact that I trade them in ever 2 years means I can expect a $10-$15k+ loss in value due to the negligence of another. Closer to $20k if I sell it now, which I will likely do.
Petty? Well, maybe, considering I could have been hurt badly and that it is just a car. But I don't think it's petty when I stand to lose that type of money through no fault of my own.