The only time I noticed a similar problem was when I was using the IFDG Airbus A320 family of aircraft in FS2002 on a system of very low specification.

I always found (and still do, although its no longer noticeable) that the IFDG A320s are far more demanding than rival aircraft of the time, such as the Project Airbus A320. If PA A320s are compatible with FS2004, maybe give them a go.