As for the A340-500/600 with GTF: The problem is twofold (attention: here comes good information for free on the public internet!).

1. The competitor, B777-200LR/300ER uses the GE90-115, which is a very efficient engine and more advanced than the CFM56-5B/C. The claimed increase in SFC would therefore be less. Additionally, a better SFC does not directly translate into better fuel burn when a complete aircraft os looked at mission-wise.
2. The A340-600 suffers not only from its engine but also from its higher OEW and some other disadvantages. Re-equipping with GTF would only increase the OEW. It remains doubtful if the A340-5/6 could beat the B777-2/3 with any useful margin.

And the A340-500/600 Trent 500 engines are in any case too big to be replaced by GTF-s. A340-300, on the other hand, is small enough to be propelled by GTF. And its competitors are B777-200ER and A330-300.

DC-8 reengining was quite worthwhile, even though new built DC-10s were competing against them. However, the production line had been shut down too thoroughly to build new DC-8-70s. Would the A340-300 fleet be worth reengining?

I recently read that P&W does not completely rule out a large thrust GTF. But the question is how the benefit compared to latest technology turbofans really is.
Would be interesting to know if a A340-300 with GTF could compete with new twins.

Indeed. See
http://www.flightglobal.com/articles/2009/03/24/324078/gtf-versus-open-rotor-which-is-best.html

240 cm diametre engine – compared to 170 cm of the best existing A320 engines. Which means reengining the A320 looks impractical. Whereas A340-500/600 already has 240 cm diametre engine, and so does A330, so the free space should be right there under A340-300 wing.

Conclusion: if you can avoid three engines, do it! The reason in the late 1960ies was that two engines were considered insufficient for long range flight and that no suitable engines existed for a twin DC-10 or L1011. Today the only reason remains the availability of suitable engines.

But they might have built a small quad DC-10 or Tristar, like A340 or Il-86. For some reason, three engines seemed better.

Why has the A340 4 engines?

The A340 has a design mission and was then consequently adapted to the available engine. The engine of choice was the CFM56, I guess because of its wide spread use.

But the engine of choice was the Superfan, which could not be built. CFM56 was emergency replacement, and an unsatisfactory one. Let´s see if Pratt and Whitney GTF works.

The only other option would have been an extremely derated PW2000 for 4-engine config or the same engine at max thrust rating in a tri-jet.

What was wrong about, say, 3X Trent 600?

McDonnell-Douglas seriously tried to buy A330/A340 wings for their MD-11 stretch. This fell through; but Airbus also might have done something similar themselves.

No need now.
The trijet was to overcome the ETOPS “No twins” attitude. Now that twins can cross the pond, and the engines are getting safer and safer…..and more economical, I doubt if we see a trijet again. 3 engines is 50% more maintenance costs than a twin, too.
You could build a twin engined design to lift the A380,

No engine gives so much thrust. You would need 3x GE90-115 to propel an A380.

But Airbus finds that the increasing bypass ratio of turbofans means that the fan size grows for a given thrust. They run out of underwing space for it. For example, the Cseries geared turbofan has 188 cm diametre fan, compared to A320 engines which are under 170 cm or so, and produces much less thrust.
For that reason, Airbus patented a trijet with H-tail, and the tail engine mounted in the middle. Only, they did not specify whether they intended a widebody or a narrowbody.