What the F-104 had against it, is that it was probably too specialized. The Mirage, Draken and MiG-21 I think has proven to be more flexible in coping with changing threats.
I don´t think that…
The F-104 could make several misions, long air atack (with aspide), CAS (with bombs, napalm and rockets), antiship (kormoran), antirunway (MW-1)….
So I think, it was so flexible like others…
1Saludo