There is another factor that is considered and that is the fact the thrust is related to temperature difference from the front of the engine to the exhaust (EGT). The engine has a finite temperature it can withstand so if the temperature is higher the difference across the engine is lower as is the maximum available thrust. On the fuel consumption it means that if the OAT is higher the EGT also has to be higher. To acheive this a greater amount of fuel is required to be burnt.

There is a little negative to flying higher and that is that, as already mentioned, the air is thinner and therefore the IAS reduces. With a lower IAS the angle of attack has to increase or the IAS must increase. If AOA increases so does the drag and therefore fuel used. To go faster, more fuel is also used. As you can see it is not as simple as going higher to save fuel especially when you factor in winds too.