As said above the key element is control surface flutter.

I once flew an aircraft which, unbeknown to me had an unauthorised mod to its ailerons i.e. somebody had replaced the fabric with thin plywood. This moved their cg aft of the hinge line. When I was flying this aircraft and let go of the stick above 90 knots, the stick immediately started to thrash from side to side, and the aircraft developed an alarming side to side wobble which stopped when I grabbed the stick. This is known as stick free instability and is different to wing aero-elastic flutter as demonstrated in the glider YouTube. Control surfaces can also flutter stick fixed, i.e. against the stiffness of the control rod/cables and this normally results in them falling off.

After landing and grounding the aircraft, the original design standard i.e. fabric, was reinstated and the problem was solved.

Fabric just will not work at high speed. When aircraft started pushing speeds beyond about 0.7 Mach, the fabric started to fail due to the formation of localised shock waves and intense pressure differences are beyond the strength of the fabric, i.e. there is a picture somewhere of an early Me 163b with no fabric left on its rudder after a high speed dive.

Now you can design with either metal or plywood covering the control surface but there’s a negative. You must add mass to the surface to restore its cg back to forward of its hinge line. The extra mass increases the control surface loads, which means heavier, stronger wing structure, and you need significantly more force to move the surface. The Gloster Meteor ailerons were said to be unmovable at it limiting Mach number. Hence the introduction of power flying controls, and yet more weight.