Part of the “problem” may be that people call them “urban explorers” (the name implies legitimacy) instead of vandals and trespasser …which is what they are.
You could brand a lot of aviation preservation societies as vandals, and amateur aviation archaeology is responsible for a fair chunk of vandalism. All these groups that went out in the ’70s and ’80s and asked farmers where aircraft came down, only to crudely hock the remains out the ground. They never found human remains, naturally.
I’m sure there is an element of the urbex community that does jimmy open windows and strip period fittings, but they seem pretty good at self-policing. The bigger crime is letting something of historic or architectural significance get into such a state that a bloke with a tripod can get in and take photos in the first place, surely?