The button has at least three parts, the black button that presses in , the centre white "spider" ( for want of a better word) that the button presses against and the spider presses into the third part, the main body of the switch.
If the black button has fallen off in the past, someone MIGHT have tried to reassemble it and then pushed it all back together again and the white "spider" could be sticking down,
One thing that MIGHT help you pin it down. The button becomes dead (electrically) when the door is closed (or if the micro-switch ion the lock has broken (but that is not likely to play a part here.
What that means is that if your flap is opening without the driver's door being open, then it is likely to be a mechanical problem with the linkage and solenoid that operates the flap opening. (if you take off the "inspection panel" where you would change the rear light bulbs) you can see the whole mechanical set up and solenoid.
You could try closing the flap then manually operating the linkage that opens the flap and determine if that is all lined up and working correctly.
Steve B