The mirror is on a plastic backing plate, this plate is pressed onto a pivot retaining ball in the housing with 3 more studs that allow the remote adjustment. Now the pivot ball and mirror plate socket are a tight fit.
I've taken several off with only one breaking the glass in the process. You have to apply pulling force all around the glass to release it other wise it cracks due to the backing plate flexing.
Is it the glass moving or the whole unit? To fix the glass moving you may need to replace the ball or socket or both. If it is the whole thing just tighten up the fixing bolts under the door cover, or the hinge on the mirror housing body.