I had the same issue and resolved it by re-touching my solder work on the home button flex ribbon - (the smaller set of joints that attach to the glass)

The speaker and mic have rubber tips that fit snugly into their sockets on the case and I believe there is glue on the mic but its very light. Also, where you removed the two screws on either side of the docking plug can be stuck or snagged, but yes it really should just pull out. If th e main board is out you should be able to just worry it/wiggle it out. It prolly just the mic that's holding it snug.