Now that you have it taken apart, use contact cleaner to clean both pieces in the red box on the right and clean the sensor in the red box on the left. Cleaning the sensor is the most important. After spraying with contact cleaner, I use a small paint brush to clean it.

The 3 black sections (blue arrows) are what recognize up/down/left/right. If there is anything on those, it causes drift. The more you use the controller, the more the black material breaks down and leaves residue behind. Also, the more you use it, the more sweat/dust/dirt find their way into the housing.

If you use the controller long enough, eventually this will breakdown and can't be fixed. This is a problem with all controllers, not just Oculus.