Help diagnosing possible wiring problem

I've been suffering from unbearable microphone background noise. I'd describe it as a buzzing more than a hiss. I have a desktop PC plugged directly into the wall, and a microphone connected to that via USB. Whenever I touch and hold the bottom of the PC case, the buzzing goes away and returns when I let go. A few vaguely similar posts on various forums suggest this could be due to a "grounding problem". I purchased a socket tester (this one to be specific) and it reports no issues, although it does note that it can't detect a reversed ground and neutral.

Does anyone have any ideas on what this could be or further steps to diagnose it? Thanks!

Is this a PC you built? 

Sounds like the casing is "floating" from a ground reference.

A USB microphone would only be sending a digital signal to the PC and so PC chassis grounding effects should find it hard to affect the microphone element or digital converter.  The symptoms sound more like an analogue mic lead problem.

Before diving too deep can you try the Mic on another PC (e.g. a laptop)  and in other USB sockets on the PC (especially at the rear if you are using a socket at the Front  and vice versa)