I'm assuming you're on an integrated sound chip?
My guess is there's some sort of interference going on between your graphics card and your sound chip (I'm not an electronics nerd so I can't speculate further as I don't know what I'm talking about

) - using nv/vesa moves graphics processing to your CPU, leaving your graphics card to merely shove out a video signal to a monitor. Try checking it's plugged in completely, blow out all dust, etc. Also try googling for eliminating sound card interference, dunno if you'll find anything.
A solution would probably be to just buy/scrounge an old dedicated sound card. Old Creative ones are very good with Alsa, X-Fis are a waste of whatever you pay for them - the current drivers provided by Creative are x86_64 only and do no more than provide basic sound. This may change with the next release of alsa but don't hold your breath.