Stuck. Need some advice?

Hi. Erm. I live in Salters Lane, and its a s***hole basically. Its where they place everyone with no prospects. And none of that particularly bothers me. But what does is the amount of grief I get from the others that live on this road. In particular the bloke above me. He moans about any little thing I do, makes up crap and god knows what else. People are advisin' me to put in for a mutual exchange but I like where I'm livin' as it is easy for me to get to work etc. But at the same time, I am getting more and more down living here. I don't want ti let him think he's won if I choose to move.... What would you do?