Hey Tony.I know what your dealing with. the way i handle it is with a can of "Dust off" It's compressed air in a can. a few short blast's and off the powder goes.
Becareful! sometimes moisture comes out first. hold it a few inches away and it works dandy. It's totally hands of the drawing, no kneaded erasers or brushes. I never blow it across my work I rotate the drawing and blow it off in what ever direction works. I've never had a problem with powder affecting the rest of my drawing.
Marc Forest.
http://www.marcforest.com