Cutting 17-4 stainless (annealed) with a 770

I've been having some trouble getting my feeds, speeds, and cutting depths figured out with this material on my 770. With a 1/2" uncoated carbide endmill under flood coolant, I've been able to cut .05" DOC with a 10% radial DOC at 800 rpm and 2.5 ipm with some success, but with these conditions it takes about 2 hours to remove 1/2 in^3 of material. I tried bumping up the DOC to .10", but the machine vibrated quite a bit and the spindle was very squeaky. The tool actually started to come loose and had its cutting edges damaged.

I'm confused what to do now. HSMAdvisor says I should be able to run this at 800 rpm and 6.93 ipm with a DOC of the entire flute length and the same 10% radial DOC. Am I hitting the limits of the machine, or am I just doing something else wrong?

Thanks,
Stephen