all I can recall is the Doctor and the Valeyard locked together Holmes and Moriarty style...

I believe so, and that was deemed too downbeat, especially as it was potentially the last ever episode. I think that's what prompted Saward to storm off (or at least was the final straw).