There was a DWM special years ago that contained a review that started by sounding like it was about The Leisure Hive before turning out to be - surprise! - The Horns of Nimon, the point being the same as yours, that the big changes to the format came later in the season. I agree that Nimon was probably the "silliest" story of the Williams era, so the contrast is stronger than if it had aired after, say, Destiny of the Daleks or The Power of Kroll (not that those are necessarily better than Nimon, but are tonally more similar). I think the different incidental music changes how the stories 'feel' too. One DVD extra has a bit of Nimon with a score by (I think) Paddy Kingsland, and it does seem like a disorientating hybrid of seasons seventeen and eighteen. The change in Tom's costume might also have a subliminal effect, more sombre and uniform.