Only one MP defected from David Miliband camp