Former Labour minister turns down coalition job