PM powerless to prevent empty Olympic seats