The crisis over Rupert Murdoch’s newspaper empire illuminates the structural crisis in Britain’s broken system of newspaper self-regulation. The closure of the News of the World, the crumbling of the edifice of obfuscations and lies erected over the past five years by News International to defend its indefensible activities, the widespread belief that more revelations about similar activities in other papers are yet to come and the manifest incompetence of self regulation in the British newspaper industry call for a radically different solution.

David Cameron, prime minister, rightly said that the Press Complaints Commission, the industry’s self-regulatory body, should be scrapped. But to create a new and better system there is a familiar dilemma about the regulation of newspapers, which must now be cut through.

On the one hand: they exist in a liberal democracy which values and guarantees freedom of opinion, freedom to publish and the right to hold powers to account. On the other: their wide distribution and power over news means that, when they lie, distort or report wrongfully – or, as with the current scandal, habitually break the law – then their actions affect the health of the society. This means they must be regulated.

This same dilemma is not true for broadcasters. They are regulated in Britain, and have a history which, for the most part, speaks to their independence, impartiality and willingness to investigate and expose abuse. Yet a significant part of the rationale for regulation is to ensure that the broadcasters are objective. Newspapers do not have that requirement, nor should they.

Regulation by the industry itself has been for two decades the only answer to calls for some form of public protection against press abuse. Yet this is clearly inadequate to the task. Those with an interest in a free and transparent press are many; currently, the only stakeholder in the PCC is the press itself, and it has failed in the largest challenge to its ability to self-regulate.

So we have a dilemma. State-backed regulation is seen as illiberal, and would be opposed (on liberal grounds) by all of the press. Yet self-regulation – paid for by the newspapers, dominated by News International and Associated Newspapers – has proved self-serving and supine.

The answer to this dilemma is not to create a new regulator with statutory backing. Instead it is to increase the group’s base of stakeholders – and to include in the number of institutions making up that base the government itself, as a representative of the public interest.

How would this work? When the PCC is replaced, a new organisation should be established, which I would call the Journalism Society, in a similar vein to the Law Society, the representative body for solicitors in England and Wales. This body should be open to being as global as the media are fast becoming. And it should be independent.

The Journalism Society’s stakeholders should include representatives of the government; the educational establishment; civil society (for instance relevant non-government organisations and policy institutes); industry and finance; and the news media. All of these would be committed, under its charter, to pluralist, independent, opinionated news media, working within the law.

This move would have a number of advantages. First, it would make clear that the interest in such news media is much wider than the press itself. Second, no one interest would dominate: the board would be so constructed as to preserve a balance of interests, and it would have no statutory powers.

Third, it would increase the financial backing for the organisation itself, allowing it to extend its scope, to conduct research and investigations and to be proactive in describing and managing the huge changes now under way in the news media – with a brief to ensure that these changes are in the public interest.

Finally, and more importantly, it would allow journalism to take itself seriously as a trade claiming a democratic mandate. It could set and patrol ethical standards, monitor training and qualifications and above all be a forum within which journalists could map out the nature and future of their craft at a time of rapid change.

Dame Onora O’Neil argued, almost a decade ago in her Reith Lectures, that the news media press the need for trust and transparency on others, but not on themselves. The News International affair will have done unintentional good if it ends that period of silence.

The writer is a contributing editor at the FT.

The age of accountability has come to the media at last

“I’m with you on the free press; it’s the newspapers I can’t stand”, as one of Tom Stoppard’s characters, Ruth, says in his play Night and Day.

The News of the World scandal, showing journalism at its criminal worst and dogged best creates an urgent task for defenders of a free press: how can its principles now be defended from the worst practices of the newspapers?

John Lloyd identifies genuine dilemmas about statutory regulation, acknowledging that they have been used too in a self-serving way by newspapers which are scathing about self-regulation in every other domain.

There is now an immediate opportunity – before any inquiry – for media practitioners to set out and publicly debate what a reformed Press Complaints Commission with teeth would look like.

But the media can now longer expect this debate to remain in-house.

And it is important to understand why cogent critics of the media’s ethos and sense of accountability have been marginalised within debate about media practice over the last decade. Figures such as Mr Lloyd or Onora O’Neill, the philosophy professor at Cambridge university, have been caricatured as lofty ivory tower academics and commentators, though their warnings about the threat to the media’s public legitimacy and licence to operate have been borne out.

Too many journalists see these as important debate for journalism schools, but not so much for newsrooms themselves. Inside every newsroom lurks the “dirty hands” theory of how to get results. It is held by liberal broadsheet editors as well as those of the red tops. The News of the World’s had legitimate scoops – such as the Pakistan bribery scandal – through subterfuge too.

The test of an effective PCC would be one that was aggressive and formidable when being stonewalled and lied to, but robust in defending aggressive journalism in the public interest.

Could that ever be captured by a rule book? It is about the ethos, powers – but it is about personalities too.

Peer review is necessary to media scrutiny. But we have a very clubbable media elite – and there is little confidence within it that really big calls would go against the proprietors or editors of the most powerful organisations.

So our media culture has been opened up by the rough and tumble of civic scrutiny and discourse. Some fear the blogosphere and social media can too easily turn into a tweeting mob. But the age of accountability has come to the media too at last.

The question of how journalism restores trust will no longer be for journalists alone.

The writer is director of a soon-to-be-launched organisation that will contribute to and inform media and public debate on migration, integration and identity. He is a former general secretary of the Fabian Society, a think tank.