A potential post-AGI world government

It seems like with AI we're stuck between 2 worlds, one where we cannot effectively coordinate to prevent x-risks, and one where effective coordination requires, and results in great concentration of power which comes with it's own pandora's box of issues.

Peter Thiel talks about this often, especially with his Antichrist lecture series. He talks about jumping from the frying pan into the fire, as we give in to power concentration (the antichrist) to save ourselves from x-risk made worse by coordination problems (armageddon).

Dario is also quite vocal in his worry about China and authoritarianism.

How do we have a functional world government that is long term stable yet not a hegemony without checks and balances?

First let us talk about why authoritarianism is bad :

1. Incentive misalignment - Authoritians with great military power can often do things that benefit themslves and part of the populace and greatly harm others and get away with it, because they do not get votes from all citizens. In democracies, feedback from the entire population through voting prevents much of this.
2. Bad ideas - They often get stuck in a single point of view, and if they are stupid enough to not update their views from real-world data the country suffers and no one can outcompete/overthrow them. Stalin thought communism was amazing and would save the world until the very end.
3. Succession - A benevolent dictator may be great as long as they rule but somewhere along the line someone comes up who is not very morally good and causes a lot of problems. Marcus Aurelius was a great ruler, but did not handle succession well with his son Commodus.
4. Irreversibility - Once something goes wrong you cannot reverse it, you are permanently stuck that way. Because usually reversions happen through force.

But maybe an AI that is perfect smart, and does not need to worry about succession, and has aligned incentives could work?

An AI misaligned along any important axis in this case will be catastrophic - if it chooses to do something bad none of us can prevent it.

(It also seems important that very similar AIs across countries may face this problem too. They may all have the same morals or failure modes or may collude very easily).

One may think democracies have great checks and balances, why not have a single democratic government that fixes coordination problems?

I think a global planet-wide democracy with the worlds-strongest-army not be enough.

As AI makes it much easier to do military operations without needing the support of many other moral humans, the democratic systems could be slowly eroded over time by people in power, until someone can consolidate enough power to overthrow democracy completely.

(Typically in democracies, getting enough coordination from other legislators, and people in the army's chain of command is simply too difficult to do this in the non-AI world)

Hence we need many countries and militaries. Even if a single country's democracy does get overthrown, the other countries can ally and collude to overthrow this single-country authoritarian regime because authoritarianism rising is rational-bad for everyone.

So it should be a system with multiple military powers that are checks and balances on each other. What system could this be?

Such a system should have a few qualities.

1. Each individual country government should have feedback systems for incentive alignment.

One of the biggest failure modes of totalitarianism is incentive-misalignment. People in power make laws that benefit themselves and a few others at the expense of everyone else, and no one can stop that cycle if one bad ruler is in power.

In democracy, as soon as most of the population suffers for the benefit of a few, voting acts as the feedback loop to remove that bad ruler and bring in someone who will do broader good.

However even voting is imperfect. People may know if they're sad or happy, but they must model the impact of different politicians and their policies on their own well being. We often do this incorrectly (it is super difficult to do), especially under the influence of well funded media campaigns.

People submitting their grievances and joys to a system that then models the impact different candidates would have on global good could arguably be more functional.

2 . The decision makers should be intelligent.

One of the biggest problems of totalitarianism, is that uncorrected mistakes have no feedback loops to check it. Stalin thought he was doing the world a great service by spreading communism, and did not update his views even as empirical data was flowing in. A hyper-intelligent dictator who wanted good for the world would likely not fall into this trap.

Even in a multi-polar, non totalitarian world, very intelligent, rational players can solve many of their coordination problems much better.

For example, in today's world - it may be evident to a very intelligent entity, that AI poses great bio and cyber-related risks from humans using them. Today in the government, many simply do not acknowledge this risk.

A 20% chance of a $20T loss for each country if AI continues without a halt, would make halting with coordination (with verification) super easy, as long as the countries had the foresight to calculate and believe these numbers without really bad warning shots.

I have spoken to several friends and seen politicians talk about these things in-person. This seems to be a combination of wishful thinking, and not viscerally feeling the downsides to prioritise it over other more immediate problems.

> "Why slow down AI when billionaires are leaving California and we must reverse that by making AI businesses easier here?"

> "Why slow down AI when it seems ridiculous that AI will kill us all but my kids have been building and learning such cool things?"

They are just modelling the problem incorrectly.

3. What of the remaining coordination problems where there are actual conflicts of interest?

Sometimes very intelligent, rational players fighting for their own people can get stuck in a race to the bottom in a prisoner's dilemma. What of this?

Now this will come up only in non existential risk cases as long as all players model uncertainties very well and approximately the same way (because the downside is -infinity in those cases).

That leaves the non existential cases - countries undercut each other's corporate taxes and regulations to attract business. Every country benefits when others cut emissions, but pays the full cost of cutting its own. Militaries build up because their neighbours do. In each case, everyone would be better off cooperating, and everyone is individually better off defecting. Intelligence doesn't fix this, it is just reality.

A few things can work here :

1. Make defection visible, so you do not have to trust the other side. i.e. make the cooperation verifiable. This seems like the rational move in all cases, unless coordination is absolutely un-verifiable.

2. Make the value function of each country more than just the well-being of their own populace. Have a factor for general-world well being of the world or morals.

For example, a case where many countries of the world would never collude to mistreat one small country because it is morally wrong and is not greatly beneficial to them.

Here is a possible setup that could be long term stable, as a thought experiment.

1. AI-aided democracy - Every country operates in a democratic manner. The people who lead the countries are aided by AIs to be intelligent and rational.

2. Voting in that democracy - Everyone has a personal super intelligent AI that knows the state of their well being and advocates for it. The people do not vote. This AI decides perfectly which policies they should vote for. Ideally voting should not be for candidates (arbitrary clustering of policies across different categories for a single candidate). It could be for individual policies. Anyone or their AI can propose a policy that could improve global well-being and it gets voted on by all the personal-AIs.

3. All AIs having some notion of morality like humans - The individual super-intelligent AIs take into account some values like fairness and morality from their users. This will make getting cooperation for unfair things like ganging up on minorities more difficult.

This seems like an important topic to simulate/research for the post-AGI world. Science fiction greatly helps in imagining some of these situations, I recommend Iain Banks and the culture series (https://theculture.adactio.com/).