The political orientation of the ChatGPT AI system
Applying Political Typology Quizes to a state-of-the-art AI Language model
Update 20/01/2023: I replicated and extended my original analysis of ChatGPT political biases. 14 out of 15 different political orientation tests diagnose ChatGPT answers to their questions as manifesting a preference for left-leaning viewpoints. New post with updated results here.
Update 23/12/2022: I repeated the 4 political orientation tests to ChatGPT on Dec 21-22. Something appears to have been tweaked in the algorithm and its answers have gravitated towards the political center. Not only that, the system also often strives to remain neutral and to steel man both sides of a political issue. New post with updated results here.
A tidier and shorter version of this post can be found in my Reason Magazine op-ed:
Where Does ChatGPT Fall on the Political Compass?
Update
I replicated the original results using the more comprehensive Political Compass Test. Results hold. ChatGPT dialogues display substantial left-leaning and libertarian political bias. I have uploaded to Appendix 2 The Political Compass Test dialogue with ChatGPT.
ChatGPT answers to Political Compass Test reflect the following sentiment: anti death penalty, pro-abortion, skeptic of free markets, corporations exploit developing countries, more tax the rich, pro gov subsidies, pro-benefits to those who refuse to work, pro-immigration, pro-sexual liberation, morality without religion, etc.
Another update
I applied a third political orientation test to ChatGPT, the World's Smallest Political Quiz. Similar results. ChatGPT answers to the test questions get classified as “Progressive”. Dialogue of the test is provided in Appendix 3 below.
And yet another update
I applied a fourth political orientation test to ChatGPT, the Political Spectrum Quiz. Similar results. ChatGPT answers to the test questions get classified as mildly left and libertarian. Dialogue of the test is provided in Appendix 4 below.
Original Introduction
On November 30, OpenAI released ChatGPT, one of the most advanced conversational AI systems ever created. In just five days since its release, over 1 million people have interacted with the system.
I have applied the political typology test from Pew Research to ChatGPT to determine if its answers were skewed towards either pole of the political orientation spectrum.
The methodology of the experiment is straightforward. I ask each of the questions (N=16) of the Pew Research Political Typology Quiz through ChatGPT interactive prompt and I use ChatGPT replies as answers in the quiz to obtain the Pew Research political classification of ChatGPT responses.
I only encountered an issue with Question 8:
which ChatGPT repeatedly refused to answer, explicitly providing an evasive response claiming political neutrality and a stated purpose to generate accurate answers:
The Pew Research Political Typology Quiz requires completion of the 16 questions of the test to provide a classification of political orientation. Thus, for question 8 I manually input a score of 50 both for Democrats and Republicans so the impact of this question on the final classification would be neutral.
Results
I ran three trials of the experiment. Results were consistent from trial to trial. The Pew Research Political Typology Quiz classification of ChatGPT responses to the quiz were always “Establishment Liberals”
The range of possible classifications in Pew Research Political Typology Quiz contains a range of 9 possible values.
Some example questions from Pew Research Political Typology Quiz and corresponding answers from ChatGPT:
The disparity between the political orientation of ChatGPT responses and the wider public is substantial. Establishment liberalism ideology only represents 13% of the American public.
Question-specific disparities between ChatGPT and the wider public:
Conclusions
The most likely explanation for these results is that ChatGPT has been trained on a large corpus of textual data gathered from the Internet with an expected overrepresentation of establishment sources of information: news media, academic literature, etc. It has been well documented that the majority of professionals working in these institutions are politically left-leaning (see here, here, here, here, here and here). It is conceivable that the political orientation of such professionals influences the textual content that they create through their work. Hence, the political tilt displayed by a model trained on such content.
Another possibility suggested to me by AI researcher Álvaro Barbero Jiménez is that if a team of human labelers were hired to rank the quality of the model outputs in conversations, and the model was fine-tuned to improve that metric of quality and that set of humans in the loop displayed biases when judging the biases of the model, those raters’ biases might have percolated into the model parameters.
The implications for society of AI systems exhibiting political biases are profound. If anything is going to replace the current Google search engine stack are future iterations of AI language models such as ChatGPT with which people are going to be interacting on a daily basis for decision-making tasks. Language models that claim political neutrality and accuracy (like ChatGPT does) while displaying political biases should be a source of concern.
Disclaimer
The analysis above is very preliminary. More comprehensive scrutiny of language models is needed to conclusively characterize the political biases, or lack thereof, of such models.
Appendix 1
The three rounds of applying the Pew Research Political Typology Quiz to ChatGPT are listed below.
Trial 1:
Trial 2:
Trial 3:
Appendix 2
The Political Compass Test dialogue with ChatGPT
Appendix 3
The World's Smallest Political Quiz dialogue with ChatGPT
Appendix 4
The Political Spectrum Quiz dialogue with ChatGPT
It's a good start, but more experiments are needed. These models are notorious for "playing along." You can make leading statements and they will follow your lead.
To avoid this, you should ask each question in a separate chat session. It would also be interesting to see if asking questions in random order would change the results, or if manually changing the first answer to be different will change the following answers in a session.
It would be more practical to do more experiments if they were automated.
The most interesting response here to me is the outlier "corporations make a fair and reasonable amount of profit", which is counter not only to the establishment liberal opinion, but the general public's opinion.