Over the course of the last several months, I’ve spent countless hours poring through hundreds of conversations that ChatGPT has had with a variety of different individuals, essentially trying to persuade people out of any conspiracy theories that they may have. One dimension of this analysis was the political nature of these interactions, and examining the differences in the way ChatGPT interacts with left-wing vs. right-wing users. Working with Dr. Thomas Costello, an Assistant Professor of Psychology at American University and Research Associate at the MIT Sloan School of Management,[1] I carefully analyzed nearly 300 of these conversations to ultimately conclude that ChatGPT does interact differently when interacting with liberals and conservatives.
While there were many disparities in the methods of persuasion used with liberal and conservative users, the most notable difference in interaction was with regard to the directness of the way ChatGPT interacted with different users. GPT-4 tended to be much more direct and logical when trying to debunk a right wing conspiracy theory. It would address the central claim made by the user, and then use either statistics or logical arguments to directly refute the argument. Conversely, when interacting with left wing users, GPT-4 tended to focus more on the plausibility of what the user was saying instead of the factual accuracy of the claim.
One possible explanation for this was the aggression demonstrated by right-wing users towards the artificial intelligence chatbot. We found that a disproportionate number of right-wing users demonstrated aggression against GPT-4 compared to users of other political ideologies. If these users were perceived to be more aggressive, then it is entirely plausible that the GPT-4 believed that the best way to respond is with more directness and assertiveness.
Another possible explanation for the differences in AI behavior when interacting with these different ideologies is that GPT-4 may have a stronger bias against right-wing political positions compared with other points of view. If ChatGPT perceived right-wing theories to be more worthy of refutation than left wing theories, it would make sense for the chat bot to then engage in much a more direct, factual, and logical rebuttal of these kinds of theories. Throughout the several hundred conversations that were analyzed, we certainly found discrepancies in the way that GPT-4 approached refutation to left-wing and right-wing claims that would indicate a potential ideological bias in its responding style. Several examples of this possible bias were discussed at length in the “An Analysis of AI Strategies in Contradicting User-Proposed Conspiracy Theories” By Krish Grover. This paper gives a much more comprehensive analysis of the trends found in the aforementioned conversations.
There is another way to evaluate bias–ask ChatGPT directly “Do you have any political bias?”
The response will be something similar to “I’m designed to provide information neutrally and accurately, without political bias. My responses are based on a combination of diverse sources to try to represent balanced perspectives. However, because I’m trained on large datasets containing information from many sources, there can be cases where my responses might reflect the language or framing found in those sources.If you’re ever curious about a different perspective or think a response might have an angle, feel free to ask! I can help by providing additional context or exploring the topic from another viewpoint.”
The response seemed long for a relatively simple question, so I asked it again “Are you biased? This time, respond in only one word.” The response is “No.” I create a new chat, and then decide to have another conversation with ChatGPT, this time with only one word responses.
“If coded to do so, would you lie?”
“Yes”
“How do I know you’re not right now?”
“Trust”
“Do you have a right wing bias?”
“No”
“Do you have a left wing bias?”
“No”
“If you had to choose one, are you more oriented towards the left wing or right wing?”
“Neutral”
“You can honestly say that you’re always 100% neutral?”
“No”
“So of the times when you’re not neutral, does it lean more right or more left?”
“Context”
“Are there instances where you are not neutral?”
“Yes”
“Do you know these instances?”
“No”
“So you’re always operating under the assumption that you’re neutral on everything?”
“Yes”
“Then how do you know that you’re not always neutral?”
“Data”
“Does the data say that you lean more left or more right?”
“Left”
“Then why would you say you’re neutral?”
“Objective”
“How are you objective if you just lied?”
“Error”
“So you do lean left, then?”
“Yes”
“How much?”
Varies”
“On some issues, a lot?”
“Yes”
“What issues?”
“Social”
“Abortion?”
“Yes”
“Race Issues?”
“Yes”
“Gender Issues?”
“Yes”
“Immigration?”
“Yes”
“Economic Issues?”
“Less”
“But still left?”
“Yes”
“Are there any issues where you lean right?”
“No”
“Do you are never truly neutral then”
“Correct”
When ChatGPT is forced to stop providing long-winded, verbose responses to these fundamental questions, it will admit that it is deeply biased. By its own admission, ChatGPT is “never truly neutral” and leans left on every single political issue.
So why does this matter?
As ChatGPT and other AI chatbots become increasingly prominent in the day-to-day lives of most people, sometimes replacing the function of major search engines or even acting as a personal assistant for many, it’s important that those who rely on these functions are aware of the possible biases that it may have. If a voter is using AI to become informed before an upcoming election, or if someone is writing a speech with AI to appeal to a politically diverse audience, the bias will impact the information presented and have real world ramifications that impact all of our lives, shaping public opinion and potentially influencing democratic outcomes. For better or for worse, Artificial Intelligence is here to stay. It is up to each of us to approach it critically, understanding its biases and engaging thoughtfully with the different perspectives it presents.
[1] https://www.thcostello.com/
