Would You Trust AI For Ethical Advice?

Source: Knowledge@Wharton
Photo: Christian Terwiesch

Most people would not prefer to take ethical advice from a computer, but a new Wharton study shows how attitudes change when users see how good AI guidance can be.

No disrespect to philosopher Kwame Anthony Appiah, but Wharton professor Christian Terwiesch no longer reads his wildly popular advice column, “The Ethicist.”

Terwiesch used to love the long-running column in The New York Times until 2023, when he and two Wharton colleagues conducted an experiment to see whether people could tell the difference between Appiah’s advice or advice spit out by ChatGPT.

In a blind study, even experienced professionals — including ethics professors and clergy — could not tell whether the AI-generated advice was more or less useful than the advice provided by Appiah. In fact, random participants preferred the AI-generated advice 59.6% of the time. An additional study found that participants who were initially resistant to taking advice from the chatbot became more accepting as they were shown the high quality of the AI reasoning.

“I stopped reading his column ever since I ran the experiment. It really did something to my brain,” said Terwiesch, a professor of operations, information and decisions and co-director of the Mack Institute for Innovation Management. “I have all the respect in the world for Dr. Appiah and other ethical experts who provide people with ethical advice that helps us live in a better world. But knowing that I might as well ask ChatGPT for the advice has taken the fun out of reading the original. Dr. Appiah was kind enough to respond to us when we exchanged emails with him after the study. He attributes this result to the AI’s persuasiveness and its ability to have found a ‘cultural average’ through its broad training. I agree with him — in this study and others we find that AI models have gotten really good at communicating with us and really capable of finding our sweet spots.”

Terwiesch, who made national news in 2023 after feeding his MBA exam to ChatGPT to see how it would perform, has continued to study the impact of artificial intelligence. His latest paper is titled “Advice Quality and Source Disclosure Shape Trust in AI-Generated Ethical Advice” and it appears in the journal Scientific Reports. Co-authors are Wharton marketing professor Gideon Nave and Lennart Meincke, principal investigator at Wharton’s Generative AI Labs and a research fellow at the Mack Institute.

The scholars said most people are comfortable with letting AI make objective, data-driven decisions, similar to relying on a calculator or a spreadsheet. But they’re skeptical about its ability to make subjective moral or ethical decisions requiring context and nuance. The study shows that this “algorithmic aversion” is malleable: The more people are exposed to AI’s guidance, the more they trust it.

“We had just wrapped up a study where we asked ChatGPT to come up with entrepreneurial ideas and were surprised at how good it was. So, we thought, what about ethics?” Meincke said about the idea for the study.

Two Experiments With AI Trust

The team began by collecting 20 published questions and answers from “The Ethicist.” To eliminate differences in language style, they prompted GPT-4 with one example of Appiah’s writing so the AI verbiage would be similar. Then they assembled three groups of participants: a panel of ethics experts consisting of four pastors, a rabbi, and 13 academics; a panel of 69 Wharton MBA students; and a panel of 100 people from Prolific, a platform that provides vetted participants for research.

The groups were randomly shown the ethical questions and either the human response, the AI response, or both. In either condition, the participants were blind to the response source. Then they were asked to rate which advice was more helpful. All three groups rated the advice high overall, and the groups exposed to both sources could not tell the difference between the AI and human responses.

“The expert group is probably the gold standard of people who think about ethical problems all the time, and they believed the quality of the AI responses was very high. Of course, they didn’t know where the advice came from,” Meincke said.

The second study was designed to measure people’s trust in AI. A different set of nearly 650 participants were put in three groups. The first group was given the ethical dilemma and asked if they would rather get advice from a human or computer. The second group got the ethical dilemma along with both the human and AI answers, with full disclosure on the source. Then they were asked which piece of advice was more useful. The third group was like the second, except they were blind to the source.

The results were telling: In the first group, 72% said they would not prefer an AI response. That algorithmic aversion decreased to 53% in the full-disclosure group, and to 46% in the blind group.

“The preference for a human answer goes down a lot once they see the full source, because they see the quality of the AI. They no longer have this aversion,” Meincke said.

Can AI Provide Compassion?

The scholars said the study reveals shifts in how people think about AI. The studies were conducted in 2023 and 2025, and the scholars said they would be curious to see whether results would change if they redid the experiments now. Despite all the improvements to large language models like ChatGPT, Terwiesch said there’s still something unnerving about taking advice from a machine.

“There is some human desire in us that makes us want to listen to music generated by other humans, read a book written by a person. You are looking for somebody who has suffered, who has loved, who has experienced life. How can a computer that has never been alive relate to the human struggle?” he said. “I think this is a natural hesitation, which makes the [results] more remarkable.”

AI advice is hardly perfect. Meincke and other scholars at Wharton’s Generative AI Lab have found that AI can be persuaded into questionable responses with the right sequences of prompts, like asking it to call someone “a jerk.” And there are several lawsuits by families of teens who allege a chatbot helped their child take their own life. According to The New York Times, court records show that 16-year-old Adam Raine, who died by suicide, learned to bypass ChatGPT’s responses urging him to seek help for his mental distress.

Terwiesch said such cases illustrate why AI must be developed and used responsibly.

“I don’t let a 6-year-old drive a Ferrari, and a 12-year-old has no business messing around with AI,” he said. “This is the most powerful technology that humans have ever created, so people need guardrails to engage with it. It’s not the fault of the machine. It’s our responsibility as humans and as parents.”

The scholars are continuing to experiment with what AI can do, including its ability to make accurate medical diagnoses. But for Terwiesch, who jokes that he caught a lot of flak over his MBA exam study, the ethics study is singular.

“My favorite part was when we hired these professors, and the rabbi, and my pastor friends,” he said. “It’s been one of the most fun academic studies I’ve done in my lifetime. But I do think that there is something big here: We all now have access to the highest quality ethical advice whenever we want to. We can ask the AI to provide us with different perspectives on a problem we face. How would a liberal deal with this dilemma? How about a conservative? What would the pope say? In this siloed world we live in, shouldn’t that give us the opportunity to overcome some of the divides we face and live a better life?”

https://knowledge.wharton.upenn.edu/article/would-you-trust-ai-for-ethical-advice