DFG project ‘Moral Hallucinations’ investigates moral errors in large language models

An interdisciplinary team led by ethicist Professor Karoline Reinhardt and computational linguist Professor Annette Hautli-Janisz from the University of Passau is investigating a phenomenon in AI hallucinations that has received little attention to date: when large language models produce moral fallacies that deviate significantly from human values.

More and more people are using large language models (LLMs) such as GPT, PaLM or Llama – not just for simple information queries, but also for advice on practical matters. It is not uncommon for so-called hallucinations to occur when using such systems. This means that the LLMs provide answers that, whilst appearing realistic, are factually incorrect or misleading.

An interdisciplinary team led by ethicist Professor Karoline Reinhardt and computational linguist Professor Annette Hautli-Janisz is conducting research into a previously little-noticed type of such fabricated AI responses as part of the DFG project ‘Moral Hallucinations in Large Language Models’. The project focuses on moral hallucinations that can arise when people use LLMs for ethically relevant questions. For example, when they ask the machine whether it is always a moral duty to keep a promise.

“Moral hallucinations thus go beyond conventional AI hallucinations. They involve not only erroneous or unreliable information, but also distortions in moral argumentation structures,” explains Professor Reinhardt, holder of the Chair of Applied Ethics at the University of Passau. “And this often occurs in situations where people are vulnerable, seeking advice, or at a loss.”

The structure and consequences of moral AI hallucinations

This interdisciplinary research project at the University of Passau combines conceptual and empirical methods from applied AI ethics and computational linguistics with the aim of theoretically analysing, ethically evaluating and automatically identifying moral hallucinations generated by large language models (LLMs). To this end, the research team is investigating questions concerning both the structure of moral hallucinations and their ethical consequences. For example: What are the key characteristics of moral hallucinations? What ethical consequences arise if moral claims made by LLMs are not merely biased but are, in fact, hallucinations? What are the argumentative structures of moral hallucinations, and how do they relate to moral judgement in philosophical theories? What are the ethical consequences of using LLMs as moral advisors? How could LLMs be designed to flag moral hallucinations?

“There is an urgent need to expand existing benchmarks and evaluate models in greater detail, particularly in the field of moral reasoning,” says Professor Hautli-Janisz, holder of the Junior Professorship in Computational Rhetoric and Natural Language Processing. She adds that this is also an area that has received little attention in computational linguistics to date.

The project is funded by the German Research Foundation (DFG) for a period of three years as part of the DFG Priority Programme “Robust Assessment & Safe Applicability of Language Modelling: Foundations for a New Field of Language Science & Technology (LaSTing)” (SPP 2556). This is a funding programme launched in 2026 that brings together interdisciplinary research at the interface of linguistics, computer science, AI and philosophy, amongst other fields.