EthosBot

MSc thesis · Vrije Universiteit Amsterdam & University of Amsterdam · 2025

EthosBot investigates whether pedagogical conversational agents can teach not just facts but higher-order skills. The study built a two-phase chatbot that first delivers structured learning modules on LLM fine-tuning, then guides students through Socratic-style discussions of real-world ethical dilemmas, and measured its effect on both information retention and moral judgment.

Research questions

1. Can pedagogical conversational agents improve students' ethical reasoning compared to traditional instructional methods?
2. Do chatbot-mediated interactions lead to greater information retention than static materials?
3. How effective is the Socratic questioning framework in developing students' moral judgment and decision-making skills?

Methodology

Nineteen CS/AI students were randomly assigned to a chatbot group (n = 10) or a static-PDF control group (n = 9). All participants completed a pre-test fine-tuning quiz and the Moral Judgment Test. The chatbot group worked through the two-phase agent (structured fine-tuning modules followed by Socratic ethical discussions), while the control group read identical content as a PDF and completed a worksheet. Afterwards everyone took an advanced fine-tuning quiz and a second MJT. Quiz gains and MJT changes were evaluated with Welch's t-tests and mixed 2×2 ANOVAs.

Implementation

Conversation history is stored in MongoDB, with each message embedded into a vector collection. On each turn the system retrieves the top-k similar embeddings as conversational context, and a prompt builder injects phase-specific templates, curriculum content, and the retrieved snippets. The LLM response is semantically evaluated and keyword-checked to unlock the next module, and a persistent flag in the user record switches the agent from structured instruction to Socratic discussion once all modules are complete.

Results

Participants using the chatbot showed greater improvements than controls in both information retention and ethical reasoning.

Conclusion

Students using the two-phase chatbot showed greater short-term gains in both technical knowledge and moral reasoning than those using static materials. The combination of formative checks and Socratic dialogue appears to foster active learning and more consistent ethical reflection. While modest in scope, the results suggest conversational agents can meaningfully support both factual and higher-order learning outcomes.

Future work

A delayed post-test to assess long-term retention of both technical knowledge and ethical reasoning, and replication with a larger, more diverse participant pool to strengthen confidence in the findings.