Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

Research as Craft, Coding as Experiment: Testing Language Models in the MediCluster Project

Qualitative research does not finish with interviews. Once conversations and observations have been conducted, transcribed, and uploaded to a QDA program (in our case, Atlas.ti), another demanding phase begins: coding, classification, and interpretation (in our project, we do this using Grounded Theory principles). In qualitative research, this often means reading, comparing, coding, recoding, and discussing material over long periods. As our interview corpus in the MediCluster project expands, now comprising more than 90 interviews across several sites and contexts, the question of how to work through this growing body of data in a careful and methodologically meaningful way has become increasingly pressing.

This blog post examines a methodological experiment within the MediCluster project: the testing of large language models (LLMs) in qualitative data coding. Its aim is not to advocate technological adoption, but to reflect on what becomes visible, altered, or contested when such systems are introduced into an interpretive and collaborative research practice. In this sense, the contribution treats AI not only as a possible tool, but also as an object of methodological and epistemic inquiry.

These issues became particularly relevant after our field trips to Samsun and Tuttlingen. With a substantial number of interviews (more than 60) already collected and coding well underway, we began to reflect not only on what our data revealed but also on the tools and procedures for analyzing this material. In this context, a closed workshop held on 5 March 2026 within the MediCluster project focused on one specific methodological experiment: the possible use of LLMs as a tool in qualitative data coding.

The aim of this workshop was not to definitively adopt or reject a new technology/method, but to explore it and improve our processes. Rather, we wanted to examine what happens when such systems enter a qualitative research process grounded in interpretive work, collaborative coding, and theoretically informed analysis. In other words, we approached LLMs not as ready-made solutions, but as objects of methodological inquiry in their own right.

An illustration of a laptop displaying coded qualitative data with colored labels such as themes and categories. Surrounding the laptop are documents, sticky notes, and icons, while a glowing digital brain on the right represents artificial intelligence assisting in analysis. The scene symbolizes human–AI collaboration in qualitative data processing.
From raw data to structured insight. This visual, generated with ChatGPT, represents the transformation of qualitative data into themes through human–AI collaboration, highlighting coding, interpretation, and pattern recognition as interconnected processes.

A new tool for qualitative data analysis

One of the fundamental stages of qualitative research is coding. The researcher reads interview transcripts or documents, identifies recurring themes, and classifies them under categories. This process is not simply a technical task; it is a practice of thinking that reflects the researcher’s theoretical approach, field experience, and interpretive skills.

In public discussions, AI tools are framed either as innovations that will transform research (Friese 2025) or as unreliable shortcuts that should be kept out of serious scholarship (Jowsey et al. 2025). In our project, we are interested in whether such tools can be meaningfully integrated into an ongoing qualitative workflow, particularly at a stage where categories have already been discussed, refined, and partially stabilized through collaborative interpretive work.

Why test such tools at all?

There are pragmatic reasons for doing so. Large qualitative projects generate substantial amounts of material, and coding is one of the most time-intensive phases of the research process. When dozens of interviews accumulate, it may be useful to explore whether computational tools can assist with certain analytical operations—for example, by applying a pre-existing codebook to a selected subset of material, identifying passages relevant to specific categories, or making patterns visible for researchers to examine more closely.

For our project, such a test is not meant to replace careful reading. It is about asking whether an LLM can function as a provisional analytical aid within a clearly defined framework.

At our workshop, we discussed this potential around the following questions:

  • (How) Can language models assist in qualitative data coding?
  • If used, what role should they play in the research process?
  • How can the researcher’s interpretive position be preserved in this process?
  • It is difficult to say that we found definitive answers to these questions. However, our experiments showed that this technology has equally promising and careful aspects.

The most obvious advantage of LLMs is their ability to quickly analyze large text collections. An initial screening process that could take researchers hours can be accomplished in minutes with these tools.

This might be especially useful for tasks such as:

  • pre-screening large datasets,
  • discovering possible themes,
  • testing a preliminary coding scheme,
  • noticing patterns that might escape the researcher’s eye,
  • processing large text corpora against a finalized codebook.

In this sense, LLMs can be thought of as assistants that speed up the analytical process rather than tools that replace the researcher.

However: being fast does not necessarily mean being accurate.

Coding is not classification alone

This matters because qualitative coding is never only a technical act of attaching labels to textual segments. It is a process of interpretation. Researchers code not only what is explicitly said, but also how it is said, in which context it emerges, and what kinds of silences, tensions, ambiguities, or implicit assumptions surround it, as well as how particular statements relate to broader field relations and theoretical concerns.

This is precisely where the limits of LLMs become visible. Such systems work through statistical pattern recognition. They can identify similarities, recurring expressions, and probable thematic clusters, but they lack field experience. They do not know the situated dynamics of an interview encounter. They cannot (yet) independently distinguish whether a phrase is central or incidental, ironic or literal, strategic or hesitant, socially embedded or analytically misleading. In this sense, they can generate useful approaches, but they don’t interpret the findings in the same way a qualitative researcher would.

Several risks follow from this…

  1. Flattening / Loss of context: statements that look similar on the surface may be assigned to the same category even when they arise from very different social or biographical situations.
  2. Overgeneralization: language models may smooth out ambiguity in favor of coherence.
  3. Mechanization of interpretation / subtle transformation of coding into an apparently neutral sorting exercise, when in fact qualitative analysis depends on theoretical sensitivity, empirical familiarity, and reflexive judgment.

An STS Perspective on Analytical Tools

From an STS perspective, this is hardly surprising. Research tools are never simply neutral instruments. They participate in shaping what becomes visible, what counts as a pattern, and how knowledge is produced. If language models are introduced into coding processes, they become part of the socio-technical arrangement through which analysis takes place.

This does not mean they determine results in a simple way. But it does mean that they are not external to epistemic practice. Their outputs depend on prompts, model architectures, parameter settings, segmentation decisions, and the prior structuring of the material. In that sense, LLM-supported coding is also a methodological intervention that requires reflection, documentation, and critical scrutiny.

One aspect we discussed in the workshop was the importance of analytical settings. In workflows based on LLMs, not only the prompt matters, but also the system’s overall configuration. Settings (such as parameters controlling consistency and variability of outputs) that influence how consistently or how openly a model responds can shape whether outputs are more stable and repeatable or more varied and exploratory. These parameters introduce an additional layer of variability, meaning that results depend not only on the data but also on the configuration chosen by the researcher. For research purposes, this raises  questions of transparency and traceability. If such systems are used in qualitative analysis, it is not enough to state that “AI was used.” Researchers need to document how it was used, the settings, the material, and the analytical purpose.

In a project such as MediCluster, this is especially relevant when working with interview excerpts and asking a language model to relate them to an existing coding structure. If the system is used, for example, to suggest thematic codes, identify recurring patterns, or connect new excerpts to already established categories, the results may differ depending on the analytical setup. In this sense, the outcome is formed not only by the empirical material itself, but also by the technical conditions under which the model is used.

This does not produce mechanical objectivity. But it can make AI-assisted analysis at least more explicit, more discussable, and more open to methodological evaluation.

Keeping interpretation with the researcher

For us, this is the key point. The interpretive responsibility must remain with the researcher. In our workshop discussions, we repeatedly returned to the fact that coding involves decisions that cannot be delegated without residue.

  • Is a category analytically meaningful?
  • Does a passage really fit the code assigned to it?
  • What remains unspoken but important?
  • Which contradictions matter?

These are not merely technical questions, but interpretive ones.

For this reason, we do not see LLMs as autonomous coders, but rather as at supervised assistants operating within a framework defined by researchers. This is also why our current test setup remains deliberately limited. Rather than submitting the entire corpus to an automated process, we want to begin with a smaller sample of five to ten interviews and compare the outputs versus a code structure that has already been collectively developed and manually applied in ATLAS.ti. This allows us to explore potentials and shortcomings in a controlled and reflexive way.

Such an approach also enables methodological comparison. Instead of treating AI-generated coding as inherently valid or invalid, we can ask more precise questions:

  • Where does the system reproduce our existing categories convincingly?
  • Where does it misread context?
  • Where does it identify relevant passages, we might have overlooked?
  • And where does it produce a false sense of coherence?

Methodological opportunity, ethical caution

Any attempt to use LLMs in research also raises ethical and infrastructural questions. Data security is one of them. Another concerns the status of model-generated labels: what kind of analytical authority, if any, should they carry? And how should such uses be reported in publications, project documentation, or methodological reflections?

These questions are not secondary. They are part of what responsible experimentation requires. As a research team, we consider transparency essential. If AI-based tools enter the research process, their role needs to be clearly stated—not only to ensure accountability, but also to enable collective discussion about the changing conditions of academic knowledge production.

Research remains a craft

One of the clearest conclusions from our workshop was that qualitative research cannot be fully automated. This is not because researchers should resist technical change on principle, but because interpretation remains a situated, relational, and often painstaking practice. Coding is not simply about processing text; it is about crafting analytical meaning in dialogue with theory, data, and field experience.

In that sense, research is still a craft. New tools may reshape this craft, accelerate certain operations, or open up new possibilities for comparison and scale. But they do not eliminate the need for judgment, reflexivity, and responsibility.

For the MediCluster project, testing LLMs in coding is therefore about experimentally probing how such tools might be incorporated into qualitative practice without losing sight of what makes this practice qualitative in the first place. The question is not whether AI will replace interpretation. The more productive question is how researchers can engage such systems critically, carefully, and on their own methodological terms.

That, for us, is where the real experiment begins.

References

Friese, Susanne, From Coding to Conversation: A New Methodological Framework for AI-Assisted Qualitative Analysis (April 27, 2025). Available at SSRN: https://ssrn.com/abstract=5232579 or http://dx.doi.org/10.2139/ssrn.5232579

Jowsey, T., Braun, V., Clarke, V., Lupton, D., & Fine, M. (2025). We reject the use of generative artificial intelligence for reflexive qualitative research. Qualitative Inquiry, 10778004251401851.