Meta-research – the study of research practices and the "the science of science" – offers insights into how the field of Human-Computer Interaction (HCI) can refine its methodological frameworks, enhance rigor, and address its challenges. We believe CHI deserves a dedicated space for meta-research.
This 1-day workshop at CHI '25 establishes an open space for HCI scholars to explore and discuss meta-research in HCI. We are equally focused on the past, present, and future: what we study, how we document it, how we evaluate, and how we distribute our work, and its collateral effects, such as mounting career pressures. Together, we will work towards a research roadmap specifically for HCI meta-research. In the long term, we hope to see this workshop be the initial spark to establishing a permanent HCI meta-research community.
As the field of Human-Computer Interaction continues to evolve, it is becoming increasingly important to thematize the growing pain points in HCI research, not only in steering and working groups, but also in rigorous scientific publications. Self-reflective and meta-scientific contributions are critically important to advance the HCI field as a whole. Yet, little research and self-reflection is being published in HCI. It is time for the CHI community to pause and reflect on how things have progressed in the past decade and where we are headed. Meta-research is the study of research practices. Meta-research and the science of science offer valuable insights into how HCI as a field can refine its methodological frameworks, enhance rigor, and address its challenges. However, meta-research contributions on HCI currently have no dedicated venue at ACM conferences, and the CHI Conference currently has no fitting subcommittee for meta-scientific investigations. In the past, such investigations have been relegated to poster contributions or adjunct proceedings ("alt.chi") where the investigations might not get all the attention they deserve. This is suboptimal and also contrasts with the field’s critical and self-reflective tradition.
This 1-day workshop aims to bring together researchers, practitioners, and reviewers within the HCI community and beyond to explore and discuss meta-research. By critically evaluating the processes underlying HCI research, we can improve the ways in which studies are conducted, reported, and assessed, fostering a culture of transparency and methodological rigor. The workshop will provide a forum for researchers to share, discuss, and brainstorm their ideas about improving the current state of meta-research in HCI. We aim to provide a common platform for meta-research in HCI, and to establish a research agenda for meta-scientific investigations in HCI.
This workshop is for anyone in the HCI community who cares about understanding, cataloging, and improving the ways we conduct research. By taking a step back and looking critically at how we design, conduct, and share our work, we can identify areas where we can improve. The goal is to create a space where we can safely and openly discuss the challenges of HCI research and develop practical ways to ensure that HCI research is as rigorous, transparent, and impactful as possible.
This workshop is intended to mark the starting point of a meta-research community in HCI that outlasts the workshop event. The workshop will be a catalyst for change in the CHI and HCI community.
The workshop topics include, but are by no means limited to, the following topics:
HCI research has recently shifted to using and focusing on large language models (LLMs) and generative systems. Community members have voiced concerns about the disproportionate number of HCI works that thematize or use generative AI, the lack of methodological frameworks, and prompt-hacking (akin to p-hacking). Going forward, it makes sense to discuss these issues, and find constructive ways of using LLMs ethically and responsibly.
The "replication crisis" in HCI has been identified but not fully addressed. Expectations of the field also have seen an increase in recent years. For instance, the number of references included in a CHI paper has increased with each year since CHI '16, and most likely this is now an implicit expectation by reviewers. It is also fascinating to understand the citation behaviors of the community.
Peer review faces a number of recognized limitations. We are hearing the HCI community's signals of how, for instance, generative AI is used, should be used, or should not be used in peer review. And the community feels the increased workload in review duties. There is a growing urgency to act on these issues.
The nature of what we evaluate has evolved, with submissions now including various adjunct materials (video figure, data sets, submission history, etc.) that may influence the peer review process in many subjective ways. It is easy to envision how the current format is not the "final optimal" way of documenting research. In addition to what we publish, there is also the consideration of how much we publish.
Science is evolving, and these changes also affect us HCI researchers. There are high incentives to publish always more and more. These mounting career pressures to publish also seem to affect the young academics disproportionately.
Rather than prescribing activities at the workshop, participants are invited to also bring their own topics to work on in the group sessions.
We aim to kick-off the event with a keynote (TBA). The first part of the workshop will then give room for participants to briefly present their submissions. Instead of prescribing activities, we provide a smorgasbord of activities and topics (see above) to work on in the second part of the workshop.
| Start | End | Program Schedule |
|---|---|---|
| 09:00 | 09:15 | Keynote |
| 09:15 | 09:45 | Welcome and introductions |
| 09:45 | 10:30 | Lightning presentation rounds |
| 10:30 | 11:00 | Coffee break |
| 11:00 | 11:45 | Lightning presentation rounds (continued) |
| 11:45 | 12:00 | Preparing afternoon sessions |
| 12:00 | 13:00 | Lunch |
| 13:00 | 14:30 | Group work sessions |
| 14:30 | 15:00 | Break |
| 15:00 | 16:30 | Group work sessions |
| 16:30 | 17:00 | Closing |
The Meta-HCI workshop invites researchers and practitioners to discuss meta-research in HCI. Meta-research focuses on studying research practices and offers insights into how HCI can enhance its methodological frameworks, improve rigor, and address the field’s growing challenges.
This 1-day workshop will serve as a platform for the HCI community to share thoughts and experiences with meta-research, and collectively examine the processes that shape the research in our field. The workshop themes include, but are not limited to, what we study and publish in HCI, how we conduct and evaluate research in HCI, how HCI research affects us, and how the broader context of science affects HCI research.
Participants are invited to submit short papers (5–6 pages, references excluded) or full papers (min. 6 pages, references excluded) in the form of studies, experiments, bibliometric and scientometric investigations, or other meta-scientific research. Since meta-research is an emerging area in HCI, we also accept short position papers (2–5 pages, references excluded). Submissions should use the CEURART 1-column template. The workshop organizers will review submissions primarily based on their potential to stimulate engaging discussions. At least one author of each accepted paper must register for the workshop and attend the event.
Manuscripts should be submitted as email attachments in pdf format by March 14, 2025 to the workshop co-chairs at meta-hci@oulu.fi.
We look forward to your contributions and to welcoming you to an exciting and productive discussion on the future of HCI research practices!