Subtopic Deep Dive

← Language, Discourse, Communication Strategies

Multimodal Conversation Analysis
Research Guide

What is Multimodal Conversation Analysis?

Multimodal Conversation Analysis examines how verbal conduct integrates with gesture, gaze, body orientation, and other embodied resources to organize interaction in video-recorded settings.

Researchers apply Conversation Analysis (CA) methods to capture turn-taking, recipient design, and semiotic diversity beyond speech alone. Key studies include Kendon's analysis of pictorial and action modes in utterance production (Kendon, 2014, 248 citations) and de Ruiter's projection of turn ends via multimodal cues (de Ruiter et al., 2006, 524 citations). Over 200 papers explore these embodied mechanisms since 2006.

Curated Papers

Key Challenges

Why It Matters

Multimodal CA reveals how non-verbal cues shape social actions like recruitment of assistance (Kendrick and Drew, 2016, 390 citations) and emotional influence in interaction (van Kleef and Côté, 2021, 342 citations). Applications include improving AI dialogue systems by modeling gaze and gesture in turn projection (Levinson, 2015, 697 citations) and analyzing cross-cultural communication where semiotic diversity affects utterance design (Kendon, 2014). It informs education by linking language to embodied culture (Kramsch, 2014, 1308 citations).

Key Research Challenges

Multimodal Transcription Complexity

Transcribing synchronized verbal, gestural, and gaze data requires detailed notation systems beyond Jefferson Transcript. Kendon (2014) notes pictorial modes challenge spoken language models. Automation lags for video analysis (de Ruiter et al., 2006).

Turn Projection Mechanisms

Debate persists on whether lexicosyntactic or prosodic-gestural cues primarily signal turn ends. Levinson (2015) argues cognitive origins integrate multimodality. Empirical verification needs large video corpora (de Ruiter et al., 2006).

Cross-Cultural Semiotic Variation

Embodied practices vary across languages, complicating universal models. Sidnell and Enfield (2012, 202 citations) link diversity to social action. Kendon (2014) highlights utterance production differences.

Essential Papers

Language and Culture

Claire Kramsch · 2014 · AILA Review · 1.3K citations

This paper surveys the research methods and approaches used in the multidisciplinary field of applied language studies or language education over the last fourty years. Drawing on insights gained i...

Turn-taking in Human Communication – Origins and Implications for Language Processing

Stephen C. Levinson · 2015 · Trends in Cognitive Sciences · 697 citations

Projecting the End of a Speaker's Turn: A Cognitive Cornerstone of Conversation

Jan-Peter de Ruiter, Holger Mitterer, N. J. Enfield · 2006 · Language · 524 citations

A key mechanism in the organization of turns at talk in conversation is the ability to anticipate or PROJECT the moment of completion of a current speaker's turn. Some authors suggest that this is ...

Recruitment: Offers, Requests, and the Organization of Assistance in Interaction

Kobin H. Kendrick, Paul Drew · 2016 · Research on Language and Social Interaction · 390 citations

In this article, we examine methods that participants use to resolve troubles in the realization of practical courses of action. The concept of recruitment is developed to encompass the linguistic ...

The Social Effects of Emotions

Gerben A. van Kleef, Stéphane Côté · 2021 · Annual Review of Psychology · 342 citations

We review the burgeoning literature on the social effects of emotions, documenting the impact of emotional expressions on observers’ affect, cognition, and behavior. We find convergent evidence tha...

An alternative model and ideology of communication for an alternative to politeness theory

Robert B. Arundale · 2015 · Pragmatics Quarterly Publication of the International Pragmatics Association (IPrA) · 324 citations

Preview this article: An alternative model and ideology of communication for an alternative to politeness theory, Page 1 of 1 < Previous page | Next page > /docserver/preview/fulltext/prag.9.1.07ar...

Mock impoliteness, jocular mockery and jocular abuse in Australian and British English

Michael Haugh, Derek Bousfield · 2012 · Journal of Pragmatics · 301 citations

Reading Guide

Foundational Papers

Start with de Ruiter et al. (2006) for turn projection mechanisms and Kendon (2014) for semiotic integration in utterances, as they establish multimodal expansions of CA.

Recent Advances

Study Levinson (2015) for cognitive implications and Kendrick and Drew (2016) for assistance organization, capturing post-2015 embodied interaction advances.

Core Methods

Jefferson Transcript for talk; ELAN/ANVIL for multimodal annotation; projection analysis via gap anticipation and body torque (de Ruiter et al., 2006; Kendon, 2014).

How PapersFlow Helps You Research Multimodal Conversation Analysis

Discover & Search

Research Agent uses citationGraph on de Ruiter et al. (2006) to map turn projection papers, exaSearch for 'multimodal turn-taking gaze gesture', and findSimilarPapers to uncover 50+ related works like Kendrick and Drew (2016).

Analyze & Verify

Analysis Agent applies readPaperContent to Kendon (2014), runPythonAnalysis for timing gaze-turn correlations via pandas on extracted data, and verifyResponse (CoVe) with GRADE scoring to validate multimodal cue claims against Levinson (2015).

Synthesize & Write

Synthesis Agent detects gaps in cross-cultural multimodal CA using contradiction flagging on Sidnell and Enfield (2012), then Writing Agent employs latexEditText, latexSyncCitations for Kramsch (2014), and latexCompile for publication-ready manuscripts with exportMermaid diagrams of turn sequences.

Use Cases

"Extract timing data from multimodal turn papers for statistical analysis"

Research Agent → searchPapers 'turn projection multimodal' → Analysis Agent → readPaperContent (de Ruiter 2006) → runPythonAnalysis (pandas correlation of gesture onset and turn ends) → matplotlib plot of latencies.

"Draft LaTeX section on semiotic diversity in conversation with citations"

Research Agent → citationGraph (Kendon 2014) → Synthesis → gap detection → Writing Agent → latexEditText 'semiotic utterance production' → latexSyncCitations (Sidnell 2012) → latexCompile PDF.

"Find GitHub repos analyzing video CA datasets"

Research Agent → searchPapers 'multimodal conversation analysis dataset' → Code Discovery → paperExtractUrls → paperFindGithubRepo → githubRepoInspect → exportCsv of repo tools for gesture annotation.

Automated Workflows

Deep Research workflow scans 50+ papers via searchPapers on 'multimodal CA turn-taking', structures report with GRADE-verified sections on gaze integration (Levinson 2015). DeepScan applies 7-step CoVe chain to verify recruitment multimodality (Kendrick 2016). Theorizer generates hypotheses on gesture-verbal interplay from Kendon (2014) and de Ruiter (2006).

Try Doxa for Multimodal Conversation Analysis Research

Frequently Asked Questions

What defines Multimodal Conversation Analysis?

It analyzes integration of speech with gesture, gaze, and body orientation in video data to study embodied turn organization (Kendon, 2014).

What are core methods?

Video transcription combines Jeffersonian verbal notation with ELAN for gesture/gaze timing; projection relies on prosodic and bodily cues (de Ruiter et al., 2006).

What are key papers?

Foundational: de Ruiter et al. (2006, 524 citations) on turn projection; Kendon (2014, 248 citations) on semiotic diversity. Recent: Kendrick and Drew (2016, 390 citations) on embodied recruitment.

What open problems exist?

Cross-cultural generalizability of multimodal cues; scalable computational modeling of embodied projection (Levinson, 2015; Sidnell and Enfield, 2012).