Towards pragmatic gestures: From repetition to construction in multimodal pragmatics

Pragmatics is concerned with the relation of linguistic items to their context of utterance, their users and interpreters, that is, with context-dependent aspects of meaning. Studies in pragmatics also recognize that non-verbal communication mechanisms (including gestures), alongside verbal aspects of communication, are at the core of the creation of meaning in the interaction context (Payrató, 2009: 175). Given this context-sensitive and multimodal view of pragmatics, ‘pragmatic gestures’ will be defined here as formally heterogeneous, multifunctional, (mostly) non-representational and (often) unintentional visible actions in language interaction, which express “aspects of utterance structure, including the status of discourse segments with respect to one another, and the character of the ‘speech act’ or interactional move of the utterance” (Kendon, 1995: 247). The aim of the present talk is to answer the following question: To what extent non-representational gestures with a pragmatic function are conventionalized? The hypothesis is that pragmatic gestures can be seen as multimodal constructions insofar as they consist in learned pairings of form with discourse function (Goldberg 2006: 5). As it has been shown for contructionalized pragmatic markers in speech (Bolly, 2014; Travis & Torres Cacoullos, 2014), I suggest to tackle pragmatic gesturing in terms of continua from idiosyncratic uses to more conventionalized ones. The focus will thus be on the conventionalization process of pragmatic units, highlighting the particular role of repetition (Bybee, 2006) in the gradual process of fixation of multimodal constructions in the linguistic system. The multimodal approach explores the function of pragmatic gestures in audio-video data taken from the CorpAGEst corpus, which is comprised of 18 semi-directed, face-to-face interviews (9 subjects; 16.8 hrs; approx. 250,000 words). Adhering to some extent to the categorization of visible actions in dialogues proposed by Bavelas et al. (1995), three types of pragmatic gestures are distinguished: self- and object-oriented actions (viz. adaptors), stereotypical signals (viz. emblems), and conversational gestures (viz. topic and interactive gestures). In line with form-based approaches to gesture (Müller, Bressem & Ladewig 2013), relations that exist between speech and nonverbal resources (including hand gestures, facial displays, gaze, head, trunk and shoulders) are also investigated. Particular attention is paid to clusters and recurrent combinations of (non)verbal parameters (e.g., gaze direction, head shake, hand location in the subjective space, occurrence of discourse marker, etc.). Preliminary results from a study of emotional and attitudinal states in the healthy elderly people (Bolly, to appear) indicate that the use of nonverbal resources is highly idiosyncratic: for instance, no clear physiological pattern seems to be emotion-specific. Yet, some regularity may still be noticed: for instance, ‘surprise’ is mainly expressed by means of eyebrow raising, often combined with an exaggerated opening of the eyes. It is worth pointing out that language variation and idiosyncratic uses are viewed here as constituting the central object of research, as being “the entering wedge for discovering the invariant, the system viewed as a living entity, an entity which takes shape and evolves through use, through the speakers as members of a group sharing a culture and a vision of the world” (Cuenca, 2003: 7).