1. User Studies and Usability Assessment
2. The Importance of User Studies in the Humanities
3. Previous Studies of Humanities Users

 

A key concern for those interested in the representation and re-presentation of texts in any form has been whether content exists independently and abstractly of its representation by an interface, or whether it exists only concretely, as the sum of its instantiations. One of the problems evident in the field today is the document-mindedness of ideas inherited from (mostly literary) hypertext theorists such as George Landow, who speak of content as links and lexias but almost never as objects, classes, and instances. On the other hand, the field also contains many examples of idealistic approaches to content. One example is the Text Encoding Initiative (TEI), an international project developing guidelines for encoding machine-readable electronic texts, with the aim of facilitating activities such as text analysis and sophisticated searching. While the TEI is well suited to encoding print objects (for example, a novel), it becomes confused when dealing with ontologically complex texts like plays, which are neither completely documents nor events. As F.W. Bateson famously phrased the problem, “if the Mona Lisa is in the Louvre, where [is] Hamlet?” (qtd. in Greetham, p. 342) And if an edition of Hamlet is an interface between the readership and the content, where then does the content begin?

 

1. User Studies and Usability Assessment

The wide adoption of “use” as a descriptor for engagement with hypermedia reflects the challenges inherent in understanding and facilitating interaction with complex, multimedia artifacts. It also points to a potential problem with research in this area: when we attempt to accommodate the digital artifact’s complexity by devising terms like “use,” which synthesize the range of processes involved in human-computer interaction, does that deter us from realizing the distinctiveness of those processes?

One component of our research will entail close examination of how literary structures and reading processes are extended and modified in digital environments. As many theorists have observed, a close look at modern print and electronic texts reveals a shift away from conventional narrative logic, and toward indeterminacy, fragmentation, and open-endedness (e.g., Landow, 1997; Bolter, 2001; Van Peer and Chatman, 2001). At the same time, current theories concerning how readers engage with texts tend to be derived from studies of readers working with “normal” prose or other conventional narratives (e.g., Chatman, 1978; Rabinowitz, 1987; Kintcsh, 1988; Zwaan, Magliano, & Graesser, 1995).

Many contemporary models of reading are built on the premise that the act of reading relies in a large part on distinguishing between significant and insignificant narrative details, so that we may generate a workable mental model of the situation described in the text. The Construction-Integration Model proposed by Kintsch (1988), for example, describes a process whereby readers construct meaning by identifying potentially relevant elements; they then develop an integration system in which appropriate elements are strengthened and inappropriate elements are weakened or discarded. Chatman (1978, p. 53 ff.) likewise proposes that readers of fiction distinguish kernels (major events) from satellites (minor events). And Rabinowitz (1987) suggests that in making sense of narrative readers follow rules of notice, which he describes as a process of identifying more significant details and separating them from less significant details.

Miall and Kuiken (1994) have critiqued models such as this, particularly Kintsch’s. They point out that such models fail to take into account the extent to which readers’ experiences of literary texts are modified by their emotional response to stylistically emphasized language, such as metaphor and alliteration, which engages the reader’s feelings and evokes “less prototypic, more personal meanings” (p. 339). According to Miall and Kuiken, understanding literary response requires a different mode of analysis from the one implicit in text theories that have been developed based on studies of “normal” (i.e., informational) prose. These models, they note, generally describe “a resource-limited system in which cognitive structures (e.g., story grammars) or procedures (e.g., integrating processes) economize comprehension by deleting irrelevant propositions, inferring relevant propositions, and building macropropositions” (p. 344). In other words, they focus on how comprehension is facilitated or economized. In this respect, Miall and Kuiken argue that the current models are too limited for the purpose of understanding response to literature, because the essence of literary text dwells at least in part in its stylistic features, and these features are less likely to economies comprehension than to complicate it “by challenging the familiar, prototypic concepts that readers initially apply to the text” (p. 344). Even theories of the reading process based on studies of readers’ engagement with literature, such as Chatman’s and Rabinowitz’s, tend to be biased toward a focus on economizing comprehension. Rabinowitz’s notion that readers engage in a process of distinguishing significant elements from less significant elements, for example, clearly privileges plot over other features of narrative, presuming that anything in the text not immediately relevant to developing the situation is marginal to understanding the text.

This view of literary reading is problematic, particularly when we consider that much contemporary fiction, both print and digital, is increasingly complex and fails to conform to conventional expectations respecting narrative logic. Van Peer and Chatman (2001) observe that the diverse narratives of the twentieth and twenty-first centuries are incompatible with existing narrative models. This is in part because most of these models reflect a western perspective, and in part because the models do not take into account new media genres. To develop appropriate models of reading complex print or digital narratives, we must examine how people engage these texts, and revise our perspectives of narrative structure and literary reading processes.

In discussing empirical research related to how people engage books and e-books, then, we envisage two scenarios: 1) venues in which economizing comprehension is the aim; and 2) venues in which engagement with an aesthetic artifact is the aim.

With respect to the first, information-seeking in the humanities is a well-researched area (e.g., Dalton and Charnigo, 2004). Nevertheless, our understanding of how humanities scholars engage computer-based information resources is in continual need of refinement as new resources are developed. An overview of research in this area is provided below.

With respect to the second scenario mentioned above, models of reading based on observing readers of complex print narratives, or emerging hypermedia genres, do not yet exist. Further, Douglas (2000) has questioned whether there are even a dozen “studies or considerations of how hypertext may transform the way we read or write texts, and, indeed, our whole conception of a satisfactory reading experience” (p. 73) In this assessment, Douglas disregards empirical studies of informational hypertext from the perspectives of interface design and software engineering. But her point is well taken, and is still valid in spite of the intervening years since she first made this observation: there exist few examinations of how reading processes may change when readers interact with complex digital genres such as hyperfiction. Members of our research team have worked to fill this significant lacuna in the literature (e.g., Dobson & Miall, 1998; Miall & Dobson, 2001; Dobson & Luce-Kapler, 2005; Luce-Kapler and Dobson, 2005; Dobson, 2006), but much research remains to be done, and appropriate models of reading are still in need of development.

2. The Importance of User Studies in the Humanities

The technical world has been slow realize that users matter, not just in the field of digital humanities, but in broader areas of system design. As long ago as 1971 Hansen called for software engineers to know their user (Hansen, 1971) and for the last thirty years the advantages of software projects whose systems are designed with an eye to the user have been well documented (Shneiderman & Plaisant 2005). Yet some are still inclined to assume that users might not know what they want, and thus it is better not to ask them, in case they answer the wrong question.

If users are integrated into system design, this may still happen late in the process. Typically the user is presented with a prototype that — the designers hope — she will like. This can be of limited use, since at this late stage system designers may be unwilling to make significant alterations. Thus, user input can only have a relatively minor impact, since a radical redesign will cost time, money and enthusiasm, all of which are often in short supply. It is much easier to make minor adjustments, and get the system into production. In scenarios of this sort, if users lose their enthusiasm for the product, it is they and not the designers who tend to be blamed. As a result, the failure rate of software in the commercial world is still staggeringly high (Dalcher & Genus, 2003; Flowers, 1996).

In the humanities, scholars have too often been branded as digitally unskilled or even backward looking, because they have been slower to adopt digital tools than scientists (Warwick, 2004). However, this is a fundamentally flawed approach: when technologies fit well with what scholars do, they will use those technologies (Bates, 2002). Several recent studies of humanities users and digital resources in the UK have found many humanities scholars reporting that they are enthusiastic users of digital resources. However, what they define as digital resources tend overwhelmingly to be generic informational resources, such as library and archive websites, or large online reference collections such as the e-DNB or Literature Online, rather than the kind of digital object which might be compared to a scholarly book (Warwick et al, forthcoming). At present general information resources are better suited to researchers’ needs. If we would like future electronic books to be used, they must be equally fit for the purpose. To produce such a resource we must understand what users do, what they like, and what they might like in future.

3. Previous Studies of Humanities Users

Scholarly inquiry employing digital resources in the humanities is a well-researched area. As Dalton and Charnigo (2004) show, in recent years there has been a flood of literature about scholars’ information needs and seeking behaviours. Although useful recent work on humanities scholars has been done by Green (2000), Talja and Maula (2003) and Ellis and Oldman (2005), much of the literature tends to conflate information seeking and information needs in relation to humanities scholars.

The earliest work on humanities users was on their information needs and patterns of use, and it is only very recently that research has been conducted into their actual behaviour in digital environments. Seminal work done by Stone (1982) and later by Watson-Boone (1994) showed that humanities users need a wide range of resources, in terms of their age and type. This is still true in a digital environment, where humanities users continue to need printed materials and manuscripts, the latter implying older materials than those generally used by scientists (British Academy, 2005). Humanities scholars also rely on face-to-face information gathering, from colleagues and at conferences. They may also use personal collections of knowledge built up over years of study. They do not necessarily expect to create new data or discover new facts, but reinterpret and re-express ideas, where the expression itself is as important as the discovery (Barrett, 2005).

A major theme of the literature about humanities users is that they are not like those in the sciences or social sciences, although many systems designers of electronic resources have assumed that they are (Bates, 2002). Humanities scholars are much more likely to use what Ellis has called “chaining”; that is, they proceed by following references that they have found in other literature (Ellis & Oldman, 2005; Green, 2000). Despite the hypertextual nature of the web, however, such activity is seldom well supported in online environments (Bates, 2002). It is also at odds with keyword queries that tend to be the norm for information systems. Oddly, this has been seen as evidence that humanities researchers’ techniques are somehow impoverished (Chu, 1999). Yet as long ago as the mid-1980s Wiberley showed that humanities scholars constructed searches using well defined terms, but these terms were different from those used by scientists: for example, humanities searches were more likely to include names of places or people (Wiberley, 1983 & 1988). Bates’ work and that of Dalton and Charnigo (2004) and Whitmire (2002) has also shown that those humanities scholars who use digital resources tend to be demanding of their quality, and are capable of constructing complex search strategies, given appropriate training. Lehmann and Renfro (1991) and Wiberley (2000) suggest that humanities scholars are receptive to technology as long as it demonstrates adequate savings in time or effort.

If we are to design an e-book that users may regard as fit for their research purposes, then it follows that we must understand what humanities scholars do in digital environments, what kind of resources they need, and even more importantly perhaps what makes humanities scholars decline to use resources. These questions are not well understood: except for the work of Bates et al., on the Getty project (Siegfried et al., 1993), very few people have studied what humanities scholars do when they carry out research in an online environment. There is even less concern with what they are doing offline and how connections may be made between the two. Some of these questions are being addressed by researchers involved in the HCI-Book project. However, a great deal more research remains to be done on such areas before we may be confident of designing an e-book that will be of genuine utility to the humanities researcher.

In thinking about how best to conduct a study of humanities users, we have benefited from the experience of two ongoing, UK-based projects. Both of them list Claire Warwick of our research team as a principal or co-investigator.

The UCIS project (User Centred Information Search in context) looks at the ways in which users interact with digital libraries. UCIS is studying humanities users and their interactions with information in physical and virtual environments. An important facet of this work is that users are studied in as naturalistic a context as possible, so as to gain a fuller understanding of the nature of their information work (Warwick et al., 2005). Humanities academics have been interviewed and observed undertaking their usual research in both physical and digital libraries. The knowledge gained from this is being used to derive requirements to the construction of alternative interfaces to the Greenstone digital library system.

The LAIRAH project conducts research into levels of use of digital resources in the arts and humanities in the UK. The project ran workshops in which it investigated user reactions to different digital resources, both widely used and neglected.

Both LARIAH and UCIS have found that humanities researchers have very sophisticated information skills, and complex mental models of their physical information environment, although they find these much more difficult to apply to the digital domain (Makri et al., forthcoming). They are very aware of the affordances, advantages, and problems of various information technologies. Scholars are concerned about accuracy and ease of use for both physical and electronic resources, and require as much information as possible about how such resources are constructed and what they contain (Warwick et al., 2006).

LARIAH research in particular found that users expect very high standards of resource content and interfaces, and are easily deterred from using digital resources. Anything that makes a resource difficult to understand — a confusing name, a challenging interface, data that must be downloaded — will deter them from using a resource. Humanities researchers also find that the pleasure of interacting with the materials that they read, whether in physical or digital space, is an important element of scholarship (Blandford et al. forthcoming). We therefore face a challenge of designing an e-book that is not only functional and appropriate for its scholarly purpose, but that, if possible, is pleasant to use, and at all costs does nothing to deter schools from using it.

We are also aware that many of those who have been surveyed about their use of digital resources are, to some extent, enthusiasts for the medium. The research for both UCIS and LAIRAH suggests that a significant number of scholars do not make significant use of specialist digital resources (beyond the generic information resources described above). It is therefore important that our research aim to survey such light users and non-users to determine what factors might deter them from using an e-book. This will allow us to gain the fullest possible picture of the user community, not simply the early adopting enthusiasts who are likely to volunteer for studies on digital resources.

In addition to the above-mentioned projects, two major Canadian undertakings have influenced our thinking. Both TAPoR (Text Analysis Portal for Research) and ECO (Early Canadiana Online/Notre Memoire En Ligne) have investigated how humanities researchers use online tools, whether text-specific or not. Two major findings emerged. First, scholars are most concerned with accomplishing their task; the interface must present things in the context of tasks that they might wish to accomplish, rather than of the tools that might do the job, or the technical details of the text that they might be using. Secondly, graduate students and more senior scholars behave differently and have different needs. This highlights the need for the HCI-Book project to identify different categories of users, in order that we meet their needs; these categories may be more fine-grained than we expect.