The Evaluation of the Outside Matter in Dictionary Reviews

Reviewing dictionaries is part of the ongoing work in lexicography, and several lexicographers have discussed the process and guidelines for reviews published in academic journals. However, few have addressed the evaluation of the outside matter and, if so, only in a cursory way. This article examines the evaluation of the outside matter in reviews published in Lexikos with a view to proposing some general principles for reviewing outer texts in printed and electronic dictionaries. The study shows that reviewers define the review object differently, some excluding the outside matter altogether, and that the way in which the outside matter is assessed differs within and between reviews. It is proposed that the separate sections of dictionaries should not only be examined independently but that their relationship to each other should also be evaluated so as to represent faithfully the lexicographic elements, i.e. wordlist, front, middle and back matter, their organisation and presentation, as well as three underlying elements: the function(s), data types and structures of the dictionary. Focus on all these elements may result in dictionary reviews that are academically sound because they treat the dictionary as a true research object.


Introduction
Dictionary reviews published in academic journals play an important role in the development of lexicography. This is particularly true of international journals such as Lexikos, which contribute to lexicography by publishing a substantial number of reviews in each issue. A natural basis for such reviews is an understanding of the findings reported in scholarly writings on lexicography, including books and articles specifically dealing with dictionary criticism. A study of the literature on dictionary reviewing shows a tendency to focus on linguistic aspects in dictionary articles and to deal mainly with printed language dictionaries for learners, for example Steiner (1979), Tomaszczyk (1986), Osselton (1989), Ripfel (1989), Hartmann (1999), Chan and Taylor (2001) and Jackson (2002). These lexicographers discuss the process involved in reviewing dictionaries and suggest guidelines for reviews published in academic journals. However, few have addressed the evaluation of the outside matter and only in a cursory way. One consequence of this is that dictionaries are often evaluated on the basis of the data in the wordlists instead of being treated as complex units with many types of data found in the often numerous texts surrounding the wordlists. This should be compared with the findings reported in Wolf (1992: 335-338) showing that more than 50% of users read user guides, findings that indicate that outer texts are important to theoretical and practical lexicographers, reviewers and users. Dictionary reviews can be either scholarly or non-scholarly depending on the actors and media involved. This article will examine reviews of dictionaries in Lexikos, which is an academic journal devoted to lexicographic research areas reflecting different theoretical schools of thought for the benefit of lexicographic specialists. So dictionary reviewing will be regarded as essentially a lexicographic exercise which may relate to cultural, factual and linguistic aspects as the case may be. On the basis of selected reviews included in Lexikos from its first issue up to issue 18 (published in 2008), the way in which reviewers have treated the outside matter will be examined and discussed. Furthermore, suggestions will be made as to how reviewers may analyse and critically evaluate the outside matter in dictionaries in an attempt to produce better reviews that increase the academic quality of dictionary criticism. This will involve three topics: the identification of the research object for dictionary reviews and its placing in a lexicographic framework; the discussion of the objective of dictionary reviewing in academic settings; and the actual approach to reviewing the outside matter.

Dictionary reviews and Lexikos
Several researchers have discussed the practice of criticising dictionaries, suggesting various ways to increase the quality of reviews. Some of the first scholarly contributions were Ripfel (1989), the eight contributions in the thematic part of Lexicographica 9-1993 as well as the seven contributions in the thematic part of LexicoNordica 10-2003, these being more recently followed by a number of individual contributions, including Chan and Loong (1999), Chan and Taylor (2001), Nielsen (2009) and Swanepoel (2008). Lexikos is an international journal based on high-quality research as pointed out in the 'Editorial Objectives' (Lexikos 2008: xv), and this is reinforced by the statement in the 'Instructions to Authors' that, among others, reviews 'must […] meet the requirements of academic writing and are evaluated by the editors with this in mind' (Lexikos 2008: 466). This means that reviews are evaluated before publication to conform to a certain standard to ensure international research quality, i.e. to contribute to the development of the discipline concerned.
There is no doubt that reviews of dictionaries can contribute to the development of lexicography as a research discipline. The criticism of lexicographic publications, including dictionaries, is one of the fundamental practices of metalexicography. This does not mean, however, that all dictionary reviews can be characterised as 'scientific', 'scholarly' or 'academic', since many reviews are not written with the intention of contributing to the development of lexicography, for example most reviews published in newspapers and popular magazines. Their primary objective is to draw attention to a new publication and to provide information to consumers in general. This does not mean that these reviews cannot contribute to the development of lexicography but they are generally written and published by and for non-academic actors in nonacademic media. In contrast, scholarly dictionary reviews are potential contributors to the development of lexicography as a research discipline, they are written and read by scholars in the natural course of their research work, and they are published in academic journals.
Research is often regarded as something special, different from one's usual ordinary activities. The question then is: What makes scholarly reviews special? A possible answer is given by Phillips and Pugh (2005: 48) who argue that 'research goes beyond description and requires analysis'. If this is extended to lexicography, it implies that dictionary reviews should not merely be descriptive but should contain more or less thorough analyses, evaluations and reflections of their objects. This is supported by the following description in the 'Instructions to Authors': 'Reviews: An analysis and critical evaluation of published scientific sources and products, such as books and computer software' (Lexikos 2008: 466). In other words, scholarly reviews that contribute to lexicographic research are at a level above everyday reflection and contain analyses, evaluations and findings that can influence research conducted by others.
This has various implications for the actual work of reviewing. Firstly, reviewers must possess knowledge about the object of review and the reality in which it is rooted that is more extensive than the knowledge of ordinary people. Secondly, reviewers must explicitly use that knowledge in their reviews. Finally, it may be said that scholarly reviewers seek to achieve some sort of intersubjectivity by using generally accepted (lexicographic) methods that enable lexicographers and researchers from other disciplines to learn from the statements made in reviews and to let these statements affect their own research. In order to produce findings that support or contradict this position, it is first necessary to delimit the scope of the study.
Throughout its history, Lexikos has contained reviews of dictionaries. For practical reasons, however, it has been necessary to limit in the following ways the number of reviews that form part of the discussion. First, only reviews of dictionaries (as defined by their name, e.g. 'dictionary', 'woordeboek', 'lexicon', 'encyclopedia' and 'glossary') found under the heading 'Resensies/Reviews' are considered. In accordance with the 'Instructions to Authors', review articles are regarded as research articles, not as reviews (Lexikos 2008: 466). Second, the number of reviews is further limited to those written in English, as they contribute to the development of lexicography in the international academic community. This results in 40 dictionary reviews published in 13 issues. Even though some issues have no dictionary reviews or no reviews that fall within the definition above, the texts examined can be said to span the entire life of the journal. However, not every possible topic will be addressed in the discussion below. Dictionaries contain many individual textual components and data types. As pointed out by Bergenholtz, Tarp andWiegand (1999: 1763), the printed dictionary is a collection of text types or genres, and the most general distinction is that between the wordlist and the outside matter. This article will focus on the outside matter and how this has been evaluated in the 40 reviews, though reference will be made to the wordlist when it is relevant for the understanding of arguments put forward. The outside matter, alternatively called outer texts, comprises the front matter, the middle matter and the back matter, this distinction being relevant for any printed dictionary. A similar distinction can be applied to online dictionaries, as they may and often do consist of a number of text files linked to the database in which searches are made, for instance the Oxford English Dictionary Online (2009) and the Merriam-Webster Online Dictionary (2009), though it is difficult to allege that a particular section is placed before or after another section, while features such as pop-up windows with written or audiovisual data may be regarded as middle matter. How, then, do reviewers perceive their object of review with specific reference to the outside matter?

4.
A dictionary is more than just a dictionary An important aspect of scholarly reviewing is the ontological position of dictionary reviewers which is basically the answer to the following question: What is the nature of the object that is being analysed, examined and evaluated, i.e. the research object? The answer to this question is significant, for research is not value-free but value-laden. No matter how objectively they attempt to conduct their analyses and present their findings, reviewers will do so within a theoretical framework. This does not pose any problem as long as they do so in an academically sound way. The ontological position directly affects the way in which reviewers treat the dictionaries under review. The object of dictionary reviewing is obviously the dictionary, and the theoretical framework within which reviewers place the dictionary is directly or indirectly reflected in their reviews. A small quantitative study may illustrate this point. Out of the 40 reviews examined 25 evaluated outer texts, whereas in 15 cases there was no evaluation of outer texts but the focus was on what the wordlists contain. It is difficult to imagine that these 15 dictionaries had no outer texts, but apparently the outside matter had no place within the ontological positions of the reviewers. This leads to the conclusion that these 15 reviews do not evaluate dictionaries as collections of text types in their entirety but reflect a restricted view of the research object. The question is whether this complies with the objective of dictionary reviewing.
It is possible to identify more than one objective, or purpose, of dictionary reviewing in the existing literature. Lexicographers generally agree that reviews should describe, analyse and examine dictionaries (e.g. Ripfel 1989: 31; Bergenholtz and Mogensen 1993: 10), Swanepoel (2008: 209) mentioning that reviews have two major goals: (1) to assist readers in their decision-making in acquiring the best dictionaries for their usage needs by presenting them with a well-founded analysis of the positive and negative qualities of a dictionary/dictionaries under review, and (2) to assist lexicographers in optimizing the functionality of their dictionaries.
The above purposes are rather specific, because they deal with individual aspects of a wider picture. Moreover, it is necessary to list several purposes to give a meaningful explanation of the objective of dictionary reviewing, and the purposes selected depend on the individual giving the explanation in each case. One solution to this problem is to introduce what may be called an overriding objective of dictionary reviewing. It is provisionally proposed that the overriding objective is to give a faithful representation of the lexicographic elements presented by the dictionary under review. It is further suggested that it is only possible to attain a faithful representation if the depiction of a dictionary is complete when it takes the whole dictionary into account. This approach complies with generally accepted academic requirements, because the dictionary is no longer just a product to be used, but a true research object to be analysed, described and examined.
The introduction of the overriding objective has at least two consequences. First, it impacts directly on the ontological position of reviewers, as it requires them to treat the dictionary as a complex whole. A dictionary is a lexicographic tool that consists of a number of distinct sections whose common function is to provide answers to users so that they can solve particular types of problem. Second, reviewers who do not take the various dictionary sections into account risk that their reviews lack too much information which is useful to readers. For instance, after having read the review of a dictionary of public administration by Marais (1998), readers still do not know what the dictionary looks like, nor do they know which sections it contains. The text of a review of a bilingual dictionary by Nkabinde (1993) is better structured than that of Marais (1998) as it is divided into numbered paragraphs with appropriate headings. However, the review gives no indication of what the actual dictionary looks like and which sections it contains. Even if dictionaries contain a minimum of sections, this would be relevant information for readers, and reviewers should therefore comment on this minimalistic approach to dictionary making.
The 25 reviewers who address the outside matter indicate their ontological position in various ways. One way in which authors of scholarly articles reveal their theoretical framework is through the bibliography or reference section. A closer study shows that only 11 out of 25 reviews contain reference sections. One interpretation of this finding could be that 14 of these reviews do not comply with the requirement that academic texts should be in a different league than reviews published in non-academic media, which are often characterised by having no bibliographies or reference sections. Table 1 summarises the findings of a more detailed examination of the 11 reviews that contain bibliographies and divides the works listed into four broad categories. 1

Reviews with references to dictionaries
Reviews with references to other reviews Reviews with references to lexicographic literature Reviews with references to nonlexicographic literature 10 2 6 7 Table 1: Number of reviews dealing with the outside matter with references divided by category.
The numbers in Table 1 give rise to the following comments. First, 10 reviews list dictionaries, both the dictionaries under review and other dictionaries, but this reveals nothing about the ontological position of the reviewers. Second, it is interesting to see that 2 reviewers refer to other dictionary reviews; this seems to underline the importance of reviews in the academic community. Third, 6 reviewers refer to lexicographic literature (books and articles). In one respect this is somewhat disappointing, because dictionary reviewing falls within the discipline of lexicography. As dictionary reviewing is a lexicographic exercise it is not unreasonable to expect that reviewers explicitly state their theoretical point of view through literature belonging to the discipline, and therefore that all reviewers refer to lexicographic literature. Finally, 7 reviews refer to non-lexicographic literature (books and articles dealing with topics such as grammar, language teaching, musical works, plants and language history). Prima facie, this indicates that most reviewers have an approach to dictionary reviewing that is not based on lexicography.
In order to corroborate the above findings, the actual number of references by category is relevant for a quantitative analysis.  Two of the numbers in Table 2 warrant consideration. It is striking that the number of non-lexicographic literature is more than four times higher than the number of lexicographic references. This reinforces the impression from the numbers in Table 1 that the reviewers do not have a lexicographically based theoretical framework but rather one influenced by other disciplines. References to non-lexicographic literature are natural in dictionary reviews but it does not seem to fit in with the fact that reviewing dictionaries is an exercise in lexicography. From a research point of view, it is important that the texts produced within a discipline are explicitly based on the fundamental theories and principles of that discipline, which also applies to academic dictionary reviewing. It is proposed that an appropriate approach is to say that dictionaries are first and foremost lexicographic products based on lexicographic principles and in the second place relate to other disciplines through their data contents and functions. Consequently, reviews in general and reviews of the outside matter in particular should have a lexicographic core basis with a supplementary basis related directly to the other discipline(s) involved, for instance economics, law, linguistics and translation. This means that the ideal reviewer should be an expert in lexicography plus something else, or that teams of reviewers with different competences are needed. However, bibliographic references are not the only indications of ontological positions of reviewers. If one looks behind the numbers in Tables 1 and 2, it turns out that a quantitative analysis like this has drawbacks. The main disadvantage is that even though there are references to lexicographic literature at the end of 6 reviews, it is difficult to pinpoint which references have been instrumental in forming the reviewers' perception of the concept of dictionary. Furthermore, the references are mostly so general that it is unclear which concern the issue of the outside matter. This indirect way of indicating references is inferior to placing references to lexicographic literature directly in the text. Ridge (1993: 293) contains a direct reference to lexicographic literature dealing with the outside matter: 'The introductory section is also kept as short as possible. As Lemmens and Wekker (1986) have shown, users of a dictionary are unlikely to consult detailed discussion in an introduction.' This is the most explicit reference to a theoretical position on the outside matter found in the material studied, but the reviews contain other clues.
Information about the treatment of outer texts can be found outside the reference sections. The use of different linguistic means in their texts directly or indirectly reveals the reviewers' perception of the notion of dictionary. For instance, after having said that 'the dictionary begins with initial information for the users', Batibo (2004: 451) continues by saying that 'the dictionary itself is divided into two main parts', by which he means that the dictionary has two bilingual wordlists. These utterances show that the reviewer has a narrow understanding of what a dictionary is. The reference work reviewed by Batibo (2004) is called a dictionary and therefore the entire book is the dictionary, i.e. the research object, and the dictionary does not start on, say, page 25. An equally awkward use of terminology is found in Chebanne (2007a: 447): 'The dictionary consists of preliminary remarks and acknowledgements, the table of contents, abbreviations and symbols, an introduction, the Khwe-English dictionary [...]'. Does it really make sense to say that a dictionary contains front-matter texts and a dictionary? At least this does not seem logical. In a similar vein, Mashamaite (2000: 329) claims that 'the dictionary has a simple structure', by which he means that 'the wordlist has a simple structure'. However, other reviewers have a broader view of the object of dictionary reviewing, as explained by Gold (1993: 305): Reviewing a dictionary means not only scrutinizing the lexical material which it provides, but also examining its front matter, back matter, covers, spine, jacket, and title, as well as the publisher's advertizing for it.
In order to meet the overriding objective of giving a faithful representation, reviewers should have a broad understanding of the object of their reviews. Of course, the publishers' advertising material is not part of the dictionary, but may help reviewers in their work. However, for the purpose of dictionary reviewing, reviewers should regard dictionaries as complex objects that contain a number of independent sections constituting a whole: the dictionary. By comparison, those who review the performance of symphony orchestras evaluate the entire orchestra, not just one or two sections. This is not to say that reviewers who have a narrow understanding of the concept of dictionary write non-academic reviews, but it may indicate that they are perhaps (slightly) off track.
In this context, it may be relevant to make a distinction between two general types of outside matter: outside matter with a simple composition containing a maximum of two outside matter sections, and outside matter with a complex com-position containing more than two outside matter sections. A typical example of simple outside matter contains a preface and a user guide, and it will be relevant to discuss whether such a simple set of outer texts is sufficient for fulfilling the function(s) of the dictionary, taking into account the linguistic, factual and cultural competences of intended users. Typically, complex outside matter contains a preface, a user guide, a subject-field introduction, a style guide, and several appendices, and reviewers should examine whether these outer texts individually and collectively live up to expectations and report their findings. In order to give useful information about the complete dictionary and represent it faithfully, these individual sections should be addressed by reviewers. So far, the discussion has had a quantitative focus, but the evaluation of the outside matter also involves a qualitative aspect.

What the reviews say and could say
In order to obtain a detailed picture of the evaluation of the outside matter in dictionary reviewing, further examination of the reviews is necessary. This will involve a content analysis of the 25 reviews that address the issue of the outside matter. In addition to the quantitative findings mentioned above, such a content analysis will reveal the degree of scholarliness of the reviews, indicate to what extent the reviews give a faithful representation of the dictionaries, and suggest whether the reviews contribute to the development of lexicography. When one moves on to a content analysis, the definition of a dictionary as a complex whole in the two-dimensional space with wordlist and outer texts cannot stand alone. In addition to its surface features, a dictionary -whether printed or electronic -has three underlying features, namely: (a) It has been designed to fulfil one or more functions. (b) It contains data that have been selected because they help to fulfil its function(s). (c) It has structures that marshal its data into the task of fulfilling its function(s).
Only when it is defined in terms of both surface and underlying features, does one have a proper account of a dictionary as an object of analysis, description, evaluation and research. And this is what the dictionary consists of for reviewing purposes. However, as pointed out by Nielsen (1999: 301) dictionaries often lack clear statements about their functions, so reviewers have to make educated guesses to assess whether the data support the functions for which the dictionaries are intended. Despite this lack of indications as to the functions of dictionaries, reviewers should be aware of the role of outer texts for, as Gouws (2007: 83) accentuates, the wordlist 'is not the only venue for the realisation of functions'. The extension of the definition of the review object entails a further specification of the overriding objective of dictionary reviewing. It is proposed that this objective is to give a faithful representation of the lexicographic elements of the surface features, i.e. wordlist and outer texts, their organisation and presentation, as well as the underlying features, i.e. the function(s), data types and structures of the dictionary. Any evaluation of the outside matter should be conducted with this objective in mind, and it will also be used as a benchmark for assessing the qualitative features of the reviews examined.
The analysis carried out reveals that there are considerable differences in the way in which reviewers comment on outer texts. Some reviewers merely acknowledge the existence of outer texts, such as Schuring (1994: 298) who identifies one outer text and says: 'The bibliography at the end of the dictionary covers eleven pages.' This is a minimalist approach to reviewing, which does not meet the requirements of critical evaluation, though it could be argued that it meets the requirement of quantitative analysis. However, the utterance is merely a description and statement of fact and it does not contribute to the qualitative development of lexicography. On the whole, this reviewer does not give any significant evaluation of the outside matter (e.g. by not relating the bibliography to the function(s) of the dictionary), which may also be an indication of a narrow understanding of the reference work under review. In a different type of review, Snyman (2000: 332) refers several times to pages in the dictionary without explaining which section he is speaking about, for instance 'an orthographic list of symbols' and 'the list of abbreviations'. The forms of the references, which mention p. i and p. iv respectively, indicate that the reviewer refers to text in the front matter, but does not expand on it. An explicit indication of the section to which he refers would have resulted in a more faithful representation, enabled readers to better judge the quality of the dictionary reviewed, and avoided unnecessary speculation and doubt.
Not all the reviewers have taken a minimalist approach to their evaluation of outer texts. In their review of a music dictionary, King and Steyn (2003: 343-344) discuss the contents of the 'Introduction' in 75 lines. Impressive though this is, readers only get a glimpse of what this introductory text contains and what its purpose is, because the reviewers focus on the musical contents of the text and ignore lexicographic aspects. The criticism is no doubt correct, but the discussion does not contribute to the development of lexicography, though it gives a detailed evaluation of the quality of the data presented in the 'Introduction'. Apart from one reference to the publishers' blurb on the back cover of the dictionary, no other outside matter is mentioned in the review. A similar treatment of the outside matter is found in Chebanne (2007b: 481-483), though in this case the two-page discussion of the introduction concerns the status and history of the language covered by the dictionary. Again, this is the only outer text the reviewer considers and the points he raises are no doubt correct, but the evaluation does not contribute to the development of lexicography, and it is therefore doubtful whether the review faithfully represents the dictionary as defined above.
Several reviewers explicitly mention all the outer texts found in the dic-tionaries. At first glance, this meets the requirement of giving a faithful representation of the entire dictionary, though some reviewers treat outer texts differently in the same review. Feinauer (1996: 280) explicitly mentions that the dictionary under review contains ten appendices, but only two are singled out for comment with the result that the informative value of the review is lower than it could have been. Benjamin (2001: 312) describes the organisation of the dictionary, saying about the user guide: 'The explanation of the entries is a valuable source of learning material and even teachers of grades 8-9 (the first two years of high school) should find the dictionary useful.' Not being an academic analysis of the purpose of user guides, this is merely a description which contributes nothing to the development of lexicography. However, Gold (1993) and Mojela (2003) are good examples of reviews that attempt to give a faithful representation of the dictionary in its entirety, addressing the individual sections in a way that is informative and useful to readers. One minor flaw can be found, though, in that Mojela (2003: 334) states that 'the dictionary has important features such as front and back matter' but does not comment on the backmatter texts. One area in which the terminology used by reviewers -who no doubt use the terminology found in the dictionaries -is ambiguous concerns the 'introduction', also referred to in the reviews as introductory remarks, introductory pages, introductory guide, initial information for users, and preliminaries. First, not all reviewers make it clear which types of data this 'introduction' contains and what its function is (e.g. Hughes 1996). Second, some reviews show that the introduction is a section giving a preliminary to a technical subject field or the language treated by the dictionary in question (e.g. Sitaram 1996). Third, a number of reviews indicate that the introduction is a collection of text types, in particular the preface and the user guide (e.g. Mojela 2003). For readers to acquire a faithful representation of the object analysed and examined, reviewers should comment on the fact that, by lumping several and distinct types of data together in one outer text, the dictionary authors have not helped users sufficiently. Reviewers should point out that hiding important information from users is not the best option. Finally, ambiguous statements fall short of the requirements of academic standards. In order to contribute to the development of lexicography, reviewers should make precise and unambiguous statements so that they do not leave readers in doubt of what is actually meant, of what was left unsaid, and why. For instance, does the dictionary reviewed by Schuring (1994) really only contain a wordlist and a bibliography? And why did Feinauer (1996) single out only two out of ten appendices?
Directly related to the ambiguous term 'introduction' and its variants, is the effect it has. Most reviewers seem to use an approach that places form over substance. This means that their focus is on the formal names of the sections, i.e. it seems to be more important to mention the sections by their formal designations than to focus on the actual data contents. The main drawback of this approach is that it often results in description rather than analysis and critical evaluation. It is proposed that a better approach is to place substance over form, i.e. to focus on what types of data are actually in the sections, how these are accessed, how these are presented, and how these support the functions of the dictionary. One way that can help change the focus from form to substance is to study the lexicographic literature where lexicographers discuss the various sections that can be found in the outside matter, some arguing that prefaces and user guides should be presented separately in dictionaries. Herberg (1989) and Nielsen (1994: 88-93;1995: 168-169) discuss prefaces in dictionaries, their functions and data types. On the basis of these discussions, it is proposed that reviewers should evaluate prefaces by indicating to what extent they (at least) answer the following questions: What is (are) the function(s) of the dictionary (e.g. text reception, text production or translation)? (b) What is the subject field of the dictionary (e.g. linguistics, law, mechanical engineering, or a multiplicity of subject fields)? (c) What is (are) the target group(s) of the dictionary? (d) What is the empirical basis of the dictionary (e.g. sources of lemma or equivalent selection)?
The existing literature also discusses user guides in dictionaries. Reviewers' lack of attention to user guides is noted by Chan and Taylor (2001: 168), who claim that most reviewers do not 'appreciate the potential contribution made by information contained in the user's guide to using a dictionary to its fullest extent'. Theoretical articles such as Kirkpatrick (1989) and Nielsen (2005;2006) discuss the function and data types in dictionary user guides. These contributions lead to the suggestion that reviewers should evaluate user guides by indicating to what extent they (at least) answer the following questions: (a) How do users best access the data when using the dictionary as an aid in, for example, reading, writing and translating texts?
What types of data are in the dictionary? (c) Where can those data types be found (e.g. in separate sections or in the wordlist)? (d) How do the data in the individual sections in the outside matter relate to, complement or supplement the data in the wordlist and the other outer texts?
In what types of user situation are the data in the sections likely to help users?
One benefit of using the substance-over-form approach in discussing the above questions is that it may alert reviewers to aspects relevant for the development of lexicography. For example, prefaces and user guides that answer few questions may be used as examples of how not to write these sections. Conversely, prefaces and user guides that answer all questions or more may be used as examples to be followed by both theoretical and practical lexicographers. Finally, critical evaluations of these two sections may lead dictionary authors to write really good prefaces and user guides, so that they are read because users find them helpful and relevant. Reviewers may also examine to what extent the factual contents of outer texts relate to the functions of dictionaries. As discussed in Bergenholtz, Tarp and Wiegand (1999: 1778-1791 and Bergenholtz and Nielsen (2006), lexicographic data are mainly found in the wordlist but may be placed anywhere in the dictionary depending on its distribution structure. The relevant data are found in the lexicographic sections of the dictionary, i.e. those textual components that contain data about its use and function(s), for example in sections that inform users of the subject field treated by the dictionary (e.g. geology (Zawada 1997)). Other textual components such as advertising texts, picture credits and single user licence agreements are extra-lexicographic sections as described in Nielsen (2006: 7-8), irrelevant for academic dictionary reviewing as they do not contain lexicographic elements. In this context, it is suggested that reviewers look at the number of subject fields treated, and, if only one, whether it is the entire field, or one or more subfields, and whether the section is integrated into the wordlist. Sections to which there are cross-references from the wordlist or vice versa are integrated sections. It would, for instance, have been relevant for readers to know whether there are cross-references from individual articles to the style guide in the dictionary reviewed by Heliel (1998) so that the guide helps users by actually supporting the function of the dictionary. This type of integration can, for instance, be found in Longman Exams Dictionary (2006) where articles contain cross-references to the 'Writing Handbook' in its middle matter. An analysis and critical evaluation of such an integration may provide valuable contributions to the development of lexicography.
An aspect that extends to surface as well as underlying features of dictionaries is the concept of lexicographic information costs. According to Nielsen (2008: 173-174), lexicographic information costs are defined as the effort that users believe or feel is associated with consulting a dictionary, an article or any other text part of a dictionary such as outer texts. The focus is on the relationship between the expected and actual information costs and the expected and actual informative value, i.e. what users gain from consulting the dictionary. It is important to note, however, that high information costs are not always a disqualification. Even if the lexicographic information costs are high in a particular case, they may be offset by a search result with a large informative value to the user. In accounting terms, the information costs represent expenses and the informative value represents income and if the income exceeds the expenses, the bottom line will post a profit, i.e. show a positive gain.
Two general types of lexicographic information costs are relevant in connection with outer texts. Search-related information costs are the efforts related to the look-up activities users have to perform when consulting outer texts to gain access to the data they are searching for; this relates to the search activities nec-essary to use both printed and electronic dictionaries. Examples of situations involving search-related information costs include cross-references to and within outer texts; the need to scroll up and down the screen to find something in outer texts; the possibility of searching for specific words in outer texts; and the time it takes to open and interact with multimedia elements (see e.g. Louw 2000: 326-327). Comprehension-related information costs are the efforts related to the user's ability to understand and interpret the data presented in outer texts. A typical example is whether the text in the subject-field section is in plain intelligible language adapted to the linguistic, factual and cultural competences of intended users. The design and organisation of outer texts may contribute to keeping lexicographic information costs down, as an inappropriate design, arrangement and layout may lead to high information costs. The actual wording and presentation of the data in outer texts, for instance the use of long and complex sentences, may increase information costs, whereas clear search routes may reduce lexicographic information costs. The ease or difficulty with which users can decode messages on sound files with, for example, explanations and pronunciations may reduce or increase lexicographic information costs.
The above discussion mainly deals with outer texts usually found in the front matter of printed dictionaries. This does not mean that some outer texts are more important than others for the purpose of dictionary reviewing. On the contrary, all outer texts that are lexicographic sections are equally important. Furthermore, prefaces, user guides and subject-field sections can be placed in the front, middle and back matter, as shown in Gold (1993: 305), though there is a tendency to place them in the front matter just as appendices are normally placed in the back matter. Some outer texts are placed on CD-ROMs that accompany printed dictionaries, while the distinction front versus back is irrelevant in online dictionaries. The important point is that each individual type of lexicographic section in the outside matter should be analysed, described, examined and critically evaluated on the basis of their substance related directly to the surface and underlying features of the dictionary, no matter where the sections are placed.

Concluding remarks
By its focus on linguistic categories in dictionary articles, existing literature on dictionary criticism gives little attention to the evaluation of outer texts. This results in a lacuna, leaving reviewers of dictionaries largely to their own devices. The result is that many reviewers exclude outer texts from their evaluation. Those who do include outer texts treat these sections very differently, ranging from a mere acknowledgement of their existence over a description to a critical evaluation. Consequently, the academic standard of treating outer texts varies much, in each case depending on the reviewer. This indicates that there is a need for general principles that can help academic reviewers analyse and critically evaluate the outside matter in printed and electronic dictionaries.
In order to encourage more reviewers to include outer texts in their reviews and thereby write texts that contribute to the development of lexicography, attention should be given to several factors. It is important to appreciate that dictionary reviewing is an exercise in lexicography. This means that a lexicographically based framework broadens the scope so that reviewers are likely to see the dictionary as a complex whole with surface and underlying features that all form the core of reviewing. This should be linked to the overriding objective of dictionary reviewing, which is to give a faithful representation of the lexicographic elements that make up the dictionary. Reviews that meet the overriding objective will not only examine the separate sections of the dictionary independently but also examine their relationship to each other and the wordlist. Finally, in their evaluation of outer texts, reviewers should place substance over form and focus on data contents and presentation. The application of all these elements may result in dictionary reviews that are academically sound because they treat the dictionary as an object of analysis, examination and evaluation, i.e. as a true research object.

Notes
1. Table 1 only shows numbers for reviews dealing with the outside matter, but these numbers may be compared with the corresponding numbers for all reviews examined. Out of 40