Reflections on Lexicographical User Research

The last decades have seen a growing interest in theoretical and practical problems related to lexicographical user research. Starting with a discussion of the concept of lexicographically relevant user needs, this contribution analyses, utilising the lexicographical function theory, various types of needs to be taken into account when doing this kind of research. It then discusses the positive and negative aspects of the various methods applied, i.e. questionnaires, interviews, observation, protocols, experiments, tests, and log files. With reference to both lexicographical and sociological literature, it raises a number of problems common to most of the lexicographical user research conducted until now and recommends the application of scientific methods in future research. Finally, it proposes a number of alternative methods in order to obtain more knowledge about the real user needs.


Introduction
It may be no exaggeration to say that the greater number of surveys conducted today are a waste of time and money.Many are simply bad surveys.Samples are biased; questions are poorly phrased; interviewers are not properly instructed and supervised; and results are misinterpreted.Such surveys are worse than none at all because the sponsor may be misled into a costly area.(Sheatsley 1974) The last decades have seen a growing theoretical interest in lexicography, and, parallel to this development, a similar interest has emerged in acquiring more knowledge of the usage of the practical products of lexicography, first of all dictionaries.This interest was especially aroused in the wake of a conference on lexicography convened in the U.S.A. in 1960, where one of the conclusions was that "dictionaries should be designed with a special set of users in mind and for their specific needs" (Householder 1967: 279).With this starting point it is understandable that a growing number of lexicographers focused on the users and their needs, and started organising research projects within this field.Notwithstanding, the take-off was very slow.In a chronology of 220 published user researches, Welker (2006) only lists three contributions before 1980.Hence, it was not without reason that Wiegand (1977) called the user the wellknown unknown.However, although the user was still treated as an unknown creature by Neubauer (1987), from that moment the situation began to change.The 1980s and 1990s were characterised by the publication of a growing number of articles on user research, mainly as individual contributions in books and journals.Starting in the late 1990s, a number of monographs, books and thematic issues exclusively dedicated to the topic were added to the list, for instance Atkins (1998) which contains a selection of contributions from various researchers, and Nesi (2000), Tono (2001), Wingate (2002), Thumb (2004), Lew (2004) and Dziemianko (2006), which are all monographs.
Apart from these generally available published contributions, various publishing houses also carried out their own user research which, however, was treated as business secrets to protect their competitive power.This practice is a violation of the ethic principle formulated by the sociologist Merton (1968), according to which scientific discoveries are always in one way or another the result of a collective effort and should therefore be considered public property to be freely used by anybody interested.Obviously the undisclosed results of these research projects cannot be subjected to a critical analysis contributing to the development of lexicographical theory, although some authors related to publishing houses, for instance Anthony P. Cowie, occasionally refer to them in their articles.
Meanwhile, the number of publications on user research has gradually reached proportions that make it increasingly difficult to keep up with them.The corresponding need to acquire an overview of these projects and easy access to their results was met by a synoptic work (Welker 2006) which con-tains a short summary and index of 220 research projects published between 1962 and 2006.
Parallel to the publication of the various research projects, a corresponding theoretical literature emerged.Tono (1986), for example, calls for a more scientific approach to this type of research.Ripfel and Wiegand (1988) provide critical comments on the value of the research projects carried out until then.Hartmann (2001), Tono (2001), Thumb (2004), and other monographs contain valuable theoretical reflections on the planning and organisation of user research.The same holds true for Wiegand (1987Wiegand ( , 1998) ) who also provides a systematic introduction to the action-theoretical basis of dictionary usage introducing a whole number of new categories and terms, of which many are very useful and ought to be assimilated by more scholars engaged in dictionary research, although others are perhaps too much integrated into his specific lexicographical theory and therefore less useful outside the framework of this theory (cf.Bergenholtz and Tarp 2002, 2003, 2004).
The following contribution refers to the ideas presented by various scholars and endeavours to develop them even further, utilising the lexicographical function theory.It will primarily discuss which types of needs users have, the advantages, disadvantages and limitations characterising the known types of research methods, and the value that can be assigned to their results.Finally, it will propose some alternative methods to supplement the ones already applied in lexicographical user research.

Dictionary usage
By way of introduction, it is necessary to make a few comments on the concept of dictionary usage, a concept which is, in fact, inappropriate in terms of lexicographical user research because it also incorporates other types of usage not relevant to lexicography.In this respect, Wiegand (1987: 197) distinguishes between five types of dictionary usage or usage-actions: the normal usage of dictionaries as reference works, -the normal usage of dictionaries as reading books about language, -the anomalous usage of dictionaries in order to learn something about dictionary usage, -the anomalous usage of dictionaries when they are not used as dictionaries, and the usage of dictionaries in order to learn normal usage.
The above classification seems to be somewhat problematic.For instance, why is the wish to acquire knowledge about dictionary usage listed as anomalous when reading dictionaries is considered normal usage?An alternative classification based on the following three criteria could be suggested: -Is the type of usage in question lexicographically relevant or not?
http://lexikos.journals.ac.za -Is the dictionary in question used as a reference work or in another relevant way? -Is the dictionary consultation in question function-specific or not?
Of the five types of dictionary usage listed by Wiegand, it is only the fourth that is lexicographically irrelevant (and unpredictable), whereas the other four are relevant in one way or another.For instance, dictionary usage with a view to acquiring skills in terms of dictionary usage is very much relevant to lexicography.Within the framework of lexicographically relevant usage, it is first of all important to distinguish between consultation of dictionaries as reference works -precisely the type of genuine usage separating dictionaries from other kinds of texts where users also look for information -and other types of usage (items 2, 3, and 5 in Wiegand's classification).Finally, it is also necessary to distinguish between a non-function-specific consultation, i.e. a dictionary consultation in general, and a function-specific consultation which takes place when users look for assistance in a lexicographical work designed to meet exactly the type of needs which may occur for users of a type similar to themselves and in the same type of extra-lexicographical situation in which they find themselves.Of course, the precondition for a function-specific consultation is that the lexicographers have analysed and decided on the functions to be displayed by the dictionary and passed this information to the interested users.
Research into lexicographically irrelevant dictionary usage is, by its very nature, irrelevant.As a rule, the same holds true when dictionaries are not used as reference works.Both function-specific and non-function-specific dictionary consultations are obvious topics for lexicographical user research as it also appears from the previously published research projects.But it would be a problem if no clear distinction were made between research into the two types of consultation because this may lead to deceptive and contradictory results.Dictionary consultation takes place when users with a specific type of need occurring in a specific type of extra-lexicographical situation think that this type of need can be satisfied by consulting a dictionary and therefore take action in this direction.If this type of users consult dictionaries specifically designed to provide assistance in the respective situations, then they are more likely to have their needs met than if they used dictionaries not designed to provide this specific assistance.
Here it is also necessary to distinguish between two entirely different situations relevant to lexicographical user research, i.e. the user situation which is an extra-or pre-lexicographical situation where the need to consult a dictionary occurs for a potential user, and the usage situation where the user, now turned into an actual user, takes action to satisfy his/her need by consulting a dictionary or another lexicographical tool.Without a clear distinction between these two completely different types of situations, there is a considerable risk of obtaining deceptive and defective results.Research into the actual usage situation may, if carried out according to scientific standards, lead to reliable information about this type of situation, whereas -as will be argued -it can only http://lexikos.journals.ac.za provide suppositions and vague ideas of the problems and needs occurring in the preceding extra-lexicographical user situation.In this respect, it is important to underline that for research into dictionary usage to be relevant, it should not only generate knowledge of how dictionaries are used, but also of who the users are, where, when and why they use dictionaries, and with which result.Hence, it is necessary to do research into:  (d) the users' usage of a dictionary, and (e) the degree of satisfaction of the user needs.
These five categories are interrelated.Hence, without knowing a user's needs, it makes no sense to investigate the degree to which these needs have been satisfied.And the same applies to dictionary usage where it is necessary not only to know these needs but also the user's general experience in dictionary usage to draw relevant conclusions.Analogously, and as it will be discussed in the following sections, it makes no sense to speak about user needs if these needs are viewed in an abstract way without relating them to specific types of users and situations.
The types of relevant user situations are the communicative ones (production, reception, translation, text revision and marking) and the cognitive ones (systematic and sporadic), to which can also be added the operative ones covered by how-tos, handbooks and manuals (cf.Tarp 2007).However, it is far more complicated to establish a user typology because the criteria for such a typology until now discussed constitute an open list and vary from dictionary to dictionary (cf.Tarp 2008: 54-56).A typology of users depends both on the user situation -i.e. which types of users will find themselves in such a situation -and on their need for differentiated lexicographical solutions.For instance, if learners at a beginner's level need assistance to produce texts in a foreign language, they will, as a rule, need a bilingual solution; whereas advanced learners who think and express themselves directly in a foreign language will frequently manage with a monolingual solution in this language.This implies that it could be relevant to investigate or at least confirm the relevant criteria for establishing user typologies by means of user research.

User needs
The potential and actual dictionary users' lexicographical needs constitute a subset of the needs which human beings have in general.With a view to specifying them, it is therefore necessary to comment upon a number of problems related to human needs in general.To do so, the discussion in this section will take its starting point in the following oppositions: - The changes in human life, collective and individual, in the course of the past 10,000 years, let alone in the past 10 generations, are too great to be explained by a wholly Darwinian mechanism of evolution via genes.They amount to the accelerating inheritance of acquired characteristics by cultural and not genetic mechanisms.I suppose it is Lamarck's revenge on Darwin via human history.
[…] Cultural and biological inheritance don't work the same way.
Lexicographical needs clearly belong to the historical-cultural needs.In a historical perspective, according to Al-Kasimi (1977: 1), dictionaries have emerged as "practical tools" designed to satisfy specific needs observed within different cultures: The major motives behind the rise of lexicography differ from one culture to another.Each culture fosters the development of dictionaries appropriate to its characteristic demands.
Like other similar needs, lexicographical needs will change in the course of time as a function of more profound changes in social life and culture.However, it is worth noting that their satisfaction may simultaneously influence and have a positive effect on the cultural and social development owing to the fact that dictionaries and other lexicographical tools are artefacts contributing to transfer acquired characteristics not only from one generation to another, but http://lexikos.journals.ac.za also among the members of one and the same generation.Consequently, research into lexicographical needs must take into account that these needs are not static but continuously changing and developing.For instance, the needs of "the Assyrians who came to Babylonia about three thousand years ago" (Al-Kasimi 1977: 1) are surely not the same as the needs of the people living in modern South Africa.In a similar way, while it makes no sense investigating the usage of electronic dictionaries only 30 years ago, this situation has changed completely today.
Recognised versus non-recognised needs.The above example concerning food indicates that it is necessary to distinguish between recognised and non-recognised needs.Few people ignore their need for food, but this does not mean that they also recognise that the need for food includes a need for nutrients like minerals, vitamins and proteins.Such recognition is primarily a product of history and the development of science.What holds true for natural needs is to a great extent also valid for historical-cultural needs.In this regard, a person may, in an extra-lexicographical situation, have a lexicographically relevant need which he/she does not recognise and therefore does not try to solve consulting a dictionary, although dictionaries designed for this specific type of need may already exist.There may also be other lexicographically relevant needs which lexicographical theory has still not recognised although they could easily be met by the conception and production of the right dictionaries or other lexicographical tools.It could therefore be relevant to lexicographical user research to try to discover and describe both the recognised and the nonrecognised needs.Analogously, there could be cases where a potential user actually recognises a need, but does not take lexicographical action because he/she does not think that it can be satisfied by means of dictionary consultation.In this respect, the recognised needs express themselves as needs that both trigger and do not trigger a lexicographical consultation, whereas the non-recognised needs never lead to any such consultation.As mentioned above, this indicates that it is not enough to do research on the actual usage situation when the purpose is to reveal the lexicographically relevant user needs.
Objective versus subjective needs.As a continuation of the previous opposition, it is also necessary to distinguish between the actual needs recognised by the potential users and the way these needs are reflected in their brains.Here it makes sense to consider an opposition or dialectical relation between the objective needs, i.e. the actual, genuine needs, and the subjective needs representing the reflection of these needs in the consciousness.This opposition or dialectical relation between objective and subjective needs is the result of a complex cognitive process including a number of pitfalls.Sometimes, the subjective and the objective needs may correspond, and in these cases the potential users are fully conscious of their genuine needs.But the users may frequently only have a vague or approximate idea of the objective needs; they may know that they need something, but not exactly what it is.In such cases, it makes no sense to ask them about their real needs because the answers will be inaccurate and unreliable.And as the researchers will never know when the subjectively recognised and the objective needs correspond completely, it can be argued that user research exclusively built on the informants' own answers or their usage of dictionaries will have only little scientific value if the purpose is to discover the genuine or objective needs occurring before the consultation process, i.e. extra-lexicographically.
Genuine versus artificial needs.Publicity as a scientific discipline was invented in the U.S.A. about a hundred years ago when the American industry needed a bigger home market as the basis for its international expansion.In the beginning, the purpose was to defeat the resistance among the many Christian sects who defended a modest and Spartan life and considered any kind of extravagance and wastefulness to be the work of the Devil.This was the starting point for an industry trying not only to promote genuine, but unsatisfied needs, but also to create new and sometimes completely artificial needs.Although a complex dialectical relation exists between genuine needs and artificial, publicity-created needs, because the latter may sometimes, in the course of time, be transformed into genuine historical-cultural needs, artificial needs should nevertheless be considered as subjective needs for they do not, by definition, correspond to objective needs.The commercial houses publishing the big lexicographical works constitute no exception from the rule when it comes to the use of publicity and the priority of profits over the satisfaction of genuine human needs.In this respect, Hausmann (1989) speaks of a gulf between market and science.This opposition has big consequences for the planning of lexicographical user research.If such research projects have commercial purposes, they will typically try to reveal the subjective needs which the potential users (and buyers) themselves think they have, because this information may be helpful to expand the market capacity of future dictionaries.On the other hand, if the purpose of the research projects is to generate new scientific knowledge, then they ought to focus more on the discovery of genuine, objective needs.It is important to have a clear vision of the differences between the two types of research projects, both when planning a new research project and when comparing the results of a project already completed.

Lexicographical needs
The discussion in the previous section showed that lexicography as a scientific discipline first of all regards lexicographical needs as objective, historical-cultural needs, no matter whether these needs are recognised by the potential dictionary users, whereas commercial lexicography mainly focuses on the subjective needs that express themselves in the market.Of course, this does not mean that science-based lexicography is not interested in the subjective needs in as much as the distance separating the objective needs from the ways these are http://lexikos.journals.ac.za expressed subjectively in the potential users' brain frames, the area where lexicographical education and training of the potential and actual users are needed.
However, lexicographical needs are much more.The most important point is that they are not abstract needs in general, but concrete, specific types of needs related to a specific type of user being in a specific type of extra-lexicographical situation as described above.It is evident that many different needs may occur in such a situation.For instance, the potential user may need new glasses when reading or a computer when writing.What separates lexicographical needs from these types of needs is that they can be met by consulting a dictionary or another lexicographical tool, no matter if this already exists or still has to be conceived.It is a matter of course that the various types of users may have very different lexicographical needs in the various types of situations, but common to all of them are that they are needs for information (cf.Tarp 2008: 56-58).However, dictionaries do not contain information, but only lexicographical data, which have been made accessible through structures in printed dictionaries and by means of links and search machines in electronic ones, and from which users with specific characteristics, through a complex mental process, may retrieve exactly the information required to meet their lexicographical needs (cf.Wiegand 1998Wiegand , 2000Wiegand , 2002)).
In this regard, it is also relevant to distinguish between two basically different types of lexicographical needs, i.e. on the one hand, the so-called primary, function-related needs which are the objective needs occurring in an extralexicographical situation, and, on the other hand, the secondary, usage-related needs which only occur during the consultation process when the users need help to find and interpret the relevant lexicographical data.Obviously these two completely different types of user needs should not be confused when preparing and carrying out a user research project.

Research methods
Within lexicography, various methods have been used to conduct empirical research into users' needs and dictionary usage.Of all these methods, questionnaires are the most commonly used, whereas log files constitute a relatively new method made possible by the introduction of http://lexikos.journals.ac.za computers and electronic dictionaries.With the possible exception of log files, all these methods are well-known within social science, and there is a comprehensive body of literature discussing and evaluating them from a more theoretical angle.These methods and their biggest advantages and disadvantages will be briefly described and discussed below with references to both lexicographical literature and social science.
Questionnaires.As already mentioned, surveys by means of questionnaires are the most commonly used method to investigate the usage of dictionaries.
There are two main types: the one with open and the other with closed questions.In the first type, the respondents or informants have to give the answers within certain predetermined categories, whereas in the second type, the respondents or informants may add categories of answers other than the predetermined ones.The advantage of questionnaires is mainly that it is possible to ask a large number of respondents and that it is relatively easy to analyse the answers, especially in the case of closed questions where no subsequent coding is necessary.But there are also serious disadvantages.This is illustrated by Welker (2006: 23) who classifies the questions typical in questionnaires into three main types: (a) about facts that can be easily remembered by the respondents (for instance: How many dictionaries do they have?When did they buy them?Why did they buy them?),(b) about the usage, and (c) about the user's opinion (for instance: Are you satisfied with the dictionaries?What type do you prefer?What improvements would you like?).Welker (2006: 23) himself believes that the answers to questions of type (a) are reasonably reliable, whereas the answers to questions of type (c) are in all cases subjective "although they may guide lexicographers and publishers to a certain extent".This confidence in retrospective questions dealing with the past and requiring a good memory is not shared by sociologists such as Hansen and Andersen (2000: 146) who comment (my translation -S.T.): Retrospective questions demanding a lot from the respondents' memory may […] cause reliability problems.
[…] However, this does not necessarily mean that such questions should be avoided when this type of reliability problems can be anticipated, but they should at least be taken into consideration when the results of the measuring are to be interpreted.
With this reservation in terms of retrospective questions, it is quite possible to agree with Welker (2006: 23), especially when he provides the following examples: -What are you looking for in the dictionary?-Under which conditions do you consult it most frequently?
http://lexikos.journals.ac.za -In which percentage of the look-ups does the dictionary provide assistance?-Which part of the information is most useful?
Welker rightly comments that these questions, when put after the usage of the dictionary, only reveal the users' perception of their consultation, not the real usage.This scepticism corresponds to the classic criticism of questionnaires by Hatherall (1984: 184): Are subjects saying here what they do, or what they think they do, or what they think they ought to do, or indeed a mixture of all three?Do they all define the categories in the same way -and in the same way as the researcher?When all is said and done, do we not, on this basis, arrive at a consensus on how subjects are likely to behave when faced with a particular questionnaire, rather than authentic data on what they use the dictionary for?[…] I conclude that, whatever the difficulties, the only reliable method of collecting data on dictionary user behaviour is by direct observation.
Although Hatherall's critical comments have often been quoted in lexicographical literature, many lexicographers still carry out user research by means of questionnaires, arriving at conclusions which even a modest sociological knowledge would show to have no scientific warranty.
Interviews.Within lexicography, it is normal to distinguish between questionnaires and interviews as two different methods to collect data.In contrast, within sociology interviews are frequently regarded as a special form of questionnaire (cf.Hansen and Andersen 2000: 98).However, it seems reasonable to maintain the distinction between questionnaires and interviews because lexicography does not only use the latter to conduct quantitative research where the questions are the same as in questionnaires, but also to carry out qualitative research where the respondents may speak their mind and where the interviewers do not base themselves on predetermined questions but, for instance, on an action pattern observed among the respondents.In this respect, Welker (2006: 26) speaks of "open interviews" in opposition to interviews undertaken by means of questionnaires.Zikmund (1997: 122) calls it an "in-depth interview" which he characterises as a "relatively unstructured, extensive interview".
Interviews have a number of clear advantages compared to questionnaires distributed to respondents to be answered at home, at work, or in a special room.For instance, the respondents cannot cheat the interviewer by consulting dictionaries in the course of the interview.Likewise, the interviewer may explain the meaning of questions not understood by the respondents, e.g. containing linguistic or lexicographical terms.On the other hand, interviews suffer from the disadvantage that they do not solve the dilemma mentioned by Hatherall (1984), i.e. whether respondents say what they do, or what they think they do, or what they think they ought to do, or a mixture of all three.Further-more, the very presence of the interviewer or his/her way of asking may influence the informants and the answers given.Finally, interviews are time-consuming and therefore frequently expensive to undertake.This makes it more difficult to reach the number of informants necessary to be representative of the given population, i.e. the group of potential and actual users, and to provide statistically valid answers.Of course, this is especially a problem related to quantitative research, whereas other criteria are valid for in-depth interviews with regard to qualitative research where they are combined with other methods such as observation, protocols and log files.
Observation.Wiegand (1998) distinguishes between several types of observation with regard to dictionary usage.Hence, the observation may be open or hidden depending on whether the informant can see the observer.Likewise, the latter may participate actively with advice and instructions, or remain passive.The observation may be carried out directly in relation to the usage of the dictionary, or afterwards when the consultation process is videotaped.Furthermore, the observation may take place in a research laboratory or at the informant's normal work site.And finally, the results of the observation may be written down in a structured manner, or freely and unstructured (cf.Wiegand 1998: 570-583).
As seen above, Hatherall recommended observation as the most reliable method of collecting data on dictionary usage.There is no doubt that a welltrained and well-prepared observer may collect useful data through this method, and that the informant's chances to cheat are strongly reduced.But like interviews, observation is very time-consuming and expensive because it requires the presence of an observer who may also disturb the informant if it is a case of open observation as Wiegand calls it.Hatherall (1984: 184) is very much aware of this limitation: Ideally, […] the researcher would actually watch the users in action.But this, too, causes problems.Under such conditions it would probably be difficult for the subjects to behave normally as users.Also, it is unlikely that all the information the researcher needs would be retrievable via the visual medium.And finally, such an exercise is so time-consuming that the sample is likely to remain unrepresentatively small.But apart from this legitimate criticism, it is also a big problem that observation only serves to study "the external aspects of the usage action", i.e. what happens, but not why it happens and with what result (cf.Wiegand 1998: 974).On the other hand, observation as method displays clear advantages compared to the other types of surveys discussed above.In this respect, Zikmund (1997: 265) writes: The major advantage of observation studies over surveys, which obtain selfreported data from respondents, is that the data do not have distortions, inaccuracies, or other response biases due to memory error, social desirability, and so on.The data are recorded when the actual behaviour takes place.
http://lexikos.journals.ac.zaProtocols.Wiegand (1998: 974) considers dictionary protocols to be a genuine metalexicographical method which embraces not only the external aspect of dictionary usage, but also the internal aspects as well as the "preceding and subsequent context".In this regard, he classifies protocols into two main types: written protocols and oral protocols.Written protocols are produced by the informants themselves, either during or after the dictionary consultation.These protocols may be verified or non-verified according to the presence of somebody to verify that the protocols actually reflect the consultation process.In addition to this, Wiegand distinguishes between structured, non-structured and semi-structured protocols depending on whether the informants have to embody their data in formulas with prepared fields, completely freely, or as a mixture of these two options.
Oral protocols are produced by means of the "think-aloud" method.The informants are invited to freely express which reflections and problems they have during the consultation process.These "thoughts" are tape-recorded and subsequently transcribed and written down in protocol form.Although one cannot "think aloud" as Wiegand (1998) rightly comments, the "think-aloud" method is nevertheless an excellent and highly appreciated method to go beyond the external aspects of the dictionary consultation process.As such, it gives the researcher an idea of the users' way of working as well as what is happening during the process, what users are looking for, what they think they are looking for, and which problems they face when trying to find and interpret the relevant data.A number of research projects performed with this method have provided valuable results, among others Wingate (2002) who did research into the usefulness of various types of definitions in learners' dictionaries, and Thumb (2004) who focused on the users' different look-up strategies and the problems they faced during the process.
A disadvantage of the protocol method -and especially the oral protocols that require further processing -is that it is very time-consuming and therefore frequently only involves a reduced number of informants.Moreover, it does not reveal the users' objective needs, but only the subjective needs they themselves believe to have.
Experiments.The purpose of an experiment is to see how the introduction of a certain factor influences the result.This factor may be teaching in dictionary usage, the introduction of a new type of definition, specific grammatical data or a group of such data, a different structuring of the dictionary article or of the dictionary as a whole, the introduction of new access routes in electronic dictionaries, etc.Only very few experiments of this kind have been conducted with regard to lexicography.Tono is probably the researcher who has worked most exhaustively with lexicographical experiments.He (Tono 2001: 70-72) classifies experiments in three different types: In the pre-experiment, a single group is tested in order to measure one variable.
Then a new factor is introduced (e.g.teaching in dictionary usage), and a new test carried out to see how this factor influences the result.The problem related to this type of pre-experimental investigation is that it is not possible to control the external factors which may invalidate the result.
In the quasi-experiment, two different groups are tested as a start.Then a new test is performed where a new factor is introduced to one of the groups, the test group, whereas this factor is not introduced into the other group.When the two results are compared, it is easier to see the effect caused by the introduction of the new factor.
The true experiment is in many ways similar to the quasi-experiment but with the important difference that the two groups of test persons are selected at random.Tono (2001: 71) argues as follows: In theory […] selection controls all possible independent variables.In practice, of course, it is only when enough subjects are included in the experiment that the principle of randomisation has a chance to operate as a powerful control.
According to Zöfgen (1994: 50), most psychologists consider experiments to be the "royal road" of empirical research.The advantage of experiments is that it is possible to measure how the introduction of certain types of data or groups of such data in dictionaries and other lexicographical tools influence the result of the consultation by various types of users.A disadvantage is that they are very time-consuming and require a large number of informants to be representative, and that it is necessary to have knowledge of statistic methods or collaborate with statisticians with a view to determining whether the research results are statistically significant.

Tests.
A test in the lexicographical sense of the word is a method to evaluate to what degree the consultation of a dictionary or other lexicographical tool can help users to satisfy their needs.As mentioned in the previous section, tests are an integrated part of experiments whose performance does not make sense if the results are not measured.Analogously, tests may contribute to avoid or mitigate some of the shortcomings mentioned with regard to other methods.As Nesi (2000: 32) rightly argues, this requires that they are properly performed: The validity of test findings very much depends on correct test administration, and appropriate test design.If this advice is followed, the advantage of tests is that they may provide reliable information about the results of dictionary consultation.In addition to http://lexikos.journals.ac.za this, they are relatively easy to undertake, especially if multiple choice is used.However, if this method is not used, the assessment of the test results may be subjective if only one person is involved, for which reason more than one person is needed to evaluate them, an aspect that constitutes a clear disadvantage.In contrast, the application of some types of multiple-choice tests may "affect the validity of results by providing a further context for each word, and therefore facilitating contextual guessing as an alternative to dictionary use" (Nesi (2000: 32).
Log files.The research into log files is, as mentioned above, a relatively new method made possible by the introduction of electronic dictionaries.There are basically two different types of log files.The first type is the registration of all movements on the user's computer, i.e. activation of the keyboard and use of the mouse.The second type is the registration of all transactions between the user's computer and the database where the dictionary is located.In contrast to the previous type, the registration is made in the database facilitating the collection of data.From a lexicographical perspective, only research of the second type is known, but this does in no way imply that the first method is irrelevant to lexicography.
Although much research has been undertaken into the usage of electronic dictionaries, most of the research projects have been carried out by means of "traditional" methods.Only a few have been based on log files, among them De Schryver and Joffe (2004), De Schryver et al. (2006), Bergenholtz andJohnsen (2005, 2007) and Almind (2008).User research by means of log files is a new method and it is therefore reasonable to believe that all its potentialities have not by far been explored, also because electronic dictionaries themselves are rapidly developing and changing.However, based on the state of the art it is nevertheless possible to evaluate this method and its results, advantages and disadvantages so far.
One advantage of using log files is that they constitute a type of observation which does not influence the user and interfere in the lexicographical consultation process.It also provides easy access to a large amount of data representing the whole population of actual users, from which reliable information about consultation can be retrieved using the proper methods.In addition to this, the processing of the collected data is relatively easy in terms of quantitative research.The collected data may also be used for qualitative purposes, for instance to reveal the individual user's look-up pattern, although this type of research is very time-consuming (cf.Almind 2008).Finally, log files may also be used to obtain information on aspects related to the user typology and user situation, especially when the users are invited to define themselves and their extra-lexicographical situation before accessing the dictionary as such (cf.Bergenholtz and Johnsen 2007).
However, the use of log files as a lexicographical research method also suffers from a number of limitations and disadvantages.Like other forms of observation, they only furnish data related to the external aspects of the consultation process, but no data capable of explaining why users do what they do.And neither do they provide data informing about the users' objective and subjective needs, or the results of the consultation.Hence, in order to acquire more qualified knowledge that may lead to improved lexicographical products, it is necessary to combine the analysis of the log files with one or more of the already discussed methods, for instance interviews that can follow up on the information obtained from the log files.

General problems
There are a number of general problems with regard to the majority of the lexicographical research projects published until now.This is especially the case with the quantitative user research which is generally characterised by the "lefthand work of the research institutions" criticised by the two sociologists Hansen and Andersen (2000: 23).As a rule, the informants have not been selected at random which is an indispensable requirement in this type of investigation.In his synoptic review of 220 lexicographical research projects, Welker (2006: 9) writes (my translation -S.T.): The majority of the informants in the research projects were university students, among whom many studied foreign languages at the bachelor or licentiate level.
In this way, they have broken the golden rule of sociology that informants should never select themselves or be selected by the researchers, but that the selection should always be made at random.Of course, there may be cases where the whole population of dictionary users are university students and where the random sample to be investigated is necessarily made up by students.But this is the exception to the rule in the 220 research projects, of which the majority has also broken another golden rule, i.e. that the number of informants should be large enough to provide statistically significant (and relevant) results.This lenient way of taking samples implies that they are not representative of the total population, and that the results cannot be generalised.It is nevertheless a fact that the researchers behind many of these projects do not hesitate to add percentages and decimals to everything that their miniature world of informants have done, looked for, wanted, etc.Another serious problem is the formulation of the questions used in various types of questionnaires and interviews.Also in this case, the lexicographical researchers have not followed the corresponding instructions put forward by sociology (cf.Hansen and Andersen 2000: 97-150).This is not only true of questions which are ambiguous, which the informants do not understand, or which require a good memory.It is also a matter of formulating the right questions.Even if the researchers have been meticulous with regard to data collection, analysis and validation -which is regrettably not always the case -they still run the risk of receiving not only wrong and dubious answers to the right http://lexikos.journals.ac.za questions, but also "the right answer to the wrong question" (cf.Zikmund 1997: 96).These weaknesses can most probably be explained by poor preliminary work in the phase where the researchers are supposed to formulate the problem and reach clarity on which type of knowledge should be pursued and how it should be produced, as discussed by Hansen and Andersen (2000: 31) (my translation -S.T.): The problem formulation refers to the process where the researcher, fixed upon the interest aroused by or the curiosity towards a phenomenon, chooses his research field, defines what he will investigate, and which aspects or problems within the field he will focus on.These considerations conclude with the formulation of the questions to be answered by the investigation.These questions should be formulated as precisely as possible so that they can form the basis of the evaluation of which form of knowledge the investigation should produce, and thus constitute the starting point for the decision on how this knowledge should be generated.
This preliminary work has to be done before deciding on the methods to be used in the research project and, if suitable, the formulation of a questionnaire.The aspects which previous lexicographical user research has tried to clarify can be divided into two main types.It seems that the best and most convincing results have been achieved when the research projects have focused on the usage situation and the different look-up strategies and search routes of the users (cf.Thumb 2004, andBergenholtz andGouws 2007).In addition to this, many projects have focused on where the users look for information and what information they look for.On the other hand, it is surprising that very few research projects endeavour to produce more knowledge on the users' needs.There is no known investigation that provides real information on the users' objective needs.And it is even more surprising that there are very few projects telling anything real and factual about the users' subjective needs, although they sometimes try to infer these needs from the generated data.However, this inference is most often built on a reconstruction of the users' needs according to the information they look for in dictionaries, i.e. an indirect determination of the user needs where all the lexicographically relevant needs not expressing themselves in any dictionary consultation are ignored.The dubious result of this circular arguing is that the users need exactly what has already been included in dictionaries.This is the same type of methodological problem mentioned by Mentrup (1984: 151) in his criticism of Wiegand. Humblé (2001: 53) has the following critical comment on this way of conducting research: Investigating the link between needs and habits is useful only to find out how learners at present tackle the problems which current dictionaries are expected to solve.They reveal what the needs are to the extent that they are soluble by already existing means.
On the whole, it is worth considering the relevance of quantitative research to lexicography.For instance, how useful exactly is the information that such and such a percentage of all dictionary users in such and such a percentage of their look-ups are searching for?What relevance has the information that 80 percent of all look-ups have to do with orthography and semantics?Of course, commercial lexicography would be delighted and would hasten to produce dictionaries and publicity material focusing on these types of data.On the other hand, scientific lexicography would above all be interested in knowing in which situations -e.g.reception and production -these needs may occur.Then it would set itself the task of uncovering the needs users have in the last 20 percent of the look-ups, i.e. in one out of five consultations.And it would not stop here, but would try to go even deeper into the problem in order to discover the needs that only show up in one out of a hundred or one out of a thousand consultations, or, even more rarely, in order to conceive dictionaries capable of meeting all the users' needs in specific types of situations.Whereto else, if not to dictionaries, should users direct themselves when they look for assistance to satisfy lexicographically relevant needs?
Although the majority of the previous quantitative user research projects have not fulfilled their objectives, this does not mean that the generated results are useless if they are critically assimilated.For instance, Tarp (2008: 154-157) has studied the results of four of these research projects - Tomaszczyk (1989), Mackintosh (1998), Varantola (1998) and Nord (2002) -with a view to confirming one of the theses of the function theory, i.e. that translators frequently need to go directly to a dictionary in the target language instead of consulting a bilingual dictionary translating from the source language into the target language.Although the projects mentioned are all based on a very small number of informants -two of them with less than ten informants -they nevertheless provide an excess of percentages and decimals showing how often the informants are using one dictionary or another.It is evident that the small sample of informants is in no way representative of the total population, and that the results have no validity outside the narrow sphere of the projects.However, by means of a qualitative analysis of the generated data, it is nevertheless possible to conclude that the informants, in all research projects, sometimes pass directly to a dictionary in the target language.This proves that such a need -i.e. the consultation of a target language dictionary -exists with regard to translation, although it does not say anything about how frequently it may occur and if the need -as must be expected -is more pronounced with welltrained and experienced translators.However, it is worth noting that this conclusion was not the direct purpose of the research, but a deduced result that does not require a lot of percentages and decimals.

Alternative methods
It is evident from the above discussion that no known user research has produced real information on the objective user needs, i.e. the needs that may occur in the extra-lexicographical situation preceding the dictionary consulta-http://lexikos.journals.ac.za tion.If this has to be accomplished, then it is necessary to move the focus from the dictionary usage situation to the extra-lexicographical user situation.In this respect, there are various methods, mostly with a view to obtain qualitative results.The only one applied so far is the deductive method used in the function theory and based on a complex of premises.However, if more empirical data is needed to substantiate the function theory, other methods can also be used.Tests and interviews could be applied to investigate how much of a given text readers have understood and which reception problems they have experienced during the reading.Text revision and marking may be used to uncover non-recognised needs occurring during text production and translation.Likewise, the activation of Microsoft Word's spelling and grammar function may also reveal a number of non-recognised needs.Another option is the use of log files, eye-tracking and "think-aloud" protocols with regard to text production and translation, i.e. independent of any dictionary consultation.And finally, in-depth interviews may also be useful to clarify certain problems related to the other methods.
Except for the deductive method applied by the function theory, many of the other methods, which probably have to be combined to generate maximum results, are time-consuming and expensive to use.It may be difficult to obtain the necessary financing, and it is therefore a real question whether this is the way to be followed by lexicography with a view to projecting itself into the future.

Conclusion
At the beginning of this contribution, the survey expert Sheatsley (1974) was quoted for writing that "it may be no exaggeration to say that the greater number of surveys conducted today are a waste of time and money".It is difficult to judge how the general development on this front has been during the last 35 years, and which improvements and progress have been achieved.But if one focuses exclusively on lexicography, it seems that almost no qualitative progress has been made.There are of course positive aspects, but it is not difficult to reach the conclusion that the majority of the previous user research is in fact "a waste of time and money".This holds especially true for the quantitative research projects.Future user research should learn from this experience.It should formulate clear objectives, incorporate the scientific methods used by modern social science, and do a meticulous planning.In this respect, the advantage of combining various methods within the framework of one and the same research project should be considered, as well as the degree to which quantitative research methods are at all relevant to lexicography.Finally, the research should be based on an advanced theory of lexicography capable of establishing scientific categories with regard to user needs, user typology, user situations, usage situations, access routes, etc., to be taken into account when planning and per-forming research projects.Atkins and Rundell (2008: 4) synthesise their view of the scientific and theoretical status of lexicography: This is not a book about 'theoretical lexicography' -for the very good reason that we do not believe that such a thing exists.But that is not to say that we pay no attention to theoretical issues.Far from it.There is an enormous body of linguistic theory which has the potential to help lexicographers to do their jobs more effectively and with greater confidence.
The deficient state of the art of lexicographical user research could perhaps be explained by the fact that many researchers still share the anti-theoretical approach defended by Atkins and Rundell and reject any attempt at establishing lexicography as an independent scientific discipline with its own concepts, methods, theory, and interdisciplinary vocation.
experimental design: the one group pretest-posttest, http://lexikos.journals.ac.za(b) a quasi-experimental design: the non-equivalent control group design, and (c) a true experimental design: the pretest-posttest group design.