Timothy C. Lethbridge

School of Information Technology and Engineering

150 Louis Pasteur, University of Ottawa

Ottawa, Canada, K1N 6N5

tcl@site.uottawa.ca

We discuss the evaluation of a tool designed to allow domain specialists to manage their own knowledge base. We present the evaluation as a two-phase process: In the first phase we assess whether the tool has met its objectives of allowing those not trained in logical formalisms to effectively represent and manipulate knowledge in a computer. By studying use of the tool by its intended users, we conclude that it has met this objective. In the second phase of the evaluation, we assess what aspects of the tool have in fact led to its success. To do this we study what tasks are performed by users, and what features of both knowledge representation and user interface are exercised. We find that features for manipulating the inheritance hierarchy and naming concepts are considered the most valuable. Our overall conclusion is that tool research must involve this two-phase approach if the others are to learn from the work – the research has much less value unless it can be determined which features should most profitably be adopted by others.

This paper presents a case study in evaluation of a domain-specialist oriented knowledge management system called CODE4 (Lethbridge, 1994; Skuce and Lethbridge, 1995). CODE4 is primarily intended to be used by domain specialists who want to organize and manipulate knowledge for the sake of better understanding and communicating that knowledge among humans. Contrasting systems rely on knowledge engineers, and involve the development of formalized models of knowledge for manipulation by software that performs reasoning.

In this paper we outline key steps in the process of developing CODE4, but the focus is placed on evaluation. Evaluation is a key phase in any software development project: In conventional software engineering, evaluation comprises such activities as validating requirement or prototypes to ensure they meet customer needs, and verifying that the final product is free of defects. The star model of software development (Hix and Hartson, 1993) places evaluation at the centre, with all other software development tasks surrounding it.

The hoped-for conclusion from such evaluation is that the system in question is in some sense good. Such conclusions are of little value, however, if others are to learn from the software development experience. If software is being developed as a research task therefore, it is necessary to go one step further and evaluate why the software is good. In this paper we present both types of evaluation.

It is hoped that readers of this paper will learn two things: The first is what kinds of features ought to be present in a domain-specialist oriented knowledge management system – as determined by our evaluation. The second is how should research regarding software tools be performed and reported – we hope that this paper highlights some important aspects of that process.

This paper is structured according to the following general approach to performing research into software tools. The intent is that these steps be performed iteratively:

It is our belief that each of these steps is important, but that much research into software tools does not effectively address one or more of the steps.

Step i is often omitted because the tool is developed to test or implement some underlying technological of scientific idea (e.g. a new knowledge representation), rather than to help perform a task. Unless the target task and users are effectively chosen, few or no conclusions about usefulness can be drawn. On the other hand, if a real-world need can be established by defining a task that users want to perform more effectively, then the resulting system can be evaluated to determine if the users are in fact more productive. Task analysis is well understood in the field of human computer interaction (see Diaper, 1989), but is not widely applied by researchers in computing disciplines

Step ii is also often omitted when software tools are developed in a research setting – the evaluation criteria are often established only during the evaluation phase, after the tool exists. However, establishing evaluation criteria can help focus the work and guide design decisions. Establishment of evaluation criteria in advance is characteristic of engineering approaches such as usability engineering (Whiteside, Bennett and Holtzblatt, 1988).

In the past there has been criticism that software developed in research environments, and reported in the literature, is inadequately evaluated. Although we are now in the era where the importance of evaluation is more widely recognized, we believe that little attention is placed on distinguishing steps iv and v in our general approach. If step iv is performed alone, readers are able to conclude that the researchers have done good work, but they cannot learn what aspects of the tool contributed to success. Readers might assume that all of the features described were useful – in fact, this is the impression conveyed by much literature that describes software tools. On the other hand, if detailed evaluations of features are performed (step v) but there is no verification that the tool as a whole is useful (step iv), then the reader is equally unable to obtain insights that can be reliably applied to subsequent work.

This paper, therefore, is structured according to the above general approach. Section 2 discusses the first three steps, presenting the task for which CODE4 was intended, as well as key aspects of its design. Section 3 discusses CODE4’s evaluation in general, focusing on the overall techniques used; section 4 presents the effectiveness evaluation and section 5 presents the analytic evaluation.

In this section we summarize those aspects of our research that preceded evaluation: steps i, ii and iii as described in section 2.1

The first step in our research was to define the task. Table 1 presents two very different tasks that might be performed under the general heading of ‘knowledge management’. The first task is by far the most widely researched: Knowledge engineers develop models of some domain, and represent the knowledge using languages rooted in formal logic. The knowledge is then used to drive formal reasoning processes in a computer program commonly called a ‘knowledge based system’.

The second task in table 1 is not so widely researched: Domain specialists, typically with no intent to create a knowledge based system (i.e. no intent to develop a system that performs automated reasoning), want to manipulate knowledge structures so as to understand better some aspects of the world. Analogous tasks would be financial analysts who want to manipulate numbers in a spreadsheet, to better visualize the data and to discover useful relationships using the analytic capabilities of the human brain. Lacking CODE4, domain specialists typically structure their knowledge using diagrams, written text, spreadsheets, outline processors etc. Our hypothesis in performing this research was that a simple tool for manipulating concepts and properties (frames and slots) would prove very useful to these people.

Both task 1 and task 2 involve the acquisition of knowledge, and will probably require a system that can represent such entities as concepts and relations. However, the nature of the tasks will lead to design decisions and evaluation criteria that are significantly different.

Why did we choose to develop a system for task 2 instead of task 1? The main reasons were that we saw an unserviced market and a lack of research into the task.

The success of tools and languages developed with task 1 (in table 1) in mind can probably best be judged by evaluating whether the inferences made with the resulting knowledge are sound and allow people to make decisions effectively. These criteria can generally be specified in objectively-evaluable terms – for example verifying that a tool comes to expected conclusions in known cases, and performs at least as well as other tools.

For our task-2 research, however, any inferences made are by humans. We are therefore forced to use more subjective evaluation techniques.

We decided that the success would be judged based on two criteria. The criteria are phrased as questions and described below:

By using this criterion we are asserting that if the system is used voluntarily by people, then there must be value in it, and our hypothesis mentioned above is found to be true. The criterion requires that the usage be by its target audience and on a variety of significant examples so that we can gain confidence that we are not drawing conclusions by chance.

Our objectives were to obtain strong positive answers to both of the above questions. The conclusions are presented in section 4.

This section outlines some key decisions made during the design of CODE4, and explains their rationale in terms of the task presented in section 2.1 and the evaluation criteria presented in section 2.2. More complete details can be found in Lethbridge 1994 – only enough is presented here so the reader can understand section 5 which discusses analytic evaluation.

To achieve our objectives, we had to design two elements: the knowledge representation and the user interface.

We had to design a knowledge representation that had sufficient power to represent the kind of information users wanted to represent, yet would not require much training to get started. In particular we did not want users to have to put up with complex textual syntaxes; nor did we want them to have to know formal logic or computer programming.

The user interface had to be able to effectively present different views of the knowledge to the users. It also had to allow them to edit knowledge without having to follow restrictive syntactic conventions.

Each of the above aspects is summarized below.

We attempted to keep the knowledge representation very simple: For the sake of uniformity, everything the users manipulate is a concept, and all concepts have common features – e.g. they are located in an inheritance hierarchy and they inherit properties (i.e. relations). Since everything is a concept, even the properties are concepts.

Concepts are divided into type concepts, that can have subconcepts (specializations) and instance concepts that represent specific things. In order to allow the user to more easily manipulate concepts, he or she can readily convert a concept between instance and type (and vice versa if it does not have any subconcepts).

In order to ensure that users could work with the knowledge without being constrained by syntax, we introduced several features that allow what we call ‘informal’ knowledge to be manipulated: Concepts can have several different terms (synonyms), and the values of properties can be expressed using arbitrary text. Users can add more formality if they choose by making the values of concepts point to other concepts. See Lethbridge and Skuce (1992) and (1994) for more on this topic.

Several features were added in order to allow users to better visualize the knowledge. One such feature was the global property hierarchy – this contains all properties in the system, with each concept inheriting a subhierarchy of it. Other features added for visualization purposes included graphical icons and graph layout information.

In order to represent information such as multiple terms, property hierarchy relations, icons and graph layout data without adding special features (i.e. without veering from the notion that the knowledge base should consist entirely of concepts), the notion of the ‘metaconcept’ was introduced. Metaconcepts are not intended to be used directly by ordinary domain experts – they serve only as special concepts that contain information describing an underlying concept as opposed to information described by the underlying concept. Each concept, conceptually at least, has a metaconcept; information about a concept is described using properties of its metaconcept that are known as facets.

A relatively small number of other features were added to the representation to solve specific problems encountered by groups of users. Among these were a) a ‘combining’ technique for automatically resolving situations where two different values of a property are inherited due to multiple inheritance; b) an ability, called ‘delegation’, to specify the value of a property by reference to another property; c) an ability to specify that two type concepts are disjoint (i.e. cannot have a common descendent in the inheritance hierarchy); and d) an ability to add ‘dimension’ or discriminator labels to distinguish among alternative ways of dividing a concepts into subconcepts. See Bowker and Lethbridge (1994a) and (1994b) for information on the latter.

The user interface was designed so that users can interact with multiple different views of the knowledge. Four so-called mediating representations were developed; i.e. visual representations that mediate between the user and the underlying knowledge representation.

Subwindows, each containing a view of part of the knowledge base, are chained together such that sets of concepts currently selected by the user in one window dynamically determine what appears in dependent windows. Each subwindow uses one of the four mediating representations and shows concepts linked by one or more properties.

The four mediating representations are as follows:

1. Graphical: This lays out concepts as nodes linked by arcs in two-dimensional space;. Users can edit the graph by adding nodes and arcs; they can also manually adjust and save the layout of the graph.

2. Outline: This lays out concepts as in the outline view of a word processor, with indentation representing the relation of one concept to another. Concepts can also be arranged alphabetically.

3. Matrix: This lays out concepts in rows and columns with the relations between concepts shown as the intersections of the rows and columns.

4. User language: This is a degenerate mediating representation that merely shows the value of a property as text.

Figure 1 gives capsule views of windows containing the various mediating representations.

The first three mediating representations have many features in common. In particular, the user can: a) add and delete concepts and relations; b) select sets of concepts in several standard ways; c) cause a subset of concepts to be shown or highlighted according to the setting of a ‘mask’.

In addition to the above, there is a control panel facility that allows the user to specify general preferences as well as load and save knowledge bases.

In order to modify knowledge, users open a knowledge base (or request to start a new one). They then edit the knowledge using one or more of the mediating representations. In order to query and explore the knowledge they can do one of the following: a) Use the masking facilities to pick knowledge of interest; and, b) open subwindows that show concepts and relations of interest. When used together, these facilities give the user quite a rich knowledge management environment.

In order to provide a basis for both the effectiveness and the analytic evaluation (sections 4 and 5) we performed a circumscribed study of a set of 12 users as they developed 25 knowledge bases. This section describes that process.

The basic steps in performing the user study were: 1) Solicit people to build knowledge bases; 2) Train them; 3) Perform any necessary enhancements to the system in order to accommodate their needs, and 4) Study the results of their knowledge base building efforts. Details of these steps follow:

Some users were solicited from the University of Ottawa student or researcher population – people who had to perform tasks for which CODE4 was suitable. Users from companies and other universities also started using the tool after they found out about the product through various publications and contacts. Section 4 discusses tool usage in more detail.

A 100-page user manual was created to train users. Additionally, users received training: 1) from graduate lectures, 2) by following a step-by-step tutorial created by members of a particular user group, and 3) by personal consultation with members of the development team.

It was not possible to have CODE4 remain static during the research. Significant new features were added over a three-year period in order to meet the requirements of users. The kinds of features added after CODE4 started to be used seriously include:

A significant percentage of CODE4 users were willing to have their work studied in detail. These users are called the participants in the following discussions.

The total number of participants was twelve; all used CODE4 at the University of Ottawa. Eleven were graduate students and one was a professor. Student usage was divided between those using CODE4 for course work, and those using it in thesis research.

The participants created 25 knowledge bases. These covered a very wide range of topics, categorized as follows:

The total amount of work involved in creating these knowledge bases was reported to be about 2000 hours, i.e. an entire person-year.

The work of the participants was studied in two ways:

The questionnaire contains six parts containing a total of 55 main questions, many of which have sub-questions: There are a total of 259 sub-questions about CODE4 in general and 130 sub-questions about each knowledge base. Most participants took several hours to complete the questionnaire and some were quite enthusiastic about it despite the amount of work involved.

The first two sections of the questionnaire ask general questions in order to ascertain how much users know about CODE4, how they learned it and how much they have used it. The subsequent two sections focus on the users’ reactions to specific features of the software. The fifth section deals with users’ experiences developing individual knowledge bases. Questions in this section were designed to complement the measurement of the knowledge bases. The final section of the questionnaire covers problems and desires users encountered in the use of CODE4.

Data resulting from the questionnaire is presented in sections 4 and 5.

The evaluation methodology described above is that of a natural experiment (i.e. trying to reach conclusions about an activity by observing as many cases of that activity as possible, but without interfering). An alternative approach would have been to design a controlled experiment.

In the field of knowledge engineering, a controlled experiment might involve a procedure like the following: First, create two versions of a knowledge management system that differ according to the presence or absence of some feature. Then have separate sets of users create knowledge bases with each system. Finally, measure the resulting knowledge bases to see if the presence or absence of the feature causes any difference in the resulting work.

Such an experiment involves varying one parameter (in this case a feature) while ensuring that differences in other factors, including ongoing changes to the system, do not influence any conclusions. This was judged impractical for the following reasons:

In conclusion, without a very significant amount of money with which to pay participants for long periods of time, it was only feasible to perform an exploratory study of the type described in the previous subsections. We believe, however, that such a study provides adequate information to allow us to effectively evaluate the tool.

In this section we discuss our assessment about whether CODE4 was successful in terms of whether it met the objectives set for it. The rationale behind the objectives themselves are described in section 2.2.

The first evaluation criterion, as described in section 2.2, is: Is the tool used on a discretionary basis by significant numbers of members of its target audience and on a variety of different and significant examples of the intended task?

A total of at least 30 individuals are known to have used CODE4 to build knowledge bases. These fall into three main groups: 1) industrial customers, 2) researchers and 3) graduate students taking courses. In addition, about ten other groups are known to have obtained copies of a demonstration version. Some examples CODE4’s use are described below; all of these involved people voluntarily choosing to use the tool.

The primary industrial customers for CODE4 have been groups headed by Jeff Bradshaw, then of Boeing Aircraft Corporation in Seattle. Bradshaw’s work primarily involved building CODE4 into other applications. In Bradshaw, Boose, Shema, Skuce and Lethbridge (1992) an application involving design rationale capture is presented. Another application involving organizational modelling is discussed in Bradshaw, Holm et al. (1992).

CODE4 was aCCODE4 was also adopted by the Cogniterm project. In this project, linguists used it for terminology research – their goal was to find ways to model multi-lingual terminology to assist translators. Some of the published literature includes Eck’s M.Sc. thesis (1993) and papers by Skuce and Meyer (Skuce 1993; Meyer, Eck and Skuce 1994). Bowker (Bowker and Lethbridge 1994a and 1994b) used CODE4 as a major tool for her Ph.D. research.

Several other student research projects and theses have used CODE4 as their major tool; these include Ghali (1993) and Wang (1994). CODE4 has also been used as a teaching tool in several courses in artificial intelligence at the University of Ottawa.

We can conclude from the above that CODE4 has achieved its first objective. The only minor counter-argument is that many of the users completely ignored certain features. However, as stated in section 1.1, we want to evaluate success of the tool as a whole and then subsequently ask question about which features contributed to this success. The latter is discussed in section 5.

The second criterion to ascertain success is: Is the system perceived to be useful, i.e. at least as useful as other techniques that could be used for the task?

To answer this question we posed three subjective questions to the participants of the study described in section 3. The first two questions ask about usefulness in general and the third compares CODE4 to other technologies.

In almost all instances, the participants report that they obtained substantial insight about the subject matter when using CODE4. On a scale where 0 indicated that constructing the knowledge base resulted in no new insights and 10 indicated that many new insights were obtained, the mean response was 7.1 (range 2 to 10; std. dev. = 2.6).

Participants were asked, for each knowledge base, whether creating it was a worthwhile exercise. Answers were on a scale where -5 meant ‘strongly disagree’, 0 meant ‘no opinion’ and 5 meant ‘strongly agree’.

The mean response was 4.2 (range 0 to 5; std. dev. 1.5), indicating a reasonably high feeling that using CODE4 was worthwhile. Users however gave significantly different answers for different knowledge bases: As would be expected, those knowledge bases used for serious research were judged more worthwhile.

For each knowledge base they developed, each participant was asked to compare CODE4 with other representational technologies with which he or she was familiar. The answers were on a scale where -5 meant that the knowledge could have been represented much more easily using the alternate technology than using CODE4; zero meant that CODE4 equalled the technology in representational ease, and 5 meant that using the alternate technology instead of CODE4 would have resulted in much more difficulty (i.e. that CODE4 was much better for the task than the alternative technology would have been be).

In all cases, the mean rating indicates CODE4 is easier for the task than alternative technologies. Table 2 summarizes the results and figure 2 shows a plot of the results. In particular it is notable that CODE4 was perceived as easier to use than both informal representation techniques (e.g. natural language) and formal representation techniques (e.g. predicate calculus and conceptual graphs). Hypertext is CODE4’s only significant competition: One user ranked hypertext as far easier than CODE4, but on average, users still found hypertext a bit harder to use for knowledge management.

Note that the figures only consider cases where participants actually know the competing technology; for example less than half of the respondents compared CODE4 to other AI tools. Also note that the figures do not mean that the competing technologies are less good; the only conclusion that can be drawn is that there is evidence that for this task, CODE4 performs better.

We can conclude from the significant numbers of discretionary users of the tool and the positive responses of users in our study, that CODE4 has met the objectives outlined in section 2.2. There would seem, therefore, to be strong evidence in favour of our hypothesis that a simple tool for the manipulation of concepts and properties would prove useful for domain specialists.

As mentioned in section 1.1, however, this does not inform us about exactly what led to success – this is discussed in the next section.

Having concluded that CODE4 has met its objectives, we now move on to the second phase of evaluation: Determining what aspects of the tool are the key contributors to that success, and, conversely, what aspects of the tool may be of lesser use, or even detrimental to success.

In this section we present the evaluation of the knowledge representation and user interface features.

We analysed knowledge representation features using the following techniques: a) Studying the knowledge bases to see which representation features are actually used; b) Studying how, in the opinion of users, each feature helps convey the content of each knowledge base, and c) Studying the overall opinions of users about each feature.

The knowledge bases created by the participants can be analysed by measuring various attributes. Table 3 shows counts of the various classes of concepts in CODE4 knowledge bases.

The following are some general observations arising from the table 3:

In addition to the above counts of concepts, various metrics were developed to better characterize knowledge bases. These are described in Lethbridge (1998). While most of the metrics are out of the scope of this paper, two are particularly relevant:

For each knowledge base, users were asked to estimate how much each feature of the knowledge representation contributed to conveying the actual content of the knowledge base (i.e. to expressing its most important information).

The results are shown in tables 4 and 5. Table 4 lists the 14 knowledge representation features about which users were asked. Note that the features are not necessarily disjoint; e.g. users were asked about values (of statements) in general as well as particular types of value.

The features are listed in decreasing order by mean, however there is no statistically significant difference between features whose means are close together. An analysis of where statistically significant differences do exist showed that the features can be divided into eight groups (i.e. equivalence classes), labelled A through H in tables 4 and 5. Table 5 can then be used to determine which groups convey significantly more knowledge than others. For example none of the aspects within group A were considered more important than others, whereas all of group A was considered more important than all of the others (groups B through F). Similarly the single aspect in group B is only more important, in the opinion of participants, than aspects in groups D and F, but not more important than any of the aspects in group C.

Some general observations about Tables 4 and 5:

Users were asked how useful they perceived various knowledge representation features to have been in their work. This set of questions differs from the last set in the following ways:

Data for this set of questions is found in tables 6 (features rated high) and 7 (features rated low). Note that features that were given an intermediate rating are excluded for compactness – their contribution to success or failure of the tool as a whole is considered likely to be neutral. The following are some comments:

This section continues the analytical evaluation of CODE4 by considering user interface features. Two types of questions were posed in order to perform the evaluation: The first type of question asked subjects to give their general opinion about the usefulness of features; the second type of question probed how much they used certain mediating representations, where alternatives were available.

In a similar manner to the questions about knowledge representation features, users were asked a series of questions about user interface features. They used the same scale as previously (where -5 means harmful and 5 means essential).

Most of the features about which questions were asked are either novel in CODE4 or are emphasized far more in CODE4 than in other knowledge management technology. Data is found in tables 8 and 9; the following are some specific observations:

The matrix representation was used relatively little, although it was developed after some users had started their work. Also, it is not (yet) possible to start creating a knowledge base using a matrix window; it can only be used as a query mechanism or to fill in missing values.

Table 10 gives the data about mediating representation usage. The statistics for the ‘user language’ mediating representation and the outline representation are likely to be too low and high respectively. This is because many users are not aware that the user language representation is considered distinct from the outline representation (the former, which is used only for entering a single property value, is usually associated with an outline but can be associated with a graph).

The previous two sections have separately analysed the various features of CODE4 to determine which are apparently most useful to the system’s success. This section continues the analysis by looking at various aspects of the work they performed using the tool – tasks that involve both knowledge representation and user interface.

For each knowledge base, users were asked how difficult various tasks were to perform. Difficulty was ranked on a scale where zero indicates extremely easy and ten indicates extremely difficult.

The data is shown in tables 11 and 12. Table 11 orders tasks by mean. Table 12 shows which groups of tasks were significantly more difficult than others. The following are general observations:

Participants were asked to judge the degree of formality of their knowledge bases on a scale where 0 means completely informal and 10 means completely formal. The mean response was 4.8 with a standard deviation of 2.7. Responses ranged from 0 to 9 for the 24 knowledge bases from which responses were received. Interestingly, individual participants gave very different responses for different knowledge bases they created. It thus appears that informality and formality are both necessary, although as was discussed earlier, users feel the informal aspects might convey more knowledge.

Participants were also asked to indicate how rapidly they would represent an idea when it occurred to them. The principle here is that informal capabilities should allow users to rapidly record their thoughts, if they so wish. The scale used was as follows: 0 means that they would represent the idea immediately and later on worry about whether it was correct; and 10 means that they would never represent the idea until they were absolutely sure they could do it correctly. The mean response was 4.2, with a range of 0 to 8 and a standard deviation of 2.4. Being in the middle of the range, this suggests that users need both facilities to rapidly enter ideas, and also facilities to rapidly help them decide whether they should enter an idea.

Users were asked what percentage of main subjects or properties did not have standardized terms; i.e. they had trouble putting a term on a concept, deciding if two terms had the same meaning or even what a term meant. The mean response was 29% (range 10% to 90%; standard deviation 22%). This indicates naming is not a straightforward task, and justifies CODE4’s attempts to provide better facilities to handle names.

As table 11 shows, users found naming properties to be significantly more difficult than naming main subjects.

Users were also asked what percentage of main subjects and properties had a name that the participant invented, as opposed to one that might be found in an ordinary dictionary or technical glossary. The mean was 17% (range 0% to 70%; standard deviation 18%).

The following summarizes the results of evaluating the importance of features – the process we call analytic evaluation. It is important to note that many features of CODE4 are interdependent. For example, users did not directly judge the uniformity of concepts to be important; they nevertheless found facilities to deal with terms, metaconcepts and properties to be useful. Treating concepts uniformly allows the latter to be designed more easily; such treatment also facilitates browsing and other useful capabilities.

On the other hand, there were a variety of features that we judge did not contribute to the success of the tool. These include instance concepts, combination, delegation and a variety of formatting options. These nevertheless involved considerable development work.

The following needs were identified during the evaluation.

This paper has presented the evaluation of a knowledge management tool called CODE4, as well as techniques for performing and presenting such evaluations in general. The design of the tool was presented in terms of task analysis, objectives and design rationale. The evaluation was then split into two sections: Effectiveness (judging success) and analytic (determining what led to success or failure).

The tool was judged to be effective because a) users from the target population (domain specialists) regarded developing their knowledge bases to be a worthwhile exercise; b) the users used CODE4 repeatedly and produced significant knowledge bases with it, and c) the users judged CODE4 to be easier to use than other tools for knowledge management.

With regard to the reasons for success, it appears the most important contributors were features that allowed easy direct-manipulation interaction with hierarchies, the names of items and a combination of formal and informal knowledge.