One human-computer interaction approach to the design of tools has been to study the cognitive processes of programmers as they attempt to understand programs [19, 20, 21]. The results of such studie s are supposed to provide the basis for designing better tools. In other words, understanding the mental processes involved in programming will permit the design of tools that mesh with the programming process.

In this vein, ESP research has identified a number of programmers’ approaches to the comprehension ‘problem’ including the top-down [12], bottom-up [4], and as-needed strategies [9], and the integrated meta-model [21].

There are three problems with this research, though, as it pertains to tool design. First, the vast majority of the research has been conducted with graduate and advanced undergraduates serving as expert programmers (but c.f., [21]). It is not at all clear that these subjects accurately represent the population of industrial programmers. Consequently, the results of studies involving students cannot be generalized to programmers in industry.

Second, to control extraneous variables, researchers have used programs that are very small (both in terms of lines of code and logic) relative to industrial software. This poses a generalization problem as well: it is not clear that ap proaches to comprehending small programs scale up to the comprehension of very large programs.

Third, there is an assumption that understanding the programmer’s mental model is an efficient route to designing effective tools. However, it is not at all obvious how to design a tool given a specification of the programmer’ s mental model. For instance, how does knowing that programmers will sometimes use a top-down strategy to understand code [12] inform tool design? It doesn’t tell us what kind of tool to build, or how to integrate that tool into the workplace or the programmer’s work. Furthermore, given this knowledge, it is not clear how to help the programmer build that mental model; how to help her apply it; or how to help her use it effectively in software engineering activities.

These three problems with the ESP approach suggest that an alternative approach to tool design may be more effective.

Currently, there is a strong focus on usability in the field of human-computer interaction [10]. That is, designers attempt to ensure that prospective users can use the software without encountering interface difficulties. For instance, it should be clear to users what action they should take at each step, preferably without referring to documentation. Another aspect of usability is the minimization of the number of steps and the amount of time neede d to accomplish a task. To determine whether software is sufficiently usable, prospective users are observed using the software for a few, or several minutes. Reaction times, errors, backtracking to previous states and failures to accomplish the task are recorded along with the conditions under which they occurred. These data are then used to fix the interface and, ideally, more of these test-redesign iterations take place until the software is sufficiently usable.

However, we see problems with this approach. While it may increase the usability of systems, it does not guarantee that the systems that are built will be genuinely useful [2, 18]. The usability approach cannot speak to the issue of whether a user will adopt and use a new tool in the workplace because that is not the point, or the focus, of usability. Moreover, several features of the usability approach prevent it from informing the designers about the acceptance of the tool in t he workplace. Usability testing usually takes place outside the normal work setting, sometimes in a room especially designed for that purpose. This method of testing prevents the user from behaving in a normal manner because it isolates him from resources that are not part of the software (such as colleagues, documentation, notes). In other words, it prevents the user from engaging in his day-to-day work practices. In addition, during usability testing, the user is essentially forced to use the software. In consequence, it is impossible to collect data on whether the user would use the software if he were given a choice between his existing work practices and the new software.

The lack of tool adoption and use is a major problem in the area of tool design for software engineering. However, because of its features and techniques, usability cannot inform designers on this issue. We believe that to build tools t hat are actually used, designers must first understand what it is that SEs do when they work. This is the reason for our focus on work practices in designing software engineering tools.

The system includes a real-time operating system and interacts with a large number of different hardware devices. The system contains several million lines of code with over 16000 routines in over 80 00 files. It is also divided into numerous layers and subsystems written in a proprietary high-level language.

The system was first fielded in the early 1980s and has since been continually updated. Its importance to the company and its evolution are expected to continue for many years to come.

Approximately 13 people actively work on various aspects of the system at the current time. Over 100 people have made changes to the source code during the life of the system.

We began this research by administering a web-based questionnaire. The questionnaire covered many different aspects of the SEs’ work. Here we report their answers to a question on what they spen d their time doing. Six SEs in the group of 13 responded. The question was open-ended, i.e., the SEs had to decide how to describe their work, rather than choosing certain activities from a list.

On average, SEs said that they spend 57% of their time fixing bugs, and 35% of their time making enhancements to the system. Table 1 shows more specifically the things they reported that they engaged in, and the percentage of people rep orting that activity.

The most reported activity was reading documentation. SEs also reported that they spend time looking at source, writing documentation, attending meetings, and writing code. Other activities include consulting, both answering and asking questions, working with the hardware, testing, designing, and fixing bugs.

Because of the questionable validity of self-reports, we felt it was extremely important to not just rely on what SEs said they did, but to actually observe them as they worked. Hence the next sections of the paper describe two studies that we undertook towards this goal.

We have been following one SE longitudinally from the time he joined the company (November, 1996). For the first six months, we spent about 1-1/2 hours per week with B. However, as B has become more expert, we have found that it makes more sense to meet once every 3 weeks. This is both because new things happen less frequently (e.g., experience with a new tool) and because B is more busy with ‘real’ tasks. B is an experienced SE (was previo usly a team-leader), thus while he is new to the company, he is certainly new to neither maintenance nor telecommunications software.

Our sessions with B consist of 3 distinct components. First we talk about what has transpired since the last time we met. This could be anything from code review to learning about a new tool to reading documentation, etc. Second, we ask B to look at a diagram of the system he previously constructed and ask him to modify it if it does not reflect his current understanding. Finally, we ‘shadow’ B as he works for 1/2 hour. In this paper, we report the data from the shadowing.

1. Method

1. Subject

B has worked in the software industry for many years. Prior to joining the telecommunications company, he worked as a team leader for a nearby competitor. There, B maintained a product in the same category as the current product, bu t developed on a much smaller scale.

B has experience in several languages, but prior to joining the company, considered himself to be an expert only in an in-house proprietary language. Likewise, while he has experience in several platforms, prior to joining the company, B considered himself to be an expert only in an in-house proprietary 68K development platform. B has worked on 5 different systems, 3 of which have involved development, 2 of which have involved maintenance.

B joined the company in November, 1996. Before then B had no experience in the company’s in-house Pascal-based proprietary language. Nor did B have any experience in Pascal, although he had programmed in other structured languages. B had utilized VI before coming to the company, but planned on switching to the Emacs editor at the company. Similarly, he had used Grep previously, but was switching to use of Egrep and Fgrep at the company. B did not have previous experience with the o ther tools available at the company

2. Procedure and Data

The shadowing data result from 14 half-hour sessions ranging from October 17, 1996 to February 27, 1997. Some days are missing because of vacation or schedule conflicts. For the most part, however, these dates reflect weekly meeting s with B.

For half an hour, we would sit behind B and write down the things he did. For instance, if he used Grep, that would be recorded (using pencil and paper). If he read documentation, or wrote notes to himself, that was written down.

We recorded B’s activities in detail, but not to the point of exactly what he typed or said. For example, we would record that B edited a file or interacted with the hardware, while not detailing the exact nature of his involvement with these activities.

A new activity was recorded each time a switch in activity occurred. So, for instance, if B did 6 Greps in a row, that was recorded as a single instance of the event Grep. Then if he did 4 Diffs, a single Diff event would be recorded. T aking that to its extreme, if all B did was Grep for 1/2 hour, that is the single activity that would have been recorded for that 1/2 hour. No time measures were taken. Thus, we do not know the duration of B’s involvement in each distinct event. We f ollowed the shadowing procedure regardless of the nature of B’s work. Sometimes that meant that we observed B reading documentation only. Other times B was engaged in a wide variety of tasks.

As a general note, there is probably some self-selection of activities involved in B’s choices of things to do. For instance, it is highly unlikely that B would have chosen to respond to personal email when he was being shadowed by us. As a rule, he was always directly involved in work activities. We do not consider this to be too much of a problem, however, because our goal is, after all, to build tools that help SEs work.

2. Results

The shadowing events were categorized into 14 distinct categories which are described in Table 2. Each of B’s events was then classified as belonging to one of these event categories.

First, Figure 1 shows the percentage of days (for a 14 day span) on which an event occurred at least once. For example, if B searched for information one day, the search count would be incremented by 1, regardless of whether B searched 1 time, 4 times, or 24 times on that particular day.

Searching and interacting with the hardware were the most likely events to occur on a daily basis, each occurring on 8 of the 14 days. B looked at the source code on 6 of the 8 days. The reason that B searched on more days than he looke d at the source code is because searching was an activity that also occurred when interacting with the hardware and debugging. B only looked at documentation on 2 of the 14 days. This is surprising because, at the time, B was still a relative novice to th e software system and it is commonly assumed that novices will spend much of their time reading the documentation to get a handle on what they are doing. The data show that this was not the strategy B pursued. However, because B was a novice, it was not s urprising to find that , editing code, compiling, and management were each only done on 1 of the 14 days.

The data were then examined in two distinct ways.

Figure 2 shows the proportion of each event type out of the total of 156 distinct events. Unlike Figure 1, Figure 2 shows the total count, so that if B searched 8 times on one day, that is counted as 8 instances of search.

Again, we see that overall B searched more often than he did anything else (37 times). He also frequently looked at the source code (33 times). While B was likely on any particular day to work with the hardware (see Figure 1), he did so on only 22 distinct occasions.

Remember, however, that these data do not include time measurements, but simply activity switches. So, for instance, while B did management activities on only 1 day, the code review that was undertaken took the entire 1/2 hour.

Thus overall, in terms of both daily activities and frequency of different activities, search for information about the system, whether through Grep, in-house search tools, or within a particular editor or debugger, figures most promine ntly. A significant amount of effort was also expended interacting with the hardware and looking at the source code.

To generalize our findings, we have conducted several studies that focus on different aspects of the work of an entire group of SEs.

We have collected four types of data from the group. First, we asked the SEs to draw a diagram or picture of their current understanding of the system, a conceptual map, if you will. Second, we conducted intensive interviews with the SE s as they solved a real problem with the software. This generally involved 1 hour interviews over the course of several days. Third, we asked the SEs to recount how they solved a recently encountered problem. Finally, we spent one hour shadowing each SE a s they went about their work. This report focuses on this fourth type of data; the shadowing data.

1. Method

1. Subjects

Eight group members participated in the shadowing study. Their experience ranged from the most expert member of the group (8 years) to the least experienced (6 months - recent college graduate). All but one of the shadowed subjects worked on the main controller of the hardware. One of the subjects worked primarily on the database component.

The subjects were expert in a wide variety of platforms and languages, and had experience in both development and maintenance environments.

2. Procedure and Data

The shadowing occurred in the same manner as for B: we sat behind the SEs and recorded the activities they engaged in. Again, a new activity was recorded when there was a switch in activity, so 9 Greps in a row counted as one instan ce of the activity search. Durations of activities were not recorded.

We recorded activities in gross, not fine, detail; e.g., we did not record the arguments to particular commands.

Shadowing schedules were not chosen to reflect any particular activity, but rather were scheduled at times convenient for the SEs. Shadowed times were relatively free from stress, i.e., SEs were not shadowed as deadlines approached.

Again, there is probably some self-selection involved in the activities that the SEs pursued. However, it was very clear that they were all working on ‘real’ problems as evidenced by their concern with the problem report’ s contents.

2. Results

Like B’s data, the shadowed events were categorized into 14 distinct categories which are described in Table 2. Each of the events was then classified as belonging to one of these event categories. 356 distinct events were reco rded.

Figure 3 shows the proportion of users who engaged in a particular type of activity at least once during the shadowed hour. All 8 SEs looked at the source, conducted a search, and changed the source code at least once during the hour. M ost of the SEs also engaged at least once in several other activities, with 5 of the 8 SEs interacting with the hardware, debugger, or the in-house tools. On the other hand, only 3 SEs looked at a call trace, while only one SE performed a management activ ity.

Figure 4 shows the percentage of times a particular type of event occurred out of the total of 357 events (totaled over the 8 SEs). Issuing a UNIX command was the most frequent activity, occurring 54 times. A close second was looking at the source which was done 52 times. Interacting with the hardware or the debugger, searching, or changing the source code was done on 36, 32, 31, and 30 occasions respectively. Configuration management, consulting, compiling, and looking at in-house tool s were each done about 20 times.

Surprisingly enough, reading the documentation, although done by 6 of the 8 SEs accounted for only 12 separate events. Clearly, the act of looking at the documentation is more salient in the SEs’ minds (as evidenced by the question naire data) than its actual occurrence would warrant.

SEs only occasionally wrote notes, looked at the call trace or did management activities. This is not to say that these events are not important, but merely that they did not occur as frequently as other events.

As B did, the group frequently examined the source code. Every SE in the group made at least one search during their shadowing session, but search was less prominent than in B’s activities. Search ranked as the most frequent event type for B, while it was the 4th most frequent for the group.

Code editing and compiling were more prominent activities in the group data. This is probably because B was still learning the system at the time we shadowed him, so he was not yet in a position to make many changes. This may also expla in the higher prominence of call trace in his data: call trace may be effective in gaining an initial understanding of a system.

Interestingly, in-house tools and documentation were both relatively infrequent activities for both the group and B.

The group data converge with B’s data to suggest that looking and searching through the source code are prominent activities for SEs. Editing and compiling also seem important. This concurs with what we would expect in that the cod e is the focus of their work.

The final study we report concerns company-wide tool usage statistics. These data were obtained from the company’s tool group. This group is responsible for acquiring, updating, and maintaining the company’s tools. Collecting usage statistics is part of their mission.

1. Results

The data presented here represent one week of Sun tool usage by 367 users in late May. Note that this week occurred before ‘vacation season,’ so is fairly representative of peak tool usage. There were 79,295 separate tool calls logged from the Sun operating system. Each call counts as one usage event. These tool calls were classified according to the scheme presented in Table 3.

Figure 5 shows the proportion of times that each type of tool was used. Compilers, which accounted for 32,422 calls, or 41% of all calls are not included in this graph. This is because the compiler data include all the automatic softwar e builds done nightly and by the various testing and verification groups. These data are therefore not representative of the SEs real work practices.

The overwhelming finding from the company data is that search is done far more often than any other activity. In fact, search accounts for 21,146 events over the course of the week, or an average of about 58 searches per individual user . Compression and un-compression tools are also used often. We never actually observed anyone using these tools. Perhaps they are used by the verification groups.

The configuration management system was activated 2819 times, accounting for approximately 4% of all events. At this company, the configuration management system is central to the work process, both for retriev ing files, filing changes, and searching through past changes (along with associated documentation).

Editors and viewers account for approximately 3190 events, or 4% of the total number of events. This low frequency could be due to counting particularities that apply only to editors. In the company tool data, an editor command is count ed only when the editor is opened. Once an editor is open, it generally stays open, regardless of how many changes are made, or how many files are viewed. In contrast, in the shadowing data, an edit was recorded each individual time the source was changed , and a source event was counted each time the source was examined, whether the editor was already open or not. Consequently, it comes as no surprise that the shadowing data edit and source frequency is higher than that of the company data.

Again, the in-house tools are not used very frequently, but that belies their importance. These tools are important because they perform necessary functions that cannot be performed by other tools.

Search is the most frequently used tool at the company wide level. Grep and its variants are the most frequently used search tools, accounting for 21,117 separate invocations. Clearly, search is an important aspect of SEs work practices .

Almost all the SEs we have studied spend a considerable proportion of their total working time in the task of trying to understand source code prior to making changes. We call the approach they use < I>Just in Time Comprehension (JITC) [14]; the reason for this label will be explained below. We choose to focus our research on this task since it seems to be particularly important, yet lacking in sufficient tool support.

The ‘changes’ mentioned in the last paragraph may be either fixes to defects or the addition of features: The type of change appears to be of little importance from the perspective of the approach the SEs use. In either case t he SE has to explore the system with the goal of determining where modifications are to be made.

A second factor that seems to make relatively little difference to the way the task is performed is class of user: Two major classes of users perform this task: Novices and experts. Novices are not familiar with the system and mu st learn it at both the conceptual and detailed level; experts know the system well, and may have even written it, but are still not able to maintain a complete-enough mental model of the details. The main differences between novice and expert SEs are tha t novices are less focused: They will not have a clear idea about which items in the source code to start searching, and will spend more time studying things that are, in fact, not relevant to the problem. It appears that novices are less focused merely b ecause they do not have enough knowledge about what to look at; they rarely set out to deliberately learn about aspects of the system that do not bear on the current problem. The vision of a novice trying to ‘learn all about the system’, therefo re seems to be a mirage.

As described in section 3, we observe that SEs repeatedly search for items of interest in the source code, and navigate the relationships among items they have found. SEs rarely seek to understand any part of the system in its entirety; they are content to understand just enough to make the change required, and to confirm to themselves that their proposed change is correct (impact analysis). After working on a particular area of the system, they will rapidly forget details when they move to some other part of the system; they will thus re-explore each part of the system when they next encounter it. This is why we call the general approach, just-in-time comprehension (JITC). Almost all the SEs we have studied confirm that JITC accurately describes their work paradigm — the only exceptions were those who did not, in fact, work with source code (e.g. requirements analysts).

As a result of our work-practices studies (section 3), we have developed a set of requirements for a tool that will support the just-in-time comprehension approach presented in the last section. Requ irements of relevance to this paper are listed and explained in the paragraphs below. Actual requirements are in italics; explanations follow in plain text.

The reader should note that there are many other requirements for the system whose discussion is beyond the scope of this paper. The following are examples:

• Requirements regarding interaction with configuration management environments and other external systems.

• Requirements regarding links to sources of information other than source code, such as documentation.

• Detailed requirements about usability.

Functional requirements. The system shall:

The SEs we have studied do this with high frequency. In the case of a file whose name they know, they can of course use the operating system to retrieve it. However, for definitions (of routines, var iables etc.) embedded in files, they use some form of search tool (see section 4.3).

We have observed SEs spending considerable time looking for information about such things as the routine call hierarchy, file inclusion hierarchy, and use and definitions of variables etc. Sometimes they do this by visually scanning source code, other times they use tools discussed in section 4.3. Often they are not able to do it at all, are not willing to invest the time to do it, or obtain only partially accurate results.

This requirement has come about because we observe users working on multiple problems and subproblems over a span of many days. We also observe them losing information they had previously found and r edoing searches.

Non-functional requirements. The system will:

As we are concerned with systems that are to be used by real industrial SEs, an engineer should be able to pick any software system and use the tool to explore it.

This is one of the hardest requirements to fulfill, but also one of the most important. In our observations, SEs waste substantial time waiting for tools to retrieve the results of source code querie s. Such delays also interrupt their thought patterns.

The SEs that we have studied use at least two languages — a tool is of much less use if it can only work with a single language. We also want to validate our tools in a wide variety of software engineering environments, and hence must be prepared for whatever languages are being used.

We want to be able to connect our tools to those of other researchers, and to other tools that SEs are already using.

We want to perform separate and independent research into user interfaces for such tools. This paper addresses only the overall architecture and server aspects, not the user interfaces.

It is important for acceptance of a tool that it neither represent a step backwards, nor require work-arounds such as switching to alternative tools for frequent tasks. In a survey of 26 SEs [7], the most frequent complaint about tools (23%) was that they are not integrated and/or are incompatible with each other; the second most common complaint was missing features (15%). In section 4.3 we discuss some tools the SEs already use for the program comp rehension task.

Some information in software might be described as ‘latent’. In other words, the software engineer might not see it unless it is pointed out. Examples of such information are the effects of conditional compilation and macros.

This simplifies implementation decisions without unduly restricting SEs.

We are restricting our focus to SEs working on large bodies of legacy code that happens to be written in non-object-oriented languages. Clearly, this decision must be subsequently lifted for the tool to become universally useful.

This is the purview of debuggers, and dynamic analysis tools. Although it would be useful to integrate these, it is not currently a requirement. We have observed software engineers spending considera ble time on dynamic analysis (tracing, stepping etc.), but they consume more time performing static code exploration.

There are several types of tools used by SEs to perform the code exploration task described in section 4.1 This section explains why, in general, they do not fulfill our requirements:

Grep: Our studies described in section 3.4 indicated that over 25% of all command executions were of one of the members of the Grep family (Grep, Egrep, Fgrep, Agrep and Zgrep). Interviews show th at it is the most widely used software engineering tool. Our observations as well as interviews show that Grep is used for just-in time comprehension. If SEs have no other tools, it is the key enabler of JITC; in other situations it provides a fall-back p osition when other tools are missing functionality.

However, Grep has several weaknesses with regard to the requirements we identified in the last section:

• It works with arbitrary strings in text, not semantic items (requirement F1) such as routines, variables etc.

• SEs must spend considerable time performing repeated Greps to trace relationships (requirement F2); and Grep does not help them organize the presentation of these relationships.

• Over a large body of source code Grep can take a large amount of time (requirements NF1 and NF2).

Search and browsing facilities within editors: All editors have some capability to search within a file. However, as with Grep they rarely work with semantic information. Advanced editors such as Emacs (used by 68% of a total of 127 users of text-editing tools in our study) have some basic abilities to search for semantic items such as the starts of procedures, but these facilities are by no means complete.

Browsing facilities in integrated development environments: Many compilers now come with limited tools for browsing, but as with editors these do not normally allow browsing of the full spectrum o f semantic items. Smalltalk browsers have for years been an exception to this, however such browsers typically do no not meet requirements such as speed (NF2), interoperability (NF4), and multiple languages (NF3). IBM’s VisualAge tools are to some ex tent dealing with the latter problem.

Special-purpose static analysis tools: We observed SEs using a variety of tools that allow them to extract such information as definitions of variables and the routine call hierarchy. The biggest problems with these tools were that they were not integrated (requirement NF6) and were slow (NF2)

Commercial browsing tools: There are several commercial tools whose specific purpose is to meet requirements similar to ours. A particularly good example is Sniff+ from Take5 Corporation [17]. Sni ff+ fulfills the functional requirements, and key non-functional requirements such as size [NF1], speed [NF2], and multiple languages [NF3]; however its commercial nature means that it is hard to extend and integrate with other tools.

Program understanding tools: University researchers have produced several tools specially designed for program understanding. Examples are Rigi [11] and the Software Bookshelf [5]. Rigi meets many of the requirements, but is not as fast [NF2] nor as easy to integrate other tools [NF6] as we would like. As we will see later it differs from what we would like in some of the details of items and relationships. The Software Bookshelf differs from our requirements in a key way: Before somebody can use a ‘bookshelf’ that describes a body of code, some SE must organize it in advance. It thus does conform fully with the ‘automatically’ aspect of requirement NF1.