The full responses are listed in Appendix D, split into the response elements used for the content analysis. For ease of reference, each category has been given a two-part number of the form x.y, where x is the category tag allocated by the evaluator and y is the number of the interviewee. Throughout this section and Appendix D, the following abbreviations will be used:
CUG The Commercial User Group.
AUG The Academic User Group.
CUx Commercial User x (where x=1 to 5).
AUx Academic User x (where x=1 to 5).
As mentioned in the previous chapter, one evaluator was a media studies graduate, the other a psychology graduate working as a researcher in computing. As the background of the evaluators might be relevant when considering distinctions between their results, they will be referred to by following abbreviations:
MSE Media Studies Evaluator
CSE Computing Studies Evaluator
The table below shows the numbers of response elements and the number of categories allocated by the two evaluators.
Table 6.1: Response categories.
| User |
|
|
|
|
|
|
||
| CU1 |
|
|
|
| CU2 |
|
|
|
| CU3 |
|
|
|
| CU4 |
|
|
|
| CU5 |
|
|
|
| Mean |
|
|
|
| SD |
|
|
|
| AU1 |
|
|
|
| AU2 |
|
|
|
| AU3 |
|
|
|
| AU4 |
|
|
|
| AU5 |
|
|
|
| Mean |
|
|
|
| SD |
|
|
|
The most obvious feature of the table is that the AUG shows a higher number of categories and a much greater standard deviation in the numbers of categories between users. However, the number of subjects is far too small for any difference between the subject groups to be significant. It may also be noted that evaluator CSE identified a larger number of categories than MSE in every case apart from AU4 and AU3. Although this might seem important, the results of the Kendall's Tau test in the previous chapter showed that there was a significantly close correlation between the two categorisations.
A possible reason for the distinction between the two sets of evaluations becomes apparent when the results are compared for the degree to which categories are shared across interviewees. The numbers of unique and shared categories for each interview are shown in the table below.
Table 6.2: Shared categories.
|
|
|
|||||||
| unique | shared | total | unique | shared | total | |||
| CU1 | 7 | 11 | 18 | CU1 | 7 | 7 | 14 | |
| CU2 | 4 | 8 | 12 | CU2 | 3 | 4 | 7 | |
| CU3 | 3 | 9 | 12 | CU3 | 8 | 3 | 11 | |
| CU4 | 2 | 11 | 13 | CU4 | 4 | 6 | 10 | |
| CU5 | 11 | 7 | 18 | CU5 | 7 | 4 | 11 | |
| Total | 27 | 46 | 73 | Total | 29 | 24 | 53 | |
| per cent shared | 63.01 | per cent shared | 45.28 | |||||
| AU1 | 8 | 15 | 23 | AU1 | 5 | 8 | 13 | |
| AU2 | 10 | 17 | 27 | AU2 | 5 | 9 | 14 | |
| AU3 | 10 | 13 | 23 | AU3 | 20 | 6 | 26 | |
| AU4 | 1 | 8 | 9 | AU4 | 7 | 4 | 11 | |
| AU5 | 11 | 9 | 20 | AU5 | 5 | 5 | 10 | |
| Total | 40 | 62 | 102 | Total | 42 | 32 | 74 | |
| per cent shared | 60.78 | per cent shared | 43.24 | |||||
It can immediately be seen that both evaluators identified similar numbers of unique categories for each interviewee, totalling 67 for CSE and 71 for MSE. By contrast, the total number of shared categories per interview is 108 for CSE against only 56 for MSE. It is possible that the greater number of categories initially identified by CSE means that each individual category is more basic and therefore more likely to match a simple category from another interviewee. For example, for AU1, CSE allocated "so if you want to make changes" to one category, with "and keep the old changes" to a second category whereas MSE allocated both parts of the phrase to a single category. Where the results of this interview matched those of other interviews, MSE marked them as having one shared category, whereas CSE marked them as two shared categories thus doubling the apparent overlap in content between the interviews.
One question which arises from this is whether, in any sense, one set of results is 'better' than the other. There is no objective reason to suppose this, but CSE's work, background and knowledge of computer systems are closer to those of the interface designers who are the target users for the technique. For this reason, examination of the qualitative data will concentrate on his analysis.
The numbers of users identifying a particular category, as tagged by CSE, are shown in the following graphs. The category tags were allocated by reference to the first interview, adapted to include the second interview and so forth. Ranking categories on this basis would therefore bias them in favour of the ordering by the initial interview in each group. To avoid this, the mean of all the category tags for a given category was calculated and the categories ranked on this basis. For example, if a category was the third category to be introduced by User 1, the fourth by User 2 and the tenth by User 4 then the value used in the ranking would be:
(3 + 4 + 10)/3 = 5.7
Figure 6.1 CUG: users sharing each category.
Figure 6.2 AUG: users sharing each category.
Perhaps the most surprising observation is the similarity in form between the two figures. The mean numbers of users per category is similar for the two groups, at 1.62 for CUG, compared to 1.66 for AUG. In both diagrams there are more common categories at the earlier layers of signification but this effect is not very strong - the gradient of the least square line is -0.015 in both cases. Despite this apparent similarity, comparison of the content of the sets of interviews shows some distinct differences which will be discussed below.
As the table above shows, the CUG interviews uncovered between 12 and 18 layers of signification, while the AUG interviews showed a much greater range, from 9 to 27 layers. However, if AU4 is excluded, the lower figure is 20. Although not the shortest interview, the interview with AU4 was very much shorter than those of the other academic users. He also gave the impression of being somewhat hostile. All other interviewees in both user groups were friendly and interested in the unusual technique.
All interviews followed a similar structure, starting with apparently low level signification and steadily moving towards higher levels with only minor backtracking. Judgement of what consituted a low or a high level of signification was subjective and is only used to illustrate the findings. This will be discussed further in Chapter 7. The interview with CU1 was a typical example, starting, "When you go into that particular screen. You go into 'Navigate' to input an invoice". The user then explicitly backtracked, "That's the first button you press, if you like, before going on to the next fields." CU1 then explained the reason for inputting invoices, "to bill one part of COMPANY to another part of COMPANY," and the reason for doing this, "Because if one part of COMPANY is doing work... then they need to be charged for it." He then moved up to the recording of the information, "So that the books... are... as correct as they can possibly be." His highest level of signification was then given, "It's a requirement under legislation." When asked, "What for?", CU1 ended the interview with, "I don't know." The interviews with AU1 and, to a lesser extent, AU4 showed a minor variation in giving two options (to change the name or to change the format), alternating between higher levels related to the two lower level significations.
All but three of the interviewees ended the interview by saying they could not answer the question, usually with "I don't know". The interviews with CU2, CU4 and AU2 were terminated by the interviewer when the interviewee began to repeat previous answers, slowing down and hesitating as if at a loss as to what to say. For example, CU4 ended by explicitly stating that she was repeating her answers, before returning to her previously stated highest signification, "As I was saying... tasks that I have to do."
Only one category was mentioned by every member of the CUG: 2.1 "you go into Navigate". This is hardly surprising in that 'Navigate' is the name of the command and the user's first action is to 'go into it'. One category was mentioned by four of the five, 8.1 "to bill one part of COMPANY to another part of COMPANY", the main purpose of the system. A number of categories were mentioned by three of the five users:
4.1 "That's the first button you press…" This represents the most basic interaction with the system, although it is expressed in terms of pressing the metaphorical 'Navigate' button rather than the physical mouse button which is actually pressed.
5.1 "…before going on to the next fields…" The 'Navigate' command is used to select the required screen, the following action being to select the required field for data entry or amendment.
6.1 "To input invoices…"
7.1 "We input invoices…" It is not clear why categories 6.1 and 7.1 were distinguished from one another, perhaps merely because the evaluator failed to spot the common signification. In combination these categories were mentioned by four of the five users. Both describe the main process through which the cross-company billing is enacted.
14.1 "It's a requirement…" The element of compulsion is mentioned by three users, with CU1 seeing the compulsion as coming from Oftel, whilst CU2 and CU4 express it more simply: "I do what I'm told to do," and "Because I have to."
5.2 "So you can navigate around the whole system." This category describes the behaviour of the 'Navigate' command.
For the commercial users, the 'What for?' technique was clearly very successful in disclosing the factors which most analysts would see as fundamental in designing the interface. At a practical level of using the system, the majority of the users mentioned the type of interaction (pressing a button), the command chosen (Navigate), the behaviour of the command (moving around the system), the reason why it is necessary to move around the system (to get to the right screen), the reason why they would want to get to a particular screen (to select a field for entry or amendment). A majority of the users also expressed an awareness of the role of the system in the business (cross-company billing) and how this is achieved (invoicing).
In contrast with this level of agreement, the interviews uncovered some interesting distinctions in higher level signification which might give concern. The actual reason why the system was built - "a requirement under legislation and under Oftel that each part of COMPANY should be responsible for their own accounting purposes" - was only mentioned by CU1, the most senior member of the team who supervises the others. I verified that this was the case with the systems analysts who had specified the system. The only other interviewee to express a higher level of signification related to the company was CU5, who justified the system incorrectly: "To provide information to shareholders. To see how good we are doing as a business."
It is widely recognised that understanding the purpose of a task is an important factor in employee motivation. Huczynski and Buchanan (1991, p.73) identify "experienced meaningfulness" as one of the "three psychological states critical to high work motivation, job satisfaction and performance," defining this state as "the extent to which the individual considers the work to be meaningful, valuable and worthwhile." For an employee to consider his or her work to have a worthwhile purpose obviously depends on the employee knowing what that purpose is. This knowledge is also likely to contribute to the third of the three critical states: knowledge of results.
For the academic users many more categories were shared by a majority of the users, with two categories mentioned by all five interviewees:
1.1 "It's for saving the contents of a file…"
8.1 "…if you've got several different versions."
1.1 is a simple statement of the functionality of the command, while 8.1 is one of the possible reasons for using the command. Two categories were raised by four of the five users:
5.1 "And one to change the type of file…"
13.1 "…other kinds of software, other packages…"
Again, these describe another possible reason for using the command. A large number of categories were mentioned by three of the five users:
4.1 "One to change the name to a different name."
9.1 "…of the same file."
16.1 "…you'd want to save…"
18.1 "…so if you want to make changes…"
19.1 "…and keep the old changes…"
21.1 "Historical record."
8.2 "For using it, I mean."
23.2 "…or whatever the purpose of the file is."
Some of these categories deserve examination, such as 4.1 and 5.1 which are not technically accurate. The 'Save as...' command does not change the name or type of a file. It saves a copy of the file under a new name or type; unless overwritten, the original file remains on disc with its original name and type. One possible reason for this will be discussed below.
One important factor to consider in assessing the practicality of the technique for potential users is the time taken for an interview. Times for each individual interview are shown in the table below.
Table 6.3: Interview duration.
|
|
|
||
|
|
Duration |
|
Duration |
|
|
1 min 13 sec |
|
1 min 49 sec |
|
|
0 min 43 sec |
|
3 min 37 sec |
|
|
1 min 20 sec |
|
2 min 35 sec |
|
|
1 min 54 sec |
|
0 min 58 sec |
|
|
2 min 1 sec |
|
1 min 26 sec |
|
|
1 min 26 sec |
|
2 min 5 sec |
On average, the interviews took less than two minutes each, the longest being under four minutes. One potential criticism of the interview technique might have been that, when probed in this way, users would reflect on the interface, inventing layers of signification to please the interviewer. It is clear, however, that the users had little chance to invent significations in this way and that the answers were given with little time for thought. As for the practicality of the technique for use by designers, even with pre-interview set-up, post-interview explanations and occasionally waiting for a user to be free, each group of interviews was completed in less than an hour. In practice this would make no significant difference to the total time taken for effective user requirements gathering and would be negligible in the overall development life cycle.
The 'save' metaphor is now dead for many users, certainly for experienced computing postgraduates such as AUG, and is included as a computing term in recent dictionaries such as Chambers (Schwarz 1988) which lists one meaning as "to store (data) on a tape or disc." It should be noted that this does not preclude the possibility of problems based on its metaphoric origins; as explained below, 'file' is also a dead metaphor but still caused confusion among users. However, the 'save' metaphor appeared to cause no problems among the users, with all members of AUG making statements in line with the dictionary definition as their first significations, such as 1.1: "It's for saving the contents of a file."
For the general population, the 'navigate' command is still a live metaphor. Chambers (Schwarz 1988) does not list it as a computing term and defines the verb as, "to direct the course of: to sail, fly, etc., over, on or through." Whether the metaphor remains alive for the users in this study cannot be answered from the interview results alone, but CUG were generally less familiar with computers than AUG and it seems likely. The general intention of the metaphor is that users should use the facility to steer their way around a system like steering a ship across an ocean, as indicated by the ship's wheel icon used to represent Netscape Navigator 4.0 (Netscape 1998).
CU1 did not mention the functionality of the command apart from the statement, "You go into 'Navigate' to input an invoice," giving no indication that he necessarily understood the meaning of the command. CU2, CU3 and CU4 all referred to the metaphorical meaning in their replies:
CU2 1.2- 7.2 Just press the 'Navigate' and then it takes you to all the other bits. So you can navigate around the system. So it'll get you to different screens to do your work.
CU3 1.3, 2.3 Just to show the system where to go.
CU4 4.4 It takes you on to another route.
The only questionable aspect of this understanding lies in way that both CU2 and CU4 used the phrase, "it takes you," as if they regard the system as being in control rather than themselves. The least experienced of the users, CU5, re-interpreted the meaning of the command as "Find, I'd assume find." Generally, however, the metaphor appeared to work effectively and none of the users apart from CU5 appeared to have any problems in understanding how to use it.
A more important metaphor issue to be considered is the users' understanding of the 'file' metaphor by AUG. Problems with this metaphor could explain the 'false' significations which all of the academic users gave. This could come from confusion as to what a file is, whether an identical copy made to another drive (3.3) is still the same file, whether a file remains the 'same' file if it changes its name (4.1) or its file type (5.1), or if its contents are changed (18.1). Although 'file' was not the term under consideration, the 'Save as...' command is an item on the 'File' menu and the object to be saved can only be a computer file. The 'file' metaphor was used by all users in the AUG, with 28 references to 'files', and only one to 'document'. In order to examine this it is necessary to look at the historical origins of the 'file' metaphor.
Before the widespread use of personal computers, the term 'file' was applied to a collection of computer data metaphorically equivalent to the physical file it replaced. For example, Kilgannon (1980, p. 97) describes equivalent processing methods a systems analyst should consider: manual processing, office machine processing, unit record processing and computer processing. He then goes on to examine manual processing as the first option to be considered, introducing the following list of files and filing equipment.
Table 6.4: Filing equipment (from Kilgannon 1980, p.99).
| Document filing | Book binders (ring or post) Box files Drawer cabinets - folders - suspended folders Lateral filing units Fire resistant cabinets Safes |
| Card filing | Plain boxes (blind filing) Rotary boxes Card wheels Visible edge card trays (Kardex) Vertical ledger card visible records |
| Plan filing | Drawer filing Vertical plan files Lateral plan files |
| Strip index | Book binders Trays Rotary trays |
| Wall charts | Plan boards - visible, magnetic Peg boards Manual entry |
| Manual punched cards | Edge punched Body punched |
At this time computer files were seen as the metaphorical equivalent of the physically organised data within an office environment. The ground for this metaphor lies in the structured organisation of the data so that, for example, typed documents are not listed as a type of file. This structured organisation is crucial to the metaphor, as shown by the following diagram in which Kilgannon lists the various data structures employed in computing.
Figure 6.3: Data structures (from Kilgannon 1980, p.59).
In the context of the other data structures, it is relatively easy to determine what a file is. A physical file has a specific location, if not a unique name, whilst a file that forms part of a mainframe data processing system will have a defined internal structure and a specific name. With the move to personal computing, the same term has been generally applied to the 'files' used on PCs, even though these do not share the formal characteristics of the physical files or mainframe computing files. Some interfaces based on the desktop metaphor have attempted to apply the more appropriate metaphor of 'document' but this has not been done consistently. For example, although the Lisa and Macintosh use the terms 'document' and 'item' to describe entities on the desktop, the 'File' menu forms a standard element of the Apple Human Interface Guidelines for applications (Apple 1987). Similarly, Microsoft Word, the application in this study, employs the standard 'File' menu but refers to the file as a 'document' in other places such as the 'Format' menu.
Wordprocessing files do not have any visible structure. Users are free to structure their documents in any manner they care to, backing up or transferring data to other programs according to whim, with no formal control. Thus, whereas mainframe control procedures would regularise back-up procedures with clear unambiguous names for back-ups or historical records, consistent with the original metaphor, personal back-ups are likely to be ad hoc. Users referred to different versions of the same file, copies of the file and back-ups without any consistency.
Assessing the highest level of signification for each user cannot be entirely objective but it is generally quite clear that when a respondent has answered that A is done because of B, then B represents a higher level of signification. There is not necessarily a single level of signification at the highest level and the following list combines some categories which formed part of a single statement. The list shows the highest levels of signification for the commercial users, with the number of other users mentioning the same categories in brackets.
CU1 14.1/25.1 It's a requirement/ under legislation. (2/0)
CU2 10.2/11.2 Because I need the money/ and I do what I'm told to do. (0/2)
CU3 13.3/14.3 To get to the screen/ I want to go to. (1/1)
CU4 14.4 Those are the tasks/ that I have to do. (1/2)
CU5 19.5/20.5 To see how good we're doing as a business/ and we're making a profit. (1/0)
It can be seen that there was a small degree of shared signification amongst the higher levels mentioned by the commercial users, with each of the issues being at least partially mentioned by other users. As discussed above, two of them saw the highest level of signification in company terms, one incorrectly. CU2 and CU4 both saw the highest level of signification in personal terms, doing what one is told to do, whilst CU3 failed to mention any signification above the level of the operation of the system.
By contrast with the commercial users, the highest levels for the academic users were all different and all related to their own areas of work:
AU1 22.1/23.1 So that you can keep your train of thought - what changes have been made. (0/0)
AU2 24.2/25.2/11.2 To make it easier for other people/ to run it/ or use it. (1/0/0)
AU3 28.3/29.3/30.3/31.3 To send these letters/ to my contacts. To receive a response from them/ if it's an invitation./ They'll contact me. (1/0/0/0)
AU4 7.4/8.4 Because it's a slightly different application you're developing/ or a different direction. (3/4)
AU5 9.5/10.5 Probably you want to continue working/ somewhere else. (0/0)
AU1 and AU5 gave totally unique significations, while AU2 and AU3's highest levels were only partially mentioned by a single other user each. However, the first of AU4's categories was mentioned by a total of four users and the second part by all five. The interview with AU4 was much shorter than the interviews with the other academic users and his highest level of signification was not comparable to the others. He failed to give any non-generalised significations, cutting the interview short with the complaint, "You're getting too wide." All other users went further on these points, elaborating on the reasons why they might wish to develop a new application or go in a different direction.
The split within the CUG between users who related the highest level of signification to their own purposes and those who related it to the business did not occur in the academic group. This is not surprising in that doctoral studies are largely self-motivated, although one student did refer to saving data in a different format "to make it easier for other people to run it or use it." The commercial system was designed for use by a specific group of users to meet a specific business function, whereas wordprocessors are designed to be used by anyone who has access to a PC, leading to the wide range of purposes that the academic users identified.
The question of whether the 'What for?' technique is of value to system developers and designers depends to a considerable degree on whether the significations uncovered are relevant to the design. In the case of the commercial users, as discussed above, the importance of the higher level significations lies more in management issues than in design issues. The interviews also revealed whether work is self-motivated or whether users are simply doing it because they "need the money." The significations uncovered among the academic users would be of more direct usefulness, showing a range of different purposes some of which might well be beyond the designers' original scenarios for possible use of the system. This will be discussed in Chapter 7.
The 'What for?' interviews were intended to test the last two assertions listed at the end of chapter 3:
| Assertion 4 | The recursive nature of signification leads to a structured model of multiple layers of signification. |
| Assertion 5 | Further layers of signification can be uncovered by asking of each layer, "What does that signify?" or, more simple, "What for?" |
The Layers of Signification (LoS) model was extensively described in chapter 3. The structure of the interviews showed considerable concordance with the LoS model in that each respondent began with lower levels of meaning, progressing to higher and higher levels. The interviews cannot prove that the LoS model lies at the centre of human thought but they do affirm that it is a valid way to look at the meaning of an interface to a user, confirming Assertion 4. The 'What for' technique was successfully used to uncover the layers in an efficient manner, confirming Assertion 5. A final question was also posed:
| Question 5 | What is the highest level of signification that matters to the user? |
The highest level of signification revealed by each user was listed in section 6.3.2. These were not necessarily the highest levels that existed for each user but can be considered to be the highest level that mattered to the user and was considered worth mentioning at that time. Two interviewees stopped at a comparatively low level and showed an unwillingness to reveal higher levels. This cannot be taken as indicating that higher levels did not exist and may well have been caused by hostility towards the interview method or embarrassment at revealing more personal aspects of the interface element's meaning.
The types of high level signification which were revealed included aspects which are rarely taken into account by interface designers, such as the users' motivation for doing their work. Whether designers should consider these factors will be further discussed in the next chapter.