Return to GeoComputation 99 Index

Augmented seriation: usability of a visual and auditory tool for geographic pattern discovery with risk perception data

Gluck, Myke, Ph.D., Lixin Yu, Ph.D., Boryung Ju, Woo-Seob Jeong, and Ching-Tung Chang
244 Shores Building, Florida State University, Tallahassee, FL 32306-2100 U.S.A.
E-mail: mgluck@lis.fsu.edu

Abstract

Geographic data are often presented in maps and tables using GIS viewers, spreadsheets, and other software. This study developed and tested a data presentation tool that integrated a map viewer, seriation matrix, and sound generator to study how users receive and perceive spatially related information to improve perception and interpretation of the information.

Seriation is the process of looking for univariate sequences in data, such as a chronological sequence at archeological sites of pottery shards. Seriation, also called reorderable or permuted matrices, may be used as a tool for discovering patterns in numerical data. Seriation data matrix cells contain icons proportional in size to data values rather than actual data values. Users of a seriation tool permute rows and columns of the matrix to visually discover graphical patterns of the data with meaningful interpretations.

Augmented seriation uses a computer graphical user interface to add interactive manipulation, color, sound, and map views to the basic seriation techniques. Augmented seriation permits simultaneous display of maps and iconized data matrices. Augmented seriation's maps and matrices are graphically linked to permit highlighting (brushing) of related cells in the matrix and map elements (points, lines, or polygons), expanding the use of seriation for more visual analyses of spatial and temporal phenomena; thus, augmented seriation permits users to perform a visual principal component-like analysis of numerical and spatial data concurrently. Augmented seriation also employs sound supplementing the visual pattern discovery techniques with aural analyses.

We conducted usability tests of our software implementation with naive geographic users and have found favorable results. Results indicate that after receiving an explanation of seriation, augmented seriation, and a basic introduction to the tool's menus, users can effectively and efficiently employ the concepts of seriation and this tool to discern numerical as well as spatial patterns. In our tests, users discerned patterns in hazards and risk data especially well with the visual tools and, to a lesser degree, the auditory tools.

1. Definition of seriation and augmented seriation

Seriation is the task of organizing a set of objects along a single dimension such as size or age. Seriation in its most simplistic form is used to test the mental development of children by asking them to arrange sticks in ascending or descending size order, or to pick out the stick that is out of such an order. It has been used for almost a century (Petrie, 1899) as a technique for visual exploration of quantitative and categorical data. The seriation process begins with the construction of a data matrix whose rows are observations and whose columns are variables. The cells of the matrix are formed with icons that are proportional in size to the value of the variables at the observations rather than raw numerical values. The analyst proceeds to permute, sort, or move one or more columns or rows sequentially to discern a visual yet meaningful pattern in the data. This technique has also been called permuted matrices or reorderable matrices (Bertin, 1981; Muller, 1983). Multi-valued data are often treated as binary data by dichotomization about a mean, median, or other critical value in seriation; however, multivalued as well as binary data are amenable to analysis using seriation.

The pre-computer approaches to seriation involved paper and pencil sketches or physically moving objects such as samples of pottery or blackened blocks to permute the rows and columns (Petrie, 1899; Bertin, 1981; Foggin & Aurillon, 1989). Figures 1, 2 and 3 present an example of seriation illustrating the "before, in process, and after" effects of seriation manipulations for a subset of risk perception data discussed later in this paper. Thus, seriation permits users to perform a visual principal component-like analysis of numerical data augmented by spatial and/or temporal analyses (Rodgers & Thompson, 1992) .

Augmented Seriation, a term we have coined (Gluck, 1994; Gluck & McRae, 1997), is a procedure that expands traditional seriation using computerized multimedia. Augmented seriation employs the power of computer graphics, digitized sound, automated spreadsheets, and geographic information systems (GIS) to assist the analyst in discerning patterns in data. Augmented seriation adds features such as color, sound, or a third dimension to the data matrix, provides mapping options, and permits communication between the data matrix and map(s). Sound and color provide the analyst additional perceptual modalities for observing patterns in the iconic data matrix and for assessing the meaning of patterns. As the analyst manipulates the data matrix, maps appearing adjacent to the data matrix provide concurrent spatial and/or temporal pattern identification, recognition, and understanding support. The display of multiple maps can provide spatio-temporal support when each map represents the spatial distribution of data at different times, as does the facility for displaying bivariate choropleth maps. A third dimension also may be added to the data display by incorporating multiple data matrices to provide additional temporal support in seeking patterns in the data when time series data are available much like tabbing in automated spreadsheets.

2. Conceptual background for seriation

The concept of developing a series along a dimension has an extensive literature in several fields. For example, in child development, Piaget & Inhelder (see Kingma,1986) developed several simple seriation tasks to assess the cognitive development of children. These tasks often involved organizing a small set of sticks in size order. Such simplistic tasks are not what this investigation is considering; however, the value of simple seriation tasks indicates the significance of ordering as a useful yet complex cognitive human skill.

The earliest recorded use of more sophisticated traditional seriation found in preparation for this work involves the analysis of Egyptian pottery by Petrie (1899). He investigated sequences of successive stages of pottery over time and later used the sequences to date other fragments by material and tool markings. Seriation has been applied to citation data, geological layer data, cartographic analysis, and general information processing problems. Bertin (1981) and subsequently Muller (Muller,1983; Muller & Honsaker, 1984) promoted the technique as a means for data analysis. Bertin and Gimeno (1983) promote seriation as a pedagogical technique for engaging grade school students in higher levels of thinking and analysis. Bertin (1981) also indicates the value of maps in the seriation process, and exhibited data matrices with the cells containing dot maps.

An earlier attempt to automate the traditional seriation process for sophisticated analysis was a computer program for Apple's Macintosh computer environment called MAT written in 1988 by Reijnders and Adams at Eindhoven University of Technology. MAT did not use multimedia (no color, sound, maps, brushing, or third dimension matrices), but is a faithful implementation of Bertin's basic seriation concepts for information processing.

3. Pattern recognition, algorithms, and meaning

The goal of manipulating and exploring the iconic matrix of data is to find meaningful patterns in the data. There are two interrelated and inherently difficult problems in this process: 1) find a pattern in the data, and 2) discern if a relevant meaning for the pattern exists in the context of the problem. These two difficult and interrelated problems are not unique to seriation. For example, the Chi-square statistical test may indicate a lack of randomness or the existence of a relationship in contingency table analyses, but a specific pattern of non-randomness may actually be elusive with even two or three variables (Reynolds, 1984; Liebetrau, 1983). Also, statistical techniques, such as cluster analysis and factor analysis, suffer from interpretation enigmas even when significant patterns emerge (see Cohen & Cohen, 1983; Johnson & Wichern, 1988). For example, factor analysis often finds several significant factors. Unfortunately, the influential factors from collapsed variables often lack meaningful descriptions in the context of the specific problem. A similar situation frequently develops with canonical correlation's canonical variables.

The predominant strategy for seriation exploration has been to diagonalize a dichotomized or bi-valued matrix. This process attempts to form upper and lower triangles with cells lacking certain properties and a diagonal band of cells with high presence of properties (see Figure 3 above). This is equivalent to maximizing the sum of elements in either the upper or lower triangle (assuming 1 and 0 are the dichotomized values underlying the cell icons). This diagonalization strategy does have an assessment criteria as well as a closed mathematical process for binary data. This is not as limiting as it may appear since binary data may result from the presence or absence of a variable at a particular site or from dichotomizing data above or below a critical value of a variable (mean, mode, national norm, etc.).

The diagonalization process with binary data follows a four step process:

  1. Compute mean position of presence of variables in each row (observations) by averaging all column numbers with the phenomena present within that row (one can assume column numbers increase from left to right)
  2. Arrange the rows in order (ascending or descending) according to these means
  3. Compute mean position of presence in columns (variables) by averaging all row numbers with the phenomena present within that column (one can assume row numbers increase from top to bottom)
  4. Arrange the columns in order (ascending or descending) according to these means.

Iterations of these four steps continue until the matrix stabilizes which occurs when an iteration leads to no change in the order of either rows or columns.

A test criterion, S, for seriated binary data assesses how well the stable seriated matrix comes to matching the diagonalization objective:

with j=1 to n, where n is the number of columns of the matrix, Aj is the number of embedded absences (0) in column j, and Rj is the number of ranges of rows of presence (1) for that column.

A perfect seriation will have a value of S=1 representing a fully diagonalized binary matrix with no embedded absences (Figure 3, is the best this data can do); however, any seriation process is problematic depending upon the starting point for manipulations. Often stability in the criteria is achieved but a more optimal solution may exist. This is similar to finding local optimums that are not global optimums. Even with this computational method for diagonally seriating a binary matrix, the interpretation of the best pattern may be problematic. Often, from a practical point of view, a less than ideal mathematical result, reflecting weaker relationships, may lead to useful interpretations while the mathematically best result may not be interpretable.

For constructing seriation criteria for multivalued matrices we have experimented with assigning weighted values between 0 and 1 to cells whose value lies between the maximum and the minimum value for that variable (column). (The interface to the software described below makes re-expressing the data categories very easy, permitting analysts to reclass data to observe both clustering and sensitivity effects.) The weight assigned to the cell is proportional to the size of the value in the cell relative to the range of the variable with 0 being the minimum and 1 as maximum. For example, if the data have ten interval valued categories, the second smallest category cells are given a value of 1/9, the next larger category 2/9, and so forth, with the second largest category given the value 8/9. A four-step process analogous to that described above may then be applied resulting in a criteria for assessing the diagonalization of a multivalued data matrix. Such assigned weights provide some assessment of seriation success but are still often problematic for interpretation. Multivalued matrices also are similarly subject to local optima stability conditions.

Since the success criteria and interpretation of automatically manipulated matrices is problematic seriation is frequently best done using visual and interactive exploration to find meaningful patterns. As a sequential process, seriation can be driven by meaningful hypotheses. An analyst can begin to see a pattern developing, perhaps by random, cognitive or perceptual processes, that generates a meaningful hypothesis for the context. The analyst may then continue manipulations to see how strong a visual image that hypothesis demonstrates, and whether or not it leads to the mathematically best seriation. Our software allows the analyst to manipulate the matrix by moving a row, column, or a block of rows and columns, sorting by column(s) or row(s) ascending or descending, and by invoking the automated seriation algorithm described above.

4. Augmented seriation: color, maps, sound, and interactivity

We have built computer software to perform seriation augmented by multimedia and to test the value of this process with a range of human subjects. Traditional seriation has used visual size as a singular variable for assisting analysts in performing seriation. We have begun using this tool to investigate how a range of analysts discern patterns in data. Based on the user testing discussed below, we have modified the tool by incorporating users' successful strategies, and creating an intelligent computer data exploration aid for pattern identification, recognition, and understanding.

We have produced the basic matrix, sorting and permutation functions including an undo function using Visual Basic on a Microsoft Windows platform. Visual Basic includes a built-in grid function called MSFlexgrid that is sufficient for our proof of concept work. Cartographic software support is achieved through ESRI's MapObjects LT linked to our other program modules. Brushing, the ability to select information in one window and have it be reflected in another window, is enabled in this software package through message passing between the map and matrix via selection event interrupts when brushing is turned on through a pull-down menu option. The user has basic configuration choices, including:

Multimedia capabilities include visual and sound augmentation to these basic seriation functions. Color saturation ranging (low to high or high to low) for multi-valued data provides redundant color to support the visual search for patterns in the data. Any of the basic hues can be saturation ranged providing gray scale support for visual exploration. We exhibit only one hue for a variable within the matrix at a time since color is not a size variable (Bertin, 1981).

Cartographic display support for spatial data permits the analysis of spatial relationships. Maps may be presented in adjacent windows on the display screen. Currently only single variable or bivariate choropleth maps are available, but dot and graduated circle (raw or psychometrically adjusted) maps as well as multi-variable maps including hue with dots, hue with graduated circles, multiple hue dots, and hypsometric are planned. Brushing is available between the map(s) and matrix windows.

Sound options are included to support the search for patterns. All the sound options currently developed are functions of the underlying data value or the number of data classes. Currently, tone or volume ranges serve to subdivide the data much like the category scheme for icon size for the matrices. Rows and columns or subsets of them may be "played" with a choice of several sound responses. Various pitches, timbre, number of tones, tone duration, and tone frequency options permit the user to richly explore the data aurally as well as visually.

The work also supports viewing temporal changes in data since the software can display multiple maps simultaneously. Display changes in attribute values at locations are illustrated by creating either multiple univariate maps or "bivariate" maps using the same variable displayed at different times. A future options will include enhancements and usability testing of "three dimensional" matrices much like several spreadsheet software products that include tabbed sheets. Such multi-dimensional views generate simultaneous visualization of more than one iconic matrix permitting display of the temporal data changes in the matrices to accompany the spatio-temporal changes in the maps. (See Figure 4 for current interface, map, matrix, and menu displays.)

5. Overview of usability assessments

The Usability of an information system or product assesses how well users are able to employ the system or product as well as assessing how useful the service or product is in permitting users to successfully achieve their intended goals. We may view usability as assessing both the efficiency of the process (Is it usable?) with the effectiveness of the product (Is it useful?). The usability approach to information system product development and evaluation evolved from a combination of human factors engineering (a subfield of industrial engineering) and applied and cognitive psychology. Human factors engineering, or ergonomics, studies the ability of humans to employ tools or affect processes in the workplace without incurring physical harm. For example, ergonomics may determine an effective instrument cluster placement within a fighter plane cockpit or modify load lifting methods to reduce the likelihood of lower back pain for workers on an assembly line. Cognitive psychology may be meaningfully applied in other scenarios to assist information workers and information seekers in reducing memory demands, cognitive processing time, and the amount of information overload. Reducing these demands and effort makes human execution of information tasks less prone to error and decreases the time for accomplishing the given tasks. The roots of usability can thus be traced to the time and motion analysis in the workplace of Frederick Taylor (1911) and to the processing demands exposed by Gestalt psychology (Kohler, 1947). As computers became more common in the workplace, design of computer interfaces and the desire to move toward more user-centered information systems led workers in the computer field to adopt and adapt the goals, concepts, and methods of human factors and psychology (e.g., Engelbart, 1963; Shneiderman, 1992; Norman, 1993). Such efforts have led to the development of a separate field called usability shared by computer science, industrial psychology, and information systems.

The traditional usability approach holds a limited yet effective view of humans as information processors because of its roots in engineering and psychology. The traditional usability approach monitors the performance of a small sample of workers on a task, typically capturing their time on tasks and their rate of errors in executing the tasks. Much work in applying the usability approach centers around the careful selection of subjects, composition of subject tested tasks, and assessing user performance by time on task and error rates. The traditional usability approach based in these roots is driven by the dual yet sometimes competing desires to be (1) the servant of users of systems as opposed to being subservient to the technical aspects, and (2) cost effective in a commercial setting. Occasionally usability laboratories will post signs stating "The User Is Always Right" or "The User, Stupid, The User." Based on these efforts and the general attitude of usability professionals, usability analysts have reasonable claims to being the user advocates for the world of information system development and deployment.

General agreement among usability analysts suggests (Rubin, 1994):

1) Effective usability demands an early focus on users and tasks as products and systems are being developed

2) Insistence upon empirical observation and as much measurement of product usage and performance as possible; however, most usability analysts also make qualitative assessments (i.e., satisfaction surveys, focus groups, short term ethnomethodological task analysis) along with these quantitative assessments, and

3) Effective usability evaluation requires multiple assessments of usability during the design and deployment of a product. This usually involves development of a design, some testing with users, modification of the design, more testing with users, and so forth, through the implementation process with testing cycles "beginning" all over again with users as new versions are developed and deployed.

Thus, the overall goal of the usability approach is to identify and suggest preferable alternatives to the weaknesses of an information system, product, or interface as noted by the users of such a system. Issues noted by users are to include both quantitative and qualitative assessments of both usableness and usefulness of a service or product. The usability approach seeks to support the development and deployment processes by assessing the ease of learning, ease of use, degree of user satisfaction, and the value to the user community of highly functional components.

6. Usability of augmented seriation: a geospatial example with risk data

6.1 Participants

During the summer of 1998, usability testing of the then current version of the augmented seriation software was conducted. Ten participants, each with at least 2 years of computer experience but without geographic or risk assessment expertise, were recruited to test the software. All participants were current or former graduate students with diverse majors from analytical chemistry to adult education. They ranged in age from 25 to 54 with six female and four male participants. Such a user group provides a baseline of usability for the augmented seriation software; those with more computer expertise or statistical backgrounds should find the tool at least as usable as the baseline subjects.

6.2 Tasks

Each participant performed four tasks with increasing specificity and complexity regarding the functionality of the software. Each session lasted 90 minutes and the users were accompanied by a research assistant while they executed the tasks. The sessions were video taped in the Florida State University Usability Center, which permitted concurrent picture-in-a-picture video of the users' upper torso and the software screen image. The major data collection tools were video tapes of think aloud protocols in which users described orally what and why they were doing what they were doing (Ericsson & Simon, 1980), researchers' observation logs of users' interaction during the task performances, and post-test debriefing survey forms in which users described their assessment of the usability of the tool in general and of particular functions specifically in written format. (Copies of the full usability report including results and testing scenarios, complete task descriptions as well as subject background and debriefing forms, are available from the first author at the address listed above.)

The first task required users to assess the help menu and a help demonstration that explained the process of seriation and its goal of diagonalizing a matrix and data exploration. The second task assessed the ease of learning regarding the basic functions of the software as outlined in the section above on augmented seriation, while the third task assessed the ease of use of the software functions for novice users with risk data. These first three tasks provided users with step-by-step detailed instructions. The fourth and last task requested users 'explore' an actual analysis problem using risk assessment data; this was a much less structured task.

6.3 Data employed

The risk perception data employed in this testing is based on survey results of risk managers from the counties of New York State who rated perceived level of risk to their county of each of a set of ten threats to their communities (Monmonier & Giordano, 1998). These data (used with permission) provided naive participants a data set of sufficient size and complexity for meaningfully testing the software's data exploration and data pattern recognition capabilities.

6.4 Findings

The usability testing revealed the following strengths of the software. All users indicated that the current implementation of the help menu and system is useful and usable. The toolbar menuing system was also seen as useful and usable by all users. Users indicated that the sort and move functions that permit data matrix manipulation are useful and helped them reach the performance goals. Regarding matrix icon colors, users preferred green and found either icon shape (round or rectangular) useful and usable. These non-experts found both the thematic choropleth maps and the bivariate map useful and usable but preferred using the single variable rather than the bivariate maps in their efforts to see patterns in the data.

The usability testing revealed the following opportunities to improve the software. Testing revealed that improvements to the help system are needed to support better understanding of the overall purpose of seriation and data exploration for naive users. (Such extensive demonstration and preparation for use should not be needed for those with high levels of expertise in statistical or geographic analysis but further testing is needed on this matter.) The tools for data exploration (color, icon shape, sound, maps, etc.) were very usable but naive users did not find them very useful in the less structured phase of the testing, apparently because their mental models of the process of data exploration and pattern discernment were not yet well developed. This lack of apparent usefulness was illustrated most clearly by the sound function; users were not at all sure how sounds could help them. Such lack of usefulness for these users reinforces the need for improvements in the help and demonstration features to better explain the goals of seriation and the potential support of the various exploration features.

6.5 Recommendations

Think aloud protocols from users reveal both high and low level needs for software improvements. Analysis of the think aloud videos and the debriefing forms suggested the following specific recommendations with eleven important recommendations:

1) Provide users with clearer representation of the purpose of seriation, 2) Provide a glossary of general and specific terminology, 3) Provide better contrast in the color schemes for the maps and legend, 4) Use darker colors for higher and larger data values and brighter colors for lower and smaller data values, 5) Provide additional explanation for the role of sound in data exploration, 6) Provide a more convenient way to delete legends from view, 7) Include a close function as well as an exit function for the help screens, 8) Include a back function in the Help demonstration, 9) Eliminate the NEXT button on the last screen of the Help demonstration; 10) Provide a more memorable mechanism to MOVE columns and rows, and 11) Provide an easier to learn and recall mechanism for thematic map and bivariate map construction.

Also suggested by the testing are six less critically needed improvements: 1) Help demonstration needs a more specific example with less generality such as replacing labels of A, B, C with more concrete labels of transportation, housing, income, etc., 2) Provide more contrast for selected (highlighted) columns or rows, 3) Provide more sound options, 4) Provide additional multivariable mapping beyond the bivariate thematic displays, 5) Provide additional matrix icon shapes and colors, and 6) Add a self-exploration section for the Help demonstration.

7. Conclusion and future work

These are typical recommendations that are rarely anticipated by designers of software. They reflect more real world use of products and the broader context in which real people work. They are usually only captured by actual user testing. All users were able to use the software giving it high marks for usableness; the issue of usefulness is more problematic. These testing sessions clearly indicated that those users with a more robust mental model of seriation were much more effective in using the software and in discerning patterns of multivariate and spatial distributions of risk perception data. We believe that this bodes well for the value of augmented seriation and for this software implementation, especially for those with more experience and expertise in the analysis of spatial and statistical data. This testing also reveals the imperative for the Help system to be more supportive of the conceptual development for naive users before turning them loose to use the tool. Statistical and spatial analysis are not highly intuitive skills, but they appear to be trainable skills. We believe augmented seriation can assist in such training as well as supporting actual data analysis and exploration.

Muller's (1983) statement that seriation matrix manipulations may ". . . lead to an interpretation of structural relationships between geographic entities and their spatial attributes" may be expanded. Our usability testing indicates strongly that augmented seriation provides enhanced visual displays that lead more readily to these important structural relationships. Sound seems more problematic in assisting users to explore data and find meaningful patterns in data. Usability testing of the auditory and musical enhancements also may add to seriation's support for data analysis.

Our future work will 'clean up' the issues reflected by users in our testing, enhance the software with additional cartographic displays such as dot and graduated circle maps as well as additional multivariate displays, and experiment with more tonal sound (music) support. Little research has been done to explore the use of tonal articulation and ornamentation, for example, as variables to enhance data exploration (e.g., Krygier, 1994). We hope to exploit such variables and their software implementations to better understand the role sound can play in data exploration. We will also continue usability testing with differing user groups as we modify and enhance the tool. In conclusion, we believe that with these more user friendly recommendations and additional tool functionality implemented, augmented seriation will be one more efficient and effective analysis tool for researchers to consider when exploring data, especially risk perception and other spatially referenced datasets.

References

Bertin, J., 1981. Graphics and Graphic Information Processing: Walter de Gruyter, New York, NY.

Cohen, Jacob and Patricia Cohen, 1983. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. (2nd Ed.): Lawrence Erlbaum Associates, Hillsdale, NJ.

Engelbart, D.C., 1963. A Conceptual Framework for the Augmentation of Man's Intellect. In Vista in Information Handling, v1. Howerton, P.D. and Weeks, D. (Eds)., Spartan Books Washington, D.C.: pp.1-29.

Ericsson, A. and H. Simon, 1980. Verbal reports as Data. Psychological Review. 87 (3): pp. 215-251.

Foggin, P. and N. Aurillon, 1989. "Respiratory health indicators and Acculturation among the Inuit and Cree of Northern Quebec: A Regional Approach Using Geographic Seriation Analysis." Soc. Sci. Med 29(5): pp. 617-626.

Gimeno, Roberto and Jacques Bertin, 1983. "The Cartography Lesson in Elementary School." In D.R.F. Taylor (Ed.) Graphic Communication and Design in Contemporary Cartography. John Wiley & Sons Ltd: pp. 231-256.

Gluck, Myke and John McRae, 1997. Augmented Seriation: Searching for Health Care Patterns with a Multimedia Cartographic Tool. In Proceedings of International Symposium on Computer Mapping in Epidemiology and Environmental Health. Tampa, FL. pp.53-59.

Gluck, Myke, 1994. "Augmented Seriation." Presentation at Association of American Geographers (AAG) Annual Meeting. San Francisco, CA. Abstracts, p. 123.

Johnson, R.A. and Dean W. Wichern, 1988. Applied Multivariate Statistical Analysis: Prentice-Hall, Englewood Cliffs, NJ.

Kingma, J., 1986. Training of Seriation in Young Kindergartners. J. of Genetic Psychology. 148(2): pp. 167-181.

Kohler, Wolfgang, 1947. Gestalt Psychology. Mentor Books, New York, NY.

Krygier, J.B., 1994. Sound and Geographic Visualization. In Visualization in Modern Cartography MacEachren, A. and Taylor D.R.F. (Eds.) New York: Pergamon. pp. 149-166.

Liebetrau, Albert, 1983. Measures of Association. Sage University Series on Quantitative Applications in the Social Sciences, series no. 32. Sage, Beverly Hills, CA and London, England.

Monmonier, M. and A. Giordano, 1998. GIS in New York State County Emergency Management Offices: User Assessment. Applied Geographical Studies, Vol. 2, No.1, pp. 95-109.

Muller, J.C., 1983. Geographic Seriation Revisited. Professional Geographer, 35: pp. 196-202.

Muller, J.C. and J.L. Honsaker, 1983. Visual vs. Computerized Seriation: The application for Automated Map Generalization. Autocarto IV Proceedings, p. 277-287.

Norman, D.A., 1993. Things that make us smart : defending human attributes in the age of the machine.: Addison-Wesley, Reading, MA.

Petrie, W.M., 1899. Sequences in Pre-Historic Remains. J. Anthropological Institute of Great Britain and Ireland. Vol. 29: pp.295-301.

Reynolds, H.T., 1984. Analysis of Nominal Data. Sage University Series on Quantitative Applications in the Social Sciences, series no. 7. Sage, Beverly Hills, CA and London, England.

Rodgers, Joseph Lee and Tony D. Thompson, 1992. Seriation and Multidimensional Scaling: A Data Analysis Approach to Scaling Asymmetric Proximity Matrices. Applied Psychological Measurement. 16(2): pp.105-117.

Rubin, Jeffery, 1994. Handbook of Usability Testing: How to Plan Design, and Conduct Effective Tests. Wiley & Sons, New York, NY.

Shneiderman, Ben, 1992. Designing the User Interface: Strategies for Effective Human Computer Interaction. 2nd., Addison-Wesley, Reading, MA.

Taylor, Frederick W. 1911. Principles of Scientific Management. Harper and Row, New York, NY.