Analytic Hierarchy Process: Difference between revisions

From Citizendium
Jump to navigation Jump to search
imported>Louis F. Sander
(→‎See also: " See also" not needed. Covered in Related Articles.)
imported>Louis F. Sander
Line 102: Line 102:
As we move ahead through the Analytical Hierarchy Process, the priorities will change but will still add to 1.000 for each group of child nodes.
As we move ahead through the Analytical Hierarchy Process, the priorities will change but will still add to 1.000 for each group of child nodes.


==Criticisms==
Despite its widespread use, there have been criticisms of the AHP: A) since there is no theoretical basis for constructing hierarchies, AHP users can construct different hierarchies for identical decision situations, possibly producing different solutions, B) AHP rankings are claimed to be arbitrary because they are based on subjective opinions using a ratio scale, a situation which can produce "rank reversal," C) there are flaws in the methods of combining individual weights into composite weights, and D) the process has no sound underlying statistical theory.


In 1990, a series of debates between critics and proponents of AHP were published in the journals Management Science<ref>Dyer, J. S. (1990): Remarks on the Analytic Hierarchy Process. In: Management Science, 36 (3), S. 249-258.</ref><ref>M. V. Mikhalevic "Remarks on the Dyer-Saaty controversy" Cybernetics and Systems Analysis,Volume 30, Number 1 / January, 1994 </ref><ref>Patrick T. Harker, Luis G. Vargas, "Reply to 'Remarks on the Analytic Hierarchy Process' by J. S. Dyer", Management Science, Vol. 36, No. 3 (Mar., 1990), pp. 269-273</ref><ref>Dyer, J.S. (1990b), "A clarification of 'Remarks on the analytic hierarchy process'", Management Science, Vol. 36 No.3, pp.274-5.</ref> and The Journal of the Operations Research Society.<ref>Holder, R.D., Some Comment on the Analytic Hierarchy Process, Journal of the Operational Research Society, 1990, 41, 11 1073-1076. </ref><ref>Thomas L. Saaty "Response to Holder's Comments on the Analytic Hierarchy Process" The Journal of the Operational Research Society, Vol. 42, No. 10 (Oct., 1991), pp. 909-914</ref><ref>R. D. Holder "Response to Holder's Comments on the Analytic Hierarchy Process: Response to the Response" The Journal of the Operational Research Society, Vol. 42, No. 10 (Oct., 1991), pp. 914-918</ref>
In these debates, AHP critic J. Dyer argues:
::"The AHP is flawed as a procedure for ranking alternatives in that the rankings produced by this procedure are arbitrary."<ref>Dyer, J. S. (1990): Remarks on the Analytic Hierarchy Process. In: Management Science, 36 (3), S. 249</ref>
Another AHP critic, R. D. Holder also concludes in 1991 that:
::"Thus questions about the validity of AHP are far from having been settled.  Also, it is clear that such discussion over a long period has not sufficiently penetrated the AHP users' community, with the result that paper are still being published using a faulty method."<ref>R. D. Holder "Response to Holder's Comments on the Analytic Hierarchy Process: Response to the Response" The Journal of the Operational Research Society, Vol. 42, No. 10 (Oct., 1991), pp. 914-918</ref>
However, even while criticizing AHP, Holder finds room for possible improvement in AHP when he states in the same source:
::"No amount of empirical evidence can truly validate a theory with such internal contradictions as the originally constituted AHP, though the theory is a good basis from which to develop." 
Of the weaknesses found in AHP, the use of arbitrary scales and certain internal inconsistencies and theoretical flaws have been discussed extensively in the literature. 
'''The use of arbitrary scales'''
AHP is based on pairwise comparisons where the relative importance of different attributes are given a value on a scale of 1 to 9 or the inverse (1/9th to 1).  These values are in practice assigned by verbal elicitation of decision makers.  For example, if a person says attribute A is "moderately more imporantant" than attribute B, A is said to have a relative weight of 3 times that of B while being "extremely more important" will give A a weight of 9 times that of B. While this scale is commonly used in AHP, it is arbitrary and alternative scales have been proposed. Empirical research has found "...that the perceived meaning of the verbal expressions varies from one subject to the next and also depends on the set of elements involved in the comparison." <ref>MARI A. PÖYHÖNEN, RAIMO P. HÄMÄLÄINEN, AHTI A. SALO "An Experiment on the Numerical Modelling of Verbal Ratio Statements" Journal of Multi-Criteria Decision Analysis,vol 6, no 1, ppg 1-10, 1997</ref>. This is a well known result (since the 1930s) from [[Thurstone]]'s work on attitude scales.{{fact}} However, these researchers felt the problem was correctable in that the scales could be based on empirical evidence of AHP user perceptions.
'''Inducement of nonexistant order'''
Another problem is the inducement of "nonexistant order" by innocuous changes even without the addition or deletion of suboptimal alternatives.  AHP critic Stan Schenkerman writes in Decision Sciences:
::"An apparently unreported problem facing decision makers who use AHP is described [in this paper]. It is demonstrated that conventional AHP and some of its variants (the ideal mode, and the pairwise aggregated approach, PAHAP) can induce ordering even when no order exists. It is also shown that all three approaches can induce different orderings and that the orderings are sensitive to innocuous changes. Thus, even absent addition or deletion of alternatives, the decision maker relying on AHP or these variants can be seriously misled." <ref>Stan Schenkerman "Inducement of nonexistent order by the analytic hierarchy process",  Decision Sciences, Spring 1997</ref>
'''Responses by AHP proponents'''
Proponents argue that in spite of these concerns, the process works well in practice and is extremely popular among decision-makers in the private and public sectors.<ref> {{Citation| first=J.E. | last=de Steiguer| coauthors=Jennifer Duberstein, Vicente Lopes| contribution=The Analytic Hierarchy Process as a Means for Integrated Watershed Management| title=First Interagency Conference on Research on the Watersheds| editor-first=Kenneth G.| editor-last=Renard| coeditors=et al| publisher=U.S. Department of Agriculture, Agricultural Research Service| place=Benson, Arizona| pages=736-740| date=October, 2003| year=| id= | contribution-url=http://www.tucson.ars.ag.gov/icrw/Proceedings/Steiguer.pdf| format=| accessdate= }}</ref>


==References==
==References==

Revision as of 23:51, 11 February 2008

This article is developed but not approved.
Main Article
Discussion
Related Articles  [?]
Bibliography  [?]
External Links  [?]
Citable Version  [?]
Addendum [?]
 
This editable, developed Main Article is subject to a disclaimer.

The Analytic Hierarchy Process (AHP) is a structured technique for dealing with complex decisions. Based on mathematics and human psychology, it was developed by Thomas L. Saaty in the 1970s and has been extensively studied and refined since then. The AHP provides a comprehensive and rational framework for structuring a problem, for representing and quantifying its elements, for relating those elements to overall goals, and for evaluating alternative solutions. It is used throughout the world in a wide variety of decision situations, in fields such as government, business, industry, healthcare, and education.

Users of the AHP first decompose their decision problem into a hierarchy of more easily comprehended sub-problems, each of which can be analyzed independently. The elements of the hierarchy can relate to any aspect of the decision problem—tangible or intangible, estimated or carefully measured, well- or poorly-understood—anything at all that applies to the decision at hand.

Once the hierarchy is built, the decision makers systematically evaluate its various elements, comparing them to one another in pairs. In making the comparisons, the decision makers can use their judgments about the elements' relative meaning and importance, or they can use concrete data about the elements. It is the essence of the AHP that human judgments, and not just the underlying information, can be used in performing the evaluations.

The AHP converts the judgments to numerical values that can be processed, evaluated and compared over the entire range of the problem. A numerical weight or priority is derived for each element of the hierarchy, allowing diverse and often incommensurable elements to be compared to one another in a rational and consistent way. This capability distinguishes the AHP from other decision making techniques.

At the end of the process, numerical priorities are derived for each of the decision alternatives. It is then a simple matter to pick the best alternative, or to rank them in order of relative preference.

Uses and applications

While it can be used by individuals working on straightforward decisions, AHP is most useful where teams of people are working on complex problems, especially those with high stakes, involving human perceptions and judgments, whose resolutions have long-term repercussions.[1] It has unique advantages where important elements of the decision are difficult to quantify or compare, or where communication among team members is impeded by their different specializations, terminologies, or perspectives. Computer software is available to assist in the application of the process.

The applications of AHP to complex decision situations have numbered in the thousands,[2] and have produced extensive results in problems involving alternative selection, planning, resource allocation, and priority setting.[1] Many such applications are never reported to the outside world, because they take place at high levels of large organizations where security and privacy considerations prohibit their disclosure. But some uses of AHP are discussed in the literature. Recently these have included:

AHP was recently applied to a project that uses video footage to assess the condition of highways in Virginia. Highway engineers first used it to determine the optimum scope of the project, then to justify its budget to lawmakers.[8]

The process is widely used in countries around the world. At a recent international conference on AHP, over 90 papers were presented from 19 countries, including the U.S., Germany, Japan, Chile, Malaysia, and Nepal. Topics covered ranged from Establishing Payment Standards for Surgical Specialists, to Strategic Technology Roadmapping, to Infrastructure Reconstruction in Devastated Countries.[9] AHP was introduced in China in 1982, and its use in that country has expanded greatly since then—its methods are highly compatible with the traditional Chinese decision making framework, and it has been used for many decisions in the fields of economics, energy, management, environment, traffic, agriculture, industry, and the military.[10]

Though using AHP requires no specialized academic training, the subject is widely taught at the university level—one AHP software provider lists over a hundred colleges and universities among its clients.[11] AHP is considered an important subject in many institutions of higher learning, including schools of engineering[12] and graduate schools of business.[13] AHP is also an important subject in the quality field, and is taught in many specialized courses including Six Sigma, Lean Six Sigma, and QFD.[14][15][16]

In China, nearly a hundred schools offer courses in AHP, and many doctoral students choose AHP as the subject of their research and dissertations. Over 900 papers have been published on the subject in that country, and there is at least one Chinese scholarly journal devoted exclusively to AHP.[10]

Steps in using the process

Model the problem as a hierarchy

The first step in the Analytic Hierarchy Process is to model the problem as a hierarchy. In doing this, participants explore the aspects of the problem at levels from general to detailed, then express it in the multileveled way that the AHP requires. As they work to build the hierarchy, they increase their understanding of the problem, of its context, and of each other's thoughts and feelings about both.[17]

Hierarchies defined

The hierarchy of authority aboard a commercial airliner. Each person is subordinate to a single other person.

A hierarchy is a system of ranking and organizing people, things, ideas, etc., where each element of the system, except for the top one, is subordinate to one or more other elements. Diagrams of hierarchies are often shaped roughly like pyramids, but other than having a single element at the top, there is nothing necessarily pyramid-shaped about a hierarchy.

Human organizations are often structured as hierarchies. In the airliner example shown to the right, the hierarchical system is used for assigning responsibilities, exercising leadership, and facilitating communication.

Familiar hierarchies of "things" include a desktop computer's tower unit at the "top," with its subordinate monitor, keyboard, and mouse. Inside the tower is another hierarchy, with the top level motherboard and its subordinate hard drive(s), CD-ROM drive, and sometimes a floppy drive or DVD writer.

In the world of ideas, we use hierarchies to help us acquire detailed knowledge of complex reality: we structure the reality into its constituent parts, and these in turn into their own constituent parts, proceeding down the hierarchy as many levels as we care to. At each step, we focus on understanding a single component of the whole, temporarily disregarding the other components at this and all other levels. As we go through this process, we increase our understanding of whatever reality we are studying.

Think of the hierarchy that medical students use while learning anatomy—they separately consider the musculoskeletal system (including parts and subparts like the hand and its constituent muscles and bones), the circulatory system (and its many levels and branches), the nervous system (and its numerous components and subsystems), etc., until they've covered all the systems and the important subdivisions of each. Advanced students continue the subdivision all the way to the level of the cell or molecule. In the end, the students understand the "big picture" and a considerable number of its details. Not only that, but they understand the relation of the individual parts to the whole. By working hierarchically, they've gained a comprehensive understanding of anatomy.

Similarly, when we approach a complex decision problem, we can use a hierarchy to integrate large amounts of information into our understanding of the situation. As we build this information structure, we form a better and better picture of the problem as a whole.[17]

AHP hierarchies explained

An AHP hierarchy is a structured means of describing the problem at hand. It consists of an overall goal, a group of options or alternatives for reaching the goal, and a group of factors or criteria that relate the alternatives to the goal. In most cases the criteria are further broken down into subcriteria, sub-subcriteria, and so on, in as many levels as the problem requires.

The hierarchy can be visualized as a diagram like the one below, with the goal at the top, the alternatives at the bottom, and the criteria filling up the middle.

A simple AHP hierarchy. In practice, many Criteria have one or more layers of subcriteria. These are not shown in this simplified diagram. To avoid clutter in these diagrams, the lines between the Alternatives and Criteria are often omitted or reduced in number. Regardless of any such simplifications in the diagram, in the actual hierarchy each Alternative is connected to every one of its covering criteria—the lowest-level criterion or subcriterion that appears immediately above it.

The design of any AHP hierarchy will depend not only on the nature of the problem at hand, but also on the knowledge, judgments, values, opinions, needs, wants, etc. of the participants in the process.

As the AHP proceeds through its other steps, the hierarchy can be changed to accommodate newly-thought-of criteria or criteria not originally considered to be important; alternatives can also be added, deleted, or changed.[17]

A simple example

In an AHP hierarchy for the simple case of buying a vehicle, the goal might be to choose the best car for the Jones family. The family might decide to consider cost, safety, style, and capacity as the criteria for making their decision. They might subdivide the cost criterion into purchase price, fuel costs, maintenance costs, and resale value. They might separate Capacity into cargo capacity and passenger capacity. The family, which for personal reasons always buys Hondas, might decide to consider as alternatives the Accord Sedan, Accord Hybrid Sedan, Pilot SUV, CR-V SUV, Element SUV, and Odyssey Minivan.

The Jones' hierarchy could be diagrammed like this:

File:AHPHierarchy2.png
AHP hierarchy for the Jones family car buying decision. The Goal is green, the Criteria and Subcriteria are yellow, and the Alternatives are pink. All the alternatives (six different models of Hondas) are shown below the lowest level of each criterion. Later in the process, each alternative (each model) will be rated with respect to the criterion directly above it.
File:AHPHierarchy2a.png
Alternatives for the Jones family car buying decision. To save space, we have represented them as stacks of papers.

Some simple terms apply to the boxes in the diagrams above. Each of the boxes is called a node. The boxes descending from any node are called its children. The node from which a child node descends is called its parent. Applying these definitions, the four Criteria are children of the Goal. The Goal is the parent of each of the four Criteria. Cargo Capacity and Passenger Capacity are the children of their parent, the Capacity criterion.

As they build their hierarchy, the Joneses should investigate the values or measurements of the different elements that make it up. If there are published safety ratings, for example, or manufacturer's specs for cargo capacity, they should be gathered as part of the process. This information will be needed later, when the criteria and alternatives are evaluated. Information about the Jones' alternatives, including color photos, can be found HERE.

Note that the measurements for some criteria, such as purchase price, can be stated with absolute certainty. Others, such as resale value, must be estimated, so must be stated with somewhat less confidence. Still others, such as style, are really in the eye of the beholder and are hard to state quantitatively at all. The AHP can accommodate all these types of criteria, even when they are present in a single problem.

Also note that the structure of the vehicle-buying hierarchy might be different for other families (ones who don't limit themselves to Hondas, or who care nothing about style, or who drive less than 5,000 miles a year, etc.). It would definitely be different for a 25-year-old playboy who doesn't care how much his cars cost, knows he will never wreck one, and is intensely interested in speed, handling, and the numerous aspects of style.[17]

Additional information. More information about hierarchies, including further reading and some complex real-world examples, can be found HERE.

Establish priorities

Once the hierarchy has been constructed, the participants use AHP to establish priorities for all its nodes. In doing so, information is elicited from the participants and processed mathematically. This activity is somewhat complex, and the participants have many options on the road to completing it. This and the following sections describe a simple, straightforward example of establishing priorities.

As our first step, we will define priorities and show how they interact.

Priorities defined

Priorities are numbers associated with the nodes of the hierarchy. By definition, the priority of the Goal is 1.000. The priorities of the Criteria (which are the children of the Goal) can vary in magnitude, but will always add up to 1.000. The priorities of the children of any Criterion can also vary but will always add up to 1.000, as will those of their own children, and so on down the hierarchy.

This illustration shows some priorities for the Jones car buying hierarchy. We'll say more about them in a moment. For now, just observe that the priorities of the children of each parent node add up to 1.000, and that there are three such groups of children in the illustration.

AHP hierarchy for the Jones family car buying decision, with some associated priorities.

If you understand what has been said so far, you will see that if we were to add a "Handling" criterion to this hierarchy, giving it five Criteria instead of four, the priority for each would be .200. You will also know that if the Safety criterion had three children, each of them would have a priority of .333.

In our example as it stands, the priorities within every group of child nodes are equal. In this situation, the priorities are called default priorities. As the analytic hierarchy process continues, the default priorities will change to reflect our judgments about the various items in each group.

As you may have guessed by now, the priorities indicate the relative weights given to the items in a given group of nodes. Depending on the problem at hand, "weight" can refer to importance, or preference, or likelihood, or whatever factor is being considered by the participants.

If all the priorities in a group of nodes are equal, each member of the group has equal weight. If one of the priorities is two times another, or three, (or whatever), that member has two, or three, (or whatever) times the weight of the other one. For example, if we judge cargo capacity to be three times as important as passenger capacity, cargo capacity's new priority will be .750, and passenger capacity's priority will be .250, because .750 = 3 × .250, and .750 + .250 = 1.000. (Don't worry—the AHP software does all this arithmetic automatically .)

AHP priorities have another important feature. The priority of any child node represents its contribution to the priority of its parent. In the diagram above, Cost, Safety, Style and Capacity each contribute .250 of the 1.000 priority of the Goal. Cargo capacity and passenger capacity each contribute half of the priority belonging to the Capacity criterion. Working through the arithmetic, Passenger Capacity contributes .500 × .250 = .125 of the 1.000 priority of the Goal.

As we move ahead through the Analytical Hierarchy Process, the priorities will change but will still add to 1.000 for each group of child nodes.

Criticisms

Despite its widespread use, there have been criticisms of the AHP: A) since there is no theoretical basis for constructing hierarchies, AHP users can construct different hierarchies for identical decision situations, possibly producing different solutions, B) AHP rankings are claimed to be arbitrary because they are based on subjective opinions using a ratio scale, a situation which can produce "rank reversal," C) there are flaws in the methods of combining individual weights into composite weights, and D) the process has no sound underlying statistical theory.

In 1990, a series of debates between critics and proponents of AHP were published in the journals Management Science[18][19][20][21] and The Journal of the Operations Research Society.[22][23][24]

In these debates, AHP critic J. Dyer argues:

"The AHP is flawed as a procedure for ranking alternatives in that the rankings produced by this procedure are arbitrary."[25]

Another AHP critic, R. D. Holder also concludes in 1991 that:

"Thus questions about the validity of AHP are far from having been settled. Also, it is clear that such discussion over a long period has not sufficiently penetrated the AHP users' community, with the result that paper are still being published using a faulty method."[26]

However, even while criticizing AHP, Holder finds room for possible improvement in AHP when he states in the same source:

"No amount of empirical evidence can truly validate a theory with such internal contradictions as the originally constituted AHP, though the theory is a good basis from which to develop."

Of the weaknesses found in AHP, the use of arbitrary scales and certain internal inconsistencies and theoretical flaws have been discussed extensively in the literature.

The use of arbitrary scales

AHP is based on pairwise comparisons where the relative importance of different attributes are given a value on a scale of 1 to 9 or the inverse (1/9th to 1). These values are in practice assigned by verbal elicitation of decision makers. For example, if a person says attribute A is "moderately more imporantant" than attribute B, A is said to have a relative weight of 3 times that of B while being "extremely more important" will give A a weight of 9 times that of B. While this scale is commonly used in AHP, it is arbitrary and alternative scales have been proposed. Empirical research has found "...that the perceived meaning of the verbal expressions varies from one subject to the next and also depends on the set of elements involved in the comparison." [27]. This is a well known result (since the 1930s) from Thurstone's work on attitude scales.Template:Fact However, these researchers felt the problem was correctable in that the scales could be based on empirical evidence of AHP user perceptions.

Inducement of nonexistant order

Another problem is the inducement of "nonexistant order" by innocuous changes even without the addition or deletion of suboptimal alternatives. AHP critic Stan Schenkerman writes in Decision Sciences:

"An apparently unreported problem facing decision makers who use AHP is described [in this paper]. It is demonstrated that conventional AHP and some of its variants (the ideal mode, and the pairwise aggregated approach, PAHAP) can induce ordering even when no order exists. It is also shown that all three approaches can induce different orderings and that the orderings are sensitive to innocuous changes. Thus, even absent addition or deletion of alternatives, the decision maker relying on AHP or these variants can be seriously misled." [28]

Responses by AHP proponents

Proponents argue that in spite of these concerns, the process works well in practice and is extremely popular among decision-makers in the private and public sectors.[29]

References

  1. 1.0 1.1 Bhushan, Navneet; Kanwal Rai (January, 2004). Strategic Decision Making: Applying the Analytic Hierarchy Process. London: Springer-Verlag. ISBN 1-8523375-6-7. 
  2. 2.0 2.1 de Steiguer, J.E. (October, 2003), The Analytic Hierarchy Process as a Means for Integrated Watershed Management, in Renard, Kenneth G., First Interagency Conference on Research on the Watersheds, Benson, Arizona: U.S. Department of Agriculture, Agricultural Research Service, at 736-740
  3. Berrittella, M. (January, 2007), An Analytic Hierarchy Process for the Evaluation of Transport Policies to Reduce Climate Change Impacts, Fondazione Eni Enrico Mattei (Milano)
  4. McCaffrey, James (June, 2005). "Test Run: The Analytic Hierarchy Process". MSDN Magazine. Retrieved on 2007-08-21.
  5. Grandzol, John R. (August, 2005). "Improving the Faculty Selection Process in Higher Education: A Case for the Analytic Hierarchy Process". IR Applications 6. Retrieved on 2007-08-21.
  6. Atthirawong, Walailak (September, 2002), An Application of the Analytical Hierarchy Process to International Location Decision-Making, in Gregory, Mike, Proceedings of The 7th Annual Cambridge International Manufacturing Symposium: Restructuring Global Manufacturing, Cambridge, England: University of Cambridge, at 1-18
  7. Dey, Prasanta Kumar (November, 2003). "Analytic Hierarchy Process Analyzes Risk of Operating Cross-Country Petroleum Pipelines in India". Natural Hazards Review 4 (4): 213-221. Retrieved on 2007-08-20.
  8. Larson, Charles D. (January, 2007), Application of the Analytic Hierarchy Process to Select Project Scope for Videologging and Pavement Condition Data Collection, 86th Annual Meeting Compendium of Papers CD-ROM, Transportation Research Board of the National Academies
  9. Participant Names and Papers, ISAHP 2005, Honolulu, Hawaii (July, 2005). Retrieved on 2007-08-22.
  10. 10.0 10.1 Sun, Hongkai (July, 2005), AHP in China, in Levy, Jason, Proceedings of the 8th International Symposium on the Analytic Hierarchy Process, Honolulu, Hawaii
  11. List of Expert Choice education clients. Retrieved on 2007-08-23.
  12. Drake, P.R. (1998). "Using the Analytic Hierarchy Process in Engineering Education". International Journal of Engineering Education 14 (3): 191-196. Retrieved on 2007-08-20.
  13. Bodin, Lawrence; Saul I. Gass (January, 2004). "Exercises for Teaching the Analytic Hierarchy Process". INFORMS Transactions on Education 4 (2). Retrieved on 2007-08-20.
  14. Hallowell, David L. (January, 2005). "Analytical Hierarchy Process (AHP) -- Getting Oriented". iSixSigma.com. Retrieved on 2007-08-21.
  15. "Analytic Hierarchy Process (AHP)". QFD Institute. Retrieved on 2007-08-21.
  16. "Analytical Hierarchy Process: Overview". TheQualityPortal.com. Retrieved on 2007-08-21.
  17. 17.0 17.1 17.2 17.3 Saaty, Thomas L. (1999-05-01). Decision Making for Leaders: The Analytic Hierarchy Process for Decisions in a Complex World. Pittsburgh, Pennsylvania: RWS Publications. ISBN 0-9620317-8-X.  (This book is the primary source for the sections in which it is cited.)
  18. Dyer, J. S. (1990): Remarks on the Analytic Hierarchy Process. In: Management Science, 36 (3), S. 249-258.
  19. M. V. Mikhalevic "Remarks on the Dyer-Saaty controversy" Cybernetics and Systems Analysis,Volume 30, Number 1 / January, 1994
  20. Patrick T. Harker, Luis G. Vargas, "Reply to 'Remarks on the Analytic Hierarchy Process' by J. S. Dyer", Management Science, Vol. 36, No. 3 (Mar., 1990), pp. 269-273
  21. Dyer, J.S. (1990b), "A clarification of 'Remarks on the analytic hierarchy process'", Management Science, Vol. 36 No.3, pp.274-5.
  22. Holder, R.D., Some Comment on the Analytic Hierarchy Process, Journal of the Operational Research Society, 1990, 41, 11 1073-1076.
  23. Thomas L. Saaty "Response to Holder's Comments on the Analytic Hierarchy Process" The Journal of the Operational Research Society, Vol. 42, No. 10 (Oct., 1991), pp. 909-914
  24. R. D. Holder "Response to Holder's Comments on the Analytic Hierarchy Process: Response to the Response" The Journal of the Operational Research Society, Vol. 42, No. 10 (Oct., 1991), pp. 914-918
  25. Dyer, J. S. (1990): Remarks on the Analytic Hierarchy Process. In: Management Science, 36 (3), S. 249
  26. R. D. Holder "Response to Holder's Comments on the Analytic Hierarchy Process: Response to the Response" The Journal of the Operational Research Society, Vol. 42, No. 10 (Oct., 1991), pp. 914-918
  27. MARI A. PÖYHÖNEN, RAIMO P. HÄMÄLÄINEN, AHTI A. SALO "An Experiment on the Numerical Modelling of Verbal Ratio Statements" Journal of Multi-Criteria Decision Analysis,vol 6, no 1, ppg 1-10, 1997
  28. Stan Schenkerman "Inducement of nonexistent order by the analytic hierarchy process", Decision Sciences, Spring 1997
  29. de Steiguer, J.E. (October, 2003), The Analytic Hierarchy Process as a Means for Integrated Watershed Management, in Renard, Kenneth G., First Interagency Conference on Research on the Watersheds, Benson, Arizona: U.S. Department of Agriculture, Agricultural Research Service, at 736-740