There was a time when virtually all students pursuing the master's degree were required to submit to and successfully complete comprehensive examinations, most of which featured both written and oral elements. In this more traditional period, the acquisition of the master's degree was in large part a qualifying ritual for admission to a Ph.D. program which, in turn, assumed a subsequent professional life in the academy. With the gradual shift in the character of the degree from the purely academic to the at least quasi-professional, alternative approaches to evaluating student performance are emerging as exit requirements for the master's degree. This paper reports the results of a survey, featuring both quantitative and qualitative components, designed to investigate the status of the traditional comprehensive examination, as well as the nature and distribution of alternative assessment methods.

Much of what occurs in colleges and universities, from curriculum to governance structure, proceeds on a normative assumption, i.e., that there are principles which are fundamental to the practice under investigation. It is assumed, for example, that students will be required to follow programs of study which are carefully crafted to adhere to what is considered "normal" for the field and to the standards and expectations of accrediting bodies. Thus, if one were to pick up a college catalogue in order to investigate which courses are required for the MA in applied mathematics, it would be surprising if many were to deviate from the commonly prescribed curricular substance and sequence.

The issue of the comprehensive examination for the master's degree appears to conform to this phenomenon. Singletary (1999) observed that "the extent of reliance on master's comprehensive examinations is undocumented but believed to be strong," suggesting that the requirement that candidates for the master's degree demonstrate competence by successfully passing an examination remains standard.

First, we needed to verify Singleton's contention that at some time in the past the reliance on a comprehensive examination was "strong." When we surveyed 50 graduate catalogs for the 1983-1984 academic year we found that most graduate schools required a comprehensive examination-either written or oral-at the master's level, with the exception of students doing a thesis. In those cases, the defense of the thesis substituted for a comprehensive exam (and may, in fact, have been considered equivalent.) Of the catalogs surveyed, 86% explicitly mention the comprehensive exam. Those that fail to mention such an exam may still have required one given the culture of the time and the implicit assumption that graduate students were supposed to take "comps."

With the master's degree considered preparation for the Ph.D., master's candidates were expected-just like doctoral students-to write theses and take grueling written examinations. If one could handle these challenges at the master's level, presumably one was a reasonably good candidate for a Ph.D. program. And, if one failed to complete the doctorate, one at least had an academically legitimate consolation prize.

A search of several databases which archive academic literature seems to confirm the suggestion that the comprehensive exam retains a level of ubiquity. A search in Academic Search Elite, which provides full text articles from 1,530 periodicals dating back to 1990 and offers abstracts and indices from 2,720 periodicals back to 1984, returned only 16 references under the subject "comprehensive examination," 11 of which were related to standardized testing in the K-12 sector. Of the remaining five references, two concerned study guides for an engineering examination, one the master's examination in economics, one comprehensive examinations at the doctoral level, and one the relationship between comprehensive and licensing examinations for the doctoral degree in psychology. When the search string was changed to "comprehensive assessment," 15 references were returned, 13 of which focused on K-12 concerns. The remaining two involved articles on a comprehensive health care network and on an analysis of the dissemination of public information conducted by the National Council of Libraries.

Expanding the search banks to include the Education Resources Information Center (ERIC) and Ebsco databases in addition to Academic Search Elite returned more of the same pattern. While the number of returns was increased (32 returns to "comprehensive examination" and 50 to "comprehensive assessment"), the vast majority maintained the focus on standardized testing at the K-12 level. Of the eight articles with any relationship to higher education, seven examined subject-specific examinations, e.g., the relationship between a comprehensive examination in nursing and subsequent performance on the National Council Licensure Examination (NCLEX), and the other the implementation of a portfolio-based assessment in an educational administration program. Searches using the string "master's" + "comprehensive" + "examination" returned no citations.

One might assume, from the apparent scarcity of citations, that Singletary's (1999) observation is correct. It seems plausible, however, to at least attempt to confirm it in light of emerging challenges to the orthodoxy implicit in assessment rhetoric at both the undergraduate and graduate levels. This discourse has been accompanied by a variety of progressive approaches, among them an increasing acceptance of qualitative measures, which some characterize as less authoritative but more illuminating than the quantitative measures which have dominated the field for literally decades. Among the more visible of these approaches is portfolio assessment. The portfolio, while still presently used primarily in conjunction with some more conventional evaluation methods (e.g., licensing examinations, certification tests, etc.), has been gaining the support of both students and faculty across disciplines, and has had a significant impact on teacher education programs (Darling-Hammond et al., 1995; Holmes Group, 1986; Moss, 1997; National Board, 1989; Schon, 1987).

Several graduate programs in our own university have been among those making the transition from the more conventional comprehensive examination to alternative forms of assessment for master's degree candidates. Completion of the master's degree in educational administration and in elementary and secondary education are among those which have undergone a transformation in their assessment practices, requiring the development of a portfolio which demonstrates students' competencies as they relate to the standards of the appropriate accrediting body. It was this shift, in both our own programs and in others of which we are aware, which prompted the current study in order to discern whether the transformation of assessment practices is, in fact, widespread, or whether reliance on those more traditional measures remain as strong as Singletary (1999) implies.

Recent decades have seen a significant increase in challenges to what is variously described as the modernist or logical positivist approach to conducting research from methods which are more closely aligned with postmodernist or postpositivist thinking. Modernist/positivist methods, most closely identified with quantitative research, generally take for granted a kind of binary logic that assumes the phenomenon in question either does or doesn't conform to preconceived hypotheses. Research questions, thus, can be neatly divided into categories with discrete and mutually exclusive responses.

People with a more postmodern approach to research, on the other hand, stress the importance of context and perspective, questioning the appropriateness of attempting to reduce complex questions to simple yes/no proportions. Many wonder if it is even possible to undertake a scientific, objective study in a field which is characterized as "uncertain, unstable, complex, and often unique" (Cunningham and Cordeiro, 2000, p. 5). We find this question legitimate. We also, however, find it unnecessarily limiting to reject traditional quantitative research methods on the ground that they are insufficiently nuanced to be of any use in a less than tightly structured study. We share with Griffiths (1995) an interest in a research agenda which is problem-focused and which adopts a theoretically diverse approach.

This study, thus, relies on a combination of both quantitative method and qualitative analysis. It blends responses to discrete questions with an examination of responses to open-ended questions in order to derive as complete a picture as possible of the state of the comprehensive assessment as the culminating experience for the acquisition of the master's degree in the institutions surveyed. It is an attempt to address what we see as two related issues: 1) the nature of the comprehensive assessment and 2) reasons for any changes institutions have made in their policies concerning that assessment.

A survey instrument featuring 11 questions, with a request for respondents to elaborate on certain responses, was sent to 450 members of the Council of Graduate Schools. In order to generate the largest possible number of completed surveys, respondents were invited to choose whether to answer questions via the Internet, where we established a Web site for the survey, or to complete and return the paper copy. Of the 11 questions, two were identifiers (i.e., whether the respondent represented a public or private institution and whether the institution is research extensive, research intensive, master's I or master's II), and one question asked whether the respondent would agree to a follow-up interview. Eight focused on the research question itself.

Of the 450 surveys mailed, 115 were returned by mail and 38 via the Internet, for a total of 153 (a response rate of 34%). Following standard quantitative research measures, such a response rate fails to conform to expectations for generalizability. Given the interest colleges and universities maintain in assessment practices, however, we thought it useful to proceed with a quantitative analysis of the data nonetheless. If the results of the research are to be used to inform decision making at the administrative levels in any way, questions of validity and reliability may be raised as a result of the return rate. In this study, we establish frequencies of responses and percentages of totals in order to quantitatively array the data ¹. Validity can be looked at as the responsiveness compared to the population surveyed, based on certain characteristics such as institutional type and geographical location.

It is in the explanations for certain responses, however, that much of the valuable and interesting data lie - in narrative which cannot be reduced to frequencies and percentages. For that reason, our analysis of the narrative data, and of the comments of respondents who agreed to subsequent interviews concerning the subject, is conducted from a qualitative perspective.

Phenomenological analysis, the qualitative approach we have chosen, involves a mutual relationship, one in which the interviewer and the interviewee(s) are equal participants or "co-researchers" (Nelson, 1989) in the effort to gain some insight. Rather than adhering to a format of carefully structured questions, the protocol features an open-ended dialogue wherein the participants can generate data together "in a social context where [they] can consider their own views in the context of others" (Patton, 1980, p. 335). It is a process we think appropriate for examining more closely this phenomenon in which we share an interest with our peers.

Lanigan (1988) describes phenomenological analysis as conceived by Merleau-Ponty, widely regarded as the seminal thinker in the field, as a three-step process requiring description, reduction and interpretation. Each step informs the next in a progressive fashion to yield what Lanigan calls "a systemic completeness" (p. 173).

• The descriptive phase involves a careful reading of or "listening to" the data absent any preconceived ideas or classifications. The intent at this stage of the analysis is to allow for "the widest possible number" of broad themes to emerge (Nelson, 1989, p. 232).



• The second step, reduction, extracts from the emerging themes those which can be seen as constitutive. The aim of reduction, as explained by Cooks and Descutner (1994), is "articulat[ing] ... a 'pattern of experience' expressed through the essential elements of the phenomenon under investigation" (p. 255).



• Phenomenological interpretation, then, requires the examination of the primary themes which emerge to discern those which effectively make explicit (in this case, the current status of the comprehensive examination) what had formerly been implicit (the assumption that its use continues to be widespread).

In short, we attempt to analyze the data from both a discrete and a holistic perspective, although a more complete analysis will occur subsequent to the follow-up interviews. After breaking down the percentages and frequencies where appropriate, we turn to the narrative data (examined in the description phase), reduce it to some primary themes, which we then re-examine in an attempt to determine which clarify in a useful or meaningful way our understanding of the state of the comprehensive examination. Having described the data-gathering procedure, we now turn to the handling of the collected data.

The survey questions involving content (as opposed to classification of institutions responding), first concerned whether there exists an institutional policy requiring some kind of comprehensive assessment experience for all master's degree students. The majority of respondents responded positively, as is demonstrated in Table 1 below.

While the comprehensive/capstone assessment remains the standard of good practice at the master's level, a university-wide policy for all programs to require a capstone experience is not as universal as it apparently once was. This appears to reflect the increasing professionalization of the master's degree and the increasingly part-time nature of graduate students. This tendency may become more pronounced with the growing success of the Sloan Foundation initiative to professionalize the M.S. in science.

Respondents reported that individual departments, programs, schools or colleges within the institution have fairly broad latitude as to the form such a comprehensive assessment will take. While some respondents cited the traditional thesis coupled with a comprehensive examination involving both written and oral components, they were few in number. Among the alternatives mentioned as exit requirements were research projects, research courses or symposia, capstone courses, creative projects or performances, exhibitions, internships, practica or clinicals, and portfolios.

If ever the comprehensive examination predominated in MBA and other business programs, it appears to have been surpassed by capstone courses and/or capstone projects which emanate from those courses. Although we did not inquire about the nature of capstone projects in business programs, we speculate that they are related to real world problems-either problems which have recently been "solved" and now stand as case studies or problems with which businesses have asked our colleges and universities to help them deal.

In science, despite the Sloan Foundation's recent initiative, the thesis rather than the comprehensive exam dominates the responses. Whether theoretical or practical, the thesis and its oral defense appears designed to give the student an intimate experience with the research methods he or she, presumably, will be using either in "the real world" or in a doctoral program (the thesis serving as practice for the dissertation).

The fine arts, reasonably enough, primarily offer students a supervised showcase for their talents. Although it may be called a "comp," in fact in many cases it is an exhibit or performance and it is unlikely the faculty would let the student get this far without the expectation the student would graduate.

The liberal arts utilize the traditional written comprehensive examination, but the thesis appears to be a more popular option. The capstone course and the capstone project have made their appearance and augur the presence of viable alternatives to the written comps and thesis.

In nursing and the health professions, clinical experiences such as internship and practicum are frequently cited. A research element, however, remains strong whether through the research report, master's paper, special project or thesis. One university listed a "scholarly inquiry endeavor."

The professional project in journalism is offered as an alternative to both the thesis and the comprehensive examination. In education, portfolio assessment has not yet replaced the comprehensive exam entirely, but there is growing respect for a process which emphasizes student products which evolve over time and reflect preparedness to enter the teaching profession.

Among the other major options is the final project, typically found in engineering and information systems/management systems programs. According to one correspondent, the final (or comprehensive) project, by involving the application of management, engineering and scientific techniques, offers "a good assessment of a student's ability to collect information and data about a particular problem, analyze this material, synthesize a solution, invoke the solution, and effectively communicate this process in both an oral and written format."

In summary, the type of closure experience which encouraged the student to cram for a multi-hour written examination no longer holds the hegemony it once did. Instead, a host of options is now available.

While most respondents indicated that the internship in and of itself is insufficient to serve as the culminating experience, more than one-third reported that their institutions do accept such arrangements. Internships have been included, of course, as an element in programmatic requirements for some time. Their emergence as the sole comprehensive experience, however, may come as a surprise to some.

Programs which allow the internship to replace the comprehensive exam presumably do so because they believe the student's performance in the internship reflects all that he or she has learned in the didactic courses, as well as the student's ability to use what he or she has learned in an appropriate and effective manner in practice. The proof of graduation is in the pudding of the internship.

The fields in which internships were implemented provide some interesting information as well. It might be expected, for example, that internships in such applied fields as business, psychology, counseling or allied health would be common. Survey participants, however, also reported the presence of internships in fine arts, physics, history, and geography.

The subject of using the comprehensive assessment/capstone experience to validate outdated coursework, Question 5, produced the most significant figures. As demonstrated in Table 4, an overwhelming number of respondents reported that their institutions do not permit such a practice. Because an explanation was invited from only respondents who indicated that their institutions allow validation via the comprehensive assessment, we can only speculate as to the reasons for non-acceptance. It seems reasonable to assume, however, that most institutions conceive of the processes as distinct. A revalidation exam is designed to demonstrate both retention of course content and in-depth currency of subject matter within a particular course, whereas a comprehensive assessment allows students to demonstrate their ability to synthesize, integrate and interrelate what they learned in a broad range of classes. The comprehensive assessment would not necessarily cover all significant content in every outdated individual class.

Respondents who reported that their institutions do accept the substitution of the comprehensive assessment for outdated coursework were invited to elaborate on their answers, but most addressed issues of specificity as opposed to justification. They wanted to clarify, for example, that only a traditional comprehensive examination would be acceptable, as opposed to an alternative form of assessment. Others mentioned that there are limitations related to the number of courses which could be validated (a maximum of two courses was the norm), as well as student performance on the assessment itself (e.g., a minimum score of 75%). One respondent noted that courses students have completed at non-accredited institutions and wish to transfer, if they reside within the confines of the institution's policy concerning acceptable age of the coursework (e.g., completed no longer than seven years prior), can be so validated.

At the master's level most institutions do not allow revalidation through means of the comprehensive exam. Instead, they tell students to retake their outdated classes, update through a special exam for each individual class, update through independent study, or substitute other coursework on the students' Plans of Study. A number, such as Marshall, are now turning to portfolio assessment as an additional option.

Question 6 also produced some fairly definitive responses. The norm for number of faculty comprising MA comprehensive assessment committees has been three for many years, and remains the most popular among our respondents. With more than one-quarter of the survey participants reporting fewer than three, however, it should be noted that the judgment as to whether a student has successfully completed the comprehensive assessment/capstone course often rests with only one or two faculty members.

Perhaps the reduction constitutes a concession to the reality that, in the face of increasing numbers of part-time faculty, too much work is being required of too few full-time faculty members. That is, if a student's performance is to be the product of a majority vote, it may be perceived a pointless allocation of a scarce resource to require a third reader if the first two agree. Whatever the reason, the concept of a three-person committee at the master's level is no longer an absolute rule.

Turning to question 7, the current mantra is that interdisciplinary teamwork results in better science and reflects the way of the "real world," whether it is business, education, nursing or whatever works. Nonetheless, the majority of deans heaved a collective gasp of horror at the idea of students' participating in a group comprehensive assessment. While acknowledging that there is a role for collaboration at a certain level, they believed that a judgment, ultimately, must be made about each individual student's success in having met the requirements for graduation. They feared that team projects reward the weak and penalize the superior students. For them, it appeared a matter of fairness.

This particular question invited elaboration from all respondents, and provided additional insight. Among respondents who indicated the existence of collaborative projects, one raised a potentially problematic issue: the incursion of technology into virtually every nook and cranny of the academy. Explaining the institution's choice to accept a collaborative project in a technology program, he noted that the institution's "multimedia M.A. students do a team project because no one person can do a professional multimedia project by himself or herself." While such a position is perhaps arguable (e.g., should a student who plans a career in multimedia have a basic operational grasp of multiple technologies and be able to demonstrate her competence across them by herself?), it is neither our intent nor our place to question institutional policy.

Rather, we found this particular observation interesting for the questions it raises concerning the use of technology in the academy in general and the expectations held by institutions and programs (imposed in large part by accrediting agencies) relative to student technological competency. If, for example, a student in a M.Ed. program is required to compose a multimedia presentation demonstrating his mastery of National Council for Accreditation of Teacher Education (NCATE) standards in partial fulfillment of degree requirements, and he is assisted by another student whose technological expertise surpasses his own, does that render the final product a collaborative one? If students are required to submit their theses electronically and do so by acquiring the assistance of a technologically savvy peer or even professional, have they failed to conform to procedure?

Certainly these are questions to which the answers are specific. If the requirement for an electronic thesis submission is made for reasons of institutional expedience rather than an attempt to assess students' technological competence, one supposes the issue of who handles the actual submission is of little import. If, however, programs are implementing requirements for culminating experiences which involve technological elements on the ground that they can assess students' competence in line with institutional or accrediting agency expectations for technological literacy, the issue could become problematic. How much assistance can a multimedia expert provide before her involvement renders the project "collaborative"?

The number of institutions allowing team or collaborative comprehensive assessments does, in any case, appear to be growing. That finding is consistent with what was revealed in responses to the previous questions: the nature of the comprehensive process is changing, from its composition to its presentation to the number of faculty who evaluate it.

A variety of methods is used to prepare students for their comprehensive assessment ranging from literature lists, providing sample (old) exams, posting topics, seminars designed to review content and applications, to discussions with (and perhaps helpful hints from) either the major professor or members of the student's graduate committee.

One thing that is clear is that the individual departments and programs take the lead here. The Graduate Dean's Office does not claim a central role in assisting students at this juncture in their academic travail.

With this explication of the quantitative elements of our study, we turn to the qualitative analysis of our data - the descriptive, reductive and interpretive elements.

Description. Lanigan (1988) reminds researchers that it is important at the description phase to attend to elements in participants' discourse which demonstrate their "awareness of what [the] phenomenon is" (p. 337). Those involved in this study revealed a clear understanding of their institutions' assessment practices. From our analysis of their comments, three broad themes emerged.

First, the question of institutional policy regarding both the value and existence of some form of comprehensive assessment appears to rest on historical assumptions and professional recommendations. The American Association of University Professors (AAUP) in 1935 recommended that the master's degree culminate in a thesis and a comprehensive examination, a recommendation which the AAUP reiterated in 1945 (pp. 10-11, Conrad).

We have witnessed more of a retreat from the thesis requirement than from the comprehensive examination. "As the master's became less a traditional research degree, especially in the arts and sciences but also in some professional fields, many master's programs either dropped the thesis requirement altogether or introduced a 'non-thesis' option [Boddy 1970]" (cited in Conrad, p. 22). While this appears increasingly to be true, the comprehensive exam has largely survived, albeit in ways that significantly redefine and transform the term.

Second, the culminating or closure experiences at American master's institutions are exceptionally diverse. It would no longer be accurate to say, for example, that "the written comprehensive examination is the most common requirement in education master's degree programs" (Jenkins and Douzenis, 1998). Specific choices concerning appropriate methods for assessing students' competencies largely reside at the college, school or departmental levels.

Third, the nature of the comprehensive assessment has changed significantly. As is becoming increasingly apparent, the goal of the comprehensive assessment at the master's level is not merely to measure what the student has learned but, just as importantly, to demonstrate how he or she applies what he or she has learned. Various projects and portfolio assessments permit the student to show competency to enter a professional field-or to earn advancement in his or her field. Some academic units, to be sure, continue to require the traditional comprehensive examination with both written and oral elements. Alternative forms of evaluation, however, appear across the disciplinary board. This development is not surprising. We might have cause to worry, in fact, if faculties failed to take advantage of research indicating that a breadth of evaluative practices is more likely to elicit an accurate view of student performance.

Not everyone is happy about the de-emphasis of the thesis. As reported by Robert M. Jackson in a December 14, 2001, e-mail , a number of California State University deans deplore this change because

"the thesis and project are viewed as requiring a sustained demonstration of independent research skills or the creation of an educational/professional product in the case of a project (software, instructional materials, training manual, etc.). This is viewed as a more demanding challenge than a few hour exam."

However, few academicians any longer view the comprehensive assessment as merely a few hour exam.

It is not unusual that such culminating experiences as research symposia, internships or practica, capstone projects and the like be part of the student's requirements for completing the master's degree. These are fairly universal elements in graduate programs, and are often used in conjunction with the routine examination. That some of these alternative approaches have replaced the conventional comprehensive examination, however, and in not only a few instances, suggests the emergence of a significant change in assessment practice.

Reduction. What the previous descriptions represent are expressions of participants' connotative understandings of comprehensive assessment practices. The reduction step in the analytical process requires us to extract from those descriptions those which can be characterized as essential elements and formulating them into "concise expressions" (Nelson, 1989, p. 235). Our analysis reduces these elements to two such expressions:

• Some form of comprehensive assessment remains imperative in the academy. While the institutional representatives who responded to the survey acknowledged changes, some of them significant, in the character of assessment practices, all expressed a commitment to maintaining the integrity of the master's degree by ensuring that students demonstrate in some fashion their acquisition of the knowledge, skills and dispositions it represents.



• Decisions concerning the nature of the comprehensive assessment should be made at the level of the academic unit which certifies that the student has mastered the requirements for the awarding of the degree.

Interpretation. The final step in the phenomenological analysis is to examine the themes and their subsequent reductions in an attempt to make explicit what has previously been implicit. In this study, implicit feelings or impressions about the changes in the nature of comprehensive assessment were transformed into explicit examination of current practices. If we were to isolate, as Cooks and Descutner (1994) recommend, "a revelatory phrase" from the interviews, something we believe is the central premise or "the signified in the discourse" (p. 260), we would need to locate a statement that makes clear the meaning of culminating assessment for people involved in institutional policy.

The respondents, as can be seen from their descriptions of assessment practices, did seem to share a sort of a priori understanding of the dimensions of comprehensive assessment, particularly as it relates to the traditional process. Unlike many phenomenological studies, wherein participants' descriptions tend to demonstrate a kind of fluid, mutable character in recognition that judgments of the phenomenon in question are highly personal, this investigation is notable for the degree of consensus it reflects. The smallest majority response, for example, was 60% (on Question 7). Other majorities, as can be seen in the previous tables, ranged from 73% to 84%.

The statement we believe best makes explicit our respondents' understanding of the status of institutional policy as it relates to the comprehensive assessment is this: Assessment is both a normative and a variable enterprise; while comprehensive assessment remains a critical element in master's level graduate education, its dimensions are quite diverse.

In reading the comments of these interviews, we find more than an occasional trace of postmodern thinking. The modernist notion of a traditional comprehensive examination which is both the exclusive and conclusive instrument for determining whether a student merits the awarding of the master's degree bears little relation to what emerged from the discourse. Postmodernism accepts the existence of indeterminacy as a given in the contemporary culture the central characteristic of which is its lack of homogeneity. It is an understanding which confirms Cunningham and Corderio's (2000) description of educational administration, the endeavor in which we are all involved, as a field which is characterized as "uncertain, unstable, complex, and often unique" (p. 7).

The compatibility of our data with this postmodern perspective leads us to two observations, both grounded in Griffith's (1995) claim that much contemporary research engages in theory-building at the expense of problem-solving. First, as is related to the frameworks under which research in educational administration is conducted, we find Griffith's advocacy of theoretical pluralism to have merit. The data we were able to glean from the kind of open-ended, dialogic approaches common to qualitative analyses generated information which would have been difficult if not impossible to acquire through a conventionally structured survey that controls for and thus limits respondents' ranges of response. The dialogic element enhanced the richness of the accompanying quantitative analysis by broadening the scope of the investigation.

Second, in terms of problem-solving, we believe the study contributes to our ability to challenge the broadly held public perception that university expectations are out of touch with the professional needs of today's students and their eventual employers. People who continue to conceive of universities as ivy-covered cloisters mired in the academic practices of the last century might even be cheered by the multitude of options available to students for demonstrating their competence.

Are there conclusions which may be safely drawn from these observations or from the study itself? As was noted at the outset, the small number of respondents relative to both the membership of this organization and to the number of institutions of higher education represented herein necessarily mitigates that possibility. Our findings, thus, are more suggestive than conclusive.

That said, however, it is reasonable to assume that these respondents are representative of institutions nationwide, both public and private, and across classifications. To that extent, the responses they made, if not generalizable, remain informative. We would be surprised, in fact, if a more thorough investigation yielded results significantly different from those reported herein. Still, a study of broader proportion is a recommendation we are inclined to make, if only to confirm the transformation of assessment practices our respondents indicate.

Were such a process to be undertaken, we would also recommend that the researchers assist the respondents by taking more care when constructing survey questions (e.g., clarifying "capstone experience" by drawing a distinction between a capstone course and a capstone project, or defining such terms as "internship" and "practicum" or "symposium"). These revisions would be helpful to researchers as well, reducing the possibility of misinterpretation. Finally, now that we feel we can claim with reasonable assurance that a transformation of assessment practice is occurring, we would be inclined to ask participants to offer their perspectives on why the more traditional comprehensive examination has been marginalized or replaced altogether in some fields which had previously used it as the sole exit requirement. Answers to that question may prove even more enlightening than what we found thus far.

The meat of our study is found in response to question 3-the type(s) of comprehensive assessment or capstone or closure experience conducted in master's institutions. The responses were remarkably diverse. Clearly, much has changed since we went to graduate school.

As recently as 1998, Jenkins and Douzenis proclaimed that: "The written comprehensive examination is the most common requirement in education master's degree programs" (1998). However, it would no longer be accurate to make this claim for education, or for any other academic discipline.

As is increasingly apparent, the goal of the comprehensive assessment at the master's level is not merely to measure what the student has learned but, just as importantly, to demonstrate how he or she applies what was learned. Various projects and portfolio assessments permit the student to show competency to enter a professional field-or to earn advancement in his or her field.

Our findings are likely to revive an old debate from a new perspective. The debate concerns the quality of graduate education at a time when institutions are competing to increase the quantity of their students. In this context, some will argue that comprehensive assessments are becoming less rigorous-that the current situation represents a dumbing down of graduate education-reflecting the fact that we are scrambling to recruit (and graduate) students who would not have been accepted into our programs in the past. Others will argue that the alternatives outlined in this paper to the traditional written comprehensive examination demonstrate that universities have developed more reliable, realistic tools for measuring what a student has learned during an extended period rather than what he or she can memorize in a week of cramming and then promptly forget. Whatever the explanation and whatever its ultimate import, the comprehensive assessment is clearly in the process of undergoing significant, pervasive change.

Given the findings of this study, it would be helpful for the Tennessee Conference of Graduate Schools or some other organization-such as CGS- to conduct a follow up to Singleton's 1999 study, this time focusing on defining the various closure requirements in different disciplines and offering best practices. What, for example, constitutes for good portfolio assessment? In fine arts, when one institution says its students must present a capstone project and another institution says students must present an exhibit or recital to graduate, what distinction-if any-are the two institutions making about the nature of their respective culminating assessments?

If an internship is used in lieu of a comprehensive exam, how does the capstone internship differ from internship courses which merely place students in so-called real-life situations? How does the culminating internship serve as an integrative summary of all the student has learned?

A further consideration is this: if all these developments have occurred in the last 10 years or so, perhaps it is time to ask if the nature of the culminating experience is changing at the doctoral level as well.

This edition of IR Apps is commendable from several perspectives, both methodological and focus. First, in terms of methodology, Deutsch and Nicholson work to bring together both the qualitative and the quantitative methodologies. In several of his works Quinn Patton argues that the strength of methodology is in the balance. The article also mentions the presence of a paradigm that guides the assessment methods in many graduate schools. Is it possible that the deans of these schools came from disciplines such as the natural sciences where the positivist or post-positivist paradigm dominates? The reader is encouraged to go to a discussion of paradigms such as that by Gupta in The Paradigm Dialog (Sage, 1991, Newbury Park, CA) and consider the relevance of other paradigms such as the Constructivism or the Critical Theory approach to the validation of learning.

The second commendable aspect of this research is the focus it takes. In some of my own previous research I have been forced to concur with the authors. There is a very serious lack of research on our graduate programs. All of our focus on learning outcomes as institutions seems to be on undergraduate experiences and learning. Even some of the professions, such as those in engineering, focus on undergraduate education. It would seem that institutions, in conjunction with professions, should establish and aggressively pursue a research agenda looking at graduate learning if we are to expect our graduate programs to be strengthened through management and assessment of learning. How should such an agenda be established? Who should do what?

The third commendable aspect of this research is its existence. Several colleagues who had a vital interest in a topic seem to have established a systematic procedure and reached a point where at least initial conclusions can be drawn. They did not use sophisticated statistics and this does not seem to be a problem. Unfortunately they did not get a high return rate and this may also be indicative of range of perspectives on the importance of their question - or it may be indicative of bad addresses. Their partnership with the Council of Graduate Schools will hopefully give the results visibility and will cause additional assessments of graduate level expectancies.

¹ It should be noted that not all participants responded to all questions. This accounts for the differences in frequencies and percentages. The tables note the total number of responses to each question.

Conrad, C. F., Haworth, J. G. & Millar, S. B. (1993). A silent success: Master's education in the United States. Baltimore: Johns Hopkins University Press.

Cooks, L., & Descutner, D. (1994). A phenomenological inquiry into the relationship between perceived coolness and communication competence. In K. Carter and M. Presnell (Eds.), Interpretive Approaches to Interpersonal Communication (pp. 247-68). Albany: State University of New York Press.

Cunningham, W., & Cordeiro, P. (2000). Educational administration: A problem-based approach. Boston: Allyn and Bacon.

Darling-Hammond, L., Ancess, J., & Falk, B. (1995). Authentic assessment in action: Studies of schools and students at work. New York: Teachers College Press.

Griffiths, D. (1995). Theoretical pluralism in educational administration. In R. Donmoyer, M. Imber & J. Scheurich (Eds.), The knowledge base in educational administration: Multiple perspectives (pp. 302-311). Albany: State University of New York Press.

Holmes Group. (1986). Tomorrow's teachers: A report of the Holmes Group. East Lansing, MI: Author. Griffiths, D. (1995). Theoretical pluralism in educational administration. In R. Donmoyer, M. Imber and J. Scheurich (Eds.), The knowledge base in educational administration: Multiple perspectives (pp. 302-11). Albany: State University of New York Press.

Jackson, R. M. E-mail to Leonard J. Deutsch, December 14, 2001.

Jenkins, S. J., & Douzenis, C. Factors related to master's comprehensive examination performance, College Student Journal. Sept. 1998, 32 (3), p. 370.

Lanigan, R. (1988). Phenomenology of communication: Merleau-Ponty's thematics in communicology and semiology. Pittsburgh: Duquesne University Press.

Moss, P. (1997, March). Developing coherence between assessment and reform in the licensing and professional development of teachers. Paper presented at the annual meeting of the American Educational Research Association, Chicago.

National Policy Board for Educational Administration. (1989). Improving the preparation of school administrators: An agenda for reform. Charlottesville, VA: Author.

Nelson, J. (1989). Phenomenology as feminist methodology: Explicating interviews. In K. Carter and C. Spitzak (Eds.), Doing research on women's communication (pp. 221-241). Norwood, NJ: Ablex.

Patton, M. (1980). Qualitative evaluation methods. Beverly Hills: Sage Publications.

Schon, D. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions . San Francisco: Jossey-Bass.

Singletary, M. (1999). Graduate examinations. Knoxville, TN: Tennessee Conference of Graduate Schools.

We would like to acknowledge: Les Sims, Tom Feldbush, Frank Biafora, and Dorothy Suomala for pilot testing the draft version of our questionnaire, and the Council of Graduate Schools for providing mailing labels. The survey was approved by Marshall University's Institutional Review Board.