I posted this over a year ago before anyone probably knew this blog even existed. I am reposting it now for two reasons: 1) I have raised this critique since then and found that a lot of my Presbyterian Church (USA) friends agree with it. 2) The issues about assessment that I raise seem to have no answers. I posting this in the hopes that those currently taking these exams don't read it and then get upset off before they finish them up this weekend. That would not sit well and create anxiety that is not helpful. But when they are done, this is a reflection worth having. I passed all exams first try which was great, but I never knew what it actually meant. I am even less sure now that I know something about educational and professional assessment.
I think that the process lacks assessment rigor and needs to change if my assumptions here are at all correct.
My point is not so much about gate-keeping as it is with how we measure observable outcomes and expectations and how those standards are used to train ministers. Do the ordination exams have standard outcomes that are regularly reviewed? Are the five exams tested for their effectiveness to meet those outcomes on a regular basis? Are these results shared with Presbyteries in order to improve preparation programs? Are there standards for preaching that Presbyteries of care and search committees can use to determine best of fit or levels of preparedness? Is it that much based on not getting a 1 0r 2 rating on your exams? Moreover, what do those ratings mean? Are they tied to a standard rubric to measure standard outcomes? Do we know that these are reliable variables across reviewers? Do we test inter-rater reliability to assess the assessment instrument if it at all exists? I suspect that the rigor I am discussing did exist 10 years ago when I sat down for exams, and I suspect that it still does not really exist.
Look at how teacher education and evaluation is performed (among other disciplines). Look at Nursing. The issue is not so much the exam, but what it measures and how we know it. These disciplines use exams and field work, but all are tied to specific assessment variables. This is crucial and lives depend on it so they take these measures seriously. Why does it seem that ministry preparation is so haphazard in terms of assessment? Are we not dealing with the most intimate layers of human living as well?
My thinking is that all candidates should have a portfolio that is assessed and scored with an effective rubric or rubrics. This would balance exams with field work, and even variables that can be measured in an assessment that the Presbytery of care should administer about the candidate based on standard variables. This ought to include a sample of preaching and it can all be done electronically and exported to a CD for search committees to review so the assessment of the candidate is tied to their PIF. I think that the PCUSA should function as an accrediting agency over the Presbyteries to ensure they are meeting support and training standards for the candidates who are under care just like NCATE does with teachers. It also means that robust assessment should be coupled with training sessions and search committees to find what they most need and desire from a pastor. Yes it's a lot of administration and work for people to do. It takes expertise in assessment and good strategic planning, but don't our young ministers and struggling congregations deserve something more explicit and clear?
I simply do not know how assessment of preparation programs is implemented or how changes are implemented as a result. It just seems to me that such rigor does not exist. I have been through the process and looking over my materials now that I have had experience with educational and institutional assessment it just seems to be disconnected and thrown together far more than I see with other professional disciplines that require standards of their candidates. What I produced was a series of disconnected documents that are not tied to any measure of assessment whatsoever. The package does not give a coherent picture of the candidate, and it is not clear what this person's level of proficiency is in any given area because those objective proficiencies are no where mentioned and so, they are no where observed and assessed.
(BTW: A good assessment can also account for variations because you will be dealing with different persons who exercise different gifts. But it can do that while making sure that all candidates are meeting very specific and clear assessment variables. So while we think about standards, let's not block the discussion with the concern over "freedom" at the get-go. I know that it comes up immediately. I have been there many times before and am discussing that with different kinds of institutional assessment even at the moment.)
The point is that we are assuming that people will be on the same page with this process and there is no evidence that I know of, perhaps there are exceptions as with any rule, that there is any consistency with the process. Just as if I flip through dozens of Congregational Information Forms (CIF's), they all start to sound the same, imagine congregations looking at Pastoral Information Forms (PIF's) that all start to sound the same. Sameness and actually producing evidence of clear competencies, it seems, has been detrimentally confused. SO if we the PCUSA want to actually get serious about assessment rigor, sign me up to help. As long as we are OK if we need to scrap the entire process if need be, we can do great things for our young ministers by tying assessment to training.
Related posts:
I agree that the ordination exams do not do an adequate job of assessing the ability of an individual to serve as a pastor. Then again, I don't really feel that the seminary education that I received at Princeton adequately trained me to become a pastor. The majority of the work of a pastor is with people, working with living human documents, not with scripture or texts or preparing sermons. Knowing how not to get stuck in dysfunctional relationships with parishioners, how to evaluate what spiritual need parishioners have and how to set realistic goals for oneself and healthy boundaries are essential to surviving in ministry. None of these things was taught in my seminary curriculum.
You mentioned nursing in your post. Many health care workers have on the job training that must be done and is carefully supervised. Physicians have to complete residencies. This program is available to clergy but it is not always mandatory. I feel that I needs to be. I am speaking of Clinical Pastoral Education. CPE has gotten a bad rap and some of the reputation is deserved, though I really feel that learning how to do good pastoral care and critical on the ground, as- you- go theological reflection is essential to becoming a good pastor. Further, the self knowledge and self awareness that can be gained if one is open to the process is invaluable to self-care and healthy boundary making.
Having had considerable experience developing assessments, both as a HS teacher and in my doctoral research in education, I have to wonder … why doesn't the PCUSA engage some expertise in developing these exams? Who develops them anyway? If the writers are seminary professors, I can't imagine they have much formal training in assessment development, and if they're simply relying on their experience developing exams for their classes, that experience isn't necessarily applicable to this sort of exam.
I'm sure there's some assessment specialist out there in the educational research community who would think this is a cool project, particularly if the PCUSA wanted to put some money up to do it.
Unfortunately, like pedagogy, most people feel they're experts in writing tests just because they've sat in classes for most of their lives.
Anyway, it isn't like this is a new problem that education experts have never thought about. The literature on portfolio creation and assessment alone is extensive.
BTW, having taken teacher education exams for certification, I'd say that, at least based on my experiences in Michigan, they're actually not a good model to emulate. Here they're simply badly written multiple choice exams.
This is a good post — which I stumbled upon from twitter. But I think you sell the presbytery's cooperative committee on examinations short. I'm not on the committee, but I've worked with them in several capacities and they do hire and use folks trained in assessment and test-making.
I've never graded exams, although I served on Committees on Preparation in several presbyteries. I keep hearing from graders that students who fail often don't answer the question that was asked. If that is why people fail, keep the test.
I share your concerns. However, I find myself unable to come up with a better option. In your opinion, what WOULD be a good assessment?
I appreciate this post and the comments. Although I have NO experience with formal preparation for ordination, I agree with some of the other posts that this is a common problem with tests. Assessment is so much more than answering questions at the end of a program or class and the reasons you cited are absolutely part of the stumbling block to changes — it takes a lot of work to create and score quality assessments. If pastors are are for our spirtual wellness what MD's are for our physical and Psych. are for our mental health then there is absolutely a need for change from the system described.
i have no issue with the written exam, but it should be tied to outcomes that are consistent throughout one's process. in other words, let's say we want to measure variables in pastoral care. these can be observed not only in the theological examination, but in rubrics that are also tied to observations in field work. the problem now is that no one has a clue what they are being measured by in the examinations, no one understands what the scores actually mean, and therefore it is not only an artificial prep. process for the exams, it is a test isolated from the rest of one's candidacy work managed by one's COM. it has to be rigorously linked from the top down so that not only does an inquirer/candidate understand what they are doing and why, but how they will be assessed on those specific activities along outcomes that are consistent throughout. that's authentic assessment and that's not what the process it right now. i think that a portfolio system should be used so that these outcomes are linked from the BoO all the way down to examination rubrics – which to my knowledge do not exist.
if you look at my response to matt's question, it's not that i do not favor an examination. i do! what i think does not work is how this examination is tied to the work of the COM and field ed, etc. which are all part of a very robust portfolio that serves to be the material to address a candidate's fit for ordination of W&S. my problem is that the measures are not clear or non-existent and none of this is tired together in a way to make the assessment authentic.
but what is good are the portfolios in teacher education that are tied to outcomes that organizations like NCATE specify. as for the assessment part, i will seriously offer help if folks want it. it's an antiquated system that needs to get recharged with the backing of all of the educational literature i have been swimming in for the past 10 years.
that's why i am suggesting a portfolio assessment tied to an exam. i think exams are solid ways of giving a baseline of knowledge. but these are the same questions that have been recycled for what, 30…40 years? we have to break it apart and start from the ground up. what are the standards we want for our candidates? why are these good standards? how will we measure them? etc…
I'm not Presbyterian, so forgive my ignorance of the process. I have had similar frustrations with my own denomination's ordination process – lack of standards of assessment (for us it is all pass/fail) and wide variances based on who makes up your committee and who asks the questions.
I must say though, I find many of the above "solutions" (including in the original post) to sound more like a secular professional world than the Church. My own opinion is that trusted pastor/priests should be chosen to mentor potential candidates, and that much should weigh upon those recommendations. It is too easy to get caught up on the theological/political agenda of a particular committee member, whether they be judging your oral or written interviews.
Perhaps this is where Bishops come in (forgive me, descendants of Calvin). They are traditionally the holders of apostolic authority and conferees of ordination. It seems to be that the office of bishop deserves more than a tacit place in this process of call/selection/ordination. But please, give it some gospel content, not just another professionally assessed inventory of skills.
thanks for responding.
my alternative, or correction as it were, is from a very admittedly secular basis. but only because the secular professions have made great strides in assessing standards and outcomes while in our processes in the church we just assume things to be reliable. and that's what i am after – reliability across assessments and outcomes.
for my own theological view calling to ministry is from god and the people simply witness to that and confirm it within mutable social structures. it means that people can fail the calling of god, and it also means that those who feel like they are called may in fact not be. that is for the presbyterian committee on ministry to decide for the candidates under their care. but there is much more to this piece of the discerning process. and that the ordination exams can make or break your entire investment in a graduate education to legitimate your calling as well as other pieces of the puzzle is a problem if they are not all intimately connected.
Thank you for this post, Drew. As one who is at the beginning of the ordination process I am just beginning the process of "figuring things out." At this point I lack the backround info to put together some of what you are saying. If I can say it as I understand it I would be happy for your feedback. Whatever the deficiencies of the system currently in place, it would be good to take them into account this early on so that I might avoid those pitfalls. You mention a "robust portfolio" that somehow connects CPE/field ed/preaching/teaching/hand on training to the ordination exams. Not having any idea how this works in other professional communities, how would one go about making these connections within the current system? As in, how can I fill in the blanks for myself? Who can I ask to help me connect the dots? From what you have written it seems to me that you are mostly addressing the connections the CPM is making in relation to what a candidate has done to how a candidate has done on the ords. Is there a way to come at this from my angle since any changes that happen will be after I am (hopefully) ordained? If my questions don't make sense no worries….like I said, I am still trying to "figure it out." : )
I'd like to know more about the system that devises the exams and what is told to candidates taking it. I took exams about 20 years ago, and they have changed since then.
In most presbyteries the Committee on Preparation for Ministry (CPM) supervises, guides, and grades candidates, not the Committee on Ministry (COM). The CPM provides graders for the exams, and when they are functioning well, they counsel and prepare their candidates to succeed. Sessions have an even bigger role, but often abandon their members when they leave for seminary.
I agree, Mack, at its best a presbytery meeting as a governing body and working through its committees does the work of a bishop. But a committee may not have the wisdom of one experienced bishop, however it can overcome one vindictive or petty bishop.
good questions. i am not sure i have any answers for ya. other folks might field that a bit better?
[...] why the pcusa ordination exam is not a good assessment – notes-from-offcenter.com/2009/08/28/why-the-pcusa-ordination-exam-is-not-a-good-assessment – view page – cached #RSS 2.0 RSS .92 Atom 0.3 » why the pcusa ordination exam is not a good assessment Comments Feed index Have Your Cake and Eat it Too? when saying "deviation against God's will" is not theological — From the page [...]
@pinkhammer:
Find someone who has recently finished the ordination process in your presbytery–they can fill you in on how your CPM "does things."
Things you can do to make the process easier: do your psych eval as soon as possible. Also, request to take ordination exams the summer after your first year or in the winter of your second year. There's not much harm in failing the exams the first time, but you'll be at a significant disadvantage if you wait until your last year and have to repeat exams then.
If you get any exceptions approved to our internships or class substitutions or anything else, get the approval in writing. Since 1/3 of the committee changes every year, your paperwork becomes the institutional memory of the committee.
Have several plans for what to do after graduation. There are a lot of things along the way that can delay your approval for circulating a PIF: psych eval, ordination exams, class schedule mixups, internship problems, committee timing. If you have other ways of supporting yourself, you'll take the pressure off and save yourself a lot of stress.
I don't mind the fact that there is an exam, but I do mind that few people can articulate a good reason for these particular exams and for how they are administered.
Probably my biggest complaint with ordination exams is that they function as a way of hazing candidates, while making those who object to the exams appear to be weak or troublesome. The rationale for cramming 4 exams into 2 days is given in the exam handbook:
"The examinations are intentionally given in two days time as a way of evaluating the candidate's stewardship of time and energy, an important aspect of life in ministry."
One important way of demonstrating stewardship of one's time and energy is to object to requirements that are abusive of one's time and energy.
Currently the exams are given twice per year in August and again in January. Results are released early November and early March. This means that if a candidate fails one exam in August, their search for a call can be delayed by six months to a year. In the event that a candidate fails the exams more than once, there are alternative exams available, but they have to be approved by the presbytery, adding a public notice of failure to everyone who might help them network.
In addition, not knowing who you are writing for nor what criteria they are using to evaluate makes it very difficult to decide what to write (or even what material to study). The readers' comments are sometimes helpful for figuring out how to take the exam better the next time. But almost 3 months pass between taking the exams and seeing the comments–after a while, the usefulness of that feedback fades. I know it takes a great deal of effort to gather the readers together, but why such a time lag?
Evaluation of a candidate's stewardship of time, ability to connect classroom theory with practical ministry, and "pastoral sensibility" in times of crisis are best measured in internships and CPE. The judgment of whether a candidate is Presbyterian enough in their theology falls to the CPM, COM and examination on the presbytery floor. And knowledge of specific content is covered by 3 years of exams and papers in a seminary.
What is the exam testing that isn't already tested by other elements of the process?
How about this? What if one pastor from every presbytery took the exams anonymously next time? It would be interesting to see what the passing rate is for experienced pastors and to find out from them what it was like after being in ministry for five years or more.
i agree totally. it is the process i have issues with. the assertion that the time is some abstract test of stewardship is the most absurd piece of straw i have heard. really. and yet we have no clue what readers are measuring when the give you their arbitrary scores. i really just want to know if readers use a rubric to assign a score.
Thanks for the comments, Katie. Good things to think about.
Agree with pretty much everything shared here, but want to add a few further pieces…
What is challenging to me about this process is that it is often not even used (even with all its MANY faults) as a basis for whether to ordain or not. While I agree with what has been shared about the problems with the current process (more on that later), it is the system we have now. What I have observed, however, both as a CPM Moderator and also as one who went through the process is that the exams are often seen as "how to help people pass" rather than using them as a basis of whether a person should continue or not. They are one of the standards for ordination currently, but I had many discussions with people about how to do "alternate" exams for people who were unable to pass the exams on their first several attempts. Now, in several of those circumstances, it was the result of language issues or of learning disabilities (again another problem with using an exam – but again more on that later). However, in several others, CPMs tried to bend the process to get an alternate exam approved so that a person could pass. Discussions about whether these should be simply used as a yes or no did not come up. Again, I do not like the usage of these exams as a basis for yes/no in ordination, but they are officially set up as that and yet many pbys don't use them in that way. With other professions and their exams, they are used as an objective standard of whether one can go forward. You don't see people getting a pass out of a bar exam just so that we can have one more lawyer, etc.
So, about the process…We have lost something by not having apprenticeships or mentoring as a part of this process. Yes, there are officially liaisons with both the CPM and the Session for each person, but in reality these individuals function more as communication links rather than as mentors. In a perfect world, I would love the process to be set up such that the exams (if there were ones) were more like the MCATs where one took them to go into seminary or as a guide for what areas one might need to focus on within their seminary training and their mentoring relationship, rather than the last hoop to go through at the end of the process. But mentoring (whether it is through an official program like CPE or something else) is a vital part because the mentor can speak to the specific situations of ministry that took place.
If I didn't have my CPE experience specifically, I am not sure what kind of pastor I would be today. It was through the mentoring that took place and the specific situations I worked with that radically transformed my understanding of what it meant to be a vessel for Jesus and to be a pastor to God's people.
mentoring and apprenticeships were the core of minister training before we began to confer special professional degrees as legitimation for the ministry. people usually think that college was the place to train ministers, but that's not accurate. colleges trained people in a classical curriculum and training in ministry came in the kind of mentoring situation you are talking about.
that's why i think a portfolio assessment is the way to go. either make the ords like MCATs or NCLEX, etc. or fold them in as a weighted percentage of a candidate's fit for the ministry to which they are pursuing a call. right now it tries to do both and fails on both accounts. first we need to determine what the hell they are for and what they are supposed to do and how we know if they are actually doing it to make any headway.
Drew, I think you are raising some interesting questions. Have you read the self-study by the PCCEC? http://www.pc-biz.org/Explorer.aspx?id=1887&pro...
Your suggestion "that the PCUSA should function as an accrediting agency over the Presbyteries to ensure they are meeting support and training standards for the candidates who are under care just like NCATE does with teachers" strikes me as such a radical shift in our polity, that it would require an amendment to the Book Of Order that would state in an explicitly measurable way what the presbyteries need to do.
When reading that report, scroll to the bottom and look at the difference between how readers of the exam, CPM moderators and recent examinees perceive the exam. A lot of weight is given to the readers' perception. Table VI tells us that less than 1/2 of the examinees "Think That Each Examination Assesses the Integration of Academic Learning and Pastoral Skills to a “Great” or “Very Great Extent." (except for the Worship exam at 52%).
Table 8 "Proportion of Respondents Judging that Successful Performance on the Four Ordination Examinations Reflected Readiness for Ministry to a “Great” or “Very Great Extent”: Readers: 63%, Recent ordinands: 22%.
This disparity leads to cynicism, and a lack of trust in the examination system Seminary education does not come cheaply in terms of money, time, or spiritual cost. By the time a candidate is in their 3rd year of seminary, he or she should have a reasonable expectation of being able to seek a call after graduation.
Regardless of the gate keeping qualities of the exams, if they are not pastoral and loving toward the most vulnerable participants, then they need to be re-evaluated.
Here's a link to the 2008 multiple choice Bible Content Exam. http://www.whitneyhq.com/biblecontent/tests/Tes...
I question whether this is what we mean when we say that we want our pastors to have Biblical literacy.
i agree with katie here. pretty clear that the confidence intervals are very low.
also, all of the data gathered was based on perceptions of examiners and examniees. NOTHING was done here to test the actual fitness of the instrument used to assess examinees on the variables as expressed in the BoO. that's my main beef. if you have an invalid instrument to assess someone, it does not matter how people feel about it.
also, the pcusa really does act as an accrediting agency since the BoO prescribes the process for the COM's and CPM's to use for the ordination process. my problem is that it does not do this well because the process lacks integration and validity. IOW it's not an authentic assessment.
Drew, I think you are raising some interesting questions. Have you read the self-study by the PCCEC? http://www.pc-biz.org/Explorer.aspx?id=1887&pro...
Your suggestion "that the PCUSA should function as an accrediting agency over the Presbyteries to ensure they are meeting support and training standards for the candidates who are under care just like NCATE does with teachers" strikes me as such a radical shift in our polity, that it would require an amendment to the Book Of Order that would state in an explicitly measurable way what the presbyteries need to do.
When reading that report, scroll to the bottom and look at the difference between how readers of the exam, CPM moderators and recent examinees perceive the exam. A lot of weight is given to the readers' perception. Table VI tells us that less than 1/2 of the examinees "Think That Each Examination Assesses the Integration of Academic Learning and Pastoral Skills to a “Great” or “Very Great Extent." (except for the Worship exam at 52%).
Table 8 "Proportion of Respondents Judging that Successful Performance on the Four Ordination Examinations Reflected Readiness for Ministry to a “Great” or “Very Great Extent”: Readers: 63%, Recent ordinands: 22%.
This disparity leads to cynicism, and a lack of trust in the examination system Seminary education does not come cheaply in terms of money, time, or spiritual cost. By the time a candidate is in the 3rd year of seminary, he or she should have a reasonable expectation of being able to seek a call after graduation.
Regardless of the gate keeping qualities of the exams, if they are not pastoral and loving toward the most vulnerable participants, then they need to be re-evaluated.
Here's a link to the 2008 multiple choice Bible Content Exam. http://www.whitneyhq.com/biblecontent/tests/Tes...
I question whether this is what we mean when we say that we want our pastors to have Biblical literacy.
i agree with katie here. pretty clear that the confidence intervals are very low.
also, all of the data gathered was based on perceptions of examiners and examniees. NOTHING was done here to test the actual fitness of the instrument used to assess examinees on the variables as expressed in the BoO. that's my main beef. if you have an invalid instrument to assess someone, it does not matter how people feel about it.
also, the pcusa really does act as an accrediting agency since the BoO prescribes the process for the COM's and CPM's to use for the ordination process. my problem is that it does not do this well because the process lacks integration and validity. IOW it's not an authentic assessment.
Hey Sparky! How about sending it to EVERY SINGLE PRESYBTERY'S CPM! That might actually do some good instead of just making your feel better. I might agree with your statements – but the reality is that I have to take them next month. That's the reality.
Since you are already "in" – do something to help those of us who are still "out."