Issue: Using Portfolio Assessment to Improve K-12 Schools and Meet the Accountability Requirements of the No Child Left Behind Legislation

Portfolio assessments are gaining favor as a learning tool. Is it possible that this passionately supported pedagogical process can also meet the accountability needs of policymakers? In an online panel discussion, four experts find portfolios to be an extraordinarily useful method for assessing student learning and academic programs. Even so, they agree the approach does not provide the reliability or validity legislated for high-stakes assessment. They also expect that attempts to bend the portfolio process into high-stakes compliance could remove what panelists felt was its very heart: student selection of work.

This Soapbox forum further explores the possibility of using portfolio assessment to demonstrate AYP, as defined by No Child Left Behind (NCLB). Each of the four panelists brings an important perspective to the conversation. Knowledge of Vermont's experiences with portfolio assessment is brought to the discussion by Douglas E. Harris, executive director of The Vermont Institutes and director of the Center for Curriculum Renewal. Two forum participants bring higher education success stories to the discussion: Gail Ring, Ph.D., director of Instructional Technology at the University of Florida College of Education, and Paul Ross, manager of the Faculty Technology Resource Center at the University of Denver Center for Teaching and Learning. Finally, Peggi Munkittrick, SunGard SCT's senior director of teaching and learning strategy, comes to the discussion with an industry perspective. SunGard SCT provides various technologies for schools and partners with Nuventive to market a portfolio product called iWebfolio.

An e-mail-based discussion sponsored by IAETE

The demands for technical quality in areas such as cognitive complexity, balance of representation, and standards alignment preclude the kind of student choice that is at the heart of good portfolio assessment. While we may (or may not) be able to meet these demands by standardizing prompts, performance tasks, and work sampling protocols, in my opinion, we would have slain the portfolio in the process.

- Doug Harris

Portfolio assessments are gaining favor as a learning tool. Is it possible that this passionately supported pedagogical process can also meet the accountability needs of policymakers? In pursuing the topic, a panel of four experts, including Douglas E. Harris, Gail Ring, Ph.D., Paul Ross, and Peggi Munkittrick, finds portfolios to be an extraordinarily useful method for assessing student learning and academic programs. Even so, they agree the approach does not provide the reliability or validity legislated for high-stakes assessment. They also expect that attempts to bend the portfolio process into high-stakes compliance could remove what panelists felt was its very heart: student selection of work.

Why the Question?
The 2002 IAETE conference (titled Assessments That Empower Student Success, The Role of Technology) explored new opportunities for technology to support assessment systems. At the conference, John Bailey, who was, at the time, director of the Office of Educational Technology at the U.S. Department of Education, mentioned Vermont's work with portfolio assessment:

States have the opportunity to set multiple indicators to help gauge whether a student in a school is making AYP [adequate yearly progress] each year. We thought some states might move toward the portfolio assessments, Vermont in particular. We didn't have any states take advantage of alternative assessment tools, partly because of some of the problems with reliability and validity. I think those are some areas that need some further thought and further research. Models that are working need to be presented to public policymakers.

This Soapbox forum further explores the possibility of using portfolio assessment to demonstrate AYP, as defined by No Child Left Behind (NCLB). Each of the four panelists brings an important perspective to the conversation. Knowledge of Vermont's experiences with portfolio assessment is brought to the discussion by Douglas E. Harris, executive director of The Vermont Institutes and director of the Center for Curriculum Renewal. Two forum participants bring higher education success stories to the discussion: Gail Ring, Ph.D., director of Instructional Technology at the University of Florida College of Education, and Paul Ross, manager of the Faculty Technology Resource Center at the University of Denver Center for Teaching and Learning. Finally, Peggi Munkittrick, SCT's senior director of teaching and learning strategy, comes to the discussion with an industry perspective. SCT provides various technologies for schools and partners with Nuventive to market a portfolio product called iWebfolio.

Teaching to the Test: The Need for New Assessments
An underlying theme expressed by IAETE conference participants and the panelists is an acknowledgement of the shortcoming of current state assessment systems--both as an effective part of the learning process and as an accurate profile of student abilities. Ring writes,

Portfolios encourage students to actively engage in the learning process, whereas these high-stakes tests encourage a more passive mode of learning. Teachers are forced to facilitate the archaic model of disseminating knowledge. In Florida, during the quarter when the FCAT is administered, all teaching pertains to the FCAT. Everything is suspended but the FCAT. Some argue that this model is reducing schools to little more than test preparation centers.

Munkittrick, in comparing the qualitative elements of portfolio assessment to the quantitative method that now prevails, writes,

Test scores, on the other hand, are a quantitative approach to assessment. They tend to be summative in nature with a focus on knowledge acquisition rather than knowledge construction. In our view, standardized test results give an incomplete picture of a student's achievements, and of a school's performance.

The Value of Portfolios
Having used portfolios in teacher education courses, Ring is convinced of multiple benefits. She writes,

We found that not only did our students learn more, but they were able to articulate more clearly (in a variety of formats: electronic media, face-to-face presentations, and written reflections) their proficiency in a given FAP (Florida Accomplished Practice). We also learned about weaknesses in our program. For example, we learned that while some courses stated that students learned about FAPs 1-12, students weren't learning about any of the FAPs in any great depth; consequently, the instructors stepped back and reevaluated their courses and adjusted assignments accordingly.

Harris's endorsement also sums up the many advantages of the assessment system:

Many of us are convinced that the power of portfolios as sources of metacognition, teacher/student dialogue, and deep understanding of students' approaches to unifying concepts and themes is too important to give up. And, the portfolio provides important school-level (program diagnostic) and student-level (individual diagnostic) information.

Technology's Role for Portfolio Assessment
Schools implement portfolios with paper, Web pages, portfolio software applications, and custom programming. Scaling the approach to a statewide accountability system, however, would require far more robust and affordable technology for search, storage, and scoring.

The state of the technology underscores that the portfolio is more potential than practice. Discussing the need for affordable, open-source technology to manage portfolios, Ring writes, "There are portfolio communities at the higher education level that are having these types of discussions (the AAHE and EDUCAUSE, to name two), but to my knowledge, similar discussions are not taking place at the K-12 levels. (View the AAHE Web site.) A quick history from Ross confirms her observation:

Up until 2002, most portfolio applications consisted of templates that the participant then modified to create a Web site. Participants needed a moderate to high level of technical skill to modify/customize the templates, as well as skills in layout and graphic design. Attempts like this were nothing more than customized personal Web sites and provided no way to aggregate data or conduct assessment. There are a number of portfolio systems in use across higher education that are establishing mechanisms to support assessment. Some of these are quite scalable for the K-12 environment.

Through IMS Global, discussions have commenced on interoperability of portfolio data, which would assist in the goal of creating systems that then have the ability to communicate with each other. There is potential to form groups around shared interests, assessment data, rubrics, and objects. A white paper was recently published with collaborators from a variety of institutions (https://portfolio.du.edu/pc/port.detail?id=14112). Organizations like the AAHE and EDUCAUSE are facilitating studies focusing on the effectiveness of portfolios in various situations. In February, the AAHE and the University of Denver will launch a revised Portfolio Clearinghouse that will facilitate the collection/dissemination of information on projects from across the country. (View the AAHE and EDUCAUSE Web sites.)

To make the leap from higher education to K-12, Ross suggests a pilot program, noting that "there has to be willingness at the state and federal levels to fund pilots that enable the testing and flow of data across different institutions tracking the progress of individuals/groups." He recommends forums to show teachers (please click for his teacher conference recommendation) and policymakers these new technologies.

Public policymakers need to experience and review current portfolio applications. In the past six months (with the move to establish enterprise systems for coordinating assessment activities across departments in institutions and to facilitate the sharing of evaluation rubrics), the landscape has changed dramatically. The University of Denver Portfolio Community recently opened up the rubric library for public access after requests from other institutions. There is a need to actively share assessment rubrics and assessment models across the community.

Scaling with Reliability and Validity
Scaling the portfolio's personal assessment approach to the magnitude required by NCLB would, to say the least, raise some objections from psychometricians. Though all panelists deeply value portfolios in the learning process, they also recognize that this assessment system does not now offer the reliability and validity required by NCLB.

Harris perhaps best states the problems facing states that want to use portfolios for federal accountability requirements when he writes, "The requirements for equitable opportunity to learn and meaningful disaggregation in NCLB are daunting. While, in my opinion, these are the most laudable parts of the law, they place demands on portfolio assessment greater than the robustness of our scoring protocols can bear."

Vermont, however, is making progress in terms of reliability and validity. Harris writes,

In our world of sound bites, the Vermont portfolio process picked up its moniker from the Rand study a decade ago: "Great instructional impact, low reliability." The good news is that, in the ensuing years, we have developed calibration protocols and scoring processes that have greatly increased the reliability of portfolio scoring, certainly within the range of acceptability for open-ended scoring on high-stakes assessments.

Harris offers a detailed discussion of calibration issues which is well worth clicking to. As his report of Vermont's experiences continue, he writes,

In terms of validity, I will focus on content, construct, and decision validity. As to content validity, it is quite feasible to establish this through research and expert review. This is not a challenge, except to be vigilant, to ensure that it gets done.

We have pretty well established the construct validity of our rubrics in writing and mathematics, though there is a lingering question about middle-school-level mathematics communication. The major construct validity question we face is related to the intersection of topics and categories of cognitive demand. For example, I know that our math portfolio can provide valid measures of mathematical communication and problem solving at the grade 4 level. But, for example, is this equally true when the student selects a piece related to geometry or functions and algebra? I suspect it is but can't prove it. But, can I also make inferences about the student's understanding of the geometry content based solely on the portfolio? And even if I can do that, can I generalize to an entire mathematical construct such as proportionality?

Scalability: Teaching Issues
Up reliability and validity, say the panelists, and one compromises the pedagogical value of the portfolio process. Writes Harris,

Computerized scoring of common prompts may be quite feasible, but computer use in a system involving student choice among many pieces may be beyond us at this time. This poses the same question I suggested yesterday: if you focus the item and eliminate student choice, are you doing portfolios anymore?

Though there are portfolio systems that do not rely heavily on student selection of materials, the panelists consider this trait central to their portfolio experiences. Ring sees this as a clear conflict with NCLB accountability definitions:

The key to reliability is setting clear criteria for the portfolio, and, as I think we have already agreed, an overly directive portfolio becomes institution/district/testing agency centered and no longer student centered, and consequently no longer a student's portfolio.

Training and support also emerge as issues for Vermont when considering the large-scale use of portfolios as annual summative assessments. Writes Harris:

I know that portfolio assessment can simultaneously improve learning and be part of the process of defining school performance . . . . However, if one adds the qualifier "as defined by current federal law" to the above question, the question is murkier. First, the law would require use of portfolios for high-stakes purposes at all grades, 3-8 at minimum . . . . It is far beyond our capacity to support all teachers to the degree necessary to meet minimal reliability and construct validity requirements across the state.

A Quick Answer: The Portfolio within a Portfolio
Unwilling to give up on portfolios, panelists see wisdom in simply having the portfolio as a part of the mix--as one of the "multiple indicators" referenced by Bailey in the comment that initiated this conversation. Of Vermont's experience, Harris writes:

We continually work to help teachers and school leaders understand that the portfolio is valid but not sufficient, and that multiple measures are needed, but I'm not sure that message always comes through.

As to decision validity, are portfolios valid for making high-stakes decisions? Certainly not alone, but in an appropriate mix of assessment strategies, almost certainly. This has been successfully done in K-12 at places like the MET Schools; in higher ed, as other colleagues describe and in places like Alverno College; and in business and the arts worldwide. The problem arises when it is perceived as necessary to reduce decisions to a single cut score or threshold performance. In that case, the portfolio is perhaps best left out of the mix. (View the MET Schools and Alverno College Web sites.)

In further support of a mix, Harris also notes that "students who do very well on traditional measures like multiple choice tests can find it more challenging to develop strong portfolios."

A collection of assessments, however, does not erase the tension with large-scale accountability requirements. Harris writes, "Thus far, we have been able to justify [portfolio assessment] in terms of cost and effort. Whether we will continue that success as resources are usurped by NCLB remains to be seen."

Meanwhile, Harris suggests he may have found another promotional route. "We also are exploring ways for our small schools (of which there are many in Vermont) to use portfolios as part of their school-based evidence, absent sufficient numbers of students per grade," he writes. "This may ultimately be the camel's nose under the tent of narrowly defined AYP."