Issue: The Online Assessment of Technology Literacy

• Martin Ripley heads the eStrategy Unit of the United Kingdom's Quality and Curriculum Authority (QCA). In that capacity, he has led groundbreaking work on assessing technology skills and higher order thinking via simulations. His work differs from other such work because it looks at large patterns of activity rather than solely at a final answer. See www.qca.org.uk and www.ks3ictpilot.com.


• Greg Pearson, program officer with the National Academy of Engineering in Washington, DC, serves as the responsible staff officer for Assessing Technological Literacy in the United States, a study funded by the National Science Foundation (NSF), and the State Educators' Symposium on Technological Literacy project, which is funded by the U.S. Department of Education. He is the lead author of Technically Speaking, Why All American Need to Know More About Technology (National Academy Press, 2002) and a related report on assessment to be published in 2005. See www.nae.edu/techlit.


• Margaret Honey is vice president of the Education Development Center (EDC) and director of EDC's Center for Children and Technology (www.edc.org/CCT). In the latter capacity, she is also the coauthor of Assessment of 21^st Century Skills: The Current Landscape, a report of assessments published by the Partnership for 21st Century Skills (www.21stcenturyskills.org).


• Kate Kemker, director of instructional technology for the Florida Department of Education, has worked with a state team that created a simulation-based assessment of technology skills for teachers known as the Inventory of Teacher Technology Skills. She now works on a similar inventory for eighth-grade students to be used in the spring of 2006. See www.flstar.org.

Defining Technology
Technically Speaking, writes Pearson, "spends many pages defining and defending an expansive notion of technology and technological literacy." Elsewhere, he summarizes technology literacy "as a process of modifying the natural world to satisfy human needs and wants, and of course the products that result from that process." For the National Academies, technology can be a computer, or it can be a shovel, a pencil, paper, or the process of identifying chromosomes on a strand of DNA. Writes Pearson, "As this panel illustrates in microcosm, when the word technology is mentioned in the context of U.S. education, it is associated almost exclusively with computers and IT (information technology) or ICT literacy. This is not wrong . . . but it is a rather narrow view of things." From his work on assessment, Pearson notes that an emerging priority is a familiarity with the "technological design process, which has many parallels with science inquiry."

Pearson also points out that, beyond this panel, the Academies' perspective has supporters.

In the interest of complete honesty, the Academies are far from the first to describe and promote this larger view of technology. Philosophers of technology such as Langdon Winner, historians of technology like Thomas Hughes, and technology critics like Neil Postman have all taken this larger view. So has the STS (science, technology, and society) movement in U.S. postsecondary education, and the International Technology Education Association, which represents technology education teachers in this country.

In contrast to Greg, I'm choosing to highlight a set of fairly specific skills and competencies that students need. I don't by any means disagree with GregI like the breadth of his argument and ideas. I'm motivated to write more specifically only because I see too much of what is NOT being taught in schools. In too many schools we are stuck with a very 1980s notion of technology competencya kind of basic facts approach to technologythat's not serving our kids well.

The discussion from my perspective is, do you measure all of these skills at once with a performance-based solution within scenario-based settings? Or do you begin with basic computer skills, then communication, basically in a hierarchal method? The situation that we face in K-12 education is that if we begin with one area then precedence has been established in which those skills tend to have greater value than the other skills.

So, can we usefully ponder what subject will replace IT, and why? My view is that the subject must change its outlook from training students predominantly in the skills and capabilities that arise from the existence of personal computers. Instead we should look for the subject IT to concern itself increasingly with the growing range of technologies (mobile devices, blogs, video). It should concern itself with the uses to which technology is put (work, leisure, recreation, purchase). And it should concern itself with the facilities (or capabilities) that those technologies provide to students and adults (voice, visual communication, decision making, choice, responsibility).

• have hands-on skills, such as using a computer for word processing and surfing the Internet and be able to operate a variety of home and office appliances,
• be able to identify and fix simple mechanical or technological problems at home or work, and
• be able to apply basic mathematical concepts related to probability, scale, and estimation to make informed judgments about technological risks and benefits."

Honey observes that students today have almost universal access to unfiltered and unsubstantiated information. No longer do they receive their information solely from teachers or librarians, or through textbooks or print-based reference materials. She continues,

To navigate through this wealth (or glut) of data, draw conclusions, and communicate with others proficiently, teachers and students require whole new sets of skills surrounding accessing, interpreting, analyzing and evaluating complex sets of images, words, numbers and sounds in meaningful ways.

She identified some of these skills as communicating effectively, analyzing and interpreting data, understanding models and simulations (computational literacy), managing and prioritizing tasks, engaging in problem solving, and ensuring security and safety. Her work with the Partnership for 21^st Century Skills identifies ICT learning skills as related to thinking and problem-solving skills, information and communication skills, and interpersonal and self-direction skills. Honey and the Partnership emphasize that these skills are a priority and can be addressed through core subjects as well as in areas they refer to as 21^st Century Content: global awareness; financial, economic, and business literacy; and civic literacy. ICT learning skills are interwoven with technology because their pursuit is so often assisted by high-tech tools.

The Promise of Simulation-Based Assessment
An assessment environment that simulates common software applications is an ideal and obvious way to investigate a student's abilities with computer applications. Assessments that utilize only one brand or model of software would be severely limited, as there is no standard for any common software applications, such as word processors, spreadsheets, or databases. A simulation also allows greater leverage for the capturing and reporting of multiple points of data, functions that would be complex or expensive to retrofit to existing software. After developing such projects, Kemker and Ripley are anxious to apply the environment to other school subjects. Kemker sees a fit with music. Ripley wants students to design Shakespearian sets. Both also see potential for science. Beyond other disciplines, however, simulations may hold promise for assessing complex cognition.

Ripley's assessments take place in the virtual world of Pepford, where work assignments arrive via e-mail and a "walled garden" provides a virtual world of Web sites and applications. Writes Ripley,

The use of simulations is a key development. A simulator provides the context within which authentic assessment tasks can be designed and delivered to students. It also facilitates the development of assessment tasks that invite students to combine a range of capabilities and skills. The combination of these two aspects enables us to assess higher order IT capabilities, such as choice or communication.

The test records and scores the actions that the student takes while completing the test. For example, a higher order capability for a 14-year-old student in England includes designing a system for someone else to use. That capability in turn includes an assessment of the end-user's requirements. This we assess dynamically in the virtual world of Pepford by collecting evidence of the student researching into those requirements, by sending and receiving emails to ask about requirements, by the student refining the system to meet requirements and so on. To achieve this we have worked with an extensive range of teachers to document the processes that students go through when producing eloquent or satisfactory responses to the task set. We use this [data from teachers] to create a matrix of plausible routes that a student will take when en route to complete a satisfactory (or better) response.

In every e-test development project I have worked on it is possible to get all of the senior staff (designers, technologists, psychometricians, trainers, teachers) together regularly. These core teams number around 15-20 individuals and can be led effectively as a single team. There is no parallel in my experience with paper-based tests, which are often developed in a linear process, with different teams (involving large numbers of people) responsible for the various stages of development, administration and marking.

The development of online assessments requires a very diverse team of individuals. It is imperative that each member of the team be an expert in specific fields, such as measurement, technology, and curriculum. Last year, as we developed the online assessment tool for teacher technology skills, the team consisted of these individuals in addition to teachers.

However, the key ingredient was to have a leader that focused on the scope of the project, or basically the project manager. This individual understood every aspect of the process, so that the focus remained the development of a performance based tool . . . .

It is critical to have the team leader be a measurement expert so that the team does not stray from the original scope of the project. In addition the team needs to be forward thinking so that the tool developed will have longevity.

Assessments which use simulations are a radical development. Delivering high-stakes assessments to all schools onscreen is a departure from current practice. This type of test development project involves systemic, nationwide innovation and change. The nature and extent of innovation in this type of redesign challenges psychometric expertise and wisdom; it requires teachers to adapt their teaching, and it requires pupils to reconsider ideas of what tests involve; it requires a national school hardware infrastructure that is as robust as some international banking systems; it requires network managers and technical staff in schools who can support the system. So, this is what I mean by 'systemic innovation.'

Another way of describing such systemic innovation is to say that it requires a clear acknowledgement, including from national political and educational sponsors, that in the beginning there are no guarantees of success and there is absolutely no track record of successful delivery. From the beginning, there needs to be flexibility within the proposed approach to reconsider, review and learn lessons. A willingness to admit mistakes and start afresh when needed. And there needs to be enough time in the early development phases and initial roll-out, to prove the conceptual approach and to build the track record.

We started our project to develop an ICT test for 13-year-olds in 2001. We aim to have it rolled-out to all of our secondary schools (high schools) by 2008. I do not believe that we could achieve our aims in less than seven years!

Greg has elsewhere made a good point about the general lack of psychometric research and empirically based evidence to support the most adventurous forms of e-test development. I see that deficiency in almost all aspects of e-test development . . . . In England I like to remind colleagues that paper-based testing has been in use for 140 years. The science and business of paper-based testing is well refined and well evidenced. A person experienced in managing, researching or administering one set of paper-based tests is likely to be able to move in a straightforward way to a similar role in relation to other paper-based tests. In contrast, there is little stability yet in e-testing. The business processes are immature. Even when the e-test consists of closed-response or multi-choice items, the on-screen delivery involves making design decisions on the basis of emergent thinking, not on industry best practice standards (which do not yet exist).

Ripley advises that research focus on "sources of difficulty" (the full range of demands carried by a test, but not part of the targeted assessment domain), the validity and reliability of measurement methods used in simulation-based assessments, and the possibilities of expanding it to domains beyond ICT.

Pearson offered one illustration of the numerous research questions that need to be considered for simulation-based assessment: "Can each action taken by an individual in a simulation or game be treated as a test item and its correctness be judged by an on-demand, real-time assessment of the circumstances in which that action is taken, or do prior actions that led to the context in which the action is taken need to be considered?" In other words, how much of a student's activities do we analyze?

Additionally, the domain needs to be understood. Beyond a definition of technology, Pearson points out, we need to know more about "how people actually learn technological concepts." He explains,

Assessment designers would benefit from having more information about such issues as expert/novice differences in knowledge acquisition, concept formation, misconceptions, and knowledge transfer, just to name some of the most obvious areas. A better understanding of these very fundamental aspects of learning will greatly improve the quality of assessment for technological literacy, as well as the potential to harness simulation effectively.

I would like to ask my colleagues whether they feel students and teachers should, in addition to simply using ICT as a learning tool, also be challenged to think about and discuss ICT in a somewhat more critical way. For example, what are some of the unintended consequences of our use of computers in education? What trade-offs are we making, intentionally or not, by placing ICT in such a prominent position in U.S. education?

The difficulty and challenge in focusing our technology ambitions on assessment is that assessment is such a blunt instrument. National assessments are filled with unintended consequences. They atrophy and become predictable within a matter of a couple of years, with the result that creativity and innovation are lost. And the processes of preparing students for high stakes national assessments are rarely witnesses of exemplary pedagogy.