Prove it: Stifling innovation with the burden of unobtainable proof

Think of something new and innovative that you are trying out in your classroom, school, or district.

Prove to me that it works.

Yep, I want you to stop reading this and think about some fancy new way that you have of educating and/or assessing students and tell me what evidence you have to prove that your new technique works.

Twice recently I’ve been faced with this demand. In the first instance, a teacher who was very excited about using portfolios after hearing my talk at NSTA15 in Chicago contacted me for help in convincing her science department to let her pilot the use of portfolios. She sent me a list of their questions that looked something like this:

1) Have you seen an increase/decrease on standardized test scores?

2) Have you seen an increase/decrease in student motivation?

3) Have you seen an increase/decrease in student competency?

A similar question popped up in the application packet for the PAEMST:

Provide evidence of your teaching effectiveness as measured by student achievement on school, district or state assessments, or other external indicators of student learning or achievement.

Here’s the problem: portfolio-based assessments like those that I employ are meant to be a replacement for standardized test scores. Portfolios are not just some labor-intensive test prep system. That would be like spending months training for a triathlon but instead finding yourself riding a mechanical bull for ten minutes. You could probably ride the bull a little better than if you hadn’t trained, but the bulk of your training would be lost on anyone watching you ride the mechanical bull (badly).

What then do you say to the science department questionnaire about the effectiveness of portfolios? What proof could I possibly provide about external indicators of student learning that could match the depth and quality of the portfolio assessments themselves? ACT data might be the closest thing to useful testing data that I see, but correlating achievement on ACT with pre- and post-portfolio implementation would be fraught with any number of the usual data snarls that we find when trying to compare different test takers from multiple school years.

We are then at an impasse. Those educators like myself that want to use portfolios for assessment will tout all the amazing things that you can observe in portfolios that you could not otherwise. Those who want to keep using standardized tests as the measuring stick for student and educator performances will decry the lack of a link between portfolios and achievement test scores.

I think that pretty soon we are going to have two different systems pop up across the country to accommodate these two assessment camps. One wing will be led by the testing juggernaut that stands to make a lot of money by continuing the current testing regime, but the other will be led by…..Kentucky? New Hampshire? Your guess is as good as mine, but I suspect (hope?) that sooner or later we’ll see some states piloting portfolios (again) as much needed replacements for the broken assessments that we currently use.

In the meantime, I hope that teachers like the one I mention above are allowed or even encouraged to try new ways of teaching and learning and that the burden of proof of effectiveness does not grind progress to a halt. New assessment systems require new systems of measurement. To expect more comprehensive forms of assessment such as portfolios to generate the same simple, supposedly comparable data as has been generated in the past is blatantly unfair to those willing to try something new.



4 thoughts on “Prove it: Stifling innovation with the burden of unobtainable proof

  1. Aaron B.

    Chris: Great post, as always. Your mechanical bull analogy is spot on and made me laugh out loud. Those who would decry the link between portfolios and testsandgrades would be hard pressed to answer those same questions about the status quo. Does current practice increase test scores, engagement, and competency? If it did, and since it has been the norm for years, then test scores, engagement, and competency should be through the roof. No one that asks those questions really wants any proof, they just want to excuse the laziness and complacency of current system.

  2. mannixl

    Hey Chris – I think I’m in a middle camp somewhere. I have not used portfolios in the past (a scientific notebook is about as close as I’ve got) but I would like to. What I worry about is whether or not I will be good at assessing students’ conceptual understanding through projects. For example, I can easily picture a situation where a student sets up a beautiful webpage on Newton’s Laws, defines them, shows a video with themselves demonstrating them, maybe a lab, etc…..but then gets a test question like the following, and others like it, consistently wrong :

    A semi collides with a car on the highway, which of the following is true? A) The car exerts more force on the semi B) The Semi exerts more force on the car C) they exert the same force on each other.

    or, since i’m mostly teaching biology now: a student could do a lot of projects with photosynthesis, make diagrams of inputs and outputs, etc etc. But when you asked them where a plants mass comes from, the soil, water, sunlight, or the air…I often find they think it comes from the soil or from sunlight.

    Full disclosure: students with regular instruction and assessment don’t exactly knock conceptual quizzes out of the park either, but how do you deal with that? Can projects and portfolios address this kind of conceptual understanding effectively? Do you still do a fair amount of conceptual quizzing and testing? Or do you feel like it is not important? Is there a way you are able to address conceptual stuff like Newton’s laws through a project more effectively than I realize?

    1. Chris Ludwig Post author


      One of the strengths of portfolios is that you can put ANYTHING in the portfolio, including grades from more traditional assessments such as quizzes and tests. Any type of assessment can find its way into the portfolio, not just the project-based or blogging variety. I don’t do it often, but I do give exams that students then can use as further proof that they understand a concept. Mostly I switched away from exams to blogs since I realized all my exams were essay questions that work in a blog format just as well.

      I do toss the occasional multiple choice, concept-heavy exam their way, though. Students post that score on the appropriate page of the portfolio. For example, when I give pre- and post-tests on mitosis, cell cycle, and cancer, students can share their results on the Cell Reproduction page of the biology portfolio as further proof of understanding the content.

      I agree that we have to be careful when determining exactly what a student knows and can do when using online tools (or offline for that matter). My strategy is to try to get multiple lines of evidence from them that form a clearer picture of what they do and don’t truly understand.

  3. Kaitlin Savage

    You give an interesting point. Who is to see which method of assessment REALLY gives us information as to how a student is actually progressing in our classroom. I’m currently taking a grad class regarding teacher leadership. As I reflect after each week, I’m noticing that the big idea I am taking away from this is that any teacher can be a teacher leader, as long as they have the skill set and specific dispositions associated with that goal.
    I really do believe that all it would take to turn this entire assessment “mess” around, it one strong teacher leader (someone who actually knows what they are talking about). Right now, the middle school I currently work at in New Jersey uses standardized tests to measure students growth. We have even purchased a program through our district that is training these students to do well on this test. Tell me we’re not teaching to the test…
    I would LOVE to have the ability to assess these students through the use of portfolios. Is everyone else feeling the same way?


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.