What goes into measuring the success of edtech?

What does “quality” mean in assessing statewide digital efforts?

The Foundation for Excellence in Education recently released its 2014 Digital Learning Report Card. According to this report, Vermont does not support digital learning. In fact, all of New England is a digital wasteland. But what does the data really say? How are these researchers quantifying “digital learning”?

And how can we use this report to look at other measurements of success with edtech?

Lies, darn lies and statistics

 

Join our fearless data guy/historian Mark Olofson as we unpack the metrics behind this report, and why no one (with the possible exception of the Washington Post) is panicking just yet. Full transcript appears below for your convenience.

 

---

 

What goes into measuring the success of edtech?

Everyone remembers that horrible sinking feeling… The first time they were handed a report card with, uh, lets say less than stellar grades on it. Maybe you were too busy drawing, or reading, or being otherwise distracted to engage with the founding of Seattle. Maybe you were allergic to anatomical diagrams of frogs. Or maybe, long division just really isn’t you thing, Ok, Mr. Paccelli? Anyway…

In this episode of The 21st Century Classroom, resident stats guru Mark Olofson and I look at a recent study by the Foundation for Excellence in Education that gave our beloved home state of Vermont a D- in Digital Learning. A D…minus. Ouch.

Audrey: Mark, we got a D- in 2014. Vermont, as a state, got a D- in EdTech. How did this happen?!

Mark: D’s are not passing, and D-, that’s so close to an F. So close to an F. Well, I mean, because it doesn’t feel like we should have a D-. It doesn’t– I don’t feel that we’re D- material.

Audrey: No!

Mark: I think it’s important for us to look at what they did say about Vermont. Um, and so, these rankings are based on ten different criteria: student eligibility; student access; personalized learning; advancement; quality content; quality instruction; quality choices; assessment and accountability; funding; and delivery. And I believe it was in the — one of these articles we did mention — that the quality content is, pretty across the board, there is quality content available. Which is definitely good to see, and that also there is a lot of high quality instruction. And so, those are places that, you know, across the nation and also here in Vermont we can point to and say “Yes, there is quality digital content out there for our students.”

The most surprising thing, and I’ll just say it say it, the most surprising thing that I saw was that in the category of Personalized Learning we got an F.

Audrey: Isn’t that surprising considering that we have to have had a whole mandate to get us towards personalized learning. Were we so far behind in personalized learning that that is what spurred Act 77 and the Personalized Learning Plan Mandate?

A little background on Act 77

In 2013 the Vermont Legislator passed the “Flexible Pathways Act,” also known as Act 77. It mandates that every student in Vermont in grades 7-12 will have personal learning plans in place by fall of 2015. All of them. By fall of 2015. Which is, uh, now…

Mark: I don’t feel that we’re really that far behind. In fact, I think you’ll see a lot of people pointing to Act 77 as being out in front of the field around personalized learning. Um, and so, this, like so many other pieces, it’s really important that you drill down and take a look at what they’re actually measuring. You know and, thinking about as, any time you give an assessment it can be tempting to just look at the final grade. Um, but it’s important not only to look at different sub-scales but also to look at what these different measures are actually measuring.

So, I’d like to dive in a little bit and click through and see, what are these things… You know, it’s easy to say “Student Eligibility,” but what does that mean to the people who are creating the measure?

This hangs its hat on three criteria: that all students must be provided opportunities to access online courses and services throughout their entire K-12 experience. That’s their entire K-12 experience. Vermont got something like a 25% on that. That means that there are times in a student’s K-12 experience that they don’t have access to online courses. And, that’s an area of growth, but that’s also a very particular metric that’s being measured.

Under the student access course, and we got a zero on this, was that all students may enroll in an unlimited number of part time, individual, online courses. Now, that’s not a question of digital access, that’s an attitude on a district-by-district and state-wide basis, right? And so, this is a criteria that is, we may not align values-wise, or the way that we think that education should be done, that digital education should be done, with the criteria. And so, this foundation is making choices on, er, is creating criteria based on their values. If our values don’t align, we’re not gonna score highly, right?

And if the way that we conceptualize of what is the best place for leveraging technology in education, we may not be aligned with what they think. Um, another great example of this is under the Advancement, and Vermont gets a zero on this, one of their criteria is that “all students are provided multiple opportunities during the year to take end of course exams.”

That is a very large pedagogical statement. That is, I mean it is very much standards based, but it’s saying that, in order to score highly on this metric, we would provide students, all students, chances to essentially test out of courses throughout the course of the year.

And that’s, for a lot of folks in education, that’s saying that only the content matters, and it’s not about the experience of the learning, and it’s not about the experience of the class, and it’s not about constructing meaning with your peers. It’s just about “can you finish a test.” And so, it’s a very value-laden criteria to toss in there, to say that, you know, in order to achieve highly on this measure, you have to provide students opportunities to test out of courses. And, we may not want to do that. So that’s why I think it’s important to drill down and really see what metrics like this, that are easy to create headlines, are all about.

I mean, if we look at the quality content measure, here Vermont get’s A’s all the way around. You know, our stuff is aligned, the digital content provided is aligned with our state standards. It’s not hard to get digital content through into being approved, and that, you know, materials are funded. Same thing with our instruction being of high quality. You know, these are places where we have high measures.

Hear that, Vermont?

Even with a D- overall, we still scored really, really highly on the quality of our educators. Give yourselves a pat on the back, you amazing educators, you!

Mark: But then when we go down to quality choices, one of the metrics is that there are full time online schools. And, we don’t have full time online schools.

Audrey: I’m trying to envision how, you know, looking at how school change occurs, I’m trying to envision how you take a traditional one room school, or a traditional “metropolitan” school, like in the Burlington area, and explain to the community that you would like them to ratify a budget for having the school be entirely online.

What does that conversation look like? And is that a conversation that you can conceive of going in a productive manner in most of Vermont?

Mark: Well, that’s interesting because, when you think about, you know, should that be something contained at a supervisory union or a school district level. Think about the amount of investment that has to go in to make an online learning program viable.

It’s hard for just one district, one supervisory union, to take that on. And so that’s why we start to see these state-wide initiatives; why we do see some digital programs stretched across the state. But then that gets to be very difficult when it comes time to say who pays for what. So it’s a really– setting up, you know, online schools is very, very difficult. In Vermont we don’t have charter schools. In a lot of states, they take a charter school approach to setting up an online school. Because, you know, thinking about it, it makes sense, because charter schools should be open to anyone in the state, and so if you’re getting funding based  on, you know, the kids that are going to sign on and take these online courses, that that would then be a state-wide initiative. Right?

In the state of Vermont, we don’t have that set-up.

Audrey: When you think of all of the valuable lessons that students can get from hands-on learning. One example that this brings to mind is, you know, our partners at the Cabot school, you know, packed up all of their students and took them to, um, the Flume Gorge in Franconia, New Hampshire, and they all had their iPads, and they were all  in the moment recording their reactions to the Flume Gorge, and many of the students had never seen this park before. And their teachers had created an entire scavenger hunt with UR codes, like through the park, based on, you know, learning more facts.

And that was, for them, really hands on, just in terms of going through the experience.

And then, having the freedom to come back and make something, with their class mates, in the classroom setting with the teachers’ guidance and support. How would that experience have translated to a 100% online school?

Mark: I don’t think that it can translate to a 100% online school. And what we’re chiseling away at here is really: what is–what’s the purpose of school? What’s the purpose of having, you know, a place, a building, where students from an area congregate and where teachers and students do education together? So, if the point of school is simply to learn a bunch of stuff, and be able to report out that you learned a whole bunch of stuff, then the fully digital model, it makes sense. We can conceptualize of what that would look like, you know, through videos, and text, and audio, and practice, and drill, and all of these different things, students can lean skills, they can acquire knowledge. There, what we’re thinking about is, you know, a student, in this very simplistic model, of a student being a bucket, then you pour stuff into the bucket, then there’s stuff in the bucket, right?

And you can walk around with the bucket. But if we think about schooling, and I think that we do, think about education in a much more broad manner, where it’s important to be having those discussions, those free-form discussions between students and teachers to really explore interests and to develop your own meaning behind, you know, the Gorge, right?

Because that gives students the opportunity to make sense of it. Instead of “learning” it, they’re making sense of it. The purpose of education, especially in our world now, and I know that’s, you know, the sort of thing that gets tossed around all the time, but: it’s about understanding problems; it’s about critical thinking; it’s about innovation; it’s about all of these things that can’t just be transmitted and, you know, regurgitated. And so, I balk at the idea of, you know, a fully online school. I think that for, there are certain places where, um, an online course makes sense. When that comes up as a metric, I do hesitate to say, “Oh, it’s bad that we’re not at 100% there.”

It just means that, you know, that’s not in alignment with what we see as the best way of doing teaching and learning. It also tells us that the folks who are scoring really highly, you know, different states, and, you know, although we’re technically in the same country, I think that Florida and Vermont cultures are quite a bit different, right?

And, although, there is a lot of rural culture in Florida, and too often we ignore that. If in Florida it makes sense for them, you know, infrastructure wise and being able to meet, you know, the needs of all these students-folks all the way out in the Everglades, they need to have a state wide online school available, then more power to them. And then they are going to score high on that metric. And that’s great. It’s just important that, you know, that we’re able to parse out and say “Ok, so, is that in alignment with our values? Does it make sense?” And, also, “Is it in alignment with our legislation?”

Another piece here, on the assessment and the accountability, is that if “Poor performing providers of digital content should be not renewed or lose their ability to serve students.” Well, first we don’t really have these state-wide digital providers, but even if we did I think that we’re going to find that we would work with them to kinda make them better, vs an accountability structure that, um, you know, “if you don’t perform, we shut you down.” Um, we don’t have that, we don’t have that culture with our schools. We don’t, you know, if-

Audrey: If you get an F, you don’t get asked to leave.

Mark: Exactly, right?

Audrey: It’s like: “How can we help you? How can we bring you up to speed and get you involved?”

Mark: Exactly. “What can we do? What else can we offer? What can we work with you?” you know? I mean, for crying out loud, if, and we have this very deep in our culture, coming from– with the AOE and Rebecca Holcomb issued about, um, the aspect testing. Um, that we’re going to use this as a  measure, not as the measure. Right?

And, as we continue to learn about, you know, this test and experiment with it, it’s not going to be the end-all be-all of how we rate kids, of how we rate teachers, and of how we rate schools. We understand that there’s a lot more intricacies to the, you know, the doing of education. Um, and they can’t just be measured on one test. And that’s very much in our educational culture in Vermont. We understand that there’s a lot of moving pieces.

What about other statewide testing scores?

Around the time this report appeared, another report appeared here in Vermont: the results of the “Smarter Balanced” standardized test, administered for the first time last year statewide. Vermont Secretary of Education Rebecca Holcombe went on Vermont Public Radio to talk about those results.

 

So there you have it.

As Mark Twain might have put it, were he a Vermonter and perhaps while leading a nearpod exercise on how far frogs jump, rumors of Vermont’s schoolings’ demise have been vastly exaggerated. This Foundation for Excellence in Education study is a measure of educational technology, but it is far from being the only measure of quality.

By the way: the story of the founding of Seattle is actually pretty darn interesting. And what the textbooks like to leave out are all the good parts, like: mud, cholera, infidelity, and exploding toilets. Google it.

The music in this episode of our podcast is used by permission, as always, from the band dirtwire, and you can find out more about them on Soundcloud or over at dirtwire.net.