Teachers, Data and Dashboards: Why Principles of Adoption and Diffusion Matter
The Research Alliance for New York City Schools released a great new report on the NYC Department of Education's $80 million Achievement Reporting and Innovation System (ARIS). This is an incredibly detailed report and I encourage anyone who is interested in student information systems or reporting dashboards to read it.
This report builds on findings of earlier reports (see John Tyler's report) that "dashboard" systems don't get a lot of play with teachers. I'm only going to focus on a couple of findings from the ARIS report in this post:
1. The average user logged on to the system 21 times during the year, for just under five minutes per session.
2. The "heavy" users "accounted for more than 80 percent of all time spent on the system" -- but they only spent four and a half hours of usage during the entire year.
I think it's fair to say that ARIS wasn't used to the extent that people had hoped it would be.
At New Visions, we know quite a bit about dashboard and student information systems and the ongoing, painstaking, never-ending training that has to accompany this type of innovation. In 2010 we rolled out a web-based student information system, DataCation, to the 75 schools in our network and reinforced that roll-out with intensive trainings for school staffs (Note: New Visions assumes the cost of providing DataCation to our network, although as a third-party vendor, it is available to schools and districts nationwide for a fee).
In the two years since we've been using DataCation, we've delivered over 260 trainings that have reached almost 3000 educators. Since 2009, DataCation has grown from 32 schools to 352 schools in NYC; and has expanded to multiple districts throughout NY State. They have also expanded nationally since 2009. DataCation platform is now being used in schools in Virginia, New Mexico, Wisconsin, and California.
So, why did DataCation go viral while ARIS did not?
For me, understanding this question hinges on at least two dimensions of adoption and diffusion (see Rogers' seminal work): 1) timing (where are you in the adoption/diffusion cycle?) and 2) compatibility (does the tool support above-the-flow or within-the-flow work structures?).
Timing: Where you are in the adoption-diffusion cycle matters
Mayor Michael Bloomberg and former Chancellor Joel Klein created a new space in the field of education -- one dominated by data. The progress reports, the School Quality Review in addition to the federal NCLB policy created a culture of accountability that necessitated real-time access to data. ARIS was designed to deliver data-on-demand that would meet schools' accountability needs. It was a groundbreaking, bold initiative. It is no easy thing to be one of the first to arrive in an innovative space. It's uncharted territory and there aren't a lot of prototypes to draw from - especially on the scale of NYC. The innovation space is a rather lonely place.
But this was also one of the problems. ARIS arrived early in that cycle of adoption and diffusion before the accountability culture was fully established. Adoption of an innovation is predicated on the idea that the innovation is: 1) compatible and aligned to work (i.e., is it relevant?) and 2) easy to use. In the early days of ARIS rollout (presentations on ARIS began in 2007-08 and rollout began in 2008-09), NYC DOE was still making the case for data-driven schools and its accountability structures were in formation. Educators were just trying to wrap their heads around the complex new structures. People aren't going to use tools until they are clear on why they need the tool. Without that clarity, the tool doesn't feel aligned to the work. In other words, the tool won't be adopted.
DataCation, on the other hand, hit the scene in a bigger, more visible way a bit later in the adoption / diffusion cycle (in 2010), when the culture of accountability was more established. DataCation didn't have to convince educators about the need to look at data. By then, school staff were clamoring for real-time, easily accessible data and they knew what sort of tools they needed. And, because they came slightly after ARIS, DataCation benefited from observing what was working and what was not working.
Once educators recognized they needed tools to manage data, why did they turn to other systems rather than embracing ARIS, a free-of-charge system? (While New Visions pays for DataCation service for the schools in our network, the other 278 NYC schools using DataCation pay full price.) There are likely a lot of reasons for this, but I'm going to focus on one in particular -- ARIS doesn't differentiate between above-the-flow and within-the-flow for different audiences.
Compatiblity: Above-the-Flow and Within-the-Flow
Tools have to be relevant and tightly aligned to the work if you want them to be adopted. In their book The New Edge in Knowledge, Carla O'Dell and Cindy Hubert draw from Michael Idinopulos' blog Transparent Office where he discusses work that is above-the-flow and within-the-flow.
They write: "Enabling employees to do their work more easily - by collaborating and capturing and sharing knowledge without an additional burden or interruption on their part" is working within-the-flow. However, when you ask "employees to stop their work process to move to another mode to reflect, capture, or share" you are asking those employees to work above-the-flow. If you want employees to work outside of their day-to-day flow, "then you will need to explain why and ensure there is an intrinsic or extrinsic payoff."
If the goal was for teachers to use ARIS, the question is -- does ARIS support their work within-the-flow?
If you have spent any time on the ARIS, then you know that it feels very IT-ish where greater weight is given to standardization over the more bubbly bottom-up, school and teacher driven customization. For teachers, ARIS is an above-the-flow, top-down system. It didn't meet teachers in the flow of their day-to-day. In fact, for teachers to access ARIS, it took them out of their day-to-day flow. Principals, data-specialists and other administrative staff might have used ARIS more frequently (the "heavy" users Research Alliance talks about) because it supported their work. So, in this way ARIS might be within-the-flow for the school administrators -- but above-the-flow for teachers' work.
DataCation, on the other hand, is driven by its gradebook function. The gradebook is one of the most relevant tools for a teacher. It is deeply aligned to their day-to-day work. What teacher doesn't use a gradebook? Teachers were front and center in the design of the DataCation platform. In fact, teachers designed the DataCation platform!
DataCation Gradebook Sample (with dummy student data)
The student and classroom data that teachers need are embedded within the gradebook portal. That is, when a teacher is creating an assignment and needs some data on a particular student, the information is right there - a click away - within the flow. The reporting needs of principals and administrators -- which are above the flow for teachers -- are also easily accessed through the gradebook portal. In this way, DataCation managed to direct access to the above-the-flow and within-the-flow rather seamlessly.
No technology is perfect. We know this first hand. Our process of rolling out DataCation to our schools had some supremely messy moments. And we learned some hard lessons. Two big lessons: 1) always know where you are in the adoption-diffusion cycle (and be a little nervous if you've arrived early) and 2) if you want teachers to use data -- deliver it within the flow of their work.
But there are many more lessons learned that we will be addressing in future posts. So stay tuned!
Susan Fairchild is director of program analysis and applied research at New Visions. Follow her on Twitter at @SKFchild.