Foundation releases executive summary of review of Idaho Instructional Management System pilot
The State of Idaho and the J.A. and Kathryn Albertson Family Foundation (JKAF) have made considerable investments in Idaho education and data systems to better serve students. One key JKAF investment has been in Schoolnet, an Instructional Management System (IMS), designed to make data easily accessible and useable by teachers to directly impact classroom and school decision-making and ultimately student success. Schoolnet, which was selected by the Idaho State Department of Education (SDE) through an independent competitive bid process, has been implemented and used successfully in school districts around the United States.
Despite significant investments in Schoolnet, JKAF has heard from the field about struggles with the roll-out of the system. JKAF was concerned there have been barriers to successful implementation of Schoolnet, and it is not yet making the expected difference for Idaho educators and, more importantly, Idaho students. JKAF set out to better understand how educators in Idaho are using Schoolnet and similar systems by contracting with the Institute for Evidence-Based Change (IEBC) to conduct an external review of school districts’ use of IMSs including Mileposts, as well as Idaho’s K-12 student longitudinal data system, the Idaho System for Educational Excellence (ISEE). This review was designed to assess the rollout, implementation, and use of these systems. In May 2013, educators across the state contributed their input through interviews and surveys administered by IEBC. IEBC identified several key areas that could be improved and recommended action steps for SDE and Pearson, the company that administers Schoolnet. Once the review was complete, IEBC facilitated meetings with SDE and Pearson to develop an actionable final year work plan to address the challenges identified in the statewide review.
This Executive Summary describes IEBC’s review and facilitation process to capture and address Idaho’s concerns and needs in the implementation of Schoolnet. This summary also presents the findings of the external review and a discussion about how SDE and Pearson have worked to address them.
Review and Facilitation Process
During May 2013, IEBC conducted the review of IMSs used by Idaho school districts. This effort included a document review, site visits with face-to-face interviews, follow-up interviews by telephone, and a survey to collect information.
IEBC conducted 11 site visits to five Schoolnet districts, four Milepost districts, and two districts that identified themselves as using both Schoolnet and Mileposts. IEBC interviewed administrators (e.g., superintendents, principals, instructional coordinators, district information technology (IT) specialists) at each district to learn about the IMS selection, implementation, and use. In addition to the district interviews, IEBC interviewed Foundation staff, Idaho SDE staff, and IMS developers (Pearson and Silverback Learning Solutions).
IEBC’s online survey instrument included questions about knowledge of IMS systems, how school districts and teachers have implemented these systems, and related questions about professional development and system use. IEBC selected all site visit districts and 18 additional districts to participate in the survey.
Once all of the data were collected, IEBC performed a qualitative review of themes found among the interviewees per site and across districts that resulted in a report delivered to JKAF. The report identified major findings and recommended action steps to expand use of the IMSs in Idaho.
IEBC then convened SDE and Pearson—including their high-level leadership—in a series of meetings to redefine the third year work plan for Schoolnet considering the review findings. IEBC recommended thisfacilitated meeting with high-level decision makers and other team members to move beyond past issues, assess current barriers to successful implementation, and establish a positive and collaborative working relationship between SDE and Pearson. Ultimately, these meetings needed to result in a specific, actionable work plan that both parties could stand behind to put Schoolnet on a path to success in this final development year.
What were the evaluation findings and how have SDE and Pearson worked to address them?
IEBC found that Schoolnet has been underused in Idaho, and most educators expressed frustration because implementation has not been seamless or as fast as districts would like. Despite frustration with the system and the available data, Schoolnet users remained hopeful that the project team was going to work out issues, especially those related to the data transfer from districts to ISEE to Schoolnet, and Schoolnet would become a useful system over time. Considering the investment of resources, sweat equity, and local commitment to date, it was prudent to work to address system issues rather than begin with a new effort.
SDE and Pearson designed the third year work plan to realize the hopes in the field and ease the frustrations. The plan identifies activities that are feasible to complete in the remaining timeframe and addresses many of the evaluation concerns. Moreover, on a project suffering from a lack of clarity around tasks, deliverables, goals and success, the submitted work plan provides a level of specificity and accountability that was lacking.
In addition to the frustration with Schoolnet, IEBC identified several other findings from the field in the Idaho district IMS review. These key findings are described below, with discussion about how SDE’s third year work plan is designed to address them.
Stronger SDE Leadership & Teamwork is Needed
The capacity of the SDE team to effectively manage this project has been a challenge, particularly in the current phase of the implementation where teacher use is paramount. The project lens needs to shift from a focus on technology implementation to supporting teachers’ use of the rich content and data to improve student learning. SDE project leadership expertise has not fulfilled the project needs; and key resources in the field (e.g., ISEE Navigators) have not provided the desired support to help districts use Schoolnet in a meaningful way.
The SDE and Pearson teams also have not had a consistently productive working relationship. While new project leadership from Pearson has improved this relationship, miscommunications between the entities continued to impede progress on the Schoolnet implementation.
The original implementation plan called for SDE to self-host the Schoolnet platform. Although self-hosting may provide a measure of independence from Pearson, this option limits how Pearson can support Schoolnet in Idaho if needed. Additionally, the necessary training and expertise to maintain the system and implement future version updates of Schoolnet may not be available to SDE staff. Finally, if SDE alters the underlying Schoolnet code as bugs or changes to accommodate user needs arise, these alterations will threaten the stability and consistency of the Schoolnet product.
Current efforts to build stronger SDE leadership and teamwork
SDE has named Roger Quarles to lead the Schoolnet project. Dr. Quarles brings a user-level perspective as a former district superintendent and teacher, and co-director of Idaho Leads. Further, SDE is committed to re-evaluating their internal team and seeking needed expertise to support the project and address current gaps in service to the field; one example is the ISEE Navigator resource. As a key SDE resource in the field, the ISEE Navigators must know how to use Schoolnet to change classroom practice and how to train others to do so.
The facilitated meetings between SDE and Pearson focused first on building trust and effective communication between the two teams. SDE and Pearson drafted and agreed to the plans for year three together. The plan identifies common expectations about the project’s status between the two groups. Finally, the key leaders on both teams committed to regular meetings to address escalating issues quickly and efficiently.
JKAF has established a payment schedule that is tied to key deliverables over the next year to provide an incentive for SDE and Pearson to continue this productive collaboration. JKAF also is encouraging SDE to ensure that no modifications to the Schoolnet code by SDE are allowed.
Data Quality is Crucial for Success
The review found that issues with data quality and access were paramount concerns among users. Most respondents do not trust their data are being processed properly in the ISEE system, the data presented in Schoolnet are not considered reliable, and the resources required to deal with data issues are costly. Data access also is problematic. Users have difficulty navigating Schoolnet, system updates intended to be responsive to user issues are confusing, and the data are not up-to-date—the timing of uploads and time lag between submission and access to data need to be addressed. All of this leads to low levels of engagement.
Current efforts to improve data quality in Schoolnet
The Schoolnet system has been used successfully to positively impact teaching and learning by many schools and school districts in other areas of the country. The key difference in Idaho’s experience from other successful implementations is the way data are loaded into the Schoolnet system. Idaho is populating data in Schoolnet from a statewide longitudinal data system (SLDS), which is a novel and complicated approach. While there are benefits to linking Schoolnet to a SLDS (e.g., cross-district student records), the SDE and Pearson teams acknowledge that increasing the rate at which data are processed through Schoolnet will improve the perceived data quality and usefulness of the system to teachers. For example, in some districts nightly data updates are necessary for teachers to have accurate class rosters display in Schoolnet. The SDE and Pearson teams are implementing two methods to improve the data quality in Schoolnet: the five file data submission and the direct connect solution.
The five file data submission process is a means to overcome the data availability and accuracy issues. Data are updated more frequently— as often as every night—so changes are apparent more quickly in the system. However, the data submitted may have some problems as the validation process is less rigorous for the five file than the monthly ISEE submission. Over 50 districts have completed data submissions using the five file process. More testing and feedback from districts will be used to see how the process affects use.
Another parallel fix to address data quality concerns is the direct connection of selected districts’ student information systems (SIS) to Schoolnet. SDE and Pearson are working with a group of districts to implement the direct connection in fall 2013. Again, testing and feedback from districts will be necessary to see if this is a viable and effective method to increase use of the system for Idaho districts.
Stronger Communication and Feedback Loop are Necessary
In the IMS review, IEBC found weak communication from SDE to the field, and also from the field to SDE. SDE is the hub for all information being disseminated about data quality, data access, system use, and professional development. It is responsible for managing expectations, as well. Thus, communications from SDE to the field and vice versa are crucial. SDE must be perceived as clear, honest, and timely. Furthermore, communication must flow in and out of SDE—they need to listen and react as much as act. These communications should dispel rumors, help market the product, and meet the real needs in the field. Furthermore, honest feedback about use and ongoing struggles with use is critical to improve the implementation of Schoolnet.
Current efforts to build stronger communication
The SDE leadership has committed to repurpose efforts in communication and marketing to focus on teachers. SDE and Pearson are working on several methods to get information out to the field, such as monthly newsletters, an Advisory Group of superintendents, and a Power User’s group. The latter two groups will be vehicles to solicit feedback from the field. In addition, SDE will target select districts that are piloting new data submission procedures for follow-up. SDE also plans to utilize ISEE Navigators to provide reports about their experience working directly with districts, highlighting district needs and feedback.
Defining the Value to Teachers is Crucial
Ultimately, the value to teachers is the prime determinant of use. Time and again interview respondents noted that if teachers do not see value in the system they will not use it. Value is defined as saving time and improving student outcomes. Furthermore, using technology simply for technology’s sake is not perceived as bringing value to teachers. The case needs to be made for some faculty that moving from paper-and-pencil to technology-based instruction and assessment brings value (e.g., interactions with a broader group of colleagues; centralized data for district accessibility; one-stop access to a broad array of items such as lesson plans, assessments, student data; ease of reporting to parents and students).
The factors that affect implementation of instructional management systems do not occur out of context. This review revealed that issues of local control and past events such as the Idaho Student Information Management System (ISIMS) rollout have affected the level of acceptance of Schoolnet among Idaho faculty, administrators, and other stakeholders. Furthermore, there are many educators using another IMS in Idaho, Mileposts. Although the review found that Mileposts users mainly use it for Response to Intervention (RTI), which differs from the more comprehensive focus of Schoolnet, ardent supports of Mileposts are resistant to Schoolnet. It is likely that the Mileposts system would experience similar problems to Schoolnet when attempting to connect to ISEE (currently Mileposts connects only to local district data systems). IEBC recommended that documenting and tailoring professional development about best practices among IMSs, regardless of system, may benefit Idaho educators.
Current efforts to define value to teachers
The current SDE team leader, Roger Quarles, is committed to connecting Schoolnet to the teacher experience. Furthermore, the Pearson team will connect Schoolnet to teacher needs by focusing on key use cases (e.g., using Schoolnet to create and administer an assessment) that will directly benefit teachers using the system. Pearson and SDE will emphasize these use cases in trainings and communications with districts.
The field was loud and clear—the implementation of Schoolnet has been struggling and changes to project management and a concerted focus on use in the classroom were possible to improve the Idaho users’ experience. With JKAF’s support, SDE and Pearson have been systematically working their way through the identified concerns with a facilitated process; and they have come a long way in the past three months toward addressing the shortcomings of the Schoolnet implementation. SDE and Pearson collaboratively created a plan forward that empowers SDE leadership to focus on teacher use, integrates needed technical changes into Pearson’s development schedule, and improves timeliness and accuracy of the data in Schoolnet.
The remaining months of the pilot implementation of this project will continue to face challenges as all pilots do. However, SDE and Pearson have created a feasible and actionable plan that focuses on Schoolnet use in the classroom—where Schoolnet can make the biggest differences for Idaho students. SDE and Pearson remain committed to the project and to working toward effective use of Schoolnet among teachers across the state. JKAF will continue to support the project with a revised payment schedule that is tied to outcomes, driving the team toward success.
The successful implementation of this project – with strong instructional impact and resultant student achievement – is critical to the future of Idaho’s education system. Accurate and timely data in support of effective instruction supported by state-of-the-art technology will enable Idaho to produce students ready for college and career and rightly take its place among the leading states in the country.