Survey

State Leaders Weigh In on Open Source Assessment

• By Scott Aronowitz
• 03/29/10

Open source assessments have great potential for cost savings, collaboration, and standards adoption, but there are also some perception barriers that stand in the way of wider adoption in the immediate future, according to a new report exploring the attitudes of state assessment and technology leaders.

Market research firm Grunwald Associates released the report based on an in-depth survey on open source assessment. The report, entitled "A Report on Education Leaders' Perceptions of Online Testing in an Open Source Environment," was sponsored by Educational Testing Service (ETS), and it reveals several interesting perceptions and preferences regarding open source online testing.

Grunwald contacted 81 state assessment and technology leaders and educational organization leaders across 27 states and used the answers provided to glean a broad spectrum of data on the topic. From among the 43 state leaders interviewed, the company divided respondents into three usage categories: those from states with widespread online administration of annual accountability assessments; those from states that have experimented with online testing, with limited participation; and those from states with limited or no experience yet with online testing. The 38 organization leaders were categorized by the organizations they represented: associations, coalitions, and foundations; state and national policy organizations; businesses and nonprofits active in education issues; and universities and research organizations.

Once the company's researchers analyzed and interpreted the gathered data, Grunwald made several observations, and extrapolated a number of perceptions and trends, related to open source online testing. The following is a brief overview of the results:

Perceived benefits of open source assessment:

• Potential cost savings based on absence of licensing fees;
• Common formatting, data standards, and development standards improve/would improve adaptability and, subsequently, efficiency; and
• Collaboration benefits, including shared resources, ideas, testing standards, and even risks.
  

Concerns about open source assessment:

• Possible hidden costs, including maintenance, technical support (sometimes a cost when using an open source product), product development necessary to make modifications, and ongoing professional development for educators using original and modified versions;
• Perception of security risks to both source code and content; and
• The potential downsides to collaboration, including lack of leadership, lack of alignment in thinking among those recognized as experts for the purposes of development and modifications, and both inherent and unforeseeable inefficiencies.

Additional observations:

• The greater a state's current investment in open source technology and its education leaders' and educators' awareness of what it offers, the greater the prevailing interest in increasing its use, in advancing its quality, and in becoming better educated about the technology and the content it propagates and has the potential to offer;
• Education leaders need to be better educated about both the benefits and risks of open source technology and its related issues;
• Quality, security, ease of use, and access to effective support are of far greater concern than cost savings to users and potential users of the technology;
• Because effective evaluation of students' comprehension, progress, and potential requires more complex and in-depth assessment, in order for the education community to embrace the technology for the long term, it must evolve to include more than multiple-choice and short answer options; and
• Many of the prevailing issues surrounding the use of open source technology for assessment can be addressed with strong leadership, reliable structure, and a well organized approach.

The complete report, which includes more detailed results, a complete explanation of the methodology used in the study, two case studies of individual states' forays into wide use of open source technology, and strategies recommended by interviewees for approaching the technology, can be found here.