Build It and They Will Not Necessarily Come

• 06/01/02

The Effectiveness of a Professional Development Web Site for Entry-Year Principals

The Northeast Ohio Principals' Academy Entry-Year Program (EYP) was one of five regional programs that received funding from the Ohio Department of Education for 1999-2001 to develop policy recommendations for state licensure of beginning principals, effective fall 2002. Each program had two university-based co-directors, and the Northeast Ohio region also used a 20-person coordinating committee consisting of superintendents, principals and representatives of higher education. All regional programs were charged to develop, pilot test, evaluate and offer recommendations for sound and workable licensure procedures. To encourage creativity, the leaders of the programs were directed to work independently.

In this project, a diverse group of 30 entry-year principals, recommended by their superintendents, and 30 active or retired principal mentors from a 14-county area received training in the Ohio Administrative Competencies. These competencies were based on the "Interstate School Leaders Licensure Consortium Standards for School Leaders" (1996), available online at www.ccsso.org/isllc.html, and adopted by the Ohio Board of Education in 1998. The participants were provided with mentoring education or support, training and practice in portfolio preparation, opportunities for informal networking, as well as a range of additional professional development activities. Some activities took place at general meetings of the program, but most occurred in monthly cluster meetings held in four geographic areas within the Northeast Ohio region. The cluster meetings were planned and led by program facilitators who were active or retired principals selected for their professional success and interpersonal skills.

The program Web site was designed to enable electronic communication to achieve four instrumental objectives:

1. Facilitate the coordination of program and cluster activities.

2. Encourage formal and informal networking among participants by overcoming the physical distances between them.

3. Reduce the personal isolation often experienced by principals.

4. Provide links and references that would support principal professional development.

The co-directors and coordinating committee viewed the Web site as potentially useful to the participants and, therefore, as an important element of the project.

Each participant was offered the use of a Compaq Prosignia 150 notebook computer or a stipend of equal value; almost everyone chose the notebook. Those who accepted the computer were allowed to keep it upon the project's completion. The computers were intended to:

Web Site Training

At the first general meeting for project participants, the Web site administrator distributed and reviewed a set of technology handouts. These materials consisted of a copy of the Web site home page and a tour of the site that provided basic instructions for using its functions. Also included in the materials were instructions for setting up the Prosignia 150, establishing an account with an ISP, connecting to the Internet, and accessing and leaving the program Web site.

The handouts did not assume extensive technological knowledge, only a basic level of personal comfort and confidence in using PC technology. This assumption was premised on the knowledge that all participants had completed advanced graduate education and that their own schools were participants in the mandatory statewide implementation of the Ohio Educational Management Information System. Following the review of the materials, the presenter encouraged use of the Web site, then invited and responded to participant questions. The notebook computers were then distributed, and all participants were encouraged to bring concerns about the use of the computer or the Web site to their cluster meetings.

At the project's second general meeting, the Web site administrator presented a technology update. He announced that the Internet server at the sponsoring university had upgraded its software to WebCT v2.2, and that each user of the site would need to create a new individual account to login. A handout describing the new login procedure with nontechnical language was distributed and reviewed.

The site administrator and two computer technicians fielded questions concerning computer use and the Web site. The announcement of this meeting had invited participants to bring their notebooks and any technology-related questions. Individuals were again encouraged to make use of the Web site and to seek needed help. Between April and early summer 2000, some participants stated they experienced difficulty in using the Web site. The summer break in the program interfered with discussion of the problems and training of the participants.

The second year of the project focused less on Web site training than on portfolio development by the entry-year principals. Therefore, only minimal time was spent on site training. However, based on experience from the first year, the program incorporated a training session on the use of computers and the Web site into the general meeting in December 2000. At that time, only a few of the participants chose to remain for the training. Those who remained received one-on-one help to resolve their difficulties. Because all participants in the second year of the project were continuing from the first year, no start-up training was provided on the computer or the Web site. Incidental attention was given to the Web site at regional meetings.

Formal Site Evaluation

A two-phase evaluation of the effectiveness of the program Web site was included in the evaluation of the Northeast Ohio Principals' Academy EYP. The evaluators collected relevant information through questionnaires, interviews and observations. Only those parts of the evaluation having to do with the program Web site are reported here. At the end of the second general meeting of the project, the evaluators asked participants to evaluate the achievement of eight training objectives, three of which related to technology. On a scale of one (low) to five (high) their ratings were to:

The participants were also asked to comment on the session. Three participants named technology as part of the session that should be improved. Two participants suggest-ed there should be a live demonstration of the Web site in a computer lab. These comments were consistent with the generally low rate of Web site use observed by the site administrator. On the other hand, two participants said they planned to access the Web site when they returned to work to apply what they had learned.

At the final meeting of the coordinating committee for 1999-2000, the evaluators asked committee mem-bers to complete a written evaluation of the project. The coordinating committee included the four program facilitators. The evaluators conducted personal interviews with these facilitators covering general aspects of the program. The facilitators twice mentioned problems with technology. One facilitator said technology was a resource that could have made communication easier, while another facilitator said "constant training" could have helped them better use the technology.

The evaluators mailed a questionnaire to all mentors and entry-year administrators in the program. One section of the questionnaire focused on the training provided to participants during the project's first year. Four-point ratings were sought in six areas, three of which concerned technology: use of the computer, use of the program Web site and use of e-mail. The objectives, procedures and results of the training were evaluated in each of these areas. Of the six areas that were evaluated, the areas relating to technology received the lowest three ratings. Among these, the highest ratings were given to training on the use of the computer, the second highest ratings were consistently given to training on the use of e-mail, and the lowest ratings were consistently given to training on the use of the program Web site.

From the information reported above, the evaluators drew several conclusions relating to the Web site. Electronic communication was found to be one of the two aspects of the program needing significant improvement. As noted earlier, findings from the first year led to the development of a training session in December 2000 to deal with technology issues. Concerning this meeting, evaluations showed the technology training piece that was so strongly requested in the first year was notable in that more than half of those who participated in the questionnaire did not respond to questions dealing with the notebook computer and use of the Internet. Everyone who responded gave the sessions a three or better rating (on a five-point scale).

A questionnaire evaluating the EYP was sent to program leaders and participants. Technology was one of the major areas evaluated. Respondents were asked to identify what aspects of the technology should continue unchanged, discontinued or changed. Responses related to technology were coded by the evaluators and labeled as positive (continued) or negative (im-proved or changed).

There were 25 comments con-cerned with the provision of notebook computers. Of these, 19 were positive and six were negative. Both positive and negative comments were evenly spread among coordinating committee members, entry-year administrators and mentors. The most frequent positive com-ments centered on the general usefulness of the notebooks. The negative comments contained the need for a more powerful computer, as well as the need for better training and help in using the computer.

There were 31 comments about the program's Web site. Of these, 12 were positive and 19 were negative. The positive comments came evenly from committee members, entry-year participants and mentors. The negative comments were unevenly spread among committee members (three), entry-year participants (nine) and mentors (seven). The suggestions were used to improve it in its present location or to move it to the state level.

There were 23 comments focused on the technology training. Of these, two were positive and 21 were negative. The comments were evenly distributed among committee mem-bers, entry-year principals and mentors. The comments favored better hands-on training that was earlier and more helpful throughout the course of the program.

In April 2001, program participants met for the last general meeting of the program. A key purpose of the meeting was to generate policy recommendations for state licensure of beginning principals. Technology was one of seven areas in which the participants developed recommendations.

Conclusions

The central question of this study was whether the Northeast Ohio Principals' Academy EYP Web site had accomplished its four objectives in 1999-2001. Multiple sources pro-vided evidence that the Web site was not effective in accomplishing its objectives. Those seeking to use a Web site for similar professional development purposes should consider the following seven lessons:

1. The leaders of the program, and the Web site administrator in particular, acted on the mistaken assumption that a good technology idea would create sufficient interest and motivation to prompt effective adoption. This was a highly consequential error. The excitement of participation in an important policy initiative, the incentive of a free notebook computer and the attractive resources available through the Web site were not sufficient enough to ensure the site would be used.

2. The training provided in the use of technology was inadequate. The level of detail that was provided in the training handouts was more than could be effectively communicated by a simple review followed by questions. Purposefully planned instruction was needed. The content should have been broken down into digestible packets, ample time should have been allotted to explication, and hands-on lab experience should have been provided to help participants interpret and use the technology information.

3. The training assumed a certain degree of comfort and confidence on the part of the participants in using PC technology. In fact, not all participants had received extensive technology exposure in their graduate education, and not all possessed familiarity with the state's educational management information system. The participants reflected a variety of readiness that influenced the level of learning and use of the technology. The training should have been more thorough and flexible to address these levels of readiness.

4. The technological update necessitated by the change in university software made a confusing situation worse for some. For these participants, the new login procedure added an undesirable layer of complexity to using the technology. The presentation of a new technology handout with time for questions, but without detailed instruction, was an inadequate response to the needs of these participants. The same may be said of the Web site administrator's response to the difficulties reported in spring 2000 by participants trying to gain access to the site. The problems of these individuals should have been handled promptly and the solutions reported to all participants.

5.The Web site administrator interpreted the unexpected low level of participant use of the Web site in the first year of the program to indicate low interest in use of the Web site. The evaluation data suggest that participant interest was high, but training and technical problems frustrated such use. Closer monitoring of participant perceptions following the first general meeting would have permitted proper corrective action to be taken.

6. The program leaders were probably unrealistic to expect the participants, especially the entry-year principals, to be able and willing to allocate substantial time to the technology once they returned to their jobs. The principalship requires complex and difficult work, and individuals new to the position face extra demands on their time, energy and psychological resources. Formal professional development activitiesfor entry-year principals certainly have a place, but much necessary professional development for entry-year principals occurs at work.

7. Given the importance assigned to the use of the Web site by the program leaders, they should have drawn from the research literature and professional advice that are available on how to implement technology, design and conduct staff development activities, and lead change efforts in general. The university affiliation of the leaders, as well as their own professional involvement in training activities, make this conclusion somewhat uncomfortable.

Features & Functions of the Program Web Site

The Northeast Ohio Principal's Academy EYP Web site was built on the framework provided by WebCT v1.3 course development software. The site offered eight program-related functions, which are identified below. (The following discussion is adapted from the WebCT v1.3 user's guide.)

1. Bulletin Board served as the main form of group communication for the project. All postings to the board were distributed to all program participants with a Web site account. The messages were also archived.

2. Mail allowed participants to send private messages to fellow participants.

3. Chat allowed participants to have real-time conversations with fellow participants. Chat included four general-purpose rooms that could be used simultaneously.

4. Overview of Program briefly described the purposes, funding and operation of the Northeast Ohio Principals' Academy EYP.

5. Participants in Program provided information on how to contact the program co-directors, fiscal agent, facilitators, entry-year principals, mentors and coordinating committee members.

6. Photos documented the general meetings of participants.

7. Electronic Resource Links provided hyperlinks to professional organizations, electronic periodicals, discussion lists and other regional EYP Web sites in Ohio.

8. Print Resource Information provided annotated references to print sources of potential interest to participants.

Lessons Learned

Important lessons can be learned from the Northeast Ohio Principals' Academy Entry-Year Program that can be incorporated into other similar endeavors: