MICHIGAN EDUCATIONAL ASSESSMENT PROGRAM

Michigan Office of the
Auditor General
Thomas H. McTavish, C.P.A.
Auditor General

EXECUTIVE DIGEST #3120099
MICHIGAN EDUCATIONAL ASSESSMENT PROGRAM

INTRODUCTION
This report contains the results of our performance audit of the Michigan Educational Assessment Program (MEAP), Department of Education and Department of Treasury.

AUDIT PURPOSE
This performance audit was conducted as part of the constitutional responsibility of the Office of the Auditor General. Performance audits are conducted on a priority basis related to the potential for improving effectiveness and efficiency.

BACKGROUND
Article VIII, Section 3 of the State Constitution vests in the State Board of Education the leadership and general supervision over all public instruction. Prior to January 1, 2000, the Office of Standards, Assessment, and Accreditation, Department of Education, was responsible for administering MEAP. Effective January 1, 2000, Executive Order No. 1999-12 transferred MEAP to the Department of Treasury.

First introduced in 1970, MEAP has been a Statewide testing program initiated by the State Board of Education and funded by the Legislature. MEAP's mission is to develop the best possible assessments of student academic knowledge and skills. MEAP's goal is to provide data to school districts to use in assessing their instructional programs and to provide information on the status and educational progress of Michigan's schools in specified subject areas to the Legislature, the State Board of Education, the Executive Office, local educators, students, and parents.

MEAP has evolved over the years in response to current research on learning, new areas of emphasis in curriculum and assessment, and increased interest in the performance and accountability of Michigan schools. MEAP is based on objective-referenced tests meeting specific standards established by the State Board of Education that are performance-based and designed to be an indicator of educational progress over time. MEAP will provide assistance to interpret, use, and report student achievement information as one basis for improving schools' instructional programs.

The tests include the MEAP essential skills mathematics and reading tests for all students in grades 4 and 7; the MEAP science, writing, and social studies tests for all students in grades 5 and 8; and the MEAP High School Test (HST) subjects of mathematics, science, reading, writing, and social studies for all students in grade 11.

For the fiscal year ended September 30, 1999, the Department of Education expended approximately $8.1 million in administering its MEAP responsibilities. As of August 5, 1999, the Department of Education had 17 full-time equated employees assigned to MEAP.

AUDIT OBJECTIVES AND CONCLUSIONS
Audit Objective: To evaluate MEAP's effectiveness in assessing students' academic knowledge and skills.

Conclusion: We concluded that MEAP was generally effective in assessing students' academic knowledge and skills. However, our evaluation disclosed one material condition:

MEAP should formally establish the necessary numerical degree of reliability that the MEAP HST should achieve in relation to statutory requirements and routinely compare test results with that degree of reliability (Finding 1).

The Department of Treasury agreed with the corresponding recommendation and informed us that it will take corrective action.
Audit Objective: To assess MEAP's effectiveness in providing MEAP data and test results to school districts to improve students' academic knowledge and skills.

Conclusion: We concluded that MEAP was generally effective in providing grades 4, 5, 7, and 8 MEAP data and test results to school districts to improve students' academic knowledge and skills. However, we also concluded that MEAP was somewhat effective in providing MEAP HST data and test results to school districts to improve students' academic knowledge and skills. Our assessment disclosed one material condition:

MEAP should increase the usefulness of MEAP HST results provided to school districts for assessing and improving their instructional programs and include all appropriate student results in the data reported to school districts (Finding 2).

The Department of Treasury agreed with the corresponding recommendation and informed us that it had taken corrective action.
Also, our assessment disclosed reportable conditions related to MEAP HST administration and MEAP HST security (Findings 3 and 4).

Audit Objective: To assess the reasonableness of and compliance with test development and test scoring procedures.

Conclusion: We concluded that MEAP generally complied with test development and test scoring procedures. However, our assessment disclosed reportable conditions related to controls over compiling and reporting test results and loss of MEAP records (Findings 5 and 6).

Audit Objective: To assess other pertinent issues related to MEAP.

Conclusion: Our assessment disclosed reportable conditions related to MEAP contract management and MEAP HST testing materials (Findings 7 and 8).

AUDIT SCOPE AND METHODOLOGY
Our audit scope was to examine the program and other records of the Michigan Educational Assessment Program. Our audit was conducted in accordance with Government Auditing Standards issued by the Comptroller General of the United States and, accordingly, included such tests of the records and such other auditing procedures as we considered necessary in the circumstances.

Our ability to achieve our third audit objective in accordance with Government Auditing Standards was impeded by the accidental disposition of documentation for the development and scoring of the current grades 4 and 7 mathematics tests first administered in school year 1991?92; the current grades 4 and 7 reading tests first administered in school year 1989-90; the current grades 5 and 8 science and writing tests first administered in school year 1995-96; and the current MEAP HST subjects of mathematics, science, reading, and writing first administered in school year 1995-96 (Finding 6). The Department of Education provided us with documentation for the development and scoring of the current grade 5, grade 8, and high school science and social studies tests first administered in school year 1998-99. Our review of these records disclosed no reportable conditions. However, we could not assess whether the destroyed documentation would have affected our conclusion on our third objective.

Our audit procedures included an examination of MEAP and selected school district records for the period September 1995 through June 1999.

Our methodology included a preliminary survey of MEAP operations. This included interviewing various MEAP staff and reviewing applicable statutes, policies and procedures, reports, and other reference materials.

We visited 27 high schools in 18 school districts and interviewed staff who administered the MEAP HST and received test results. We observed testing practices and assessed security over testing materials at the high schools. We examined student records related to MEAP tests and grades earned since grade 4. Also, we interviewed 11 middle school administrators in the school districts visited who administered the MEAP tests and utilized the test results.

We reviewed methods used by MEAP to measure and evaluate the effectiveness of its tests. Also, we performed a correlation analysis of subject grades and the corresponding subject MEAP test scores for a random sample of students who participated in the MEAP HST during school year 1997-98. In addition, we examined independent assessments of MEAP tests.

We analyzed MEAP data to determine compliance with statutes and contractual provisions and populations of tested students. We reviewed MEAP's processes for developing its tests, assessing the tests for reliability and validity, and establishing the scores used in determining testing performance levels.

AGENCY RESPONSES
Our audit report includes 8 findings and 10 corresponding recommendations. The Department of Treasury's preliminary response indicated that it agreed with all of the recommendations.

Full Audit Report - #3120099 - MICHIGAN EDUCATIONAL ASSESSMENT PROGRAM

Documents prefaced by require the Adobe Acrobat Reader®, a free application available on the Adobe homepage.

This page was created on July 5, 2001.


INTRODUCTION This report contains the results of our performance audit of the Michigan Educational Assessment Program (MEAP), Department of Education and Department of Treasury.

AUDIT PURPOSE This performance audit was conducted as part of the constitutional responsibility of the Office of the Auditor General. Performance audits are conducted on a priority basis related to the potential for improving effectiveness and efficiency.

BACKGROUND Article VIII, Section 3 of the State Constitution vests in the State Board of Education the leadership and general supervision over all public instruction. Prior to January 1, 2000, the Office of Standards, Assessment, and Accreditation, Department of Education, was responsible for administering MEAP. Effective January 1, 2000, Executive Order No. 1999-12 transferred MEAP to the Department of Treasury. First introduced in 1970, MEAP has been a Statewide testing program initiated by the State Board of Education and funded by the Legislature. MEAP's mission is to develop the best possible assessments of student academic knowledge and skills. MEAP's goal is to provide data to school districts to use in assessing their instructional programs and to provide information on the status and educational progress of Michigan's schools in specified subject areas to the Legislature, the State Board of Education, the Executive Office, local educators, students, and parents. MEAP has evolved over the years in response to current research on learning, new areas of emphasis in curriculum and assessment, and increased interest in the performance and accountability of Michigan schools. MEAP is based on objective-referenced tests meeting specific standards established by the State Board of Education that are performance-based and designed to be an indicator of educational progress over time. MEAP will provide assistance to interpret, use, and report student achievement information as one basis for improving schools' instructional programs. The tests include the MEAP essential skills mathematics and reading tests for all students in grades 4 and 7; the MEAP science, writing, and social studies tests for all students in grades 5 and 8; and the MEAP High School Test (HST) subjects of mathematics, science, reading, writing, and social studies for all students in grade 11. For the fiscal year ended September 30, 1999, the Department of Education expended approximately $8.1 million in administering its MEAP responsibilities. As of August 5, 1999, the Department of Education had 17 full-time equated employees assigned to MEAP.

AUDIT OBJECTIVES AND CONCLUSIONS Audit Objective: To evaluate MEAP's effectiveness in assessing students' academic knowledge and skills. Conclusion: We concluded that MEAP was generally effective in assessing students' academic knowledge and skills. However, our evaluation disclosed one material condition: MEAP should formally establish the necessary numerical degree of reliability that the MEAP HST should achieve in relation to statutory requirements and routinely compare test results with that degree of reliability (Finding 1). The Department of Treasury agreed with the corresponding recommendation and informed us that it will take corrective action. Audit Objective: To assess MEAP's effectiveness in providing MEAP data and test results to school districts to improve students' academic knowledge and skills. Conclusion: We concluded that MEAP was generally effective in providing grades 4, 5, 7, and 8 MEAP data and test results to school districts to improve students' academic knowledge and skills. However, we also concluded that MEAP was somewhat effective in providing MEAP HST data and test results to school districts to improve students' academic knowledge and skills. Our assessment disclosed one material condition: MEAP should increase the usefulness of MEAP HST results provided to school districts for assessing and improving their instructional programs and include all appropriate student results in the data reported to school districts (Finding 2). The Department of Treasury agreed with the corresponding recommendation and informed us that it had taken corrective action. Also, our assessment disclosed reportable conditions related to MEAP HST administration and MEAP HST security (Findings 3 and 4). Audit Objective: To assess the reasonableness of and compliance with test development and test scoring procedures. Conclusion: We concluded that MEAP generally complied with test development and test scoring procedures. However, our assessment disclosed reportable conditions related to controls over compiling and reporting test results and loss of MEAP records (Findings 5 and 6). Audit Objective: To assess other pertinent issues related to MEAP. Conclusion: Our assessment disclosed reportable conditions related to MEAP contract management and MEAP HST testing materials (Findings 7 and 8).

AUDIT SCOPE AND METHODOLOGY Our audit scope was to examine the program and other records of the Michigan Educational Assessment Program. Our audit was conducted in accordance with Government Auditing Standards issued by the Comptroller General of the United States and, accordingly, included such tests of the records and such other auditing procedures as we considered necessary in the circumstances. Our ability to achieve our third audit objective in accordance with Government Auditing Standards was impeded by the accidental disposition of documentation for the development and scoring of the current grades 4 and 7 mathematics tests first administered in school year 1991?92; the current grades 4 and 7 reading tests first administered in school year 1989-90; the current grades 5 and 8 science and writing tests first administered in school year 1995-96; and the current MEAP HST subjects of mathematics, science, reading, and writing first administered in school year 1995-96 (Finding 6). The Department of Education provided us with documentation for the development and scoring of the current grade 5, grade 8, and high school science and social studies tests first administered in school year 1998-99. Our review of these records disclosed no reportable conditions. However, we could not assess whether the destroyed documentation would have affected our conclusion on our third objective. Our audit procedures included an examination of MEAP and selected school district records for the period September 1995 through June 1999. Our methodology included a preliminary survey of MEAP operations. This included interviewing various MEAP staff and reviewing applicable statutes, policies and procedures, reports, and other reference materials. We visited 27 high schools in 18 school districts and interviewed staff who administered the MEAP HST and received test results. We observed testing practices and assessed security over testing materials at the high schools. We examined student records related to MEAP tests and grades earned since grade 4. Also, we interviewed 11 middle school administrators in the school districts visited who administered the MEAP tests and utilized the test results. We reviewed methods used by MEAP to measure and evaluate the effectiveness of its tests. Also, we performed a correlation analysis of subject grades and the corresponding subject MEAP test scores for a random sample of students who participated in the MEAP HST during school year 1997-98. In addition, we examined independent assessments of MEAP tests. We analyzed MEAP data to determine compliance with statutes and contractual provisions and populations of tested students. We reviewed MEAP's processes for developing its tests, assessing the tests for reliability and validity, and establishing the scores used in determining testing performance levels.

AGENCY RESPONSES Our audit report includes 8 findings and 10 corresponding recommendations. The Department of Treasury's preliminary response indicated that it agreed with all of the recommendations.