Analyzing CMT and CAPT results

By Bruce Kimmel

Norwalk school officials are in the midst of a detailed analysis of this past year’s CMT and CAPT tests. So far, based on preliminary discussions, it seems the district has been treading water; improvement in some areas, regression in others. It does appear, however, that there has been some improvement in closing the achievement gap.

Now that we have a new Superintendent, I believe it would be worthwhile to look at these test results differently than in the past. In recent years, most discussions of CMT scores have focused on the percentages of students in different categories and then comparing these percentages to past years. For instance, if the percentage of fourth graders at the proficiency or goal levels increased, the district would conclude that there has been improvement.

But this type of conclusion is superficial and can mask trends and accomplishments among students. While it is a necessary first step, it barely scratches the surface of what’s going on in our schools. In fact, relying on this type of comparative analysis — which is, in part, a consequence of federal law — can create a variety of problems. New York State provides an excellent example of what can happen if test scores are viewed exclusively through the framework imposed on the nation by the federal No Child Left Behind legislation.

According to federal law, districts are judged by the percentages of students moving from one category to another, such as basic to proficiency, or proficiency to goal, and very little else. The key for states, districts and individual schools, if they are to remain in good standing with the federal government, is to have a certain percentage of students above proficiency. How much students actually improve over the course of a year — generally called a Growth Model — has not yet replaced the federal framework, even though New York City has been piloting such a model for a few years, and with excellent results.

But back to New York State: Several years ago, state education officials noticed there were extremely large clusters of students just below the proficiency level on the reading and math exams. So they adjusted the cut downward; that is, they made it easier to reach the proficiency level. Scores, of course, improved drastically across the state and many schools were able to avoid federal sanctions, even though students were not learning more.

But the problem didn’t stop there. New York, like Connecticut and other states, does not change the types of questions on its standardized tests from year to year. This is done to facilitate valid comparisons over time. However, as districts, schools and individual teachers inevitably become familiar with certain types of questions, they are better able to prepare their students and scores go up. The tests administered last year in New York and Connecticut have not changed much in the last five years.

As information leaked out regarding New York’s testing procedures, outside experts began to examine what was going on. One examination concluded that because of the decision to lower the cuts (which, by the way, also happened in other states), New York’s proficiency level was actually lower than the basic level on the federal government’s main test, called the National Assessment of Educational Progress. And recently, a study by Harvard commissioned by New York found that the recent improvements were “illusory.” As a result, New York officials have announced they will recalibrate how they score their tests.

(Two or three years ago, thousands of New York students answered the same number or a few less questions correctly compared to the previous year’s test, but still managed to move from basic to proficient. To put all this in perspective: Twenty years ago, New York elementary students who scored below the 20th percentile nationally in math were usually held over. In 2009, according to Harvard researchers, students who scored roughly in that same percentile were counted as proficient.)

Back to Norwalk: I do not believe state officials adjusted the cuts to artificially raise scores. But nonetheless, relying on the percentages of students in different categories sheds little light on student achievement. What the district needs to do is perform detailed analyses of the raw scores, or what are sometimes referred to as scale scores, of our students.

By focusing on this type of data, which I assume is available, the BOE can examine the progress of individual students, cohorts or subgroups as they move through the system. It also provides information about the effectiveness of schools and teachers. Put in more traditional terms, the current system of scoring does not fairly distinguish between the student who goes from 69 to 70 and the student who goes from 40 to 69. The former is applauded for reaching proficiency; the latter is (statistically speaking) deemed a failure.

The BOE spends months on the operating and capital budgets. In contrast, the academic achievement of our students, as measured by standardized tests, is discussed one or two times during a school year. It would probably prove productive, not only for Board members but also for the general public, to examine these test scores with the same degree of diligence that is reserved for the budget.

12 comments

  1. For the Kids

    I believe that in the past, most of the BOE members took what Dr. Corda said as scripture and moved on. If he said everything was going well, then it must have been. In reality, some subgroups were making progress, but as whole we were not improving. We all need to spend more time reviewing the data and translating into language the parents can understand as well.

  2. Braemar

    While knowing that each grade’s cmt test is not exactly the same in difficulty, you can look at one class cohort going from third to fourth to fifth over several and so forth to see if they continue at the same level, or move up or down– for students who remained constant.

    BUT, do not forget that if you make changes in curriculum or practice, change programs, then you have years to await the sign of improvement. Grade 8 will not suddenly improve because you instituted a new program in K-3. CAPT changes will take at least 4 years.

    I do not know your level movement in and out of town, schools.
    How many students start K and continue to stay in that school and take the grade three and such. How much responsibility can there be if you have many students in classes for only a year or two of pre-K through 8.

    So what are you in control of?
    Certainly not if student do homework, or pay attention or try hard on testing.
    You control what you can and hope the culture in the schools (staff, students and family) embrace the idea of working for student success on these test (if they want to!).

  3. Bruce Kimmel

    I mentioned in my column that New York educational officials announced, after much criticism, that they would recalibrate how state tests are scored. Well, they did. See the front page of today’s New York Times.

    Scores took a nosedive across the state. One example of what happened: Previously, on the fourth grade math test, 37 out of 70 points was deemed proficient. This year, after the recalibration, 51 out of 70 points was deemed proficient.

    Also, New York City now knows that an additional 8,500 students should have gone to summer school.

  4. Braemar

    So 53% was proficient in NY for their equivalent to grade 4 math CMT. Now that is changed to 73%.

    Proficient has been a reporting problem for many reasons.

    In CT, although Proficient and Goal are given to ‘media’, they focus reporting on GOAL results. That is about 88- 89% in most grades. (Just how many students do we realistically expect to be in that range? Oh sure, 100% by 2014. )

    Proficient has not been treated as acceptable although if was not ‘failing’, but more like a C- to B+.
    In most grade levels Proficient has been approximately between 68% to 88%.
    Since different math area questions are much more difficult and statisticians bounce the numbers it is difficult to be certain. Each year a different test in the generation is given and so no year is a repeat. Each of the four CMT generations has different versions for each year.

    Connecticut’s tests have questions that must be responded to in writing. Yes, even the math test. This is not all multiple choice roulette. We do have one of the most difficult tests of the 50 states. And then our results are judged on par with other states. Always a problem.

    Should we make our CT tests easier to match other states more closely? Connecticut education people have always said, “No.”

    Your ‘Man on the Street’ in Connecticut has little knowledge of the details. He only wants top results.

  5. anon

    Braemer-Proficient on the CMT or CAPT is not acceptable. Goal is acceptable. I don’t know where you got your C- to B+ for proficient information. Proficient is NOT grade level, it is below average. Why would be want any of our kids to be BELOW grade level and then think that’s OK?

  6. Braemar

    I got it at the state dept of ed.
    Goal is like getting an A.
    If you earn proficient level scores on the grade level you are proficient. You are not below grade level.

    We do want students to go for GOAL.
    But, do not sell proficient short.

    Have someone at your school district show you generation three (not current) CMT and CAPT tests. try a few questions even at grade 6 level CMT.

    How would all students, no matter the disability or language understanding, achieve GOAL? That is not realistic. The Federal Govt’s annual yearly progress is based on having improvement in proficiency level Connecticut is choosing to use GOAL.

  7. Bruce Kimmel

    I do not believe Connecticut’s tests are still considered among the most difficult in the nation. The disparity in the scores on our state tests and our scores on the National Assessment of Educational Progress indicate we are not up to snuff when it comes to difficulty.

    I read a month or so ago that 31 states in the nation have set their proficiency cut below the NAEP’s basic level. The story, which I believe was in the NY Times, mentioned Massachusetts as one of the few states that have held the line on difficulty. I assumed from the article that Conn. adjusted its cuts. But I could be wrong.

    Many if not most states today have open ended questions that require lots of writing on all their tests, including Math. New York has been doing that for years.

  8. Braemar

    I understand that other states also have open ended or essay questions as well.

    Massachusetts has top scores in many areas. Some states report attendance/tardiness as one of the three AYP measures.

    I’ll tell you that teachers from Massachusetts coming into the building where I was teaching were surprised at how much harder our math test was. The degree of detail and multiple parts to include in written answers even at grade three was the surprise.

    I agree that we can not be complacent skating on an old perception that Connecticut is a great state for education. We must keep our noses and the noses of students to the grindstone.

    Connecticut must work at developing the appreciation for education in every home. A culture of valuing education and expecting that students will put forth a best effort does wonders for learning.

    Students who are not in the room or building to learn raise havoc and intimidate those who are. This kind of situation derails learning in an insidious and dangerous way.

    Being certain that schools are great places for learning is paramount to test scores. Working in various schools is like working in different planets. Giving studnets best practice to help them learn to read with comprehension and understand the logic of mathematics is a top priority.

    Let’s not forget that being a student is a job and it is work. It is the student who must do that work.

  9. D.O.E. is paying attention

    A big thank you, from all of us working hard to effect positive progressive education reform, to the D.O.E. Inspector General for acknowledging the work being done here in Norwalk.

    The D.O.E. awarded Sgt. Art Weisgerber and Detective David Orr of N.P.D. for thier investigative work of the mismangament and incompetency at Norwalk Public Schools.

    The parents deserve acknowledgement also for their inexhaustable and continuiung efforts to push and push hard for accounbtability and professionlisim from our education system and advocate for the ones with no voice, no seat at the table.

    The strides we have made are truely amazing but with each step forward a deeper understanding illuminates the challenges
    we atill face.

    Long way to go, no time to get there.

  10. That's oversight?

    Can never quite figure out why the apologists for those who think that grossly inconsistent test scores among schools doesn’t reflect poorly on the CENTRAL ADMINISTRATION.
    The high schools did poorly. This is surely due to the lack of leadership and guidance to the 2 principals (and a sign that the principals at NHS and BMHS just don’t seem to get that they need to lead their staffs).
    Don’t get me started on the middle schools- one middle school does great and one does poorly. Yet, the PRMS Principal keeps getting a free ride for her lack of guidance to her staff. Once again no oversight from central office– no true work by the person in charge of these principals.

  11. Aunt Bertha

    Again it comes back to no assessment plan for the Principals if you want to go there Mr. Oversight. No real assessment for department chairs other than they have seniority so those making the calls on books and strategies may not be the ones who are most knowledgeable. If schools want to make a positive change they should revamp the departments and get the best teachers involved in the testing strategies. All teachers should be teaching all levels and should not capsulate themselves in the high end classes, (Honors, AP). If they are that talented they should have some lower end classes to build students up and motivate their fellow teachers. Oh, I also think that just a little thank you for a job well done when it is in order can go a very long way. There is a lot of good out here in the schools that seems to go unnoticed.