The NYC School Survey: Changes over time leading to possibly skewed results and how the survey could be improved

The following was written by the new research associate for Class Size Matters, Emily Carrazana. Take a look!

Since 2007, the NYC Department of Education has issued something called the “Learning Environment Survey,” administered to students, parents, and teachers to collect their views about each school's quality and the system as a whole. Every year between 2007 and 2014, when parents were presented with ten choices, class size came out as the top priority of 21% to 24% of parents when they were asked “Which of the following improvements would you most like your school to make (Choose ONE)?”  

In 2015, this question related to parents’ top priority was completely left out of the survey, possibly because then-Chancellor Farina seemed uninterested in listening to parents in general and especially on the need to lower class size. In fact, her peculiarly unique view was that class sizes in NYC schools were too small.   

  After significant pushback from Class Size Matters and others, this question was resurrected the following year, but with a few other "tweaks". While class size was still offered as one of the choices, options relating to more or less state test preparation were removed, and “a safer school environment” option was added. The phrasing on the choice of more enrichment programs was expanded from “Stronger enrichment programs” to “Stronger enrichment programs (e.g. afterschool programs, clubs, teams)”. 

This choice now became the only one that provided any additional descriptive information, including specific examples of what might qualify as an enrichment program. Before that point, the percent of parents choosing “enrichment programs” as their top priority had significantly fallen, from 19% in 2007 to 12% in 2014.  Unsurprisingly, in 2016 for the first time, the number of parents who chose enrichment exceeded those who chose class size –with enrichment the top choices of 23% to 27% of parents from that year onward, compared to class size at 20% to 21%.

(Take note of the sharp uptick in enrichment programs in year 2016, the same year that the wording changed)

This is not to downplay the importance that schools should place on improving their students’ access to extracurricular and afterschool programs, especially to parents, many of whom work a full day; but to highlight the way in which the question seems to have been altered that might have skewed the results.

For instance, it is reasonable to assume that had the option for smaller classes been elaborated on in a similar fashion, for instance - "Smaller class sizes (e.g. more small group instruction, one-on-one feedback from teachers,  resulting in a more cohesive class culture and fewer disruptions, all of which have been proven to result from class size reduction), more parents might have chosen this option as their highest priority. Admittedly, expanding each option into a long-winded description is not optimal or efficient for a survey design, and might have led to even more slanted results. Instead, all options should be presented in a manner that does not persuade respondents to distort their own priorities. 

Another change that may have resulted in smaller class sizes being knocked down from first place citywide was the rapid expansion of the preK program starting in the 2014-2015 school year, and the inclusion of the responses of thousands more parents whose children attended preK that year and the following one.  Moreover, only in 2015 did the DOE begin to include the responses in the survey results of parents at hundreds of private preK centers operated by community based organizations. 

PreK classes  are strictly limited by state law to class sizes of 18-20 students per class, depending on whether there are one or two teachers plus an aide; so it is not surprising that preK parents would not choose class size reduction as their top priority. The survey data also includes the responses from hundreds of thousands of parents of students with special needs who are in self-contained classes and/or attend District 75 programs, whose class sizes are limited to only 8 to 15 students.  

It is therefore understandable to see why these parents might not view lowering the size of their children’s classes as important as those whose children are crammed into classes of 25, 30 students or more; conditions which are quite common in NYC public schools.  To be more informative, it would be preferable to disaggregate and report separately the priorities of parents of students in District 75 programs and especially those enrolled in preK classes.

Information on how DOE officials analyze or use these survey results, if at all, is virtually untraceable. Any follow-up conducted by school administrators seem to be done on an entirely voluntary basis. While the DOE provides a few pre-populated worksheets and  PowerPoint decks with suggestions on how the overall results can be presented to and discussed by parents, nowhere in these documents can you find any mention that their priorities as reflected by their responses to this key question should be addressed. 

This is a disappointing omission, especially for an administration that claims to base their policies on a “Framework of Great Schools” centered on mutual respect and strong family ties,  and a Chancellor who often insists he wants to focus on “parent empowerment”.

If the Chancellor really wants this survey to be more useful, the actual structure of the question might also be changed. Parents are asked to choose from a list of nine options, all of importance, and select only one that they would most like to see improved in their child's school.  Instead of asking them to do the mental backflips needed to weigh the pros and cons of a safer school environment vs. more arts programs or hands-on learning, the question might allow them to assign corresponding priority levels to each area. For instance, 5 = Highest Priority, 4 = High Priority, 3 = Priority, 2 = Somewhat of a Priority, 1 = 1 Low Priority. This would be advantageous for administrators to inform decision-making at their schools.

How the question left out of the DOE parent survey was the most interesting of all

More on the results of the parent survey and the question left out in Schoolbook and the NY Daily News. 

When the results of the DOE's Learning Environment Survey were released yesterday,  the administration took credit for high rates of parent satisfaction, though reporters pointed out the satisfaction rates were the same for the last three years and this year, only about half of all parents bothered to respond to the survey.
 
Yet what was most striking was the question that was left out.  This question had been asked of parents since the survey was first given:  Which of the following improvements would you MOST like your school to make?  Every year since 2007,  which was the first year the survey was given, smaller classes have been the top priority of parents by far, among ten choices.

The question was included in the survey because of the insistence of parent members of the focus groups organized by Jim Liebman, then head of the Accountability office, to give feedback on the survey's design back in 2006.    This year, the DOE revised the survey without holding any focus groups at all.

If you look at the 2014 data, smaller classes were even more clearly a top priority of NYC parents;  as it was either their #1 or #2 priority in all but two out of 32 districts:

In 2007, parents actually wanted separate questions on each of these issues, especially as regards class size and testing, but instead DOE decided to group them together in this way.  In any event, the results have been an important source of data that could have been used if the Chancellor or the Mayor actually wanted to be responsive to parent concerns.   Unfortunately, since 2007 class sizes have increased instead.

In 2007, Bloomberg tried to obscure the responses to this question during his press conference by grouping together many other options in a new category, called "more or better programs."  I and others, including Patrick Sullivan, called him out on this naked attempt to obscure that class size reduction had been first.  Here is the NY Times article from the time: 

Mr. Bloomberg lumped together several categories to note that for 45 percent of parents, “more or better programs,” not class size, was the top priority, despite a fierce lobbying campaign for smaller classes by some parents groups and the teachers’ union.
“When somebody stands up and says, ‘I speak for all parents and we want smaller class sizes,’ that’s just not true,” Mr. Bloomberg said.
Leonie Haimson of Class Size Matters, an advocacy group at which the mayor was clearly taking aim, said the survey pointed to the opposite conclusion. “It’s a transparent attempt to minimize the importance of an issue that is staring everybody in the face as the top priority of parents.”

In 2012, the DOE stopped including the responses to this question in their citywide summaries, though the data was still easy to find in their reports.  Now for the DOE to stop asking this question at all, after eight years, especially for an administration that claims to care about parent input,,  is hugely disappointing.  Here are quotes from yesterday's DOE press release:

Schools Chancellor Carmen Fariña: "... I look forward to working with school communities as they use these results to identify areas of improvement, and develop the right supports and solutions to address them. The more we listen to the feedback of students, parents, and teachers, the better our schools are going to be."

But how can they listen to parent feedback if they refuse to ask them what changes need to be made?  And this quote:

“The retooled survey focuses on dimensions of the school community that past research has found to be critical for improving student outcomes. The goal of this work is to equip schools with actionable information they can use to support teachers and serve students more effectively….” said James Kemple, Executive Director, Research Alliance for New York City Schools at New York University.

I don’t know what is any more “actionable information” or research-based than class size reduction, one of the few reforms proven to help kids succeed, unless the administration is determined not to take action to lower class size, no matter what parents want or the evidence shows .   In fact, class size reduction is one of only a handful of reforms proven to work through rigorous evidence, according to the Institute of Education Sciences.

Of course, the DOE can keep on asking silly questions instead like the following ones, included in this year's parent survey, which are neither “actionable” nor ones that most parents would probably be able to answer:

Sadly, the omission  of the critical question of what changes parents would like to see in their children's schools is yet another sign that though the administration's rhetoric focuses on the importance of parents having "trust" in school leadership, it is difficult to trust their leadership  when they don't  even ask the right questions.

How the DOE ignoring class size puts kids at risk, according to education professionals and parents

Check out the oped in Schoolbook by Jacqueline Shannon and Mark Lauterbach, professors of education, urging the Chancellor and the Mayor to lower class size in the city's schools:  De Blasio Must put Reducing Class Sizes at Top of His Agenda.  As the authors point out, the trend towards larger class sizes every year for the past six will underminethe success of the Mayor's other education initiatives, including special education inclusion, expanding preK and creating community schools.  

They also show how the union contractual limits have not altered in forty years, despite the far more extensive research in recent years, including  studies summarized in this NEPC report, showing the multiple benefits of smaller classes in terms of academic and life outcomes: 

“Students who were originally assigned to small classes did better than their school-mates who were assigned to regular-sized classes across a variety of outcomes, including juvenile criminal behavior, teen pregnancy, high school graduation, college enrollment and completion, quality of college attended, savings behavior, marriage rates, residential location and homeownership.”

The Schoolbook oped also links to a letter to the Chancellor and the Mayor, signed by 73 professors of education, reflecting the strong consensus among experts in the field that the issue of class size is fundamental to the opportunity to succeed, particularly for at-risk students, "including those in children of color, those in poverty, English language learners, and students with special needs."  

The reality is that even in the city's most struggling schools, like Boys and Girls High School, where staff is being asked to re-apply to keep their positions, class sizes remain much too large -- with many classes at the union maximum of 34 students per class, according to the DOE's own class size reports from last fall. 

The chart at the right reveals that class sizes at the Boys and Girls were inordinately high, particularly in 9th grade classes, where it's most important to keep students on track.  Class sizes were lower in the upper grades, presumably because many of the school's students didn't get that far.  Many of the special ed classes even violated the 12/1/1  limits, according to the DOE reports (see the last column for the size of the largest classes in each of these categories.)

Moreover, lowering class size remains the top priority of parents to improve their schools, according to the DOE's own Learning Environment Survey.   And yet, there are only two parents on the committee to decide on staffing for Boys and Girls and Automotive HS, one to be chosen by the teachers union and the other by the principals union rather than other parents; and NO parent members on the committees to decide on the improvement strategies.

The DOE claims to be listening to parents more than the past administration; I'm not sure what that means if they continue to ignore their views on what their schools need to succeed.

2011 Survey results: Teachers are dissatisfied with the chancellor, and parents want smaller classes more than ever!!

The 2011 DOE learning environment surveys are out; here is where you can find the results for your child's school; here is the citywide summary; here are some charts. 

Two findings that immediately stood out for me:

Teachers really didn't like Cathy Black and/or Walcott; above is a chart showing their feelings about the chancellor, when the survey closed on April 15.  (Strangely, the DOE claims on p. 11 here that Walcott's tenure started on April 18; but actually he started April 7. ).  The majority of teachers were unsatisfied or very unsatisfied with his/her performance. 

Parents were happier, with more than 50% satisfied or very satisfied with the chancellor -- though less so than last year, when over 60% were satisfied.

Even more than ever, though, parents are dissatisfied with their children's class sizes; as smaller class sizes continue to be their top preference for what improvement they'd like to see in their children's schools, as it has been  every single year that the survey has been given.

And yet class sizes have gone up sharply every year of the last three, and are now larger in the early grades than they have been in more than a decade.

Which really shows the lie to the administration's supposed interest in catering to parent "choices", no?  And really undermines the apparent purpose of this survey.