Why Do Two Good Polls Get Different Results?

Martin West and Paul E. Peterson
Year of publication: 
August 31, 2015

Two major public opinion polls have just been released. First, Education Next (EdNext) released its ninth annual survey of over 4,083 respondents, which is administered by Knowledge Networks. (Along with Michael Henderson, we are responsible for the design and analysis of this survey.) Shortly thereafter, Phi Delta Kappan (PDK) released its own survey of 3,499 respondents, which is administered by Gallup.

The two surveys are complementary, because they ask about different topics. But when they ask about the same topic, they sometimes get different results even when asking questions that appear to be quite similar. The curious reader is bound to wonder: Who is right? Our answer: Both are correct, but to understand the findings, you must pay close attention to how the questions were asked.

The issue is not the sampling techniques that are used. If questions are worded the same way, PDK and EdNext get almost identical results. For example, PDK finds that 21% of Americans think the nation’s schools deserve either an A or a B, while EdNext finds that 23% do. PDK reports that 51% of Americans give an A or a B to schools in their community, while EdNext finds that 52% do. Statistically speaking, the results are identical.

Yet PDK’s results differ from EdNext’s on several topics. These differences almost certainly arise from differences in the way the question and response options are worded. Let’s take a look at a few specifics.