The Narrative: The Controversy and the Conversation

Introduction

Issue and Questions

Through this review of literature, I will be exploring the answers to the following questions:

1. What do global standardized tests such as Programme for International Student Assessment (PISA), Progress in International Reading Literacy Study (PIRLS), and Trends in International Mathematics and Science (TIMSS) measure and who participates?
2. What are some misconceptions related to global standardized test scores, especially when countries are ranked against each other?
3.  Are schools using global standardized tests to design curriculum?

The term “globalization” is defined as “the development of an increasingly integrated global economy” (Merriam-Webster, 2012). In our globalized world, nations naturally compare successes and setbacks. There is widely held perspective that education is directly related to economy and so comparing student learning with other countries becomes not only natural, but essential to understand their “human capital” (Cavanagh, 2012). “Competitiveness in the global economy has led to competition in the educational area, since one is assumed to be based on the other” (Dall, 2011). This point of view has led to the creation of international standardized tests designed specifically to rank the level of student achievement in participating countries. The actual problem that I want to address is the high level of misunderstanding that then results from these test scores; this huge gap in understanding leads to many problems that school professionals encounter every day when interacting with students and their parents. Parents lose faith in their schools and therefore the teachers and administrators in these schools. Economists and Policymakers use these tests to create educational law that is based off of inaccurate interpretations that can ultimately negatively impact student learning.

Due to the increased dependence on the results of these test scores, policy makers begin to view an effective school system as one in a country with a high ranking on one of these tests. (Dall, 2011). The reliance upon standardized test scores has been leading to a tightening of curriculum in schools in order to compete globally. Unfortunately, the scores themselves are taken out of context which leads to changes being made without all of the information. Tests results provide information surrounding the context of the child’s education but media tends to simplify results to an overall ranking of countries (Topping, 2006).

An imprecise understanding of what is actually assessed on standardized tests exists. While the Programme for International Student Assessment (PISA), a widely used international standardized test, claims to “assesses to what extent students near the end of compulsory education have acquired some of the knowledge and skills essential for full participation in society” (OECD, 2012); research shows that these tests are no different than other standardized tests in that it assesses the ability of the student to achieve in the singular situation of survey test (Dohn, 2007). Also of note is that standardized tests assess students in a “cross-sectional” manner (single point in time) and not through a longitudinal analysis which is the preferred method of study for growth in student achievement (Schneider, 2009).

Rationale

I believe that globalization is an opportunity to grow as an international community that celebrates the diversity of each country while taking the opportunity to learn from each other. However, even in our age of information many choose to ignore important aspects of a problem, in this case it is the belief that the American education system is inferior to other national education systems, in order to reach a solution. We are so focused on a specific outcome, that many times the plan to get to that outcome is poorly thought out and implemented. “One of the most common mistakes that policymakers make in interpreting international test results is focusing on one aspect of high-performing nations’ school systems and assuming it can be replicated wholesale in American schools” (Cavanagh, 2012).

Globalization as a concept carries with it the threat of a “homogenized world where nations and cultures are neutralized” (Dall, 2011). This threat is a very real fear that manifests in our school environment through the desperate cling to the status quo and by the way that we treat students and parents of other cultures, whether positive or negative. Undeniably our schools are a testing culture through legislation and under constant pressure to enact change in the form of improved student performance on standardized tests. "Quantification now defines progress in education. If a variable cannot be measured, no progress can be evidenced” (Dall, 2011). We must ask ourselves if this is truly the best way to assess student learning.

Nature of the Controversy

                There are three major players that I will discuss regarding the controversy of globalized standardized testing. They are School Professionals, Parents, and Economists & Educational Policymakers.  While each of these groups have the best of intentions, it becomes problematic when their views are opposing and begin to work against each other. For clarity of the sometimes dry and confusing information I will be providing, from this point forward I will be organizing my outline of viewpoints, critique and discussion, and my conclusions and recommendations surrounding the research questions.

Viewpoints

1.       What do global standardized tests such as Programme for International Student Assessment (PISA), Progress in International Reading Literacy Study (PIRLS), and Trends in International Mathematics and Science (TIMSS) measure and who participates?

• Brief Description of Standardized Tests and the Organizations Behind the Tests:
• Programme for International Student Assessment (PISA)
• “The Organisation for Economic Co-operation and Development (OECD) conducts PISA every three years, emphasizing one of the subjects on a revolving basis” (Schneider, 2009)
• "Assesses to what extent students near the end of compulsory education have acquired some of the knowledge and skills essential for full participation in society” (OECD, 2012).
• “Evaluates reading, math, and science “literacy” among 15-year-olds” (Schneider, 2009).
• Since the year 2000 over 70 countries and economies have participated in PISA  (OECD, 2012).
• Progress in International Reading Literacy Study (PIRLS)
• “Administered under the auspices of the International Association for the Evaluation of Educational Achievement (IEA)” (Schneider, 2009).
• “Collects data to provide information on trends in reading literacy achievement of fourth-grade students, while providing baseline data for new countries” (PIRLS, 2012).
• “Assessment of 4^th-grade reading administered every five years” (Schneider, 2009)
• For 2011 there were 49 countries and 9 benchmarking entities, such as individual US states, participating (TIMSS & PIRLS, 2012).
• Trends in International Mathematics  and Science Study (TIMSS)
• Just as PIRLS is, this assessment is administered by IEA (Schneider, 2009).
• “A globally cooperative enterprise, TIMSS conducts comprehensive state-of the-art assessments of student achievement supported with extensive data about country, school, and classroom learning environments” (TIMSS, 2012).
• “Assessment of student achievement in 4^th- and 8^th-grade science and math and is conducted every four years” (Schneider, 2009).
• For 2011 there were 63 countries and 14 benchmarking entities, such as individual US states, participating (TIMSS & PIRLS, 2012).

2.       What are some misconceptions related to global standardized test scores, especially when countries are ranked against each other?

• School Professionals
• If over 40 countries use this test, it must be an accurate assessment of learning.
• What are we doing wrong?
• Parents
• Test scores are an easy way to see if our public schools are teaching our children; by looking at an international scale, we can see to what extent American schools are failing our children!
• It is obvious that because of these scores, we need more math, science, and reading time in our schools in order for my child to get a job when they graduate.
• Economists & Educational Policymakers
• Global standardized tests accurately compare one nation to another.
• Using these test scores, you can simply look at what the top nations are doing and “transplant” that method here to achieve similar results in student achievement.
• These test scores can be used to make decisions about how effective our schools are.
• The nation’s economy directly relates to the nation’s test scores.

3.       Are schools using global standardized tests to design curriculum?

• School Professionals
• We have to design our curriculum to reflect the test or else we will fail our students because they will have to know the exact same things as their peers in other countries.
• Parents
• Doesn’t the test actually reflect the current curriculum? 
  
• Economists & Educational Policymakers
• We should be using these tests as a guide to curriculum design in order to prepare our students for their future careers that compete internationally.
  
  

Critique and Discussion

1.       What do global standardized tests such as Programme for International Student Assessment (PISA), Progress in International Reading Literacy Study (PIRLS), and Trends in International Mathematics and Science (TIMSS) measure and who participates?

• “In PISA terms, the qualities that prepare students to meet the challenges of today’s knowledge societies and the challenges of adult life in the real world are clearly competency in reading, mathematical and scientific literacy. Consequently, effective education systems are defined by how well students are performing in the PISA surveys” (Dall, 2011). This statement clearly articulates the problem that educational systems are beginning to face internationally. No longer are people asking if these tests are a fair assessment of student knowledge, instead they begin to ask why their children have not learned enough to achieve higher on these tests.
  
• It is essential to remember that each of these tests are unique in what they assess, but by nature they only assess a moment in a student’s education; because these assessments are designed to test different academics, it is important to clarify what these tests are designed for.

o   PISA addresses (curricular differentiation) by focusing on application of skills beyond the school context, but even that strategy does not entirely solve the problem” (Koretz, 2009).

o   Surveys reading literacy of fourth-grade level students (Unknown, 2008)

o   “TIMSS is intended to follow school curricula reasonably closely” (Koretz, 2009).

2.       What are some misconceptions related to global standardized test scores, especially when countries are ranked against each other?

•   School Professionals
• If over 40 countries use this test, it must be an accurate assessment of learning.
• What are we doing wrong?
• It is important to remember with any assessment that there are human beings writing the test itself and therefore is subject to human error. The most subtle and disheartening is the cultural bias that exists with any standardized test due to the obvious lack of a standardized culture. “With every standardized test, one has to consider the possibility of a cultural bias, especially when these assessments are intended to be used globally. ‘Even when one approaches the items with an attitude of tolerant understanding of the complexities of the assessment process, however, one is dismayed at the number of test items that are culturally biased, ambiguously formulated, confusing on account of misprints, down-right erroneous or furnished with highly questionable answering keys” (Dohn, 2007).
  
•  Another symptom of a testing culture is the acceptance of the results of these tests without considering the context of student achievement level. “The data also provide rich information about children’s early literacy experiences, instruction, and reading habits outside of school, which has implications for how performance might be improved” (Topping, 2006). Comparing test results globally shows at best a very general guide of achievement, but the next inevitable conclusion of which country has a better education system is truly speculative however, is often accepted as fact. More accurate would be comparing the results between countries with similar linguistic characteristics, average socioeconomic status, or levels of immigration. It is interesting that even though test score and student context information from PISA, PIRLS, and TIMSS is widely available, the only information ever heard is the over simplified comparative score results. There is not usually a mention of where you can view the scores for yourself.
• Go to PISA and the PIRLS/ TIMSS websites to view these scores and context reports for yourself!
• Parents
• Test scores are an easy way to see if our public schools are teaching our children; by looking at an international scale, we can see to what extent American schools are failing our children!
• It is obvious that because of these scores, we need more math, science, and reading time in our schools in order for my child to get a job when they graduate.
• Western culture seeks an ultimate answer to any question posed. Are our schools successful, yes or no? Well it’s not always that easy. Successful in what way? Are they meeting the social needs of the child? Are they teaching creative problem solving? Are teachers taking the time to ensure that students comprehend information instead of overwhelming them with facts that will only apply to a standardized test? “International comparisons can easily be simplified, especially when reported in popular media” (Topping, 2006).
•  An easy and even logical assumption is that a low score on any test means that students are not learning all that they need to know and therefore, will not have the skills to compete later in life. “PISA gives a relatively reliable assessment of ‘knowledge and skills for PISA’, that is, of how well students exercise competence within the PISA focus areas in one—and only one—‘real life’ situation, the PISA test situation” (Dohn, 2007). This concept is not exclusive to PISA, standardized tests sample student knowledge of facts at that moment. The data produced by standardized tests is what is known as “cross-sectional” which means that it measures a single point in time, as opposed to a longitudinal analysis which allows comparisons of student grown and achievement over time (Schneider, 2009). Just because a child does not succeed on a standardized test in no way predicts their academic and/ or career success.
• Economists & Educational Policymakers
• Global standardized tests accurately compare one nation to another.
• Using these test scores, you can simply look at what the top nations are doing and “transplant” that method here to achieve similar results in student achievement.
• These test scores can be used to make decisions about how effective our schools are.
• The nation’s economy directly relates to the nation’s test scores.
• “Comparisons with an ‘international average,’ however, are nearly meaningless. An average is useful if it represents a clear comparison group… however, in the case of international comparisons, the ‘international mean’ reflects the collection of countries that happened to participate in a given assessment in a given year” (Koretz, 2009). This means the US can move up or down on the master list of educational achievement simply depending on who participates that year!
• “On average, close to 60 percent of the difference in reading performance between schools is accounted for by the socioeconomic status of the students attending the schools” (Rotberg, 2011). What is so often overlooked is the student, school, and community contextual information. Not only does this clarify scores of each nation, but it also presents an issue when attempting to compare scores internationally.  Unfortunately, many times the easy answer is the generalization and so that is what is accepted. “PISA reports often list the known problems with the data, but then the policy advice flows as though those problems didn’t exist” (Schneider, 2009).
• “You can’t take one element or one variable out of a system and expect it to work” (Cavanagh, 2012). It is almost laughable at times when you hear discussions of what America “should” be doing since a high achieving country such as Finland or South Korea is doing it. Those countries have created a system that works for them, not one that will work for the United States. We can’t expect to glean the “reward” of high test scores without putting in the time and effort towards designing classrooms that best meet student needs “There is no reason to believe that if one dropped students from the United States into schools in Singapore, their performance would match that of Singaporean students, or vice versa, even if one adjusted for the limited range of other factors about which data were collected in these studies” (Koretz, 2009).
• The identification of the causal relationship between education and the economy is not entirely accurate. The data does not “identify causal relationships” (Topping, 2006). There is definitely a correlation between the two, but one does not immediately impact the other. “Many of the countries that ranked high on test scores rank lower than the United States on competitiveness—for example South Korea, No. 22, and Finland, No.7” (Rotberg, 2011). While it is comforting to believe that the solution to a problem is so simple, our education system is not failing students in this way. There is a need for reform, just not economic ones

3.       Are schools using global standardized tests to design curriculum?

• School Professionals
• We have to design our curriculum to reflect the test or else we will fail our students because they will have to know the exact same things as their peers in other countries.
• “Educational value is today measure by what competencies can benefit the individual and the society in the global competition” (Dall, 2011). While it is important to consider global competition, it is also important to teach students life skills that will benefit everyone in the long run. These tests are better at assessing current knowledge than guiding education systems toward reform (Schneider, 2009).
  
  
• Parents
• Doesn’t the test actually reflect the current curriculum? 
  
• These tests are designed to appeal to a global market of school systems and so it is not possible to reflect each curriculum being assessed.
  
  
• Economists & Educational Policymakers
• We should be using these tests as a guide to curriculum design in order to prepare our students for their future careers that compete internationally.
• While utilizing benchmarking for student knowledge is important, teaching to the test does not benefit anyone. “Thus quantification now defines progress in education… consequently, effective education systems are defined by how well students are performing in the PISA surveys“ (Dall, 2011). Do we really want an education system literally defined by how high students have achieved on these tests?
• “[Arne Duncan] praised U.S. officials work on the common standards, while underscoring that the federal government would not prescribe a national curriculum as part of its support for the venture” (Sawchuk, 2011). But through the culture of fear that comes with accountability, hasn’t that already happened?
  

Conclusions

1.       What do global standardized tests such as Programme for International Student Assessment (PISA), Progress in International Reading Literacy Study (PIRLS), and Trends in International Mathematics and Science (TIMSS) measure and who participates?

•  As with any news source or research paper it is essential to consider multiple sources of information.
  
• Because the findings of the two tests are sometimes inconsistent…” it is essential that we consider data from both sources (OECD and IEA) (Koretz, 2009). More importantly, what we usually see is an individual’s or organization’s interpretation of these score results. Seek them out on your own to better form an opinion of student achievement.

2.       What are some misconceptions related to global standardized test scores, especially when countries are ranked against each other? 

• There are so many misconceptions about what actually happens in education. “Newspapers never fail to highlight their disappointing results” (Koretz, 2009) and educational professionals continue to fail shouting from the rooftops about all of the amazing achievements in education we see daily! “International assessments measure very broad domains of achievement” (Koretz, 2009). Because the information is so broad, it can miss high levels of achievement in a non-assessed area such as arts or physical abilities in athletics. We also need to consider what is being assessed and what is most important about education. Is it more important that kids can spew facts or that they can become valued and contributing members of society? It’s not necessarily an either or question unless we make it so. Also, I cannot stress enough to take into account the context of student achievement. “On average, close to 60 percent of the difference in reading performance between schools is accounted for by the socioeconomic status of the students attending the schools” (Rotberg, 2011)

3.       Are schools using global standardized tests to design curriculum?

• “No one test, however well designed, should be treated as a ‘gold standard’” (Koretz, 2009). The short answer to this is yes, but it is not always intentional. Many times the pressure to succeed on these assessments leads to a compromise in curriculum goals and instructional time.
• “Comparisons at the high school level are problematic is curricular differentiation…because having useful comparisons across countries in a broad subject area, such as mathematics, requires agreement about the goals of mathematics instruction” (Koretz, 2009). Can we really agree globally what the goal of a mathematics curriculum should be for the entire world?

Recommendations

• When considering these test scores: “First, one should ignore small differences among countries, as they are too likely to be the result of sampling or unimportant characteristics of the tests. Second, one should ignore the “international average” and select other nations that provide informative comparisons, such as those that are similar or that are particularly high-achieving. Third, when possible, one should consider data from more than one source” (Koretz, 2009).
• Teach, teach, teach! Don’t just teach your students, teach their parents, teach the community, teach the nation! Tell them about all of the great things our schools provide while working on improving those areas that need reform. Correct any misconceptions you hear and be a resource when you can be.
  
  
• Bring the world into your classroom! I have provided a great list of easy ways to bring international education to the feet of our students. We also have an often underutilized resource in parents and community members who would love to come into the classroom to talk about their culture.

References

Merriam-Webster. (2012). Retrieved May 20, 2012, from Dictionary: http://www.merriam-webster.com/dictionary/globalization

OECD. (2012, May 16). Retrieved May 20, 2012, from OECD Programme for International Student Assessment: http://www.pisa.oecd.org/pages/0,2987,en_32252351_32235731_1_1_1_1_1,00.html

PIRLS. (2012). Retrieved May 28, 2012, from TIMSS & PIRLS: http://timssandpirls.bc.edu/pirls2011/index.html

TIMSS. (2012). Retrieved May 28, 2012, from TIMMS & PIRLS: http://timssandpirls.bc.edu/timss2011/index.html

TIMSS & PIRLS. (2012). Retrieved May 28, 2011, from IEA International Study Center: http://timssandpirls.bc.edu/

Cavanagh, S. (2012, January 12). Complex policy options abound amid international comparisons. Education Week, 6-10.

Dall, A. (2011). Is PISA counter-productive to building successful educational systems? Social Alternatives, 30(4), 10-14.

Dohn, N. B. (2007). Knowledge and skills for PISA -- assessing the assessment. Journal of Philosophy of Education, 41(1), 1-15.

Koretz, D. (2009). How do american students measure up? Making sense of international comparisons. The Future of Children, 19(1), 37-51.

Rotberg, I. C. (2011). International test scores, irrelevant policies. Education Week, 32-26.

Sawchuk, S. (2011). U.S. reforms out of sync with top-performing nations' education path. Education Week.

Schneider, M. (2009). The international PISA test. A risky investment for states. Education Next(Fall), 69-74.

Topping, K. (2006, March). PISA/ PIRLS data on reading achievement: transfer into international policy and practice. International Voices, 59(6), 588-590.

Unknown. (2008, February/March). PISA, PIRLS spotlight global trends. Reading Today, 25(4), 3-4.