The Value of Living By Numbers and Not Opinions With a Special Needs Child
“I feel…” “I think…” “In my opinion…” Listen closely at your next Planning and Placement Team (PPT) meeting and make note of the number of times these phrases are used throughout the discussion.
Now, consider the number of graphs presented. If these two figures don’t match, subjectivity may be inadvertently guiding decision-making. In navigating the PPT process, parents should strive to adopt a certain level of trust in the expertise of school-based teams while also working together to create a system that measures change and progress objectively, so student progress is rooted in hard evidence.
While using opinions rooted in objective evidence to direct treatment decisions can be helpful, it is important to understand that opinions are merely verbal expressions of preferences. The formal definition of an opinion is, “a view or judgment formed about something, not necessarily based on fact or knowledge (Opinion, n.d.).” Therefore, opinions about student performance can be biased and, as such, inherently inaccurate. All people, despite their best intentions, develop generalized opinions based on limited perspectives and preconceived judgments.
Take, for example, the following statement, “I have tried every strategy out there and explained the consequences to Little Johnny repeatedly, but he just does not get it and continues to be aggressive. I strongly feel that this is not the right class for Little Johnny.” It is important to pose the following questions when considering the above statement: What does every strategy mean? Was the plan implemented consistently, and how was fidelity ensured? Was data collected on the targeted behavior for decrease and the desired replacement behavior for increase? If the answer is no to any of these questions, then changing the intervention or placement solely based on an opinion could be detrimental.
In addition, when we rely upon opinion to drive the IEP process, we may be working off of information that is not truly representative of the child’s performance and needs. Consider a scenario in which informally observing on-task behavior is added to the long list of educator responsibilities. Can we demonstrate with certainty that a child is on-task 80 percent of the time without actually tracking it? Are these risks we are willing to assume when making critical decisions that greatly impact our children’s futures?
Data is the better way
So, what should we do instead? How do we improve upon commonplace practices to develop improved decision-making processes? The answer lies in data, which, in contrast to opinion, is defined as“factual information (such as measurements or statistics) used as a basis for reasoning, discussion, or calculation.” Using data as the foundation for decision making allows teams to analyze information for signs of progress and determine whether treatment modifications are required or if instruction should continue as currently implemented.
That being said, data is only useful if it is accurate and reliable in nature. Fortunately, the field of Applied Behavior Analysis has developed procedures for verifying the quality of data, thus increasing our confidence in the information we use to make decisions. Simply put, the data doesn’t lie. As a result, replacing opinions with data-based conversations empowers teams to make informed, educated decisions that are truly in the best interest of the children they serve.
Discerning Fact from Opinion
So, how do you know if a PPT discussion if fact or opinion-based? To support teams and families in better discerning whether or not a data-based decision-making process is in place, we have developed the following checklist that can be used as a discussion guide throughout a meeting or a reflective tool upon conclusion.
|Programming Decisions: Data/Opinion Checklist
You are at risk of using an opinion-based decision-making process if any of the following apply:
You recognize words to be wary of. Watch for subjective language indicating the use of an opinion. Common examples include, “we think,” “we feel,” “we believe,” “it seems like,” and “in our opinion.” These words suggest that someone may be making an assumption without supporting data.
No written documentation is provided. Progress or lack of progress is verbally referenced, but printed reports are not available to support mastery or lack thereof. The PPT meeting involves opinion-based conversation and does not review written documentation.
Documentation is provided, but it is all words and no numbers. A written summary of student progress uses keywords referencing pre- and post- instructional levels of performance, for example, “compared to baseline data” and “once we started the program,” however hard data is not provided or available when requested.
Documentation is provided that includes numbers, but no graphs are available for visual analysis. Data is available on durations, percentages, or frequencies but decisions are based on the individual numbers instead of the patterns those numbers create when graphed and visually inspected.
You are likely using a data-based decision-making process if all of the following apply:
Data analytic language is used to guide conversation. The discussion indicates that the team not only collected but also reviewed data to inform the thought process surrounding teaching strategies. Common examples include “the data shows,” “when we look at this graph,” and “here you can see.”
The team is able to explain the data collection process. Clear information is available regarding the type of data collected (e.g., rate, percentage, duration), who collected the data, when/where the data was collected, and how long it was collected for.
Numbers have been translated into line graphs. In presenting a line graph summarizing a child’s performance, teams are able to review progress over time in an efficient manner and identify patterns in behavior. You will quickly be able to see where a child started, growth achieved, challenges encountered, and present levels of performance. Graphs may include vertical phase changes lines that visually highlight extended breaks from school, changes in medication, updated teaching procedures, and other relevant adjustments. This allows for a greater understanding of the impact such changes might have had on the child’s progress.
The team provides a brief verbal or written summary of the graph. A solid analysis would include baseline levels of performance, or simply put, what could the child do before any teaching occurred? In addition, the team reflects on the trend noted in the graph following intervention; is the skill to be acquired increasing, decreasing, or stagnant? Finally, periods of variability are noted and compared to times at which performance was more stable.
Click here to find out more
Data in Disguise
To add a caveat, it is important not to mistake misleading figures that may meet some or all of criteria outlined above for truly meaningful data. After all, our decisions can only be as good as data supporting them. Here, we have identified several “data points” sometimes erroneously used to guide the PPT process without fully understanding the critical flaws that render them unreliable.
1. Results of subjective rating scales or Likert scales measure what someone writes on a survey, not a child’s actual performance. Any approach that asks users to rate their opinion on a scale ranging from strongly agree to strongly disagree based solely on memories of past experiences runs the risk of great inaccuracy. Some scales transform results into numbers, or even graphs, that represent average ratings. As a rule of thumb, ask yourself, did this tool measure my child’s behavior or someone else’s opinion? If someone’s opinion lies at the core, truly data-based decision making is not in effect.
2. A work sample provides an example of your child’s performance on one occasion. While work samples can be captivating because they are “real life” representations of your child’s skills, they do not provide all the information needed to make informed decisions and therefore beg further inquiry. Did your child do the work independently or did he require prompts? How long did it take to complete? Is this how he does the work every time?
That being said, a work sample might be transformed into an effective measure if permanent product recording, the process of saving and inspecting a work sample after it is completed, is repeatedly used to assess your child’s work. Keep in mind, data derived from analysis of a permanent product must be graphed over time in order to be useful in decision making; providing a pile a work sample equipped with permanent product recording data does not suffice.
3. Scores on subjective rubrics, in their condensed form, are often opinions transformed into numbers. For example, assigning a numeric value to represent a student’s use of excellent grammar, average grammar, or poor grammar is all based on opinion. One teacher’s understanding of excellent grammar may vary widely from another’s. Rubrics can be useful data collection tools, however, if they are designed with an objective framework. Consider a grammar rubric governed by the number of errors instead. A student might receive a higher score for having zero to one grammatical error and a lower score for having five or more. Through such a system, a rubric might represent an effective data collection tool, but careful review is required to determine whether objectivity is embraced.
4. Anecdotal summaries of teacher observations may include numbers embedded in paragraph form that can be misleading. Typically, such numbers are flawed in one of two ways. First, they may be irrelevant and communicate little information about your child’s performance. For example, consider this statement “This evaluator observed the student for 30 minutes.”
The number 30 carries little value about the behavior observed. Second, numbers may represent estimates, which are not the most accurate measures of performance. Consider this statement: “Johnny was off-task two times as often as his peers.” Unless there is data specifying the time frame to which this applies, such as two times more off task per minute versus per week, then the data has little significance. Without the supporting figures, a subjective estimate does not represent solid data.
5. The provision of one or two data points does not allow for effective decision making because you cannot be sure that such a small sample represents typical performance; it very well could have been a good day or a bad day. At a minimum, a graph should include three data points in order to allow for visual analysis, and a graph with an insufficient data set may be deceiving.
While these “data points” may supplement discussions by providing specific examples of student performance, we suggest using other forms of data in which there lies greater confidence as the primary rationale supporting team decisions.
Summing It Up
While the use of data is considered “best practice” by board-certified behavior analysts, it is critical across all domains of learning, not just behavior. Data-based decision making is relevant for academic services, occupational therapy, speech and language therapy, social skills instruction, physical therapy, and counseling too. As such, we encourage the use of this tool in a comprehensive manner across providers within the school setting.
Parents, while sharing this resource with your school-based teams, remember this may serve as a helpful self-monitoring resource for you as well. As the most active and influential advocates in our children’s lives, we have developed our own opinions, and while they are of great value, we too must remember to look back to the data for confirmation. After all, in the genius words of Sherlock Holmes, “It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.”
Opinion. (n.d.) In English Oxford Living dictionary. Retrieved from https://en.oxforddictionaries.com/definition/opinion
Meghan Cave, M.Ed., BCBA, LBA, Justyna Balzar, M.Ed., BCBA, LBA, and Keri Spielmann, M.A., BCBA, LBA are three Board Certified Behavior Analysts with unique backgrounds, who seek to use the principles of Applied Behavior Analysis to affect meaningful behavior change within the education system and among professionals responsible for advocating for students with special needs and their families. By objectively defining goals, referencing real-life examples from their practice, and workshopping solutions rooted in Applied Behavior Analysis, they seek to develop evidence-based task analyses that will unify advocates, lawyers, families, school districts, and related service providers, thus creating an evidence-based forum in which socially significant progress is achieved through collaboration, trust, and science.
This article was featured in Issue 85 – Top Strategies for Supporting your Family