Originally published on February 13, 2020
Happy Thursday!
Okay, when I last left you, I shared all of the front-end work that I had to do in order to be able to write the test questions. This week I’ll share with you what happened when I was writing test questions for three days.
January 22, 2020
By the time we got there, everyone had been assigned a grade level and content area. I was assigned 6th grade ELA. We all were together for the initial welcome and background information session. We all had to sign two different confidentiality agreements—one for Questar and one for NYSED. Then we went to our breakout rooms.
In my breakout room, we were assigned a laptop computer and given our passages and a list of questions we had to write for the various standards. Everyone was assigned sixteen total questions they had to write in those three days. Let me say that again. We were given three days to write sixteen questions. Does that sound crazy to you? It probably would have to me too had I not gone through it. Though most of us finished a little early on the third day (but we were not released early), it still took about two and a half days to write just sixteen questions. Why?
1. First you need to read the passage and understand what it’s saying.
2. Then you need to “unpack” the passage. Unpacking it means that you are looking at it to see what parts might be applicable for specific standards. During this step, you are brainstorming different prompts potentially. Here’s the good news. They have a very helpful document that deconstructs the standards and even gives samples of different prompt stems for each type of question (multiple choice, 2-point constructed response, and 4-point extended response). Here’s the bad news. They don’t give you sample answer stems so you have to write those yourself.
Side bar. Please answer this multiple choice question:
Which type of test question is the hardest to write?
a. Multiple choice
b. Extended response
c. Constructed response
d. Only crazy people write test questions
You might think it’s easier to write multiple choice questions than constructed or extended response questions but you would be gravely wrong. Though you have to also write sample responses to the writing questions, it’s not hard for an adult to write a response that’s at an 8th grade level or lower. It is, however, really difficult to come up with multiple choice answers that are wrong but not so wrong that they would be laughable (see choice D above).
3. Okay, now you’ve unpacked the passage and looked at possible stems, etc.. Next you get to dive in and write the actual question, come up with the right answer and some good bad answers, and then write the rationale as to why a student would select the bad answers and also why the student should select the right answer. Rationales cannot be something like, “Because this is the wrong answer” or “The student may select this answer because they didn’t clearly read” or “This is the right answer because that’s what the word means in the question.” You have to write things like, “The student may select this answer because they confused the narrator with the character in the first paragraph” or “The student may select this answer because the story does mention the red wheelbarrow in the beginning but at this point in the story, the focus in on the red pick-up truck.”
By the way, you can’t use words like “can’t” in the questions. Contractions are not allowed unless you are quoting the text and the text has the contraction in it. You should say, “story” if it’s a story, not use the term “passage.” You cannot have answer choices that are only one word. You cannot say “Line 2” as an answer choice, rather you should actually write out line two in quotation marks and then write “Line 2” in parentheses after the quote. You can say things like, “Which is the best example of…[whatever]” but you cannot say, “Which of these is not an example of…[whatever]” These are just a few of the restrictions that are needed when writing the questions.
You have to determine what level (1, 2, 3 or 4) the question is written to. In other words, since the students receive a score of 1, 2, 3, or 4 for the test, there needs to be questions that are designed at each level. The Level 4 questions are designed to be answered correctly by students who would score a Level 4. The Level 1 questions should be answered correctly by all students.
You also have to be careful about what words you use so that the vocabulary you are using in your questions and choices are grade-level appropriate. You are able to use a word that is up to two grade levels above the test’s intended grade level if the question is designed to be at a Level 4 (to do this we used the attached document which I’d never seen before—start out on page 47 to see words alphabetically leveled by grade).
4. There are times throughout this process that you can confer with your partner, or the person who read your passage as their back-up so they know what you’re doing, but they are writing questions for a different passage. Though this person is nice to have, they’re in the same boat as you and this is just as tricky for them.
5. You have to enter the questions, choices, and rationales or sample answers into a secure system. This is because the questions will be reviewed and edited many times over before they end up on a field test or, more likely, on the cutting-room floor.
So here are some of the things I learned from doing this…
1. NYS certified teachers are truly involved in the process of writing the NYS 3-8 Assessments all along the way. Really. I’m not sure if it was always so teacher-involved, but it is now. The passages that we were given to create the questions had been approved already by a team of NYS teachers who went through some rigorous training that was probably just as intense as the training I went through. A new set of NYS teachers will review the questions that were generated by this first-round team. Etc. In fact, before the field or actual Assessments go out, they have teams of NYS teachers who review them one more time. It’s really a very labor-intensive process that is very structured and secure.
2. Though I have received training in creating tests, I have never created tests with such thoughtfulness as I saw occurring during this process. The amount of effort and energy going into creating truly valid and reliable assessments that truly measure the standards made me reflect on how difficult it would be to truly do this at a local level.
3. All is not lost at a local level. Though this level of drafting and revision is likely never going to happen locally, the attention to the standards and writing questions that are aligned to the standards could.
4. We should be focusing A LOT more on the standards than we probably are now. Even after three extreme days, I would not say that I am fluent enough in the standards as I would like to be. It really take dedicated effort and focus to be able to say that the difference between ELA standard 3 versus standard 5 is X because they are very close. Also standard R4 asks about the impact of the vocabulary’s meaning to the overall text whereas L4 asks about just the meaning of the vocabulary independent of the meaning of the overall text.
5. Working with a partner to do this work makes the work easier and the outcome better.
6. Don’t put too much pressure on yourself. Try. Fail. Learn. Apply learning. Repeat. This is the process. You don’t grow if you don’t try.
~Heather