Finding the joy in AsTTle

writing
It might not be as bad as you think. ( Maybe)

Why are you here? What can you expect to get out of it?
A better understanding of the asTTle writing tool therefore:
More accurate marking
Less remarking
More accuracy for our learners
More

accurate support for you OTJs

It can be genuinely useful to you and your learners.

Before we start: Some space to get it off your chest

Put your wonderings on here I will do my best to answer them
or point you in the right direction. I will do this at the
start of tomorrows session.

The AsTTle writing tool: What is it

The only tool that NZ has to compare achievement between
schools
Complex and rigorously tested
Accurate
Able to be used formatively or summatively. It was designed
to be formative and used with AFL practice.

What is it not?
Something you can mess with.
A complete picture of everything that learner can do.
Something you can argue with: It works how it works. If you
want accurate results you have to use the tool accurately.
Your OTJ.

How can you use the writing tool ?

As formative assessment that works in conjunction with AFL
practice to identify target areas for learning design.
As a regular part of the learning programme to help learners
to develop effective assessment strategies.
As a regular part of the learning programme to help develop
assessment capable learners
As a one off summative test to give a curriculum level.

Some things you might not know about e-asTTle

Recalibrated in 2013 -For accuracy. The change was ONLY in
the calibration of scores to levels.
This was applied to all previous scores
Ongoing evaluation and improvement process since the early
2000
Carefully selected prompts that are extensively trialed,
R scores have nothing at all to do with curriculum levels.

Research and development of the writing tool

Here is a doc containing research, development and
evaluation papers for the e asTTle writing tool.

It cost 17
level they
opinion of
would have

million dollars. If, for an accurate curriculum

could have just asked an individual teacher's
the curriculum level a piece of writing was, They
saved the money. They couldnt.

Why Test? What can you get out of it

To give you regular information that allows you target
learners and learning design to meet identified needs.
To track the writing progress of your class.
To map learners writing achievement to the curriculum.
To build assessment capability and strategies with your
learners.

What can your learners get out of it?

Improved assessment capability.
A clear picture of what they need to do to make progress and
what progress looks like.
Visible progress.
An opportunity to develop effective independent assessment
practices.

A note on making a judgments

An evaluation or any judgment you make uses a criteria to
Internal Criteria.
External Criteria.

Assessments like asTTle and PAT are attempts to make

criteria objective, universal and consequently comparable.
We cannot compare our internal criteria: they are different.

The actual assessment

Picking prompts
The test
Marking the test

Interesting things about using internal criteria

People will apply it more or less consistently
People will norm against themselves.
The research on the National Standards showed this really
clearly.

Picking a Prompt: they are not all created equal

The prompts are designed for a range of ages and a calibrated
differently. It is really hard to score highly on some.
You can compare genre: here is why.
There is no prompt or genre that is easier than another. Here is
why
Choice of prompt will give you the most accurate results because the
learner will choose something they think they can write about giving
them the best shot of showing what they can do accurately

Prompt stuff that will make your assessment inaccurate

You cant reuse the same prompt consecutively.
You cant Practice the test with the prompt.
You cant use the prompts as teaching tools

Sitting the test.

Here are the instructions
You CAN discuss the prompt. You cant make written records
of the discussion.
Digital or handwritten is fine (Spell check off)
Any special conditions you would usually use for learners
with learning difference apply.

Marking the test: The Rubric

There is one rubric for all writing purposes. Here it is.
The old rubric, and genre specific rubrics give inaccurate
results.They are still floating about online.
The rubric has been developed from the NZ Curriculum, the
Literacy Learning Progressions, and an analysis of student
responses to writing prompts.

The role of the exemplars

There are 2 sets. Generic exemplars and genre specific
exemplars - scroll down :-)
The exemplars that match the rubric area at each level are
underneath.
The genre specific ones cover what that genre looks like in
each level of the rubric.

BUT THEY DONT MATCH!!!!

The answer to this is in how the tool was tested
This means that only 48% agreement of moderators over all
rubric areas but 91% agreement within a sub level.
We are not supposed to agree on every single aspect. Not
agreeing (Moderator bias) is actually built into the
calibration of the tool.
They dont match in all areas with your internal criteria.
They do match the tool and how it works.

We have to use the exemplars as they are

Regardless of whether or not we agree with every aspect of
the example.
If you dont, the tool will not give you or your learners
an accurate result. It is like Justin making flapjack. You
muck with the oven temperature and you get burnt raisins.

Marking the test. Using the rubric and the exemplars

Use the exemplars as benchmark comparisons
What R score do you think the piece of writing is?
Look at the exemplars that have that R score. Is your piece
better or worse than that?
Look at one up or down from that. Is your Piece better or
worse than that?

The Structure and Language Notes

Here They are
These are specific to the structure and language area of the
rubric only.
It is not a checklist, more given examples of what structure
and language features for this genre COULD look like.

Some Important Stuff

Remember the rubric goes from Year 1 learners that can
barely write a sentence to year 10 learners.
Giving an R6 in an area does not mean the writing could not
be any better in that area.
R levels are not curriculum levels. They are not related at
all.
If the writing has it you MUST give it.

The key to your R scores being valid is being able to

back your judgments with evidence from the exemplars
There is no one voice of authority: even amongst marking
experts
Any one persons opinion is not enough.
The Tool takes this into account when calculating overall
levels
However: It is HIGHLY unusual for markers to disagree beyond
a sublevel. So Shared understanding is key.

Heres how it looks in action

Click here

Quick tips
Familiarise yourself with the exemplars before marking each
set.
Use Ctrl F and the digital copies for speed.
Develop go tos in the examples for the levels you most
regularly mark.

Marking , Rubric and exemplar stuff that will make your assessment
inaccurate.
Not using the exemplars.
Using your internal criteria (Instinct) to level writing
before marking using the criteria.
Tweaking.

Do I have to use the exemplars for each sample

It takes so long
You have to recalibrate from your internal criteria and
calibrate to the external asTTle criteria. Using the
exemplars at the start of each asTTle marking session will
help you do this.
It gets quicker.

Tweaking
People are doing this intending to improve accuracy BUT..
Gives inaccurate results: hopefully the information today
has helped clarify why.
Not fair to learners or parents.

So I have accurate data!

What can I do with it?

Using the data to make an OTJ

You can use the data with the Literacy Learning Progressions
to back up an OTJ. The results summary can be really useful
here.

Here are the Literacy Learning Progressions

Using e-asTTle formatively

To Identify target areas and areas of strength. You can use
the learning pathway for this.
Build assessment capability in learners, giving a clear
picture of what success looks like. Mark the writing with
the learners, using the exemplars, discussing their writing.

Cont...
Develop strategies to reduce assessment anxiety. Making
assessment a normal part of the literacy programme and
helping students to develop strategies to succeed.
If used regularly, you can use e-asTTle data to track
writing progress over time.

The aim of tomorrow

Developing our individual understandings
Moving toward a shared understanding of using the asTTle
tool and marking against the rubric.