sarahlangridge

[sarahlangridge]

Standardized tests vs. PBLA

Standardized tests help people develop rote learning and memorization, which can be useful at some points in life, but this usually doesn’t help people get and maintain employment. 

LINC programs are geared towards new immigrants who will settle in Canada and need to develop an understanding of the society and culture here. In this way, PBLA  is about helping people achieve real-life goals and develop skills that they will need outside of the classroom. PBLA is a way for them to track their progression and reflect on it. It is also a monitoring system for teachers to use in order to reference that students have or have not demonstrated ability in a certain skill area (and consequently, competency area). 

--

Noam Chomsky questions whether standardized tests are artificial because they drive people to achieve a rank rather than a skill. 

Linda Darling-Hammond claims that standardized tests predict almost nothing about future success.

Some teachers believe that standardized tests undermine good teaching and there is no correlation between test scores and work performance. 

Are standardized tests producing more fear in students? How does this contrast to the PBLA model?

https://www.youtube.com/watch?v=9JVVRWBekYo- Noam Chomsky

https://www.youtube.com/watch?v=2G_vWcS1NTA- Linda Darling-Hammond

[sarahlangridge]

Accountability in PBLA

What accountability exists in PBLA? 

All assessments should be criterion-referenced to the CLBs. If they are not, they students will not be achieving their benchmark goals. Without achieving these things, students will miss out on opportunities for work and other academics. 

Other teachers see the portfolios. If assessments do not meet the ‘standard’ or are not assessing what they should, then other people will notice. The interconnectedness of PBLA allows for more accountability to take place. 

Teachers are regularly reviewed by their school’s PBLA lead and usually have to submit 1-2 binders each year to an IRCC representative to demonstrate the learning, progression and projects their students completed throughout the term. 

Students can also be accountable to each other and themselves for their learning. Because PBLA is very integrated and demands that students be involved in the process, they can monitor themselves and read feedback from other students about their progression. (Peer-feedback is another form of ‘other’ artefact that can be placed in the portfolios). 

[sarahlangridge]

Potential abstract for the final project

PBLA (Portfolio Based Language Assessment) is a new initiative in Canada, funded by Immigration, Refugee, Citizenship Canada (IRCC) and used in all of the federally funded language classes, primarily in LINC (Language Instruction for Newcomers to Canada). PBLA uses continual feedback and on-going assessments to give learners many opportunities to display their abilities in a given language skill: reading, writing, listening and speaking. PBLA is criterion-referenced to the Canadian Language Benchmarks (CLBs), which are the national language standards for both English and French second language learners. LINC classes are structured around learner needs and the feedback students provide to the teacher throughout the term. PBLA is a way for students and teachers to track progress and map achievements. It also gives teachers evidence to support their benchmark decisions at the end of a term and this system increases Hermenuetic reliability because any ESL teacher should be able to review the portfolio and draw the same conclusions. PBLA provides reliable results and it can increase program reputation and success. 

[sarahlangridge]

Is there a shared interpretation of the assessment criteria?

Thought of the day

[sarahlangridge]

Literature

Some literature (there are over 80 papers and books) that PBLA was based on:

Gladwell, M. (2000). The Tipping Point. Boston, MA: Little, Brown.

Gardner, J. (Ed.). (2012). Assessment and Learning (2nd ed.). London, UK: Sage.

Fullan, M. (2004). System Thinkers in Action: Moving beyond the Standards Plateau. Nottingham, UK: Department for Education and Skills Innovation Unit. Retrieved from http://www.michaelfullan.ca/media/13396063090.pdf

Dörnyei, Z. (2001a). Motivational Strategies in the Classroom. Cambridge, UK: Cambridge University Press.  (https://www.slideshare.net/carlachavezs/motivatonal-strategies-in-the-language-classroom-dornyei-zoltan)

Delett, J. S., Barnhardt, S., & Kevorkian, J. A. (2001). A framework for portfolio assessment in the foreign language classroom. Foreign Language Annals, 34(6), 559-568.

Cotterall, S. (2000). Promoting learner autonomy through the curriculum: Principles for designing language courses. ELT Journal, 54, 109-117.

[sarahlangridge]

Haters gonna Hate

[sarahlangridge]

PBLA TESL Ontario Blog

In this blog, Sridatt Lakhan discusses the two shortcomings she has experienced with PBLA. 1- the  issue of needs assessment and tasks based on authentic materials that are used in real-life situations. 2- the ‘Guide for Teachers and Programs’ fails to address the question of differentiation. Students’ autonomy, individualized competencies, needs and assessment techniques – the foundations of PBLA – necessitate the differentiation of everything, from feedback, materials, and even instructional practices.  The guide states that “teachers need to adjust their lesson plans to accommodate PBLA activities appropriately and effectively” (p. 35).

Unfortunately, this blog does not elaborate much on her arguments, but others post good materials in the comments like: 

Professor Norm Friesen’s talk: https://vimeo.com/220251988 - where he discusses the broad claims for the superiority of PBLA over other instructional approaches are not clearly substantiated by the evidence cited in their support. More robust research findings show that teacher accountability measures like PBLA result in greater workplace stress and teacher attrition. This raises many questions about the mandated use of PBLA in LINC and other programs across the country: What does the research actually say about the effectiveness of PBLA and similar programs? Are all students in a position to be accountable for their learning? What is the impact of accountability measures like PBLA on teacher workload and morale? 

Source: http://blog.teslontario.org/pbla-a-need-for-task-based-needs-analysis-and-a-study-in-task-differentiation/

[sarahlangridge]

Questions to answer about IDAF- May 12

Who are the stakeholders that will be impacted by this assessment procedure?

-new immigrants (the students)

- the teachers, administrators

-school/program reputation

-government body deciding how to spend money

-society (CLBs determine opportunities)

How will the development and use of this assessment promote equity of opportunity?

  The CLBs are the governing body and the standards are the same for all people. Opportunities are then based on the CLB levels. 

The assessment is standardized by using the same measuring tool for everyone at a particular level (the same expectations). There is no favouritism or preference...someone cannot just skip a level. 

What are the information needs about students, teachers, or administrators that need to be filled?

?What do they want to learn and why

?What do they need to know to get there

?How can I teach those things

?How many classes do I need at this level (administration)

?What do they know and what do they need to improve on

What inferences about student learning or program functioning will be made from the assessment information?

St: At CLB (x) they can achieve the communicative objectives of most of that benchmark. Evidence has been provided to support this through the portfolio

Program: PBLA is a reliable way to assess and should be funded   

How will this assessment information be used?

Consequential

Who are the audiences for this information?

Businessess, employers, schools, administrators, government

What construct(s) are targeted in the assessment?

Whatever has been taught in class and a skill [R, W, L, S]

>objectives, theme, life skill, concept, grammar point, etc.

How will the construct sample be defined?

Per unit taught in the classroom

What is the content domain being measured?

Theme as provided or created against- Getting things done, Understanding instructions, Communicating with others, Comprehending information

What is the range of approaches you can use to collect the information you want?

Games, test, quiz, short-answer, paragraph, following instructions (do something), create something, instruct others, presentation, critical thinking and analysis, field trip, rubric, recordings, etc. 

How will the approach to collecting this information impact student learning and/or program deliver?

Continual assessment. A lot of work for the teachers, so class materials or assessments may be hindered. Continual feedback -> greater improvement + direction for future lessons. Program becomes flexible based on the needs of the learners. 

In what ways do the selected approaches introduce irrelevant variance into the measurement process? 

??*Only if the teacher doesn’t create assessments that assess what was taught in class. Because everything is created/chosen by a teacher based on what was taught in class, there shouldn’t be much, if any, irrelevant variance. 

What form will the information take?

Multiple, limited only by teacher creativity.

What are your scoring criteria?

70%+, using the CLBs for that level

What is the range of scoring procedures available to you?

Anything the teacher wants to create that shows achievement, improvement

How will scoring procedures achieve consistency (reduce irrelevant variance) while also supporting construct validity?

 -Everyone has to get 70+ in the end (criterion-referenced): it is based on achievement from the course and all the assessments the teacher collected throughout the term

-All students can learn a variety of things in their classes across a city, but they are all measured to the same standards in the end

How will scoring procedures influence: assessment outcomes, student populations, district/school/classroom experiences?

It means assessments lead to reliable outcomes despite a variety of teaching styles and classroom themes. 

Developmental needs and speed can be more easily met. 

Each level must achieve a certain consistency.

More homogenous and easier to teach. 

 More variety and self-expression and exploration. 

Scoring inferences- What claims about performances on the test items themselves does the information enable you to make?

 Whether or not a students was able to complete this task. If a student has developed these skills to a working level or not. 

Any cultural references or biases are representative of what was covered in the classroom. 

Generalization inference- Reflecting on construct sample, what claims about performances on the broader construct domain does the information enable?

 If a student can complete this task, they can/should be able to to it outside the classroom and can be given the appropriate benchmark.

They are ready for something more challenging. 

They can help others accomplish the same thing. 

Extrapolation inference- What claims about performance beyond the specific test context, construct, or set of test tasks does the information enable?

 Completion of the activity in real-life, outside of the classroom. All of the skills are meant to be transferable to other situations and help the students trouble-shoot and figure out harder or similar problems in other contexts.

 All of this leads us to believe they are ready for certain opportunities at a certain level of language achievement. 

Within the population being assessed, have the populations who might be disparately impacted by the assessment been identified?

 Yes and no. 

Yes: A teacher can cater to those who need extra help or more time to complete a task. Most assessments are not timed or limited to one sitting. A teacher may choose to give an assessment over an entire week. It just means that those who could be disparately impacted may take longer to achieve the benchmark. 

No: The CLBs make no exceptions for those with or without special circumstances. All new Canadians between ages 18-55 have to have CLB 4+ in order to take the citizenship exam, and there is no extra time or help for those with disabilities.

Taken collectively, does the evidence gathered indicate that the assessment has achieved the purpose or goals for which is was designed?

  Absolutely. Using the portfolios not only allows the students to see their performance, but it gives the teachers something to back up their decisions with and it gives future teachers something to look through and reference in order to know what the student has already learned and make their own assessment about abilities. 

Taken collectively, does the evidence provide an understanding of unintended impact, whether positive, negative or unknown?

If a teacher focuses primarily on speaking assessments and it can be seen that the student showed little improvement in writing, then it is obvious what the next teacher needs to focus on. 

One unintended impact is that teachers and administrators begin to stereotype what a certain level can achieve, and can forget that learning is fluid. It doesn’t take everyone the same amount of time to learn a language, but someone at the “bottom” and “top” of a benchmark are together in the same class and this may not allow the “top” person to feel challenged and improve as much as the “bottom” person. Teachers often forget that when people are working on the next benchmark, we should be teaching at the next benchmark level, in order to help students achieve the next steps, rather than ‘completing’ their current benchmark. A person does not have to master everything at one level to move on to the next. 

I would say the evidence provides somewhat of an understanding of the unintended impact, because it may be highly influenced by the teacher’s personal style, and of course this could be evidenced by the portfolios and student achievement.

Questions taken from: 

https://uottawa.brightspace.com/content/enforced/77748-EDU5499[T]_20185/Slomp%202016.pdf?_&d2lSessionVal=K7iTWRqLVx4ocs293wf4UhHKO

Slomp, D. (2016).  An Integrated Design and Appraisal Framework for Ethical Writing Assessment. 

[sarahlangridge]

IDAF in brief

Phase 1: Fairness, Aims, Context

Phase 2: Identify elements foundational to the assessment design.

Phase 3: Identify and select information collection methods.

Phase 4: Develop the scoring system

Phase 5: Develop a plan for analyzing assessment results--

scoring, generalization, extrapolation, decision inferences

critical questions regarding the scoring procedures should be explored

Phase 6: Develop a plan for analyzing consequences 

Source: https://uottawa.brightspace.com/content/enforced/77748-EDU5499[T]_20185/Slomp%202016.pdf?_&d2lSessionVal=K7iTWRqLVx4ocs293wf4UhHKO 

Slomp, D. (2016). An integrated design and appraisal framework for ethical writing assessment. Journal of Writing Assessment, 9 (1). 

[sarahlangridge]

Validity of PBLA

PBLA is both reliable and quite valid.  Construct validity- assessments are always based on what the teacher has done in the class and the students get multiple opportunities to demonstrate their knowledge in a subject area.  Concurrent validity- Skill building tasks are meant to be predictive of how a student will do on a Skill using task. All success is referenced to the CLBs.  Consequential validity- PBLA is meant to be use real-world-task assessment, so the transferrability of the skills in the classroom to real-life is meant to be very high. Because PBLA uses many exemplars and a standardized test to evaluate benchmarks, it is low-stakes and very useful for instructional design.  Content validity may be questionable if a teacher is new to creating assessments of lazy. This is the only area where a teacher may "cheat the system" and not assess learning outcomes (though it would be pointless because there would be no evidence for changing benchmarks at the end of the term). 

[sarahlangridge]

May 9

PBLA is about equal access to opportunity for all in a course, and it is criterion referenced because the students are all measured against the CLBs (which are made and evaluated by experts and were originally created based on research and recommendations from experts). 

PBLA is used throughout a term and then reviewed and referenced at the end (students keep a portfolio in their Language Companion binder). 

IDAF for PBLA

1. Assessment Aims: 

central to classroom practice

benefit the learner

reflect the curriculum developed in response to the needs and goals of the learners

yield accurate, reliable results

be part of effective planning

inform teaching

enhance exemplary teaching

2. Construct being assessed: Language skills in regards to-

Interacting with others

Getting things done

Comprehending instructions

Comprehending information

3. Assessment methods/approaches:

Learner centerd, RWT

Skill building/skill using

8-10 artefacts in each skill

informal notes, anecdotal comments, checklist

A learner has to demonstrate the level of communicative ability associated with most or all (70-100%) of the descriptors for the benchmark assigned. 

Analytic and Holistic - does the learner demonstrate success consistently?

??What progress has the learner made towards his/her personal language learning goals??

6. Assessment Consequences:

Work

School opportunities (post-secondary)

Settlement

Further educational and programming opportunities

Confidence

Volunteer opportunities

Quality of life

Source: http://pblaepg.language.ca/part-b-pbla-in-my-classroom/preparing-for-progress-reports-and-conferences/

[sarahlangridge]

This image was taken from the Emerging Practices document and was originally published in Davison and Leung (2009, pp 402-403). 

[sarahlangridge]

This video goes quickly through the online PBLA training that new LINC teachers are required to take before using PBLA in their classrooms. 

[sarahlangridge]

CLB Flow chart for PBLA.

[sarahlangridge]

Alignment to the Canadian Language Benchmarks

As outlined in the introduction of Canadian Language Benchmark: English as a Second Language for Adults, the CLB standard provides the following:

In the Canadian Language Benchmarks, a language task is understood to be a communicative “real-world” instance of language use to accomplish a specific purpose in a particular context.

– Canadian Language Benchmarks, p IX

A national standard for planning curricula for language instruction in a variety of contexts

A framework of reference for learning, teaching, programming, and assessing adult English as a Second Language (ESL) in Canada

A common yardstick for assessing learning outcomes

A set of descriptive statements about successive levels of achievement on the continuum of ESL performance

Descriptions of communicative competencies and performance tasks through which the learner demonstrates application of language knowledge (competence) and skill (proficiency)

A descriptive scale of communicative proficiency in ESL expressed as 12 benchmarks or reference pointsA framework of reference for learning, teaching, programming, and assessing adult English as a Second Language (ESL) in Canada

CLB Principles

The CLB standard also reflects fundamental principles about second language learning, teaching, assessment, and evaluation:

The CLB standard is learner-centred.

Instruction is based on the needs and goals of learners

Learners are informed and involved in decision making.

The CLB standard is task-based.

Performance is best determined through task-based assessment.

Instruction is task-based.

Tasks are based on real-world issues and events and use authentic text.

The CLB standard stresses community, study, and work-related tasks.

The CLB outcomes are free of context; therefore, they are taught in context through various topics or themes.

The CLB standard is competency based.

Competency statements describe what learners can do.

Communicative competence requires organizational knowledge, including grammatical and textual knowledge; pragmatic knowledge, including functional and sociolinguistic knowledge; and strategic competence.

These principles are embedded in CLB-based curriculum, teaching, and assessment practices; indeed, CLB principles are fundamental to PBLA.

Source: http://pblaepg.language.ca/part-a-pbla-background/theoretical-foundations-of-pbla/ 

[sarahlangridge]

Reliability and Validity of Classroom-Based Assessment

Assessment by teachers has the potential for providing valid and reliable information about learners’ achievements since teachers can build up a picture of learners’ attainments across the full range of activities and goals. In the past, critics have raised concerns that teacher-based assessment is unreliable and subject to bias. In 2004, in response to these concerns, Wynne Harlen of the University of Cambridge undertook a systematic review of 30 papers for the EPPI Centre of the University of London, England, in order to provide research evidence for the dependability of summative assessment by teachers and the conditions that affect it. He found that classroom-based assessment carried out by teachers is both reliable and valid when the following criteria are met:

Programs display a constructive and positive assessment culture.

Assessment is related to learners’ learning goals.

There are clear standards, required protocols, and appropriate resources to conduct assessment.

There are rigorous procedures for quality assurance and quality control of teachers’ judgments.

There is access to sufficient and appropriate assessment tasks, tools, and other resources.

There is professional development for teachers and supportive monitoring of assessment practices (Harlen, 2004).

PBLA protocols, resources, supports and monitoring procedures are intended to ensure these critical conditions are in place to support the reliability and validity of PBLA in the classroom.

Source: http://pblaepg.language.ca/part-a-pbla-background/theoretical-foundations-of-pbla/

[sarahlangridge]

Evaluating a Portfolio to Assign Benchmarks

The PBLA portfolio tells a story of learner growth and achievement.  Telling this story involves gathering evidence, reviewing the evidence and making decisions.  We will focus on the evidence and processes that support informed professional judgment when reviewing learner portfolios to make decisions.

How the Portfolio Demonstrates Achievement

To show evidence of achievement of benchmark standards

THE EVIDENCE:

8 – 10 artefacts in each of the skill areas (Listening, Speaking, Reading and Writing).  Portfolio artefacts will include a combination of assessment tasks and skill-using activities.  (See: PART B: Portfolio Contents for fuller explanation. )

Informal notes, anecdotal comments or checklists that have been collected over the semester to document what a learner can do aside from regular classroom activities.

Optional: Results from formal exit tasks (e.g., SAM, CLB 5 – 10 Exit Assessment Tasks) may be included if appropriate to what has been taught in the class.

Principles Guiding a Review of the Evidence:

Ask: What benchmarks have been achieved?

Before beginning the review process, there are several critical issues to keep in mind:

It is important to ensure that assessments are consistent with the standards outlined in the CLB. CLB expectations for the level and stage are found in the Profiles of Ability Across a Stage, Some Features of Communication Across a Stage and the specific descriptors for each benchmark level.  As well, reviewing the exemplars in the CLB Support Kit will help to keep expectations realistic when assessing learner performance.

Remember that we are not expecting 100% mastery. National Placement guidelines state that “As a general rule, the benchmarks assigned to a learner at the time of placement assessment, summative in-class assessment, or high-stakes language test, mean that the learner has achieved, and demonstrated, the level of communicative ability associated with most or all (traditionally, 70 to 100%) of the descriptors for the benchmarks assigned in each of the four skills. (National Placement and Progression Guidelines, 2013, p.3).Reviewing the evidence in the skill areas to assign benchmarks is not simply a matter of tabulating scores or percentages. The process requires informed professional judgment embedded in a thorough understanding of the CLB expectations and consideration of the factors identified below.

Portfolio review uses a combination of analytic and holistic processes. Portfolio review is analytic – focusing on collecting the necessary artefacts and looking at the performance on individual entries.  But it is also holistic which entails taking a step back and widening our lens to look at ALL of the entries in any skill as a WHOLE and exercising professional judgment to answer the question:  Does the learner demonstrate success consistently, i.e. most of the time at the benchmark level?

Steps in Reviewing Each Skill Area

Before you begin, ask these broad questions:

Review the evidence. Has the learner met Benchmark level expectations?  What progress has been made?

Notice the learner’s strengths and challenges. What suggestions will you make to the learner to help him/her improve his or her skills?

Once you’ve considered these questions, you are ready to begin the review, following these steps:

Check the number of entries.  Generally, you are reviewing portfolios after learners have been in class a minimum of  250 hours and you have about 8 – 10 entries for each skill area. (If a learner has recently moved to your class and was not assigned benchmarks upon leaving the previous class, you may consider evidence in the learner’s portfolio since benchmarks were last assigned).

Check the distribution of entries. The entries should be distributed across the four competency areas and reflect a range of tasks and criteria appropriate to the CLB level.

Review the assessment tasks.  Look at the tasks over time.  More recent assessment tasks should give a clearer indication of current language ability than tasks completed earlier in the semester.  Look for trends or particular areas of improvement or concern.

Review the skill-using activities.  The skill-using activities also provide evidence of what learners are able to do.  Skill-using activities are an opportunity for learners to practise – sometimes learner performance will be weaker on the practice activities and stronger on the subsequent assessment tasks, providing evidence of learning. Sometimes the skill-using activities will be stronger than the assessment task.  There may be many reasons for this.  For example, consider the amount of scaffolding provided. Scaffolding should be appropriate to the CLB level and help move learners to more independent performance.

Review anecdotal comments.  Informal notes about a learner’s performance not directly related to assessment tasks (e.g., the learner’s ability to give an appropriate apology when coming into class late, his/her ability to follow written or oral classroom instructions, the learner’s dependence on an electronic translator, etc.

Consider optional information.  You may also include results from tests and formal exit tasks (e.g., SAM, CLB 5 – 10 Exit Assessment Tasks, etc.) if they are appropriate to what has been taught in the class.  The results should be viewed holistically as part of the assessment evidence. They should not be given any 0greater weight than other data collected.

ASSIGN BENCHMARK LEVELS.

After reviewing the portfolio entries assign a benchmark level.  What benchmark has the learner achieved in each skill?

How the Portfolio Demonstrates Growth

To show progress related to learner goals..

THE EVIDENCE:

About Me section of the portfolio including:

Autobiography,

Needs assessment,

Learner Goal statement,

Review of Goal statement

Learning Reflections

REVIEW THE EVIDENCE:

Ask: What progress has the learner made towards his/her personal language learning goals?

Review the learner’s autobiography or piece of personal writing, needs assessment information, goal statement and review, and learning reflections.  This is an area in which the learner should generally take the “lead” in the learner conference but make notes so you will be able to support learners when discussing their growth related to their goals.

Source: http://pblaepg.language.ca/part-b-pbla-in-my-classroom/preparing-for-progress-reports-and-conferences/