November 2006 Volume 2
Article 4.
PDF File

Article Title
          Using Descriptive Assessment Rubrics as Teaching and Learning Tools       

Author
David F. Dalton. MA. App Ling; Dip Ed; Dip TESOL
Senior Communications Lecturer, Petroleum Institute, Abu Dhabi, UAE

Bio Data
The author is a senior lecturer in Communications at the Petroleum Institute in Abu Dhabi. He has sixteen years experience of teaching language and communications at universities in the UK, Spain, Mexico and the UAE. He also has extensive experience of curriculum design and educational management. His current work focuses on teaching a range of practical, synthesized communications and research skills to students who will later work in the oil and gas industry as engineers, managers and administrators. Current research interests are:
1 Core concepts in learning in the context of engineering, communications and language.
2 Discoures (features) analysis of written and oral texts in the context of engineering degree programmes and the oil and gas industry.       E-mail: ddalton@pi.ac.ae

Abstract 

The concept of learning outcomes is probably as old as learning itself. It has been accepted practice in education in some areas for at least the last three decades and was previously referred to as ‘vocational education’, with the National Vocational Qualifications in the UK being one of the prime examples.
Apart from the initial difficulty of effectively articulating the outcomes, there are often specific issues when it comes to assessment. Common concerns are:

• Teacher learning curve (professional development)
• Design of effective rubrics
• Increase in assessment load and feedback time
• Student buy in

The writer believes that outcomes are not universally appropriate but context -determined and that some contexts lend themselves more naturally to learning outcomes than others. The PI being an engineering institute preparing graduates to work in the oil and gas industry has a well-defined context. Clarity of context makes dealing with some of the concerns indicated above, much easier. This thesis will be explained by looking at outcomes assessment as teaching and learning in the context of the Communications Program offered at the Petroleum Institute. There will be specific reference to assessment as learning; its value as a teaching and repair tool; issues of transparency and learner buy-in.

 Introduction.

The program context.
The Abu Dhabi National Oil Company (ADNOC) operates in all areas of the oil and gas industry. It was established in 1971 and has steadily widened its endeavors through its subsidiaries, establishing an integrated oil and gas industry in Abu Dhabi. The Company manages and oversees oil production of over two million barrels a day and is among the top ten oil and gas companies in the world. It has substantial and impressive upstream and downstream operations in transportation, shipping, marketing and distribution.
Five years ago ADNOC announced the establishment of The Petroleum Institute, in Abu Dhabi. The decree, establishing the Petroleum Institute, provides for an entity that will bring together the best from education and industry to create a world class regional center for engineering, applied science and research.

The Institute offers five baccalaureate degree programs: chemical engineering, petroleum engineering, petroleum geosciences engineering, mechanical engineering and electrical engineering. As the undergraduate programs mature, the PI will eventually evolve into a fully-fledged research and educational institution providing programs leading to Master of Science and Doctor of Philosophy degrees. Planning is now underway for graduate and research programs.

For some time now there has been a growing concern in our industry to ensure a range of skills and competencies amongst prospective employees that go well beyond the content-specific knowledge base appropriate for a particular field. Today there is more of an emphasis on the whole person, exemplified by concerns that students exiting institutes such as ours should be able to call upon a range of competencies of a more behavioral nature. Our teaching focus here is guided by ADNOC’s Competence Assurance Management System. Section D of this extensive document relates explicitly to the behavioral competencies (so-called soft skills) delivered through the Communications Programme.
Recognizing that the most immediate and dominant product of the Petroleum Institute will be baccalaureate graduates, the institutional educational goals for baccalaureate degrees identify attributes that reflect this whole-person development of the graduating student. This is stated in our graduate profile which includes the following description:

• The graduating student will appreciate the critical role played by verbal, written and graphical communications in engineering practice and project management, and will have the corresponding skills to communicate with a range of audiences, and the skills to employ information technologies where appropriate.
• The graduating student should acknowledge that technologies, economies and societies are in a continuous state of evolution, and should therefore have the flexibility to manage a career path that changes over time, and that is supported by life-long learning, critical thinking, teamwork, leadership and the ability to span several disciplines.
• The graduating student should understand the global nature of modern engineering and business, and in order to succeed in this international arena the student should have an awareness of customary practices in different countries and the influence of diverse cultures.

The programmme.
To facilitate and nurture the development of such skills, students also choose from a growing range of electives of a more social sciences/humanities nature such as history, economics, political science, German and Islamic studies, as well as being required to complete a two-part Communications Programme which includes most of the behavioral competencies and project management skills identified above.

Through qualitative and quantitative research projects and presentations, the students learn and apply a range of communication and project management skills, the mastery of which becomes essential to their success on the programme as well as being transferable to other areas of study and life. These include:

• Time management and planning
• Critical thinking and problem solving
• Effective oral and written communication
• Teamwork and collaborative learning
• Leadership
• Cultural awareness
• Technology and media awareness

The program is learning outcomes based. The concept of learning outcomes is probably as old as learning itself. It has been accepted practice in education for at least three decades and was previously referred to as ‘vocational education’, with the National Vocational Qualifications in the UK being one of the prime examples.

 Universities throughout the world have increasingly come to favour a learning outcomes approach to their educational programmes. This has come about as a consequence of an increased interest from end users in what learners are really able to do, rather than what programmes and educators state as objectives. Employers in the oil and gas industry (ADNOC, SHELL, BP, and TOTALFINAELF) increasingly express explicit interest in skills, competencies and performance. Accrediting bodies (ABET,) want to know about levels, standards, outcomes and assessment. Government and the broader society have similar interests. The learning outcomes approach developed as a response to and analysis of this range of needs, emphasizing the application of knowledge and the integration of competencies in the context of performance.
 We have the luxury of a clear context for student learning and we know:

• What (communication) skills students need to be successful in undergraduate study in our  institute
• What client courses require of the students
• What (communication) skills student will require in the workplace

This has allowed the expression of learning outcomes with a very practical and real world focus. This makes it easier to dialogue with students about the value of the program and the skills they will be learning as they are able to see that these are practical, life-long and transferable.

Because we are dealing with the development of skills and competencies, assessment needs to de descriptive and formative. For example, assessing grammar with a multiple choice test may give us limited information about a learner’s knowledge of elements of the system of grammar, but tells us nothing about their ability to apply that knowledge in different settings, for different purposes with different participants and audiences. There is thus no synthesis or integration. It is likely that if testing occurs in this way, then so does teaching. What then of learning? We view assessment as the basis of a dialogue with the learner about their development. In order for this to be effective we need documents ‘to talk from’. These are the descriptive rubrics we use for feedback. In this section I will discuss the use of a set of in-house writing descriptors (page 5) used for assessing and responding to technical report writing.

Table 1: writing indicator.

What follows is a description of how such rubrics are introduced to the students and applied. The overarching learning outcome of the program is: A PI student who successfully completes this course will have the critical thinking, problem solving and communication skills to be able to successfully function in an academic environment at the undergraduate level.

The course therefore prepares the students to be able to do this. We use indicators as in Table 1 to describe how students make progress towards meeting the outcome in different skill areas. In the context of writing, the following indicator is expressed: The ‘written text’ referred to under enabling skills can be of a number of genres.  Descriptive rubrics are then used to assess these indicators, in this case using the bands referred to under ‘Standard’ above. The rubric on page five is used in the assessment of technical reports such as the research report mentioned above .We utilize such rubrics (albeit text specific) for every piece of oral or written communication our students produce. These rubrics are hyperlinked to our electronic course syllabus and students are instructed to download them at the beginning of the semester.

The course outline is used as a critical reading text during the first week of each semester. The concept of learning outcomes and skills and competencies is worked with in the classroom. Our approach to assessment (purpose, rational and process) is discussed and questioned and the students are introduced to the concept of descriptive rubrics and how they are applied. In this way the assessment is initially approached as a teaching/learning dynamic. This is something new for our students. Since we believe that it is very important, rather than being approached as a component, it is dealt with as a theme and dynamic within the program and is therefore frequently returned to as a learning focus. It is also integrated into the description and approach to each assignment.

Assessment rubrics in teaching and learning.
An obvious starter question for the students is: “If you know exactly how you are going to be assessed, how can this help you when framing and planning a deliverable (assignment)?” The rubric below is used in the assessment of a basic research report and other forms of technical writing.

Table 2: Report writing descriptors

The rubric above is simply a text, in that it is a piece of communication used for a specific purpose and delivering information to an identified audience. As such it can be used as a reading text allowing exploration of concepts, content and lexicogrammar. All of these, of course, are elements of the texts students will produce.

Thus before the assessment process begins, the tool which will be applied to make ‘judgements’ about quality of writing is front-loaded as a teaching/learning tool. A number of writers (e.g. Osborne, 2005; Walvrood, 1998) have addressed the value of this. The text is analysed in terms of the concepts it contains. This may relate to some of the titular vocabulary such as ‘purpose’ and ‘organization’ and it also relates to the descriptive and conceptual language such as ‘assumption’ and ‘implication’. Notions of what constitutes a ‘clear structure or pattern’ and ‘style and expression’ can be explored and discussed resulting in common understanding and agreement.

Elements of lexicogrammar (this writer agrees with Lewis, 1993 page V1, that the “Grammar/ vocabulary dichotomy is invalid”) can be explored and explicitly focussed on as a significant feature of successful writing and as a focus for future teaching. Collocation is an obvious focus here and there are many such as ‘acceptable appearance’, ‘generally relevant’ which can be worked with in the context of the rubric and then generalized to other texts.
Exploring such elements also helps develop the students’ language awareness.

Content can begin to be looked at in terms of deciding on relevance of the information contained in the text and how one might decide what is relevant in the context of different types of writing. Prior experience of students can be drawn on here. Questions and discussion can be generated by examining what is meant by supporting and explaining opinions. Our students are generally effective when it comes to description but perhaps not always so developed when it comes to higher order notions such as clear exemplification and the differences between description and explanation.

Schema can be referred to by discussion focussing on what is already known about the text from previous writing experience. Students generally have the idea that any text is written for a purpose with a particular audience in mind. They may not know how this should affect their choice of text language or organization. What is partially known and what is unknown can also be worked with. The ‘threat’ of new information can then be explored and allow the students to individually prioritize learning objectives and which knowledge they need to further develop to have a better understanding of the rubric and therefore a better schema for producing their own texts. Starting with what the learner already knows (constructivism) allows the learner a degree of security and familiarity with what can be rather complex documents or concepts

One of the useful wash back effects of this is approach is  that the focus is taken away from writing as  pure ‘product’ and through discussion, analysis it becomes established that writing is obviously a process beginning with a purpose and with an audience in mind. This is particularly important for our students as they have largely been exposed to a ‘model/reproduce’ approach with very little interest in why and how to write. It is also critical reading with a clear instrumental purpose for the reader- an improved grade!

Feedback and professional development

The first contact the student has with the assessment rubric is therefore not post-production, but pre-production, not reactive but proactive. It is consequently seen as a formative tool which can be worked with as the student is involved in the process of producing a text. It can guide the development of that text and be used to proofread and edit the text before submission.

During the feedback sessions, the rubric is then used to focus the discussion between the instructor and the student as to the qualitative description of the text and the resulting quantitative grade. Again the process has a teaching/learning focus as discussion inevitably highlights points of weakness that need development to improve subsequent text products. Students are able to compare text development and thus chart their own development as writers.

All too often such rubrics are simply handed out at the start of a course, (or not at all) filed away by the learner and never returned to. Sadly, this is a lost opportunity for learning (for both teachers and students) of a kind which has a direct instrumental benefit for the student and therefore results in a generally high buy-in. As I have identified above, much useful cognitive and language work can take place here. The examples I have given are only a few of the rich possibilities.

For the teacher, the use of the rubric also serves as a professional development tool. Initial familiarization with the content and process of working with the rubrics may take time (as any new learning does) but as Stevens and Levi have clearly pointed out, once mastered, the effective use of rubrics saves the teacher time.

Teaching and transparency

A major component of our program is delivering information through presentations. These range from short oral progress reports to lengthy (40 min) multi-media presentations involving substantial and lengthy planning. The rubric on the following page is one applied to presentations and again is utilized as a learning tool. The rubric can also be cognitively and linguistically cross-referenced to the written report rubric, thereby allowing the teacher to demonstrate the connectedness of skills we are teaching and the transferability of knowledge across areas of learner activity.

Our students sometimes have a tendency to see what they are learning as useful in only one context, this approach helps them think outside the box and become accustomed to using similar skills in different environments for (apparently) different purposes. Certain elements are used as milestones to focus the students on strategic elements of the presentation. The notion and skills of narrowing a topic for example is worked with in the context of the written research report. Here students can see their transference to a slightly different environment. Guiding vocabulary such as appropriate, logical, focussed and interesting can be worked with in the context of developing the presentation.

Concepts of relevance and appropriateness re appear (see Pages 5 and 9). The competency description on page 8 (currently being expanded) allows discussion with the students of the notion of an ‘acceptable’ presentation. It also begs the (learning) question: ‘If this is acceptable, how would descriptions of ‘unacceptable’ and ‘excellent’ look?’ The linguistic and cognitive benefits of such an approach do not need to be laboured on, but one obvious advantage is that it makes explicit the sometimes abstract concept of critical thinking (problem solving for a purpose?).

For example, students have a piece of evidence (the description) and then need to be able to realistically generalize from that, make assumptions and educated guesses to arrive at some decision which demonstrates some evidence of logical relationship, hierarchy and gradation as they develop criteria for the categories mentioned above.  These are considered higher order thinking and language skills and require the student to be able to describe and justify the criteria used to reach these decisions. This is a very useful language and cognitive exercise and process. I t also make the assessment of the presentation completely transparent and in a sense, enfranchises the learner.

Further, for those learners who are able and desirous, there is a shift from the ‘you gave me a C attitude’, to accepting personal responsibility for the grade by being able to evaluate their own performance because the student has internalized the description.
Once internalized, students are also able to see at what degree of proficiency the requisite skills need to be expressed in order to get that elusive ‘A’.

The table on the following lays out these descriptors.

Table 3: Oral presentation descriptors

The following description of competence is attached to the rubric to aid the students and again as a focus for classroom work.

Table 4: Presentation competence description

Again, the rubrics should make the skills to be assessed explicit and give a description of performance and level. In this way the rubrics capture the skills to be learned and later expressed by the students and assessed by the teacher. They also allow the learner to become involved in a discussion with the teacher about the latter’s response to performance. Seventy five percent on an exam does not leave the learner much to go on while a description of why the grade was given permits an informed discussion of the teachers’ assessment and the grade. The learners are therefore much more informed and have information useful to their development and improvement. They also have levels of performance which can be compared and reviewed. Without this it is often difficult for a learner to measure in what ways they have developed and improved.

Much of what we know about our current state is based on an ability to describe and measure past performance. The application of rubrics as learning and assessment tools clearly assists in developing this awareness.

It is important to mention that rubrics are constantly being fine-tuned. They are never ‘finished’. Indeed, the two examples above are currently being reworked to make the criteria more comprehensive and specific. They are also being linked to show elements of similarity between oral and written text and reformatted to have a more similar physical appearance. They will replace existing rubrics and be used this coming semester. This is part of our on-going professional and program development. Further, this kind of continual program modification is a feature we discuss with our students. This, plus the inclusion of assessment in teaching, the feedback and negotiation process and overall transparency lead to a very high learner buy in reflected in consistently positive comments and perceptions of the program expressed in the course survey we administer to students at the end of each semester. For example, below is a snapshot from our survey from last semester.

Overall feedback on the improvement of language skills was reassuringly positive. Feedback from client courses reinforces this. We believe that the approach to assessment used in our program and the application of the rubrics in the fashion described in the paper, is a strong affective factor in this highly positive perception.

References

1 Osborne. N.M. (2001).Rubrics for Elementary Assessment. Osborne Press. Mich. USA.

2 Lewis.M. (1993) The Lexical Approach. LTP. Hove. England.

3 Stevens.D and Levi .A. (2004) An Introduction to Rubrics:An Assessment Tool to Save Grading Time, Convey Effective Feedback and Promote Student Learning .Stylus Publishing.USA.
 
4 Walvrood.B.E. and Anderson.V.J. (1998) Effective Grading: A Tool for Learning and Assessment Jossey-Bass. San Fransisco.USA.

5 Weimer. M. (2002) Learner Centered Teaching: Five Key Changes to Practice. Jossey-Bass. San Fransisco.USA