Peer Assessment System (PAS)

written August 2003 being revised May 2006
Google site search

ContentsIntroductionBasicIntermediateAdvancedFuturePolicyInfrastructure

1   Introduction

This is a prototype example of a peer assessment system.

The assignment author writes a set of questions on the material or topic being assessed. Up to 30 questions can be written, anymore than this and the author should think about dividing the questions into two sets. This assignment is added to a library of such assignments.

The readers of the material access the system and select the assignment from the library. They answer the question set and their set of answers is stored under their name awaiting marking.

The author of the assignment marks a number of the answer sets until such point as there are readers who demonstrate sufficient understanding of the topic or material to act as peers. This is signalled by their achieving a certain assessment level specified by the author.

In addition the author sets a lower level trigger which allows readers access to answers that gained high assessment. The author can use this as a hint mechanism by providing an answer set consisting of hints rather than answers and marking them excellent. The author should label each answer as a hint in this case. Answer sets to not have to be complete, the author can select questions for the hint set.

The author can continue to mark question sets or rely on peers to do this. (Need to email the author that an answer set has been submitted).

A complete set can be deleted, in which case a copy of the assignment is emailed to the author and the readers get an email copy of the question set and their answer set.

2   The Question Set

This assessment system is better suited to questions which test comprehension and require opinion, inference and other fuzzy-type answers rather than yes or no. In answering the question the reader demonstrates understanding of the material being assessed. The reader is assessed on the level of understanding which is represented in the answers, rather than on whether the answer is right or wrong. Thus questions which have both yes and no answers can be asked and assessed. The computer does not do any marking or assessing, but simply provides a web based framework for the communication of a questions set and the process of assessing it, world-wide, over a time period which can be set, and for an indefinite readership.

Each question has a degree of difficulty which indicates how long it might take to answer the question, either in terms of thought or creativity. The degree of difficulty is taken from D.E.Knuth's 'The Art of Computer Programming Vol 1' and a short summary is presented here.

00Immediate
10Simple (one minute)
20Medium (quarter hour)
30Moderately Hard ( several hours )
40Term Project
50Research Problem
This scale informs the readers and the peers of how much creative thought is required in the answer. The author would be expecting an answer of a few words for a [00] question while a report of several pages would be expected for a [40] question.

3   Assessment

Because there are few right answers, assessment is done by peers who have already completed the assignment and demonstrated that they understand the material well enough to assess someone else's responses. One assignment can be marked by several peers and the marks averaged out to avoid bias. SO there can be several marks (mark records) per answer

Peers can learn more about the material by marking assignments and reading other peoples views of the material.

The question set can be given a mark legend. This is an array of mark steps and a description of criteria for assigning each mark as a guide to the markers. Because the questions tend to be fuzzy a resonably fuzzy (5 step) marking system is used. Ypu can weight the value of answers in the whole question set by using the difficulty indicator with each question. Here is a sample mark legend.

    Answers are marked on a scale of 1 to 5 :
  1. lost the plot
  2. poor
  3. ok
  4. good ( reasonably complete, accurate, thoughtful )
  5. excellent ( Gee, thats very good - I hadn't thought of that )
The Mark Legend is entered in the Question Set Record as a Tcl name-value-list:
{ 1 {lost the plot} 2 poor 3 ok 4 {good (reasonably complete, accurate and thoughtful)} 5 {excellent (Gee, thats very good - I hadn't thought of that)} }

Here is a database layout for the question sets. There will be a seperate table which will have the personal details of teacher and students. They will need to logon to the system with password and have enail address for contact. This table is more part of a general logon system, and the question sets hook into this.

create table qset (
    qset         integer,
    title          char(30),
    author      integer,
    datec       integer,
    dateu       integer,
    alevel       integer,
    mlevel      integer,
    url           char(40),
    marklegend char(60),
    status      char(4)
    primary key ( qset )
);

insert into dbtable ( dbtable, dbkey ) values ('qset', 1 );
delete from dbfield where dbfield glob 'qset *';
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset qset', 0, 'integer', '-width 12 -text QSet' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset title', 1, 'char', '-width 32 -text Title' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset author', 2, 'integer', '-width 12 -text Author' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset datec', 3, 'date', '-width 12 -text {Date Created} -format {%d %M %Y}' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset dateu', 4, 'date', '-width 12 -text {Data of Last Update} -format {%d %M %Y}' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset alevel', 5, 'numeric', '-width 4 -text {A Level} -format %0.2f ' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset mlevel', 6, 'numeric', '-width 4 -text {M Level} -format %0.2f' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset url', 7, 'char', '-width 40' -text URL );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset marklegend', 8, 'char', '-width 40' -text {Mark Legend} );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset status', 9, 'char', '-width 4' -text Status );

create table question (
    qset        integer,
    number      integer,
    difficulty  char(4),
    question    char(60)
    marks       integer,
    primary key (qset, number)
);

insert into dbtable ( dbtable, dbkey ) values ('question', 2 );
delete from dbfield where dbfield glob 'question *';
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset qset', 0, 'char', '-width 12 -text QSet' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset number', 1, 'integer', '-width 2 -text {Question Number} ' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset difficulty', 2, 'char', '-width 4 -text Difficulty' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset question', 3, 'char', '-width 60 -text Question' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('qset marks', 4, 'integer', '-width 2 -text Marks' );

create table answer (
    student integer,
    qset    integer,
    number  integer,
    answer  char(40),
    date   integer,
    primary key (student, qset, number)
);

insert into dbtable ( dbtable, dbkey ) values ('answer', 3 );
delete from dbfield where dbfield glob 'answer *';
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('answer student', 0, 'integer', '-width 12 -text Student' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('answer qset', 1, 'integer', '-width 12 -text {Question Set} ' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('answer number', 2, 'integer', '-width 2 -text {Question Number} ' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('answer marker', 3, 'integer', '-width 12 -text Marker' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('answer answer', 4, 'char', '-width 40 -text Answer ' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('answer date', 5, 'date', '-width 12 -text {Answer Date} -format {%h:%m $d $M}' );


create table mark (
    student integer,
    qset    integer,
    number  integer,
    marker  integer,
    mark    integer,
    note    char(20),
    date    integer,
    primary key (student, qset, number, marker)
);

insert into dbtable ( dbtable, dbkey ) values ('mark', 4 );
delete from dbfield where dbfield glob 'mark *';
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('mark student', 0, 'integer', '-width 12 -text Student' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('mark qset', 1, 'integer', '-width 12 -text {Question Set} ' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('mark number', 2, 'integer', '-width 2 -text {Question Number} ' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('mark marker', 3, 'integer', '-width 12 -text Marker' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('mark mark', 4, 'integer', '-width 4 -text Mark ' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('mark note', 5, 'char', '-width 20 -text Note ' );
insert into dbfield ( dbfield, dbposition, dbdomain, dbattrs ) values ('mark date', 6, 'date', '-width 12 -text {Mark Date} -format {%h:%m %d %M} ' );

Here is a question set specified at a low level of Tcl. The question set and questions have been phrased a Tcl commands. So [qset] and [question] are commands that create their respective records in the database. The data definition provides as much information as is needed to write these two procedures, they dont have to be written by hand.

set qset [qset  "The Perils of ICT 1"  -author webscool -alevel .3  -mlevel .6 -url "http://webscool.org/itp/Advanced.html#Putting it all together" -status publish ]
set i 0
question $qset [incr i] -difficulty 00 -question "What is Susan teaching?"
question $qset [incr i] -difficulty 00 -question "Who are Susan's students?"
question $qset [incr i] -difficulty 05 -question "At what level of their training are the teacher trainees ?"
question $qset [incr i] -difficulty 00 -question "Do the students go to the same school?"
question $qset [incr i] -difficulty 15 -question "Where are the students?"
question $qset [incr i] -difficulty 20 -question "Do the students attend a real physical school? List several other options for a 'school' that might be available in 2012. For each, provide characteristics which distiguish it from each of the others."
question $qset [incr i] -difficulty 20 -question "For each school paradigm in question [expr $i-1] list pros and cons that their organisational and technical setup engender." 
question $qset [incr i] -difficulty 10 -question "Do the students know each other personally?"
question $qset [incr i] -difficulty 10 -question "Who are Paynter and Brooke?"
question $qset [incr i] -difficulty 05 -question "Had Susan instructed the students to read Paynter and Brooke? Give the reason for your answer."
question $qset [incr i] -difficulty 10 -question "Why did some of the students read Paynter and Brooke?"
question $qset [incr i] -difficulty 05 -question "Can you access and read Paynter or Brooke? If yes please provide reference."
question $qset [incr i] -difficulty 20 -question "Will a student in 2012 be able to access and read Paynter or Brooke? Carefully argue the reasons for both answers."
question $qset [incr i] -difficulty 15 -question "Why was the session with the students logged and the session with the teacher trainees not logged?"
question $qset [incr i] -difficulty 15 -question "What is it that Susan did not \"really need to know\" ?"
question $qset [incr i] -difficulty 15 -question "Is <i>Romeo and Juliet</i> an appropriate play for 17 year olds to study in school? Give reasons for and against."
question $qset [incr i] -difficulty 20 -question "Reflect on the students take on <i>Romeo and Juliet</i>"
question $qset [incr i] -difficulty 15 -question "Do you think the students business plan has any merit? Give your reasons"
question $qset [incr i] -difficulty 00 -question "Did Susan require a unanimous vote ?"
question $qset [incr i] -difficulty 05 -question "Did the students know that a unanimous vote was required?"
question $qset [incr i] -difficulty 00 -question "Do you know the Rules of Westminster?"
question $qset [incr i] -difficulty 10 -question "At what age or level do you think students should be taught the Rules of Westminster? Give reasons for your answer."
question $qset [incr i] -difficulty 20 -question "To what extent are teachers trained to handle student interest conflicts in collaborative projects?"
question $qset [incr i] -difficulty 35 -question "Provide a set of criteria which a teacher could use to help resolve conflict of student interests in collaborative projects."

set qset [qset  "The Perils of ICT 2"  -author webscool -alevel .3  -mlevel .6 -url "http://webscool.org/itp/Advanced.html#Putting it all together" -status draft ]

Exercise - Write the two procedures [qset] and [question].
Exercise - Write a generic SQLite create record procedure, based on the data defunition.

4   Question Sets Available

Checkout the [tablelist] procedure and write the command to display the available question sets.

Answer - tablelist $questiondb qset {} status=publish

5   Question Sets Authored

Each author has access to the question sets they have authored so they can change, withdraw, or reissue each question set. They can also check on assignments returned by students and marking progress. Once a student has completed an assignment it becomes locked and the student can not go back and change it.

Student markers who get a mark in excess of the mlevel (marker level)n are given access to other students answers to mark. This is not too different to the tutorial system used in Universities, except this is more immediate.

Exercise - Discuss the pros and cons of this arrangement.

©2000 - 2006 WEBSCOOL This page last updated 16 May 2006. All rights reserved - including copying or distribution of any portion of this document in any form or on any medium without authorisation. For more regarding the copyright.