OnlineJudge

OnlineJudgeAndyKurniaa,*,AndrewLima,BrendaCheangbaDepartmentofComputerScience,NationalUniversityofSingapore,3ScienceDrive2,117543SingaporebI-OPT,5JalanBesar,OngBanHongLeongBldg,03-01A,208785SingaporeReceived7August2000;accepted13January2001AbstractThisreportdescribesandevaluatestheimplementationandapplicabilityofanautomaticprogrammingassignmentgradingsystemwenamedtheonlinejudge.Wecomparedthiswiththemanualgradingsystemthatiscurrentlybeingusedandshowedthattheautomaticgradingsystem,whenimplementedcarefully,ismoreconvenient,fairer,andmoresecurethantheformer.Wehavesuccessfullytestedthesystemontwocourses.However,furtherstudiesneedtobeconductedtoimprovetheeffectivenessoflearningthroughthissystem.#2001ElsevierScienceLtd.Allrightsreserved.Keywords:Automaticgradingsystem;Onlinejudge;Programmingassignments1.IntroductionProgrammingisoneofthemajorskillstaughtintheDepartmentofComputerScience.Vir-tuallyeveryComputerSciencemoduleinvolvessomeformofprogramming.Needlesstosay,someComputerSciencemodulesrequiremoreprogrammingthanothers.Programmingisaskillacquiredthroughpractice.Tohelpstudentsinthisrespect,program-mingassignmentsaregiventostudents.Thenumberofprogrammingassignmentsrevealshowmuchemphasisonprogrammingamodulehas.Mostofthehigher-levelmoduleswouldassumethatstudentsarealreadycomfortablewithprogramminginvirtuallyanyprogramminglanguage,andthushasonlyonemajorprogrammingprojectthatemphasizesonthetopicbeingcovered,suchasclient-servernetworkingandstatespacesearching.However,modulesthatemphasizeprogramming,suchasProgrammingMethodology,DataStructuresandAlgorithms,andCom-petitiveProgramming,aimtonurturestudentsinprogramming,andhavemanyprogrammingassignmentswithquestionsofvaryingdifficulties.0360-1315/01/$-seefrontmatter#2001ElsevierScienceLtd.Allrightsreserved.PII:S0360-1315(01)00018-5Computers&Education36(2001)299–315*Correspondingauthor.E-mailaddresses:[e-mail](A.Kurnia),[e-mail](A.Lim).Consideringhowanundergraduatestudentwouldapplytheeconomicprincipleofmaximizinggainwithminimumefforttobeeffective,programmingassignmentshavetobegradedasother-wisetheywouldnotevenbeattempted.Consequently,teachingstaffneedtogradethestudents’attemptsontheprogrammingassignments.Normally,gradingwouldinvolveseveralpaidgraderstomanuallygothroughallthesourcefilesthatstudentssubmit,eitherinprintedformorelectronically.Thegoinggetstougherasmorestudentstakethemoduleandasmoreprogrammingassignmentsarethrownintothemodule,becausethegradersgetmoreworktodo,possiblywithoutanincreaseintheirpayrate.Manualgrading,asillustratedinSection2,hasitsproblems.Assuch,itwouldbeaworthyandenrichingexperiencetotryanotherapproach.Naturally,theotherapproachtothisproblemwouldbetohavegradingdoneautomatically.Aprogramwouldtakeinanotherprogramandreportitsjudgement,whichwouldthenbetranslatedintonumericalgrades.Thisprogram,likeahumangrader,wouldhavetoknowtheprogrammingquestions.Interestinglyenough,programmingassignmentsforthesemodulesarealgorithmicinnature,andthereforedonotrequirespecialuserinterfaces.Theywouldnot,forexample,needtodelveintotheintricaciesofmachine-specificimplementationssuchasgraphicaluserinterface,mouseinput,prin-teroutput,audiooutput,andsoon.Mostprogramsthatarealgorithmicinnaturewouldonlyneedtotakesomeinputfromthestandardinputinsomeprescribedformat,justasifthisinputweretheoutputofanotherprogram.Theywouldthenprocesstheminmemory,andoutputtheresultsofthecomputationinsomeprescribedformat,readyforyetanotherprogramtoreadintheresultsandmakeuseofthecomputation.Inshort,theywouldactlikeafilter.Byhavingaprogramthatwouldpassintheinput,andanotherprogramthatwouldtakeintheoutputandverifythemforcorrect-ness,itbecomespossibletohavegradingofprogrammingassignmentsdoneautomatically.Whilemanualgradingcanonlybedoneoffline,automaticgradingenablesgradingtobedoneeitherofflineoronline.Beforethedeadline,anofflineautomaticgraderwouldtakeinsubmissions,butwouldotherwisebeidle.Whenthedeadlineisreached,itwouldstartcheckingallsubmissionsthathavebeenreceivedandproduceareportonitsjudgementsonthesesubmissions.Onthecontrary,beforethedeadline,anonlineautomaticgraderwouldtakeinsubmissionsandproduceitsjudgementimmediately,allowingthestudentstodecideiftheywouldliketohaveanotherattemptontheques-tiontoobtainabettermark.Anonlineautomaticgraderwouldonlybeidleafterthedeadline.Inthisproject,weareinvestigatinghowhavinganonlinejudgewouldhelpgrading.WeusedthismethodofgradinginJuly–November1999and2000forCS3233CompetitiveProgramming,inOctober1999forACMProgrammingContestselection,andinJanuary-April2000forCS1102DataStructuresandAlgorithms.C++wastheprogramminglanguageusedforthefirsttwo,andJavawasusedforthethird.Theuseofcomputertechnologiestohelpingradingassignmentsisnotnew.SomesuchworkshavebeenreportedinArnow(1995),ArnowandBradshaw(1999),Kay(1998),Kay,Scott,Isaacson,andReek(1994)andReek(1996).2.FlawsinmanualgradingTraditionally,gradinghasalwaysbeendonemanually.Thisrequiredthestudentstosubmittheirsourcefiles,beitelectronically,printed,orhandwritten.Afterthedeadline,gradingwould300A.K