Evaluating Capture-Recapture Models with Two Inspe

V13–17/05/991EvaluatingCapture-RecaptureModelswithTwoInspectorsKhaledElEmamOliverLaitenbergerNationalResearchCouncil,CanadaInstituteforInformationTechnologyBuildingM-50,MontrealRoadOttawa,[e-mail]ExperimentalSoftwareEngineeringSauerwiesen6D-67661KaiserslauternGermany+49(0)[e-mail]-recapture(CR)modelshavebeenproposedasanobjectivemethodforcontrollingsoftwareinspections.CRmodelswereoriginallydevelopedtoestimatethesizeofanimalpopulations.Theyhavealsobeenusedtoestimatethenumberofdefectsinaninspectedartifact.Armedwiththisestimate,onecandecidewhethertheartifactrequiresareinspectiontoensurethataminimalinspectioneffectivenesslevelhasbeenattained.LittleevaluativeresearchhasbeenperformedthusfarontheutilityofCRmodelsforinspectionswithtwoinspectors.Furthermore,thesestudieshavefocusedontherelativeerrorofthedefectcontentestimatesexclusively.InthispaperwereportonanextensiveMonteCarlosimulationthatevaluatedsixcapture-recapturemodelsfortwoinspectorsassumingacodeinspectionscontext.Inadditiontorelativeerror,weevaluatetheaccuracyofthereinspectiondecision.Thelatterismorecongruentwiththemannerinwhichthesemodelswouldbeusedinpractice.Ourresultsindicatethatthemostappropriatecapture-recapturemodelfortwoinspectorsisanestimatororiginallydevelopedbyChapmanthatallowsforinspectorswithdifferentcapabilities.Thiswillhavearelativelyhighdecisionaccuracyandwillperformbetterthanthedefaultdecisionofnoreinspections.Furthermore,weidentifytheconditionsunderwhichthisestimatorwillperformbest.1IntroductionArecentliteraturereviewfoundthat,onaverage,softwareinspectionsfindonly57%ofdefectsincodeanddesigndocuments[8].Giventhesubstantialdefectdetectioncostsavingsthatcanbeaccruedbyincreasingtheeffectiveness1ofinspections[8],contemporaryresearchhasfocusedonimprovedreadingtechniques(e.g.,see[33][3][19][41])andonreinspections(e.g.,see[24])formaximizingeffectiveness.Thefocusofthispaperisonmaximizinginspectioneffectivenessthroughreinspections.Reinspectionscanbeconsideredpartofthegeneralproblemofwhentostopinspections.Asisthecasewithtesting,oneneedsacriterionbywhichtodecidewhetheradocumentshouldbeinspectedanew,orwhetheritcanpasstothesubsequentphase.Mostorganizationshavenotinstitutionalizedproceduresfordecidingwhentostopsoftwareinspections.Thosethatdohaveutilized,forexample,historicalnormssothatiftoomanydefectsarefoundcomparedtothenormthenthisistakenasevidenceofapoordocument,whiletoofewaretakenasevidenceofapoorinspection[24].However,thisapproachassumesthatvariationsamongreviewsarelargerthanvariationsamongdocuments.Ifthisisnotthecasethenthiscanleadtoreinspectionsofhighqualitydocuments,andlowqualitydocumentsmayeasilypass.Toaddressthesepotentialproblems,onecanuseCapture-Recapture(CR)models.CRmodelswereinitiallydevelopedtoestimatethesizeofanimalpopulations(e.g.,see[38][51]).Inasoftwareengineeringcontext,theyhavebeenappliedincontrollingthetestingprocess[4][30][36][21][37],andmorerecentlytheyhavebeenusedincontrollingtheinspectionprocess[23][24].1Effectivenessisdefinedastheproportionofdefectsinadocumentthatwerefoundduringtheinspection.V13–17/05/992Whenappliedtosoftwareinspections,CRmodelscanbeusedtoestimatethenumberofdefectsintheinspecteddocument.Usingthisestimateandtheknownnumberofdefectsfound,thenumberofremainingdefectsintheinspecteddocumentcanbeestimated.Subsequently,armedwiththisinformation,theinspectionteamcanmakethedecisionastowhetherthedocumentshouldbereinspectedtoreduceitsdefectcontentbeforepassingitontothenextphaseofthelifecycle.ResearchersatBellLabsfirstappliedCRmodelsforrequirementsanddesigninspections[23][24][25].However,inthesestudiesthetruenumberofdefectswasunknownandthereforeanevaluationoftheirtrueefficacywasnotpossible.LaterworkconsistedofaMonteCarlosimulationtoevaluatetherobustnessofdifferentCRestimatorstoviolationsoftheirassumptions[50].ObjectiveempiricalevaluationofCRmodelsstartedwiththestudyofWohlinetal.[53].However,thisstudywasconductedwithnon-softwareengineeringdocuments.Subsequentworkusedsoftwareengineeringartifacts[10][12][35][44].Alloftheaboveworkutilizedmodelsthatwereoriginallydevelopedinwildliferesearch.OtherresearchersconsideredtheincorporationofBayesianmethodstoestimatedefectcontent[5],performedfurtherevaluationsofassumptionviolationswhenusingCRestimates[48],andevaluatedtheapplicabilityofCRmodelstoperspective-basedreading[12][49].Analternativeapproachwasproposedin[54],theDetectionProfileMethod(DPM).TheDPMisanintuitivelyappealingapproachthatcanbeeasilyexplainedgraphicallytononspecialists.AlaterstudysuggestedamethodforselectingbetweenaCRmodelandtheDPM[9],andthiswassubsequentlyfurtherevaluatedin[39].InadditiontotheexperiencesreportedbytheresearchersatBellLabs,theuseoftheDPMataninsurancecompanyinGermanywasreportedin[11],andtheapplicationofCRmodelsintelecommunicationsprojects[1].Therefore,thereisagrowingadoptionofdefectcontentestimationmodelsinindustrialpractice,andspecificallyCRmodels.LittleempiricalinvestigationoftheutilityofCRmodelsforinspectionswithtwoin