Multiple vehicle detection and tracking in hard re

CAR-TR-834CS-TR-3667UMIACS-TR-96-52N00014-95-1-0521DASG-60-92-C-0055July1996MultipleVehicleDetectionandTrackinginHardRealTimeMargritBetke,EsinHaritaogluandLarryS.Davis.ComputerVisionLaboratoryCenterforAutomationResearchandInstituteforAdvancedComputerStudiesUniversityofMarylandCollegePark,MD20742-3275AbstractAvisionsystemhasbeendevelopedthatrecognizesandtracksmultiplevehiclesfromse-quencesofgray-scaleimagestakenfromamovingcarinhardrealtime.Recognitionisaccomplishedbycombiningtheanalysisofsingleimageframeswiththeanalysisofthemo-tioninformationprovidedbymultipleconsecutiveimageframes.Insingleimageframes,carsarerecognizedbymatchingdeformablegray-scaletemplates,bydetectingimagefeatures,suchascorners,andbyevaluatinghowthesefeaturesrelatetoeachother.Carsarealsorecognizedbydierencingconsecutiveimageframesandbytrackingmotionparametersthataretypicalforcars.Thevisionsystemutilizesthehardreal-timeoperatingsystemMarutiwhichguaran-teesthatthetimingconstraintsonthevariousvisionprocessesaresatised.Thedynamiccreationandterminationoftrackingprocessesoptimizestheamountofcomputationalre-sourcesspentandallowsfastdetectionandtrackingofmultiplecars.Experimentalresultsdemonstraterobust,real-timerecognitionandtrackingoverthousandsofimageframes.ThesupportofDARPA,theOceofNavalResearch,theArmyStrategicDefenseCommand,andPhilipsLaboratoriesunderContractsN00014-95-1-0521andDASG-60-92-C-0055isgratefullyacknowledged.1IntroductionWehavedevelopedavisionsystemthatrecognizesandtrackscarsinhardrealtimefromsequencesofgray-scaleimagestakenfromamovingcar.Recognizingandtrackingobjectsinimagestakenbyamovingcamera(oraxedcameracarriedbyamovingcar)ismuchmorechallengingthanreal-timetrackingwithastationarycamera.Notonlyistheremotionoftheobjectsintheimages,butalsorelativemotionbetweenthecamera,theobjects,andtheenvironment.Ourmethodusestherelativemotionbetweenthecamera-assistedcaranditsenvironmenttodetectpotentialcars.Tohandlecaseswherethereislittlerelativemotion,ourmethodsearchesforfeaturesthataretypicalforcars.Recognitionofacarisveriedifanobjectivefunctionyieldsahighvalue.Theobjectivefunctiondeneshowlikelyitisthatanobjectwithcertainparametersisacar.Theobjectivefunctioncombinesevaluationofthehistoryoftrackingapotentialcarwithcorrelationofthepotentialcarwithadeformabletemplateofacarcreatedon-lineusingthemethoddescribedinRef.[3].Variousapproachesforrecognizingand/ortrackingcarsfromamovingcamerahavebeensuggestedintheliterature{forexample,detectingsymmetry[33,29,4],approximatingopticalow[32,13,15,22],exploitingbinocularortrinocularstereopsis[23,4,26,18],matchingtemplates[26],andtraininganeuralnet[27].Relatedproblemsareautonomousconvoydriving[31,11],roaddetectionandfollowing[1,2,5{9,12,16,19,20,23,28],lanetransition[17],andimagestabilization[25].(Sincesomeoftheseareashavebeenresearchedextensively,weonlyrefertosomeofthemorerecentapproaches.)Therehasalsobeenworkontrackingvehiclesusingstationarycameras(e.g.,[21,14,10]).Acollectionofarticlesonvision-basedvehicleguidancecanbefoundinRef.[24].Unlikesomemethodsdescribedintheliterature,ourvisionsystemcantrackmorethanonecaratatime.Inaddition,itdoesnotneedanyinitializationbyahumanoperator,butrecognizesthecarsittracksautomatically.Ourmethodalsodoesnotrelyonhavingtoestimateroadparameters(asdoesRef.[23]).Unlikeothermethods[4,13,22,23,26,27,29,32],ourvisionsystemprocessesthevideodatainrealtimewithoutanyspecializedhardware.Allweneedisanordinaryvideocameraandalow-costPC.Simplicityisthekeytothereal-timeperformanceofourmethod.Wedevelopedasystemthatissimpleenough1tobefast,butsophisticatedenoughtoworkrobustly.Weareinterestedinapplicationsthatimprovetracsafetyforcamera-assistedorvision-guidedvehicles.Suchvehiclesmustreacttodangeroussituationsimmediately.Thisrequiresthesupportingvisionsystemnotonlytobeextremelyfast,butalsotobeguaranteedtoreactwithinaxedtimeframe.Therefore,weuseahardreal-timesystemthatcanpredictinadvancehowlongitscomputationstake.Mostoftherelatedresearchaimsat\virtualreal-timeperformance,i.e.,fastprocessingwithouttimingguarantees.Incontrast,weutilizetheadvantagesofMaruti,ahardreal-timeoperatingsystemdevelopedattheUniversityofMaryland[30].Marutindsanecientscheduleforallocatingresourcestothevariousprocessesthatsearchfor,recognize,andtrackthecars.Maruti’sschedulingguaranteesthattherequireddeadlinesaremet.Itreservestherequiredresourcesforeachtaskpriortoexecution.Thispaperisorganizedasfollows.AnoverviewofthevisionsystemisgiveninSec-tion2.Section3summarizesthefeaturesofthehardreal-timeoperatingsystemthatareimportanttoourwork.ThecomponentofourvisionsystemthatdetectscarsisdescribedinSection4andthecomponentthattrackscarsisdescribedinSection5.Section6reportsourexperimentalresultsandSection7summarizesourconclusions.2VisionsystemoverviewGivenaninputofavideosequencetakenfromamovingcar,thevisionsystemoutputsanon-linedescriptionofthelocationsandsizesofothercarsintheimages.Thisdescriptioncouldbeusedtoestimatethepositionsofthecarsintheenvironmentandtheirdistancesfromthecamera-assistedcar.Thevisionsystemcontainsthreemaincom