Building a hand posture recognition system from mu

BuildingAHandPostureRecognitionSystemFromMultipleVideoImages:ABottom-UpApproachDeclanMurphycARTLab,CNR,[e-mail]15March2002AbstractThisreportpresentsanoverviewofworkcarriedoutduringavisittothecARTlabofCNUCE,CNR,Pisa,undertheMOSARTTMRproject.Thetaskwastoinvestigatethefeasibilityofdevel-opingasystemforaccuratereal-timerecognitionofhandposturefrommultiplevideoimages,andtosetaboutdevelopingsuchasystem.Thissystemistobesuitableforapplicationtoreal-timecontrolofcomputerandelectronicallygeneratedmusic.Keywords:HandPosture,GestureCapture,ComputerVision,GesturalControl,MusicalInterface.AcknowledgmentsIwanttothankverymuchLeonelloTarabella,GrazianoBertini,GabrieleBoschi,andalltheteaminPisafortheirgeneroushospitalityandhelp.MyvisittoPisawasbothenjoyableandproductive,andIhopethiscomesacrossinthisreporttosomeextent.IowesomefurtherthankstoClausB.MadsenandMadsS¿rensenofºAlborgUniversityforprovidingmewithsomeusefulpointerstorelatedback-groundwork,andforthestimulatingexchangeofideas.IalsowishtothankAntonioCamurriandalltheteamattheInfoMuslab,DIST,UniversityofGenoa,forallowingmethe°exibilityto¯nishtheconstraintsinx4.2duringmyvisitthere.[Thissectionwasrevisedshortlyaftertheoriginalreport.]IwouldalsoliketoexpressthankstoJensArnspangandKristo®erJensenforinitiatingtheMOSARTproject,andparticularlyfor\adoptingmeastheirPhDstudent.1Contents1Introduction51.1Background............................51.1.1ManualDexterityandExpressivity...........51.1.2TheHandelSystem....................61.1.3TheApproach.......................61.2LiteratureReview.........................72Platform92.1Background............................92.2Linux...............................102.3EyesWeb..............................103ImagePre-Processing123.1CannyEdgeDetection......................123.1.1HowitWorks.......................133.1.2TheY-JunctionE®ect..................133.2EdgeRanking...........................133.2.1PerimeterExtraction...................143.2.2BridgingGaps.......................153.2.3Ranking..........................163.3FingerRecognition........................174PhysicalModel184.1TheModel.............................184.2Constraints............................194.3ThePalm.............................205FittingtheView(s)totheModel235.1Calibration............................235.2TheGeometryofReconstructingthe3DImage........245.2.1TheTrivialCase,APoint................2425.2.2ALineSegment......................255.2.3TwoViews,ConvexPolyhedra..............255.2.4FleshySegments.....................265.2.5Articulation........................275.2.6TheDigitalCalculus...................276FurtherWorkandConclusion296.1FurtherWork...........................296.1.1Completion........................296.1.2Animation.........................296.1.3Audio/MusicalParameters................296.2Conclusion.............................293ListofFigures2.1EyesWebPatchusingtheImagePre-Processingblock.....113.1CannyEdgeDetectionandits\Y-Junctione®ect.......133.2EdgeRanks............................143.3Maintainingthe\lastouterpixel.Eachpixeloftheperimeterisrepresentedbyasquare,withastraightlinesegmentfromitscentretothecentreofitslastouterpixel,rotatinganticlockwise.153.4Bridgingtheperimetergaps....................163.5Acounterexampleofwhyrotationmustbebackwards,notonwardsthroughtheedge.....................174.1Transformationsbetweenrepresentationsofthehand......194.2ThePhysicalModeloftheHand.................215.1Howeyesandcamerasperceive3Dprojectedonto2D.....235.2TheTrivialCase:asinglepoint..................255.3TwinView,ConvexPolyhedraFitting..............265.4ThickVectorFitting........................285.5ArticulatedSegmentFitting....................284Chapter1IntroductionThisreportoutlinesthestatusoftheresearchcarriedoutbytheauthorduringavisittotheComputerArtLab(cART)ofCNUCE,CNR,Pisa,undertheMOSARTTransferandMobilityofyoungResearchers(TMR)EUresearchnetworkproject.Theobjectoftheprojectwastoinvestigatethefeasibilityofdevelopingasystemforaccuratereal-timerecognitionofhandposture1frommultiplevideoimages,andtosetaboutdevelopingsuchasystem.Thissystemistobesuitableforapplicationtoreal-timecontrolofcomputerandelectronicallygeneratedmusic.1.1Background1.1.1ManualDexterityandExpressivityManyevolutionistsconsiderthatthedexterityofthehumanhand(inpartic-ular,itsopposablethumb)wasthesinglemostdecisivefactorcontributingtoouradvantageousevolutionamongsttheprimates.Thehandiscertainlythemostarticulatepartofthehumanbodywhenitcomestophysicalma-nipulation2,anditiswellknowninanatomythatthehumanhandhas,foritssize,manytimesmorenerveendingsthanalmostanyotherpartofthebody.1Thewordpostureisusedinthisreporttorefertoaparticular(mutuallyrelative)positionofthehandsand¯ngers,asopposedtothewordposewhichismorecommonlyusedinitssteadintheliterature.Innon-technicalEnglish,thewordposerefers{notsomuchtothepositionitself{buttoitsa®ectedimpressiononotherpeople.Thisisnot(yet)aconcernatthelowlevelofthesystemdescribedinthisreport.Therefore,postureispreferred.2ThewordmanipulationcomesfromtheLatinma