ODI_Solutions_For_BigData_On_Hadoop

ODI–HadoopLabExercisesIntroduction..................................................................................................................................................4Pre-Requisites...............................................................................................................................................4Lab-1-LoadunstructureddatafromFile(LocalfilesystemorHDFS)intoHive.....................................7UnderstandingtheLab:............................................................................................................................7Executingthelab.....................................................................................................................................13Lab2-TransformandvalidatestructureddataonHive............................................................................15UnderstandingtheLab:..........................................................................................................................16Executingthelab.....................................................................................................................................18Lab3-TransformunstructureddataonHive.............................................................................................19UnderstandingtheLab:..........................................................................................................................20Executingthelab.....................................................................................................................................21Lab4-LoaddatafromHadoop(HDFSorHivetable)toOracleusingOracleLoaderforHadoop..........22UnderstandingtheLab:..........................................................................................................................22Executingthelab.....................................................................................................................................24IntroductionTheODIsolutionforBigDatalabwork,broadlyconsistsoffourlabs,eachofwhichhighlightsthecapabilitiesofODIKnowledgeModuleswhicharedevelopedtosimplifytheprocessingofBigDataonHadoop.Itshouldbenoted,however,thelabsdonotprovideacompletecoverageonalltheoptionsavailableinODIandtheknowledgemodules.Itishighlyrecommendedtousethelabsasastartingpointandrefertothedocumentation–ODIApplicationAdaptorforHadoopforadditionaldetailstodevelopcustomsolutions.SincethefocusonthelabsessionisonODIsolutionsforBigDataonHadoop,ItisassumedthatuserhasaknowledgeofODIandHadooptechnology,specificallyHDFSandHive.TheknowledgemodulesweredevelopedtosimplifyprocessingofunstructuredandstructureddataonHadoop.TheknowledgemodulesweredevelopedtofacilitatethefollowingprocessTheknowledgemodulesthathelpinfacilitatingtheaboveprocessare:LoadunstructureddatafromFile(LocalfilesystemorHDFS)intoHive–Box1TransformandvalidatestructureddataonHive–Box2TransformunstructureddataonHive–Box2LoadprocesseddatainHivetoOracle–Box3Pre-RequisitesTheODIsolutionforBigDataonHadoopisdeliveredthroughVMwithalltherequiredsoftwareandtheenvironmentpre-configured.AfterverifyingtheVMisupandrunningdothefollowing:BringupacommandterminalandenterthecommandtostartthehiveserverhiveserverBringupanothercommandterminalandenterthecommandtostartODIodiLoadDataIntoHiveTransformandvalidateDatainHiveLoadProcesseddatafromHiveintoOracleoClickon‘ConnectToRepository’inthe‘DesignerTab’oSelectOKonthepopupboxAfterselectingOK,youwillseethefollowingprojects.Thegeneralstepstoconfiguringallofthelabsarethefollowing.Sincethelabsarepre-configuredyouwillnotbeneededtoperformthesesteps,butitisimportanttobeawareofthesestepswhendevelopingcustomizedsolutions.Thedetailsofthesestepswillbeillustratedineachofthelabs.Createthemodelmeta-datawhichwillcontainthedatastoresandtheassociationtothelogicalschema.Duringexecution,dependinguponthecontextselected,thelogicalschemawillbemappedtotheappropriatephysicalschema.Thisallowsyoutoswitchcontextfromdevelopmenttotesttodeployment.Forthepurposesofthislabexercisethecontext‘global’willbeusedthroughout.Thedatastoremeta-datadescribeeitherthesourceortargetdatastore.Thedatastoresareusedintheinterfacetospecifythesourceandtarget.Definethelogicalarchitecture,physicalarchitectureandtheexecutionContextstoassociatelogicalandphysicalarchitecture.Designtheinterface.Theinterfacespecifiesthesource,target,mappingsandtherules,theKMs.Executingtheinterfacewithaspecifiedcontextwillmigratethedatafromthesourcephysicallocationintothetargetphysicallocation.Designthepackage.Thepackageallowstocreateaprocesswhichwouldexecuteinterfaces,proceduresandotherlogicasrequired.Thepackagesdesignedforthelabincludesthestepsthatgeneratedataandvalidateresults.ThegoaloftheeachofthelabsexerciseshighlighttheKnowledgemoduleslistedabove.Forthepurposesofthislabexercise,thestepstounderstandingthelabsandexecutinglargelyremainsthesame.Thereforeeachlabhastwoparts–UnderstandingtheLabandExecutingtheLab.Itis,however,importanttounderstandthedeeperpurposeofeachoftheKMsandusageofitsoptions.Whenexecuti