大数据介绍英文方案

BIGDATAEVERYMINUTE…1,388cabs2,777privatecarsDidirideshailed:EVERYMINUTE…395,833PeopleloginToWeChat194,444peoplearevideooraudiochattingEVERYMINUTE…625,000YoukuTudouvideosbeingwatchedEVERYMINUTE…64,814postsandrepostsonWeiboSEARCH4,166,667searchqueriesEVERYMINUTE…774peoplebuysomethingonAlibaba’smarketplacesUS$1,133,942spentonAlibaba1Definition2Characteristic3NoSQL4RDBMS5MapReduceCONTENTS6Applications1Definition1DefinitionBIGDATAvolumeofdataimportantdataonaday-to-daybasisforbetterdecisions2Characteristic2CharacteristicVolumeThequantityofgeneratedandstoreddata.VarietyThetypeandnatureofthedata.Thequalityofcaptureddatacanvarygreatly,affectingaccurateanalysis.VelocityInthiscontext,thespeedatwhichthedataisgeneratedandprocessedtomeetthedemandsandchallengesthatlieinthepathofgrowthanddevelopment.VariabilityInconsistencyofthedatasetcanhamperprocessestohandleandmanageit.Veracity3NoSQL3NoSQLNoSQLreferstodocument-orienteddatabasesSQLdoesn’tscalewellhorizontally.Itisschemaless.Butnotformless(JSONformat).JSON:datainterchangeformatMongoDatabaseCouchDatabase3NoSQLBasicAvailabilityspreaddataacrossmanystoragesystemswithahighdegreeofreplication.SoftStateEventualConsistencyBaseModeldataconsistencyisthedeveloper'sproblemandshouldnotbehandledbythedatabase.atsomepointinthefuture,datawillconvergetoaconsistentstate.Noguaranteesaremade“when”.3NoSQL{field1:value1,field2:value2…fieldN:valueN}varmydoc={_id:ObjectId(5099803df3f4948bd2f98391),name:{first:Alan,last:Turing},birth:newDate('Jun23,1912'),death:newDate('Jun07,1954'),contribs:[Turingmachine,Turingtest,…],views:NumberLong(1250000)}JSONStructure3NoSQLRDBMSvsNoSQL•XszcRowDB:001:10,Smith,Joe,40000;002:12,Jones,Mary,50000;003:11,Johnson,Cathy,44000;004:22,Jones,Bob,55000;index:001:40000;002:50000;003:44000;004:55000;ColumnDB:10:001,12:002,11:003,22:004;Smith:001,Jones:002,Johnson:003,Jones:004;Joe:001,Mary:002,Cathy:003,Bob:004;40000:001,50000…;Smith:001,Jones:002,004,Johnson:003;…3NoSQLBenefits•Column-orientedorganizationsaremoreefficientwhenanaggregateneedstobecomputedovermanyrowsbutonlyforanotablysmallersubsetofallcolumnsofdata,becausereadingthatsmallersubsetofdatacanbefasterthanreadingalldata.•Column-orientedorganizationsaremoreefficientwhennewvaluesofacolumnaresuppliedforallrowsatonce,becausethatcolumndatacanbewrittenefficientlyandreplaceoldcolumndatawithouttouchinganyothercolumnsfortherows.•Row-orientedorganizationsaremoreefficientwhenmanycolumnsofasinglerowarerequiredatthesametime,andwhenrow-sizeisrelativelysmall,astheentirerowcanberetrievedwithasinglediskseek.•Row-orientedorganizationsaremoreefficientwhenwritinganewrowifallofthecolumndataissuppliedatthesametime,astheentirerowcanbewrittenwithasinglediskseek.3NoSQLSQLvsNonSQLAgoodcompromiseistodesignyoursystemwith3logicalDBs1.NormalSQLDBusedbyyouradminapplicationtocreatecontent.2.No-SQLDBforfront-end/public/high-volumeapplicaitonusedbythepublicinternet.3.ThelastDBisforanalyticalreportingsystemusingcubesandallthatgoodstuff.ThendataflowsfromtheAdminDBtotheclientNo-SQLDBwhensomeonePublishesapieceofcontent,theclient(NoSQL)dbprovidesveryfastreadaccessandrecordsuserinteractionswiththecontent.ThenyouhaveascheduledjobthatpullsthedatafromtheclientDBintothereportingsystem.SinceAdmin,client,andreportingareoftenseparateapps,eachapplicationteamcanworkwithdataintheformatthatbestservestheapplicationandthetransitionfromonesystemtotheotherishandledintheservicelayers.4RDBMS4RDBMSfixed-schema,row-orienteddatabaseswithACIDpropertiesandasophisticatedSQLqueryengineTheemphasisisonstrongconsistency,referentialintegrity,abstractionfromthephysicallayer,andcomplexqueriesthroughtheSQLlanguage.easilycreatesecondaryindexes,performcomplexinnerandouterjoins,count,sum,sort,group,andpageyourdataacrossanumberoftables,rows,andcolumns.5MapReduceDividingandconqueringHighlyfaulttolerantEverydatablockreplicatedon3nodesDifficulttoimplement5MapReduce5ComparisonRDBMSMapReduceDatasizeGBPBAccessInteractiveandBatchBatchUpdatesRead/WritemanytimesWriteonce,ReadmanytimesStructureStaticSchemaDynamicSchemeIntegratedHigh(ACID)LowScalingNolinerLinerDBARatio1：401：30005HowdoesMapReduceworkMapReduceuseskey/valuepairs.(Traditionallyusingrowsandcolumns)--Mapalltheintermediatevaluesforagivenoutputkeyarecombinedtogetherintoalist.--ReduceThereducefunctionthencombinestheintermediatevaluesintooneormorefinalvaluesforthesamekey.--ReduceTwosteps:MapandReduce6Application6GovernmentTheuseandadoptionofbigdatawithingovernmentalprocessesisbeneficialandallowsefficienciesintermsofcost,productivity,andinnovation,butdoesnotcomewithoutitsflaws.Dataanalysisoftenrequiresmultiplepartsofgovernment(centralandlocal)toworkincollaborationandcreatenewandinnovativeprocessestodeliverthedesiredoutcome.Belowarethethought[bywhom?]leadingexampleswithinthegovernmentalbigdataspace.6HealthcareBigdataanalyticshashelpedhealthcareimprovebyprovidingpersonalizedmedicineandprescriptiveanalytics,clinicalriskinterventionandpredictiveanalytics,wasteandcarevariabilityreduction,automatedexternalandinternalreportingofpatientdata,stand