了解IO协议栈

了解IO协议栈核心系统数据库组余锋@淘宝褚霸2012-03-181提纲•IO子系统架构图•IO子系统各层分解•IO请求事件跟踪点•blktrace/btt解释2IO子系统架构图3$stap-l'ioscheduler.*'ioscheduler.elv_add_requestioscheduler.elv_completed_requestioscheduler.elv_next_requestblktrace$stap-l'ioblock.*'ioblock.endioblock.requestDM层块层框图4bufferediommapdirectio思考IO子系统有几层？各层的输入输出分别是什么？5块层probeioblock.request–FireswhenevermakingagenericblockI/Orequest.probeioblock.end–FireswheneverablockI/Otransferiscomplete.6DM层7•LVM2（LinuxVolumeManager2version)•EVMS(EnterpriseVolumeManagementSystem)•dmraid(DeviceMapperRaidTool)请求队列/电梯probeioscheduler.elv_add_request.kp-kprobebasedprobetoindicatethatarequestwasaddedtotherequestqueueprobeioscheduler.elv_next_request–Fireswhenarequestisretrievedfromtherequestqueueprobeioscheduler.elv_completed_request–Fireswhenarequestiscompleted8调度器参数微调9文档参考：Documentation/block/deadline-iosched.txt#cat/sys/block/sda/queue/schedulernoopanticipatory[deadline]cfq思考电梯算法的核心作用是什么？10驱动程序•中断平衡–/proc/irq/IRQ/smp_affinity•软中断平衡–/sys/block/DEV/queue/rq_affinity11块请求关键事件点12block:block_rq_abortblock:block_rq_requeueblock:block_rq_completeblock:block_rq_insertblock:block_rq_issueblock:block_bio_bounceblock:block_bio_completeblock:block_bio_backmergeblock:block_bio_frontmergeblock:block_bio_queueblock:block_getrqblock:block_sleeprqblock:block_plugblock:block_unplug_timerblock:block_unplug_ioblock:block_splitblock:block_remapblock:block_rq_remap$perflist|grep“block:”或者$trace-cmdlist|grep'block:*‘$stap-l‘kernel.trace(“block_*”)‘Tracepoint解释C--completeApreviouslyissuedrequesthasbeencompleted.D--issuedArequestthatpreviouslyresidedontheblocklayerqueueorinthei/oschedulerhasbeensenttothedriver.I--insertedArequestisbeingsenttothei/oschedulerforaddi-tiontotheinternalqueueandlaterservicebythedriver.Q--queuedThisnotesintenttoqueuei/oatthegivenlocation.B--bouncedThedatapagesattachedtothisbioarenotreachablebythehardwareandmustbebouncedtoalowermemorylocation.13Tracepoint解释(续）M--backmergeApreviouslyinsertedrequestexiststhatendsontheboundaryofwherethisi/obegins,sothei/oschedulercanmergethemtogether.F--frontmergeSameasthebackmerge,exceptthisi/oendswhereapreviouslyinsertedrequestsstarts.G--getrequestTosendanytypeofrequesttoablockdevice,astructrequestcontainermustbeallocatedfirst.S--sleepNoavailablerequeststructureswereavailable,sotheissuerhastowaitforonetobefreed.14Tracepoint解释(续）P--plugWheni/oisqueuedtoapreviouslyemptyblockdevicequeue,Linuxwillplugthequeueinanticipationoffutureiosbeingaddedbeforethisdataisneeded.U--unplugSomerequestdataalreadyqueuedinthedevice,startsendingrequeststothedriver.T--unplugduetotimerIfnobodyrequeststhei/othatwasqueuedafterpluggingthequeue,Linuxwillautomaticallyunplugitafteradefinedperiodhaspassed.X--splitOnraidordevicemappersetups,anincomingi/omaystraddleadeviceorinternalzoneandneedstobehoppedupintosmallerpiecesforservice.A--remapForstackeddevices,incomingi/oisremappedtodevicebelowitinthei/ostack.15思考如何可视化IO请求生命期？16IO行为观察17不觉得信息量太少吗？blktrace架构图18blktrace可过滤事件barrier:barrierattributecomplete:completedbydriverfs:requestsissue:issuedtodriverpc:packetcommandeventsqueue:queueoperationsread:readtracesrequeue:requeueoperationssync:synchronousattributewrite:writetracesnotify:tracemessagesdrv_data:additionaldriverspecifictrace19btrace第一感20blkiomon21btt#blktrace/dev/sdb#blkparse-isdb-dsdb.bin#blkrawverifysdb#btt-isdb.bin-A22btt:LifeofanI/O•Q2I–timeittakestoprocessanI/Opriortoitbeinginsertedormergedontoarequestqueue–Includessplit,andremaptime•I2D–timetheI/Ois“idle”ontherequestqueue•D2C–timetheI/Ois“active”inthedriverandonthedevice•Q2I+I2D+D2C=Q2C–Q2C:TotalprocessingtimeoftheI/O23btt解读====================AllDevices====================ALLMINAVGMAXN-----------------------------------------------------------------Q2Q0.0000070980.0853237521.18953484914Q2G0.0000006850.0000017370.00000475712G2I0.0000002720.0000017240.00000424012Q2M0.0000004750.0000010360.0000013623I2D0.0000025020.0002446330.00223865112M2D0.0000048700.0000650110.0001787223D2C0.0000554880.0001457200.00021906815Q2C0.0000620480.0003574050.0023037581524btt解读(续)====================DeviceOverhead====================DEV|Q2GG2IQ2MI2DD2C----------|---------------------------------------------(8,16)|0.3889%0.3859%0.0580%54.7575%40.7717%----------|---------------------------------------------Overall|0.3889%0.3859%0.0580%54.7575%40.7717%25btt解读(续)====================DeviceMergeInformation====================DEV|#Q#DRatio|BLKminBLKavgBLKmaxTotal----------|-----------------------|--------------------------------(8,16)|15121.2|8102412026btt解读(续)====================DeviceQ2QSeekInformation====================DEV|NSEEKSMEANMEDIAN|MODE----------|---------------------------------------------|---------------(8,16)|15620978236.70|0(5)----------|---------------------------------------------|---------------Overall|NSEEKSMEANMEDIAN|MODEAverage|15620978236.70|0(5)====================DeviceD2DSeekInformation====================DEV|NSEEKSMEANMEDIAN|MODE----------|---------------------------------------------|---------------(8,16)|1277622