Defcon Kaklkaklkekl (Wiki)

====== Testing Linux software raid configuration ====== ** Work in progress, these are rough notes...** <code> Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 5698 14 5692 5 4596 8 27752 63 70331 34 171.0 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 613 12 +++++ +++ 441 8 582 12 +++++ +++ 410 8 filez,8G,5698,14,5692,5,4596,8,27752,63,70331,34,171.0,0,16,613,12,+++++,+++,441,8,582,12,+++++,+++,410,8 </code> {{:testing:raidtest-char-read.png}} {{:testing:raidtest-block-read.png}} {{:testing:raidtest-char-write.png}} {{:testing:raidtest-block-write.png}} {{:testing:raidtest-seq-rewrite.png}} {{:testing:raidtest-random-seek.png}} {{:testing:raidtest-file-ops.png}} <code> En Hitachi-disk på Adaptec-kontroller, satt opp som Volume med Write-cache på kontroller. En partisjon på disk, 16GB.. Filsystemet er EXT3, opprettet med mkfs.ext3 -b 4096 -i 4096 -m 0 /dev/sdb1 Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 7375 35 7442 24 5718 11 28081 71 63743 45 219.4 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 5968 97 +++++ +++ 7104 99 6789 93 +++++ +++ 8057 99 filez,8G,7375,35,7442,24,5718,11,28081,71,63743,45,219.4,1,16,5968,97,+++++,+++,7104,99,6789,93,+++++,+++,8057,99 An Hitachi-disk på Asus-onbard AHCIkontroller, samme partisjonsoppsett som over: Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 21308 81 30172 69 37773 48 39839 94 127851 87 261.0 2 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6175 96 +++++ +++ 6875 99 6974 96 +++++ +++ 7023 99 filez,8G,21308,81,30172,69,37773,48,39839,94,127851,87,261.0,2,16,6175,96,+++++,+++,6875,99,6974,96,+++++,+++,7023,99 SÅ, over SATA2 skal hver disk kunne levere 20(29)/40(125) i det minste... mdadm \ --create --verbose /dev/md0 \ --level=raid5 \ --metadata=1 \ --raid-devices=4 \ /dev/sdb \ /dev/sdc \ /dev/sdd \ /dev/sde \ --chunk=256 pvcreate /dev/md0 vgcreate -s 32M raid5v /dev/md0 lvcreate -n test -L 32G raid5v mkfs.ext3 -b 4096 -E stride=64 -i 4096 -m 0 /dev/raid5v/test Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 9513 49 14860 50 16499 40 22149 74 31935 41 300.6 3 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 3885 98 +++++ +++ 4938 91 2157 53 +++++ +++ 2358 43 filez,8G,9513,49,14860,50,16499,40,22149,74,31935,41,300.6,3,16,3885,98,+++++,+++,4938,91,2157,53,+++++,+++,2358,43 Samme RAID5-sett rebuild/init var ferdig: Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 14559 69 20433 60 20453 50 26155 92 50916 87 441.5 5 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 3880 96 +++++ +++ 5074 91 3918 98 +++++ +++ 5181 94 filez,8G,14559,69,20433,60,20453,50,26155,92,50916,87,441.5,5,16,3880,96,+++++,+++,5074,91,3918,98,+++++,+++,5181,94 Samme RAID5-sett, men mkfs.ext3 uten -E stride=64 opsjonen Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 13725 67 17437 59 19201 49 26382 92 52236 86 455.7 4 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 3916 94 +++++ +++ 4963 91 4877 97 +++++ +++ 5486 92 filez,8G,13725,67,17437,59,19201,49,26382,92,52236,86,455.7,4,16,3916,94,+++++,+++,4963,91,4877,97,+++++,+++,5486,92 mdadm \ --create --verbose /dev/md0 \ --level=raid5 \ --metadata=1 \ --raid-devices=4 \ /dev/sdb \ /dev/sdc \ /dev/sdd \ /dev/sde \ --chunk=128 pvcreate /dev/md0 vgcreate -s 32M raid5v /dev/md0 lvcreate -n test -L 32G raid5v mkfs.ext3 -b 4096 -E stride=32 -i 4096 -m 0 /dev/raid5v/test under build Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 7881 34 11826 31 17959 28 22926 61 33085 24 153.5 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1528 27 +++++ +++ 1783 23 1751 27 +++++ +++ 2629 36 filez,8G,7881,34,11826,31,17959,28,22926,61,33085,24,153.5,1,16,1528,27,+++++,+++,1783,23,1751,27,+++++,+++,2629,36 Etter build Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 16305 68 21675 55 23201 44 31438 92 78377 86 447.8 4 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7215 97 +++++ +++ 7768 99 8037 93 +++++ +++ 8177 99 filez,8G,16305,68,21675,55,23201,44,31438,92,78377,86,447.8,4,16,7215,97,+++++,+++,7768,99,8037,93,+++++,+++,8177,99 mdadm \ --create --verbose /dev/md0 \ --level=raid5 \ --metadata=1 \ --raid-devices=4 \ /dev/sdb \ /dev/sdc \ /dev/sdd \ /dev/sde \ --chunk=64 pvcreate /dev/md0 vgcreate -s 32M raid5v /dev/md0 lvcreate -n test -L 32G raid5v mkfs.ext3 -b 4096 -E stride=16 -i 4096 -m 0 /dev/raid5v/test Under build: Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 7009 33 11222 32 14150 16 16201 47 21514 28 164.9 2 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1543 20 28677 42 1705 24 2142 26 27715 16 2317 29 filez,8G,7009,33,11222,32,14150,16,16201,47,21514,28,164.9,2,16,1543,20,28677,42,1705,24,2142,26,27715,16,2317,29 Etter build-complete Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 16100 66 21336 54 22765 45 28652 93 61033 89 440.1 4 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7692 98 +++++ +++ 7406 98 7324 92 +++++ +++ 7893 98 filez,8G,16100,66,21336,54,22765,45,28652,93,61033,89,440.1,4,16,7692,98,+++++,+++,7406,98,7324,92,+++++,+++,7893,98 echo 8192 > /sys/block/md0/md/stripe_cache_size Status for cache skal kunne leses fra /sys/block/md0/md/stripe_cache_active Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 15685 64 21053 54 21318 44 27955 91 58725 87 436.2 4 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7190 97 +++++ +++ 7234 96 7140 92 +++++ +++ 7928 95 filez,8G,15685,64,21053,54,21318,44,27955,91,58725,87,436.2,4,16,7190,97,+++++,+++,7234,96,7140,92,+++++,+++,7928,95 echo 512 > /sys/block/md0/md/stripe_cache_size Version 1.03d ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP filez 8G 15716 65 21201 53 22391 46 29170 91 64344 87 442.4 5 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7258 96 +++++ +++ 7224 91 7335 97 +++++ +++ 7691 91 filez,8G,15716,65,21201,53,22391,46,29170,91,64344,87,442.4,5,16,7258,96,+++++,+++,7224,91,7335,97,+++++,+++,7691,91 </code>