5日ほど経過してL2ARCの熟成具合をみてみると、Hit Ratioが49%となかなか高い。でもARC Sizeが2.57GiBと小さいのはなぜだろう。Target Sizeは4.34GiBなので全然埋まってない。
あと気になるのはtop -aHS (-aはargv表示、-Hはスレッド表示、-Sはシステムプロセスも表示)で観察していると、l2arc_feed_threadが1%ほどCPUを喰いつづけている。vfs.zfs.l2arc_feed_secs=1秒間隔でチェックしてるだけで1%ということは1回まわるのに1msかかっているということで、けっこう重い処理なんだなぁ。WITNESSを有効にしているので一時的にdebug.witness.watch=0にしてみて負荷が軽くなるかとおもったら、まったく変化なし。
% zpool iostat -v tank
capacity operations bandwidth
pool alloc free read write read write
----------- ----- ----- ----- ----- ----- -----
tank 884G 1.07T 43 13 1.87M 213K
raidz1 884G 1.07T 43 13 1.87M 192K
ada1s2 - - 15 6 1.07M 97.2K
ada2s2 - - 16 6 1.07M 97.2K
ada3s2 - - 15 6 1.07M 97.2K
logs - - - - - -
label/zil 1.44M 1.95G 0 0 3 21.1K
cache - - - - - -
ada4 46.6G 72.6G 9 2 54.9K 144K
----------- ----- ----- ----- ----- ----- -----
% arc_summary ------------------------------------------------------------------------ ZFS Subsystem Report Mon Dec 12 10:48:33 2011 ------------------------------------------------------------------------ System Memory: 1.60% 174.12 MiB Active, 2.58% 280.89 MiB Inact 84.55% 8.99 GiB Wired, 0.14% 15.25 MiB Cache 11.12% 1.18 GiB Free, 0.02% 2.02 MiB Gap Real Installed: 12.00 GiB Real Available: 91.54% 10.98 GiB Real Managed: 96.82% 10.64 GiB Logical Total: 12.00 GiB Logical Used: 87.74% 10.53 GiB Logical Free: 12.26% 1.47 GiB Kernel Memory: 304.88 MiB DATA: 91.44% 278.79 MiB TEXT: 8.56% 26.08 MiB KMAP: 5.61 GiB FREE: 4.69% 269.40 MiB Page: 1 ------------------------------------------------------------------------ ARC Summary: (THROTTLED) Storage pool Version: 28 Filesystem Version: 5 Memory Throttle Count: 7 ARC Misc: Deleted: 3.41m Recycle Misses: 2.56m Mutex Misses: 12.40k Evict Skips: 12.40k ARC Size: 42.77% 2.57 GiB Target Size: (Adaptive) 72.30% 4.34 GiB Min Size (Hard Limit): 66.67% 4.00 GiB Max Size (High Water): 1:1 6.00 GiB ARC Size Breakdown: Recently Used Cache Size: 12.40% 550.78 MiB Frequently Used Cache Size: 87.60% 3.80 GiB ARC Hash Breakdown: Elements Max: 1.14m Elements Current: 98.64% 1.12m Collisions: 3.11m Chain Max: 17 Chains: 241.78k Page: 2 ------------------------------------------------------------------------ ARC Efficiency: 97.95m Cache Hit Ratio: 93.06% 91.15m Cache Miss Ratio: 6.94% 6.80m Actual Hit Ratio: 80.38% 78.73m Data Demand Efficiency: 94.23% 30.25m Data Prefetch Efficiency: 56.08% 457.86k CACHE HITS BY CACHE LIST: Anonymously Used: 9.75% 8.88m Most Recently Used: 5.75% 5.24m Most Frequently Used: 80.62% 73.49m Most Recently Used Ghost: 1.51% 1.37m Most Frequently Used Ghost: 2.37% 2.16m CACHE HITS BY DATA TYPE: Demand Data: 31.28% 28.51m Prefetch Data: 0.28% 256.78k Demand Metadata: 52.71% 48.04m Prefetch Metadata: 15.73% 14.34m CACHE MISSES BY DATA TYPE: Demand Data: 25.68% 1.75m Prefetch Data: 2.96% 201.08k Demand Metadata: 40.86% 2.78m Prefetch Metadata: 30.50% 2.07m Page: 3 ------------------------------------------------------------------------ L2 ARC Summary: (HEALTHY) Passed Headroom: 2.51m Tried Lock Failures: 6.73m IO In Progress: 38.63k Low Memory Aborts: 33 Free on Write: 3.36k Writes While Full: 4.03k R/W Clashes: 541 Bad Checksums: 0 IO Errors: 0 SPA Mismatch: 539.89m L2 ARC Size: (Adaptive) 36.97 GiB Header Size: 0.48% 181.02 MiB L2 ARC Breakdown: 6.80m Hit Ratio: 48.99% 3.33m Miss Ratio: 51.01% 3.47m Feeds: 340.70k L2 ARC Buffer: Bytes Scanned: 722.28 TiB Buffer Iterations: 340.70k List Iterations: 21.76m NULL List Iterations: 769.00k L2 ARC Writes: Writes Sent: 100.00% 33.61k Page: 4 ------------------------------------------------------------------------ File-Level Prefetch: (HEALTHY) DMU Efficiency: 288.43m Hit Ratio: 73.56% 212.17m Miss Ratio: 26.44% 76.26m Colinear: 76.26m Hit Ratio: 0.01% 7.12k Miss Ratio: 99.99% 76.25m Stride: 210.90m Hit Ratio: 99.97% 210.84m Miss Ratio: 0.03% 64.77k DMU Misc: Reclaim: 76.25m Successes: 0.23% 174.33k Failures: 99.77% 76.08m Streams: 1.33m +Resets: 0.06% 841 -Resets: 99.94% 1.33m Bogus: 0 Page: 5 ------------------------------------------------------------------------ VDEV Cache Summary: 11.57m Hit Ratio: 57.01% 6.60m Miss Ratio: 22.79% 2.64m Delegations: 20.20% 2.34m Page: 6 ------------------------------------------------------------------------ ZFS Tunable (sysctl): kern.maxusers=384 vfs.zfs.l2c_only_size=36178498560 vfs.zfs.mfu_ghost_data_lsize=352977920 vfs.zfs.mfu_ghost_metadata_lsize=343705600 vfs.zfs.mfu_ghost_size=696683520 vfs.zfs.mfu_data_lsize=826750976 vfs.zfs.mfu_metadata_lsize=424857088 vfs.zfs.mfu_size=1475247616 vfs.zfs.mru_ghost_data_lsize=192281088 vfs.zfs.mru_ghost_metadata_lsize=3768553472 vfs.zfs.mru_ghost_size=3960834560 vfs.zfs.mru_data_lsize=318336000 vfs.zfs.mru_metadata_lsize=55524352 vfs.zfs.mru_size=389883904 vfs.zfs.anon_data_lsize=0 vfs.zfs.anon_metadata_lsize=0 vfs.zfs.anon_size=3495936 vfs.zfs.l2arc_norw=1 vfs.zfs.l2arc_feed_again=1 vfs.zfs.l2arc_noprefetch=1 vfs.zfs.l2arc_feed_min_ms=200 vfs.zfs.l2arc_feed_secs=1 vfs.zfs.l2arc_headroom=10 vfs.zfs.l2arc_write_boost=8388608 vfs.zfs.l2arc_write_max=8388608 vfs.zfs.arc_meta_limit=1610612736 vfs.zfs.arc_meta_used=1610368104 vfs.zfs.arc_min=4294967296 vfs.zfs.arc_max=6442450944 vfs.zfs.dedup.prefetch=1 vfs.zfs.mdcomp_disable=0 vfs.zfs.write_limit_override=0 vfs.zfs.write_limit_inflated=35384819712 vfs.zfs.write_limit_max=1474367488 vfs.zfs.write_limit_min=33554432 vfs.zfs.write_limit_shift=3 vfs.zfs.no_write_throttle=0 vfs.zfs.zfetch.array_rd_sz=1048576 vfs.zfs.zfetch.block_cap=256 vfs.zfs.zfetch.min_sec_reap=2 vfs.zfs.zfetch.max_streams=8 vfs.zfs.prefetch_disable=0 vfs.zfs.mg_alloc_failures=8 vfs.zfs.check_hostid=1 vfs.zfs.recover=0 vfs.zfs.txg.synctime_ms=1000 vfs.zfs.txg.timeout=5 vfs.zfs.scrub_limit=10 vfs.zfs.vdev.cache.bshift=16 vfs.zfs.vdev.cache.size=104857600 vfs.zfs.vdev.cache.max=16384 vfs.zfs.vdev.write_gap_limit=4096 vfs.zfs.vdev.read_gap_limit=32768 vfs.zfs.vdev.aggregation_limit=131072 vfs.zfs.vdev.ramp_rate=2 vfs.zfs.vdev.time_shift=6 vfs.zfs.vdev.min_pending=4 vfs.zfs.vdev.max_pending=10 vfs.zfs.vdev.bio_flush_disable=0 vfs.zfs.cache_flush_disable=0 vfs.zfs.zil_replay_disable=0 vfs.zfs.zio.use_uma=1 vfs.zfs.version.zpl=5 vfs.zfs.version.spa=28 vfs.zfs.version.acl=1 vfs.zfs.debug=0 vfs.zfs.super_owner=0 vm.kmem_size=8589934592 vm.kmem_size_scale=1 vm.kmem_size_min=0 vm.kmem_size_max=329853485875 Page: 7 ------------------------------------------------------------------------
