（dat の URL に変換した上で）crawld に投げてるだけなんですが，
----------------------------------------------------------------------
last pid: 74880; load averages: 0.02, 0.06, 0.04 up 15+18:27:52 15:22:53
170 processes: 1 running, 169 sleeping
CPU states: 0.2% user, 0.0% nice, 0.5% system, 0.2% interrupt, 99.2% idle
Mem: 64M Active, 1659M Inact, 196M Wired, 81M Cache, 112M Buf, 2996K Free
Swap: 2048M Total, 2048M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
74627 c22chio 1 4 0 5452K 4120K kqread 0 5:37 4.54% crawld
----------------------------------------------------------------------
CPU の能力的には余裕っぽいですね．ただ，
----------------------------------------------------------------------
[crawld statistics] Tue, 19 Dec 2006 14:04:38.945 (JST)
user CPU time = 0:00:11.052, system CPU time = 0:00:33.811
elapsed time = 0:13:18.542, CPU load = 5.62%
total workers = 8, idle workers = 0
minor page faults = 3656, major page faults = 0, swaps = 0
block inputs = 3837, block outputs = 3329
messages sent = 28359, messages received = 691574
signals = 5, vol ctx switches = 664364, invol ctx switches = 60839
URLs: input = 55614, done = 25802, error = 1445
----------------------------------------------------------------------
[crawld statistics] Tue, 19 Dec 2006 15:23:19.866 (JST)
user CPU time = 0:01:25.717, system CPU time = 0:04:12.259
elapsed time = 1:31:59.462, CPU load = 6.12%
total workers = 8, idle workers = 0
minor page faults = 63546, major page faults = 0, swaps = 0
block inputs = 111588, block outputs = 17339
messages sent = 385824, messages received = 3511819
signals = 7, vol ctx switches = 3126563, invol ctx switches = 500464
URLs: input = 376259, done = 355261, error = 20952
----------------------------------------------------------------------
動かし始めの dat をガンガン転送してる時は URL の input に対し
done が追い付いてない感じ，一方ずっと動かしてて重複した URL への
304 レスポンスが増えてくると差が縮まって追い付いてきてる感じかなぁ．
これを見ると，ネックはネットワーク帯域？
