leandrof/linux
1
0
Fork 0
forked from Minki/linux
Code Issues Pull Requests Packages Projects Releases Wiki Activity
86bba269d0
linux/net/core
History
Eric Dumazet 86bba269d0 [PATCH] NET : convert IP route cache garbage collection from softirq processing to a workqueue 
When the periodic IP route cache flush is done (every 600 seconds on
default configuration), some hosts suffer a lot and eventually trigger
the "soft lockup" message.

dst_run_gc() is doing a scan of a possibly huge list of dst_entries,
eventually freeing some (less than 1%) of them, while holding the
dst_lock spinlock for the whole scan.

Then it rearms a timer to redo the full thing 1/10 s later...
The slowdown can last one minute or so, depending on how active are
the tcp sessions.

This second version of the patch converts the processing from a softirq
based one to a workqueue.

Even if the list of entries in garbage_list is huge, host is still
responsive to softirqs and can make progress.

Instead of resetting gc timer to 0.1 second if one entry was freed in a
gc run, we do this if more than 10% of entries were freed.

Before patch :

Aug 16 06:21:37 SRV1 kernel: BUG: soft lockup detected on CPU#0!
Aug 16 06:21:37 SRV1 kernel:
Aug 16 06:21:37 SRV1 kernel: Call Trace:
Aug 16 06:21:37 SRV1 kernel:  <IRQ>  [<ffffffff802286f0>] wake_up_process+0x10/0x20
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff80251e09>] softlockup_tick+0xe9/0x110
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803cd380>] dst_run_gc+0x0/0x140
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff802376f3>] run_local_timers+0x13/0x20
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff802379c7>] update_process_times+0x57/0x90
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff80216034>] smp_local_timer_interrupt+0x34/0x60
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff802165cc>] smp_apic_timer_interrupt+0x5c/0x80
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff8020a816>] apic_timer_interrupt+0x66/0x70
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803cd3d3>] dst_run_gc+0x53/0x140
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803cd3c6>] dst_run_gc+0x46/0x140
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff80237148>] run_timer_softirq+0x148/0x1c0
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff8023340c>] __do_softirq+0x6c/0xe0
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff8020ad6c>] call_softirq+0x1c/0x30
Aug 16 06:21:37 SRV1 kernel:  <EOI>  [<ffffffff8020cb34>] do_softirq+0x34/0x90
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff802331cf>] local_bh_enable_ip+0x3f/0x60
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff80422913>] _spin_unlock_bh+0x13/0x20
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803dfde8>] rt_garbage_collect+0x1d8/0x320
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803cd4dd>] dst_alloc+0x1d/0xa0
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803e1433>] __ip_route_output_key+0x573/0x800
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803c02e2>] sock_common_recvmsg+0x32/0x50
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803e16dc>] ip_route_output_flow+0x1c/0x60
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff80400160>] tcp_v4_connect+0x150/0x610
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803ebf07>] inet_bind_bucket_create+0x17/0x60
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff8040cd16>] inet_stream_connect+0xa6/0x2c0
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff80422981>] _spin_lock_bh+0x11/0x30
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803c0bdf>] lock_sock_nested+0xcf/0xe0
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff80422981>] _spin_lock_bh+0x11/0x30
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803be551>] sys_connect+0x71/0xa0
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803eee3f>] tcp_setsockopt+0x1f/0x30
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803c030f>] sock_common_setsockopt+0xf/0x20
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff803be4bd>] sys_setsockopt+0x9d/0xc0
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff8028881e>] sys_ioctl+0x5e/0x80
Aug 16 06:21:37 SRV1 kernel:  [<ffffffff80209c4e>] system_call+0x7e/0x83

After patch : (RT_CACHE_DEBUG set to 2 to get following traces)

dst_total: 75469 delayed: 74109 work_perf: 141 expires: 150 elapsed: 8092 us
dst_total: 78725 delayed: 73366 work_perf: 743 expires: 400 elapsed: 8542 us
dst_total: 86126 delayed: 71844 work_perf: 1522 expires: 775 elapsed: 8849 us
dst_total: 100173 delayed: 68791 work_perf: 3053 expires: 1256 elapsed: 9748 us
dst_total: 121798 delayed: 64711 work_perf: 4080 expires: 1997 elapsed: 10146 us
dst_total: 154522 delayed: 58316 work_perf: 6395 expires: 25 elapsed: 11402 us
dst_total: 154957 delayed: 58252 work_perf: 64 expires: 150 elapsed: 6148 us
dst_total: 157377 delayed: 57843 work_perf: 409 expires: 400 elapsed: 6350 us
dst_total: 163745 delayed: 56679 work_perf: 1164 expires: 775 elapsed: 7051 us
dst_total: 176577 delayed: 53965 work_perf: 2714 expires: 1389 elapsed: 8120 us
dst_total: 198993 delayed: 49627 work_perf: 4338 expires: 1997 elapsed: 8909 us
dst_total: 226638 delayed: 46865 work_perf: 2762 expires: 2748 elapsed: 7351 us

I successfully reduced the IP route cache of many hosts by a four factor
thanks to this patch. Previously, I had to disable "ip route flush cache"
to avoid crashes.

Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-10-10 16:49:15 -07:00
..
datagram.c [NET]: Do not dereference iov if length is zero 2007-09-11 10:29:07 +02:00
dev_mcast.c [NET]: Make the device list and device lookups per namespace. 2007-10-10 16:49:10 -07:00
dev.c [NET]: Implement network device movement between namespaces 2007-10-10 16:49:12 -07:00
dst.c [PATCH] NET : convert IP route cache garbage collection from softirq processing to a workqueue 2007-10-10 16:49:15 -07:00
ethtool.c [NET]: Make the device list and device lookups per namespace. 2007-10-10 16:49:10 -07:00
fib_rules.c [NET]: Make the device list and device lookups per namespace. 2007-10-10 16:49:10 -07:00
filter.c [SK_BUFF]: Convert skb->tail to sk_buff_data_t 2007-04-25 22:26:28 -07:00
flow.c mm: Remove slab destructors from kmem_cache_create(). 2007-07-20 10:11:58 +09:00
gen_estimator.c [NET]: gen_estimator deadlock fix 2007-07-18 01:46:50 -07:00
gen_stats.c [SK_BUFF]: Convert skb->tail to sk_buff_data_t 2007-04-25 22:26:28 -07:00
iovec.c [PATCH] remove many unneeded #includes of sched.h 2007-02-14 08:09:54 -08:00
kmap_skb.h
link_watch.c [NET] link_watch: Always schedule urgent events 2007-05-10 23:45:28 -07:00
Makefile [NET]: Basic network namespace infrastructure. 2007-10-10 16:49:03 -07:00
neighbour.c [NET]: Make the device list and device lookups per namespace. 2007-10-10 16:49:10 -07:00
net_namespace.c [NET]: #if 0 out net_alloc() for now. 2007-10-10 16:49:14 -07:00
net-sysfs.c [NET]: Make NAPI polling independent of struct net_device objects. 2007-10-10 16:47:45 -07:00
netevent.c [NET]: net/core/netevent.c should #include <net/netevent.h> 2007-07-05 17:40:27 -07:00
netpoll.c [NET]: Make the device list and device lookups per namespace. 2007-10-10 16:49:10 -07:00
pktgen.c [NET]: Make the device list and device lookups per namespace. 2007-10-10 16:49:10 -07:00
request_sock.c
rtnetlink.c [NET]: netlink support for moving devices between network namespaces. 2007-10-10 16:49:13 -07:00
scm.c O_CLOEXEC for SCM_RIGHTS 2007-07-16 09:05:45 -07:00
skbuff.c mm: Remove slab destructors from kmem_cache_create(). 2007-07-20 10:11:58 +09:00
sock.c [NET]: Make the device list and device lookups per namespace. 2007-10-10 16:49:10 -07:00
stream.c
sysctl_net_core.c [XFRM]: Allow XFRM_ACQ_EXPIRES to be tunable via sysctl. 2007-05-31 01:23:23 -07:00
user_dma.c [NET]: Revert sk_buff walker cleanups. 2007-04-27 15:21:23 -07:00
utils.c [NET] net/core/utils: fix sparse warning 2007-08-07 18:02:43 -07:00