leandrof/linux
1
0
Fork 0
forked from Minki/linux
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8b27dae5a2
linux/include/net
History
Eric Dumazet 8b27dae5a2 tcp: add one skb cache for rx 
Often times, recvmsg() system calls and BH handling for a particular
TCP socket are done on different cpus.

This means the incoming skb had to be allocated on a cpu,
but freed on another.

This incurs a high spinlock contention in slab layer for small rpc,
but also a high number of cache line ping pongs for larger packets.

A full size GRO packet might use 45 page fragments, meaning
that up to 45 put_page() can be involved.

More over performing the __kfree_skb() in the recvmsg() context
adds a latency for user applications, and increase probability
of trapping them in backlog processing, since the BH handler
might found the socket owned by the user.

This patch, combined with the prior one increases the rpc
performance by about 10 % on servers with large number of cores.

(tcp_rr workload with 10,000 flows and 112 threads reach 9 Mpps
 instead of 8 Mpps)

This also increases single bulk flow performance on 40Gbit+ links,
since in this case there are often two cpus working in tandem :

 - CPU handling the NIC rx interrupts, feeding the receive queue,
  and (after this patch) freeing the skbs that were consumed.

 - CPU in recvmsg() system call, essentially 100 % busy copying out
  data to user space.

Having at most one skb in a per-socket cache has very little risk
of memory exhaustion, and since it is protected by socket lock,
its management is essentially free.

Note that if rps/rfs is used, we do not enable this feature, because
there is high chance that the same cpu is handling both the recvmsg()
system call and the TCP rx path, but that another cpu did the skb
allocations in the device driver right before the RPS/RFS logic.

To properly handle this case, it seems we would need to record
on which cpu skb was allocated, and use a different channel
to give skbs back to this cpu.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Acked-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-03-23 21:57:38 -04:00
..
9p 9p: Add refcount to p9_req_t 2018-09-08 01:39:47 +09:00
bluetooth Bluetooth: Add quirk for reading BD_ADDR from fwnode property 2019-02-26 10:08:26 +01:00
caif caif: reduce stack size with KASAN 2018-01-19 14:02:12 -05:00
iucv net/af_iucv: locate IUCV header via skb_network_header() 2018-09-26 09:56:07 -07:00
netfilter netfilter: nf_tables: bogus EBUSY when deleting set after flush 2019-03-11 13:19:24 +01:00
netns ipv6: Add icmp_echo_ignore_anycast for ICMPv6 2019-03-20 16:29:37 -07:00
nfc NFC: Fix the number of pipes 2018-09-18 19:55:01 -07:00
phonet phonet: fix building with clang 2019-02-21 16:23:56 -08:00
sctp sctp: get sctphdr by offset in sctp_compute_cksum 2019-03-18 18:16:12 -07:00
tc_act net: Change TCA_ACT_* to TCA_ID_* to match that of TCA_ID_POLICE 2019-02-10 09:28:43 -08:00
6lowpan.h
act_api.h net: Change TCA_ACT_* to TCA_ID_* to match that of TCA_ID_POLICE 2019-02-10 09:28:43 -08:00
addrconf.h ipv6_stub: add ipv6_route_input stub/proxy. 2019-02-13 18:27:55 -08:00
af_ieee802154.h ieee802154: add rx LQI from userspace 2018-07-13 12:18:18 -04:00
af_rxrpc.h Revert "rxrpc: Allow failed client calls to be retried" 2019-01-15 21:33:36 -08:00
af_unix.h net: split out functions related to registering inflight socket files 2019-02-28 08:24:23 -07:00
af_vsock.h vsock: split dwork to avoid reinitializations 2018-08-07 12:39:13 -07:00
ah.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
arp.h ipv4: Make neigh lookup keys for loopback/point-to-point devices be INADDR_ANY 2018-01-15 14:53:43 -05:00
atmclip.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ax25.h ax25: fix possible use-after-free 2019-01-23 11:18:00 -08:00
ax88796.h net-next: ax88796: add interrupt status callback to platform data 2018-04-19 16:11:11 -04:00
bond_3ad.h bonding: count master 3ad stats separately 2019-01-24 22:18:48 -08:00
bond_alb.h
bond_options.h
bonding.h bonding: avoid possible dead-lock 2018-09-26 20:22:19 -07:00
busy_poll.h net: remove sock_poll_busy_flag 2018-07-30 09:10:25 -07:00
calipso.h net, calipso: convert calipso_doi.refcount from atomic_t to refcount_t 2017-07-04 22:35:16 +01:00
cfg80211-wext.h
cfg80211.h cfg80211: allow sending vendor events unicast 2019-02-22 13:46:43 +01:00
cfg802154.h
checksum.h Remove 'type' argument from access_ok() function 2019-01-03 18:57:57 -08:00
cipso_ipv4.h net, ipv4: convert cipso_v4_doi.refcount from atomic_t to refcount_t 2017-07-04 01:29:04 -07:00
cls_cgroup.h
codel_impl.h
codel_qdisc.h
codel.h
compat.h net: remove compat_sys_*() prototypes from net/compat.h 2018-04-02 20:16:17 +02:00
datalink.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dcbevent.h
dcbnl.h net: dcb: Add priority-to-DSCP map getters 2018-07-27 13:17:50 -07:00
devlink.h devlink: Add support for direct reporter health state update 2019-03-04 11:00:43 -08:00
dn_dev.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dn_fib.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dn_neigh.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dn_nsp.h net/decnet: Convert timers to use timer_setup() 2017-10-18 12:39:36 +01:00
dn_route.h decnet: Move dn_next into decnet route structure. 2017-11-30 09:54:25 -05:00
dn.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2017-11-04 09:26:51 +09:00
dsa.h net: dsa: add KSZ9893 switch tagging support 2019-03-03 13:48:49 -08:00
dsfield.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dst_cache.h net: core: dst_cache_set_ip6: Rename 'addr' parameter to 'saddr' for consistency 2018-03-05 12:52:45 -05:00
dst_metadata.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2017-11-04 09:26:51 +09:00
dst_ops.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dst.h net: dst: remove gc leftovers 2019-03-21 13:39:25 -07:00
erspan.h erspan: set bso bit based on mirrored packet's len 2018-05-20 18:31:42 -04:00
esp.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ethoc.h inet: whitespace cleanup 2018-02-28 11:43:28 -05:00
failover.h net: Introduce generic failover module 2018-05-28 22:59:54 -04:00
fib_notifier.h net: Add extack to fib_notifier_info 2017-11-01 11:50:43 +09:00
fib_rules.h net: fib_rules: add extack support 2018-04-23 10:21:24 -04:00
firewire.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
flow_dissector.h net/flow_dissector: correct comments on enum flow_dissector_key_id 2018-11-30 13:21:52 -08:00
flow_offload.h flow_offload: fix block stats 2019-02-13 21:07:47 -08:00
flow.h route: Add multipath_hash in flowi_common to make user-define hash 2019-02-27 12:50:17 -08:00
fou.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
fq_impl.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2017-10-30 21:09:24 +09:00
fq.h fq: support filtering a given tin 2017-10-11 09:49:34 +02:00
garp.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
gen_stats.h net: align gnet_stats_basic_cpu struct 2018-11-17 21:37:29 -08:00
genetlink.h genetlink: make policy common to family 2019-03-22 10:38:23 -04:00
geneve.h net: add netif_is_geneve() 2018-11-07 23:00:23 -08:00
gre.h net: Add netif_is_gretap()/netif_is_ip6gretap() 2018-12-10 15:53:04 -08:00
gro_cells.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
gtp.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
gue.h fou: fix some member types in guehdr 2017-12-11 14:10:06 -05:00
hwbm.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
icmp.h net: Add __icmp_send helper. 2019-02-25 14:32:35 -08:00
ieee80211_radiotap.h wireless-drivers-next patches for 5.1 2019-02-22 12:56:24 -08:00
ieee802154_netdev.h
if_inet6.h net/ipv6: Add anycast addresses to a global hashtable 2018-11-02 23:54:56 -07:00
ife.h net: sched: ife: handle malformed tlv length 2018-04-22 21:12:00 -04:00
ila.h
inet6_connection_sock.h
inet6_hashtables.h net: allow binding socket in a VRF when there's an unbound socket 2018-11-07 16:12:38 -08:00
inet_common.h net: use indirect call wrappers at GRO transport layer 2018-12-15 13:23:02 -08:00
inet_connection_sock.h tcp: change pingpong threshold to 3 2019-01-27 13:29:43 -08:00
inet_ecn.h inet: Refactor INET_ECN_decapsulate() 2018-10-17 17:45:07 -07:00
inet_frag.h net: remove unused struct inet_frag_queue.fragments field 2019-02-26 08:27:05 -08:00
inet_hashtables.h net: dccp: fix kernel crash on module load 2018-12-24 15:27:56 -08:00
inet_sock.h net: ensure unbound stream socket to be chosen when not in a VRF 2018-11-07 16:12:38 -08:00
inet_timewait_sock.h net-tcp: remove useless tw_timeout field 2018-06-05 10:45:24 -04:00
inetpeer.h net: ipv4: use a dedicated counter for icmp_v4 redirect packets 2019-02-08 21:50:15 -08:00
ip6_checksum.h
ip6_fib.h ipv6: Change addrconf_f6i_alloc to use ip6_route_info_create 2019-03-21 10:16:54 -07:00
ip6_route.h net: Add struct for fib dump filter 2018-10-16 00:13:12 -07:00
ip6_tunnel.h udp: Support for error handlers of tunnels with arbitrary destination port 2018-11-08 17:13:08 -08:00
ip_fib.h net: ipv4: Fix memory leak in network namespace dismantle 2019-01-15 13:33:44 -08:00
ip_tunnels.h route: Add multipath_hash in flowi_common to make user-define hash 2019-02-27 12:50:17 -08:00
ip_vs.h ipvs: avoid indirect calls when calculating checksums 2019-01-28 11:15:58 +01:00
ip.h ipv4: Allow amount of dirty memory from fib resizing to be controllable 2019-03-21 13:29:53 -07:00
ipcomp.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ipconfig.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ipv6_frag.h net: IP6 defrag: use rbtrees for IPv6 defrag 2019-01-25 21:37:11 -08:00
ipv6.h ipv6: factor out protocol delivery helper 2018-11-07 16:23:05 -08:00
ipx.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
iw_handler.h net: Spelling s/stucture/structure/ 2018-03-27 09:51:23 +02:00
kcm.h
l3mdev.h ipvlan, l3mdev: fix broken l3s mode wrt local routes 2019-01-30 22:13:34 -08:00
lag.h net: Add lag.h, net_lag_port_dev_txable() 2018-07-11 23:10:19 -07:00
lapb.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
lib80211.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
llc_c_ac.h net: LLC: Convert timers to use timer_setup() 2017-10-25 12:06:25 +09:00
llc_c_ev.h
llc_c_st.h
llc_conn.h llc: delete timers synchronously in llc_sk_free() 2018-04-22 14:55:03 -04:00
llc_if.h
llc_pdu.h
llc_s_ac.h
llc_s_ev.h
llc_s_st.h
llc_sap.h
llc.h llc: avoid blocking in llc_sap_close() 2018-09-13 09:04:58 -07:00
lwtunnel.h bpf: implement BPF_LWT_ENCAP_IP mode in bpf_lwt_push_encap 2019-02-13 18:27:55 -08:00
mac80211.h mac80211: notify driver on subsequent CSA beacons 2019-02-22 13:45:51 +01:00
mac802154.h
mip6.h
mld.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
mpls_iptunnel.h
mpls.h
mrp.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ncsi.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ndisc.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
neighbour.h neighbour: register rtnl doit handler 2018-12-19 13:37:34 -08:00
net_failover.h net: Introduce net_failover driver 2018-05-28 22:59:54 -04:00
net_namespace.h net: xsk: track AF_XDP sockets on a per-netns list 2019-01-25 01:50:03 +01:00
net_ratelimit.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
netevent.h net: ipv4: Notify about changes to ip_forward_update_priority 2018-08-01 09:52:30 -07:00
netlabel.h net: convert netlbl_lsm_cache.refcount from atomic_t to refcount_t 2017-07-01 07:39:09 -07:00
netlink.h netlink: reduce NLA_POLICY_NESTED{,_ARRAY} arguments 2019-02-01 11:06:55 +01:00
netprio_cgroup.h
netrom.h proc: introduce proc_create_seq{,_data} 2018-05-16 07:23:35 +02:00
nexthop.h net: fix rtnh_ok() 2018-04-07 22:32:31 -04:00
nl802154.h
nsh.h openvswitch: enable NSH support 2017-11-08 16:12:33 +09:00
p8022.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
page_pool.h xdp: introduce xdp_return_frame_rx_napi 2018-05-24 18:36:15 -07:00
ping.h ipv{4,6}/ping: simplify proc file creation 2018-05-16 07:23:35 +02:00
pkt_cls.h net: sched: set dedicated tcf_walker flag when tp is empty 2019-02-25 10:18:17 -08:00
pkt_sched.h net: sched: extend Qdisc with rcu 2018-09-25 20:17:35 -07:00
pptp.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
protocol.h net: Convert protocol error handlers from void to int 2018-11-08 17:13:08 -08:00
psample.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
psnap.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
raw.h net: fix raw socket lookup device bind matching with VRFs 2018-11-07 16:12:39 -08:00
rawv6.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
red.h net_sched: red: Avoid illegal values 2017-12-05 14:37:13 -05:00
regulatory.h cfg80211: make wmm_rule part of the reg_rule structure 2018-08-28 11:11:47 +02:00
request_sock.h tcp: free request sock directly upon TFO or syncookies error 2019-03-19 14:13:01 -07:00
rose.h proc: introduce proc_create_seq{,_data} 2018-05-16 07:23:35 +02:00
route.h net-ipv4: remove 2 always zero parameters from ipv4_redirect() 2018-09-26 20:30:55 -07:00
rsi_91x.h Bluetooth: btrsi: add new rsi bluetooth driver 2018-03-13 18:37:02 +02:00
rtnetlink.h net: Add extack argument to rtnl_create_link 2018-11-06 15:00:45 -08:00
sch_generic.h net: sched: add empty status flag for NOLOCK qdisc 2019-03-23 21:52:36 -04:00
scm.h pids: Compute task_tgid using signal->leader_pid 2018-07-21 10:43:12 -05:00
secure_seq.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
seg6_hmac.h rhashtable: split rhashtable.h 2018-06-22 13:43:27 +09:00
seg6_local.h bpf: add End.DT6 action to bpf_lwt_seg6_action helper 2018-07-31 09:22:48 +02:00
seg6.h net: seg6.h: remove an unused #include 2018-12-20 16:56:04 -08:00
slhc_vj.h slip: Check if rstate is initialized before uncompressing 2018-04-11 10:33:46 -04:00
smc.h net/smc: add smcd support to the pnet table 2019-02-21 10:34:37 -08:00
snmp.h
sock_reuseport.h bpf: Enable BPF_PROG_TYPE_SK_REUSEPORT bpf prog in reuseport selection 2018-08-11 01:58:46 +02:00
sock.h tcp: add one skb cache for rx 2019-03-23 21:57:38 -04:00
Space.h net/mac89x0: Convert to platform_driver 2018-03-01 21:21:36 -05:00
stp.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
strparser.h strparser: Add __strp_unpause and use it in ktls. 2018-06-06 14:07:53 -04:00
switchdev.h switchdev: Remove unused transaction item queue 2019-03-01 21:35:19 -08:00
tcp_states.h tcp: remove the hardcode in the definition of TCPF Macro 2018-02-21 15:06:05 -05:00
tcp.h tcp: convert tcp_md5_needed to static_branch API 2019-02-26 13:16:03 -08:00
timewait_sock.h
tipc.h flow_dissector: do not rely on implicit casts 2018-05-08 00:02:41 -04:00
tls.h net/tls: Add support of AES128-CCM based ciphers 2019-03-20 11:02:05 -07:00
transp_v6.h ipv6: fold sockcm_cookie into ipcm6_cookie 2018-07-07 10:58:49 +09:00
tso.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
tun_proto.h vxlan: factor out VXLAN-GPE next protocol 2017-08-29 15:16:52 -07:00
udp_tunnel.h udp_tunnel: add config option to bind to a device 2018-12-03 14:15:26 -08:00
udp.h net: Convert protocol error handlers from void to int 2018-11-08 17:13:08 -08:00
udplite.h udplite: fix partial checksum initialization 2018-02-16 15:57:42 -05:00
vsock_addr.h
vxlan.h vxlan: add extack support for create and changelink 2019-02-26 08:54:37 -08:00
wext.h lift handling of SIOCIW... out of dev_ioctl() 2018-01-24 19:13:45 -05:00
wimax.h
x25.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
x25device.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
xdp_sock.h xsk: fix umem memory leak on cleanup 2019-03-16 01:27:51 +01:00
xdp.h xdp: export xdp_rxq_info_unreg_mem_model 2018-08-29 12:25:53 -07:00
xfrm.h xfrm: destroy xfrm_state synchronously on net exit path 2019-02-05 06:29:20 +01:00