linux/arch
Raghavendra K T d6abfdb202 x86/spinlocks/paravirt: Fix memory corruption on unlock
Paravirt spinlock clears slowpath flag after doing unlock.
As explained by Linus currently it does:

                prev = *lock;
                add_smp(&lock->tickets.head, TICKET_LOCK_INC);

                /* add_smp() is a full mb() */

                if (unlikely(lock->tickets.tail & TICKET_SLOWPATH_FLAG))
                        __ticket_unlock_slowpath(lock, prev);

which is *exactly* the kind of things you cannot do with spinlocks,
because after you've done the "add_smp()" and released the spinlock
for the fast-path, you can't access the spinlock any more.  Exactly
because a fast-path lock might come in, and release the whole data
structure.

Linus suggested that we should not do any writes to lock after unlock(),
and we can move slowpath clearing to fastpath lock.

So this patch implements the fix with:

 1. Moving slowpath flag to head (Oleg):
    Unlocked locks don't care about the slowpath flag; therefore we can keep
    it set after the last unlock, and clear it again on the first (try)lock.
    -- this removes the write after unlock. note that keeping slowpath flag would
    result in unnecessary kicks.
    By moving the slowpath flag from the tail to the head ticket we also avoid
    the need to access both the head and tail tickets on unlock.

 2. use xadd to avoid read/write after unlock that checks the need for
    unlock_kick (Linus):
    We further avoid the need for a read-after-release by using xadd;
    the prev head value will include the slowpath flag and indicate if we
    need to do PV kicking of suspended spinners -- on modern chips xadd
    isn't (much) more expensive than an add + load.

Result:
 setup: 16core (32 cpu +ht sandy bridge 8GB 16vcpu guest)
 benchmark overcommit %improve
 kernbench  1x           -0.13
 kernbench  2x            0.02
 dbench     1x           -1.77
 dbench     2x           -0.63

[Jeremy: Hinted missing TICKET_LOCK_INC for kick]
[Oleg: Moved slowpath flag to head, ticket_equals idea]
[PeterZ: Added detailed changelog]

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Reported-by: Sasha Levin <sasha.levin@oracle.com>
Tested-by: Sasha Levin <sasha.levin@oracle.com>
Signed-off-by: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Cc: Andrew Jones <drjones@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Dave Jones <davej@redhat.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Fernando Luis Vázquez Cao <fernando_b1@lab.ntt.co.jp>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Ulrich Obergfell <uobergfe@redhat.com>
Cc: Waiman Long <Waiman.Long@hp.com>
Cc: a.ryabinin@samsung.com
Cc: dave@stgolabs.net
Cc: hpa@zytor.com
Cc: jasowang@redhat.com
Cc: jeremy@goop.org
Cc: paul.gortmaker@windriver.com
Cc: riel@redhat.com
Cc: tglx@linutronix.de
Cc: waiman.long@hp.com
Cc: xen-devel@lists.xenproject.org
Link: http://lkml.kernel.org/r/20150215173043.GA7471@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-02-18 14:53:49 +01:00
..
alpha vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
arc arc: mm: Fix build failure 2015-01-30 10:31:14 -08:00
arm Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2015-02-09 14:28:42 -08:00
arm64 Merge branch 'x86-efi-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2015-02-09 17:53:53 -08:00
avr32 vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
blackfin arch/blackfin/mach-bf533/boards/stamp.c: add linux/delay.h 2015-01-08 15:10:52 -08:00
c6x net, lib: kill arch_fast_hash library bits 2014-12-10 15:17:46 -05:00
cris vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
frv vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
hexagon Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rkuo/linux-hexagon-kernel 2014-12-19 17:57:51 -08:00
ia64 vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
m32r vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
m68k vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
metag vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
microblaze vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
mips Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2015-02-09 14:28:42 -08:00
mn10300 mm/debug_pagealloc: fix build failure on ppc and some other archs 2015-02-05 13:35:30 -08:00
nios2 nios2: fix unhandled signals 2015-02-09 09:47:05 +08:00
openrisc vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
parisc vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
powerpc Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2015-02-09 14:28:42 -08:00
s390 Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2015-02-09 14:28:42 -08:00
score vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
sh vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
sparc mm/debug_pagealloc: fix build failure on ppc and some other archs 2015-02-05 13:35:30 -08:00
tile Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2015-02-09 14:28:42 -08:00
um vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
unicore32 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next 2014-12-11 14:27:06 -08:00
x86 x86/spinlocks/paravirt: Fix memory corruption on unlock 2015-02-18 14:53:49 +01:00
xtensa vm: add VM_FAULT_SIGSEGV handling support 2015-01-29 10:51:32 -08:00
.gitignore
Kconfig