Peter Zijlstra
2c61002271
locking/qspinlock: Fix spin_unlock_wait() some more
...
While this prior commit:
54cf809b95 ("locking,qspinlock: Fix spin_is_locked() and spin_unlock_wait()")
... fixes spin_is_locked() and spin_unlock_wait() for the usage
in ipc/sem and netfilter, it does not in fact work right for the
usage in task_work and futex.
So while the 2 locks crossed problem:
spin_lock(A) spin_lock(B)
if (!spin_is_locked(B)) spin_unlock_wait(A)
foo() foo();
... works with the smp_mb() injected by both spin_is_locked() and
spin_unlock_wait(), this is not sufficient for:
flag = 1;
smp_mb(); spin_lock()
spin_unlock_wait() if (!flag)
// add to lockless list
// iterate lockless list
... because in this scenario, the store from spin_lock() can be delayed
past the load of flag, uncrossing the variables and loosing the
guarantee.
This patch reworks spin_is_locked() and spin_unlock_wait() to work in
both cases by exploiting the observation that while the lock byte
store can be delayed, the contender must have registered itself
visibly in other state contained in the word.
It also allows for architectures to override both functions, as PPC
and ARM64 have an additional issue for which we currently have no
generic solution.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Giovanni Gherdovich <ggherdovich@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Pan Xinhui <xinhui.pan@linux.vnet.ibm.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Waiman Long <waiman.long@hpe.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: stable@vger.kernel.org # v4.2 and later
Fixes: 54cf809b95 ("locking,qspinlock: Fix spin_is_locked() and spin_unlock_wait()")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-06-08 14:29:08 +02:00
..
bitops
bitops: Do not default to __clear_bit() for __clear_bit_unlock()
2016-03-21 10:50:48 +01:00
4level-fixup.h
mm, asm-generic: define PUD_SHIFT in <asm-generic/4level-fixup.h>
2015-02-11 17:06:03 -08:00
asm-offsets.h
asm-generic: Add common asm-offsets.h
2015-06-23 13:35:49 +09:00
atomic-long.h
include/asm-generic/atomic-long.h: force inlining of some atomic_long operations
2016-03-17 15:09:34 -07:00
atomic.h
atomic, arch: Audit atomic_{read,set}()
2015-09-23 09:54:28 +02:00
atomic64.h
atomic: Provide atomic_{or,xor,and}
2015-07-27 14:06:24 +02:00
audit_change_attr.h
audit: Modify a set of system calls in audit class definitions
2014-01-17 17:01:46 -05:00
audit_dir_write.h
audit: support the "standard" <asm-generic/unistd.h>
2011-05-04 14:41:28 -04:00
audit_read.h
audit: support the "standard" <asm-generic/unistd.h>
2011-05-04 14:41:28 -04:00
audit_signal.h
audit_write.h
audit: Modify a set of system calls in audit class definitions
2014-01-17 17:01:46 -05:00
barrier.h
asm-generic: implement virt_xxx memory barriers
2016-01-12 20:46:59 +02:00
bitops.h
arch: Prepare for smp_mb__{before,after}_atomic()
2014-04-18 11:40:30 +02:00
bitsperlong.h
UAPI: (Scripted) Disintegrate include/asm-generic
2012-10-04 18:20:15 +01:00
bug.h
asm-generic changes for 4.6
2016-03-24 23:13:48 -07:00
bugs.h
asm-generic: add generic versions of common headers
2009-06-11 21:02:37 +02:00
cache.h
asm-generic: add generic NOMMU versions of some headers
2009-06-11 21:02:50 +02:00
cacheflush.h
asm-generic/cacheflush.h: flush icache when copying to user pages
2011-05-25 08:39:37 -07:00
checksum.h
ipv4: Update parameters for csum_tcpudp_magic to their original types
2016-03-13 23:55:13 -04:00
clkdev.h
asm-generic: COMMON_CLK defines __clk_{get,put}
2014-09-25 18:00:45 -07:00
cmpxchg-local.h
LLVMLinux: Remove warning about returning an uninitialized variable
2014-04-09 13:44:35 -07:00
cmpxchg.h
asm-generic: cmpxchg: avoid warnings from macro-ized cmpxchg() implementations
2015-10-15 00:21:13 +02:00
cputime.h
cputime: Generic on-demand virtual cputime accounting
2013-01-27 19:23:27 +01:00
cputime_jiffies.h
sched, time: Fix build error with 64 bit cputime_t on 32 bit systems
2014-10-03 05:46:55 +02:00
cputime_nsecs.h
cputime: Prevent 32bit overflow in time[val|spec]_to_cputime()
2016-02-02 15:24:38 +01:00
current.h
asm-generic: add generic versions of common headers
2009-06-11 21:02:37 +02:00
delay.h
asm-generic: delay.h fix udelay and ndelay for 8 bit args
2011-07-22 18:45:33 +02:00
device.h
Driver Core: Add platform device arch data V3
2009-07-22 00:28:38 +02:00
div64.h
__div64_32(): make it overridable at compile time
2015-11-16 14:42:12 -05:00
dma-contiguous.h
asm-generic: Add dma-contiguous.h
2014-09-22 13:35:51 +02:00
dma.h
asm-generic: add legacy I/O header files
2009-06-11 21:02:42 +02:00
early_ioremap.h
Merge branch 'akpm' (patches from Andrew)
2015-09-08 17:52:23 -07:00
emergency-restart.h
exec.h
Split arch_align_stack() out from asm-generic/system.h
2012-03-28 18:30:03 +01:00
fb.h
asm-generic: add generic versions of common headers
2009-06-11 21:02:37 +02:00
fixmap.h
asm-generic: Fix local variable shadow in __set_fixmap_offset
2016-02-16 15:10:44 +00:00
ftrace.h
asm-generic headers: add ftrace.h
2011-03-17 09:19:04 +08:00
futex.h
asm-generic/futex: Re-enable preemption in futex_atomic_cmpxchg_inatomic()
2016-04-21 11:06:09 +02:00
getorder.h
bitops: Add missing parentheses to new get_order macro
2012-02-24 10:39:27 -08:00
gpio.h
gpio: allow setting ARCH_NR_GPIOS from Kconfig
2016-02-19 00:22:54 +01:00
hardirq.h
Fix IRQ flag handling naming
2010-10-07 14:08:55 +01:00
hugetlb.h
mm: Fix generic hugetlb pte check return type.
2013-10-02 20:02:35 -04:00
hw_irq.h
asm-generic: add legacy I/O header files
2009-06-11 21:02:42 +02:00
ide_iops.h
int-ll64.h
UAPI: (Scripted) Disintegrate include/asm-generic
2012-10-04 18:20:15 +01:00
io.h
io-64-nonatomic: Add relaxed accessor variants
2016-05-03 18:23:02 +01:00
ioctl.h
include/asm-generic/ioctl.h: fix _IOC_TYPECHECK sparse error
2014-06-06 16:08:13 -07:00
iomap.h
x86/mm, asm-generic: Add ioremap_wt() for creating Write-Through mappings
2015-06-07 15:28:56 +02:00
irq.h
asm-generic: add generic versions of common headers
2009-06-11 21:02:37 +02:00
irq_regs.h
core: Replace __get_cpu_var with __this_cpu_read if not used for an address.
2010-12-17 15:07:19 +01:00
irq_work.h
irq_work: Introduce arch_irq_work_has_interrupt()
2014-09-13 18:38:07 +02:00
irqflags.h
Fix IRQ flag handling naming
2010-10-07 14:08:55 +01:00
Kbuild.asm
UAPI: Set up uapi/asm/Kbuild.asm
2012-10-02 18:01:56 +01:00
kdebug.h
asm-generic: kdebug.h: Checkpatch cleanup
2010-10-09 21:51:44 +02:00
kmap_types.h
asm-generic: remove km_type definitions
2012-07-24 15:27:30 +08:00
kvm_para.h
KVM: add kvm_para_available to asm-generic/kvm_para.h
2013-06-05 13:21:29 +03:00
libata-portmap.h
linkage.h
asm-generic: add generic versions of common headers
2009-06-11 21:02:37 +02:00
local.h
atomic: use <linux/atomic.h>
2011-07-26 16:49:47 -07:00
local64.h
atomic: use <linux/atomic.h>
2011-07-26 16:49:47 -07:00
mcs_spinlock.h
locking/mcs: Allow architecture specific asm files to be used for contended case
2014-02-09 21:18:52 +01:00
memory_model.h
mm: add PHYS_PFN, use it in __phys_to_pfn()
2016-01-14 16:00:49 -08:00
mm-arch-hooks.h
mm: clean up per architecture MM hook header files
2015-07-17 16:39:53 -07:00
mm_hooks.h
mm/core, x86/mm/pkeys: Differentiate instruction fetches
2016-02-18 19:46:29 +01:00
mmu.h
asm-generic/mmu.h: Add support for FDPIC
2012-12-09 23:14:14 +01:00
mmu_context.h
asm-generic: Remove asm-generic arch_bprm_mm_init()
2014-11-22 21:52:08 +01:00
module.h
Make most arch asm/module.h files use asm-generic/module.h
2012-09-28 14:31:03 +09:30
msi.h
asm-generic: Add msi.h
2014-11-23 13:01:47 +01:00
mutex-dec.h
locking/mutex: Use acquire/release semantics
2015-10-06 17:28:20 +02:00
mutex-null.h
arch: Make __mutex_fastpath_lock_retval return whether fastpath succeeded or not
2013-06-26 12:10:55 +02:00
mutex-xchg.h
locking/mutex: Use acquire/release semantics
2015-10-06 17:28:20 +02:00
mutex.h
asm-generic: add generic versions of common headers
2009-06-11 21:02:37 +02:00
page.h
asm-generic: page.h: Remove useless get_user_page and free_user_page
2016-02-26 15:24:55 +01:00
param.h
UAPI: (Scripted) Disintegrate include/asm-generic
2012-10-04 18:20:15 +01:00
parport.h
include: remove __dev* attributes.
2013-01-03 15:57:16 -08:00
pci.h
PCI: Remove unused pcibios_select_root() (again)
2015-06-08 07:56:21 -05:00
pci_iomap.h
PCI: Add pci_iomap_wc() variants
2015-08-25 09:59:45 +02:00
percpu.h
percpu: preffity percpu header files
2014-06-17 19:12:40 -04:00
pgalloc.h
asm-generic: add generic NOMMU versions of some headers
2009-06-11 21:02:50 +02:00
pgtable-nopmd.h
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
2009-07-27 12:10:38 -07:00
pgtable-nopud.h
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
2009-07-27 12:10:38 -07:00
pgtable.h
arch: fix has_transparent_hugepage()
2016-05-19 19:12:14 -07:00
preempt.h
sched/preempt: Fix preempt_count manipulations
2016-05-17 12:24:21 +02:00
ptrace.h
asm-generic/ptrace.h: start a common low level ptrace helper
2011-05-26 17:12:36 -07:00
qrwlock.h
locking/qrwlock: Make use of _{acquire|release|relaxed}() atomics
2015-08-12 11:59:06 +02:00
qrwlock_types.h
locking/qrwlock: Rename ->lock to ->wait_lock
2015-09-18 09:27:29 +02:00
qspinlock.h
locking/qspinlock: Fix spin_unlock_wait() some more
2016-06-08 14:29:08 +02:00
qspinlock_types.h
locking/qspinlock: Move __ARCH_SPIN_LOCK_UNLOCKED to qspinlock_types.h
2016-02-29 10:02:43 +01:00
resource.h
asm-generic: remove _STK_LIM_MAX
2014-05-15 00:32:09 +01:00
rtc.h
rtc: cmos: century support
2015-09-05 13:19:09 +02:00
rwsem.h
locking/rwsem: Introduce basis for down_write_killable()
2016-04-13 10:42:20 +02:00
seccomp.h
seccomp: Get compat syscalls from asm-generic header
2016-05-13 14:02:00 +02:00
sections.h
asm/sections: add helpers to check for section data
2016-01-16 11:17:24 -08:00
segment.h
asm-generic: add generic NOMMU versions of some headers
2009-06-11 21:02:50 +02:00
serial.h
asm-generic: add legacy I/O header files
2009-06-11 21:02:42 +02:00
siginfo.h
SIGNAL: Move generic copy_siginfo() to signal.h
2016-05-13 14:02:10 +02:00
signal.h
unify default ptrace_signal_deliver
2012-11-29 00:01:23 -05:00
simd.h
crypto: create generic version of ablk_helper
2013-09-24 06:02:24 +10:00
sizes.h
ARM: 7430/1: sizes.h: move from asm-generic to <linux/sizes.h>
2012-06-28 17:14:34 +01:00
spinlock.h
asm-generic: add generic versions of common headers
2009-06-11 21:02:37 +02:00
statfs.h
UAPI: (Scripted) Disintegrate include/asm-generic
2012-10-04 18:20:15 +01:00
string.h
asm-generic: add generic versions of common headers
2009-06-11 21:02:37 +02:00
switch_to.h
Split the switch_to() wrapper out of asm-generic/system.h
2012-03-28 18:30:03 +01:00
syscall.h
syscall.h: fix doc text for syscall_get_arch()
2014-09-23 16:20:00 -04:00
syscalls.h
burying unused conditionals
2013-02-14 09:21:15 -05:00
termios-base.h
asm-generic: rename termios.h, signal.h and mman.h
2009-06-11 21:01:52 +02:00
termios.h
UAPI: (Scripted) Disintegrate include/asm-generic
2012-10-04 18:20:15 +01:00
timex.h
asm-generic: add legacy I/O header files
2009-06-11 21:02:42 +02:00
tlb.h
treewide: Remove old email address
2015-11-23 09:44:58 +01:00
tlbflush.h
BUG: headers with BUG/BUG_ON etc. need linux/bug.h
2012-03-04 17:54:34 -05:00
topology.h
topology: alternate fix for ia64 tiger_defconfig build breakage
2010-08-09 20:44:57 -07:00
trace_clock.h
tracing,x86: Add a TSC trace_clock
2012-11-13 15:48:27 -05:00
uaccess-unaligned.h
asm-generic: rename page.h and uaccess.h
2009-06-11 21:02:17 +02:00
uaccess.h
asm-generic: {get,put}_user ptr argument evaluate only 1 time
2015-11-08 22:44:42 +09:00
unaligned.h
asm-generic: allow generic unaligned access if the arch supports it
2014-05-08 10:22:23 +02:00
unistd.h
We get rid of the general module prefix confusion with a binary config option,
2013-05-05 10:58:06 -07:00
user.h
asm-generic/user.h: Fix spelling in comment
2011-03-01 15:49:39 +01:00
vga.h
asm-generic: add legacy I/O header files
2009-06-11 21:02:42 +02:00
vmlinux.lds.h
x86/asm: Make sure verify_cpu() has a good stack
2016-04-13 11:52:19 +02:00
vtime.h
include/asm-generic/vtime.h: avoid zero-length file
2013-09-30 14:31:02 -07:00
word-at-a-time.h
Make asm/word-at-a-time.h available on all architectures
2015-07-08 16:41:55 -04:00
xor.h
asm-generic: xor: mark static functions as __maybe_unused
2012-10-03 21:21:06 +02:00