Minki/linux
1
0
Fork 1
mirror of https://github.com/torvalds/linux.git synced 2024-12-28 22:02:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
0f3fffd8ab
linux/arch
History
Ingo Molnar 0888f06ac9 [PATCH] sched: fix bad missed wakeups in the i386, x86_64, ia64, ACPI and APM idle code 
Fernando Lopez-Lezcano reported frequent scheduling latencies and audio
xruns starting at the 2.6.18-rt kernel, and those problems persisted all
until current -rt kernels. The latencies were serious and unjustified by
system load, often in the milliseconds range.

After a patient and heroic multi-month effort of Fernando, where he
tested dozens of kernels, tried various configs, boot options,
test-patches of mine and provided latency traces of those incidents, the
following 'smoking gun' trace was captured by him:

                 _------=> CPU#
                / _-----=> irqs-off
               | / _----=> need-resched
               || / _---=> hardirq/softirq
               ||| / _--=> preempt-depth
               |||| /
               |||||     delay
   cmd     pid ||||| time  |   caller
      \   /    |||||   \   |   /
  IRQ_19-1479  1D..1    0us : __trace_start_sched_wakeup (try_to_wake_up)
  IRQ_19-1479  1D..1    0us : __trace_start_sched_wakeup <<...>-5856> (37 0)
  IRQ_19-1479  1D..1    0us : __trace_start_sched_wakeup (c01262ba 0 0)
  IRQ_19-1479  1D..1    0us : resched_task (try_to_wake_up)
  IRQ_19-1479  1D..1    0us : __spin_unlock_irqrestore (try_to_wake_up)
  ...
  <idle>-0     1...1   11us!: default_idle (cpu_idle)
  ...
  <idle>-0     0Dn.1  602us : smp_apic_timer_interrupt (c0103baf 1 0)
  ...
   <...>-5856  0D..2  618us : __switch_to (__schedule)
   <...>-5856  0D..2  618us : __schedule <<idle>-0> (20 162)
   <...>-5856  0D..2  619us : __spin_unlock_irq (__schedule)
   <...>-5856  0...1  619us : trace_stop_sched_switched (__schedule)
   <...>-5856  0D..1  619us : trace_stop_sched_switched <<...>-5856> (37 0)

what is visible in this trace is that CPU#1 ran try_to_wake_up() for
PID:5856, it placed PID:5856 on CPU#0's runqueue and ran resched_task()
for CPU#0. But it decided to not send an IPI that no CPU - due to
TS_POLLING. But CPU#0 never woke up after its NEED_RESCHED bit was set,
and only rescheduled to PID:5856 upon the next lapic timer IRQ. The
result was a 600+ usecs latency and a missed wakeup!

the bug turned out to be an idle-wakeup bug introduced into the mainline
kernel this summer via an optimization in the x86_64 tree:

    commit 495ab9c045
    Author: Andi Kleen <ak@suse.de>
    Date:   Mon Jun 26 13:59:11 2006 +0200

    [PATCH] i386/x86-64/ia64: Move polling flag into thread_info_status

    During some profiling I noticed that default_idle causes a lot of
    memory traffic. I think that is caused by the atomic operations
    to clear/set the polling flag in thread_info. There is actually
    no reason to make this atomic - only the idle thread does it
    to itself, other CPUs only read it. So I moved it into ti->status.

the problem is this type of change:

        if (!hlt_counter && boot_cpu_data.hlt_works_ok) {
-               clear_thread_flag(TIF_POLLING_NRFLAG);
+               current_thread_info()->status &= ~TS_POLLING;
                smp_mb__after_clear_bit();
                while (!need_resched()) {
                        local_irq_disable();

this changes clear_thread_flag() to an explicit clearing of TS_POLLING.
clear_thread_flag() is defined as:

        clear_bit(flag, &ti->flags);

and clear_bit() is a LOCK-ed atomic instruction on all x86 platforms:

  static inline void clear_bit(int nr, volatile unsigned long * addr)
  {
          __asm__ __volatile__( LOCK_PREFIX
                  "btrl %1,%0"

hence smp_mb__after_clear_bit() is defined as a simple compile barrier:

  #define smp_mb__after_clear_bit()       barrier()

but the explicit TS_POLLING clearing introduced by the patch:

+               current_thread_info()->status &= ~TS_POLLING;

is not an atomic op! So the clearing of the TS_POLLING bit is freely
reorderable with the reading of the NEED_RESCHED bit - and both now
reside in different memory addresses.

CPU idle wakeup very much depends on ordered memory ops, the clearing of
the TS_POLLING flag must always be done before we test need_resched()
and hit the idle instruction(s). [Symmetrically, the wakeup code needs
to set NEED_RESCHED before it tests the TS_POLLING flag, so memory
ordering is paramount.]

Fernando's dual-core Athlon64 system has a sufficiently advanced memory
ordering model so that it triggered this scenario very often.

( And it also turned out that the reason why these latencies never
  triggered on my testsystems is that i routinely use idle=poll, which
  was the only idle variant not affected by this bug. )

The fix is to change the smp_mb__after_clear_bit() to an smp_mb(), to
act as an absolute barrier between the TS_POLLING write and the
NEED_RESCHED read. This affects almost all idling methods (default,
ACPI, APM), on all 3 x86 architectures: i386, x86_64, ia64.

Signed-off-by: Ingo Molnar <mingo@elte.hu>
Tested-by: Fernando Lopez-Lezcano <nando@ccrma.Stanford.EDU>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-12-22 08:55:51 -08:00
..
alpha [PATCH] fdtable: Make fdarray and fdsets equal in size 2006-12-10 09:57:22 -08:00
arm [ARM] 4062/1: S3C24XX: Anubis and Osiris shuld have CONFIG_PM_SIMTEC 2006-12-19 22:54:53 +00:00
arm26 [PATCH] getting rid of all casts of k[cmz]alloc() calls 2006-12-13 09:05:58 -08:00
avr32 Merge branch 'for-linus' of git://www.atmel.no/~hskinnemoen/linux/kernel/avr32 2006-12-11 18:28:59 -08:00
cris [PATCH] getting rid of all casts of k[cmz]alloc() calls 2006-12-13 09:05:58 -08:00
frv [PATCH] sysctl: remove unused "context" param 2006-12-10 09:55:41 -08:00
h8300 [PATCH] getting rid of all casts of k[cmz]alloc() calls 2006-12-13 09:05:58 -08:00
i386 [PATCH] sched: fix bad missed wakeups in the i386, x86_64, ia64, ACPI and APM idle code 2006-12-22 08:55:51 -08:00
ia64 [PATCH] sched: fix bad missed wakeups in the i386, x86_64, ia64, ACPI and APM idle code 2006-12-22 08:55:51 -08:00
m32r [PATCH] LOG2: Implement a general integer log2 facility in the kernel 2006-12-08 08:28:51 -08:00
m68k [PATCH] getting rid of all casts of k[cmz]alloc() calls 2006-12-13 09:05:58 -08:00
m68knommu [PATCH] LOG2: Implement a general integer log2 facility in the kernel 2006-12-08 08:28:51 -08:00
mips [PATCH] MIPS: Fix COW D-cache aliasing on fork 2006-12-13 09:27:08 -08:00
parisc [PATCH] getting rid of all casts of k[cmz]alloc() calls 2006-12-13 09:05:58 -08:00
powerpc Merge master.kernel.org:/pub/scm/linux/kernel/git/gregkh/pci-2.6 2006-12-21 00:01:47 -08:00
ppc [PATCH] getting rid of all casts of k[cmz]alloc() calls 2006-12-13 09:05:58 -08:00
s390 [S390] Save prefix register for dump on panic 2006-12-15 17:18:27 +01:00
sh sh: Fixup SH-2 BUG() trap handling. 2006-12-12 09:11:45 +09:00
sh64 [PATCH] Generic ioremap_page_range: sh64 conversion 2006-12-08 08:28:52 -08:00
sparc [SPARC]: Make bitops use same spinlocks as atomics. 2006-12-17 16:18:47 -08:00
sparc64 [SPARC64]: Mirror x86_64's PERCPU_ENOUGH_ROOM definition. 2006-12-17 14:06:59 -08:00
um [PATCH] getting rid of all casts of k[cmz]alloc() calls 2006-12-13 09:05:58 -08:00
v850 kconfig: Standardize "depends" -> "depends on" in Kconfig files 2006-12-12 20:04:19 +01:00
x86_64 [PATCH] sched: fix bad missed wakeups in the i386, x86_64, ia64, ACPI and APM idle code 2006-12-22 08:55:51 -08:00
xtensa [PATCH] Xtensa: Add ktermios and minor filename fix 2006-12-13 09:05:50 -08:00