Tejun Heo
4c16bd327c
workqueue: implement NUMA affinity for unbound workqueues ...
Currently, an unbound workqueue has single current, or first, pwq
(pool_workqueue) to which all new work items are queued. This often
isn't optimal on NUMA machines as workers may jump around across node
boundaries and work items get assigned to workers without any regard
to NUMA affinity.
This patch implements NUMA affinity for unbound workqueues. Instead
of mapping all entries of numa_pwq_tbl[] to the same pwq,
apply_workqueue_attrs() now creates a separate pwq covering the
intersecting CPUs for each NUMA node which has online CPUs in
@attrs->cpumask. Nodes which don't have intersecting possible CPUs
are mapped to pwqs covering whole @attrs->cpumask.
As CPUs come up and go down, the pool association is changed
accordingly. Changing pool association may involve allocating new
pools which may fail. To avoid failing CPU_DOWN, each workqueue
always keeps a default pwq which covers whole attrs->cpumask which is
used as fallback if pool creation fails during a CPU hotplug
operation.
This ensures that all work items issued on a NUMA node is executed on
the same node as long as the workqueue allows execution on the CPUs of
the node.
As this maps a workqueue to multiple pwqs and max_active is per-pwq,
this change the behavior of max_active. The limit is now per NUMA
node instead of global. While this is an actual change, max_active is
already per-cpu for per-cpu workqueues and primarily used as safety
mechanism rather than for active concurrency control. Concurrency is
usually limited from workqueue users by the number of concurrently
active work items and this change shouldn't matter much.
v2: Fixed pwq freeing in apply_workqueue_attrs() error path. Spotted
by Lai.
v3: The previous version incorrectly made a workqueue spanning
multiple nodes spread work items over all online CPUs when some of
its nodes don't have any desired cpus. Reimplemented so that NUMA
affinity is properly updated as CPUs go up and down. This problem
was spotted by Lai Jiangshan.
v4: destroy_workqueue() was putting wq->dfl_pwq and then clearing it;
however, wq may be freed at any time after dfl_pwq is put making
the clearing use-after-free. Clear wq->dfl_pwq before putting it.
v5: apply_workqueue_attrs() was leaking @tmp_attrs, @new_attrs and
@pwq_tbl after success. Fixed.
Retry loop in wq_update_unbound_numa_attrs() isn't necessary as
application of new attrs is excluded via CPU hotplug. Removed.
Documentation on CPU affinity guarantee on CPU_DOWN added.
All changes are suggested by Lai Jiangshan.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reviewed-by: Lai Jiangshan <laijs@cn.fujitsu.com>
2013-04-01 11:23:36 -07:00
KGDB/KDB fixes and cleanups
2013-03-02 08:31:39 -08:00
hlist: drop the node parameter from iterators
2013-02-27 19:10:24 -08:00
kernel/gcov: remove depends on CONFIG_EXPERIMENTAL
2013-01-11 11:39:33 -08:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-02-26 20:16:07 -08:00
Driver core patches for 3.9-rc1
2013-02-21 12:05:51 -08:00
sched: replace PF_THREAD_BOUND with PF_NO_SETAFFINITY
2013-03-19 13:45:20 -07:00
Merge branch 'release' of git://git.kernel.org/pub/scm/linux/kernel/git/rzhang/linux
2013-02-28 19:48:26 -08:00
ImgTec Meta architecture changes for v3.9-rc1
2013-03-03 12:06:09 -08:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-02-26 20:16:07 -08:00
Merge branch 'for-3.9' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq
2013-02-19 22:01:33 -08:00
kernel/audit.c: avoid negative sleep durations
2013-01-11 14:54:56 -08:00
audit: optimize audit_compare_dname_path
2012-10-12 00:32:02 -04:00
audit: catch possible NULL audit buffers
2013-01-11 14:54:55 -08:00
audit: catch possible NULL audit buffers
2013-01-11 14:54:55 -08:00
audit: fix auditfilter.c kernel-doc warnings
2013-01-10 14:35:23 -08:00
audit: catch possible NULL audit buffers
2013-01-11 14:54:55 -08:00
userns: Teach inode_capable to understand inodes whose uids map to other namespaces.
2012-05-15 14:59:24 -07:00
sched: replace PF_THREAD_BOUND with PF_NO_SETAFFINITY
2013-03-19 13:45:20 -07:00
cgroup: rename ->create/post_create/pre_destroy/destroy() to ->css_alloc/online/offline/free()
2012-11-19 08:13:38 -08:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal
2013-02-23 18:50:11 -08:00
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-02-19 18:19:48 -08:00
Merge branch 'smp-hotplug-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-02-19 19:04:55 -08:00
kernel/cpu_pm.c: fix various typos
2012-05-31 17:49:27 -07:00
sched: replace PF_THREAD_BOUND with PF_NO_SETAFFINITY
2013-03-19 13:45:20 -07:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace
2012-12-18 10:55:28 -08:00
cputime: Use accessors to read task cputime stats
2013-01-27 19:23:31 +01:00
Remove all #inclusions of asm/system.h
2012-03-28 18:30:03 +01:00
coredump: use a freezable_schedule for the coredump_finish wait
2013-02-27 19:10:11 -08:00
extable: Skip sorting if sorted at build time.
2012-04-19 15:06:55 -07:00
fork: unshare: remove dead code
2013-02-27 19:10:12 -08:00
freezer: change ptrace_stop/do_signal_stop to use freezable_schedule()
2012-10-26 14:27:49 -07:00
more file_inode() open-coded instances
2013-02-27 16:59:05 -05:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal
2013-02-23 18:50:11 -08:00
userns: Convert in_group_p and in_egroup_p to use kgid_t
2012-05-03 03:29:33 -07:00
Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-02-19 19:05:45 -08:00
hung task debugging: Inject NMI when hung and going to panic
2012-04-25 12:39:25 +02:00
Merge branch 'nohz/printk-v8' into irq/core
2013-02-05 00:48:46 +01:00
itimer: Use printk_once instead of WARN_ONCE
2012-04-10 11:00:30 +02:00
jump_label: Export jump_label_rate_limit()
2012-08-06 19:00:35 +03:00
vsprintf: fix %ps on non symbols when using kallsyms
2012-05-29 16:22:32 -07:00
kcmp: include linux/ptrace.h
2012-12-20 17:40:19 -08:00
locking: Adjust spin lock inlining Kconfig options
2012-09-13 17:56:13 +02:00
locking/kconfig: Simplify INLINE_SPIN_UNLOCK usage
2012-03-23 13:18:57 +01:00
kexec: avoid freeing NULL pointer in image_crash_alloc()
2013-02-27 19:10:12 -08:00
Merge branch 'master' into for-3.9-async
2013-01-23 09:31:01 -08:00
hlist: drop the node parameter from iterators
2013-02-27 19:10:24 -08:00
Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2012-12-11 18:10:49 -08:00
sched: replace PF_THREAD_BOUND with PF_NO_SETAFFINITY
2013-03-19 13:45:20 -07:00
brlocks/lglocks: turn into functions
2012-05-29 23:28:41 -04:00
lockdep: check that no locks held at freeze time
2013-02-27 19:10:11 -08:00
lockdep: Use KSYM_NAME_LEN'ed buffer for __get_key_name()
2012-10-24 12:39:09 +02:00
Merge branch 'akpm' (final batch from Andrew)
2013-02-27 20:58:09 -08:00
MODSIGN: Avoid using .incbin in C source
2012-12-14 13:06:44 +10:30
keys: use keyring_alloc() to create module signing keyring
2012-12-20 17:40:21 -08:00
MODSIGN: Move the magic string to the end of a module and eliminate the search
2012-10-19 17:30:40 -07:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-02-26 20:16:07 -08:00
MODSIGN: Don't use enum-type bitfields in module signature info block
2012-12-05 11:27:24 +10:30
sched/rt: Move rt specific bits into new header file
2013-02-07 20:51:08 +01:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2013-02-26 20:16:07 -08:00
padata: use __this_cpu_read per-cpu helper
2012-12-06 17:16:23 +08:00
taint: add explicit flag to show whether lock dep is still OK.
2013-01-21 17:17:57 +10:30
params: replace printk(KERN_<LVL>...) with pr_<lvl>(...)
2012-05-04 17:28:18 -07:00
hlist: drop the node parameter from iterators
2013-02-27 19:10:24 -08:00
pidns: Stop pid allocation when init dies
2012-12-25 16:10:05 -08:00
Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-02-19 19:05:45 -08:00
posix-timers: convert to idr_alloc()
2013-02-27 19:10:19 -08:00
Merge branch 'drm-next' of git://people.freedesktop.org/~airlied/linux
2013-02-25 16:46:44 -08:00
profiling: Remove unused timer hook
2013-01-24 15:37:26 +01:00
uprobes: Add exports for module use
2013-02-08 17:47:13 +01:00
rcu: Provide RCU CPU stall warnings for tiny RCU
2013-01-28 22:06:21 -08:00
Merge branches 'doctorture.2013.01.29a', 'fixes.2013.01.26a', 'tagcb.2013.01.24a' and 'tiny.2013.01.29b' into HEAD
2013-01-28 22:25:21 -08:00
Merge branches 'doctorture.2013.01.29a', 'fixes.2013.01.26a', 'tagcb.2013.01.24a' and 'tiny.2013.01.29b' into HEAD
2013-01-28 22:25:21 -08:00
rcu: Provide RCU CPU stall warnings for tiny RCU
2013-01-28 22:06:21 -08:00
rcu: Allow rcutorture to be built at low optimization levels
2013-02-04 12:18:20 -08:00
Merge branches 'doctorture.2013.01.29a', 'fixes.2013.01.26a', 'tagcb.2013.01.24a' and 'tiny.2013.01.29b' into HEAD
2013-01-28 22:25:21 -08:00
Merge branches 'doctorture.2013.01.29a', 'fixes.2013.01.26a', 'tagcb.2013.01.24a' and 'tiny.2013.01.29b' into HEAD
2013-01-28 22:25:21 -08:00
rcu: Make rcu_nocb_poll an early_param instead of module_param
2013-01-08 14:12:19 -08:00
rcu: Separate accounting of callbacks from callback-free CPUs
2012-11-16 10:05:57 -08:00
new helper: file_inode(file)
2013-02-22 23:31:31 -05:00
res_counter: return amount of charges after res_counter_uncharge()
2012-12-18 15:02:12 -08:00
kernel/resource.c: fix stack overflow in __reserve_region_with_split()
2012-10-06 03:05:31 +09:00
sched/rt: Move rt specific bits into new header file
2013-02-07 20:51:08 +01:00
sched/rt: Move rt specific bits into new header file
2013-02-07 20:51:08 +01:00
sched/rt: Move rt specific bits into new header file
2013-02-07 20:51:08 +01:00
lockdep, rwsem: provide down_write_nest_lock()
2013-01-11 14:54:55 -08:00
seccomp: Make syscall skipping and nr changes more consistent
2012-10-02 21:14:29 +10:00
semaphore: fix improper comment reference to mutex
2012-04-05 17:15:55 -07:00
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal
2013-03-02 19:32:06 -08:00
smp: make smp_call_function_many() use logic similar to smp_call_function_single()
2013-02-21 17:22:20 -08:00
hlist: drop the node parameter from iterators
2013-02-27 19:10:24 -08:00
smpboot: Provide infrastructure for percpu hotplug threads
2012-08-13 17:01:07 +02:00
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
2013-02-20 18:58:50 -08:00
locking/kconfig: Simplify INLINE_SPIN_UNLOCK usage
2012-03-23 13:18:57 +01:00
srcu: use ACCESS_ONCE() to access sp->completed in srcu_read_lock()
2013-02-07 15:19:36 -08:00
stop_machine: Use smpboot threads
2013-02-14 15:29:38 +01:00
usermodehelper: cleanup/fix __orderly_poweroff() && argv_free()
2013-02-27 19:10:09 -08:00
module: add syscall to load module from fd
2012-12-14 13:05:22 +10:30
Initial ARC Linux port with some fixes on top for 3.9-rc1
2013-03-02 07:58:56 -08:00
sysctl: fix null checking in bin_dn_node_address()
2013-02-27 19:10:21 -08:00
task_work: task_work_add() should not succeed after exit_task_work()
2012-09-13 16:47:34 +02:00
taskstats: cgroupstats_user_cmd() may leak on error
2012-10-06 03:05:31 +09:00
time: don't inline EXPORT_SYMBOL functions
2013-02-21 17:22:19 -08:00
kernel: Replace timeconst.pl with a bc script
2013-02-16 23:17:25 +01:00
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-02-19 18:19:48 -08:00
hlist: drop the node parameter from iterators
2013-02-27 19:10:24 -08:00
cputime: Use accessors to read task cputime stats
2013-01-27 19:23:31 +01:00
userns: Convert setting and getting uid and gid system calls to use kuid and kgid
2012-05-03 03:28:41 -07:00
hlist: drop the node parameter from iterators
2013-02-27 19:10:24 -08:00
hlist: drop the node parameter from iterators
2013-02-27 19:10:24 -08:00
userns: Allow any uid or gid mappings that don't overlap.
2013-01-26 22:12:04 -08:00
kernel/utsname.c: fix wrong comment about clone_uts_ns()
2013-02-27 19:10:22 -08:00
kernel/utsname_sysctl.c: put get/get_uts() into CONFIG_PROC_SYSCTL code block
2013-02-27 19:10:22 -08:00
propagate name change to comments in kernel source
2012-12-06 10:39:54 +01:00
Merge branch 'core-locking-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2013-02-22 19:25:09 -08:00
workqueue: implement NUMA affinity for unbound workqueues
2013-04-01 11:23:36 -07:00
workqueue: directly restore CPU affinity of workers from CPU_ONLINE
2013-03-19 13:45:21 -07:00
4c16bd327c