Steven Rostedt
77ae365eca
ring-buffer: make lockless ...
This patch converts the ring buffers into a completely lockless
buffer recording system. The read side still takes locks since
we still serialize readers. But the writers are the ones that
must be lockless (those can happen in NMIs).
The main change is to the "head_page" pointer. We write to the
tail, and read from the head. The "head_page" pointer in the cpu
buffer is now just a reference to where to look. The real head
page is now kept in the head_page->list->prev->next pointer.
That is, in the list head of the previous page we set flags.
The list pages are allocated to be aligned such that the lowest
significant bits are always zero pointing to the list. This gives
us play to put in flags to their pointers.
bit 0: set when the page is a head page
bit 1: set when the writer is moving the page (for overwrite mode)
cmpxchg is used to update the pointer.
When the writer wraps the buffer and the tail meets the head,
in overwrite mode, the writer must move the head page forward.
It first uses cmpxchg to change the pointer flag from 1 to 2.
Once this is done, the reader on another CPU will not take the
page from the buffer.
The writers need to protect against interrupts (we don't bother with
disabling interrupts because NMIs are allowed to write too).
After the writer sets the pointer flag to 2, it takes care to
manage interrupts coming in. This is discribed in detail within the
comments of the code.
Changes in version 2:
- Let reader reset entries value of header page.
- Fix tail page passing commit page on reader page test.
- Always increment entries and write counter in rb_tail_page_update
- Add safety check in rb_set_commit_to_write to break out of infinite loop
- add mask in rb_is_reader_page
[ Impact: lock free writing to the ring buffer ]
Signed-off-by: Steven Rostedt <srostedt@redhat.com>
2009-07-07 18:36:12 -04:00
gcov: enable GCOV_PROFILE_ALL for x86_64
2009-06-18 13:03:58 -07:00
Merge branch 'irq-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-06-20 11:30:01 -07:00
mm, PM/Freezer: Disable OOM killer when tasks are frozen
2009-06-16 19:47:40 -07:00
Merge branch 'timers-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-06-20 10:51:44 -07:00
ring-buffer: make lockless
2009-07-07 18:36:12 -04:00
[CVE-2009-0029] System call wrappers part 04
2009-01-14 14:15:19 +01:00
async: Fix lack of boot-time console due to insufficient synchronization
2009-06-08 12:31:53 -07:00
Audit: remove spaces from audit_log_d_path
2009-04-05 13:49:04 -04:00
fixing audit rule ordering mess, part 1
2009-01-04 15:14:41 -05:00
Switch collect_mounts() to struct path
2009-06-11 21:36:01 -04:00
inotify: use GFP_NOFS in kernel_event() to work around a lockdep false-positive
2009-05-06 16:36:09 -07:00
Audit: remove spaces from audit_log_d_path
2009-04-05 13:49:04 -04:00
[CVE-2009-0029] System call wrappers part 04
2009-01-14 14:15:19 +01:00
cgroups: forbid noprefix if mounting more than just cpuset subsystem
2009-06-18 13:03:46 -07:00
debug cgroup: remove unneeded cgroup_lock
2009-04-02 19:04:54 -07:00
freezer_cg: disable writing freezer.state of root cgroup
2008-11-12 17:17:16 -08:00
signals: implement sys_rt_tgsigqueueinfo
2009-04-30 19:24:24 +02:00
mm/init: cpu_hotplug_init() must be initialized before SLAB
2009-06-22 21:18:12 -07:00
cpuset,mm: update tasks' mems_allowed in time
2009-06-16 19:47:31 -07:00
CRED: Inaugurate COW credentials
2008-11-14 10:39:23 +11:00
CRED: Rename cred_exec_mutex to reflect that it's a guard against ptrace
2009-05-11 08:15:36 +10:00
schedstat: consolidate per-task cpu runtime stats
2008-12-18 13:54:01 +01:00
dma-coherent: Restore dma_alloc_from_coherent() large alloc fall back policy.
2009-01-21 18:51:53 +09:00
Get rid of indirect include of fs_struct.h
2009-03-31 23:00:27 -04:00
ptrace: wait_task_zombie: do not account traced sub-threads
2009-06-19 16:46:06 -07:00
Merge branch 'tracing-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-04-05 11:04:19 -07:00
copy_process(): remove the unneeded clear_tsk_thread_flag(TIF_SIGPENDING)
2009-06-18 13:03:52 -07:00
freezer_cg: use thaw_process() in unfreeze_cgroup()
2008-10-30 11:38:45 -07:00
futex: fix restart in wait_requeue_pi
2009-05-20 10:34:32 +02:00
CRED: Use RCU to access another task's creds and to release a task's own creds
2008-11-14 10:39:19 +11:00
groups: move code to kernel/groups.c
2009-06-16 19:47:48 -07:00
Merge branch 'linux-next' of git://git.infradead.org/ubifs-2.6
2009-06-17 09:46:33 -07:00
softlockup: ensure the task has been switched out once
2009-02-11 11:04:16 +01:00
timers: split process wide cpu clocks/timers
2009-02-05 13:04:33 +01:00
kernel/kallsyms.c: replace deprecated __initcall with device_initcall and fix whitespace
2009-06-09 22:37:52 +02:00
rcu: provide RCU options on non-preempt architectures too
2008-12-25 09:31:28 +01:00
PM core: rename suspend and resume functions
2009-06-12 21:32:31 +02:00
kernel/kfifo.c: replace conditional test with is_power_of_2()
2009-06-16 19:47:47 -07:00
sysrq, intel_fb: fix sysrq g collision
2009-05-15 07:56:24 -05:00
kmod: Release sub_info on cred allocation failure.
2009-05-26 12:11:19 -07:00
kprobes: Clean up insn_pages by using list instead of hlist
2009-07-01 10:43:41 +02:00
kernel/ksysfs.c:fix dependence on CONFIG_NET
2009-01-06 10:44:31 -08:00
kthreads: rework kthread_stop()
2009-06-18 13:03:54 -07:00
sched, latencytop: incorporate review feedback from Andrew Morton
2009-02-11 10:18:04 +01:00
Merge branch 'linus' into tracing/core
2009-05-07 11:17:34 +02:00
lockdep: increase MAX_LOCKDEP_ENTRIES and MAX_LOCKDEP_CHAINS
2009-05-12 19:59:52 +02:00
lockstat: warn about disabled lock debugging
2009-02-14 23:28:28 +01:00
lockdep: move state bit definitions around
2009-02-14 23:27:59 +01:00
ring-buffer: Make it generally available
2009-06-25 10:31:30 +02:00
markers/tracpoints: fix non-modular build
2008-11-16 09:52:03 +01:00
kernel: constructor support
2009-06-18 13:03:57 -07:00
mutex: implement adaptive spinning
2009-01-14 18:09:02 +01:00
mutex: implement adaptive spinning
2009-01-14 18:09:02 +01:00
Merge branch 'linus' into perfcounters/core
2009-06-11 17:55:42 +02:00
mutex: implement adaptive spinning
2009-01-14 18:09:02 +01:00
Merge commit 'v2.6.28-rc6' into core/debug
2008-11-26 08:22:50 +01:00
cgroups: relax ns_can_attach checks to allow attaching to grandchild cgroups
2009-04-02 19:04:53 -07:00
nsproxy: extract create_nsproxy()
2009-06-18 13:03:56 -07:00
Fix caller information for warn_slowpath_null
2009-05-16 13:41:28 -07:00
module_param: allow 'bool' module_params to be bool, not just int.
2009-06-12 21:46:58 +09:30
perf_counter: Push perf_sample_data through the swcounter code
2009-06-20 12:30:30 +02:00
pids: clean up find_task_by_pid variants
2009-06-18 13:03:55 -07:00
pidns: rewrite copy_pid_ns()
2009-06-18 13:03:55 -07:00
kernel/posix-cpu-timers.c: fix sparse warning
2009-04-30 08:08:31 +02:00
[CVE-2009-0029] System call wrappers part 05
2009-01-14 14:15:20 +01:00
printk: Add KERN_DEFAULT printk log-level
2009-06-16 11:02:28 -07:00
page allocator: do not check NUMA node ID when the caller knows the node is valid
2009-06-16 19:47:32 -07:00
ptrace: don't take tasklist to get/set ->last_siginfo
2009-06-18 13:03:52 -07:00
kmemtrace, rcu: fix linux/rcutree.h and linux/rcuclassic.h dependencies
2009-04-03 12:23:02 +02:00
RCU: Don't try and predeclare inline funcs as it upsets some versions of gcc
2009-04-15 13:55:14 -07:00
rcu: rcu_sched_grace_period(): kill the bogus flush_signals()
2009-05-05 20:28:05 +02:00
"Tree RCU": scalable classic RCU implementation
2008-12-18 21:56:04 +01:00
cpumask: convert rcutorture.c
2009-03-30 22:05:16 +10:30
rcu: Add __rcu_pending tracing to hierarchical RCU
2009-04-14 11:33:43 +02:00
kmemtrace, rcu: fix rcu_tree_trace.c data structure dependencies
2009-04-03 12:23:03 +02:00
rcu: Add __rcu_pending tracing to hierarchical RCU
2009-04-14 11:33:43 +02:00
Merge branch 'tracing-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-04-05 11:04:19 -07:00
memcg: add interface to reset limits
2009-06-18 13:03:48 -07:00
Remove 'recurse into child resources' logic from 'reserve_region_with_split()'
2009-04-18 21:44:24 -07:00
trivial: fix ETIMEOUT -> ETIMEDOUT typos
2009-06-12 18:01:50 +02:00
rt_mutex: add proxy lock routines
2009-04-06 11:14:02 +02:00
Merge branch 'perfcounters-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-06-20 11:29:32 -07:00
sched: Fix fallback sched_clock()'s offset when using jiffies
2009-05-09 10:08:19 +02:00
sched: Remove unneeded __ref tag
2009-06-17 16:08:04 +02:00
cpumask: remove cpumask_t from core
2009-03-30 22:05:17 +10:30
sched: Hide runqueues from direct refer at source code level
2009-06-17 18:29:42 +02:00
sched: Fix out of scope variable access in sched_slice()
2009-06-17 18:37:54 +02:00
Merge branch 'locking-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-03-30 17:17:35 -07:00
sched, timers: move calc_load() to scheduler
2009-05-15 15:32:45 +02:00
cpumask: alloc zeroed cpumask for static cpumask_var_ts
2009-06-09 22:30:27 +09:30
sched: remove unused fields from struct rq
2009-03-24 23:16:51 +01:00
x86-64: seccomp: fix 32/64 syscall hole
2009-03-02 15:41:30 -08:00
ptrace: do_notify_parent_cldstop: fix the wrong ->nsproxy usage
2009-06-18 13:03:52 -07:00
slow-work: use round_jiffies() for thread pool's cull and OOM timers
2009-06-16 19:47:49 -07:00
cpumask: alloc zeroed cpumask for static cpumask_var_ts
2009-06-09 22:30:27 +09:30
softirq: introduce statistics for softirq
2009-06-18 13:03:40 -07:00
softlockup: decouple hung tasks check from softlockup detection
2009-01-16 14:06:04 +01:00
Allow rwlocks to re-enable interrupts
2009-04-02 19:05:11 -07:00
stacktrace: provide save_stack_trace_tsk() weak alias
2008-12-25 11:44:43 +01:00
cpumask: remove cpumask_t from core
2009-03-30 22:05:17 +10:30
groups: move code to kernel/groups.c
2009-06-16 19:47:48 -07:00
Merge commit 'v2.6.29-rc2' into perfcounters/core
2009-01-21 16:37:27 +01:00
sysctl.c: remove unused variable
2009-06-18 13:03:54 -07:00
net: add ARP notify option for devices
2009-02-01 01:04:33 -08:00
cpumask: convert rest of files in kernel/
2009-01-01 10:12:28 +10:30
kprobes: add tests for register_kprobes
2009-01-06 15:59:20 -08:00
[CVE-2009-0029] System call wrappers part 01
2009-01-14 14:15:18 +01:00
Merge branch 'timers-for-linus-migration' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
2009-06-15 10:06:19 -07:00
tracepoints: dont update zero-sized tracepoint sections
2009-03-18 19:55:00 +01:00
Fix fixpoint divide exception in acct_update_integrals
2009-03-09 08:13:35 -07:00
[CVE-2009-0029] System call wrappers part 19
2009-01-14 14:15:26 +01:00
smp_call_function_single(): be slightly less stupid, fix #2
2009-01-12 16:04:37 +01:00
sched: delayed cleanup of user_struct
2009-06-15 21:30:23 -07:00
Fix recursive lock in free_uid()/free_user_ns()
2009-02-27 16:26:21 -08:00
utsns: extract creeate_uts_ns()
2009-06-18 13:03:55 -07:00
proc_sysctl: use CONFIG_PROC_SYSCTL around ipc and utsname proc_handlers
2009-04-02 19:05:01 -07:00
wait: don't use __wake_up_common()
2009-04-14 17:17:16 +02:00
ftrace, workqueuetrace: make workqueue tracepoints use TRACE_EVENT macro
2009-06-02 01:10:40 +02:00
77ae365eca