mirror of
https://github.com/a52xq-dev/kernel.git
synced 2026-10-22 18:28:11 +07:00
| SHA1 | ||
|---|---|---|
Userspace processes often have multiple allocators that each do
anonymous mmaps to get memory. When examining memory usage of
individual processes or systems as a whole, it is useful to be
able to break down the various heaps that were allocated by
each layer and examine their size, RSS, and physical memory
usage.
This patch adds a user pointer to the shared union in
vm_area_struct that points to a null terminated string inside
the user process containing a name for the vma. vmas that
point to the same address will be merged, but vmas that
point to equivalent strings at different addresses will
not be merged.
Userspace can set the name for a region of memory by calling
prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, start, len, (unsigned long)name);
Setting the name to NULL clears it.
The names of named anonymous vmas are shown in /proc/pid/maps
as [anon:<name>] and in /proc/pid/smaps in a new "Name" field
that is only present for named vmas. If the userspace pointer
is no longer valid all or part of the name will be replaced
with "<fault>".
The idea to store a userspace pointer to reduce the complexity
within mm (at the expense of the complexity of reading
/proc/pid/mem) came from Dave Hansen. This results in no
runtime overhead in the mm subsystem other than comparing
the anon_name pointers when considering vma merging. The pointer
is stored in a union with fieds that are only used on file-backed
mappings, so it does not increase memory usage.
Includes fix from Jed Davis <jld@mozilla.com> for typo in
prctl_set_vma_anon_name, which could attempt to set the name
across two vmas at the same time due to a typo, which might
corrupt the vma list. Fix it to use tmp instead of end to limit
the name setting to a single vma at a time.
Bug: 120441514
Change-Id: I9aa7b6b5ef536cd780599ba4e2fba8ceebe8b59f
Signed-off-by: Dmitry Shmidt <dimitrysh@google.com>
[AmitP: Fix get_user_pages_remote() call to align with upstream commit
Colin Cross
authored and
 Alistair Strachan
commited |
533e4ed309 | |
A ~10% regression has been reported for UnixBench's execl throughput test by Aaron Lu and Ye Xiaolong: https://lkml.org/lkml/2018/10/30/765 That test is pretty simple, it does a "recursive" execve() syscall on the same binary. Starting from the syscall, this sequence is possible: do_execve() do_execveat_common() __do_execve_file() sched_exec() select_task_rq_fair() <==| Task already enqueued find_idlest_cpu() find_idlest_group() capacity_spare_wake() <==| Functions not called from cpu_util_wake() | the wakeup path which means we can end up calling cpu_util_wake() not only from the "wakeup path", as its name would suggest. Indeed, the task doing an execve() syscall is already enqueued on the CPU we want to get the cpu_util_wake() for. The estimated utilization for a CPU computed in cpu_util_wake() was written under the assumption that function can be called only from the wakeup path. If instead the task is already enqueued, we end up with a utilization which does not remove the current task's contribution from the estimated utilization of the CPU. This will wrongly assume a reduced spare capacity on the current CPU and increase the chances to migrate the task on execve. The regression is tracked down to: commit 
Patrick Bellasi
authored and
Alistair Strachan
commited |
5558f0e81f | |
Bug: 118439987 Bug: 120440300 Change-Id: I46a509df5e3bcb5253717d083f90679e7a72d378 Signed-off-by: Alistair Strachan <astrachan@google.com>
Alistair Strachan
commited |
409c3ce064 | |
[ Upstream commit 40fa3780bac2b654edf23f6b13f4e2dd550aea10 ]
When running on linux-next (8c60c36d0b8c ("Add linux-next specific files
for 20181019")) + CONFIG_PROVE_LOCKING=y on a big.LITTLE system (e.g.
Juno or HiKey960), we get the following report:
[ 0.748225] Call trace:
[ 0.750685] lockdep_assert_cpus_held+0x30/0x40
[ 0.755236] static_key_enable_cpuslocked+0x20/0xc8
[ 0.760137] build_sched_domains+0x1034/0x1108
[ 0.764601] sched_init_domains+0x68/0x90
[ 0.768628] sched_init_smp+0x30/0x80
[ 0.772309] kernel_init_freeable+0x278/0x51c
[ 0.776685] kernel_init+0x10/0x108
[ 0.780190] ret_from_fork+0x10/0x18
The static_key in question is 'sched_asym_cpucapacity' introduced by
commit:
df054e8445a4 ("sched/topology: Add static_key for asymmetric CPU capacity optimizations")
In this particular case, we enable it because smp_prepare_cpus() will
end up fetching the capacity-dmips-mhz entry from the devicetree,
so we already have some asymmetry detected when entering sched_init_smp().
This didn't get detected in tip/sched/core because we were missing:
commit cb538267ea1e ("jump_label/lockdep: Assert we hold the hotplug lock for _cpuslocked() operations")
Calls to build_sched_domains() post sched_init_smp() will hold the
hotplug lock, it just so happens that this very first call is a
special case. As stated by a comment in sched_init_smp(), "There's no
userspace yet to cause hotplug operations" so this is a harmless
warning.
However, to both respect the semantics of underlying
callees and make lockdep happy, take the hotplug lock in
sched_init_smp(). This also satisfies the comment atop
sched_init_domains() that says "Callers must hold the hotplug lock".
Reported-by: Sudeep Holla <sudeep.holla@arm.com>
Tested-by: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Dietmar.Eggemann@arm.com
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: morten.rasmussen@arm.com
Cc: quentin.perret@arm.com
Link: http://lkml.kernel.org/r/1540301851-3048-1-git-send-email-valentin.schneider@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
Valentin Schneider
authored and
 Greg Kroah-Hartman
commited |
b1e814e482 | |
[ Upstream commit 28c2fae726bf5003cd209b0d5910a642af98316f ] While 
Daniel Borkmann
authored and
Greg Kroah-Hartman
commited |
1a7ccf42cb | |
This reverts commit
Greg Kroah-Hartman
commited |
6b18878334 | |
commit 568fb6f42ac6851320adaea25f8f1b94de14e40a upstream. Since commit 
Christophe Leroy
authored and
Greg Kroah-Hartman
commited |
401182ae09 | |
commit dded2e159208a9edc21dd5c5f583afa28d378d39 upstream.
On a powerpc 8xx, 'btc' fails as follows:
Entering kdb (current=0x(ptrval), pid 282) due to Keyboard Entry
kdb> btc
btc: cpu status: Currently on cpu 0
Available cpus: 0
kdb_getarea: Bad address 0x0
when booting the kernel with 'debug_boot_weak_hash', it fails as well
Entering kdb (current=0xba99ad80, pid 284) due to Keyboard Entry
kdb> btc
btc: cpu status: Currently on cpu 0
Available cpus: 0
kdb_getarea: Bad address 0xba99ad80
On other platforms, Oopses have been observed too, see
https://github.com/linuxppc/linux/issues/139
This is due to btc calling 'btt' with %p pointer as an argument.
This patch replaces %p by %px to get the real pointer value as
expected by 'btt'
Fixes:
Christophe Leroy
authored and
Greg Kroah-Hartman
commited |
47052af237 | |
[ Upstream commit 59158ec4aef7d44be51a6f3e7e17fc64c32604eb ] Current kprobe event doesn't checks correctly whether the given event is on unloaded module or not. It just checks the event has ":" in the name. That is not enough because if we define a probe on non-exist symbol on loaded module, it allows to define that (with warning message) To ensure it correctly, this searches the module name on loaded module list and only if there is not, it allows to define it. (this event will be available when the target module is loaded) Link: http://lkml.kernel.org/r/153547309528.26502.8300278470528281328.stgit@devbox Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org> Signed-off-by: Sasha Levin <sashal@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
Masami Hiramatsu
authored and
Greg Kroah-Hartman
commited |
784c2eb37b | |
commit 1ae80cf31938c8f77c37a29bbe29e7f1cd492be8 upstream. The map-in-map frequently serves as a mechanism for atomic snapshotting of state that a BPF program might record. The current implementation is dangerous to use in this way, however, since userspace has no way of knowing when all programs that might have retrieved the "old" value of the map may have completed. This change ensures that map update operations on map-in-map map types always wait for all references to the old map to drop before returning to userspace. Signed-off-by: Daniel Colascione <dancol@google.com> Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Chenbo Feng <fengc@google.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
Daniel Colascione
authored and
Greg Kroah-Hartman
commited |
34d9615249 | |
commit d2f007dbe7e4c9583eea6eb04d60001e85c6f1bd upstream.
The current logic first clones the extent array and sorts both copies, then
maps the lower IDs of the forward mapping into the lower namespace, but
doesn't map the lower IDs of the reverse mapping.
This means that code in a nested user namespace with >5 extents will see
incorrect IDs. It also breaks some access checks, like
inode_owner_or_capable() and privileged_wrt_inode_uidgid(), so a process
can incorrectly appear to be capable relative to an inode.
To fix it, we have to make sure that the "lower_first" members of extents
in both arrays are translated; and we have to make sure that the reverse
map is sorted *after* the translation (since otherwise the translation can
break the sorting).
This is CVE-2018-18955.
Fixes:
Jann Horn
authored and
Greg Kroah-Hartman
commited |
9a7a80fb02 | |
commit 18858511fd8a877303cc34c06efa461b26a0e070 upstream.
Return -ENOENT error if there is no target synthetic event.
This notices an operation failure to user as below;
# echo 'wakeup_latency u64 lat; pid_t pid;' > synthetic_events
# echo '!wakeup' >> synthetic_events
sh: write error: No such file or directory
Link: http://lkml.kernel.org/r/154013449986.25576.9487131386597290172.stgit@devbox
Acked-by: Tom Zanussi <zanussi@linux.intel.com>
Tested-by: Tom Zanussi <zanussi@linux.intel.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Rajvi Jingar <rajvi.jingar@intel.com>
Cc: stable@vger.kernel.org
Fixes:
Masami Hiramatsu
authored and
Greg Kroah-Hartman
commited |
2b3171ec5a | |
commit 746a923b863a1065ef77324e1e43f19b1a3eab5c upstream. Commit 
Lukas Wunner
authored and
Greg Kroah-Hartman
commited |
e6d2f788cb | |
commit 277fcdb2cfee38ccdbe07e705dbd4896ba0c9930 upstream. log_buf_len_setup does not check input argument before passing it to simple_strtoull. The argument would be a NULL pointer if "log_buf_len", without its value, is set in command line and thus causes the following panic. PANIC: early exception 0xe3 IP 10:ffffffffaaeacd0d error 0 cr2 0x0 [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.19.0-rc4-yocto-standard+ #1 [ 0.000000] RIP: 0010:_parse_integer_fixup_radix+0xd/0x70 ... [ 0.000000] Call Trace: [ 0.000000] simple_strtoull+0x29/0x70 [ 0.000000] memparse+0x26/0x90 [ 0.000000] log_buf_len_setup+0x17/0x22 [ 0.000000] do_early_param+0x57/0x8e [ 0.000000] parse_args+0x208/0x320 [ 0.000000] ? rdinit_setup+0x30/0x30 [ 0.000000] parse_early_options+0x29/0x2d [ 0.000000] ? rdinit_setup+0x30/0x30 [ 0.000000] parse_early_param+0x36/0x4d [ 0.000000] setup_arch+0x336/0x99e [ 0.000000] start_kernel+0x6f/0x4ee [ 0.000000] x86_64_start_reservations+0x24/0x26 [ 0.000000] x86_64_start_kernel+0x6f/0x72 [ 0.000000] secondary_startup_64+0xa4/0xb0 This patch adds a check to prevent the panic. Link: http://lkml.kernel.org/r/1538239553-81805-1-git-send-email-zhe.he@windriver.com Cc: stable@vger.kernel.org Cc: rostedt@goodmis.org Cc: linux-kernel@vger.kernel.org Signed-off-by: He Zhe <zhe.he@windriver.com> Reviewed-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Signed-off-by: Petr Mladek <pmladek@suse.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
He Zhe
authored and
Greg Kroah-Hartman
commited |
e945325eb7 | |
commit 6a32c2469c3fbfee8f25bcd20af647326650a6cf upstream. Building any configuration with 'make W=1' produces a warning: kernel/bounds.c:16:6: warning: no previous prototype for 'foo' [-Wmissing-prototypes] When also passing -Werror, this prevents us from building any other files. Nobody ever calls the function, but we can't make it 'static' either since we want the compiler output. Calling it 'main' instead however avoids the warning, because gcc does not insist on having a declaration for main. Link: http://lkml.kernel.org/r/20181005083313.2088252-1-arnd@arndb.de Signed-off-by: Arnd Bergmann <arnd@arndb.de> Reported-by: Kieran Bingham <kieran.bingham+renesas@ideasonboard.com> Reviewed-by: Kieran Bingham <kieran.bingham+renesas@ideasonboard.com> Cc: David Laight <David.Laight@ACULAB.COM> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
Arnd Bergmann
authored and
Greg Kroah-Hartman
commited |
1e36467f65 | |
commit a36700589b85443e28170be59fa11c8a104130a5 upstream.
While fixing an out of bounds array access in known_siginfo_layout
reported by the kernel test robot it became apparent that the same bug
exists in siginfo_layout and affects copy_siginfo_from_user32.
The straight forward fix that makes guards against making this mistake
in the future and should keep the code size small is to just take an
unsigned signal number instead of a signed signal number, as I did to
fix known_siginfo_layout.
Cc: stable@vger.kernel.org
Fixes:
Eric W. Biederman
authored and
Greg Kroah-Hartman
commited |
04eb71942e | |
[ Upstream commit 3597dfe01d12f570bc739da67f857fd222a3ea66 ]
Instead of playing whack-a-mole and changing SEND_SIG_PRIV to
SEND_SIG_FORCED throughout the kernel to ensure a pid namespace init
gets signals sent by the kernel, stop allowing a pid namespace init to
ignore SIGKILL or SIGSTOP sent by the kernel. A pid namespace init is
only supposed to be able to ignore signals sent from itself and
children with SIG_DFL.
Fixes:
Eric W. Biederman
authored and
Greg Kroah-Hartman
commited |
cd0852081d | |
[ Upstream commit a9c676bc8fc58d00eea9836fb14ee43c0346416a ]
Edward Cree says:
In check_mem_access(), for the PTR_TO_CTX case, after check_ctx_access()
has supplied a reg_type, the other members of the register state are set
appropriately. Previously reg.range was set to 0, but as it is in a
union with reg.map_ptr, which is larger, upper bytes of the latter were
left in place. This then caused the memcmp() in regsafe() to fail,
preventing some branches from being pruned (and occasionally causing the
same program to take a varying number of processed insns on repeated
verifier runs).
Fix the instability by clearing bpf_reg_state in __mark_reg_[un]known()
Fixes:
Alexei Starovoitov
authored and
Greg Kroah-Hartman
commited |
e12da4273c | |
[ Upstream commit 819319fc93461c07b9cdb3064f154bd8cfd48172 ] Make reuse_unused_kprobe() to return error code if it fails to reuse unused kprobe for optprobe instead of calling BUG_ON(). Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Cc: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com> Cc: David S . Miller <davem@davemloft.net> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Naveen N . Rao <naveen.n.rao@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: http://lkml.kernel.org/r/153666124040.21306.14150398706331307654.stgit@devbox Signed-off-by: Ingo Molnar <mingo@kernel.org> Signed-off-by: Sasha Levin <sashal@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Masami Hiramatsu
authored and
Greg Kroah-Hartman
commited |
82b0f70fd3 | |
[ Upstream commit 22839869f21ab3850fbbac9b425ccc4c0023926f ] The sigaltstack(2) system call fails with -ENOMEM if the new alternative signal stack is found to be smaller than SIGMINSTKSZ. On architectures such as arm64, where the native value for SIGMINSTKSZ is larger than the compat value, this can result in an unexpected error being reported to a compat task. See, for example: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=904385 This patch fixes the problem by extending do_sigaltstack to take the minimum signal stack size as an additional parameter, allowing the native and compat system call entry code to pass in their respective values. COMPAT_SIGMINSTKSZ is just defined as SIGMINSTKSZ if it has not been defined by the architecture. Cc: Arnd Bergmann <arnd@arndb.de> Cc: Dominik Brodowski <linux@dominikbrodowski.net> Cc: "Eric W. Biederman" <ebiederm@xmission.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Oleg Nesterov <oleg@redhat.com> Reported-by: Steve McIntyre <steve.mcintyre@arm.com> Tested-by: Steve McIntyre <93sam@debian.org> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Sasha Levin <sashal@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
Will Deacon
authored and
Greg Kroah-Hartman
commited |
5445a4b0ff | |
[ Upstream commit 9506a7425b094d2f1d9c877ed5a78f416669269b ] It was found that when debug_locks was turned off because of a problem found by the lockdep code, the system performance could drop quite significantly when the lock_stat code was also configured into the kernel. For instance, parallel kernel build time on a 4-socket x86-64 server nearly doubled. Further analysis into the cause of the slowdown traced back to the frequent call to debug_locks_off() from the __lock_acquired() function probably due to some inconsistent lockdep states with debug_locks off. The debug_locks_off() function did an unconditional atomic xchg to write a 0 value into debug_locks which had already been set to 0. This led to severe cacheline contention in the cacheline that held debug_locks. As debug_locks is being referenced in quite a few different places in the kernel, this greatly slow down the system performance. To prevent that trashing of debug_locks cacheline, lock_acquired() and lock_contended() now checks the state of debug_locks before proceeding. The debug_locks_off() function is also modified to check debug_locks before calling __debug_locks_off(). Signed-off-by: Waiman Long <longman@redhat.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Link: http://lkml.kernel.org/r/1539913518-15598-1-git-send-email-longman@redhat.com Signed-off-by: Ingo Molnar <mingo@kernel.org> Signed-off-by: Sasha Levin <sashal@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
Waiman Long
authored and
Greg Kroah-Hartman
commited |
117d5fbddd | |
commit 53c613fe6349994f023245519265999eed75957f upstream. STIBP is a feature provided by certain Intel ucodes / CPUs. This feature (once enabled) prevents cross-hyperthread control of decisions made by indirect branch predictors. Enable this feature if - the CPU is vulnerable to spectre v2 - the CPU supports SMT and has SMT siblings online - spectre_v2 mitigation autoselection is enabled (default) After some previous discussion, this leaves STIBP on all the time, as wrmsr on crossing kernel boundary is a no-no. This could perhaps later be a bit more optimized (like disabling it in NOHZ, experiment with disabling it in idle, etc) if needed. Note that the synchronization of the mask manipulation via newly added spec_ctrl_mutex is currently not strictly needed, as the only updater is already being serialized by cpu_add_remove_lock, but let's make this a little bit more future-proof. Signed-off-by: Jiri Kosina <jkosina@suse.cz> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Josh Poimboeuf <jpoimboe@redhat.com> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: "WoodhouseDavid" <dwmw@amazon.co.uk> Cc: Andi Kleen <ak@linux.intel.com> Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: "SchauflerCasey" <casey.schaufler@intel.com> Cc: stable@vger.kernel.org Link: https://lkml.kernel.org/r/nycvar.YFH.7.76.1809251438240.15880@cbobk.fhfr.pm Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
Jiri Kosina
authored and
Greg Kroah-Hartman
commited |
233b9d7df0 | |
commit a3ceed87b07769fb80ce9dc6b604e515dba14c4b upstream. early_cma does not check input argument before passing it to simple_strtoull. The argument would be a NULL pointer if "cma", without its value, is set in command line and thus causes the following panic. PANIC: early exception 0xe3 IP 10:ffffffffa3e9db8d error 0 cr2 0x0 [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.19.0-rc3-yocto-standard+ #7 [ 0.000000] RIP: 0010:_parse_integer_fixup_radix+0xd/0x70 ... [ 0.000000] Call Trace: [ 0.000000] simple_strtoull+0x29/0x70 [ 0.000000] memparse+0x26/0x90 [ 0.000000] early_cma+0x17/0x6a [ 0.000000] do_early_param+0x57/0x8e [ 0.000000] parse_args+0x208/0x320 [ 0.000000] ? rdinit_setup+0x30/0x30 [ 0.000000] parse_early_options+0x29/0x2d [ 0.000000] ? rdinit_setup+0x30/0x30 [ 0.000000] parse_early_param+0x36/0x4d [ 0.000000] setup_arch+0x336/0x99e [ 0.000000] start_kernel+0x6f/0x4e6 [ 0.000000] x86_64_start_reservations+0x24/0x26 [ 0.000000] x86_64_start_kernel+0x6f/0x72 [ 0.000000] secondary_startup_64+0xa4/0xb0 This patch adds a check to prevent the panic. Signed-off-by: He Zhe <zhe.he@windriver.com> Reviewed-by: Marek Szyprowski <m.szyprowski@samsung.com> Cc: stable@vger.kernel.org Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
He Zhe
authored and
Greg Kroah-Hartman
commited |
953983cbbf | |
commit 0962590e553331db2cc0aef2dc35c57f6300dbbe upstream.
ALU operations on pointers such as scalar_reg += map_value_ptr are
handled in adjust_ptr_min_max_vals(). Problem is however that map_ptr
and range in the register state share a union, so transferring state
through dst_reg->range = ptr_reg->range is just buggy as any new
map_ptr in the dst_reg is then truncated (or null) for subsequent
checks. Fix this by adding a raw member and use it for copying state
over to dst_reg.
Fixes:
Daniel Borkmann
authored and
Greg Kroah-Hartman
commited |
d93080cdba | |
When tasks come and go from a runqueue quickly, this can lead to boost being applied and removed quickly which sometimes means we cannot raise the CPU frequency again when we need to (due to the rate limit on frequency updates). This has proved to be a particular issue for RT tasks and alternative methods have been used in the past to work around it. This is an attempt to solve the issue for all task classes and cpufreq governors by introducing a generic mechanism in schedtune to retain the max boost level from task enqueue for a minimum period - defined here as 50ms. This timeout was determined experimentally and is not configurable. A sched_feat guards the application of this to tasks - in the default configuration, task boosting only applied to tasks which have RT policy. Change SCHEDTUNE_BOOST_HOLD_ALL to true to apply it to all tasks regardless of class. It works like so: Every task enqueue (in an allowed class) stores a cpu-local timestamp. If the task is not a member of an allowed class (all or RT depending upon feature selection), the timestamp is not updated. The boost group will stay active regardless of tasks present until 50ms beyond the last timestamp stored. We also store the timestamp of the active boost group to avoid unneccesarily revisiting the boost groups when checking CPU boost level. If the timestamp is more than 50ms in the past when we check boost then we re-evaluate the boost groups for that CPU, taking into account the timestamps associated with each group. Idea based on rt-boost-retention patches from Joel. Change-Id: I52cc2d2e82d1c5aa03550378c8836764f41630c1 Suggested-by: Joel Fernandes <joelaf@google.com> Reviewed-by: Patrick Bellasi <patrick.bellasi@arm.com> Signed-off-by: Chris Redpath <chris.redpath@arm.com> [forward ported from android-4.9-eas-dev proposal] (cherry picked from commit a485e8b7bf8e95759e600396feeb7bfb400b6e46) [ - Trivial cherry-pick conflicts in include/trace/events/sched.h ] Signed-off-by: Quentin Perret <quentin.perret@arm.com> 
Chris Redpath
authored and
 Quentin Perret
commited |
4cb65352a3 | |
rt tasks are currently not eligible for schedtune boosting. Make it so
by adding enqueue/dequeue hooks.
For rt tasks, schedtune only acts as a frequency boosting framework, it
has no impact on placement decisions and the prefer_idle attribute is
not used.
Also prepare schedutil use of boosted util for rt task boosting
With this change, schedtune accounting will include rt class tasks,
however boosting currently only applies to the utilization provided by
fair class tasks. Sum up the tracked CPU utilization applying boost to
the aggregate util instead - this includes RT task util in the boosting
if any tasks are runnable.
Scenario 1, considering one CPU:
1x rt task running, util 250, boost 0
1x cfs task runnable, util 250, boost 50
previous util=250+(50pct_boosted_250) = 887
new util=50_pct_boosted_500 = 762
Scenario 2, considering one CPU:
1x rt task running, util 250, boost 50
1x cfs task runnable, util 250, boost 0
previous util=250+250 = 500
new util=50_pct_boosted_500 = 762
Scenario 3, considering one CPU:
1x rt task running, util 250, boost 50
1x cfs task runnable, util 250, boost 50
previous util=250+(50pct_boosted_250) = 887
new util=50_pct_boosted_500 = 762
Scenario 4:
1x rt task running, util 250, boost 50
previous util=250 = 250
new util=50_pct_boosted_250 = 637
Change-Id: Ie287cbd0692468525095b5024db9faac8b2f4878
Signed-off-by: Chris Redpath <chris.redpath@arm.com>
(cherry picked from commit 8e266aebf737262aeca9662254a3e61ccc7f8dec)
[ - Fixed conflicts in sugov related to PELT signals of all classes
- Fixed conflicts related to boosted_cpu_util being in a different
location
- Moved the CFS/RT util aggregation out of schedutil_freq_util to
ease the integration with SchedTune ]
Signed-off-by: Quentin Perret <quentin.perret@arm.com>
Chris Redpath
authored and
Quentin Perret
commited |
cc2c83709c | |
In preparation for the introduction of the boost hold feature in SchedTune, make the cpu_util_* signals available even without sugov, hence making the stune integration a lot easier without breaking other platforms Change-Id: I77c14b6fd470a41ffb3ac510b76395ebc116ddfd Signed-off-by: Quentin Perret <quentin.perret@arm.com>
Quentin Perret
commited |
55679f6419 | |
Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com> Signed-off-by: Andres Oportus <andresoportus@google.com> (cherry picked from commit 8e45d941282039d5379f4e286e5bd0a2044e105c) [ - Trivial cherry pick issues - Changed commit title for consistency ] Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: I78fb5e82223558def0cf16105c233591cda81d5c
Patrick Bellasi
authored and
Quentin Perret
commited |
6dfaed989e | |
We want to be able to track rt_rq signals same as we do for other RQs. Signed-off-by: Chris Redpath <chris.redpath@arm.com> (cherry-picked from commit 574a2d189695c334ae290f522b098f05398a3765) [ - Fixed conflicts with the refactored RT util_avg tracking - Changed commit title for consistency with other tracepoint patches ] Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: I38b64baa50e1ff86019ca4b8b0a04af994880b35
Chris Redpath
authored and
Quentin Perret
commited |
3124a5b9d0 | |
Suggested-by: Patrick Bellasi <patrick.bellasi@arm.com> Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: I2c43bcb37f57844a07aa36e339da00180e65b6c2
Quentin Perret
commited |
8c850534bb | |
Adapated from the existing trace event from android-4.14. Change-Id: I9785e692fb0af087c236906d7f47fed1b20690f5 Signed-off-by: Quentin Perret <quentin.perret@arm.com>
Quentin Perret
commited |
171db7dcae | |
Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com> Change-Id: I65e294c454369cbc15a29370d8a13ce358a95c39
Patrick Bellasi
authored and
Quentin Perret
commited |
8eb64d5f73 | |
The trace event key load is mapped to:
(1) load : cfs_rq->tg->load_avg
The cfs_rq owned by the task_group is used as the only parameter for the
trace event because it has a reference to the taskgroup and the cpu.
Using the taskgroup as a parameter instead would require the cpu as a
second parameter. A task_group is global and not per-cpu data. The cpu
key only tells on which cpu the value was gathered.
The following list shows examples of the key=value pairs for:
(1) a task group:
cpu=1 path=/tg1/tg11/tg111 load=517
(2) an autogroup:
cpu=1 path=/autogroup-10 load=1050
We don't maintain a load signal for a root task group.
The trace event is only defined if cfs group scheduling support
(CONFIG_FAIR_GROUP_SCHED) is enabled.
Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Change-Id: I7de38e6b30a99d7c9887c94c707ded26b383b5f8
Dietmar Eggemann
authored and
Quentin Perret
commited |
915679307f | |
The following trace event keys are mapped to:
(1) load : se->avg.load_avg
(2) rbl_load : se->avg.runnable_load_avg
(3) util : se->avg.util_avg
To let this trace event work for configurations w/ and w/o group
scheduling support for cfs (CONFIG_FAIR_GROUP_SCHED) the following
special handling is necessary for non-existent key=value pairs:
path = "(null)" : In case of !CONFIG_FAIR_GROUP_SCHED or the
sched_entity represents a task.
comm = "(null)" : In case sched_entity represents a task_group.
pid = -1 : In case sched_entity represents a task_group.
The following list shows examples of the key=value pairs in different
configurations for:
(1) a task:
cpu=0 path=(null) comm=sshd pid=2206 load=102 rbl_load=102 util=102
(2) a taskgroup:
cpu=1 path=/tg1/tg11/tg111 comm=(null) pid=-1 load=882 rbl_load=882 util=510
(3) an autogroup:
cpu=0 path=/autogroup-13 comm=(null) pid=-1 load=49 rbl_load=49 util=48
(4) w/o CONFIG_FAIR_GROUP_SCHED:
cpu=0 path=(null) comm=sshd pid=2211 load=301 rbl_load=301 util=265
The trace event is only defined for CONFIG_SMP.
The helper functions __trace_sched_cpu(), __trace_sched_path() and
__trace_sched_id() are extended to deal with sched_entities as well.
Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
[ Fixed issues related to the new pelt.c file ]
Signed-off-by: Quentin Perret <quentin.perret@arm.com>
Change-Id: Id2e4d1ddb79c13412c80e4fa4147b9df3b1e212a
Dietmar Eggemann
authored and
Quentin Perret
commited |
4290369491 | |
The following trace event keys are mapped to:
(1) load : cfs_rq->avg.load_avg
(2) rbl_load : cfs_rq->avg.runnable_load_avg
(2) util : cfs_rq->avg.util_avg
To let this trace event work for configurations w/ and w/o group
scheduling support for cfs (CONFIG_FAIR_GROUP_SCHED) the following
special handling is necessary for a non-existent key=value pair:
path = "(null)" : In case of !CONFIG_FAIR_GROUP_SCHED.
The following list shows examples of the key=value pairs in different
configurations for:
(1) a root task_group:
cpu=4 path=/ load=6 rbl_load=6 util=331
(2) a task_group:
cpu=1 path=/tg1/tg11/tg111 load=538 rbl_load=538 util=522
(3) an autogroup:
cpu=3 path=/autogroup-18 load=997 rbl_load=997 util=517
(4) w/o CONFIG_FAIR_GROUP_SCHED:
cpu=0 path=(null) load=314 rbl_load=314 util=289
The trace event is only defined for CONFIG_SMP.
The helper function __trace_sched_path() can be used to get the length
parameter of the dynamic array (path == NULL) and to copy the path into
it (path != NULL).
Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
[ Fixed issues related to the new pelt.c file ]
Signed-off-by: Quentin Perret <quentin.perret@arm.com>
Change-Id: I1107044c52b74ecb3df69f3a45c1e530f0e59b1b
Dietmar Eggemann
authored and
Quentin Perret
commited |
b540360a9c | |
Define autogroup_path() even in the !CONFIG_SCHED_DEBUG case. If CONFIG_SCHED_AUTOGROUP is enabled the path of an autogroup has to be available to be printed in the load tracking trace events provided by this patch-stack regardless whether CONFIG_SCHED_DEBUG is set or not. Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Ingo Molnar <mingo@kernel.org> Change-Id: Iecf5466fc837f428ee545ddabe70024c152ff38d
Dietmar Eggemann
authored and
Quentin Perret
commited |
31d01ca5d4 | |
If we can classify the group as overloaded, that overrides any classification as misfit but we may still have misfit tasks present. Check the rq we're looking at to see if this is the case. Signed-off-by: Chris Redpath <chris.redpath@arm.com> [Removed stray reference to rq_has_misfit] Signed-off-by: Valentin Schneider <valentin.schneider@arm.com> Change-Id: Ida8eb66aa625e34de3fe2ee1b0dd8a78926273d8
Chris Redpath
authored and
Quentin Perret
commited |
df6cd51a54 | |
When load balancing in a system with misfit tasks present, if we always pull a misfit task to the local group this can lead to pulling a running task from a smaller capacity CPUs to a bigger CPU which is busy. In this situation, the pulled task is likely not to get a chance to run before an idle balance on another small CPU pulls it back. This penalises the pulled task as it is stopped for a short amount of time and then likely relocated to a different CPU (since the original CPU just did a NEWLY_IDLE balance and reset the periodic interval). If we only do this unconditionally for NEWLY_IDLE balance, we can be sure that any tasks and load which are present on the local group are related to short-running tasks which we are happy to displace for a longer running task in a system with misfit tasks present. However, other balance types should only pull a task if we think that the local group is underutilized - checking the number of tasks gives us a conservative estimate here since if they were short tasks we would have been doing NEWLY_IDLE balances instead. Signed-off-by: Chris Redpath <chris.redpath@arm.com> Change-Id: I710add1ab1139482620b6addc8370ad194791beb
Chris Redpath
authored and
Quentin Perret
commited |
de526a3400 | |
In some systems the capacity and group weights line up to defeat all the small imbalance correction conditions in fix_small_imbalance, which can cause bad task placement. Add a new condition if the existing code can't see anything to fix: If we have asymmetric capacity, and there are more tasks than CPUs in the busiest group *and* there are less tasks than CPUs in the local group then we try to pull something. There could be transient small tasks which prevent this from working, but on the whole it is beneficial for those systems with inconvenient capacity/cluster size relationships. Signed-off-by: Chris Redpath <chris.redpath@arm.com> Change-Id: Icf81cde215c082a61f816534b7990ccb70aee409
Chris Redpath
authored and
Quentin Perret
commited |
a5ece57c96 | |
The rate-limit tunable in the schedutil governor applies to transitions to both lower and higher frequencies. On several platforms it is not the ideal tunable though, as it is difficult to get best power/performance figures using the same limit in both directions. It is common on mobile platforms with demanding user interfaces to want to increase frequency rapidly for example but decrease slowly. One of the example can be a case where we have short busy periods followed by similar or longer idle periods. If we keep the rate-limit high enough, we will not go to higher frequencies soon enough. On the other hand, if we keep it too low, we will have too many frequency transitions, as we will always reduce the frequency after the busy period. It would be very useful if we can set low rate-limit while increasing the frequency (so that we can respond to the short busy periods quickly) and high rate-limit while decreasing frequency (so that we don't reduce the frequency immediately after the short busy period and that may avoid frequency transitions before the next busy period). Implement separate up/down transition rate limits. Note that the governor avoids frequency recalculations for a period equal to minimum of up and down rate-limit. A global mutex is also defined to protect updates to min_rate_limit_us via two separate sysfs files. Note that this wouldn't change behavior of the schedutil governor for the platforms which wish to keep same values for both up and down rate limits. This is tested with the rt-app [1] on ARM Exynos, dual A15 processor platform. Testcase: Run a SCHED_OTHER thread on CPU0 which will emulate work-load for X ms of busy period out of the total period of Y ms, i.e. Y - X ms of idle period. The values of X/Y taken were: 20/40, 20/50, 20/70, i.e idle periods of 20, 30 and 50 ms respectively. These were tested against values of up/down rate limits as: 10/10 ms and 10/40 ms. For every test we noticed a performance increase of 5-10% with the schedutil governor, which was very much expected. [Viresh]: Simplified user interface and introduced min_rate_limit_us + mutex, rewrote commit log and included test results. [1] https://github.com/scheduler-tools/rt-app/ Signed-off-by: Steve Muckle <smuckle.linux@gmail.com> Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org> (applied from https://marc.info/?l=linux-kernel&m=147936011103832&w=2) [trivial adaptations] Signed-off-by: Juri Lelli <juri.lelli@arm.com> [updated rate limiting & fixed conflicts] Signed-off-by: Chris Redpath <chris.redpath@arm.com> (cherry picked from commit 50c26fdb74563ec0cb4a83373d42667f4e83a23e) [Trivial cherry-pick conflicts] Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: I18720a83855b196b8e21dcdc8deae79131635b84 
Steve Muckle
authored and
Quentin Perret
commited |
61ac701960 | |
Schedutil always requests max frequency whenever a RT task is running. Now that we have a better estimate of the utilization of RT runqueues, it is possible to make a less conservative decision and scale frequency according to the needs of the RT tasks. To do so, protect the RT-go-to-max code with a new sched_feature. The sched_feature is disabled by default, hence favoring energy savings as required in mobile environments. Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: Ic9f01c8703d4f843addaa0d684012a422fe9f3b8
Quentin Perret
commited |
ad271f2d0d | |
Wakeup balancing uses cpu capacity awareness and needs to know the system-wide maximum cpu capacity. Patch "sched: Store system-wide maximum cpu capacity in root domain" finds the system-wide maximum cpu capacity during scheduler domain hierarchy setup. This is sufficient as long as maximum frequency invariance is not enabled. If it is enabled, the system-wide maximum cpu capacity can change between scheduler domain hierarchy setups due to frequency capping. The cpu capacity is changed in update_cpu_capacity() which is called in load balance on the lowest scheduler domain hierarchy level. To be able to know if a change in cpu capacity for a certain cpu also has an effect on the system-wide maximum cpu capacity it is normally necessary to iterate over all cpus. This would be way too costly. That's why this patch follows a different approach. The unsigned long max_cpu_capacity value in struct root_domain is replaced with a struct max_cpu_capacity, containing value (the max_cpu_capacity) and cpu (the cpu index of the cpu providing the maximum cpu_capacity). Changes to the system-wide maximum cpu capacity and the cpu index are made if: 1 System-wide maximum cpu capacity < cpu capacity 2 System-wide maximum cpu capacity > cpu capacity and cpu index == cpu There are no changes to the system-wide maximum cpu capacity in all other cases. Atomic read and write access to the pair (max_cpu_capacity.val, max_cpu_capacity.cpu) is enforced by max_cpu_capacity.lock. The access to max_cpu_capacity.val in task_fits_max() is still performed without taking the max_cpu_capacity.lock. The code to set max cpu capacity in build_sched_domains() has been removed because the whole functionality is now provided by update_cpu_capacity() instead. This approach can introduce errors temporarily, e.g. in case the cpu currently providing the max cpu capacity has its cpu capacity lowered due to frequency capping and calls update_cpu_capacity() before any cpu which might provide the max cpu now. Signed-off-by: Ionela Voinescu <ionela.voinescu@arm.com>* Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com> ( Fixed cherry-pick issues ) Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: Idaa7a16723001e222e476de34df332558e48dd13
Dietmar Eggemann
authored and
Quentin Perret
commited |
9c83d8435a | |
To be able to scale the cpu capacity by this factor introduce a call to the new arch scaling function arch_scale_max_freq_capacity() in update_cpu_capacity() and provide a default implementation which returns SCHED_CAPACITY_SCALE. Another subsystem (e.g. cpufreq) or architectural or platform specific code can overwrite this default implementation, exactly as for frequency and cpu invariance. It has to be enabled by the arch by defining arch_scale_max_freq_capacity to the actual implementation. Signed-off-by: Ionela Voinescu <ionela.voinescu@arm.com> Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com> ( Fixed conflict with scaling against the PELT-based scale_rt_capacity ) Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: I770a8b1f4f7340e9e314f71c64a765bf880f4b4d
Dietmar Eggemann
authored and
Quentin Perret
commited |
6c401bf9b0 | |
If the only pre-selected candidate CPU in find_energy_efficient_cpu() happens to be prev_cpu, there is not point in computing the system energy since we have nothing to compare it against, so we currently bail out early. The same logic can be extended when prefer_idle tasks are routed in the energy-aware wake-up path: if the only candidate is idle for a prefer_idle task, just select it no matter what the energy impact is. That should help speeding-up wake-ups of prefer_idle tasks, at least when find_best_target() is used for them. Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: Idd0e387e4a766061cc05d2584df3a31e4dabfd09
Quentin Perret
commited |
c06013206c | |
Use the upstream slow path to find an idle cpu for prefer-idle tasks. This slow-path is actually faster than the EAS path we are currently going through (compute_energy()) which is really slow. No performance degradation is seen with this and it reduces the delta quite a bit between upstream and out of tree code. It's not clear yet if using the mainline slow path task placement when a task has the schedtune attribute prefer_idle=1 is the right thing to do for products. Put the option to disable this behind a sched feature so we can try out both options. Signed-off-by: Joel Fernandes <joelaf@google.com> (refactored for 4.14 version) Signed-off-by: Chris Redpath <chris.redpath@arm.com> (cherry picked from commit c0ff131c88f68e4985793663144b6f9cf77be9d3) [ - Refactored for 4.17 version - Adjusted the commit header to the new function names ] Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: Icf762a101c92c0e3f9e61df0370247fa15455581
Chris Redpath
authored and
Quentin Perret
commited |
1d00c33d8b | |
This patch adds a parameter to select_task_rq, sibling_count_hint
allowing the caller, where it has this information, to inform the
sched_class the number of tasks that are being woken up as part of
the same event.
The wake_q mechanism is one case where this information is available.
select_task_rq_fair can then use the information to detect that it
needs to widen the search space for task placement in order to avoid
overloading the last-level cache domain's CPUs.
* * *
The reason I am investigating this change is the following use case
on ARM big.LITTLE (asymmetrical CPU capacity): 1 task per CPU, which
all repeatedly do X amount of work then
pthread_barrier_wait (i.e. sleep until the last task finishes its X
and hits the barrier). On big.LITTLE, the tasks which get a "big" CPU
finish faster, and then those CPUs pull over the tasks that are still
running:
v CPU v ->time->
-------------
0 (big) 11111 /333
-------------
1 (big) 22222 /444|
-------------
2 (LITTLE) 333333/
-------------
3 (LITTLE) 444444/
-------------
Now when task 4 hits the barrier (at |) and wakes the others up,
there are 4 tasks with prev_cpu=<big> and 0 tasks with
prev_cpu=<little>. want_affine therefore means that we'll only look
in CPUs 0 and 1 (sd_llc), so tasks will be unnecessarily coscheduled
on the bigs until the next load balance, something like this:
v CPU v ->time->
------------------------
0 (big) 11111 /333 31313\33333
------------------------
1 (big) 22222 /444|424\4444444
------------------------
2 (LITTLE) 333333/ \222222
------------------------
3 (LITTLE) 444444/ \1111
------------------------
^^^
underutilization
So, I'm trying to get want_affine = 0 for these tasks.
I don't _think_ any incarnation of the wakee_flips mechanism can help
us here because which task is waker and which tasks are wakees
generally changes with each iteration.
However pthread_barrier_wait (or more accurately FUTEX_WAKE) has the
nice property that we know exactly how many tasks are being woken, so
we can cheat.
It might be a disadvantage that we "widen" _every_ task that's woken in
an event, while select_idle_sibling would work fine for the first
sd_llc_size - 1 tasks.
IIUC, if wake_affine() behaves correctly this trick wouldn't be
necessary on SMP systems, so it might be best guarded by the presence
of SD_ASYM_CPUCAPACITY?
* * *
Final note..
In order to observe "perfect" behaviour for this use case, I also had
to disable the TTWU_QUEUE sched feature. Suppose during the wakeup
above we are working through the work queue and have placed tasks 3
and 2, and are about to place task 1:
v CPU v ->time->
--------------
0 (big) 11111 /333 3
--------------
1 (big) 22222 /444|4
--------------
2 (LITTLE) 333333/ 2
--------------
3 (LITTLE) 444444/ <- Task 1 should go here
--------------
If TTWU_QUEUE is enabled, we will not yet have enqueued task
2 (having instead sent a reschedule IPI) or attached its load to CPU
2. So we are likely to also place task 1 on cpu 2. Disabling
TTWU_QUEUE means that we enqueue task 2 before placing task 1,
solving this issue. TTWU_QUEUE is there to minimise rq lock
contention, and I guess that this contention is less of an issue on
big.LITTLE systems since they have relatively few CPUs, which
suggests the trade-off makes sense here.
Signed-off-by: Brendan Jackman <brendan.jackman@arm.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Josef Bacik <josef@toxicpanda.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Matt Fleming <matt@codeblueprint.co.uk>
( - Applied from https://patchwork.kernel.org/patch/9895261/
- Fixed trivial conflict in kernel/sched/core.c
- Fixed select_task_rq_idle, now in kernel/sched/idle.c
- Fixed trivial conflict in select_task_rq_fair )
Signed-off-by: Quentin Perret <quentin.perret@arm.com>
Change-Id: I3cfc4bf48c3d7feef969db4d22449f4fbb4f795d
Brendan Jackman
authored and
Quentin Perret
commited |
88d05f9257 | |
Since we don't do energy-aware wakeups when we are overutilized, always honoring sync wakeups in this state does not prevent wake-wide mechanics overruling the flag as normal. This patch is based upon previous work to build EAS for android products. sync-hint code taken from commit 4a5e890ec60d "sched/fair: add tunable to force selection at cpu granularity" written by Juri Lelli <juri.lelli@arm.com> Signed-off-by: Chris Redpath <chris.redpath@arm.com> (cherry-picked from commit f1ec666a62dec1083ed52fe1ddef093b84373aaf) [ Moved the feature to find_energy_efficient_cpu() ] Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: I4b3d79141fc8e53dc51cd63ac11096c2e3cb10f5
Chris Redpath
authored and
Quentin Perret
commited |
af362094df | |
find_best_target started life as an optimised energy cpu selection
algorithm designed to be efficient and high performance and integrate
well with schedtune and userspace cpuset configuration. It has had many
many small tweaks over time, and the version added here is forward
ported from the version used in android-4.4 and android-4.9. The linkage
to the rest of EAS is slightly different, however.
This version is split into the three main use-cases and addresses
them in priority order:
A) latency sensitive tasks
B) non latency sensitive tasks on IDLE CPUs
C) non latency sensitive tasks on ACTIVE CPUs
Case A) Latency sensitive tasks
Unconditionally favoring tasks that prefer idle CPU to improve latency.
When we do enter here, we are looking for:
- an idle CPU, whatever its idle_state is, since the first CPUs we
explore are more likely to be reserved for latency sensitive tasks.
- a non idle CPU where the task fits in its current capacity and has
the maximum spare capacity.
- a non idle CPU with lower contention from other tasks and running at
the lowest possible OPP.
The last two goals try to favor a non idle CPU where the task can run
as if it is "almost alone". A maximum spare capacity CPU is favored
since the task already fits into that CPU's capacity without waiting for
an OPP change.
For any case other than case A, we avoid CPUs which would become
overutilized if we placed the task there.
Case B) Non latency sensitive tasks on IDLE CPUs.
Find an optimal backup IDLE CPU for non latency sensitive tasks.
Here we are looking for:
- minimizing the capacity_orig,
i.e. preferring LITTLE CPUs
If IDLE cpus are available, we prefer to choose one in order to spread
tasks and improve performance.
Case C) Non latency sensitive tasks on ACTIVE CPUs.
Pack tasks in the most energy efficient capacities.
This task packing strategy prefers more energy efficient CPUs (i.e.
pack on smaller maximum capacity CPUs) while also trying to spread
tasks to run them all at the lower OPP.
This assumes for example that it's more energy efficient to run two tasks
on two CPUs at a lower OPP than packing both on a single CPU but running
that CPU at an higher OPP.
This code has had many other contributors over the development listed
here as Cc.
Cc: Ke Wang <ke.wang@spreadtrum.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Patrick Bellasi <patrick.bellasi@arm.com>
Cc: Valentin Schneider <valentin.schneider@arm.com>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Srinath Sridharan <srinathsr@google.com>
Cc: Todd Kjos <tkjos@google.com>
Cc: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Chris Redpath <chris.redpath@arm.com>
(cherry picked from commit f240e44406558b17ff7765f252b0bcdcbc15126f)
[ - Removed tracepoints
- Took capacity_curr_of() from "7f6fb82 ANDROID: sched: EAS: take
cstate into account when selecting idle core"
- Re-use sd_ea from find_energy_efficient_cpu() / removed start_cpu()
- Mark candidates with a cpumask given by feec()
- Squashed Ionela's tri-gear fbt fixes from android-4.14
- Squashed Patrick's changes related to util_est from android-4.14 ]
Signed-off-by: Quentin Perret <quentin.perret@arm.com>
Change-Id: I9500d308c879dd53b08adeda8f988238e39cc392
Chris Redpath
authored and
Quentin Perret
commited |
c27c56105d | |
find_energy_efficient_cpu() is composed of two steps; we first look for the CPU with the max spare capacity in each frequency domain, and then the impact on energy of each candidate is estimated. In order to make it easier to implement other CPU selection policies, let's factor the candidate selection algorithm out of find_energy_efficient_cpu(), and mark the candidates in a mask. This should result in no functional difference. Signed-off-by: Quentin Perret <quentin.perret@arm.com> Change-Id: I85a28880f01fcd11d7af28f9fbc1fe0cf4f197cf
Quentin Perret
commited |
9378697a9f | |
Introduce a new sysctl for this option, 'sched_cstate_aware'.
When this is enabled, the scheduler can make use of the idle state
indexes in order to break the tie between potential CPU candidates.
This patch is based on 7f6fb825d6bc ("ANDROID: sched: EAS: take cstate
into account when selecting idle core") from android-4.14. All the
credits goes to the authors.
Change-Id: Ia076cf32faff91e90905291fa6f7924dc3dd6458
Signed-off-by: Quentin Perret <quentin.perret@arm.com>
Quentin Perret
commited |
2e88529cf8 |