Pointer to BTF object is a pointer to kernel object or NULL.
Such pointers can only be used by BPF_LDX instructions.
The verifier changed their opcode from LDX|MEM|size
to LDX|PROBE_MEM|size to make JITing easier.
The number of entries in extable is the number of BPF_LDX insns
that access kernel memory via "pointer to BTF type".
Only these load instructions can fault.
Since x86 extable is relative it has to be allocated in the same
memory region as JITed code.
Allocate it prior to last pass of JITing and let the last pass populate it.
Pointer to extable in bpf_prog_aux is necessary to make page fault
handling fast.
Page fault handling is done in two steps:
1. bpf_prog_kallsyms_find() finds BPF program that page faulted.
It's done by walking rb tree.
2. then extable for given bpf program is binary searched.
This process is similar to how page faulting is done for kernel modules.
The exception handler skips over faulting x86 instruction and
initializes destination register with zero. This mimics exact
behavior of bpf_probe_read (when probe_kernel_read faults dest is zeroed).
JITs for other architectures can add support in similar way.
Until then they will reject unknown opcode and fallback to interpreter.
Since extable should be aligned and placed near JITed code
make bpf_jit_binary_alloc() return 4 byte aligned image offset,
so that extable aligning formula in bpf_int_jit_compile() doesn't need
to rely on internal implementation of bpf_jit_binary_alloc().
On x86 gcc defaults to 16-byte alignment for regular kernel functions
due to better performance. JITed code may be aligned to 16 in the future,
but it will use 4 in the meantime.
Change-Id: I7b16ea50910e1044fbb097301292947debe81ef6
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Andrii Nakryiko <andriin@fb.com>
Acked-by: Martin KaFai Lau <kafai@fb.com>
Link: https://lore.kernel.org/bpf/20191016032505.2089704-10-ast@kernel.org
Signed-off-by: Sorayukii <sorayukii69@gmail.com>
Alexei Starovoitovauthored and Sorayukiicommited 2026-05-15 17:58:58 +07:00
While not an issue now, eventually we will have independent users of
the extable.h file and we will stop sourcing it via module.h header.
In testing that pending work, with very sparse builds, characteristic
of an "allnoconfig" on various architectures, we can sometimes hit an
instance where the very basic standard definitions aren't present,
resulting in:
include/linux/extable.h:26:9: error: 'NULL' undeclared (first use in this function)
To be clear, this isn't a regression, since currently extable.h is
only used by module.h -- however, we will need this addition present
before we start migrating exception table users off module.h and onto
extable.h during the next release cycle.
Cc: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Paul Gortmakerauthored and Rusty Russellcommited 2016-08-04 10:16:56 +09:30
For historical reasons (i.e. pre-git) the exception table stuff was
buried in the middle of the module.h file. I noticed this while
doing an audit for needless includes of module.h and found core
kernel files (both arch specific and arch independent) were just
including module.h for this.
The converse is also true, in that conventional drivers, be they
for filesystems or actual hardware peripherals or similar, do not
normally care about the exception tables.
Here we fork the exception table content out of module.h into a
new file called extable.h -- and temporarily include it into the
module.h itself.
Then we will work our way across the arch independent and arch
specific files needing just exception table content, and move
them off module.h and onto extable.h
Once that is done, we can remove the extable.h from module.h
and in doing it like this, we avoid introducing build failures
into the git history.
The gain here is that module.h gets a bit smaller, across all
modular drivers that we build for allmodconfig. Also the core
files that only need exception table stuff don't have an include
of module.h that brings in lots of extra stuff and just looks
generally out of place.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Paul Gortmakerauthored and Rusty Russellcommited 2016-08-04 10:16:54 +09:30
Alexei Starovoitov
authored and
Sorayukii
commited 
Paul Gortmaker
authored and
Rusty Russell
commited