This is the 4.9.81 stable release
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEZH8oZUiU471FcZm+ONu9yGCSaT4FAlqCzaQACgkQONu9yGCS
aT62mRAAiN+5Q/aCQcyRCnUYiD7k0WQQvNYaqb1aAuiX+4kz4/NZ4BW6xI/BENYI
32Px98b98mKenpIDNg2JaAj5ZdoSRkls4lHvJ/KfPG5uTKoXwdBbDrR5K8ASZzfI
Z7PUSoCC6NsiKszSgexK8xvfiJ/tfhz1WqpA5X6a2UfgZqCfG/H6G9Dtatdjrgfr
+V0qE0XUM+eWVy+E0lBCccrM5s8ssr+ON4V6QOgmZh1WlaMuCpouOvTYBW/6PBKu
aYjQKvJuXlxTa7IoqvHTd5WwcM+4wYwG6S+/8f3Xprgn8K3juYZ0EmACgSXm2AuR
vu089LyRDyLsG0J7eRoMZU753B/uV51vH23wfaitfbHxzFsFQihOuivhTttww10u
7v7RNcfEQBuAjD9KfZCf4/iSs2QIDKKpLQVy4hUuGv0gcj31WjOzU3uKlvvJUvN+
HsFRXrE5RvS4TaRkiI3fnV3NAFj/l9fkW+V2YV5nWg3C8zBbiGVOBNODgUKLMLvE
mJTQ42oVqhQfuzFDCSE3jPQMe4VL3OMQ1MC9Yc99/UvtYHZeV5PpgCC0kcMGh+r8
Gf9n0hBuFwr38djK/bewrjcIims3lgRbHdlzqtCHFRmaH5bJMkUWD8WakjbjRI89
SYKc6TuVzvzRoaMjNj14dHQSdiTz0Fod+U82KMccZ4F7SflhdQo=
=nbsq
-----END PGP SIGNATURE-----
Merge 4.9.81 into android-4.9-o
Changes in 4.9.81
powerpc/pseries: Add H_GET_CPU_CHARACTERISTICS flags & wrapper
powerpc/64: Add macros for annotating the destination of rfid/hrfid
powerpc/64: Convert fast_exception_return to use RFI_TO_USER/KERNEL
powerpc/64: Convert the syscall exit path to use RFI_TO_USER/KERNEL
powerpc/64s: Convert slb_miss_common to use RFI_TO_USER/KERNEL
powerpc/64s: Add support for RFI flush of L1-D cache
powerpc/64s: Support disabling RFI flush with no_rfi_flush and nopti
powerpc/pseries: Query hypervisor for RFI flush settings
powerpc/powernv: Check device-tree for RFI flush settings
powerpc/64s: Wire up cpu_show_meltdown()
powerpc/64s: Allow control of RFI flush via debugfs
auxdisplay: img-ascii-lcd: add missing MODULE_DESCRIPTION/AUTHOR/LICENSE
pinctrl: pxa: pxa2xx: add missing MODULE_DESCRIPTION/AUTHOR/LICENSE
ASoC: pcm512x: add missing MODULE_DESCRIPTION/AUTHOR/LICENSE
kaiser: fix intel_bts perf crashes
x86/pti: Make unpoison of pgd for trusted boot work for real
kaiser: allocate pgd with order 0 when pti=off
serial: core: mark port as initialized after successful IRQ change
ip6mr: fix stale iterator
net: igmp: add a missing rcu locking section
qlcnic: fix deadlock bug
qmi_wwan: Add support for Quectel EP06
r8169: fix RTL8168EP take too long to complete driver initialization.
tcp: release sk_frag.page in tcp_disconnect
vhost_net: stop device during reset owner
tcp_bbr: fix pacing_gain to always be unity when using lt_bw
cls_u32: add missing RCU annotation.
ipv6: Fix SO_REUSEPORT UDP socket with implicit sk_ipv6only
soreuseport: fix mem leak in reuseport_add_sock()
x86/asm: Fix inline asm call constraints for GCC 4.4
x86/microcode/AMD: Do not load when running on a hypervisor
media: soc_camera: soc_scale_crop: add missing MODULE_DESCRIPTION/AUTHOR/LICENSE
b43: Add missing MODULE_FIRMWARE()
KEYS: encrypted: fix buffer overread in valid_master_desc()
x86/retpoline: Remove the esp/rsp thunk
KVM: x86: Make indirect calls in emulator speculation safe
KVM: VMX: Make indirect call speculation safe
module/retpoline: Warn about missing retpoline in module
x86/cpufeatures: Add CPUID_7_EDX CPUID leaf
x86/cpufeatures: Add Intel feature bits for Speculation Control
x86/cpufeatures: Add AMD feature bits for Speculation Control
x86/msr: Add definitions for new speculation control MSRs
x86/pti: Do not enable PTI on CPUs which are not vulnerable to Meltdown
x86/cpufeature: Blacklist SPEC_CTRL/PRED_CMD on early Spectre v2 microcodes
x86/speculation: Add basic IBPB (Indirect Branch Prediction Barrier) support
x86/nospec: Fix header guards names
x86/bugs: Drop one "mitigation" from dmesg
x86/cpu/bugs: Make retpoline module warning conditional
x86/cpufeatures: Clean up Spectre v2 related CPUID flags
x86/retpoline: Simplify vmexit_fill_RSB()
x86/spectre: Check CONFIG_RETPOLINE in command line parser
x86/entry/64: Remove the SYSCALL64 fast path
x86/entry/64: Push extra regs right away
x86/asm: Move 'status' from thread_struct to thread_info
Documentation: Document array_index_nospec
array_index_nospec: Sanitize speculative array de-references
x86: Implement array_index_mask_nospec
x86: Introduce barrier_nospec
x86: Introduce __uaccess_begin_nospec() and uaccess_try_nospec
x86/usercopy: Replace open coded stac/clac with __uaccess_{begin, end}
x86/uaccess: Use __uaccess_begin_nospec() and uaccess_try_nospec
x86/get_user: Use pointer masking to limit speculation
x86/syscall: Sanitize syscall table de-references under speculation
vfs, fdtable: Prevent bounds-check bypass via speculative execution
nl80211: Sanitize array index in parse_txq_params
x86/spectre: Report get_user mitigation for spectre_v1
x86/spectre: Fix spelling mistake: "vunerable"-> "vulnerable"
x86/cpuid: Fix up "virtual" IBRS/IBPB/STIBP feature bits on Intel
x86/paravirt: Remove 'noreplace-paravirt' cmdline option
x86/kvm: Update spectre-v1 mitigation
x86/retpoline: Avoid retpolines for built-in __init functions
x86/spectre: Simplify spectre_v2 command line parsing
x86/pti: Mark constant arrays as __initconst
x86/speculation: Fix typo IBRS_ATT, which should be IBRS_ALL
KVM: nVMX: kmap() can't fail
KVM: nVMX: vmx_complete_nested_posted_interrupt() can't fail
KVM: nVMX: mark vmcs12 pages dirty on L2 exit
KVM: nVMX: Eliminate vmcs02 pool
KVM: VMX: introduce alloc_loaded_vmcs
KVM: VMX: make MSR bitmaps per-VCPU
KVM/x86: Add IBPB support
KVM/VMX: Emulate MSR_IA32_ARCH_CAPABILITIES
KVM/VMX: Allow direct access to MSR_IA32_SPEC_CTRL
KVM/SVM: Allow direct access to MSR_IA32_SPEC_CTRL
crypto: tcrypt - fix S/G table for test_aead_speed()
ASoC: simple-card: Fix misleading error message
ASoC: rsnd: don't call free_irq() on Parent SSI
ASoC: rsnd: avoid duplicate free_irq()
drm: rcar-du: Use the VBK interrupt for vblank events
drm: rcar-du: Fix race condition when disabling planes at CRTC stop
x86/microcode: Do the family check first
Linux 4.9.81
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
This commit is contained in:
Greg Kroah-Hartman
commit
1b9d719b0b
93 changed files with 2026 additions and 789 deletions
|
|
@ -2811,8 +2811,6 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
|
|||
norandmaps Don't use address space randomization. Equivalent to
|
||||
echo 0 > /proc/sys/kernel/randomize_va_space
|
||||
|
||||
noreplace-paravirt [X86,IA-64,PV_OPS] Don't patch paravirt_ops
|
||||
|
||||
noreplace-smp [X86-32,SMP] Don't replace SMP instructions
|
||||
with UP alternatives
|
||||
|
||||
|
|
|
|||
90
Documentation/speculation.txt
Normal file
90
Documentation/speculation.txt
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
This document explains potential effects of speculation, and how undesirable
|
||||
effects can be mitigated portably using common APIs.
|
||||
|
||||
===========
|
||||
Speculation
|
||||
===========
|
||||
|
||||
To improve performance and minimize average latencies, many contemporary CPUs
|
||||
employ speculative execution techniques such as branch prediction, performing
|
||||
work which may be discarded at a later stage.
|
||||
|
||||
Typically speculative execution cannot be observed from architectural state,
|
||||
such as the contents of registers. However, in some cases it is possible to
|
||||
observe its impact on microarchitectural state, such as the presence or
|
||||
absence of data in caches. Such state may form side-channels which can be
|
||||
observed to extract secret information.
|
||||
|
||||
For example, in the presence of branch prediction, it is possible for bounds
|
||||
checks to be ignored by code which is speculatively executed. Consider the
|
||||
following code:
|
||||
|
||||
int load_array(int *array, unsigned int index)
|
||||
{
|
||||
if (index >= MAX_ARRAY_ELEMS)
|
||||
return 0;
|
||||
else
|
||||
return array[index];
|
||||
}
|
||||
|
||||
Which, on arm64, may be compiled to an assembly sequence such as:
|
||||
|
||||
CMP <index>, #MAX_ARRAY_ELEMS
|
||||
B.LT less
|
||||
MOV <returnval>, #0
|
||||
RET
|
||||
less:
|
||||
LDR <returnval>, [<array>, <index>]
|
||||
RET
|
||||
|
||||
It is possible that a CPU mis-predicts the conditional branch, and
|
||||
speculatively loads array[index], even if index >= MAX_ARRAY_ELEMS. This
|
||||
value will subsequently be discarded, but the speculated load may affect
|
||||
microarchitectural state which can be subsequently measured.
|
||||
|
||||
More complex sequences involving multiple dependent memory accesses may
|
||||
result in sensitive information being leaked. Consider the following
|
||||
code, building on the prior example:
|
||||
|
||||
int load_dependent_arrays(int *arr1, int *arr2, int index)
|
||||
{
|
||||
int val1, val2,
|
||||
|
||||
val1 = load_array(arr1, index);
|
||||
val2 = load_array(arr2, val1);
|
||||
|
||||
return val2;
|
||||
}
|
||||
|
||||
Under speculation, the first call to load_array() may return the value
|
||||
of an out-of-bounds address, while the second call will influence
|
||||
microarchitectural state dependent on this value. This may provide an
|
||||
arbitrary read primitive.
|
||||
|
||||
====================================
|
||||
Mitigating speculation side-channels
|
||||
====================================
|
||||
|
||||
The kernel provides a generic API to ensure that bounds checks are
|
||||
respected even under speculation. Architectures which are affected by
|
||||
speculation-based side-channels are expected to implement these
|
||||
primitives.
|
||||
|
||||
The array_index_nospec() helper in <linux/nospec.h> can be used to
|
||||
prevent information from being leaked via side-channels.
|
||||
|
||||
A call to array_index_nospec(index, size) returns a sanitized index
|
||||
value that is bounded to [0, size) even under cpu speculation
|
||||
conditions.
|
||||
|
||||
This can be used to protect the earlier load_array() example:
|
||||
|
||||
int load_array(int *array, unsigned int index)
|
||||
{
|
||||
if (index >= MAX_ARRAY_ELEMS)
|
||||
return 0;
|
||||
else {
|
||||
index = array_index_nospec(index, MAX_ARRAY_ELEMS);
|
||||
return array[index];
|
||||
}
|
||||
}
|
||||
2
Makefile
2
Makefile
|
|
@ -1,6 +1,6 @@
|
|||
VERSION = 4
|
||||
PATCHLEVEL = 9
|
||||
SUBLEVEL = 80
|
||||
SUBLEVEL = 81
|
||||
EXTRAVERSION =
|
||||
NAME = Roaring Lionus
|
||||
|
||||
|
|
|
|||
|
|
@ -128,6 +128,7 @@ config PPC
|
|||
select ARCH_HAS_GCOV_PROFILE_ALL
|
||||
select GENERIC_SMP_IDLE_THREAD
|
||||
select GENERIC_CMOS_UPDATE
|
||||
select GENERIC_CPU_VULNERABILITIES if PPC_BOOK3S_64
|
||||
select GENERIC_TIME_VSYSCALL_OLD
|
||||
select GENERIC_CLOCKEVENTS
|
||||
select GENERIC_CLOCKEVENTS_BROADCAST if SMP
|
||||
|
|
|
|||
|
|
@ -209,5 +209,11 @@ exc_##label##_book3e:
|
|||
ori r3,r3,vector_offset@l; \
|
||||
mtspr SPRN_IVOR##vector_number,r3;
|
||||
|
||||
#define RFI_TO_KERNEL \
|
||||
rfi
|
||||
|
||||
#define RFI_TO_USER \
|
||||
rfi
|
||||
|
||||
#endif /* _ASM_POWERPC_EXCEPTION_64E_H */
|
||||
|
||||
|
|
|
|||
|
|
@ -51,6 +51,59 @@
|
|||
#define EX_PPR 88 /* SMT thread status register (priority) */
|
||||
#define EX_CTR 96
|
||||
|
||||
/*
|
||||
* Macros for annotating the expected destination of (h)rfid
|
||||
*
|
||||
* The nop instructions allow us to insert one or more instructions to flush the
|
||||
* L1-D cache when returning to userspace or a guest.
|
||||
*/
|
||||
#define RFI_FLUSH_SLOT \
|
||||
RFI_FLUSH_FIXUP_SECTION; \
|
||||
nop; \
|
||||
nop; \
|
||||
nop
|
||||
|
||||
#define RFI_TO_KERNEL \
|
||||
rfid
|
||||
|
||||
#define RFI_TO_USER \
|
||||
RFI_FLUSH_SLOT; \
|
||||
rfid; \
|
||||
b rfi_flush_fallback
|
||||
|
||||
#define RFI_TO_USER_OR_KERNEL \
|
||||
RFI_FLUSH_SLOT; \
|
||||
rfid; \
|
||||
b rfi_flush_fallback
|
||||
|
||||
#define RFI_TO_GUEST \
|
||||
RFI_FLUSH_SLOT; \
|
||||
rfid; \
|
||||
b rfi_flush_fallback
|
||||
|
||||
#define HRFI_TO_KERNEL \
|
||||
hrfid
|
||||
|
||||
#define HRFI_TO_USER \
|
||||
RFI_FLUSH_SLOT; \
|
||||
hrfid; \
|
||||
b hrfi_flush_fallback
|
||||
|
||||
#define HRFI_TO_USER_OR_KERNEL \
|
||||
RFI_FLUSH_SLOT; \
|
||||
hrfid; \
|
||||
b hrfi_flush_fallback
|
||||
|
||||
#define HRFI_TO_GUEST \
|
||||
RFI_FLUSH_SLOT; \
|
||||
hrfid; \
|
||||
b hrfi_flush_fallback
|
||||
|
||||
#define HRFI_TO_UNKNOWN \
|
||||
RFI_FLUSH_SLOT; \
|
||||
hrfid; \
|
||||
b hrfi_flush_fallback
|
||||
|
||||
#ifdef CONFIG_RELOCATABLE
|
||||
#define __EXCEPTION_RELON_PROLOG_PSERIES_1(label, h) \
|
||||
mfspr r11,SPRN_##h##SRR0; /* save SRR0 */ \
|
||||
|
|
|
|||
|
|
@ -189,4 +189,19 @@ void apply_feature_fixups(void);
|
|||
void setup_feature_keys(void);
|
||||
#endif
|
||||
|
||||
#define RFI_FLUSH_FIXUP_SECTION \
|
||||
951: \
|
||||
.pushsection __rfi_flush_fixup,"a"; \
|
||||
.align 2; \
|
||||
952: \
|
||||
FTR_ENTRY_OFFSET 951b-952b; \
|
||||
.popsection;
|
||||
|
||||
|
||||
#ifndef __ASSEMBLY__
|
||||
|
||||
extern long __start___rfi_flush_fixup, __stop___rfi_flush_fixup;
|
||||
|
||||
#endif
|
||||
|
||||
#endif /* __ASM_POWERPC_FEATURE_FIXUPS_H */
|
||||
|
|
|
|||
|
|
@ -240,6 +240,7 @@
|
|||
#define H_GET_HCA_INFO 0x1B8
|
||||
#define H_GET_PERF_COUNT 0x1BC
|
||||
#define H_MANAGE_TRACE 0x1C0
|
||||
#define H_GET_CPU_CHARACTERISTICS 0x1C8
|
||||
#define H_FREE_LOGICAL_LAN_BUFFER 0x1D4
|
||||
#define H_QUERY_INT_STATE 0x1E4
|
||||
#define H_POLL_PENDING 0x1D8
|
||||
|
|
@ -306,6 +307,17 @@
|
|||
#define H_SET_MODE_RESOURCE_ADDR_TRANS_MODE 3
|
||||
#define H_SET_MODE_RESOURCE_LE 4
|
||||
|
||||
/* H_GET_CPU_CHARACTERISTICS return values */
|
||||
#define H_CPU_CHAR_SPEC_BAR_ORI31 (1ull << 63) // IBM bit 0
|
||||
#define H_CPU_CHAR_BCCTRL_SERIALISED (1ull << 62) // IBM bit 1
|
||||
#define H_CPU_CHAR_L1D_FLUSH_ORI30 (1ull << 61) // IBM bit 2
|
||||
#define H_CPU_CHAR_L1D_FLUSH_TRIG2 (1ull << 60) // IBM bit 3
|
||||
#define H_CPU_CHAR_L1D_THREAD_PRIV (1ull << 59) // IBM bit 4
|
||||
|
||||
#define H_CPU_BEHAV_FAVOUR_SECURITY (1ull << 63) // IBM bit 0
|
||||
#define H_CPU_BEHAV_L1D_FLUSH_PR (1ull << 62) // IBM bit 1
|
||||
#define H_CPU_BEHAV_BNDS_CHK_SPEC_BAR (1ull << 61) // IBM bit 2
|
||||
|
||||
#ifndef __ASSEMBLY__
|
||||
|
||||
/**
|
||||
|
|
@ -433,6 +445,11 @@ static inline unsigned long cmo_get_page_size(void)
|
|||
}
|
||||
#endif /* CONFIG_PPC_PSERIES */
|
||||
|
||||
struct h_cpu_char_result {
|
||||
u64 character;
|
||||
u64 behaviour;
|
||||
};
|
||||
|
||||
#endif /* __ASSEMBLY__ */
|
||||
#endif /* __KERNEL__ */
|
||||
#endif /* _ASM_POWERPC_HVCALL_H */
|
||||
|
|
|
|||
|
|
@ -205,6 +205,16 @@ struct paca_struct {
|
|||
struct sibling_subcore_state *sibling_subcore_state;
|
||||
#endif
|
||||
#endif
|
||||
#ifdef CONFIG_PPC_BOOK3S_64
|
||||
/*
|
||||
* rfi fallback flush must be in its own cacheline to prevent
|
||||
* other paca data leaking into the L1d
|
||||
*/
|
||||
u64 exrfi[13] __aligned(0x80);
|
||||
void *rfi_flush_fallback_area;
|
||||
u64 l1d_flush_congruence;
|
||||
u64 l1d_flush_sets;
|
||||
#endif
|
||||
};
|
||||
|
||||
#ifdef CONFIG_PPC_BOOK3S
|
||||
|
|
|
|||
|
|
@ -340,4 +340,18 @@ static inline long plapr_set_watchpoint0(unsigned long dawr0, unsigned long dawr
|
|||
return plpar_set_mode(0, H_SET_MODE_RESOURCE_SET_DAWR, dawr0, dawrx0);
|
||||
}
|
||||
|
||||
static inline long plpar_get_cpu_characteristics(struct h_cpu_char_result *p)
|
||||
{
|
||||
unsigned long retbuf[PLPAR_HCALL_BUFSIZE];
|
||||
long rc;
|
||||
|
||||
rc = plpar_hcall(H_GET_CPU_CHARACTERISTICS, retbuf);
|
||||
if (rc == H_SUCCESS) {
|
||||
p->character = retbuf[0];
|
||||
p->behaviour = retbuf[1];
|
||||
}
|
||||
|
||||
return rc;
|
||||
}
|
||||
|
||||
#endif /* _ASM_POWERPC_PLPAR_WRAPPERS_H */
|
||||
|
|
|
|||
|
|
@ -38,6 +38,19 @@ static inline void pseries_big_endian_exceptions(void) {}
|
|||
static inline void pseries_little_endian_exceptions(void) {}
|
||||
#endif /* CONFIG_PPC_PSERIES */
|
||||
|
||||
void rfi_flush_enable(bool enable);
|
||||
|
||||
/* These are bit flags */
|
||||
enum l1d_flush_type {
|
||||
L1D_FLUSH_NONE = 0x1,
|
||||
L1D_FLUSH_FALLBACK = 0x2,
|
||||
L1D_FLUSH_ORI = 0x4,
|
||||
L1D_FLUSH_MTTRIG = 0x8,
|
||||
};
|
||||
|
||||
void __init setup_rfi_flush(enum l1d_flush_type, bool enable);
|
||||
void do_rfi_flush_fixups(enum l1d_flush_type types);
|
||||
|
||||
#endif /* !__ASSEMBLY__ */
|
||||
|
||||
#endif /* _ASM_POWERPC_SETUP_H */
|
||||
|
|
|
|||
|
|
@ -240,6 +240,10 @@ int main(void)
|
|||
#ifdef CONFIG_PPC_BOOK3S_64
|
||||
DEFINE(PACAMCEMERGSP, offsetof(struct paca_struct, mc_emergency_sp));
|
||||
DEFINE(PACA_IN_MCE, offsetof(struct paca_struct, in_mce));
|
||||
DEFINE(PACA_RFI_FLUSH_FALLBACK_AREA, offsetof(struct paca_struct, rfi_flush_fallback_area));
|
||||
DEFINE(PACA_EXRFI, offsetof(struct paca_struct, exrfi));
|
||||
DEFINE(PACA_L1D_FLUSH_CONGRUENCE, offsetof(struct paca_struct, l1d_flush_congruence));
|
||||
DEFINE(PACA_L1D_FLUSH_SETS, offsetof(struct paca_struct, l1d_flush_sets));
|
||||
#endif
|
||||
DEFINE(PACAHWCPUID, offsetof(struct paca_struct, hw_cpu_id));
|
||||
DEFINE(PACAKEXECSTATE, offsetof(struct paca_struct, kexec_state));
|
||||
|
|
|
|||
|
|
@ -251,13 +251,23 @@ BEGIN_FTR_SECTION
|
|||
END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
|
||||
|
||||
ld r13,GPR13(r1) /* only restore r13 if returning to usermode */
|
||||
ld r2,GPR2(r1)
|
||||
ld r1,GPR1(r1)
|
||||
mtlr r4
|
||||
mtcr r5
|
||||
mtspr SPRN_SRR0,r7
|
||||
mtspr SPRN_SRR1,r8
|
||||
RFI_TO_USER
|
||||
b . /* prevent speculative execution */
|
||||
|
||||
/* exit to kernel */
|
||||
1: ld r2,GPR2(r1)
|
||||
ld r1,GPR1(r1)
|
||||
mtlr r4
|
||||
mtcr r5
|
||||
mtspr SPRN_SRR0,r7
|
||||
mtspr SPRN_SRR1,r8
|
||||
RFI
|
||||
RFI_TO_KERNEL
|
||||
b . /* prevent speculative execution */
|
||||
|
||||
syscall_error:
|
||||
|
|
@ -859,7 +869,7 @@ BEGIN_FTR_SECTION
|
|||
END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
|
||||
ACCOUNT_CPU_USER_EXIT(r13, r2, r4)
|
||||
REST_GPR(13, r1)
|
||||
1:
|
||||
|
||||
mtspr SPRN_SRR1,r3
|
||||
|
||||
ld r2,_CCR(r1)
|
||||
|
|
@ -872,8 +882,22 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
|
|||
ld r3,GPR3(r1)
|
||||
ld r4,GPR4(r1)
|
||||
ld r1,GPR1(r1)
|
||||
RFI_TO_USER
|
||||
b . /* prevent speculative execution */
|
||||
|
||||
rfid
|
||||
1: mtspr SPRN_SRR1,r3
|
||||
|
||||
ld r2,_CCR(r1)
|
||||
mtcrf 0xFF,r2
|
||||
ld r2,_NIP(r1)
|
||||
mtspr SPRN_SRR0,r2
|
||||
|
||||
ld r0,GPR0(r1)
|
||||
ld r2,GPR2(r1)
|
||||
ld r3,GPR3(r1)
|
||||
ld r4,GPR4(r1)
|
||||
ld r1,GPR1(r1)
|
||||
RFI_TO_KERNEL
|
||||
b . /* prevent speculative execution */
|
||||
|
||||
#endif /* CONFIG_PPC_BOOK3E */
|
||||
|
|
|
|||
|
|
@ -655,6 +655,8 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX)
|
|||
|
||||
andi. r10,r12,MSR_RI /* check for unrecoverable exception */
|
||||
beq- 2f
|
||||
andi. r10,r12,MSR_PR /* check for user mode (PR != 0) */
|
||||
bne 1f
|
||||
|
||||
/* All done -- return from exception. */
|
||||
|
||||
|
|
@ -671,7 +673,23 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX)
|
|||
ld r11,PACA_EXSLB+EX_R11(r13)
|
||||
ld r12,PACA_EXSLB+EX_R12(r13)
|
||||
ld r13,PACA_EXSLB+EX_R13(r13)
|
||||
rfid
|
||||
RFI_TO_KERNEL
|
||||
b . /* prevent speculative execution */
|
||||
|
||||
1:
|
||||
.machine push
|
||||
.machine "power4"
|
||||
mtcrf 0x80,r9
|
||||
mtcrf 0x01,r9 /* slb_allocate uses cr0 and cr7 */
|
||||
.machine pop
|
||||
|
||||
RESTORE_PPR_PACA(PACA_EXSLB, r9)
|
||||
ld r9,PACA_EXSLB+EX_R9(r13)
|
||||
ld r10,PACA_EXSLB+EX_R10(r13)
|
||||
ld r11,PACA_EXSLB+EX_R11(r13)
|
||||
ld r12,PACA_EXSLB+EX_R12(r13)
|
||||
ld r13,PACA_EXSLB+EX_R13(r13)
|
||||
RFI_TO_USER
|
||||
b . /* prevent speculative execution */
|
||||
|
||||
2: mfspr r11,SPRN_SRR0
|
||||
|
|
@ -679,7 +697,7 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX)
|
|||
mtspr SPRN_SRR0,r10
|
||||
ld r10,PACAKMSR(r13)
|
||||
mtspr SPRN_SRR1,r10
|
||||
rfid
|
||||
RFI_TO_KERNEL
|
||||
b .
|
||||
|
||||
8: mfspr r11,SPRN_SRR0
|
||||
|
|
@ -1576,6 +1594,92 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
|
|||
bl kernel_bad_stack
|
||||
b 1b
|
||||
|
||||
.globl rfi_flush_fallback
|
||||
rfi_flush_fallback:
|
||||
SET_SCRATCH0(r13);
|
||||
GET_PACA(r13);
|
||||
std r9,PACA_EXRFI+EX_R9(r13)
|
||||
std r10,PACA_EXRFI+EX_R10(r13)
|
||||
std r11,PACA_EXRFI+EX_R11(r13)
|
||||
std r12,PACA_EXRFI+EX_R12(r13)
|
||||
std r8,PACA_EXRFI+EX_R13(r13)
|
||||
mfctr r9
|
||||
ld r10,PACA_RFI_FLUSH_FALLBACK_AREA(r13)
|
||||
ld r11,PACA_L1D_FLUSH_SETS(r13)
|
||||
ld r12,PACA_L1D_FLUSH_CONGRUENCE(r13)
|
||||
/*
|
||||
* The load adresses are at staggered offsets within cachelines,
|
||||
* which suits some pipelines better (on others it should not
|
||||
* hurt).
|
||||
*/
|
||||
addi r12,r12,8
|
||||
mtctr r11
|
||||
DCBT_STOP_ALL_STREAM_IDS(r11) /* Stop prefetch streams */
|
||||
|
||||
/* order ld/st prior to dcbt stop all streams with flushing */
|
||||
sync
|
||||
1: li r8,0
|
||||
.rept 8 /* 8-way set associative */
|
||||
ldx r11,r10,r8
|
||||
add r8,r8,r12
|
||||
xor r11,r11,r11 // Ensure r11 is 0 even if fallback area is not
|
||||
add r8,r8,r11 // Add 0, this creates a dependency on the ldx
|
||||
.endr
|
||||
addi r10,r10,128 /* 128 byte cache line */
|
||||
bdnz 1b
|
||||
|
||||
mtctr r9
|
||||
ld r9,PACA_EXRFI+EX_R9(r13)
|
||||
ld r10,PACA_EXRFI+EX_R10(r13)
|
||||
ld r11,PACA_EXRFI+EX_R11(r13)
|
||||
ld r12,PACA_EXRFI+EX_R12(r13)
|
||||
ld r8,PACA_EXRFI+EX_R13(r13)
|
||||
GET_SCRATCH0(r13);
|
||||
rfid
|
||||
|
||||
.globl hrfi_flush_fallback
|
||||
hrfi_flush_fallback:
|
||||
SET_SCRATCH0(r13);
|
||||
GET_PACA(r13);
|
||||
std r9,PACA_EXRFI+EX_R9(r13)
|
||||
std r10,PACA_EXRFI+EX_R10(r13)
|
||||
std r11,PACA_EXRFI+EX_R11(r13)
|
||||
std r12,PACA_EXRFI+EX_R12(r13)
|
||||
std r8,PACA_EXRFI+EX_R13(r13)
|
||||
mfctr r9
|
||||
ld r10,PACA_RFI_FLUSH_FALLBACK_AREA(r13)
|
||||
ld r11,PACA_L1D_FLUSH_SETS(r13)
|
||||
ld r12,PACA_L1D_FLUSH_CONGRUENCE(r13)
|
||||
/*
|
||||
* The load adresses are at staggered offsets within cachelines,
|
||||
* which suits some pipelines better (on others it should not
|
||||
* hurt).
|
||||
*/
|
||||
addi r12,r12,8
|
||||
mtctr r11
|
||||
DCBT_STOP_ALL_STREAM_IDS(r11) /* Stop prefetch streams */
|
||||
|
||||
/* order ld/st prior to dcbt stop all streams with flushing */
|
||||
sync
|
||||
1: li r8,0
|
||||
.rept 8 /* 8-way set associative */
|
||||
ldx r11,r10,r8
|
||||
add r8,r8,r12
|
||||
xor r11,r11,r11 // Ensure r11 is 0 even if fallback area is not
|
||||
add r8,r8,r11 // Add 0, this creates a dependency on the ldx
|
||||
.endr
|
||||
addi r10,r10,128 /* 128 byte cache line */
|
||||
bdnz 1b
|
||||
|
||||
mtctr r9
|
||||
ld r9,PACA_EXRFI+EX_R9(r13)
|
||||
ld r10,PACA_EXRFI+EX_R10(r13)
|
||||
ld r11,PACA_EXRFI+EX_R11(r13)
|
||||
ld r12,PACA_EXRFI+EX_R12(r13)
|
||||
ld r8,PACA_EXRFI+EX_R13(r13)
|
||||
GET_SCRATCH0(r13);
|
||||
hrfid
|
||||
|
||||
/*
|
||||
* Called from arch_local_irq_enable when an interrupt needs
|
||||
* to be resent. r3 contains 0x500, 0x900, 0xa00 or 0xe80 to indicate
|
||||
|
|
|
|||
|
|
@ -37,6 +37,7 @@
|
|||
#include <linux/memblock.h>
|
||||
#include <linux/memory.h>
|
||||
#include <linux/nmi.h>
|
||||
#include <linux/debugfs.h>
|
||||
|
||||
#include <asm/io.h>
|
||||
#include <asm/kdump.h>
|
||||
|
|
@ -678,4 +679,142 @@ static int __init disable_hardlockup_detector(void)
|
|||
return 0;
|
||||
}
|
||||
early_initcall(disable_hardlockup_detector);
|
||||
|
||||
#ifdef CONFIG_PPC_BOOK3S_64
|
||||
static enum l1d_flush_type enabled_flush_types;
|
||||
static void *l1d_flush_fallback_area;
|
||||
static bool no_rfi_flush;
|
||||
bool rfi_flush;
|
||||
|
||||
static int __init handle_no_rfi_flush(char *p)
|
||||
{
|
||||
pr_info("rfi-flush: disabled on command line.");
|
||||
no_rfi_flush = true;
|
||||
return 0;
|
||||
}
|
||||
early_param("no_rfi_flush", handle_no_rfi_flush);
|
||||
|
||||
/*
|
||||
* The RFI flush is not KPTI, but because users will see doco that says to use
|
||||
* nopti we hijack that option here to also disable the RFI flush.
|
||||
*/
|
||||
static int __init handle_no_pti(char *p)
|
||||
{
|
||||
pr_info("rfi-flush: disabling due to 'nopti' on command line.\n");
|
||||
handle_no_rfi_flush(NULL);
|
||||
return 0;
|
||||
}
|
||||
early_param("nopti", handle_no_pti);
|
||||
|
||||
static void do_nothing(void *unused)
|
||||
{
|
||||
/*
|
||||
* We don't need to do the flush explicitly, just enter+exit kernel is
|
||||
* sufficient, the RFI exit handlers will do the right thing.
|
||||
*/
|
||||
}
|
||||
|
||||
void rfi_flush_enable(bool enable)
|
||||
{
|
||||
if (rfi_flush == enable)
|
||||
return;
|
||||
|
||||
if (enable) {
|
||||
do_rfi_flush_fixups(enabled_flush_types);
|
||||
on_each_cpu(do_nothing, NULL, 1);
|
||||
} else
|
||||
do_rfi_flush_fixups(L1D_FLUSH_NONE);
|
||||
|
||||
rfi_flush = enable;
|
||||
}
|
||||
|
||||
static void init_fallback_flush(void)
|
||||
{
|
||||
u64 l1d_size, limit;
|
||||
int cpu;
|
||||
|
||||
l1d_size = ppc64_caches.dsize;
|
||||
limit = min(safe_stack_limit(), ppc64_rma_size);
|
||||
|
||||
/*
|
||||
* Align to L1d size, and size it at 2x L1d size, to catch possible
|
||||
* hardware prefetch runoff. We don't have a recipe for load patterns to
|
||||
* reliably avoid the prefetcher.
|
||||
*/
|
||||
l1d_flush_fallback_area = __va(memblock_alloc_base(l1d_size * 2, l1d_size, limit));
|
||||
memset(l1d_flush_fallback_area, 0, l1d_size * 2);
|
||||
|
||||
for_each_possible_cpu(cpu) {
|
||||
/*
|
||||
* The fallback flush is currently coded for 8-way
|
||||
* associativity. Different associativity is possible, but it
|
||||
* will be treated as 8-way and may not evict the lines as
|
||||
* effectively.
|
||||
*
|
||||
* 128 byte lines are mandatory.
|
||||
*/
|
||||
u64 c = l1d_size / 8;
|
||||
|
||||
paca[cpu].rfi_flush_fallback_area = l1d_flush_fallback_area;
|
||||
paca[cpu].l1d_flush_congruence = c;
|
||||
paca[cpu].l1d_flush_sets = c / 128;
|
||||
}
|
||||
}
|
||||
|
||||
void __init setup_rfi_flush(enum l1d_flush_type types, bool enable)
|
||||
{
|
||||
if (types & L1D_FLUSH_FALLBACK) {
|
||||
pr_info("rfi-flush: Using fallback displacement flush\n");
|
||||
init_fallback_flush();
|
||||
}
|
||||
|
||||
if (types & L1D_FLUSH_ORI)
|
||||
pr_info("rfi-flush: Using ori type flush\n");
|
||||
|
||||
if (types & L1D_FLUSH_MTTRIG)
|
||||
pr_info("rfi-flush: Using mttrig type flush\n");
|
||||
|
||||
enabled_flush_types = types;
|
||||
|
||||
if (!no_rfi_flush)
|
||||
rfi_flush_enable(enable);
|
||||
}
|
||||
|
||||
#ifdef CONFIG_DEBUG_FS
|
||||
static int rfi_flush_set(void *data, u64 val)
|
||||
{
|
||||
if (val == 1)
|
||||
rfi_flush_enable(true);
|
||||
else if (val == 0)
|
||||
rfi_flush_enable(false);
|
||||
else
|
||||
return -EINVAL;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int rfi_flush_get(void *data, u64 *val)
|
||||
{
|
||||
*val = rfi_flush ? 1 : 0;
|
||||
return 0;
|
||||
}
|
||||
|
||||
DEFINE_SIMPLE_ATTRIBUTE(fops_rfi_flush, rfi_flush_get, rfi_flush_set, "%llu\n");
|
||||
|
||||
static __init int rfi_flush_debugfs_init(void)
|
||||
{
|
||||
debugfs_create_file("rfi_flush", 0600, powerpc_debugfs_root, NULL, &fops_rfi_flush);
|
||||
return 0;
|
||||
}
|
||||
device_initcall(rfi_flush_debugfs_init);
|
||||
#endif
|
||||
|
||||
ssize_t cpu_show_meltdown(struct device *dev, struct device_attribute *attr, char *buf)
|
||||
{
|
||||
if (rfi_flush)
|
||||
return sprintf(buf, "Mitigation: RFI Flush\n");
|
||||
|
||||
return sprintf(buf, "Vulnerable\n");
|
||||
}
|
||||
#endif /* CONFIG_PPC_BOOK3S_64 */
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -132,6 +132,15 @@ SECTIONS
|
|||
/* Read-only data */
|
||||
RODATA
|
||||
|
||||
#ifdef CONFIG_PPC64
|
||||
. = ALIGN(8);
|
||||
__rfi_flush_fixup : AT(ADDR(__rfi_flush_fixup) - LOAD_OFFSET) {
|
||||
__start___rfi_flush_fixup = .;
|
||||
*(__rfi_flush_fixup)
|
||||
__stop___rfi_flush_fixup = .;
|
||||
}
|
||||
#endif
|
||||
|
||||
EXCEPTION_TABLE(0)
|
||||
|
||||
NOTES :kernel :notes
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@
|
|||
#include <asm/sections.h>
|
||||
#include <asm/setup.h>
|
||||
#include <asm/firmware.h>
|
||||
#include <asm/setup.h>
|
||||
|
||||
struct fixup_entry {
|
||||
unsigned long mask;
|
||||
|
|
@ -115,6 +116,47 @@ void do_feature_fixups(unsigned long value, void *fixup_start, void *fixup_end)
|
|||
}
|
||||
}
|
||||
|
||||
#ifdef CONFIG_PPC_BOOK3S_64
|
||||
void do_rfi_flush_fixups(enum l1d_flush_type types)
|
||||
{
|
||||
unsigned int instrs[3], *dest;
|
||||
long *start, *end;
|
||||
int i;
|
||||
|
||||
start = PTRRELOC(&__start___rfi_flush_fixup),
|
||||
end = PTRRELOC(&__stop___rfi_flush_fixup);
|
||||
|
||||
instrs[0] = 0x60000000; /* nop */
|
||||
instrs[1] = 0x60000000; /* nop */
|
||||
instrs[2] = 0x60000000; /* nop */
|
||||
|
||||
if (types & L1D_FLUSH_FALLBACK)
|
||||
/* b .+16 to fallback flush */
|
||||
instrs[0] = 0x48000010;
|
||||
|
||||
i = 0;
|
||||
if (types & L1D_FLUSH_ORI) {
|
||||
instrs[i++] = 0x63ff0000; /* ori 31,31,0 speculation barrier */
|
||||
instrs[i++] = 0x63de0000; /* ori 30,30,0 L1d flush*/
|
||||
}
|
||||
|
||||
if (types & L1D_FLUSH_MTTRIG)
|
||||
instrs[i++] = 0x7c12dba6; /* mtspr TRIG2,r0 (SPR #882) */
|
||||
|
||||
for (i = 0; start < end; start++, i++) {
|
||||
dest = (void *)start + *start;
|
||||
|
||||
pr_devel("patching dest %lx\n", (unsigned long)dest);
|
||||
|
||||
patch_instruction(dest, instrs[0]);
|
||||
patch_instruction(dest + 1, instrs[1]);
|
||||
patch_instruction(dest + 2, instrs[2]);
|
||||
}
|
||||
|
||||
printk(KERN_DEBUG "rfi-flush: patched %d locations\n", i);
|
||||
}
|
||||
#endif /* CONFIG_PPC_BOOK3S_64 */
|
||||
|
||||
void do_lwsync_fixups(unsigned long value, void *fixup_start, void *fixup_end)
|
||||
{
|
||||
long *start, *end;
|
||||
|
|
|
|||
|
|
@ -35,13 +35,63 @@
|
|||
#include <asm/opal.h>
|
||||
#include <asm/kexec.h>
|
||||
#include <asm/smp.h>
|
||||
#include <asm/tm.h>
|
||||
#include <asm/setup.h>
|
||||
|
||||
#include "powernv.h"
|
||||
|
||||
static void pnv_setup_rfi_flush(void)
|
||||
{
|
||||
struct device_node *np, *fw_features;
|
||||
enum l1d_flush_type type;
|
||||
int enable;
|
||||
|
||||
/* Default to fallback in case fw-features are not available */
|
||||
type = L1D_FLUSH_FALLBACK;
|
||||
enable = 1;
|
||||
|
||||
np = of_find_node_by_name(NULL, "ibm,opal");
|
||||
fw_features = of_get_child_by_name(np, "fw-features");
|
||||
of_node_put(np);
|
||||
|
||||
if (fw_features) {
|
||||
np = of_get_child_by_name(fw_features, "inst-l1d-flush-trig2");
|
||||
if (np && of_property_read_bool(np, "enabled"))
|
||||
type = L1D_FLUSH_MTTRIG;
|
||||
|
||||
of_node_put(np);
|
||||
|
||||
np = of_get_child_by_name(fw_features, "inst-l1d-flush-ori30,30,0");
|
||||
if (np && of_property_read_bool(np, "enabled"))
|
||||
type = L1D_FLUSH_ORI;
|
||||
|
||||
of_node_put(np);
|
||||
|
||||
/* Enable unless firmware says NOT to */
|
||||
enable = 2;
|
||||
np = of_get_child_by_name(fw_features, "needs-l1d-flush-msr-hv-1-to-0");
|
||||
if (np && of_property_read_bool(np, "disabled"))
|
||||
enable--;
|
||||
|
||||
of_node_put(np);
|
||||
|
||||
np = of_get_child_by_name(fw_features, "needs-l1d-flush-msr-pr-0-to-1");
|
||||
if (np && of_property_read_bool(np, "disabled"))
|
||||
enable--;
|
||||
|
||||
of_node_put(np);
|
||||
of_node_put(fw_features);
|
||||
}
|
||||
|
||||
setup_rfi_flush(type, enable > 0);
|
||||
}
|
||||
|
||||
static void __init pnv_setup_arch(void)
|
||||
{
|
||||
set_arch_panic_timeout(10, ARCH_PANIC_TIMEOUT);
|
||||
|
||||
pnv_setup_rfi_flush();
|
||||
|
||||
/* Initialize SMP */
|
||||
pnv_smp_init();
|
||||
|
||||
|
|
|
|||
|
|
@ -450,6 +450,39 @@ static void __init find_and_init_phbs(void)
|
|||
of_pci_check_probe_only();
|
||||
}
|
||||
|
||||
static void pseries_setup_rfi_flush(void)
|
||||
{
|
||||
struct h_cpu_char_result result;
|
||||
enum l1d_flush_type types;
|
||||
bool enable;
|
||||
long rc;
|
||||
|
||||
/* Enable by default */
|
||||
enable = true;
|
||||
|
||||
rc = plpar_get_cpu_characteristics(&result);
|
||||
if (rc == H_SUCCESS) {
|
||||
types = L1D_FLUSH_NONE;
|
||||
|
||||
if (result.character & H_CPU_CHAR_L1D_FLUSH_TRIG2)
|
||||
types |= L1D_FLUSH_MTTRIG;
|
||||
if (result.character & H_CPU_CHAR_L1D_FLUSH_ORI30)
|
||||
types |= L1D_FLUSH_ORI;
|
||||
|
||||
/* Use fallback if nothing set in hcall */
|
||||
if (types == L1D_FLUSH_NONE)
|
||||
types = L1D_FLUSH_FALLBACK;
|
||||
|
||||
if (!(result.behaviour & H_CPU_BEHAV_L1D_FLUSH_PR))
|
||||
enable = false;
|
||||
} else {
|
||||
/* Default to fallback if case hcall is not available */
|
||||
types = L1D_FLUSH_FALLBACK;
|
||||
}
|
||||
|
||||
setup_rfi_flush(types, enable);
|
||||
}
|
||||
|
||||
static void __init pSeries_setup_arch(void)
|
||||
{
|
||||
set_arch_panic_timeout(10, ARCH_PANIC_TIMEOUT);
|
||||
|
|
@ -467,6 +500,8 @@ static void __init pSeries_setup_arch(void)
|
|||
|
||||
fwnmi_init();
|
||||
|
||||
pseries_setup_rfi_flush();
|
||||
|
||||
/* By default, only probe PCI (can be overridden by rtas_pci) */
|
||||
pci_add_flags(PCI_PROBE_ONLY);
|
||||
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@
|
|||
#include <linux/export.h>
|
||||
#include <linux/context_tracking.h>
|
||||
#include <linux/user-return-notifier.h>
|
||||
#include <linux/nospec.h>
|
||||
#include <linux/uprobes.h>
|
||||
|
||||
#include <asm/desc.h>
|
||||
|
|
@ -201,7 +202,7 @@ __visible inline void prepare_exit_to_usermode(struct pt_regs *regs)
|
|||
* special case only applies after poking regs and before the
|
||||
* very next return to user mode.
|
||||
*/
|
||||
current->thread.status &= ~(TS_COMPAT|TS_I386_REGS_POKED);
|
||||
ti->status &= ~(TS_COMPAT|TS_I386_REGS_POKED);
|
||||
#endif
|
||||
|
||||
user_enter_irqoff();
|
||||
|
|
@ -277,7 +278,8 @@ __visible void do_syscall_64(struct pt_regs *regs)
|
|||
* regs->orig_ax, which changes the behavior of some syscalls.
|
||||
*/
|
||||
if (likely((nr & __SYSCALL_MASK) < NR_syscalls)) {
|
||||
regs->ax = sys_call_table[nr & __SYSCALL_MASK](
|
||||
nr = array_index_nospec(nr & __SYSCALL_MASK, NR_syscalls);
|
||||
regs->ax = sys_call_table[nr](
|
||||
regs->di, regs->si, regs->dx,
|
||||
regs->r10, regs->r8, regs->r9);
|
||||
}
|
||||
|
|
@ -299,7 +301,7 @@ static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs)
|
|||
unsigned int nr = (unsigned int)regs->orig_ax;
|
||||
|
||||
#ifdef CONFIG_IA32_EMULATION
|
||||
current->thread.status |= TS_COMPAT;
|
||||
ti->status |= TS_COMPAT;
|
||||
#endif
|
||||
|
||||
if (READ_ONCE(ti->flags) & _TIF_WORK_SYSCALL_ENTRY) {
|
||||
|
|
@ -313,6 +315,7 @@ static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs)
|
|||
}
|
||||
|
||||
if (likely(nr < IA32_NR_syscalls)) {
|
||||
nr = array_index_nospec(nr, IA32_NR_syscalls);
|
||||
/*
|
||||
* It's possible that a 32-bit syscall implementation
|
||||
* takes a 64-bit parameter but nonetheless assumes that
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue