Re: exercising current aarch64 kprobe support with systemtap

[David Long <dave dot long at linaro dot org>]

On 06/09/2016 03:52 PM, William Cohen wrote:
> On 06/09/2016 12:17 PM, William Cohen wrote:
> > I have been exercising the current kprobes and uprobe patches for
> > arm64 that are in the test_upstream_arm64_devel branch of
> > https://github.com/pratyushanand/linux with systemtap.  There are a
> > two issues that I have seen on this kernel with systemtap.  There are
> > some cases where kprobes fail to register at places that appear to be
> > reasonable places for a kprobe.  The other issue is that kernel starts
> > having soft lockups when the hw_watch_addr.stp tests runs.  To get
> > systemtap with the newer kernels need the attached hack because of
> > changes in the aarch64 macro args.
> >
> > EINVAL for seemingly valid kprobe registration
> >
> > Below shows the bz1027459.stp failing because of the some of the kprobes not registering.
> >
> > # make installcheck RUNTESTFLAGS="--debug systemtap.base/bz1027459.exp"
> > ...
> >
> >
> > spawn stap /root/systemtap_write/systemtap/testsuite/systemtap.base/bz1027459.stp
> > WARNING: probe kernel.function("SyS_set_tid_address@kernel/fork.c:1236").call (address 0xfffffc00080c9578) registration error (rc -22)
> > WARNING: probe kernel.function("SyS_sched_setaffinity@kernel/sched/core.c:4690").call (address 0xfffffc0008104d58) registration error (rc -22)
> > WARNING: probe kernel.function("SyS_sched_get_priority_min@kernel/sched/core.c:5013").call (address 0xfffffc0008105250) registration error (rc -22)
> > WARNING: probe kernel.function("SyS_sched_get_priority_max@kernel/sched/core.c:4986").call (address 0xfffffc00081051e8) registration error (rc -22)
> > hi
> > FAIL: bz1027459 -p5 (0)
> >
> > area around  Sys_set_tid_address
> >
> > fffffc00080c956c:	d503201f 	nop
> > fffffc00080c9570:	08dc4c80 	.word	0x08dc4c80
> > fffffc00080c9574:	fffffc00 	.word	0xfffffc00
> >
> > fffffc00080c9578 <SyS_set_tid_address>:
> > fffffc00080c9578:	a9be7bfd 	stp	x29, x30, [sp,#-32]!
> > fffffc00080c957c:	910003fd 	mov	x29, sp
> >
> > area around SyS_sched_setaffiniity
> >
> > fffffc0008104d4c:	17ffff73 	b	fffffc0008104b18 <sched_setaffinity+0x438>
> > fffffc0008104d50:	08dd9d80 	.word	0x08dd9d80
> > fffffc0008104d54:	fffffc00 	.word	0xfffffc00
> >
> > fffffc0008104d58 <SyS_sched_setaffinity>:
> > fffffc0008104d58:	a9bb7bfd 	stp	x29, x30, [sp,#-80]!
> > fffffc0008104d5c:	910003fd 	mov	x29, sp
> >
> > area around SyS_sched_get_priority_min
> >
> > fffffc0008105244:	f9400bf3 	ldr	x19, [sp,#16]
> > fffffc0008105248:	a8c27bfd 	ldp	x29, x30, [sp],#32
> > fffffc000810524c:	d65f03c0 	ret
> >
> > fffffc0008105250 <SyS_sched_get_priority_min>:
> > fffffc0008105250:	a9be7bfd 	stp	x29, x30, [sp,#-32]!
> > fffffc0008105254:	910003fd 	mov	x29, sp
> >
> >
> > area around SyS_sched_get_priority_max
> >
> >
> > fffffc00081051dc:	17ffffe8 	b	fffffc000810517c <sys_sched_yield+0x34>
> > fffffc00081051e0:	08dd9d80 	.word	0x08dd9d80
> > fffffc00081051e4:	fffffc00 	.word	0xfffffc00
> >
> > fffffc00081051e8 <SyS_sched_get_priority_max>:
> > fffffc00081051e8:	a9be7bfd 	stp	x29, x30, [sp,#-32]!
> > fffffc00081051ec:	910003fd 	mov	x29, sp
> >
> >
> > The stp (store pair) instructions at the beginning of these functions
> > should be fine to instrument.  One thing that I could think of causing
> > a problem is the test to make sure that the instruction is not inside
> > a load exclusive/store exclusive region.  The test might be mistaking
> > some of the data before the start of the function as load exclusive
> > instructions.
>
>
> I verified that the cause of kprobes not being registered is the scan
> backward for load exclusive instructions.  For one example have:
>
> ...
> fffffc00080c98cc:	d503201f 	nop
> fffffc00080c98d0:	08dc4c80 	.word	0x08dc4c80
> fffffc00080c98d4:	fffffc00 	.word	0xfffffc00
>
> fffffc00080c98d8 <SyS_set_tid_address>:
> fffffc00080c98d8:	a9be7bfd 	stp	x29, x30, [sp,#-32]!
> fffffc00080c98dc:	910003fd 	mov	x29, sp
>
> The previous function has 0xfffffc0008dc4c80 as data at the end of the
> function. The scan backwards from the beginning of the current
> function Sys_set_tid_address stumbles into that data and interprets
> the 0x08dc4c80 as load exclusive instructions.  This causes the kprobe
> registration to fail.
>
> Disabled the is_probed_address_atomic() scan for atomic instructions
> allows the test to work:
>
> make installcheck RUNTESTFLAGS="--debug systemtap.base/bz1027459.exp"
> ...
> Running target unix
> Running /root/systemtap_write/systemtap/testsuite/systemtap.base/bz1027459.exp ...
> PASS: bz1027459 -p5
>
> 		=== systemtap Summary ===
>
> # of expected passes		1

Interesting coincidence.  Thanks for isolating the cause of that problem.

> Somehow the is_probed_address_atomic and arm_kprobe_decode_insn
> functions need to avoid scanning past the beginning of a function.
>
> -Will

I'm not sure we're going to be able to do that.  We could interpret a 
"ret" to end the scan but that's probably going to be on the other side 
of the data. And can we be certain the compiler only puts literals 
between functions?  Maybe the "stp x29,x30,[sp,...]" instruction is 
definitive enough to use as a flag for the beginning of functions.  This 
is the approach I'm now thinking might be the best fix.

-dl