Re: exercising current aarch64 kprobe support with systemtap

[Pratyush Anand <panand at redhat dot com>]

Hi Will,

On 04/08/2016:09:56:45 AM, William Cohen wrote:
> On 08/04/2016 12:42 AM, Pratyush Anand wrote:
> > Hi Will,
> > 
> > On 03/08/2016:01:39:47 PM, William Cohen wrote:
> >> On 08/03/2016 09:13 AM, Pratyush Anand wrote:
> >>> On 07/07/2016:03:58:37 PM, Frank Ch. Eigler wrote:
> >>>> David Long <dave.long@linaro.org> writes:
> >>>>
> >>>>> [...]
> >>>>>> - bug_handler() calls report_bug() which calls __warn()
> >>>>>> - __warn() does lot of pr_warn()  which invokes print_worker_info()
> >>>>>> where we have a kprobe instrumented.
> >>>>>> - Therefore, we are encountering this issue.
> >>>
> >>> Hi Will,
> >>>
> >>> Can you please cross-check if following branch works perfectly with
> >>> kprobes_onthefly.exp and other systemtap tests.
> >>>
> >>> https://github.com/pratyushanand/linux/tree/uprobe/upstream_arm64_devel_v1.1
> >>>
> >>> Following patch in above branch should solve this issue.
> >>> https://github.com/pratyushanand/linux/commit/d0dcc6477f1279ab0bd99aefc30efdecb16c586e
> >>>
> >>> However, I am not yet sure that above modification could be the best solution,
> >>> so discussing at arm kernel list.
> >>>
> >>> ~Pratyush
> >>>
> >>
> >> I have an AMD seattle machine set up Fedora24, the upstream_arm64_devel_v1.1 branch kernel locally built, and a locally built checkout of systemtap (systemtap rpm in fc24 doesn't generate models for linux 4.6 and newer kernels).  Tried to run the systemtap tests with:
> >>
> >>  make installcheck RUNTESTFLAGS="--debug systemtap.onthefly/kprobes_onthefly.exp"
> >>
> >> However at some point the kernel starts having problems:
> >>
> >> http://paste.stg.fedoraproject.org/5375/
> > 
> > Yes, this is what you were getting with earlier code as well, but now it is not
> > going to infinite unexpected EL1, so at least proposed kprobe improvement seems
> > fine to me.
> > 
> > In this failing test we are getting oom and the oom_killer is called. So,
> > I think, this is another point of investigation that why this OOM occurs. 
> > 
> > ~Pratyush
> > 
> 
> Hi,
> 
> The OOM errors came before the otf_stress_hard_iter_5000 test that previous triggered the infinite unexpected EL1, so can't really say that the proposed patch has fixed the problem.

Yes, yes, previously also we were getting OOM, and then that OOM was triggering
infinite unexpected EL1, because OOM message uses WARN_ON() to print, and
WARN_ON() uses "BRK BUG_BRK_IMM". Now when it is printing though BRK, we were
hitting kprobe at print_worker_info() which was resulting in unexpected EL1.

Proposed patch fixes kprobe tracing within none kprobe BRK context such as
uprobe or WARN_ON() breakpoint handler etc. So, now a kprobe at
print_worker_info() will work while printing message of WARN_ON().


> 
> Any thoughts on how to track down the oom issue?  Are you able to replicate it running the systemtap onthefly/kprobes_onthefly.exp tests?

Sure, will look into. Have reserved a seattle.

~Pratyush