Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()

[William Cohen <wcohen at redhat dot com>]

On 06/22/2015 10:15 AM, Mark Wielaard wrote:
> On Mon, 2015-06-22 at 09:53 -0400, William Cohen wrote:
>> The original traceback was:
>>
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Call trace:
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fbe80>] unwind+0xbc/0x148 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fc104>] _stp_stack_user_get+0x9c/0x1d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fd7f4>] probe_2718+0x234/0x524 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fde48>] stapiu_probe_prehandler+0x1d4/0x384 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00001a2af0>] uprobe_notify_resume+0x3b4/0x8fc
>> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00000972ec>] do_notify_resume+0x80/0x8c
>>
>> Adjusting the addresses for the module by itself and Using addr2line the call trace maps back to:
>> a2e0
>> /root/systemtap_write/install/share/systemtap/runtime/unwind.c:429
>> b3f4
>> /root/systemtap_write/install/share/systemtap/runtime/unwind.c:1287
>> be80
>> /root/systemtap_write/install/share/systemtap/runtime/unwind.c:1518
>> c104
>> /root/systemtap_write/install/share/systemtap/runtime/stack.c:491
>> d7f4
>> /root/systemtap_write/install/share/systemtap/runtime/stack.c:591
>> de48
>> /tmp/stapUGCcwL/l100_src.c:867
>>
>> I hope that provides a bit more insight into what is going wrong in the systemtap unwinder runtime.
>  
> Most certainly. Assuming that maps to current git sources the crash is
> at the end of the following case statement, at the memcpy:


Yes, this for a systemtap built from the current git repository the last commit in there is:

commit 7e05b29acbc5bf84b860499ab1e3d1fa648ad086
Author: David Smith <dsmith@redhat.com>
Date:   Fri Jun 19 13:26:41 2015 -0500

    Fixed PR18563 by updating mbrwatch.meta.
    
    * testsuite/systemtap.examples/io/mbrwatch.meta: Update to ignore any
      devices called 'sr[0-0]', which on ppc64 aren't disk drives.



 
>   case DW_CFA_restore_extended:
>           value = get_uleb128(&ptr.p8, end);
>           if (compat_task) {
>                   dbug_unwind(1, "map DW_CFA_restore_extended value %ld to reg_info idx %ld\n",
>                               value, COMPAT_REG_MAP(DWARF_REG_MAP(value)));
>                  value = COMPAT_REG_MAP(DWARF_REG_MAP(value));
>           } else {
>                   dbug_unwind(1, "map DW_CFA_restore_extended value %ld to reg_info idx %ld\n",
>                               value, DWARF_REG_MAP(value));
>                   value = DWARF_REG_MAP(value);
>           }
>           memcpy(&REG_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
>           break;
> 
> Note how value comes from DWARF_REG_MAP. It maps the DWARF register
> number to the number we use in the arrays. It is defined as follows in
> runtime/unwind/arm64.h
> 
> #define DWARF_REG_MAP(r) \
>         ((r >= 0 && r <= 31) ? r /* regs[0-30] + sp */  \
>          : 9999)
> 
> So, if it doesn't recognize (or more accurately, isn't interested in)
> the DWARF register number it will return a ridiculously large number.
> Which we will use directly to index the register array. Oops...
> 
> In all other cases, except this and DW_CFA_restore, an helper function
> is called (set_*_rule) which will only use the reg value if (reg <
> ARRAY_SIZE(REG_STATE.regs)).
> 

I have built the systemtap code with the patch below and exercising with:

while [ 1 ]; do /root/systemtap_write/install/bin/stap systemtap.examples/memory/last_100_frees.stp -B CONFIG_DEBUG_INFO=y -c "/root/systemtap_write/install/bin/stap -V" -d /root/systemtap_write/install/bin/stap --ldd;done

Thanks again for the fix Mark.

-Will


> So we will need something like:
> 
> diff --git a/runtime/unwind.c b/runtime/unwind.c
> index d38363b..4dbab33 100644
> --- a/runtime/unwind.c
> +++ b/runtime/unwind.c
> @@ -426,7 +426,8 @@ static int processCFI(const u8 *start, const u8 *end, unsigned long targetLoc,
>                                                     value, DWARF_REG_MAP(value));
>                                         value = DWARF_REG_MAP(value);
>                                 }
> -                               memcpy(&REG_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
> +                               if (value < ARRAY_SIZE(REG_STATE.regs))
> +                                       memcpy(&REG_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
>                                 break;
>                         case DW_CFA_undefined:
>                                 value = get_uleb128(&ptr.p8, end);
> @@ -641,7 +642,8 @@ static int processCFI(const u8 *start, const u8 *end, unsigned long targetLoc,
>                                             value, DWARF_REG_MAP(value));
>                                 value = DWARF_REG_MAP(value);
>                         }
> -                       memcpy(&REG_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
> +                       if (value < ARRAY_SIZE(REG_STATE.regs))
> +                               memcpy(&REG_STATE.regs[value], &state->cie_regs[value], sizeof(struct unwind_item));
>                         break;
>                 }
>                 dbug_unwind(1, "targetLoc=%lx state->loc=%lx\n", targetLoc, state->loc);
> 
> Does that help in your case?
> 
> Thanks,
> 
> Mark
>