Re: Systemtap do_filp_open failure on a few linux packages

[Henrik /KaarPoSoft <henrik at kaarposoft dot dk>]

Hi David,

Thank you very much for going through my input and providing an 
excellent answer.
Your input nailed it for me (although with a twist; see below)

On 06/17/2013 08:16 PM, David Smith wrote:
> On 06/15/2013 04:52 PM, Henrik /KaarPoSoft wrote:
> > Dear all,
> >
> > I have experienced a very strange issue related to systemtap.
> > Any insights or help you might be able to provide to help me debug this
> > further would be most appreciated.
> >
> > I am developing a linux distribution called KaarPux:
> > http://kaarpux.kaarposoft.dk/
> >
> > Using a few scripts, some 600+ linux packages are build and installed.
> > Generally, this works like a charm.
> >
> > In order to automatically collect package dependencies, I have created
> > a small systemtap script to show files opened for reading:
> > http://sourceforge.net/p/kaarpux/code/ci/be342bf5667253421f562b7bc29bab8e0a2560aa/tree/master/chroot_scripts/kx_open.stp
>
> The script looks reasonable. One small note, you shouldn't need the
> '@defined($return)' check since $return should always be defined in
> 'kernel.function("do_filp_open").return'.

OK, thanks, will remove it later...
Kept it for now: better safe than sorry.

> > For most of the 600+ packages, building is successfull, and the probe
> > returns what seems to be reasonable results.
> >
> > However, for a few packages, building fails:
> > - firefox
> > - thunderbird
> > - libreoffice
> > - ghc-binary
> > - ghc
>
> Here's my thought. I don't think it is the probe itself that is causing
> the problem, my guess would be that it is our staprun loader (a setuid
> executable). When the module gets loaded, we unset several environment
> variables, like 'IFS', 'CDPATH', 'ENV', 'BASH_ENV' for security reasons.
>
> I'd guess those failing packages depend on something set in the environment.
>
> If this is the case, to work around this problem you might be able to
> modify the script you use to run staprun (linux_functions.shinc) to add
> the environment variable back in. Here's that line from
> linux_functions.shinc (split up a bit):
>
> ====
>   staprun /lib/modules/$(uname -r)/systemtap/kx_open.ko \
>    -c "./scripts/${PASS}/${PKG}_${STEP}.sh" \
>    -o "${PIPE}" > ./log/${PASS}/${PKG}_${STEP}.log 2>&1
> ====
>
> You could change that to
>
> ====
>    staprun /lib/modules/$(uname -r)/systemtap/kx_open.ko \
>     -c "env 'FOO=BAR' ./scripts/${PASS}/${PKG}_${STEP}.sh" \
>     -o "${PIPE}" > ./log/${PASS}/${PKG}_${STEP}.log 2>&1
> ====
>
> Your other option would be to put the missing environment variable in
> scripts/${PASS}/${PKG}_${STEP}.sh.
>
> This might not be the problem, but it is certainly worth investigating.

My problem turned out to be environment variables indeed!
Thank you for pointing me in this direction.

However, the culprit was that LD_LIBRARY_PATH got killed.

Simply adding env LD_LIBRARY_PATH=... to the staprun solved the problem:
http://sourceforge.net/p/kaarpux/code/ci/master/tree/master/shinc/linux_functions.shinc

So, again:
THANKS A MILLION FOR THE HINT !!!

/Henrik