Safety in Systemtap Brad Chen Intel Corporation brad dot chen at intel dot com Draft of 25 March 2005 INTRODUCTION Safety is an important goal for Systemtap. Both performance tool users and kernel debuggers are better served by a tool that protects them from system crashes and unwanted side-effects. Safety is key design point of tools to which Systemtap will be compared, including DProbes and DTrace. The Systemtap team should have a clear notion of how our tool's safety properties compares to other relevant systems. The goals of this paper are: - to support collection of and agreement on safety goals for Systemtap - to explore implementation options, tradeoffs and challenges - to raise awareness of safety issues and help create a common understanding of all the issues, which we will address, and how The paper is organized into the following sections: - Candidate Safety Goals. Reviews specific safety properties that Systemtap may have as goals. This is intended to be a superset of the safety properties of comparable systems. - Implementation Options. Discusses a number of implementation options and how they might help with one or more safety goals. - Proposal. A proposed technical approach describing some core functionality with a couple enhancements. - Appendix: Usage Models. Provide motivation for Safety Goals and a yardstick for evaluating proposed designs. - Appendix: Feature/Implementation Matrix. Work in progress. CANDIDATE SAFETY GOALS This section enumerates a set of relevant safety goals for Systemtap. These are intended to be a superset of the safety properties of Solaris DTrace, and should all be supported by usage scenarios in the appendix. There is no presumption that Systemtap will support all of these safety features. G1. "Safety" and "Guru" modes: Sets of safety-related features are enabled or disabled using a single option such as a command-line flag. The flag might change the behavior of the compiler, the loader, or the runtime. A "guru" mode would enable a set of features deemed unnecessary for or too dangerous for the most common uses of the system. This would be useful for experts, kernel debugging, etc. A "safety" mode would impose additional safety constraints. This would be useful for use on production systems, for running 3rd party or precompiled scripts, or for novice or unpriviledged users. Unless otherwise indicated this document assumes the system is not being used in "guru" mode. G2. Finite loops: Infinite loops cause a script to be safely terminated. G3. Finite loops: Systemtap supports only restricted looping constructs that make it impossible to write an infinite loop. G4. Finite recursion: Recursion beyond a pre-defined limit causes a script to be safely terminated. G5. No procedures. The scripting language does not support procedure definition. Similar to DTrace. Note this also makes recursion impossible. G6. Restricted kernel calls: The system restricts the kernel subroutines that can be invoked directly by a script. Various options are possible; these are discussed in the implementation section. G7. No non-Algol control flow. Excludes goto, longjmp, and other non-Algol from the language definition or the language implementation. G8. No external writes. Disallows writes to kernel or user memory outside the immediate Systemtap environment. This restriction is lifted for "guru" mode. G9. Restricted reads. Disallows reads as per some policy TBD. For example, user memory read access might be restricted to processes belonging to the user running the script. G10. Remote compilation. Allow scripts to be compiled on one machine for use on a different machine. Systemtap might or might not require the machines have identical configurations. G12. Graceful recovery from illegal memory references. Fault-handling mechanisms insure that read or writes to invalid memory locations result in a graceful failure of the script without otherwise disrupting the system. The next three "features" are somewhat ephemeral. G13. Crashproof. Systemtap might use a combination of safety features to make it impossible or more difficult to crash or disable a system through accidental or deliberate misuse of the Systemtap. This is an important feature for performance tool users who frequently are not kernel experts or experienced system administrators, but it has a lot of implications. DTrace appears to be crashproof. Its very restricted language and interpreted contribute to this goal. G14. Bug resistant. Systemtap might include features such that a bug in some part of the system will tend to cause a script to fail gracefully rather than crash the system. G15. Easy to trust. Systemtap might choose safety mechanisms that are easy to understand and/or validate. As an example, its easy to confirm that a language has no looping construct, and is much harder to be confident that a compiler is bug-free. Systemtap might choose a design where only a single component had to be trusted (such as an in-kernel interpreter) rather than requiring a chain of multiple components to be trusted to achieve safety. Systemtap could be made easier to trust by avoiding mechanisms that rely on arcane theory or are otherwise hard to understand, and by prefering mechanisms that are intuitive to a programmer of relatively ordinary skill. IMPLEMENTATION OPTIONS FOR SYSTEMTAP SAFETY FEATURES This section describes implementation options for some of the above safety goals. Language Design The Systemtap language could be designed so as to make certain kinds of problems impossible. It might not support procedure declarations, preventing recursion. It might not support a general purpose looping construct, preventing infinite loops. It might not support general-purpose heap-allocated memory. Heap memory might still be required to support dynamic structures such as maps, but would only be available in these "managed" forms, with no pointers or general-purpose "malloc" facility. It might limit use of arrays such that bounds checks can be implemented for all array references. It might limit external procedure calls to routines from a pre-defined runtime API. The compilation system could sign binaries to confirm their source. Optionally this signature might be cryptographically secure. An insecure signature would enhance safety, protecting against accidental misuse and making deliberate misuse somewhat more complicated. Static Checking A static analyzer would inspect Systemtap binary modules immediately before loading into the kernel, checking headers, disassembling the module, and examining it for disallowed code sequences and other problems. Properties that might be checked include: - Signatures. Check the signature applied by the Systemtap compiler. Apart from making taps tamper resistant it would also prevent accidental loading of binaries not generated by Systemtap. - Version/build checks: Check to confirm version consistency between kernel, runtime, compiler, translator and other critical components. - Illegal instructions: The tap could be disassembled to confirm that it does not include priviledged system instructions or instructions that are not allowed in the kernel (floating point, MMX, SSE, etc.) - Illegal external instruction references. The static checker could examine external references and confirm that they conform to a given policy. Example policies include: - no external calls - all external references must go through a runtime portal that provides dynamic checks - Illegal external data references. The static checker could examine memory reference instructions, and confirm that they conform to a given security policy. The policy might restrict writes, reads, and might require uncheckable references occur only via a runtime portal that provides dynamic checks. - Illegal control flow. The static checker might disallow non-local non-procedural control flow. It might disallow backward branches, assuming the source language were loop-free. - Failsafe: The static checker could declare a module unsafe if it could not be fully checked. The tap compiler would be required to generate checkable code. Language Implementation This section describes safety features that could be supported by the compiler and runtime. Note that most of these features require support from other components in order to be effective. The Systemtap runtime might use a timeout or counting mechanism to limit time spent in loops, thereby preventing infinite loops. The implementation might require the compiler to generate some special registration code prior to entry into a loop, and deregistration afterwards. DProbes uses an interpreted language, with the interpreter limiting the number of jumps taken by a script. DTrace takes an different approach, omitting a general purpose looping construct. The language implementation might provide a similar counting or timeout mechanism to prevent recursion-related faults. The runtime might provide a kernel call portal that would check external subroutine invocations from Systemtap scripts to insure they satisfy a given safety policy. The runtime might also provide a memory access portal that would check conformance of memory reads and writes to a given safety policy. To be fully effective, this language implementation would have to use these portals, and the binary would need to be checked by a static checker or loader to confirm that all external references used the portal. PROPOSAL This design proposal is structured as a minimal set of core functionality, followed various extensions that incrementally improve safety. The "Core" is intended to describe what is currently being discussed for Systemtap. Extensions A and B describe additional safety facilities that could enhance Systemtap to approach the safety of other systems. Core: Trusted Language Implementation This core description is an attempt to capture what being planned for Systemtap as of March 2005. Systemtap supports the following safety features: - finite loops - finite recursion - limited kernel API - pre-compiled scripts - version checking It guarentees these features: - pre-compiled scripts - version checking Some of the 'supported' features could be defeated due to accidental or deliberate misuse. Language Design: This design assumes a Systemtap scripting language with the following features: - general-purpose looping constructs comparable to C - procedure definition allowed - recursive procedures allowed - general-purpose array definitions allowed - unrestricted reading of external data - no kernel calls outside of pre-defined Systemtap runtime - no writes to external data - no non-algol control flow transfers - no explicit pointers; no general-purpose malloc Language Implementation: - compiler-based instrumentation of loops and recursion to support counters or timeout mechanism - remote compilation support. Compiled scripts can be moved between machines. This requires careful attention that the build system configuration is synchronized with that of the execution system. Runtime: - version checking of kernel, API, runtime etc. - counting or timeout facility for finite loops and recursion - kernel support to render harmless illegal memory references Deficiencies relative to other systems: - Hard to trust. Safety of the system depends on many components of the system being bug-free including the elaborator, compiler, linker, kernel and runtime. It also requires that "guru" features are not used, but doesn't provide a simple way to disable those features. - Remote compilation support introduces the possibility that an arbitrarily problematic binary image might be loaded into the kernel. Other systems avoid this risk by using an interpreter. Extension A: Enhanced Runtime Checking A kernel call/read/write portal supports implementation of policies with various restrictions. The implementation of non-trivial policies is TBD; in the mean time trivial policies (no external writes, no external calls) would be useful. This would only work of course if the language implementation used the portal. Extension B: Static Checker A static checker confirms safety properties of a compiled Systemtap script immediately prior to loading into the kernel. The static checker checks the following properties: - version checks - no priviledged instructions - no floating-point or other illegal kernel instructions - all loops (backward branches) apply the loop timeout mechanism - all recursive procedures apply the recursion timeout mechanism - all external references through a kernel call portal - all external reads through a kernel mem portal - all external writes through a kernel mem portal The static checker makes the system easier to trust by reducing the number of components that must be trusted to trust the system. The components require trust would be the checker, the runtime, and the kernel. The checker would validate the output of the compiler, elaborator and linker so those components would no longer need to be trusted. The static checker would also prevent many of the failure modes introduced by remote compilation, by enforcing most of the constraints of "safety" and "guru" modes in a single place, rather than having the implementation distributed broadly across a tool chain. If it were implemented using a tool like ATOM, the static checker could be a relatively short piece of code (< 1000 lines) which would make visual inspection feasable for people who cared. APPENDIX: USAGE MODELS This is an attempt to catalogue common scenarios pertaining to safety and security. M1: kernel version mismatch M2: systemtap API version mismatch M3: systemtap runtime version mismatch Script was compiled on an old version of the system, then the system is upgraded, and is not inconsistent with current APIs, symbols, etc. M4: Wrong binary In the heat of troubleshooting, a user specifies the wrong binary to load into the kernel. Assume the user is root and the binary is not deliberately malicious but otherwise arbitrarily bad. Specific cases might include: a: binary includes priviledged instructions b: IA32 binary includes illegal ring0 instructions c: binary writes randomly into kernel memory M5: buggy script A user who is learning to use Systemtap writes many buggy scripts. Can the we guarentee that the system will never crash? Consider memory leak, recursion, infinite loop, stale pointers, bogus pointers, jumps into arbitrary locations in kernel code, etc. M6: a bug in the compiler, translator, runtime Does the system crash or just print a warning message? M7: Remote compilation of script. Compiler not installed on production machine. a: binary includes priviledged instructions b: IA32 binary includes illegal ring0 instructions c: binary writes randomly into kernel memory Some of these cases might come up if a kernel debugging script were confused with a performance script. M8: 3rd party script as a binary form M9: 3rd party script as a source file A Systemtap user downloads a script from a public bboard or web site. The script sounds terribly interesting but the user is not completely confident in the source. How worried does he/she need to be about trying it? M10: malicious attack: corrupted kernel Probably nothing much you can do here. M11: malicious attack: corrupted compiler, corrupted script- launching system We do our users a favor by minimizing the number of software components they need to trust or secure in order to make the overall system secure. M12: malicious attack: bad binary script. Could have a stack overwrite, code hidden in data segments, who knows... Here's a few non-safety usage scenarios, which I'm listing here but which aren't otherwise considered in this document. M13: Fault injection M14: Patching kernel data M15: Patching kernel code APPENDIX: FEATURE/IMPLEMENTATION MATRIX This table is a tool for thinking about implementation options for various safety properties. Think of it as a placeholder for something useful. Perhaps this is more than should have been attempted in ASCII. +------------------------ finite loops | +------------------------ finite recursion | | +------------------------ Restricted kernel calls | | | +------------------------ no non-Algol control flow | | | | +------------------------ restricted reads | | | | | +------------------------ restricted writes | | | | | | +------------------------ pre-compiled scripts | | | | | | | +------------------------ version checking | | | | | | | | +------------------------ bug resistant | | | | | | | | | +------------------------ crashproof | | | | | | | | | | +------------------------ easy to trust | | | | | | | | | | | +------------------------ guru mode | | | | | | | | | | | | ===================================================================== Language Design and Implementation no procedures no general-purpose loops no recursion no heap memory array bounds checks no non-algol control flow no kernel calls no kernel reads no kernel writes kernel call portal kernel memory portal sandboxed reads/writes sandboxed calls sandboxed branches ===================================================================== Static Checks signature check version check illegal instructions external call external read external write backward branches illegal control flow failsafe ===================================================================== Runtime Checks loop counters or timeouts recursion counters or timeouts kernel call portal kernel memory portal trap invalid references ===================================================================== Ephemeral Techniques bug-free code faith =====================================================================