Safety in Systemtap Brad Chen Intel Corporation brad dot chen at intel dot com Draft of 29 March 2005 INTRODUCTION Safety is an important goal for Systemtap. Both performance tool users and kernel debuggers are better served by a tool that protects them from system crashes and unwanted side-effects. Safety is a key design point of tools to which Systemtap will be compared, including DProbes and DTrace. This paper defines the relevant safety problems for Systemtap, how the system will address them, and how Systemtap safety compares to safety of other systems. Being a great kernel debugging tool means Systemtap needs to provide these capabilities to users who really want them: - reads of any kernel or user memory - writes to any kernel memory - calls to any kernel subroutine We need to design Systemtap such that these capabilities are conveniently accessible for kernel debuggers, while not interfering with the safety needs of performance tool users. Our intention is to make Systemtap a very safe tool to use, such that performance tool users with a range of skill levels can use it without accidentally crashing, disabling or otherwise negatively impacting their systems. While safety is an important goal, securing the system against malicious users is not. We note that most uses of systemtap require root priviledges on the monitored machine. As such, scenarios that involve deliberate misuse of Systemtap, or where the security of the system has already been compromised are out-of-scope. While features that contribute to security are certainly welcome, creating a secure system is not an goal of this work. The rest of this paper is organized into the following sections: - PROBLEM DEFINITION. Reviews specific safety issues that were considered in the Systemtap design. Also provides a high-level overview of Systemtap architecture. For more detail on the architecture see the Systemtap architecture paper. - DESIGN OPTIONS. Reviews the various components of the Systemtap architecture that might be used to achieve various safety goals. - PROPOSED DESIGN. For each safety issue, describes how Systemtap safety features will address it. Also describes system defaults, Guru mode, and Safe mode. - COMPARISON TO OTHER SYSTEMS. Identifies safety advantages and disadvantages of Systemtap as compared to other systems. - FUTURE WORK - Appendix: Usage Models. Provide motivation for Safety Goals and a yardstick for evaluating proposed designs. PROBLEM DEFINITION This section reviews the safety issues that were considered as a part of the Systemtap design. We assume that Systemtap supports a "guru" mode that enables a set of potentially unsafe capabilities for advanced users. This mode is disabled by default, to discourage unnecessary use of guru features. Disabling guru features by default is analogous to how UNIX users are generally discouraged from working with root priviledges when the task at hand can be accomplished with normal user priviledges. When not in guru mode, Systemtap should protect users from the following safety issues: - infinite loops. Systemtap should limit looping constructs such that they cannot continue to loop beyond some reasonable limit. A script with an infinite loop should fail with a useful error message and otherwise cause no ill-effects to the system. - infinite recursion. Systemtap should limit recursive procedure calls, both direct and indirect, to prevent them from recursing indefinitely. A script with infinite recursion should fail with a useful error message and otherwise cause no ill-effects to the system. - division by zero. An attempted division-by-zero in a Systemtap script should cause a graceful failure of the script, with a useful error message and otherwise no ill-effects to the system. - array bounds errors. An attempt to reference an array with an invalid reference should result in a graceful failure. - invalid pointer errors. An attempt to dereference an invalid pointer should either be impossible or result in a graceful failure. - heap memory bugs. These include memory leaks, stale pointers, etc. These should be impossible or result in a graceful failure. - illegal user-mode instructions. Systemtap scripts should not attempt to execute machine instructions that are disallowed in the kernel. Typical examples are floating-point and vector operations. - priviledged instructions. Systemtap scripts should not attempt to execute priviledged kernel instructions. Typical examples are the HALT instruction and instructions that manipulate the hardware machine state. We acknowledge that the Systemtap runtime might provide runtime subroutines for executing priviledged instructions. - memory read restrictions. If a script is invoked on behalf of an unpriviledged user, it would be desireable to prevent the tap from reading memory for a process belonging to different user. [Note: this is a security feature, not a safety feature.] It would also be desireable to disallow reads from volatile memory where reading has side-effects such as a mapped I/O device. There may be other cases where read restrictions would be useful. - memory write restrictions. A Systemtap script should not be able to induce system side-effects by writing kernel or user memory. - restrictions on invoking kernel code. A Systemtap script should only be able to invoke kernel subroutines included in a predefined Systemtap runtime API. It should not be possible to invoke arbitrary kernel routines directly from a Systemtap script. - version alignment. Attempts to invoke a Systemtap script with the wrong kernel, API, or other version mismatch should fail gracefully. [IS ANYTHING MISSING FROM THIS LIST?] The data and code reference restrictions described above would impact only references outside the linked tap. References to code and data within the tap should proceed without checking. When executing in "guru" mode, many of the above restrictions would be lifted. Systemtap might also have a "safe" mode that would provide additional safety restrictions and safety checks. DESIGN OPTIONS Before suggesting some plans to address the above safety issues, we review the system components which may be of use towards this goal. - Language Design. Although the Systemtap language will borrow from familiar languages like C and AWK, but might exclude or restrict some constructs such as code pointers, GOTO, and heap-allocated memory. - Elaboration. The elaborator might generate code to implement safety checks or support downstream safety features while translating from the Systemtap language to C. - Compilation, Linking. Systemtap will use standard C compilers and Linux linkers. We will assume these tools are generally free of bugs that would impact Systemtap safety. The use of popular, standard tools increases the likelyhood that they will not be too buggy. - Systemtap command. The systemtap command may implement some consistency or version checks before loading a compiled tap into the kernel. - Loading. Systemtap will use the insmod system call to dynamically load a binary module into the kernel. insmod may provide some consistency checks to prevent loading an invalid kernel module. - Runtime. The Systemtap runtime provides an opportunity to implement a number of runtime checks including version consistency, loop counters, and various kinds of protected operations. - Runtime portal. A runtime "portal" could be used to interpret restricted operations in a safe environment at runtime. The portal would be used only for code compiled from scripts; the runtime itself would be presumed trustworthy and run without runtime checks. The interpreter would make it possible to defer certain safety checks that are difficult to check statically, and to set safety policy at runtime rather than at tap compilation. There are several useful policies that are simple to implement, including: - guru mode: all writes and code references allowed - default: no external writes or code references allowed Other policies might be added as we accumulate experience with systemtap: - unpriviledged mode: restrict memory reads based on UID - device safety mode: no reads to volatile memory - profile guru mode: write permission extended to kernel data structures that maintain sample-based profile data. Priviledged instructions for manipulating hardware performance collection state also allowed Although we might have a hard time figuring out how to implement such policies initially, time should help us understand better if they are useful and how they might be interpreted. - Static Checker. A static checker could provide additional safety checks based on a static analysis of a disassembled module prior to execution, providing some resistance to design errors, user errors, and bugs in the tool chain used to create a tap. In general, memory reference checks are difficult to implement and other checks are more straight-forward. PROPOSED DESIGN Given the above safety issues, here's a proposal for how to address them in Systemtap: - infinite loops - The elaborator will add loop instrumentation to invoke a runtime loop counter/timer. - Assumption: The C compiler will not generate any unsafe loops on its own. - The runtime will terminate loops that run too long. The counter/timer limit might be specified as a part of a safety policy. - For safe mode, a static checker will confirm that all loops in code compiled from scripts are instrumented. - infinite recursion - Elaboration recognizes recursion and generates an error or a warning such as "recursion only in guru mode". - The elaborator will instrument all recursive calls. - For safe mode, the static checker will confirm that all recursive calls are instrumented - The runtime will disallow recursion or terminate recursion that continues for too long. The recursion limit might be set as a part of a safety policy. - division by zero - Elaboration recognizes division/modulus and implements it using a call-out to the systemtap runtime - No C division or mod operations will appear in the code generated by the elaborator, so there will be no division instructions in the resulting machine code. [ARE WE SURE ABOUT THIS?] - Assumption: No tap will be linked against a module that includes unprotected division operations. [CAN WE ACTUALLY PREVENT THIS?] - Assumption: The compiler will never include a feature/bug that would cause it to generate unsafe division. - The systemtap runtime division subroutine will perform a zero check before performing division. - Optionally, the static checker will validate non-existance of division instructions in code compiled from scripts. - array-bounds errors - Elaborator will emit explicit array-bounds checks for all array references. - If an array-bounds error occurs during script execution, an error handling routine provided by the runtime will report the error and exit. - Systemtap might provide a way to disable array bounds checks for guru mode. - invalid pointer dereferences - The Systemtap language will not support pointer variables or refs. In this way invalid pointers will not be possible for memory objects created by Systemtap. - Pointers references to external memory are covered below under "memory read and write restrictions." - heap memory bugs - The Systemtap language will not support dynamically allocated heap memory. In this way heap memory bugs will not be possible for memory objects created by Systemtap. - illegal kernel instructions - Illegal kernel instructions typically include floating point arithmetic, system calls, MMX, SSE or other vector operations. - The systemtap language will not support inline assembler - The systemtap language will not support any constructs that compile to illegal kernel instructions - Systemtap will invoke the compiler in such a way that it will not generate illegal kernel instructions from input source code. - Assumption: No tap will be linked against a module that includes illegal kernel instructions. [CAN WE ACTUALLY PREVENT THIS?] - Optionally, the static checker will validate non-existance of illegal kernel instructions. - priviledged instructions - The systemtap language will not support inline assembler - The systemtap language will not support any constructs that compile to priviledged instructions - Assumption: the compiler will generate priviledged instructions from C source code. - Assumption: No tap will be linked against a module that includes priviledged instructions. [CAN WE ACTUALLY PREVENT THIS?] - Optionally, the static checker will validate non-existance of priviledged instructions. - memory read and write restrictions - The Systemtap language/runtime can reference any kernel data structure, and can read from user memory. - The Systemtap language implementation might reject kernel and user memory data structures as l-values and as procedure call arguments so that disallowed writes can be detected at compile time. - The elaborator will recognize references to kernel data structures and user memory, directing them through the portal. - Memory references from the Systemtap runtime will be trusted and not require checking or redirection through the portal. - The Systemtap runtime memory portal will apply the current safety policy. The portal might provide a null pointer check, or leave the trap handler to catch these failures. - When Systemtap runtime API routines read or write via procedure call parameters, they should use the portal to perform these memory accesses, so that the current memory policy is applied. - The kernel trap handler will provide for graceful failures from illegal reads and writes, including null pointer dereferences, from Systemtap scripts. - Optionally, the static checker will confirm that all external memory references use the runtime portal. This check will be difficult to implement and might prove to be impractical. - restrictions on invoking kernel code - The Systemtap language will not support code pointers - The language/elaborator will be designed such that external calls to are recognized during elaboration. - Kernel calls from compiled scripts will be directed by the elaborator through the kernel call portal. - Kernel calls from the Systemtap runtime will be assumed safe and not checked. - At runtime, the kernel call portal will apply the current safety policy. - The kernel trap handler will provide graceful failures for illegal code refs from a tap. - Optionally, the static checker will confirm that all external code references are directed through the runtime portal. - version alignment - The Systemtap runtime will do a complete check for version alignment before running any script, and will fail gracefully if versions don't match. - The insmod system call might do some of this checking for us. - Optionally, the static checker might also provide some version checks. This would be useful in cases where insmod might fail with an error message that is hard to interpret. Given the above implementation plan, the Systemtap "guru" mode code be implemented as follows: - safety policy allows all read, write, and code references - safety policy disables loop iteration and recursion limits - Priviledged insructions can be invoked using routines provided by the Systemtap runtime. Similarly, "safe" mode could be implemented as follows: - safety policy disallows all external write and code references from compiled scripts - safety policy enforces loop iteration and recursion limits - a static checker is applied prior to attempts to load kernel module to provide redundant checks against some errors - Runtime routines for executing priviledged instructions will be disabled COMPARISON TO OTHER SYSTEMS Solaris DTrace includes a number of unusual features intended to enhance the safety and security of the system. These features include: - very restricted scripting language. The D language does not support procedure declarations or a general purpose looping construct. This avoids a number of safety issues in scripts including infinite loops and infinite recursion. - interpreted language. The D scripts compile to a RISC abstract machine language that executes in an interpreter embedded in the Solaris kernel. Because D scripts are interpreted rather than executed directly, it is impossible for them to include illegal or priviledged instructions or to invoke code outside of the DTrace execution environment. The interpreter can also catch invalid pointer dereferences, division by zero, and other run-time errors. Features such as these enhance the perceived safety of DTrace. Systemtap will support kernel debugging features that DTrace does not, including ability to write arbitrary locations in kernel memory and ability to invoke arbitrary kernel subroutines. Since the language infrastructure used by Systemtap is common to all C programs, it may be better tested and more robust than the special-purpose infrastructure used by DTrace. The embedding of an interpreter in the Solaris kernel represents significant additional kernel functionality. This introduces an increased risk of kernel bugs that could lead to security or reliability issues. Like DTrace, DProbes uses an interpreted language. [would an DProbes expert like to finish this part?] FUTURE WORK We believe the initial safety design for Systemtap gives us adequate basic safety for most performance tool users. Going forward we see a number of opportunities for enhancing this basic safety model: - refinement of runtime policies. We could implement non-trivial data and code reference policies - static checking of memory reads and writes. This problem is likely to be left unsolved in the initial system release but should be resolveable in later releases. - security. There is much work to do to understand the relevant security threats for Systemtap and design appropriate solutions. - crashproof. We believe that Systemtap as designed should be extremely crash resistant. Experience with the system will help us recognize and remedy the most important outstanding vulnerabilities. APPENDIX: USAGE MODELS This is an attempt to catalogue common scenarios pertaining to safety and security. M1: kernel version mismatch M2: systemtap API version mismatch M3: systemtap runtime version mismatch Script was compiled on an old version of the system, then the system is upgraded, and is not inconsistent with current APIs, symbols, etc. M4: Wrong binary In the heat of troubleshooting, a user specifies the wrong binary to load into the kernel. Assume the user is root and the binary is not deliberately malicious but otherwise arbitrarily bad. Specific cases might include: a: binary includes priviledged instructions b: IA32 binary includes illegal ring0 instructions c: binary writes randomly into kernel memory M5: buggy script A user who is learning to use Systemtap writes many buggy scripts. Can the we guarentee that the system will never crash? Consider memory leak, recursion, infinite loop, stale pointers, bogus pointers, jumps into arbitrary locations in kernel code, etc. M6: a bug in the compiler, translator, runtime Does the system crash or just print a warning message? M7: Remote compilation of script. Compiler not installed on production machine. a: binary includes priviledged instructions b: IA32 binary includes illegal ring0 instructions c: binary writes randomly into kernel memory Some of these cases might come up if a kernel debugging script were confused with a performance script. M8: 3rd party script as a binary form M9: 3rd party script as a source file A Systemtap user downloads a script from a public bboard or web site. The script sounds terribly interesting but the user is not completely confident in the source. How worried does he/she need to be about trying it? M10: malicious attack: corrupted kernel Probably nothing much you can do here. M11: malicious attack: corrupted compiler, corrupted script- launching system We do our users a favor by minimizing the number of software components they need to trust or secure in order to make the overall system secure. M12: malicious attack: bad binary script. Could have a stack overwrite, code hidden in data segments, who knows... Here's a few non-safety usage scenarios, which I'm listing here but which aren't otherwise considered in this document. M13: Fault injection M14: Patching kernel data M15: Patching kernel code