More SystemTap Thoughts

I've read the SystemTap architecture paper. My initial reactions, which might be muddied by misunderstandings, misreadings, general ignorance, etc.:

1. Every script execution invokes the GCC toolchain to produce a kernel module, then loads that module, executes for a time, and unloads the module. This makes implementation more tractable, because the SystemTap folks don't have to write a different back-end for every CPU, nor do they have to define a little bytecode VM for user-level to communicate with the kernel, as DTrace did. However, this compile-link-load cycle may take a user-perceptible chunk of time, especially if a script is invoked repeatedly from some other script. Worrying about this sort of incremental friction might seem like premature optimization, but when you rely for runtime performance on a big piece of software that is not optimized for runtime performance, such as the GCC compile/link cycle (which, after all, is optimized to produce fast binaries, not to produce binaries fast), you're throwing away a lot of flexibility right out of the gate.
   
2. I'm not in love with the language. Like D, it uses C's expression syntax. However, unlike D, it doesn't use C's type syntax. This isn't just an inconvenience. D scripts can #include header files right out of a source tree, or the kernel, or wherever, and can use those types in a natural way. This can be very helpful when instrumenting a C application.
   
3. On the other hand, when not instrumenting a C application, the architecture doesn't seem to anticipate external sources of probe points. They discuss being able to probe user-level applications, but only in terms of tracing specific program counters. This doesn't always make sense. If, e.g., the target application is a scheme interpreter, the programmer will want to interact with his program's control flow in terms of source-level function entry/exit, rather than random program counters within the interpreter. While the core functionality of SystemTap can be extended via "TapSets," it sounds like these tapsets are stuck on the wrong side of the application being probed to do this sort of thing well. (I.e., instead of the scheme interpreter publishing a semantic interface to its internals, the TapSet has to contain enough knowledge about the scheme interpreter to reverse engineer its current state.)
   

This last point, if I understand it right, is really unfortunate. It basically limits SystemTap's full powers to the kernel; applications can only be instrumented at the machine-language level. Some of the more powerfully convincing DTrace demos involve following an execution path all the way from application-level into the nittiest-grittiest kernel guts. The ability to telescope from the ethereally high level of a scripted language all the way down to the kernel grovelling around in the APIC and back again is one of the more exciting things about DTrace. I hope the SystemTap folks haven't given up on achieving this for Linux.

Or, of course, maybe I'm just not getting it. Perhaps some SystemTappers out there can set me straight?