What is VProbes' relationship to DTrace?

VProbes has a similar set of design goals to DTrace: to be safe-yet-programmable, and be of global scope. DTrace is permissively licensed open source, that pre-dates the existence of VProbes, so why in heaven's name don't they share code?

The short version: virtual machine monitors aren't kernels, even though they run at CPL 0. Those curious about the hair, though, please read on...

Before breaking ground on what would eventually become VProbes, I investigated a straight port of DTrace. I went so far as meeting with VMware legal to figure out whether the CDDL said what it appeared to say ("steal this code and make a fortune selling it for all we care, as long as you're not Linux"). However, there are some daunting impedance mismatches between DTrace as it exists today and the world in which our VMM must live. An incomplete list of gotchas includes:

1. Our VMM needs to fit in 4MB of virtual address space, about 3 of which are needed for the translation cache. So, we care passionately about code and data size in ways that would be simply moronic for any other software project. Our internal mailing list regularly has knock-down, drag-out arguments about changes that bloat the monitor by 700 bytes; unless you're developing firmware, you should not even waste the time worrying about anything that impacts real-world memory usage by less than a MB or so.
   
   VProbes' current runtime impact is around 50KB, and we frankly consider it slightly overweight. Perhaps DTrace's data structures could be trimmed to fit in this environment, but text is more challenging. That's not to imply that DTrace should be tinier; DTrace is bigger for all the right reasons, mostly that it does more than VProbes does.


2. Our VMM does not provide any dynamic allocation facilities. This is one of the major reasons that efforts to port third-party software, even kernel software, to the VMM fails. Reading the BigAdmin discussion, DTrace appears to self-tune many of its buffers' capacities automagically, which is great, and makes DTrace better for its users; however, it's just not feasible for our constrained environment.


3. We care about applications where probes are always enabled. VMware's VMM doesn't have any third-party extension mechanism, safe or otherwise, at present. VProbes provides a few toeholds for programmatic extension. This "extension" use of VProbes is different from the "monitoring" use that it shares with DTrace. It drives performance trade-offs that would simply be inappropriate for DTrace. E.g., rather than interpret bytecodes, VProbes compiles user probes to x86-64 machine code. Assuming you're on an x86 makes perfect sense for VMware, and would be completely insane for a system that also needs to work on sparc.

Of course, DTrace, being software, is infintely malleable, and you could probably start with a tarball extracted from the OpenSolaris project and eventually end up with something that satisfies all of the constraints our VMM imposes. But the result would be ... un-DTrace-like. Suppose you contort DTrace so that it doesn't care whether interrupts are enabled, sacrifices some speed, flexibility and features so that it fits in less than 100KB of virtual address space, and picks (tiny) static buffer sizes. How interested would the upstream DTrace developers, and ports of DTrace to other kernels, be in integrating my diffs? How hard would it be to integrate the stream of bug-fixes from DTrace-at-large into our heavily hacked code base? Would DTrace's owners even consider bugs that only arose in my insanely contorted set-up to be "bugs"? It seemed we'd be a dead-end fork of DTrace, neither able to help the parent project, nor selfishly benefit from its further development.

Some have said that rolling our own runtime is an instance of not-invented-here syndrome; I take the charge seriously, and can see how it would seem that way from the outside. I strongly disagree. I realize that good work happens outside VMware, and consider DTrace shining example of such. I agonized over this decision, not least because I respect Bryan, Mike and Adam both as engineers and human beings. I am no Torvalds-esque DTrace hater. There are things that DTrace will continue to be hugely superior at. Then again, not having to wait for Microsoft to port DTrace to Windows is pretty neat too...