tag:blogger.com,1999:blog-176916672024-03-07T19:57:13.614-08:00Gimme Hardware/Software Interface.Computers, often from a low-level systems perspective. Note that I speak for myself, not my employer.Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.comBlogger63125tag:blogger.com,1999:blog-17691667.post-34935665492896829022009-07-14T10:13:00.001-07:002009-07-14T10:29:37.106-07:00Ch-ch-changes.As some of you know, I'm no longer at <a href="http://www.vmware.com">VMware</a>. It feels a bit like leaving home. VMware took a chance on me as a new college grad way back in 2000, and taught me a lot along the way. I will miss my colleagues, many of whom are excellent, and will always be proud of what we accomplished together. I expect VMware will continue to be successful both technically and as a business, and they treated me very well. I did not leave due to a long list of complaints; it simply had been nine years, and I had decided it was time to try something different.<br /><br />I've been waiting to update this blog until I had a clear idea of what that "something different" was. My long fast in the woods is over: I'm working in the search team at <a href="http://www.facebook.com">facebook</a>, trying to make a whole lot of data available to a whole lot of users. You might think that the title of this blog is less relevant than it once was; search sounds, initially, like an application, pretty high up the abstraction stack from the hardware. This is true, but search is also a special application, enormously demanding in computational resources. We're pushing the hardware close to its limits in several dimensions simultaneously, and some knotty systems issues arise. I'm not ready to talk about them in any depth yet, not out of secrecy, but ignorance. I'm still learning the ropes here, and any attempt to drop science at this point would just be silly.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com3tag:blogger.com,1999:blog-17691667.post-10388167359112816662009-02-13T09:16:00.000-08:002009-02-13T09:21:26.216-08:00Irfan's blogging.My colleague over in VMkernel Irfan Ahmad has <a href="http://virtualscoop.org/">a blog</a>. Irfan often has good ideas, and is a clear and effective writer; if you find the content here interesting, you might enjoy him. I've added a link to this blog's marginalia.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com0tag:blogger.com,1999:blog-17691667.post-44457902183543029272009-02-10T13:57:00.000-08:002009-02-10T13:59:42.185-08:00VProbes community forumVMware is now hosting a <a href="http://communities.vmware.com/community/developer/vprobes">community forum for VProbes</a>. Come party with us.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com0tag:blogger.com,1999:blog-17691667.post-38880053470972372982008-11-04T15:32:00.000-08:002008-11-04T17:08:47.462-08:00Cantrill and Bonwick get all concurrent-y up in there...<a href="http://blogs.sun.com/bmc">Bryan Cantrill</a> and <a href="http://blogs.sun.com/bonwick">Jeff Bonwick</a> of Sun Microsystems co-authored an excellent <a href="http://doi2.acm.org/1454456.1454462">position piece on concurrency</a> in the ACM Queue. It's about time someone stood up to those short-selling concurrency bears. Take heart, young coders! Correct concurrent software does appear out there from time to time, no matter what your grad TA tells you!<br /><br />The section on hard-won lessons from developing concurrent software warmed my heart; VMware's virtual machine monitor team has learned the same lessons, and I for one can endorse their list wholeheartedly.<br /><br />With any legit publication like this, the authors struggle to balance completeness and concision, and some things did not make the cut. So, don't take it as criticism that I would add just one item to their list, that has powerfully increased our ability to ship concurrent code (drumroll): <b>lock ranks</b>. Huh? Lock ranks! We have imposed an explicit, compiled-in partial order on the locks in our system. Acquiring locks out of order causes a runtime failure in debug builds. I've seen some engineers regard this discipline as a sort of "training wheels" for concurrency, but it has benefits even for seasoned vets.<br /><ol><br /><li>Ranks force developers to think about layering <i>before</i> writing synchronization code. Will this subsystem be touching guest memory? Will it require synchronous remote execution on other CPUs? Will the user-level portion of our software ever need to acquire this lock? Etc.<br /><li>This has strangled many a rare deadlock in the crib. I agree with Bryan and Jeff that, in practice, a deadlock is not too hard to debug post-mortem; however, post-mortem is too late if the core comes from a customer site. A rank violation is much, <i>much</i> easier to catch in testing than a deadlock, since it only depends on the single-threaded path leading an order violation, as opposed to a concurrent race. This performs the magic trick of reducing a concurrent QA problem (exercise every lock interleaving) to a serial QA problem (cover every path that acquires a lock).<br /><li>The same discipline can be applied to other synchronous services in the system. For instance, like many other concurrent systems, our VMM has a "crosscall" primitive, which lets a caller synchronously run code in a remote VCPU. This can lead to lock-like dependency cycles, where, e.g., VCPU 0 holds lock A and crosscalls VCPU 1 which is blocked waiting for A. To prevent this, we encorce the rank metaphor for crosscalls, too; a crosscall while holding dangerous locks fails just as a misorder lock acquisition would.<br /></ol><br /><br />While the rank discipline was initially adopted for deadlock avoidance, the structure it imposes on the system is even more valuable. The lock ranks, in isolation, provide a succinct description of the system's organization. If you take the partial order of the lock ranks, associate them with their controlling subsystems, you can construct a "wedding cake" chart of the system automatically. I.e., if subsystem A relies on subsystem B, then locks protecting A's data will always have lower rank than B's. The top of the cake are "pure" clients, which are never entered from other subsystems; the tippy-top layer of frosting are the handful of points that we enter the VMM from the guest, where we always know that no VMM locks are held. The bottom of the cake, corresponding to the highest ranked, "leaf" locks, from which no other code is reachable, correspond to the most basic, omnipresent life-support facilities: logging, deferring work to a less pressing time, panic'ing, etc.<br /><br />Bryan advertised the article in an excellent (though more in-character) <a href="http://blogs.sun.com/bmc/entry/concurrency_s_shysters">blog post</a>. Out here in the wild wild web, Bryan lets the invective against transactional memory advocates ("shysters", in his estimation) flow a bit more freely. I am not sure the hype surrounding TM is malicious; I suspect it is your garden-variety, low-grade academic over-enthusiasm, born partly from needing to exaggerate in grant proposals and paper abstracts, and partly from honest optimism. Think "microkernels," not "<a href="http://www.quackwatch.org/01QuackeryRelatedTopics/Cancer/laetrile.html">laetril</a>."<br /><br />Hey, remember microkernels? For about a decade, systems journals were dominated by papers that <i>took for granted</i> that microkernels would dominate general purpose OS'es. This religious belief that running parts of the operating system in separate address spaces would make everything easier and better begat all sorts of meaty sub-problems, like doing fast synchronous cross-address-space RPC, and building little IDL compilers that provide binary compatibility for clients when driver binaries change and ... so on. I suspect TM will take a similar course, dying not with a bang, but a whimper, and ten years hence.<br /><br />It's interesting to me that, as with microkernels, one of the principle reasons TM will fail is the messy, messy reality of peripheral devices. One of the claims made by microkernel proponents is that, since microkernel drivers are "just user-level processes", they'll survive driver failures. And this is almost true, for some definition of "survive." Suppose you're a microkernel, and you restart a failed user-level driver; the new driver instance has no way of knowing what state the borked-out driver left the actual, physical hardware in. Sometimes, a blind reset procedure can safely be carried out, but sometimes it can't. Also, the devices being driven are DMA masters, so they might very well have done something horrible to the kernel even though the buggy driver was "just a user-level app." And if there were I/Os in flight at failure time, have they happened, or not? Remember, they might not be idempotent... I'm not saying that some best-effort way of dealing with many of these problems is impossible, just that it's unclear that moving the driver into userspace has helped the situation at all.<br /><br />Similarly, peripherals are a gotcha for TM, because, as Bryan and Jeff point out, they're <i>not memory</i>. There will never be a way to make TM transactions persist across system calls, unless you find a way to <i>un</i>transmit network packets. That means that TM is now <i>and always will be</i>* useful only for raw computation: AVL trees, mp3 encoding, sort, and whatnot. Those are important facilities, but they're also the sort of thing that are coded once, by an expert, and used forever thence from a library, so "ease of programming" is not the highest priority. Even if "ease of programming mp3 encoders" is your metric, it's unclear that TM is an across-the-board win. What happens when you set a debug watchpoint on memory that's accessed transactionally? Oh, sorry! Didn't mean to spoil the punchline from someone's PhD thesis...<br /><br />Not so simple anymore, eh?. If you can remember way back to your undergrad days, locks probably seemed easy at first, too.<br /><br />* I'm exhausted with that weasel phrase "current TM implementations"; transactions will never, <i>bloody ever</i> span I/Os, and that's the relevant limitation.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com1tag:blogger.com,1999:blog-17691667.post-13689874162466176012008-10-08T10:00:00.000-07:002008-11-04T15:29:04.888-08:00Done projecting. Time to reflect?Well, I've been back from U of I for over a month, so it's more than about time I got to writing it up. Interacting with people who are just as passionate about computing, but looking at it from completely different perspectives, is a soothing tonic for a systems person. It reminds me that these systems we build don't just blink in datacenters anonymously; when they function right, they do real, useful things that <i>matter</i> deeply to others.<br /><br />I think my talk went fine, as things go. If I'd had one thing to do over, I would probably not have pounced so furiously on Crispin Cowan's Q&A flamebait about how everybody knows VMs are slow; Crispin, that's one for you, there.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com0tag:blogger.com,1999:blog-17691667.post-47232985833275868772008-10-03T07:56:00.000-07:002008-10-03T08:00:17.541-07:00Reflections|Projections 2008I'm going to be at <a href="http://www.acm.uiuc.edu/conference/2008/speakers">Reflections|Projections</a> 2008 this weekend. I'll be talking about the role virtual machine monitors can play in systems research, using VProbes as both an example and exploratory tool. It's an honor to have my name on the same web page as some of my fellow speakers. Curious parties should swing on by UIUC's1404 Siebel at 4pm Saturday.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com3tag:blogger.com,1999:blog-17691667.post-64224773142336752022008-08-01T15:55:00.000-07:002008-08-01T16:42:47.257-07:00Why I am not a C++ programmer: ten years on.As a far-too-full-of-myself CS undergrad, I wrote, <a href="http://web.archive.org/web/20010122104900/www.cs.brown.edu/~kma/cplusplus.html">"Many C++ critiques are available online, but none of them address what I regard as C++'s fundamental failure..."</a> Man, it's painful linking the above. I wrote it ten years ago, as a <a href="http://www.cs.brown.edu/courses/cs169/asgn.shtml">TA</a> for a course that had previously been taught in C++. Despite trying <i>way</i> too hard to project an unearned air of worldly cool, in most ways I stand by my claims. C++ is a broken notation: its attempts at being high level make it inappropriate for serious systems work, and its attempts at being low level make it even worse for applications. Incidentally, it has a truly unfortunate culture attached to it, rife with books, tutorials, even conferences which purport to be about "design," but are in fact about about "delegation," "factories," etc., and Cartesian products thereof. None for me, thanks.<br /><br />Why the stroll down memory lane? Because today a coworker introduced me to <a href="http://yosefk.com">Yossi Kreinin's</a> <a href="http://yosefk.com/c++fqa/defective.html">C++ Frequently Questioned Answers</a>. I can no longer claim that "none of [the C++ critiques online] address ... C++'s fundamental failure," as Yossi's kills them all. Folks, we are in the presence of greatness. I feel as Salieri must have upon first encountering the works of Mozart.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com2tag:blogger.com,1999:blog-17691667.post-62463527238309070322008-07-24T12:45:00.000-07:002008-07-24T13:46:51.047-07:00What is VProbes' relationship to DTrace?<a href="http://x86vmm.blogspot.com/2008/06/vprobe-demo.html">VProbes</a> has a similar set of design goals to <a href="http://www.sun.com/bigadmin/content/dtrace/">DTrace</a>: to be safe-yet-programmable, and be of global scope. DTrace is permissively licensed open source, that pre-dates the existence of VProbes, so why in heaven's name <i>don't</i> they share code?<br /><br />The short version: virtual machine monitors aren't kernels, even though they run at CPL 0. Those curious about the hair, though, please read on...<br /><br />Before breaking ground on what would eventually become VProbes, I investigated a straight port of DTrace. I went so far as meeting with VMware legal to figure out whether the CDDL said what it appeared to say ("steal this code and make a fortune selling it for all we care, as long as you're not Linux"). However, there are some daunting impedance mismatches between DTrace as it exists today and the world in which our VMM must live. An incomplete list of gotchas includes:<br /><br /><ol><li>Our VMM needs to fit in 4MB of virtual address space, about 3 of which are needed for the translation cache. So, we care passionately about code and data size in ways that would be simply moronic for any other software project. Our internal mailing list regularly has knock-down, drag-out arguments about changes that bloat the monitor by 700 bytes; unless you're developing firmware, you should not even waste the time worrying about anything that impacts real-world memory usage by less than a MB or so.<br /><br />VProbes' current runtime impact is around 50KB, and we frankly consider it slightly overweight. Perhaps DTrace's data structures could be trimmed to fit in this environment, but text is more challenging. That's not to imply that DTrace should be tinier; DTrace is bigger <i>for all the right reasons</i>, mostly that it does <i>more</i> than VProbes does.</li><br /><li>Our VMM does not provide any dynamic allocation facilities. This is one of the major reasons that efforts to port third-party software, even kernel software, to the VMM fails. Reading the BigAdmin discussion, DTrace appears to self-tune many of its buffers' capacities automagically, which is <i>great</i>, and makes DTrace better for its users; however, it's just not feasible for our constrained environment.</li><br /><li>We care about applications where probes are <i>always enabled</i>. VMware's VMM doesn't have any third-party extension mechanism, safe or otherwise, at present. VProbes provides a few toeholds for programmatic extension. This "extension" use of VProbes is different from the "monitoring" use that it shares with DTrace. It drives performance trade-offs that would simply be inappropriate for DTrace. E.g., rather than interpret bytecodes, VProbes compiles user probes to x86-64 machine code. Assuming you're on an x86 makes perfect sense for VMware, and would be completely insane for a system that also needs to work on sparc.</li></ol><br />Of course, DTrace, being software, is infintely malleable, and you could probably start with a tarball extracted from the OpenSolaris project and eventually end up with something that satisfies all of the constraints our VMM imposes. But the result would be ... un-DTrace-like. Suppose you contort DTrace so that it doesn't care whether interrupts are enabled, sacrifices some speed, flexibility and features so that it fits in less than 100KB of virtual address space, and picks (tiny) static buffer sizes. How interested would the upstream DTrace developers, and ports of DTrace to other kernels, be in integrating my diffs? How hard would it be to integrate the stream of bug-fixes from DTrace-at-large into our heavily hacked code base? Would DTrace's owners even consider bugs that only arose in my insanely contorted set-up to be "bugs"? It seemed we'd be a dead-end fork of DTrace, neither able to help the parent project, nor selfishly benefit from its further development.<br /><br />Some have said that rolling our own runtime is an instance of not-invented-here syndrome; I take the charge seriously, and can see how it would seem that way from the outside. I strongly disagree. I realize that good work happens outside VMware, and consider DTrace shining example of such. I agonized over this decision, not least because I respect Bryan, Mike and Adam both as engineers and human beings. <a href="http://slashdot.org/~kma/journal/">I am no Torvalds-esque DTrace hater</a>. There are things that DTrace will continue to be hugely superior at. Then again, not having to wait for Microsoft to port DTrace to Windows is pretty neat too...<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com2tag:blogger.com,1999:blog-17691667.post-85323791307113807112008-06-13T15:27:00.000-07:002008-06-13T15:33:50.213-07:00VProbe demo<a href="http://blip.tv/file/950760">Can you tell I'm not used to giving talks sitting down</a>? Sorry about the continual side-to-side oscillation; I promise to insist on standing up in the future.<br /><br />If this VProbe stuff looks fun to you, perhaps it's time<a href="http://www.vmware.com/communities/content/beta/ws65/registration.html">you got beta'ed up</a>?<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com0tag:blogger.com,1999:blog-17691667.post-68786630764798082008-06-13T10:04:00.000-07:002008-06-13T11:30:30.378-07:00Windows Host Quoting Gotcha, and the VProbe ToolkitWorkstation 6.5 is now in <a href="http://www.vmware.com/communities/content/beta/ws65/registration.html">public beta</a>. It includes experimental support for VProbes in the guest, virtual machine monitor, and a few spots in the virtual hardware. We'd like to invite anybody curious about vprobes to give it a whirl.<br /><br />A common problem came up on the <a href="http://communities.vmware.com/message/971080#971080">forums</a> yesterday: if you're using VProbes from Windows' native command-line, you'll soon notice that the following:<br /><pre>C:\> vmrun vprobeLoad foo.vmx '(vprobe VMM1Hz (printf "greetings windows\n"))'</pre>produces the cryptic error message:<br /><pre>vprobeLoad: error: unknown ident windows\n<br />vprobeLoad: 0 warnings, 1 errors</pre>What gives? In short, cmd's weird quoting. As far as I can tell, single quotes are basically the same as double-quotes, except cmd will only nest like-with like, so cmd ends up passing along the string '(vprobe VMM1Hz (printf greetings windows\n))'. Check the forum post for the band-aid solution for cmd; I predict you'll only be able to tolerate the syntax for cmd one-liners so long before you give up and install cygwin.<br /><br />Once you embrace cygwin, you might as well go whole-hog and <a href="http://sourceforge.net/svn/?group_id=224179">get the vprobe-toolkit</a>. The abstraction level of vp is very low; while the parentheses might remind you of scheme for a while, pretty soon the semantics will remind you of assembly language. This was an intentional choice: since bugs in the vp compiler can nuke your system, we're motivated to keep it as simple as possible. While it's possible to use vp directly for machine-level things (e.g., instruction or stack profilers), for more complex tasks automatic code generation is more appropriate.<br /><span style="font-size:130%;"><br />Introducing Emmett</span><br /><br />Emmett is a small language that provides C-like type, expression, and conditional operators, some syntactic sugar for aggregation, and automatic inference of type for undeclared variables. The vprobe-toolkit provides a wrapper script, named "vprobe", that invokes the Emmett compiler on your input, loads the resulting VP into the target VM, and runs user-specified pretty-printers on your output. E.g., "vprobe -a" pretty-prints sorted aggregates, and "vprobe -s" pretty-prints stacks.<br /><br />Where in VP, you'd have to write:<br /><pre>(defaggr a 1 1)<br />(vprobe USEC:1001<br /> (do<br /> (aggr a (VCPUID) ((curprocname)) 1)<br /> (logaggr a )))<br /></pre><br />Emmett produces the more readable:<br /><pre><br />USEC:1001<br />{<br /> a[VCPUID, curprocname()]++;<br /> logaggr(a); # OK, this is a lot of output.<br />}<br /></pre><br /><br />There are a bunch of examples in the cookbook subdirectory.<br /><br />I want to emphasize that Emmett is a replaceable component. If you're a python programmer, and curly braces make your shoulder blades clench with rage, feel free to write your own little language...<br /><br /><span style="font-size:130%;">Pragmatics</span><br /><br />vprobe-toolkit is open source. We've licensed it with a variant of the BSD license, which VMware's lawyers assure me means you can basically do whatever you want with it; if you're feeling paranoid, though, <span style="font-style: italic;">please</span> run the license file past your local lawfolk rather than taking my word for it. You'll notice the link above is to a subversion repository, not a download page. In its current state, it would be inappropriate to expose it to those unwilling to hack a bit; if ws 6.5 is currently in "beta", this is probably somewhere past "prototype" but not really "alpha" yet. Join us!<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com2tag:blogger.com,1999:blog-17691667.post-30457032739946014512008-04-05T10:25:00.000-07:002008-04-05T15:03:20.328-07:00Herlihy & Shavit: The Art of Multiprocessor Programming(Disclaimer: I've taken courses from Maurice Herlihy at Brown.)<br /><br />So! Concurrency's the hot thing. <a href="http://blogs.intel.com/research/2007/06/multicore_processors_an_inflec.php">Intel</a> says so. The technical bookshelves are groaning with books about concurrent programming. Most of these books are occult thread package tutorials; they walk you through using Java's monitors, or pthreads' conditions and mutexes, or Windows' Events or what have you, often with the same shopworn collection of toy problems: philosophers producing elves who dine on ... eh.<br /><br /><a href="http://www.amazon.com/Art-Multiprocessor-Programming-Maurice-Herlihy/dp/0123705916">The Art of Multiprocessor Programming</a> is different. It is, in part, a theory book. Don't panic! The use of formal reasoning is tasteful; a teaspoon or so of actual math is necessary when designing concurrent programs, because our intuitions can fool us. However, the authors don't overwhelm the reader; an undergraduate level background in CS theory is not necessary to appreciate the book. If you can read the word "tuple" without giggling, you'll be fine.<br /><br />More importantly, though, the actual data structures and algorithms that are highlighted in this book are not the same-old. Sure, mutual exclusion is discussed, but it's with a focus on <i>implementing</i> mutual exclusion, which is a topic <a href="http://www1.uspto.gov/web/patents/patog/week40-old/OG/html/1311-1/US07117481-20061003.html">dear to me</a>. Most strikingly, the majority of the examples in the book are of lock-free data structures. You'll find lock-free linked lists, hash tables, trees, etc.<br /><br />In my day job, I treat lock-free concurrent data structures as, well, not <i>quite</i> black magic, but certainly Powerful Medicine. They take a lot of thought, bugs can hide in them for years, reasoning about their performance is non-trivial, even compared to locking, and frankly, not every programmer is up to maintaining such code; but when you need them, you need them. If the multi-core popular press is to be believed, more and more of us will be needing them in the near future. In as much as this is the first off-the-shelf collection of useful, debugged lock-free data structures, it's an invaluable contribution. Go buy it. Even if you never use a counting network, it will make you a better programmer. It's also the most writerly book about programming you'll read all year; Herlihy's deadpan humor is a welcome break from the super-serious tone programming books usual adopt.<br /><br />However. ...<br /><br />Reading between the lines a bit, I see this book as part of a larger project, which I refer to as "pop concurrency*". "PC" is a loose confederation of researchers in programming languages, compilers, architecture and operating systems whose implicit goal is to make concurrent programming, well, "easy." Much, much easier, anyway. Of course, sequential programming isn't all that easy. Before the current fashion in concurrency, the "software engineering" community was haranguing us about the expense, difficulty, and intractability of programming. We've made only incremental progress in making sequential programming easier; garbage collectors and safe languages are probably the most important contributions so far, ad they are not order-of-magnitude revolutions over traditional tools.<br /><br />It's cheap and easy to make negative predictions about ambitious research agendas. E.g., making fun of <a href="http://singinst.org/">strong AI people</a> stopped being fun for me about 10 years ago. But at the risk of seeming to make a cheap shot, I am really, really skeptical about how easy we can make concurrent programming. Even if the thicket of hardware and software problems around <a href="http://www.cs.wisc.edu/trans-memory/">transactional memory</a>** resolve, I suspect TM only <i>seems</i> easy because it's the devil we don't know. After all, locks sound easy too: just identify the shared data and wrap mutexes around it! Sounds simple enough, doesn't it? It took years, and lots of big systems being constructed this way before problems like priority inversion were framed, and many years thereafter before they were solved. Paraphrasing Fred Brooks, there's no silver bullet; concurrency is hard not because we're Western linear Cartesian dualist squares, or because of bad notation, but because it's <i>actually hard</i>. Like all software, the easy parts are quickly discovered and factored, and the code that's left to be written represents real problems to solve.<br /><br />* Warning: I made this term up. Do not use it when trying to sound intelligent.<br />** E.g.: how do you debug a TM program? When you're stepping through a transaction, won't it basically always fail? System calls during transactions? I/O? What if you've DMA'ed into or out of a piece of physical memory that's being transacted-upon? Smarter people than me are thinking about these problems, but man. They are non-trivial.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com3tag:blogger.com,1999:blog-17691667.post-76812185718094655022008-03-15T04:47:00.000-07:002008-03-15T06:06:24.661-07:00dtrace.conf(08)DTrace's (first?) unconference in San Francisco yesterday was a blast. I attempted a <a href="http://www.vmware.com/products/beta/ws/vprobes_reference.pdf">VProbes</a> demo with Thursday morning's Fusion bits from our development branch, which I would rate a qualified success. The VMware Fusion UI crashed right after plugging into the projector, but the VProbe stuff worked. In case it was unclear to anybody in attendance, yes, there really was a Windows VM struggling on in the background; sorry you couldn't see its screen. I'm told video will be up shortly, so stay tuned. In the meantime, I let slip that we're attempting to ship VProbes in Workstation 6.5; all of you ESX users who asked me about VProbes on ESX, I have no official news for you, but yes, we feel your pain.<br /><br />I enjoyed meeting like-minded folks, putting faces to names, and getting a glimpse of the DTrace community's roadmap. Having slept on it a bit, both of <a href="http://blogs.sun.com/bmc">Bryan's</a> demos stand out. <br /><br />As Bryan admits, distributed DTrace as implemented thus far is nothing that you couldn't do with ssh and a lot of elbow grease. But I'm excited about the implications of this project. A wire format for generalized aggregation could provide a <i>lingua franca</i> between DTrace and other sources of aggregation data (*cough* VProbes). You can actually imagine "aggregation" in this VProbes/DTrace sense overlapping with "aggregation" in the RSS sense: administrators "publish" streams of worthwhile data from individual systems, and trees of aggregators pool together broader and broader summaries of these aggregations by talking the same protocol upstream.<br /><br />Which brings me to dtrace.conf's well-planned climax, Bryan's graphical DTrace demo.<br /><br />VProbes' textual UI feels like a throwback to me; when showing it to a skeptic, you can almost read the thought balloon floating over his head: "OK, you just typed something weird into an xterm, and it spit something out. It's 2008. I wonder why he expects me to care. Actually, I wonder what the stock market's doing. Oh, wait, this VProbe guy is still talking." But what can I do? I'm a programmer, it's a programming language; how else do I demonstrate its power and more importantly, its flexibility? But people (quite correctly) expect data to be presented visually, because our visual systems are the most reliable, highest bandwidth way of getting quantitative data our brains.<br /><br />Most graphical tools, though, have only one trick: plotting some scalar data against time. It's amazing how little we've really improved past xload in this regard. Bryan's demo was the first graphical tool I've seen that captures some of the <i>combinatorial</i> and <i>iterative</i>power of a dynamic instrumentation system: finding an anomaly, then breaking it down by <i>foo</i>, then by <i>bar</i>, backtracking a bit because <i>foo</i> isn't as important as you initially thought, then keeping only the purple <i>bar</i>s, etc. Sure, there will always have to be something like a programming language underneath for full generality, but a tool like this has the potential to capture much of the benefit with much less investment of intellectual energy. We professional programmers tend to, err, overestimate others' passions for learning new languages.<br /><br />It took courage and generosity of spirit on the part of my hosts to provide a platform for what <i>could be</i> viewed as a competing technology at their conference. So, thanks guys. I would (and did) argue that VProbes and DTrace are complements. Users need visibility into the virtualization stack, and we're going to try to give them as much insight into that layer as possible with VProbes, but we're happy to cede higher levels to OS tools, including DTrace. Consider for example TCP. In the grand scheme of a real application, TCP is a very low layer, but from the virtual hardware's point of view, TCP is a modestly high-level protocol to reconstruct from what's sitting on the wire and in guest memory. DTrace will always provide superior visibility into what is really an OS level of abstraction. In my mind, the interesting question is how to weave together these different instrumentation engines for different levels in the stack. Hopefully we'll have something to show for our efforts at dtrace.conf(09).<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com2tag:blogger.com,1999:blog-17691667.post-33778090861561472832008-01-23T09:00:00.000-08:002008-01-23T09:15:24.463-08:00MacOS X/DTrace tryst ends in tears.See <a href="http://blogs.sun.com/ahl/entry/mac_os_x_and_the">Adam's break-up note</a>. This also got picked up on slashdot.<br /><br />My thoughts, in no particular order:<br /><ol><br /><li>Shame on Apple. What is the point of building system-wide infrastructure that any random process can opt out of? Forget about DRM; what about legitimate uses of DTrace for system health monitoring? Won't all malware be sure to set the "don't dtrace me" bit now?<br /><li>This highlights the value of having system-wide instrumentation at the <i>hardware</i> level. Systems like <a href="http://x86vmm.blogspot.com/2007/09/presenting-vprobes.html">VProbes</a> do not allow a process, or indeed even a kernel, to "opt out." If the VM's user wants something traced, he gets it traced, by golly.<br /><li>On the other hand: if code <i>really</i> wants to evade VProbes, DTrace, debuggers, etc., the arms race is heavily stacked in favor of the sneaky code. E.g., suppose you've got some super-s3kr3t, double-plus DRM code that you absolutely don't want instrumented with DTrace's pid provider; on x86's, you could easily checksum the text to find smashed-in <a href="http://faydoc.tripod.com/cpu/int3.htm">int3</a>'s. More elaborate evasion/detection schemes are possible too, similar in spirit to the VMM detection techniques summarized in <a href="http://www.vmware.com/files/pdf/vmm-detection-hotos07.pdf">our HOTOS paper</a> about VMM detectors from last year. We concluded that creating a <i>completely</i> invisible VMM is an ill-posed problem, and trying too hard is a waste of time. I believe a similar dynamic is at work with dynamic tracing systems.<br /></ol><div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com2tag:blogger.com,1999:blog-17691667.post-41833512007606831072007-10-25T15:02:00.000-07:002007-10-25T15:16:58.401-07:00Ouch, Theo!Slashdot, by way of kerneltrap, picked up a <a href="http://kerneltrap.org/OpenBSD/Virtualization_Security">hilarious</a> flamewar over on the OpenBSD mailing list. The short version is that some guy thinks virtualization involves code, and code has bugs, and therefore virtualization is only ever bad. Or something. I'll leave the histrionics to those who think there's a substantial argument lurking in there somewhere, but I just couldn't help myself when I saw the following appeal to authority:<br /><blockquote><br />While x86 hardware has the same page-protection hardware that an IBM 390 architecture machine has, modern PC machines are a mess. They are architecturally so dirty, that parts of the video, keyboard, and other IO devices are interfaced with even to do simple things like context switching processes and handling interrupts. Those of us who have experience with the gory bits of the x86 architecture can clearly say that we know what would be involved in virtualizing it, and if it was so simple, we would not still be fixing bugs in the exact same area in our operating system going on 12 years.<br /></blockquote><br /><br />I'd like to call "BS" on the above claim of inscrutability. No, you do not dork around with, e.g., the video card, to switch contexts. You may indeed mess about with an I/O device in "handling interrupts," but only because, gee, there's an interrupt to handle. In my seven years of debugging every goofball OS written for the PC on our VMM, I haven't encountered a single task switch that wasn't basically some variation of disabling interrupts, taking a couple spin locks, switching stacks, and changing the page table pointer. You know, kind of like it is on every architecture on the planet.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com1tag:blogger.com,1999:blog-17691667.post-88190541530469735002007-09-05T07:44:00.001-07:002007-09-05T08:13:56.870-07:00Denier's DilemmaShankar Vendatam at the <a href="http://www.washingtonpost.com/"><span style="font-style: italic;">Washington Post</span></a> writes:<br /><blockquote><p> The federal <a href="http://www.washingtonpost.com/ac2/related/topic/Centers+for+Disease+Control+and+Prevention?tid=informline" target="">Centers for Disease Control and Prevention</a> recently issued a flier to combat myths about the flu vaccine. It recited various commonly held views and labeled them either "true" or "false." Among those identified as false were statements such as "The side effects are worse than the flu" and "Only older people need flu vaccine."</p> <a href="http://www.washingtonpost.com/ac2/related/topic/University+of+Michigan?tid=informline" target="">When University of Michigan</a> social psychologist Norbert Schwarz had volunteers read the CDC flier, however, he found that within 30 minutes, older people misremembered 28 percent of the false statements as true. Three days later, they remembered 40 percent of the myths as factual.<br /><br />...<br /></blockquote><br />(By way of the often excellent, and often nutty, <a href="http://www.overcomingbias.com/">Overcoming Bias</a>.) I'm reminded of the <a href="http://x86vmm.blogspot.com/2007/08/hmm-thats-not-what-i-think-i-wrote.html">difficulty</a> I've had explaining that VMs and security don't really have much to do with one another. Is it possible that, in attempting to explain something really convoluted to a crowd that barely cares, we sometimes do more harm than good? If you find yourself in a situation where you're in possession of a counter-intuitive result, that's really hard to explain, but important none the less, what's the ethical thing to do, given that saying "X" will register in a lot of folks' minds as "not X"?<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com6tag:blogger.com,1999:blog-17691667.post-53783125768726562122007-09-01T11:35:00.000-07:002007-09-02T09:17:19.567-07:00Presenting VProbesVMworld 2007 is coming up, and I'm more excited than usual this time around. Alex Mirgorodskiy, Robert Benson and I will be demonstrating a piece of technology called VProbes.<br /><br />VProbes was born from an experience that many of you have probably shared: I was wondering why my computer was so slow. I have whiled away many hours over the years investigating this question, and I have rarely gotten a solid answer. Sometimes the slowness is intermittent, and by the time I've developed a theory, it has already disappeared. Other times, the diagnostic tool I want to run (top, or strace, or ltrace, or what have you) isn't installed in the VM, because it's a purpose-built VM with a constrained set of user-land tools. Or, the tools are there, but the system is not in a state to allow me to use them; e.g., it's hung early in boot, and I can't log in to the system. Still other times, I never really scratch the surface of the problem because it's Windows and I don't know my way around.<br /><br />VProbes attempts to provide a set of tools for answering the question, "What the heck is this computer doing?" It's an open-ended question, so vprobes is accordingly open-ended, as well. In its current form, it provides an interactive, safe way of instrumenting a running VM at any level: from user-level processes down to the kernel, and even into VMware's VMM and hypervisor, if need be.<br /><br />We also attempt to solve the "unfamiliar tools" problem, by papering over (to the degree possible) the differences among operating systems. There is a broad consensus among Windows and the UNIX flavors about the abstractions that operating systems provide: all major operating systems provide threads, a tree of named processes, named files in a hierarchical namespace with byte contents, sockets, etc. VProbes tries to get by on enough guest-specific knowledge to "wrap" those differences. Thus, for Windows, Linux, and any other operating system that VProbes can be "taught" to understand, the following vprobe script provides a rough and ready approximation of the top UNIX utility:<br /><br /><span style="font-family:courier new;"> Guest_TimerIRQ</span><span style="font-family:courier new;"> ticks[curprocname()] <- 1;</span><br /><span style="font-family:courier new;"></span><br /><br />"Big deal," you might be thinking. "I had top already." Well, you didn't have top on Windows. And you didn't have it while the machine was booting, or shutting down. And you certainly didn't have it if the machine in question was a virtual appliance that doesn't even allow logins.<br /><br />Anyway, if this sounds interesting to you, please consider attending our VMworld breakout session on the 13th, from 3:30 to 4:30, numbered<span style="text-decoration: underline;"></span> TA70. I should warn that this is basically a research prototype: we haven't committed to putting VProbes in any upcoming VMware product, and indeed, might never ship it at all. We're putting what we've got in front of the public now because we are at the point where we need feedback from real users to improve VProbes.<br /><br />In the "credit where it's due" department: we owe an enormous debt in our thinking about this problem to our <a href="http://blogs.sun.com/bmc/">colleagues</a> <a href="http://blogs.sun.com/mws/">at</a> <a href="http://blogs.sun.com/ahl/">Sun</a>. I've never hidden my admiration for <a href="https://www.sun.com/bigadmin/content/dtrace/">DTrace</a>. We hope to improve on it. First, we are aiming to provide a Dtrace-like tool for other commercially important operating systems than Solaris. Second, VProbes can combine with other virtualization-based techniques in powerful ways. For example, VProbes and <a href="http://www.arctic.umn.edu/%7Ejjyi/MoBS/2007/program/01C-Xu.pdf">deterministic replay</a> combine to make the most potent tool that I'm aware of for debugging intermittent performance anomalies. I'll leave applications to VMotion, snapshots, etc., as tantalizing hints...<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com4tag:blogger.com,1999:blog-17691667.post-27234533860276789462007-08-01T09:14:00.000-07:002007-08-01T09:37:08.860-07:00Hmm. That's not what I think I wrote...I'm perplexed by this peculiar <a href="http://www.techworld.com/security/news/index.cfm?newsID=9653&pagtype=all">interpretation</a> of our HotOS paper:<br /><blockquote><br />Security has been touted as one of the benign by-products of virtualisation – but according to a recent study, that’s no longer the case.<br /></blockquote><br />Huh? "Security" is a big concept, covering many possible applications. Our paper is just a tiny footnote in a complicated discusssion about the role of VMMs in providing security; in fact, to the extent we challenge conventional wisdom at all, we suggest that the purported security <a href="http://www.eweek.com/article2/0,1895,2152137,00.asp">threat</a> posed by VMM-based rootkits is non-existent. To get out the sock puppets: VMMs are always detectable, which is a <span style="font-style:italic;">good</span> thing.<br /><br />I think I see where the author got a bit tripped up reading our paper, though. Currently, there is a trend for malware to disable itself in the presence of VMMs. We argue that this trend cannot continue, not because VMMs are becoming undetectable, but because VMMs are becoming too ubiquitous for malware authors to ignore. Note that this current fad among malware authors of refusing to do dirty business in a VM is <i>not</i> an inherent security benefit of VMs!<br /><br />A nice thought experiment when thinking about security applications of virtual machines is to replace "VM" with "laptop." Suppose, for the sake of argument, security researchers started building honeynets out of laptops, because it was more economical to do so. For a while, malware authors might decide to detect that they're running on a laptop, and refuse to do so, in order to thwart the security researchers. (Note, of course, that it's completely trivial for user-level software to figure out what sort of hardware platform it's running on; this information is intentionally, usefully exposed by, e.g., /proc nodes in Linux, or the \Device directory on NT, etc.). The answer to this situation is not to attempt to cloak the laptop-iness of the hardware platform, though; attempting to do so is a bottomless pit of wasted effort. Instead, we simply observe that there are a lot of laptops out there, and figure that, in the long run, malware that refuses to run on laptops will be giving up too many targets to represent a <a href="http://en.wikipedia.org/wiki/Nash_equilibrium">Nash equilibrium</a> for the black hats.<br /><br />Honestly, though, I'm not that happy with the paper. We're telling a complicated story, and I'm not sure we tell it clearly enough. Even if Manek Dubash's misinterpretation were intentional*, the fact that it the paper can even be plausibly distorted to read the way he suggests means that, at some basic level, we've failed. Good technical writing is hard.<br /><br />* Purely speculating, here. I'm not casting aspersions on anyone's intentions; an honest misunderstanding is, sadly, possible.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com4tag:blogger.com,1999:blog-17691667.post-8347262425126221602007-07-12T08:15:00.000-07:002007-07-12T08:53:17.927-07:00Overcoming biasNot strictly virtualization-related, but <a href="http://www.overcomingbias.com/">overcoming bias</a> is one of the more consistently interesting cross-disciplinary blogs I've come across. The same sorts of cognitive biases that cause people to make bad decisions about their happiness, finances, political preferences, etc., often cause errors in software engineering. Pawing through <a href="http://en.wikipedia.org/wiki/List_of_cognitive_biases">Wikipedia's list of cognitive biases</a>, I can associate almost every single bias with one (or more) mistakes I've made in my professional life. <a href="http://en.wikipedia.org/wiki/Optimism_bias">Optimism bias</a>? Zounds, what engineer isn't guilty of it on a daily bias? <a href="http://en.wikipedia.org/wiki/Information_bias">Information bias</a> has often led me to waste hours in debugging collecting useless trivia. Some have even entered the lore of computing; <a href="http://en.wikipedia.org/wiki/Ingroup_bias">ingroup bias</a> is nothing but "NIH syndrome."<br /><br />While "overcoming bias" is certainly a laudable goal in many professional fields, I wonder about its tractability. These biases are clearly hardwired, and I suspect no amount of mindfulness will prevent us from committing them. Getting rid of one of these well-attested biases would probably require the neurological equivalent of, e.g., making humans able to digest grass, or giving a human the <a href="http://www.tufts.edu/vet/sports/oxygen.html">VO2Max</a> of a horse. Oh well; if nothing else, at least being aware of our limitations as thinking machines might prevent us from trying anything too impossibly stupid...<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com2tag:blogger.com,1999:blog-17691667.post-69855379205089445222007-07-02T11:09:00.000-07:002007-07-02T12:19:22.977-07:00BluePill detection in two easy steps.In <a href="http://www.usenix.org/events/hotos07/tech/full_papers/garfinkel/garfinkel.pdf">our HotOS submission</a>, we alluded to a large number of theoretically possible methods of detecting the presence of a hypervisor. I'm not sure which of these many ways the well-publicised <a href="http://rdist.root.org/2007/06/28/undetectable-hypervisor-rootkit-challenge/"> BluePill challengers</a> have chosen, but I thought I'd make things a bit more concrete.<br /><br />In our paper, we outline family of detection methods based on resource consumption. In short, the hypervisor must use cache, memory bandwidth, TLB entries, etc., in the course of multiplexing the CPU. A guest can be made intentionally sensitive to these resources in order to detect a hypervisor. For instance, you can obtain a good estimate of the effective size of the hardware TLB with psuedo-code like the following:<br /><br /><pre><br />PPN oldPhysPage, newPhysPage = Alloc...();<br />VA oldVirtAddr = MapSomewhere(oldPhysPage);<br />VA newVirtAddr = MapSomewhere(newPhysPage);<br />memset(oldVirtAddr, 0x11, PAGE_SIZE);<br />memset(newVirtAddr, 0x22, PAGE_SIZE);<br />PTE=base of hardware page table;<br />for (i = 0; i < BIGNUM; i++) {<br /> PTE[i] = MAKE_PTE(oldPhysPage); // map old page<br /> (void)*(volatile char*)(i * PAGE_SIZE); // bring it into the TLB<br />}<br />/*<br />* TLB now contains between 0 and BIGNUM old mappings.<br />* Now modify the PTEs *without* flushing the TLB,<br />* and wait for the hardware to leak an entry<br />*/<br />for (i = 0; i < BIGNUM; i++) {<br /> PTE[i] = MAKE_PTE(newPhysPage); // map old page<br /> if ((*(volatile char*)(i * PAGE_SIZE)) == 0x22) {<br /> printf("apparent tlb size: %d\n", i);<br /> break;<br /> }<br />}</pre><br />OK, so you can size the TLB. So what? Well, just run the above loop twice, but the second time, stick an instruction that's guaranteed to trap out to the hypervisor (e.g., <span face="courier new">CPUID</span>) before the loop. On bare hardware, this inocuous instruction will have no impact on the TLBs. In the presence of a hypervisor, the instruction will either flush the TLB entirely (on current Intel VT implementations), or, if you're on a clever SVM hypervisor that uses the ASID facility to avoid TLB flushes on every hypervisor invocation, at least 1 fewer TLB entry. (<span style="font-size:85%;">Every hypervisor must at least read the VMCB's exit code, requiring 1 data TLB entry. For increased sensitivity, you could also detect the VMM's code footprint by rejiggering the above to measure ITLB entries, an exercise left to the reader. Hint: the return instruction is a single byte on x86</span>.)<br /><br />This detection technique uses only TLB capacity, but a more sophisticated approach could easily include the hardware eviction policy. We could also apply these same technique to instruction and data caches. (<span style="font-size:85%;">Even the x86's coherent data caches can be sized using the <span style="font-family:courier new;">INVD</span> instruction, which discards cache contents without writing back dirty data.</span>) A less naive approach that uses the hardware's event-based performance monitoring support (RDPMC and friends) would have an even richer menu of options for noticing virtualization-induced perturbations.<br /><br />I've seen zero evidence that Rutkowska has pondered resource-based detection attempts like this, or indeed, any attacks more sophisticated than a "go-slow" loop between reads of the PIT. It is hard for me to imagine a "hypervisor" worthy of the name that doesn't leave noticable traces in resource usage. In fact, to the degree that a hypervisor goes to heroic lengths to prevent such a detection technique, e.g., by running a hardware-accurate cache simulator on every guest memory access, it will only open up wider timing discrepancies for the guest's HV-detector to exploit.<br /><br />I can only conclude that in 2006 Rutkowska was either naive about the possibilities of such attacks, or that she consciously chose to make an outrageous and indefensible claim ("undetectable malware!!!!111") in order to draw attention to herself and her company. Given the peripheral evidence of Rutkowska's competence, I think the evidence favors the latter, but I'd simply love to hear from her on the subject...<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com10tag:blogger.com,1999:blog-17691667.post-59693064973233444992007-06-29T06:00:00.000-07:002007-06-29T06:08:51.108-07:00BluePill hype going the way of all fleshI'm <a href="http://rdist.root.org/2007/06/28/undetectable-hypervisor-rootkit-challenge/"> not alone</a> in <a href="http://www.usenix.org/events/hotos07/tech/full_papers/garfinkel/garfinkel.pdf">my skepticism</a> about BluePill's claims of undetectability. Let's pop some popcorn, folks! This could get good...<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com0tag:blogger.com,1999:blog-17691667.post-90631110444145938422007-02-27T16:39:00.000-08:002007-03-23T09:25:18.383-07:00Please indulge me...I've tried to be fairly disciplined about keeping this blog technical. Never the less, I would ask for your indulgence to announce the birth of Emmett Joseph Adams on February 27th, 2007 at 11:48am.<br /><br />The little dude is three amazing weeks old now. Everybody is as healthy and happy as can be expected. A few observations from very, <span style="font-style: italic;">very </span>early parenthood:<br /><ol><li>I'm amazed in the photo below how bloody <span style="font-style: italic;">peppy</span> I look. And that was after being awake all night! I've already come to regard bloodshot eyes, 5 o'clock shadow, and coffee stains on clothes I've slept in as the norm. When I ask other parents when I can expect that to change, they invariably say something smarmy, like, "oh, in about eighteen years."<br /></li><li>When I play guitar and sing for him, he kind of zones out for a while and then starts bouncing his legs and arms in time while making these little happy grunts.</li><li>Emmett's mother, Jean, is the most amazing person in the world.</li></ol><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8KhP65pMRhZKIwLrmnjOt8Mr0mm-Ms1fOsLvevGx4M0626_wqvC-gMexGu2DMnpXTAI4y8LdicUe9gAJUw3LAM-qgAyo5PSQTEFIeRxq0Bw6UqmflLWIK7CaQKYlvuvjboKj5_A/s1600-h/Emmett+064.jpg"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8KhP65pMRhZKIwLrmnjOt8Mr0mm-Ms1fOsLvevGx4M0626_wqvC-gMexGu2DMnpXTAI4y8LdicUe9gAJUw3LAM-qgAyo5PSQTEFIeRxq0Bw6UqmflLWIK7CaQKYlvuvjboKj5_A/s320/Emmett+064.jpg" alt="" id="BLOGGER_PHOTO_ID_5036378767951809938" border="0" /></a><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipn3cnURbaYeQrKp5T3LIBakVViKTbtTNHrMzW-DNfvzxDIS6BPaVo4XtagAR-oybtcJqRb-Vae5RMNnn3JfjjUUUM-WuUhYq3uwEP2gmVP9RbgJhxwxfDIQvZD3uvTP0SVIoAMw/s1600-h/Emmett+047.jpg"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipn3cnURbaYeQrKp5T3LIBakVViKTbtTNHrMzW-DNfvzxDIS6BPaVo4XtagAR-oybtcJqRb-Vae5RMNnn3JfjjUUUM-WuUhYq3uwEP2gmVP9RbgJhxwxfDIQvZD3uvTP0SVIoAMw/s320/Emmett+047.jpg" alt="" id="BLOGGER_PHOTO_ID_5036378772246777250" border="0" /></a><br />OK, back to our regularly scheduled computer stuff!<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com4tag:blogger.com,1999:blog-17691667.post-1172167147485574862007-02-22T09:16:00.000-08:002007-02-22T09:59:07.503-08:00Microsoft swallows the BluePillWhy doesn't Microsoft want users to run Vista in a VM? Why, because t<a href="http://biz.yahoo.com/ap/070222/microsoft_virtual_vista.html?.v=2">he presence of a VMM could mean someone has l33tly r00t3d your b0x0r</a>! So nice of Microsoft to protect us shivering, ignorant, childlike customers from choosing how we'd like to run Windows. Ahh, the bliss of having all my decisions made for me. Thank you. I wonder if I should be just as worried about the inherent insecurity of running on a hypervisor once Longhorn ships? Oops, I almost forgot: best not to think of these things! Better to let Microsoft do the worrying for me.<br /><br />Now that the BluePill pseudo-sploit has been widely publicized for over a year now, I'm still aware of no actual attacks. This, despite the fact that SVM and VT hardware are now extremely common, and getting more so by the day. Why would that be?<br /><br />Well, first of all, SVM and VT make possible nothing that was not already possible before; VMware's software-only products are an existence proof. The BluePill-istas don't claim that SVM/VT make new exploits possible <span style="font-style:italic;">per se</span>; rather, the claim is that SVM/VT make it possible to <span style="font-style:italic;">cloak</span> the presence of a VMM rootkit completely.<br /><br />Allow me to go on record: this claim is pure fantasy. In practice, it is <span style="font-style:italic;">always</span> possible to detect the presence of a VMM, via timing attacks. "Aha!", my hand-wringing BluePill interlocutor exclaims: "But the VMM lets us hook RDTSC and the PIT and the APIC timer, etc., so we can show the VM whatever time we want! Or whatever!" The details of getting such an approach right in practice are tough beyond belief. Once you start thinking it through, you quickly realize the deck is stacked in favor of the would-be VMM detector. As a practical example, VMware's software goes to extremely clever lengths in coordinating and manipulating the various virtual time sources <a href="http://lkml.org/lkml/2006/11/16/259">just to get certain versions of the linux kernel to boot</a>. It took some smart people <span style="font-style:italic;">years</span> to really drive a stake through the heart of the "pester mingo" boot-time failure in SMP VMs. This was the result of a real, naturally occurring guest code that had nothing to do with VM detection; you can imagine that a dedicated attack on the VMM's virtual time sources could be many orders of magnitude more effective.<br /><br />So, I'm claiming you can always find out that there's a VMM underneath you. Should that worry us, too? Since some members of the security community are using VMs to study malware, the arms race between malware creators and investigators has temporarily taken a turn wherein some malware refuses to run in a VM. This has led some to think of the <span style="font-style:italic;">detectability</span> of a VMM as a deficiency in the virtualization layer.<br /><br />I don't see it that way. As VMs become more commonplace, malware that refuses to run in VMs will be naturally selected out. We don't expect, e.g., a Dell laptop to cloak the version of Windows it's running, or which chipset or video card is present, or for that matter which version of Windows is running. It's perfectly legit for software to query these aspects of its execution environment, having an orderly way to enumerate and version them is a valuable thing. As VM proliferation progresses, I suspect that we will come to think of VMMs in the same way: as another facet of the stack of hardware and software on which you happen to be running, whose presence is neither surprising nor secret.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com4tag:blogger.com,1999:blog-17691667.post-1161877999560295322006-10-26T08:46:00.000-07:002006-10-26T09:23:41.806-07:00Survived ASPLOS XII!Well, I got to present my first <a href="http://www.princeton.edu/~asplos06/">conference</a> <a href="http://www.vmware.com/pdf/asplos235_adams.pdf ">paper</a>, and lived to tell the tale. Getting to take a question from Dave <a href="http://www.amazon.com/Computer-Architecture-Fourth-Quantitative-Approach/dp/0123704901/ref=pd_lpo_k2_dp_k2a_1_txt/104-0235784-6125539">"AQA" </a>Patterson gave me goosebumps. On Tuesday, a grad student whose name escapes me (sorry!) asked Ole and me, "So, the two of you did all this work yourselves?" Of course not! I'm ashamed to have left that impression on anybody. The VMware VMM group has around 100 alumni at this point; our results, at least as regards the software VMM, stand on the shoulders of many giants. The hardware VMM has also had significant contributions from Jim Mattson, Meenali Rungta, and Michael Ho.<br /><br />ASPLOS was biking distance from my home this year, so perhaps I'll present a receipt for some chain lube as a travel expense or something. It's amazing how hard it is to get motivated for the "extra-curriculars" at a conference when you can just go home any time you like; instead of going to the welcome reception, I stayed home and made meatloaf. Whatcha gonna do...<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com0tag:blogger.com,1999:blog-17691667.post-1159370396706864422006-09-27T08:08:00.000-07:002006-09-27T09:43:28.316-07:00Going back to Rhody... Rhody... Rhody...<a href="http://en.wikipedia.org/wiki/Ll_cool_j">I don't think so.</a><br /><br />I'm going back to my <a href="http://www.cs.brown.edu/">alma mater</a> to give <a href="http://www.cs.brown.edu/events/talks/adams.html">a talk</a>. I figure I might as well do what little I can to boost attendance, so pretty please, New England virtualizanti, do what you can to make it.<br /><br />While I'm waxing nostalgic/faux hip-hop about my misspent college youth, it has been pointed out to me that a shout-out is due to my erstwhile <a href="http://www.cs.brown.edu/courses/cs169">CS 169 TA</a> and all-around <a href="http://cryptnet.net/mirrors/texts/kissedagirl.html">bad-boy</a> of the hardware/software interface, Bryan Cantrill. <a href="http://online.wsj.com/public/article/SB115755300770755096-Puh3Kr2L9dGEhvkWyO94UivIRwA_20070910.html"><i>Chapeau</i></a>, Bryan, Adam, and Mike.<br /><br />As I've pointed out again and again, to the point of eye-rolling boredom on the part of my colleagues, DTrace really is outrageously cool. The improvement in leverage that DTrace yields feels comparable to the improvement going from debugging with printf to a source-level interactive debugger. Those who haven't tried it yet, run, do not walk, to download <a href="http://www.opensolaris.org">recent build of OpenSolaris</a>, <a href="http://www.vmware.com/products/player">install it in a VM</a>, and start working your way through the <a href="http://developers.sun.com/solaris/articles/dtrace_for_dev.html">numerous tutorials</a>. You'll be glad you did.<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com0tag:blogger.com,1999:blog-17691667.post-1158786319310002672006-09-20T13:52:00.000-07:002006-09-20T14:18:28.760-07:00Yes, Virginia. We deliver GP's on control flow transfers at the wrong EIP.<a href="http://eeyeresearch.typepad.com/blog/2006/09/another_vmware_.html">Busted!</a> This is just one of those tough situations where "perfect" software and "useful" software are in unresolvable conflict. While we could, in theory, get this corner case completely right, no customer would want us to. To patch up this hole we'd have to make <i>all</i> indirect kernel control flow transfers slower. Since literally <span style="font-style:italic;">nothing</span> we've ever run into in the wild cares, we just get this wrong. C'est la vie. Though, interestingly enough, the one piece of software out there that we know relies on this behavior: the VMware VMM itself.<br /><br />Since there are both architected (<a href="http://chitchat.at.infoseek.co.jp/vmware/vmtools.html">the backdoor port</a>) and unarchitected (i.e., the s[gi]dt/lsl instructions) ways of discovering you're in a VM, this isn't really "news," per se, but interesting x86 trivia none the less. If it makes you upset, you have a few options to make the behavior in a VM more like that of real hardware. As Derek discovered, you can tick the "disable acceleration" checkbox in your VM. You can also buy a shiny, new Intel processor and drop "monitor_control.vt32=TRUE" into your config file, but <a href="http://www.vmware.com/community/message.jspa?messageID=376400">expect lousy performance</a>. (VT32 really only exists as a check on the correctness of our VT implementation; all the VT performance work has been sunk into running 64-bit guests.)<br /><br />At the end of the day, discovering you're in a VM, in general, is always likely to be easy, just because of timing attacks. These pieces of code that folks write to detect that you're in a VM are sort of modern-day equivalents of the little assembly programs that were used for CPU identification before the CPUID instruction came along; these terribly clever little DOS programs would try to infer the size of the write buffer, time a few instructions, etc., and come up with a "fingerprint" of the CPU. These programs are serious fun to write, and have some educational value. I haven't seen any legitimate, practical uses of the VM detection programs thus far, though. I.e., congratulations, you now know you're on a VMware Workstation 4.5 VM. What are you going to do about it?<div class="blogger-post-footer">Keith Adams slings code for a living. This "blog" thing is a sideline, not some sort of officially sanctioned PR stunt. So, please don't mistake his opinions, biases,
ignorance, or taste in clothing with those of Facebook.</div>Keith Adamshttp://www.blogger.com/profile/11187291538765358570noreply@blogger.com0