Wednesday, January 23, 2008

MacOS X/DTrace tryst ends in tears.

See Adam's break-up note. This also got picked up on slashdot.

My thoughts, in no particular order:

  1. Shame on Apple. What is the point of building system-wide infrastructure that any random process can opt out of? Forget about DRM; what about legitimate uses of DTrace for system health monitoring? Won't all malware be sure to set the "don't dtrace me" bit now?
  2. This highlights the value of having system-wide instrumentation at the hardware level. Systems like VProbes do not allow a process, or indeed even a kernel, to "opt out." If the VM's user wants something traced, he gets it traced, by golly.
  3. On the other hand: if code really wants to evade VProbes, DTrace, debuggers, etc., the arms race is heavily stacked in favor of the sneaky code. E.g., suppose you've got some super-s3kr3t, double-plus DRM code that you absolutely don't want instrumented with DTrace's pid provider; on x86's, you could easily checksum the text to find smashed-in int3's. More elaborate evasion/detection schemes are possible too, similar in spirit to the VMM detection techniques summarized in our HOTOS paper about VMM detectors from last year. We concluded that creating a completely invisible VMM is an ill-posed problem, and trying too hard is a waste of time. I believe a similar dynamic is at work with dynamic tracing systems.

2 Comments:

Blogger Adam said...

To clarify, we're not breaking up -- I just wish Apple had told us they wanted an open relationship.

And I'm quite surprised to hear that you think sneaky code will win out. My guess is that you're colored by trying to produce static software which can't be hoodwinked, as I'd argue is the advantage is to the party that responds last. If a vendor hands over their bits, no amount of cleverness will keep a l33ty hacker from circumventing any protections. Similarly, once a virus or root kit is out in the wild it's just a matter of time before someone cooks up a countermeasure.

11:33 AM  
Blogger Keith Adams said...

And I'm quite surprised to hear that you think sneaky code will win out. My guess is that you're colored by trying to produce static software which can't be hoodwinked, as I'd argue is the advantage is to the party that responds last.

It seems like an arms race, where the last mover should win. But it's massively easier to detect than to hide, so given comparable resources the detectors will "win" the race.

Why this assymmetry? And what do I mean by "win"? The "detector" has a much easier time responding than the "hider" in this game, because the detector has truth on its side. If DTrace really is running, it will leave some footprints, somewhere (access and dirty bits of the page table entries it uses; the cache line pressure it exerts on instrumented paths; TLB entries consumed; etc.). Beyond a certain point, heroics to plug up all the channels between DTrace and the outside world become contrary to DTrace's primary purpose. Once the detectors have pushed the game this far, they've effectively won, unless you agree to break DTrace.

Now, granted, I'm used to thinking of scenarios where the "detector" is running in kernel mode, and so can more reliably detect microsecond-level jitter, has access to hardware performance counters, etc. Perhaps user mode is a noisy enough place that these sorts of detectors are impractical. Never the less, something as simple as scanning the process table for the presence of "dtrace" and changing behavior in its presence is a rough and ready way of approximating what Apple seems to be doing for iTunes.

1:30 PM  

Post a Comment

<< Home