DTrace Topics Guide
 DTrace Topics: Guide
This article is a guide to programming in DTrace, and is part of the DTrace Topics collection. A general understanding of DTrace is assumed knowledge, which can be studied from the DTrace Topics: Intro section. DTrace is a dynamic troubleshooting and analysis tool first introduced in the Solaris 10 and OpenSolaris operating systems.
 DTrace Guide
This DTrace Guide focuses on summaries, examples and suggested usage.
For a reference on DTrace, see the official docs.sun.com DTrace Guide which is also an excellent source of examples.
Only root or users with DTrace privileges can run DTrace. If an unprivileged user trys to run /usr/sbin/dtrace,
$ id uid=1001(user1) gid=1(other) $ /usr/sbin/dtrace -n 'syscall::exece:return' dtrace: failed to initialize dtrace: DTrace requires additional privileges
Non-root users need one or more of the following privileges to use DTrace:
|dtrace_user||to examine their syscalls, and to profile their own processes|
|dtrace_proc||to trace their own processes|
|dtrace_kernel||to examine the kernel. Danger!|
The dtrace_kernel privilege is dangerous as it allows the user read only access across the kernel. With some DTrace expertise, any data that the server acts upon could be viewed, including plaintext passwords and files. Fortunately, the dtrace_user and dtrace_proc privileges may be sufficient for many users of DTrace, especially developers who wish to DTrace their own applications.
 Adding Privileges
Interfaces for privilege control include ppriv(1), usermod(1M) and user_attr(4). Adding DTrace privileges to the user1 account can be performed in the following way:
As user1, $ id uid=1001(user1) gid=1(other) $ ppriv $$ 214885: -sh flags = <none> E: basic I: basic P: basic L: all $ exit As root, # grep user1 /etc/user_attr # usermod -K defaultpriv=basic,dtrace_user,dtrace_proc user1 # grep user1 /etc/user_attr user1::::type=normal;defaultpriv=basic,dtrace_user,dtrace_proc # su - user1 $ ppriv $$ 285007: -sh flags = <none> E: basic,dtrace_proc,dtrace_user I: basic,dtrace_proc,dtrace_user P: basic,dtrace_proc,dtrace_user L: all
The user1 account begins as an unprivileged user (UID != 0 and no entry in /etc/user_attr). root then runs usermod to add two DTrace privileges to the user1 account. This puts an entry in /etc/user_attr.
Probes are the instrumentation point from which DTrace can collect data. Their names are a four-tuple: provider:module:function:name,
|Provider||A library of related probes.|
|Module||The module the function belongs to, either a kernel module or user segment.|
|Function||The function name that contains the probe.|
|Name||The name of the probe.|
At the command line,
- Listing probes: dtrace -ln probename
- Tracing probes: dtrace -n probename
 Specifing Probe Names
A blank field in a probe name is a wildcard, as is '*'. The following are examples of this:
- List everything from the syscall provider.
- List all syscall entry probes when the function begins with “open”. This matches open and open64.
The following uses -ln to list the matched probe names:
# dtrace -ln 'syscall::open*:entry' ID PROVIDER MODULE FUNCTION NAME 75947 syscall open entry 76335 syscall open64 entry
 Understanding Probe Names
The best way is to take the provider name, then look up the related chapter in the DTrace Guide (docs.sun.com).
Examples of probe name interpretation:
- The start probe in the io provider, for bdev_strategy() in module genunix. (DTrace Guide, chapter 27, io).
- The BEGIN probe in the dtrace provider. (DTrace Guide, chapter 17, dtrace).
 Finding Probes
With thousands of probes available, it can take some time to find which probes are relevant.
You may get some mileage from grep(1) and regular expressions. Eg, what traces kill(1) activity?:
# dtrace -l | grep kill [...42 lines truncated...] 63725 fbt ufs ufsfx_kill_fix_failure_thread entry 63726 fbt ufs ufsfx_kill_fix_failure_thread return 63930 fbt ufs logmap_kill_roll entry 63931 fbt ufs logmap_kill_roll return 71717 fbt lx_brand lx_kill entry 71718 fbt lx_brand lx_kill return 71799 fbt lx_brand lx_tkill entry 71800 fbt lx_brand lx_tkill return 75527 fbt md_mirror resync_kill_pending entry 75528 fbt md_mirror resync_kill_pending return 76011 syscall kill entry 76012 syscall kill return 76221 syscall lwp_kill entry 76222 syscall lwp_kill return # dtrace -l | grep kill | wc -l 56
This approach helped (in this case), filtering ~70,000 probes down to 56 potentials.
 Finding Probes Checklist
Approaches for finding probes,
- Use dtrace -ln with '*', or dtrace -l with grep(1).
- Run your workload, watch what fires.
- if it doesn't fire, it is almost certainly not related.
- Run a known workload with a count of n (where n is a moderately sized prime), see what fires by a factor of n.
- Eg, 23 database transactions.
- Think of triggered events, and study their stack traces.
- Eg, the stacks for io:::start shows who caused I/O.
- Read the source, if available.
- Ask on firstname.lastname@example.org.