DTrace Topics Guide

From Siwiki

Jump to: navigation, search


[edit] DTrace Topics: Guide

This article is a guide to programming in DTrace, and is part of the DTrace Topics collection. A general understanding of DTrace is assumed knowledge, which can be studied from the DTrace Topics: Intro section. DTrace is a dynamic troubleshooting and analysis tool first introduced in the Solaris 10 and OpenSolaris operating systems.

Completion Image:trafficlight_red02.png
Difficulty Image:coffeemug01.png Image:coffeemug01.png
Audience All DTrace users

[edit] DTrace Guide

This DTrace Guide focuses on summaries, examples and suggested usage.

For a reference on DTrace, see the official docs.sun.com DTrace Guide which is also an excellent source of examples.

[edit] Privileges

Only root or users with DTrace privileges can run DTrace. If an unprivileged user trys to run /usr/sbin/dtrace,

$ id
uid=1001(user1) gid=1(other)
$ /usr/sbin/dtrace -n 'syscall::exece:return'
dtrace: failed to initialize dtrace: DTrace requires additional privileges

Non-root users need one or more of the following privileges to use DTrace:

Privilege Name Description
dtrace_user to examine their syscalls, and to profile their own processes
dtrace_proc to trace their own processes
dtrace_kernel to examine the kernel. Danger!

The dtrace_kernel privilege is dangerous as it allows the user read only access across the kernel. With some DTrace expertise, any data that the server acts upon could be viewed, including plaintext passwords and files. Fortunately, the dtrace_user and dtrace_proc privileges may be sufficient for many users of DTrace, especially developers who wish to DTrace their own applications.

[edit] Adding Privileges

Interfaces for privilege control include ppriv(1), usermod(1M) and user_attr(4). Adding DTrace privileges to the user1 account can be performed in the following way:

As user1,
$ id
uid=1001(user1) gid=1(other)
$ ppriv $$
214885: -sh
flags = <none>
        E: basic
        I: basic
        P: basic
        L: all
$ exit

As root,
# grep user1 /etc/user_attr 
# usermod -K defaultpriv=basic,dtrace_user,dtrace_proc user1
# grep user1 /etc/user_attr
# su - user1
$ ppriv $$
285007: -sh
flags = <none>
        E: basic,dtrace_proc,dtrace_user
        I: basic,dtrace_proc,dtrace_user
        P: basic,dtrace_proc,dtrace_user
        L: all

The user1 account begins as an unprivileged user (UID != 0 and no entry in /etc/user_attr). root then runs usermod to add two DTrace privileges to the user1 account. This puts an entry in /etc/user_attr.

[edit] Probes

Probes are the instrumentation point from which DTrace can collect data. Their names are a four-tuple: provider:module:function:name,

Provider A library of related probes.
Module The module the function belongs to, either a kernel module or user segment.
Function The function name that contains the probe.
Name The name of the probe.

At the command line,

  • Listing probes: dtrace -ln probename
  • Tracing probes: dtrace -n probename

[edit] Specifing Probe Names

A blank field in a probe name is a wildcard, as is '*'. The following are examples of this:

  • syscall:::
    List everything from the syscall provider.
  • syscall::open*:entry
    List all syscall entry probes when the function begins with “open”. This matches open and open64.

The following uses -ln to list the matched probe names:

# dtrace -ln 'syscall::open*:entry'
   ID   PROVIDER            MODULE                          FUNCTION NAME
75947    syscall                                                open entry
76335    syscall                                              open64 entry

[edit] Understanding Probe Names

The best way is to take the provider name, then look up the related chapter in the DTrace Guide (docs.sun.com).

Examples of probe name interpretation:

  • io:genunix:bdev_strategy:start
    The start probe in the io provider, for bdev_strategy() in module genunix. (DTrace Guide, chapter 27, io).
  • dtrace:::BEGIN
    The BEGIN probe in the dtrace provider. (DTrace Guide, chapter 17, dtrace).

[edit] Finding Probes

With thousands of probes available, it can take some time to find which probes are relevant.

You may get some mileage from grep(1) and regular expressions. Eg, what traces kill(1) activity?:

# dtrace -l | grep kill
[...42 lines truncated...]
63725        fbt               ufs     ufsfx_kill_fix_failure_thread entry
63726        fbt               ufs     ufsfx_kill_fix_failure_thread return
63930        fbt               ufs                  logmap_kill_roll entry
63931        fbt               ufs                  logmap_kill_roll return
71717        fbt          lx_brand                           lx_kill entry
71718        fbt          lx_brand                           lx_kill return
71799        fbt          lx_brand                          lx_tkill entry
71800        fbt          lx_brand                          lx_tkill return
75527        fbt         md_mirror               resync_kill_pending entry
75528        fbt         md_mirror               resync_kill_pending return
76011    syscall                                                kill entry
76012    syscall                                                kill return
76221    syscall                                            lwp_kill entry
76222    syscall                                            lwp_kill return
# dtrace -l | grep kill | wc -l

This approach helped (in this case), filtering ~70,000 probes down to 56 potentials.

[edit] Finding Probes Checklist

Approaches for finding probes,

  1. Use dtrace -ln with '*', or dtrace -l with grep(1).
  2. Run your workload, watch what fires.
    if it doesn't fire, it is almost certainly not related.
  3. Run a known workload with a count of n (where n is a moderately sized prime), see what fires by a factor of n.
    Eg, 23 database transactions.
  4. Think of triggered events, and study their stack traces.
    Eg, the stacks for io:::start shows who caused I/O.
  5. Read the source, if available.
  6. Ask on dtrace-discuss@opensolaris.org.
Solaris Internals
Personal tools
The Books
The Ads