DTraceToolkit

From Siwiki

[edit] DTraceToolkit

This article is about the DTraceToolkit, and is also a topic in the DTrace Topics collection. For a general understanding of DTrace, please see the DTrace Topics: Intro.

DTrace is a dynamic troubleshooting and analysis tool first introduced in the Solaris 10 and OpenSolaris operating systems. The DTraceToolkit is a collection of DTrace scripts that can be useful for performance analysis and troubleshooting.


Completion
Difficulty
Audience	Everyone

[edit] Download the DTraceToolkit

The DTraceToolkit can be downloaded from its OpenSolaris website, or Brendan's DTrace page. The toolkit contains over 100 DTrace based scripts, and we hope it proves useful for performance observability and debugging.

After downloading,

gunzip and "tar xvf" the file. cd to the toolkit directory
run ./install (optional, you can use the toolkit without doing this)
read Guide to find out how to get started
a list of scripts is in Docs/Contents
enjoy!

[edit] About the DTraceToolkit

The DTraceToolkit is a collection of DTrace based scripts for performance observability and troubleshooting on Solaris 10 and OpenSolaris systems.

[edit] Major Components

The major components of the DTraceToolkit are:

The scripts themselves
A man page for every script
An examples file for every script

[edit] Contents

The top level directory contains the top dozen or so most useful scripts. Other directories and files are,

   DTraceToolkit-X.XX/
        Bin/                    Symlinks to the scripts
        Apps/                   Application specific scripts
        Cpu/                    Scripts for CPU analysis
        Disk/                   Scripts for disk I/O analysis
        Docs/                   Documentation
           Contents             Command list for the Toolkit
           Examples/            Examples of command usage
           Faq                  Frequently asked questions
           Links                Further DTrace links
           Notes/               Notes on Toolkit commands
           Readme               Readme for using the docs
        Extra/                  Misc scripts
        Guide                   This file!
        Kernel/                 Scripts for kernel analysis
        License                 The CDDL license
        Locks/                  Scripts for lock analysis
        Man/                    Man pages
           man1m/               Man pages for the Toolkit commands
        Mem/                    Scripts for memory analysis
        Net/                    Scripts for network analysis
        Proc/                   Scripts for process analysis
        System/                 Scripts for system analysis
        User/                   Scripts for user based activity analysis
        Zones/                  Scripts for analysis by zone
        Version                 DTraceToolkit version
        install                 Install script, use for installs only

[edit] Scripts

The scripts examine numerous areas of system behaviour, such as activity of the CPUs, disks, memory system, network interfaces, kernel and user-land code.

If a script end in a ".d" suffix, then it is a pure DTrace script (and will start with #!/usr/sbin/dtrace). If it doesn't, it is a DTrace script wrapped in either shell or Perl code to provide enhanced functionality, such as commnad line options including "-h" for USAGE.

The script code has been written to follow most of the standards from the cstyle tool (a C code checker). The headers are also carefully written, and follow a toolkit standard to neatly convey essential details. For example:

#!/usr/bin/ksh
#
# iopending - Print a plot for the number of pending disk I/O events.
#             Written using DTrace (Solaris 10 3/05).
#
# This is measuring disk events that have made it past system caches.
# By plotting a distribution graph of the number of pending events, the
# "serialness" or "parallelness" of disk behaviour can be distinguished.
#
# 20-Apr-2006, ver 0.61
#
# USAGE:        iopending [-c] [-d device] [-f filename]
#                         [-m mount_point] [interval [count]]
#
#               -c              # clear the screen
#               -d device       # instance name to snoop (eg, dad0)
#               -f filename     # full pathname of file to snoop
#               -m mount_point  # this FS only (will skip raw events)
#       eg,
#               iopending       # default output, 5 second intervals
#               iopending 1     # 1 second samples
#               iopending -c    # clear the screen
#               iopending 5 12  # print 12 x 5 second samples
#
# FIELDS:
#               value           number of pending events, 0 == idle
#               count           number of samples @ 1000 Hz
#               load            1 min load average
#               disk_r          total disk read Kbytes for sample
#               disk_w          total disk write Kbytes for sample
#
# SEE ALSO: iosnoop, iotop
#
# IDEA: Dr Rex di Bona (Sydney, Australia)
#
# COPYRIGHT: Copyright (c) 2005, 2006 Brendan Gregg.
#
# CDDL HEADER START
#
#  The contents of this file are subject to the terms of the
#  Common Development and Distribution License, Version 1.0 only
#  (the "License").  You may not use this file except in compliance
#  with the License.
#
#  You can obtain a copy of the license at Docs/cddl1.txt
#  or http://www.opensolaris.org/os/licensing.
#  See the License for the specific language governing permissions
#  and limitations under the License.
#
# CDDL HEADER END
#
# Author: Brendan Gregg  [Sydney, Australia]
#
# 01-Nov-2005   Brendan Gregg   Created this.
#

[edit] One Liners

Apart from scripts, the DTraceToolkit contains a list of useful one-liners (I was always a fan of the sed one-liners). These are great for a number of reasons:

no towing scripts around, just copy-n-paste
helps you learn DTrace in small easy steps
one liners may have a faster site approval than open source scripts

DTraceToolkit-0.97> more Docs/oneliners.txt
#
#   DTrace OneLiners
#

DTrace One Liners,

# New processes with arguments,
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

# Files opened by process name,
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

# Files created using creat() by process name,
dtrace -n 'syscall::creat*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

# Syscall count by process name,
dtrace -n 'syscall:::entry { @num[execname] = count(); }'

# Syscall count by syscall,
dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'

# Syscall count by process ID,
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

# Read bytes by process name,
dtrace -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }'

# Write bytes by process name,
dtrace -n 'sysinfo:::writech { @bytes[execname] = sum(arg0); }'

# Read size distribution by process name,
dtrace -n 'sysinfo:::readch { @dist[execname] = quantize(arg0); }'

# Write size distribution by process name,
dtrace -n 'sysinfo:::writech { @dist[execname] = quantize(arg0); }'
...

See the Docs/oneliners.txt file in the DTraceToolkit for the full and latest list.

[edit] `Man` directory

This contains man pages for every script. It took a long time to write these - so please read them! :-)

They can be viewed using,

$ MANPATH=Man man iosnoop
Reformatting page.  Please Wait... done

USER COMMANDS                                         iosnoop(1m)

NAME
     iosnoop - snoop I/O events as they occur. Uses DTrace.

SYNOPSIS
     iosnoop [-a|-A|-Deghinostv] [-d device]  [-f  filename]  [-m
     mount_point] [-n name] [-p PID]

DESCRIPTION
     iosnoop prints  I/O  events  as  they  happen,  with  useful
     details such as UID, PID, block number, size, filename, etc.

     This is useful to  determine  the  process  responsible  for
     using  the  disks,  as  well as details on what activity the
     process is requesting. Behaviour such as random  or  sequen-
     tial I/O can be observed by reading the block numbers.
...

As with the script headers, the content of the man pages is standardised.

[edit] `Docs/Examples` directory

This directory contains examples for every script in action, and discusses their output. When reading man pages, many people go straight for the examples (if any) to get a quick understanding for what the command does. This is encouraged in the DTraceToolkit, with entire files just for examples.

This is the first few dozen lines from the fsrw.d example file,

DTraceToolkit-0.97> more Docs/Examples/fsrw_example.txt
The following are demonstrations of the fsrw.d script.


Here the fsrw.d script was running while a 50 Kbyte file was read,

   # ./fsrw.d
   Event            Device RW     Size Offset Path
   sc-read               .  R     8192      0 /extra1/50k
     fop_read            .  R     8192      0 /extra1/50k
       disk_io       cmdk0  R     8192      0 /extra1/50k
       disk_ra       cmdk0  R     8192      8 /extra1/50k
   sc-read               .  R     8192      8 /extra1/50k
     fop_read            .  R     8192      8 /extra1/50k
       disk_ra       cmdk0  R    34816     16 /extra1/50k
   sc-read               .  R     8192     16 /extra1/50k
     fop_read            .  R     8192     16 /extra1/50k
   sc-read               .  R     8192     24 /extra1/50k
     fop_read            .  R     8192     24 /extra1/50k
   sc-read               .  R     8192     32 /extra1/50k
     fop_read            .  R     8192     32 /extra1/50k
   sc-read               .  R     8192     40 /extra1/50k
     fop_read            .  R     8192     40 /extra1/50k
   sc-read               .  R     8192     48 /extra1/50k
     fop_read            .  R     8192     48 /extra1/50k
   sc-read               .  R     8192     50 /extra1/50k
     fop_read            .  R     8192     50 /extra1/50k
   ^C

By looking closely at the Offset (Kbytes) and Size of each transaction, we
can see how the read() system calls (sc-read) were satisfied by the file
system. There were 8 read() system calls, and 3 disk events - 2 of which were
UFS read-ahead (disk_ra). The final read-ahead was for 34 Kbytes and began
with an offset of 16 Kbytes, which read the remaining file data (34 + 16 = 50
Kbytes). The subsequent read() system calls and corresponding fop_read() calls
returned from the page cache.
...

[edit] `Docs/FAQs` directory

The DTraceToolkit has man pages for formal documentation and example files for verbose examples; however neither of these places allow miscellaneous tool nuances to be listed, the sort that you need to know to understand the output of, say, sar. The Docs/FAQs directory serves this purpose by containing individual tool FAQs, and is a dumping ground for miscellaneous notes. For example:

DTraceToolkit-0.97> more Docs/FAQs/ALLsnoop_faq.txt
The following are additional notes on ALL of the *snoop programs (such as
execsnoop, iosnoop, ..., and dapptrace, dtruss),


* The output seems shuffled?

Beware - due to the way DTrace works, on multi-CPU systems there is no
guarentee that if you print traced events the output is in the same order that
the events occured.

This is because events details are placed in per-CPU buffers, and then
combined by the DTrace consumer (/usr/sbin/dtrace) and printed out. The
DTrace consumer reads and prints the buffers one by one, it doesn't combine
them and sort them.
...

[edit] `Bin` directory

This directory contains symlinks to all scripts. This directory is handy to use for grepping for examples of DTrace functions, as it links to over 100 scripts. Here we search for examples of lquantize(),

DTraceToolkit-0.97/Bin> grep lquantize *
cpuwalk.d:      @sample[pid, execname] = lquantize(cpu, 0, MAXCPUID, 1);
dexplorer:              @length[cpu] = lquantize(this->num, 0, 100, 1);
diskhits:       @Location = lquantize(this->kb, 0, FILE_KB_MAX, SCALE_KB);
dispqlen.d:         lquantize(curthread->t_cpu->cpu_disp->disp_nrunnable, 0, 64, 1);
dnlcps.d:       @Result[execname, pid] = lquantize(this->code, 0, 1, 1); 
...

[edit] Testing

Each script is tested for a variety of workloads on a variety of systems. Where possible, a known workload is created and the numbers are compared to what DTrace has measured. Far more effort goes into testing the scripts than actually writing them.

[edit] Performance Impact

Enabling DTrace to monitor events has little effect on the system, especially when compared to the behaviour of truss (See DTrace vs truss for a comparison).

It really boils down to how often the events occur that you are monitoring. The following numbers have been provided as an approximation:

Fixed rate scripts

Usually scripts that use the profile::: provider. For example, dispqlen.d samples at 1000 hz. The impact will be negligible, close to 0% CPU. (in testing, 0.1% CPU).

Demand rated scripts

For example, iosnoop probes disk I/O events. The impact depends on the rate of events, for many servers the disk events would be slow enough for this to be less than 0.2% CPU. Scripts such as execsnoop would expect even fewer events, their impact would be close to 0.0% CPU.

Scripts that monitor potentially very rapid events will have a greater impact, for example running dapptrace on Xorg (over 6000 lines of output per second) was consuming around 10% of a CPU to do so. Both dapptrace and dappprof can consume much more, if the event frequency is higher.

Heavyweight scripts

A few scripts in the toolkit must probe either a ton of different events, or very rapid events, or both. They are going to hurt and there is no way around it. The worst would be cputimes and cpudists, they chew around 5% of the CPUs.

[edit] Support

As the DTraceToolkit is an open source product, and there is no official company offering support for this. Sun Microsystems does not support this. If you post messages to the OpenSolaris DTrace discuss mailing list, a volunteer may help you out.

[edit] Using the DTraceToolkit

[edit] Top Scripts to Run

If you have the DTraceToolkit on a misbehaving server and you don't know where to start, the following list of tools will provide the most valuable info in the shortest time:

./execsnoop -v
./iosnoop
./opensnoop -e
./errinfo -c
./procsystime -aT
./iotop -PCt8
./rwtop -Ct8
./Disk/iopattern 1

execsnoop

Look for many processes being executed quickly, as many short lived processes add considerable overhead.

iosnoop

Watch what is happening on the disks for any unexpected activity. Check who is using the disks and the size of the disk events.

opensnoop

Run opensnoop, learn lots. It is amazing what interesting problems opensnoop has unearthed. Many things are files (config files, data files, device files, libraries), and watching the open events provides a useful insight on what is really happening.

errinfo

Perhaps you have a performance issue due to an application "silently failing". errinfo will show which app is experiencing what type of syscall failure, and the frequency. Of course, the app may take these errors into account - and there may be no problem. Best to check anyway.

procsystime

This will show elapsed time, on-CPU time and counts for system calls. Very useful, and either -p PID or -n execname can be used to narrow examination to your target application only.

iotop

If disk events occured too quickly, iotop can provide a rolling summary.

rwtop

Rather than looking at the disk event level (iosnoop, iotop), rwtop examines at the syscall layer. This is application I/O. Much of this may be to the file system cache, some may make it to disk, and some may be for IPC.

Disk/iopattern

There are several scripts in the Disk directory for further analysis. iopattern provides several useful metrics for understanding disk access behaviour.

[edit] Cool Scripts

See Docs/Examples for demos of these scripts.

bitesize.d
fddist.d
rwbbypid.d
fsrw.d
iofile.d
iopending.d
pathopens.d
pfilestat
rfileio.d
threaded.d
dispqlen.d
pridist.d
nfswizard.d
dvmstat
shellsnoop

[edit] Presentations

[edit] Brendan Gregg

The slides used for a DTrace Workshop in London have been put online by Context-Switch. This DTrace presentation begins with a summary of performance monitoring tools in Solaris 9 and their limitations, covers the DTrace language, and introduces the DTraceToolkit.

The slides also document many of Brendan's strategies for DTrace analysis. They are,

DTrace Strategies

Snoop or Summary
Drill Down Analysis
Frequency Count
Known Count
Aggregate Stacks
Elapsed Time
On-CPU Time
Milestones

[edit] Stefan Parvu

Im posting a presentation about DTrace framework, found under Solaris 10, and DTraceToolkit: a big collection of D scripts. This presentation is available to anyone under the terms of the PUBLIC DOCUMENTATION LICENSE (PDL), Version 1.01. You can obtain a copy of the license at http://www.opensolaris.org/os/licensing/pdl

I'm working to add more examples to the current presentation: DTT for Database Administrators, Java programmers gather as supplements. The current version is 0.96 and does not include the supplements.

There is currently some effort to translate this presentation into Japanese and Romanian language.

Download DTT 0.96 English

[edit] Courses

There are currently no courses that are 100% DTraceToolkit based. There are courses which cover DTrace, and may discuss the DTraceToolkit as appropriate.

[edit] Sun's DTrace Course: SA-327-S10

[edit] Context Switch's DTrace Workshop

[edit] DTraceToolkit in Finland

For folks living in Finland: If you are interested about having 1,2 days class about DTT please let me know: stefan.parvu@sun.com or stefanparvu14@yahoo.com. The course can be organized at your site and can be easily adjusted to meet your requirements. For more details please visit the Course Overview section (Work in Progress)

[edit] See Also

DTrace Topics is a collection of DTrace how-tos hosted here on the Performance wiki.
DTrace Topics: DTraceToolkit - PDF presentation of this page

Retrieved from "http://www.solarisinternals.com/wiki/index.php/DTraceToolkit"

DTraceToolkit

From Siwiki

Contents

[edit] DTraceToolkit

[edit] Download the DTraceToolkit

[edit] About the DTraceToolkit

[edit] Major Components

[edit] Contents

[edit] Scripts

[edit] One Liners

[edit] `Man` directory

[edit] `Docs/Examples` directory

[edit] `Docs/FAQs` directory

[edit] `Bin` directory

[edit] Testing

[edit] Performance Impact

[edit] Support

[edit] Using the DTraceToolkit

[edit] Top Scripts to Run

[edit] Cool Scripts

[edit] Presentations

[edit] Brendan Gregg

[edit] Stefan Parvu

[edit] Courses

[edit] Sun's DTrace Course: SA-327-S10

[edit] Context Switch's DTrace Workshop

[edit] DTraceToolkit in Finland

[edit] See Also

Views

Personal tools

Navigation

Forums

Communities

Search

The Books

Toolbox

The Ads

DTraceToolkit

From Siwiki

Contents

[edit] DTraceToolkit

[edit] Download the DTraceToolkit

[edit] About the DTraceToolkit

[edit] Major Components

[edit] Contents

[edit] Scripts

[edit] One Liners

[edit] Man directory

[edit] Docs/Examples directory

[edit] Docs/FAQs directory

[edit] Bin directory

[edit] Testing

[edit] Performance Impact

[edit] Support

[edit] Using the DTraceToolkit

[edit] Top Scripts to Run

[edit] Cool Scripts

[edit] Presentations

[edit] Brendan Gregg

[edit] Stefan Parvu

[edit] Courses

[edit] Sun's DTrace Course: SA-327-S10

[edit] Context Switch's DTrace Workshop

[edit] DTraceToolkit in Finland

[edit] See Also

Views

Personal tools

Navigation

Forums

Communities

Search

The Books

Toolbox

The Ads

[edit] `Man` directory

[edit] `Docs/Examples` directory

[edit] `Docs/FAQs` directory

[edit] `Bin` directory