DTrace Topics Cheatsheets Checklists

From Siwiki

Jump to: navigation, search

Contents

[edit] DTrace Topics: Cheatsheets and Checklists

This article is about DTrace, and is part of the DTrace Topics collection. A general understanding of DTrace is assumed knowledge, which can be studied from the DTrace Topics: Intro section.

DTrace is a dynamic troubleshooting and analysis tool first introduced in the Solaris 10 and OpenSolaris operating systems.

Completion Image:trafficlight_red02.png
Difficulty Image:coffeemug01.png Image:coffeemug01.png Image:coffeemug01.png
Audience DTrace users

[edit] Cheatsheets

Some of the following cheatsheets have been sourced from the DTrace Guide and which is the official reference for this information, and Bryan's DTrace Quick Reference

[edit] DTrace USAGE

# dtrace  
Usage: dtrace [-32|-64] [-aACeFGhHlqSvVwZ] [-b bufsz] [-c cmd] [-D name[=def]]
        [-I path] [-L path] [-o output] [-p pid] [-s script] [-U name]
        [-x opt[=val]] [-X a|c|s|t]

        [-P provider [[ predicate ] action ]]
        [-m [ provider: ] module [[ predicate ] action ]]
        [-f [[ provider: ] module: ] func [[ predicate ] action ]]
        [-n [[[ provider: ] module: ] func: ] name [[ predicate ] action ]]
        [-i probe-id [[ predicate ] action ]] [ args ... ]

        predicate -> '/' D-expression '/'
           action -> '{' D-statements '}'

        -32 generate 32-bit D programs and ELF files
        -64 generate 64-bit D programs and ELF files

        -a  claim anonymous tracing state
        -A  generate driver.conf(4) directives for anonymous tracing
        -b  set trace buffer size
        -c  run specified command and exit upon its completion
        -C  run cpp(1) preprocessor on script files
        -D  define symbol when invoking preprocessor
        -e  exit after compiling request but prior to enabling probes
        -f  enable or list probes matching the specified function name
        -F  coalesce trace output by function
        -G  generate an ELF file containing embedded dtrace program
        -h  generate a header file with definitions for static probes
        -H  print included files when invoking preprocessor
        -i  enable or list probes matching the specified probe id
        -I  add include directory to preprocessor search path
        -l  list probes matching specified criteria
        -L  add library directory to library search path
        -m  enable or list probes matching the specified module name
        -n  enable or list probes matching the specified probe name
        -o  set output file
        -p  grab specified process-ID and cache its symbol tables
        -P  enable or list probes matching the specified provider name
        -q  set quiet mode (only output explicitly traced data)
        -s  enable or list probes according to the specified D script
        -S  print D compiler intermediate code
        -U  undefine symbol when invoking preprocessor
        -v  set verbose mode (report stability attributes, arguments)
        -V  report DTrace API version
        -w  permit destructive actions
        -x  enable or modify compiler and tracing options
        -X  specify ISO C conformance settings for preprocessor
        -Z  permit probe descriptions that match zero probes

[edit] Providers

DTrace Providers are libraries of related probes.

Name Comments
dtrace The meta-DTrace provider. Provides probes related to the recording of DTrace probes:
  • BEGIN. Probe triggered before any other probe. The BEGIN probe is triggered once, and on an arbitrary CPU.
  • END. Probe triggered after tracing has stopped due to an explicit termination of tracing from the DTrace consumer. When using dtrace(1M), the END probe is triggered after receipt of a SIGINT or SIGTERM (e.g., ^C). The END probe is not triggered after an exit action.
  • ERROR. Probe triggered after an error has occurred. If the ERROR action triggers an error, no further ERROR probe will be triggered. Some knowledge of the DTrace implementation is required to process ERROR probes: arg0 is a pointer to the current dtrace_state_t, arg1 is the enabled probe ID, arg2 is a pointer to the current dtrace_probe_t.
fbt The Function Boundary Tracing provider. Provides a probe, entry, for every function entry and a probe, return, for every function return in the kernel. Note that nothing is done by FBT to mitigate the counterintuitive effects of tail-call optimization: in the event of this optimization, one will see the return probe of the caller before the entry probe of the callee.
lockstat The lock statistics provider. Provides probes relating to the acquisition of, contention on, and release of synchronization primitives. Probes are provided for each of adaptive mutexes, spin locks, readers/writer locks, and dispatcher locks. The lockstat(1M) command uses this provider to obtain its raw data.
profile The Profile provider. Provides a series of probes named profile-n, where n denotes the per-second interrupt rate of the probe: when enabled the profile probe fires at high interrupt level (level-14) n times per second. The address of the interrupted instruction is available in arg0. A set of default rates is provided; if a custom rate is desired, one should simply enable "profile-desired-rate": the profile probe of the designated rate will be create automatically.
sdt The Statically Defined Tracing provider. Provides probes explicitly defined with a variant of the DTRACE_PROBE macro. Currently the only SDT-provided probes are interrupt-start, interrupt-complete, and nfs-dispatch.
syscall The system call provider. Provides a probe, entry, for every system call entry and a probe, return, for every system call return. Unlike FBT, the syscall provider is not affected by the presence of tail-call optimization.
sysinfo The sysinfo provider. Provides a probe at every point where a member of the cpu_sysinfo structure is changed. The members of this structure (defined in /usr/include/sys/sysinfo.h) are used to fill in some members of the per-CPU kstat "sys". Using this provider's probes, one may determine where (and why) a specific member of this kstat is being changed.
vminfo The sysinfo provider. Provides a probe at every point where a member of the cpu_vminfo structure is changed. The members of this structure (defined in /usr/include/sys/vminfo.h) are used to fill in some members of the per-CPU kstat "sys". Using this provider's probes, one may determine where (and why) a specific member of this kstat is being changed.
vtrace The VTRACE provider. Provides a probe at every legacy VTRACE point. (VTRACE is a tracing framework originally developed in SunOS and subsequently ported to Solaris. It has historically required a special "TRACE" build.) The format string specified with the VTRACE probe is ignored. VTRACE is only available in DEBUG kernels.

[edit] Functions

The following functions can be called in the D programming language when a probe fires.

[edit] Tracing Functions

Name Arguments Comments
breakpoint void Force the kernel to breakpoint and enter the debugger. This action is equivalent to forcing the kernel thread to call debug_enter().
chill int nsecs Delay (spin) for the specified number of nanoseconds. One may not delay for longer than 500 million nanoseconds.
commit int id Commit the trace data stored in the speculative trace buffer associated with the given identifier.
discard int id Discard the trace data stored in the speculative trace buffer associated with the given identifier.
exit int status Exit dtrace(1M) command with the specified status when the trace record associated with this enabled probe is processed.
panic void Force the kernel to panic. This action is equivalent to forcing the kernel thread to call panic().
printa const char *format, @aggregation Snapshot the specified aggregation and then print the tuple using the specified printf(3C) format string. If the special %@ format flag is used, the

corresponding conversion prints the aggregation's value. The format is repeated for each tuple in the aggregation.

printf const char *format, ... Trace the specified arguments as if by trace(), and then print them to standard output using the specified printf(3C) format string.
speculate int id Enable speculative tracing for the remainder of the current clause using the speculative buffer identified by id.
stack uint_t depth Trace a maximum of depth stack frames and display each saved %pc as a symbol name + offset.
stop void Force the thread that hit the enabled probe to stop when it next leaves the kernel, as if it had been stopped by /proc. By default, this will cause the entire process to stop as if by pstop(1).
trace D-expr value Trace the value of the specified D expression and report its value. Integer values are displayed in decimal (base 10).
tracemem D-expr addr, size_t len Trace len bytes of raw data at the virtual address specified by the D expression addr.
ustack uint_t depth Trace a maximum of depth stack frames from the user process stack associated with the current thread and display each saved %pc as a symbol name + offset.

[edit] Aggregating Functions

Aggregating functions summarize data in an efficient way, and must be called as the r-val of an aggregation assignment. They are printed using printa.

Name Arguments Comments
avg D-expr Compute a running average of the specified values.
count none Compute a simple count of the specified values.
lquantize D-expr, begin, end [ , step ] Compute a linear quantization of the specified values. The integer begin and end parameters define the range over which the expression should be linearly

quantized. The optional integer step parameter can be used to set the size of each quantization level; the default is one.

max D-expr Compute the maximum of the specified values.
min D-expr Compute the minimum of the specified values.
sum D-expr Compute the sum of the specified values.
quantize D-expr Compute a power-of-two quantization of the specified values.

[edit] Builtin Functions

Name Prototype Comments
copyin void *copyin(uintptr_t addr, size_t size) Copy the specified size in bytes from the specified user address into a DTrace scratch buffer, and return the address of this buffer. The user address is interpreted as an address in the space of the process associated with the current thread. The resulting buffer pointer is guaranteed to have 8-byte alignment. If insufficient scratch space is available, NULL is returned.
copyinstr string copyinstr(uintptr_t addr) Copy a nul-terminated C string from the specified user address into a DTrace scratch buffer, and return the address of this buffer. The user address is interpreted as an address in the space of the process associated with the current thread. The string length is limited to the default DTrace string length limit. If insufficient scratch space is available, NULL is returned.
copyout void copyout(void *buf, uintptr_t addr, size_t size) Copy the specified buf of size bytes from the kernel to the specified user-space addr in the address space of the process associated with the current thread.
copyoutstr void copyoutstr(char *str, uintptr_t addr, size_t len) Copy the nul-terminated C string str from the kernel to the specified user-space addr in the address space of the process associated with the current thread. The string is copied up to and including the terminating nul byte, or to a maximum of len bytes.
mutex_owned int mutex_owned(kmutex_t *) Return 0 if the specified adaptive mutex is currently unowned, or 1 if it is currently held by a thread.
mutex_owner kthread_t *mutex_owner(kmutex_t *) Return the thread pointer of the current owner of the specified adaptive lock. The mutex_owner function returns NULL if the specified adaptive mutex is

currently unowned, or if the specified mutex is a spin mutex.

mutex_type_adaptive int mutex_type_adaptive(kmutex_t *) Return non-zero if the specified mutex is of type MUTEX_ADAPTIVE, or zero if it is not.
mutex_type_spin int mutex_type_spin(kmutex_t *) Return non-zero if the specified mutex is of type MUTEX_SPIN, or zero if it is not.
offsetof size_t offsetof(type-name, member-name) Return the byte offset of the specified member of the specified struct or union type. The value of the function is computed at compile time by the D

compiler. The offsetof function may not be applied to bit-field members.

progenyof int progenyof(pid_t) Return non-zero if the calling process (the process associated with the thread that is currently triggering the matched probe) is among the progeny of the specified process ID.
raise void raise(int) Raise the specified signal on the LWP associated with the current thread. The DTrace raise function does not support signal queuing: only one DTrace signal can be pending at a time.
rand int rand(void) Return a pseudo-random integer.
rw_iswriter int rw_iswriter(krwlock_t *) Return non-zero if the specified reader-writer lock is either held or desired by a writer, or zero otherwise.
rw_read_held int rw_read_held(krwlock_t *) Return non-zero if the specified reader-writer lock is currently held by one or more readers, or zero otherwise.
rw_write_held int rw_write_held(krwlock_t *) Return non-zero if the specified reader-writer lock is currently held by a writer, or zero otherwise.
speculation int speculation(void) Reserve a speculative trace buffer for use with speculate( ) and return an identifier for this buffer.
strlen size_t strlen(const char *) Return the length of the specified string in bytes, excluding the terminating nul byte.

[edit] Variables

[edit] Builtin Variables

Variable Name Data Type Comments
args dynamically typed array of probe arguments For function entry probes, the arguments array will provide access to all function input arguments. For function return probes, the arguments array provides the return offset in args[0], and the return value of the function in args[1]. The args[] array is not completely implemented in the current prototype. You can use the temporary variables "arg0", "arg1", "arg2", "arg3", and "arg4" in the prototype to access arguments as type "void *".
caller uintptr_t The program counter address of the previous function call on the stack of the current thread.
curlwpsinfo lwpsinfo_t * Pointer to the current thread's translated lwpsinfo information. The lwpsinfo data is described in proc(4).
curpsinfo psinfo_t * Pointer to the current process's translated psinfo information. The psinfo data is described in proc(4).
curthread kthread_t * Pointer to the current thread's kthread_t structure.
epid uint_t The Enabled Probe ID (see DTrace KTD Presentation).
errno int The saved errno value associated with the current thread's LWP. The <sys/errno.h> defines (e.g. ENOENT) may also be used as identifiers in a D program.
execname string The file name of the current process's executable, as specified in the first argument to exec(2).
id uint_t The ID of the current probe.
ipl uint_t The current interrupt priority level.
NULL long The value 0 as an integer constant.
pid pid_t The PID of the current process.
probefunc string The name of the current probe's function.
probemod string The name of the current probe's module.
probename string The name of the current probe.
probeprov string The name of the current probe's provider.
self dynamic pointer This special pointer can be used to instantiate or access the value of D Thread-Local Storage (TLS) variables.
stackdepth uint_t The depth of the stack in frames at the time dtrace_probe() was called.
this dynamic pointer This special pointer can be used to instantiate or access the value of D ECB-local variables. Local variables may only be used as scalars; associative arrays are not permitted.
tid id_t The thread ID (LWPID) of the current thread.
timestamp uint64_t High-resolution nanosecond timestamp, as returned by gethrtime(3C).
vtimestamp uint64_t High-resolution nanosecond virtual timestamp, expressed as number of nanoseconds that the current thread has been on-CPU since some arbitrary time in the past. (Only relative differences are meaningful; absolute values should not be interpreted.)

[edit] Macro Variables

Variable Name Data Type Comments
$0, $1, $2, ... integer, identifier, or string The value of the corresponding macro parameter as determined by the arguments passed to dtrace(1M).
$egid integer The effective group id of the dtrace command.
$euid integer The effective user id of the dtrace command.
$gid integer The real group id of the dtrace command.
$pid integer The process id of the dtrace command.
$pgid integer The process group id of the dtrace command.
$ppid integer The parent process id of the dtrace command.
$projid integer The project id of the dtrace command.
$sid integer The session id of the dtrace command.
$taskid integer The task id of the dtrace command.
$uid integer The real user id of the dtrace command.

[edit] Data Types

Integer
Type Name 32–bit Size 64–bit Size
char 1 byte 1 byte
short 2 bytes 2 bytes
int 4 bytes 4 bytes
long 4 bytes 8 bytes
long long 8 bytes 8 bytes
Float
Type Name 32–bit Size 64–bit Size
float 4 bytes 4 bytes
double 8 bytes 8 bytes
long double 16 bytes 16 bytes

[edit] Languages


[edit] C

The following is a quick reference for DTracing the C programming language. See DTrace Topics: C for further details.

[edit] Providers
Provider Relavence
pid Tracing user-level C functions including library functions, and CPU instructions
profile Sampling stack traces
syscall Tracing system calls by user-level code
fbt Tracing kernel C functions (an unstable interface; check if stable versions exist for your events)
io Tracing physical io events
[edit] Probes
Probe Description
pid$target:[segment]:[function]:entry A user-level function is entered
pid$target:[segment]:[function]:return A user-level function completes
pid$target:[segment]:[function]:0x### A user-level instruction fires
fbt:[module]:[function]:entry A kernel function is entered
fbt:[module]:[function]:return A kernel function completes
[edit] Capabilities: user-land
Capability Exists Format Example
Function entries yes Probe: pidPID:[segment]:[function]:entry dtrace -n 'pid$target:a.out:main:entry' -p PID
Function entry arguments yes Variables: arg0, arg1 .. argN, int64_t dtrace -n 'pid$target:libc:close:entry { printf("%d", arg0); }' -p PID
Function returns yes Probe: pidPID:[segment]:[function]:return dtrace -n 'pid$target:libc:fopen:return' -p PID
Function return value yes Variable: arg1, int64_t dtrace -n 'pid$target:libc:fclose:return { trace(arg1); }' -p PID
Function instructions yes Probe: pidPID:[segment]:[function]:[offset] dtrace -n 'pid$target:libc:strlen:' -c date
Stack traces yes Action: ustack() dtrace -n 'pid$target:libc:fopen:entry { ustack() }' -p PID
[edit] Capabilities: kernel
Capability Exists Format Example
Function entries yes Probe: fbt:[module]:[function]:entry dtrace -n 'fbt:genunix:bdev_strategy:entry'
Function entry arguments yes Variables: args[0], args[1] .. args[N], casted dtrace -n 'fbt::bdev_strategy:entry { trace(args[0]->b_bcount); }'
Function returns yes Probe: fbt:[module]:[function]:return dtrace -n 'fbt:genunix:bdev_strategy:entry'
Function return value yes Variable: arg1, int64_t dtrace -n 'fbt::bdev_strategy:return { trace(arg1); }'
Function instructions no[1]
Stack traces yes Action: stack() dtrace -n 'fbt:genunix:bdev_strategy:entry { stack() }'

[1] not directly, although many indirect methods exist. See Limitations.


[edit] Java

The following is a quick reference for DTracing the Java programming language. See DTrace Topics: Java for further details.

[edit] Providers
Provider Relavence
hotspot Tracing JVM events including Java method execution for JDK 1.6
dvm/djvm Tracing JVM events including Java method execution for JDK 1.4.2 and JDK 1.5
profile Sampling Java execution at fixed rates
syscall Tracing system calls that Java code caused
pid Tracing raw JVM operation
[edit] Probes
Probe Description
hotspot*:::method-entry A method begins (needs VM flag ExtendedDTraceProbes)
hotspot*:::method-returns A method completes (needs VM flag ExtendedDTraceProbes)
hotspot*:::object-alloc An object was created (needs VM flag ExtendedDTraceProbes)
hotspot*:::gc-begin System wide GC begins
hotspot*:::gc-ends System wide GC ended
hotspot*:::thread-start A thread begins execution
hotspot*:::thread-stop A thread completes

This is a small selection from about 500 probes available.

[edit] Capabilities
Capability Exists Format Example
Function entries yes Probe: hotspot:::method-entry # dtrace -qn 'hotspot*:::method-entry { printf("%s.%s\n",

stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }'

Function entry arguments no[1]
Function returns yes Probe: hotspot:::method-return # dtrace -qn 'hotspot*:::method-entry { printf("%s.%s\n",

stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }'

Function return value no[1]
Function instructions no[1]
Stack traces yes Action: jstack() dtrace -n 'syscall::read:entry /pid == $target/ { jstack(); }' -p PID

[1] this can probably be achieved with some effort using the pid provider to examine raw JVM execution.


[edit] JavaScript

The following is a quick reference for DTracing the JavaScript programming language. See DTrace Topics: JavaScript for further details.

[edit] Providers
Provider Relavence
mozilla Tracing JavaScript events such as function calls and object allocation
profile For rough analysis, such as looking for libmozjs in stack traces
syscall Tracing system calls that JavaScript code caused
pid Tracing raw JavaScript engine operation
[edit] Probes
Probe Description
*mozilla*:::js_function-entry A JavaScript function began
*mozilla*:::js_function-return A JavaScript function completed
*mozilla*:::js_execute-start JavaScript code began execution
*mozilla*:::js_execute-done JavaScript code completed execution
*mozilla*:::js_object-create A JavaScript object was created
*mozilla*:::js_object-create-start A JavaScript object began creation
*mozilla*:::js_object-create-done A JavaScript object completed creation
*mozilla*:::js_object-finalize A JavaScript object destroyed (GC)
[edit] Capabilities
Capability Exists Format Example
Function entries yes Probe: *mozilla*:::js_function-entry # dtrace -n '*mozilla*:::js_function-entry { trace(copyinstr(arg2)); }'
Function entry arguments no[1]
Function returns yes Probe: hotspot:::method-return # dtrace -n '*mozilla*:::js_function-return { trace(copyinstr(arg2)); }'
Function return value no[1]
Function instructions no[1]
Stack traces no[2]

[1] this may be possible with some effort using the pid provider to examine raw JavaScript engine execution.
[2] same as [1], but even harder.

[edit] Checklists

Solaris Internals
Personal tools
The Books
The Ads