DTrace Topics Cheatsheets Checklists
From Siwiki
Contents |
[edit] DTrace Topics: Cheatsheets and Checklists
This article is about DTrace, and is part of the DTrace Topics collection. A general understanding of DTrace is assumed knowledge, which can be studied from the DTrace Topics: Intro section.
DTrace is a dynamic troubleshooting and analysis tool first introduced in the Solaris 10 and OpenSolaris operating systems.
Completion 
Difficulty 
Audience DTrace users
[edit] Cheatsheets
Some of the following cheatsheets have been sourced from the DTrace Guide and which is the official reference for this information, and Bryan's DTrace Quick Reference
[edit] DTrace USAGE
# dtrace
Usage: dtrace [-32|-64] [-aACeFGhHlqSvVwZ] [-b bufsz] [-c cmd] [-D name[=def]]
[-I path] [-L path] [-o output] [-p pid] [-s script] [-U name]
[-x opt[=val]] [-X a|c|s|t]
[-P provider [[ predicate ] action ]]
[-m [ provider: ] module [[ predicate ] action ]]
[-f [[ provider: ] module: ] func [[ predicate ] action ]]
[-n [[[ provider: ] module: ] func: ] name [[ predicate ] action ]]
[-i probe-id [[ predicate ] action ]] [ args ... ]
predicate -> '/' D-expression '/'
action -> '{' D-statements '}'
-32 generate 32-bit D programs and ELF files
-64 generate 64-bit D programs and ELF files
-a claim anonymous tracing state
-A generate driver.conf(4) directives for anonymous tracing
-b set trace buffer size
-c run specified command and exit upon its completion
-C run cpp(1) preprocessor on script files
-D define symbol when invoking preprocessor
-e exit after compiling request but prior to enabling probes
-f enable or list probes matching the specified function name
-F coalesce trace output by function
-G generate an ELF file containing embedded dtrace program
-h generate a header file with definitions for static probes
-H print included files when invoking preprocessor
-i enable or list probes matching the specified probe id
-I add include directory to preprocessor search path
-l list probes matching specified criteria
-L add library directory to library search path
-m enable or list probes matching the specified module name
-n enable or list probes matching the specified probe name
-o set output file
-p grab specified process-ID and cache its symbol tables
-P enable or list probes matching the specified provider name
-q set quiet mode (only output explicitly traced data)
-s enable or list probes according to the specified D script
-S print D compiler intermediate code
-U undefine symbol when invoking preprocessor
-v set verbose mode (report stability attributes, arguments)
-V report DTrace API version
-w permit destructive actions
-x enable or modify compiler and tracing options
-X specify ISO C conformance settings for preprocessor
-Z permit probe descriptions that match zero probes
[edit] Providers
DTrace Providers are libraries of related probes.
| Name | Comments |
| dtrace | The meta-DTrace provider. Provides probes related to the recording of DTrace probes:
|
| fbt | The Function Boundary Tracing provider. Provides a probe, entry, for every function entry and a probe, return, for every function return in the kernel. Note that nothing is done by FBT to mitigate the counterintuitive effects of tail-call optimization: in the event of this optimization, one will see the return probe of the caller before the entry probe of the callee. |
| lockstat | The lock statistics provider. Provides probes relating to the acquisition of, contention on, and release of synchronization primitives. Probes are provided for each of adaptive mutexes, spin locks, readers/writer locks, and dispatcher locks. The lockstat(1M) command uses this provider to obtain its raw data. |
| profile | The Profile provider. Provides a series of probes named profile-n, where n denotes the per-second interrupt rate of the probe: when enabled the profile probe fires at high interrupt level (level-14) n times per second. The address of the interrupted instruction is available in arg0. A set of default rates is provided; if a custom rate is desired, one should simply enable "profile-desired-rate": the profile probe of the designated rate will be create automatically. |
| sdt | The Statically Defined Tracing provider. Provides probes explicitly defined with a variant of the DTRACE_PROBE macro. Currently the only SDT-provided probes are interrupt-start, interrupt-complete, and nfs-dispatch. |
| syscall | The system call provider. Provides a probe, entry, for every system call entry and a probe, return, for every system call return. Unlike FBT, the syscall provider is not affected by the presence of tail-call optimization. |
| sysinfo | The sysinfo provider. Provides a probe at every point where a member of the cpu_sysinfo structure is changed. The members of this structure (defined in /usr/include/sys/sysinfo.h) are used to fill in some members of the per-CPU kstat "sys". Using this provider's probes, one may determine where (and why) a specific member of this kstat is being changed. |
| vminfo | The sysinfo provider. Provides a probe at every point where a member of the cpu_vminfo structure is changed. The members of this structure (defined in /usr/include/sys/vminfo.h) are used to fill in some members of the per-CPU kstat "sys". Using this provider's probes, one may determine where (and why) a specific member of this kstat is being changed. |
| vtrace | The VTRACE provider. Provides a probe at every legacy VTRACE point. (VTRACE is a tracing framework originally developed in SunOS and subsequently ported to Solaris. It has historically required a special "TRACE" build.) The format string specified with the VTRACE probe is ignored. VTRACE is only available in DEBUG kernels. |
[edit] Functions
The following functions can be called in the D programming language when a probe fires.
[edit] Tracing Functions
| Name | Arguments | Comments |
| breakpoint | void | Force the kernel to breakpoint and enter the debugger. This action is equivalent to forcing the kernel thread to call debug_enter(). |
| chill | int nsecs | Delay (spin) for the specified number of nanoseconds. One may not delay for longer than 500 million nanoseconds. |
| commit | int id | Commit the trace data stored in the speculative trace buffer associated with the given identifier. |
| discard | int id | Discard the trace data stored in the speculative trace buffer associated with the given identifier. |
| exit | int status | Exit dtrace(1M) command with the specified status when the trace record associated with this enabled probe is processed. |
| panic | void | Force the kernel to panic. This action is equivalent to forcing the kernel thread to call panic(). |
| printa | const char *format, @aggregation | Snapshot the specified aggregation and then print the tuple using the specified printf(3C) format string. If the special %@ format flag is used, the
corresponding conversion prints the aggregation's value. The format is repeated for each tuple in the aggregation. |
| printf | const char *format, ... | Trace the specified arguments as if by trace(), and then print them to standard output using the specified printf(3C) format string. |
| speculate | int id | Enable speculative tracing for the remainder of the current clause using the speculative buffer identified by id. |
| stack | uint_t depth | Trace a maximum of depth stack frames and display each saved %pc as a symbol name + offset. |
| stop | void | Force the thread that hit the enabled probe to stop when it next leaves the kernel, as if it had been stopped by /proc. By default, this will cause the entire process to stop as if by pstop(1). |
| trace | D-expr value | Trace the value of the specified D expression and report its value. Integer values are displayed in decimal (base 10). |
| tracemem | D-expr addr, size_t len | Trace len bytes of raw data at the virtual address specified by the D expression addr. |
| ustack | uint_t depth | Trace a maximum of depth stack frames from the user process stack associated with the current thread and display each saved %pc as a symbol name + offset. |
[edit] Aggregating Functions
Aggregating functions summarize data in an efficient way, and must be called as the r-val of an aggregation assignment. They are printed using printa.
| Name | Arguments | Comments |
| avg | D-expr | Compute a running average of the specified values. |
| count | none | Compute a simple count of the specified values. |
| lquantize | D-expr, begin, end [ , step ] | Compute a linear quantization of the specified values. The integer begin and end parameters define the range over which the expression should be linearly
quantized. The optional integer step parameter can be used to set the size of each quantization level; the default is one. |
| max | D-expr | Compute the maximum of the specified values. |
| min | D-expr | Compute the minimum of the specified values. |
| sum | D-expr | Compute the sum of the specified values. |
| quantize | D-expr | Compute a power-of-two quantization of the specified values. |
[edit] Builtin Functions
| Name | Prototype | Comments |
| copyin | void *copyin(uintptr_t addr, size_t size) | Copy the specified size in bytes from the specified user address into a DTrace scratch buffer, and return the address of this buffer. The user address is interpreted as an address in the space of the process associated with the current thread. The resulting buffer pointer is guaranteed to have 8-byte alignment. If insufficient scratch space is available, NULL is returned. |
| copyinstr | string copyinstr(uintptr_t addr) | Copy a nul-terminated C string from the specified user address into a DTrace scratch buffer, and return the address of this buffer. The user address is interpreted as an address in the space of the process associated with the current thread. The string length is limited to the default DTrace string length limit. If insufficient scratch space is available, NULL is returned. |
| copyout | void copyout(void *buf, uintptr_t addr, size_t size) | Copy the specified buf of size bytes from the kernel to the specified user-space addr in the address space of the process associated with the current thread. |
| copyoutstr | void copyoutstr(char *str, uintptr_t addr, size_t len) | Copy the nul-terminated C string str from the kernel to the specified user-space addr in the address space of the process associated with the current thread. The string is copied up to and including the terminating nul byte, or to a maximum of len bytes. |
| mutex_owned | int mutex_owned(kmutex_t *) | Return 0 if the specified adaptive mutex is currently unowned, or 1 if it is currently held by a thread. |
| mutex_owner | kthread_t *mutex_owner(kmutex_t *) | Return the thread pointer of the current owner of the specified adaptive lock. The mutex_owner function returns NULL if the specified adaptive mutex is
currently unowned, or if the specified mutex is a spin mutex. |
| mutex_type_adaptive | int mutex_type_adaptive(kmutex_t *) | Return non-zero if the specified mutex is of type MUTEX_ADAPTIVE, or zero if it is not. |
| mutex_type_spin | int mutex_type_spin(kmutex_t *) | Return non-zero if the specified mutex is of type MUTEX_SPIN, or zero if it is not. |
| offsetof | size_t offsetof(type-name, member-name) | Return the byte offset of the specified member of the specified struct or union type. The value of the function is computed at compile time by the D
compiler. The offsetof function may not be applied to bit-field members. |
| progenyof | int progenyof(pid_t) | Return non-zero if the calling process (the process associated with the thread that is currently triggering the matched probe) is among the progeny of the specified process ID. |
| raise | void raise(int) | Raise the specified signal on the LWP associated with the current thread. The DTrace raise function does not support signal queuing: only one DTrace signal can be pending at a time. |
| rand | int rand(void) | Return a pseudo-random integer. |
| rw_iswriter | int rw_iswriter(krwlock_t *) | Return non-zero if the specified reader-writer lock is either held or desired by a writer, or zero otherwise. |
| rw_read_held | int rw_read_held(krwlock_t *) | Return non-zero if the specified reader-writer lock is currently held by one or more readers, or zero otherwise. |
| rw_write_held | int rw_write_held(krwlock_t *) | Return non-zero if the specified reader-writer lock is currently held by a writer, or zero otherwise. |
| speculation | int speculation(void) | Reserve a speculative trace buffer for use with speculate( ) and return an identifier for this buffer. |
| strlen | size_t strlen(const char *) | Return the length of the specified string in bytes, excluding the terminating nul byte. |
[edit] Variables
[edit] Builtin Variables
| Variable Name | Data Type | Comments |
| args | dynamically typed array of probe arguments | For function entry probes, the arguments array will provide access to all function input arguments. For function return probes, the arguments array provides the return offset in args[0], and the return value of the function in args[1]. The args[] array is not completely implemented in the current prototype. You can use the temporary variables "arg0", "arg1", "arg2", "arg3", and "arg4" in the prototype to access arguments as type "void *". |
| caller | uintptr_t | The program counter address of the previous function call on the stack of the current thread. |
| curlwpsinfo | lwpsinfo_t * | Pointer to the current thread's translated lwpsinfo information. The lwpsinfo data is described in proc(4). |
| curpsinfo | psinfo_t * | Pointer to the current process's translated psinfo information. The psinfo data is described in proc(4). |
| curthread | kthread_t * | Pointer to the current thread's kthread_t structure. |
| epid | uint_t | The Enabled Probe ID (see DTrace KTD Presentation). |
| errno | int | The saved errno value associated with the current thread's LWP. The <sys/errno.h> defines (e.g. ENOENT) may also be used as identifiers in a D program. |
| execname | string | The file name of the current process's executable, as specified in the first argument to exec(2). |
| id | uint_t | The ID of the current probe. |
| ipl | uint_t | The current interrupt priority level. |
| NULL | long | The value 0 as an integer constant. |
| pid | pid_t | The PID of the current process. |
| probefunc | string | The name of the current probe's function. |
| probemod | string | The name of the current probe's module. |
| probename | string | The name of the current probe. |
| probeprov | string | The name of the current probe's provider. |
| self | dynamic pointer | This special pointer can be used to instantiate or access the value of D Thread-Local Storage (TLS) variables. |
| stackdepth | uint_t | The depth of the stack in frames at the time dtrace_probe() was called. |
| this | dynamic pointer | This special pointer can be used to instantiate or access the value of D ECB-local variables. Local variables may only be used as scalars; associative arrays are not permitted. |
| tid | id_t | The thread ID (LWPID) of the current thread. |
| timestamp | uint64_t | High-resolution nanosecond timestamp, as returned by gethrtime(3C). |
| vtimestamp | uint64_t | High-resolution nanosecond virtual timestamp, expressed as number of nanoseconds that the current thread has been on-CPU since some arbitrary time in the past. (Only relative differences are meaningful; absolute values should not be interpreted.) |
[edit] Macro Variables
| Variable Name | Data Type | Comments |
| $0, $1, $2, ... | integer, identifier, or string | The value of the corresponding macro parameter as determined by the arguments passed to dtrace(1M). |
| $egid | integer | The effective group id of the dtrace command. |
| $euid | integer | The effective user id of the dtrace command. |
| $gid | integer | The real group id of the dtrace command. |
| $pid | integer | The process id of the dtrace command. |
| $pgid | integer | The process group id of the dtrace command. |
| $ppid | integer | The parent process id of the dtrace command. |
| $projid | integer | The project id of the dtrace command. |
| $sid | integer | The session id of the dtrace command. |
| $taskid | integer | The task id of the dtrace command. |
| $uid | integer | The real user id of the dtrace command. |
[edit] Data Types
| Type Name | 32–bit Size | 64–bit Size |
| char | 1 byte | 1 byte |
| short | 2 bytes | 2 bytes |
| int | 4 bytes | 4 bytes |
| long | 4 bytes | 8 bytes |
| long long | 8 bytes | 8 bytes |
| Type Name | 32–bit Size | 64–bit Size |
| float | 4 bytes | 4 bytes |
| double | 8 bytes | 8 bytes |
| long double | 16 bytes | 16 bytes |
[edit] Languages
[edit] C
The following is a quick reference for DTracing the C programming language. See DTrace Topics: C for further details.
[edit] Providers
| Provider | Relavence |
| pid | Tracing user-level C functions including library functions, and CPU instructions |
| profile | Sampling stack traces |
| syscall | Tracing system calls by user-level code |
| fbt | Tracing kernel C functions (an unstable interface; check if stable versions exist for your events) |
| io | Tracing physical io events |
[edit] Probes
| Probe | Description |
| pid$target:[segment]:[function]:entry | A user-level function is entered |
| pid$target:[segment]:[function]:return | A user-level function completes |
| pid$target:[segment]:[function]:0x### | A user-level instruction fires |
| fbt:[module]:[function]:entry | A kernel function is entered |
| fbt:[module]:[function]:return | A kernel function completes |
[edit] Capabilities: user-land
| Capability | Exists | Format | Example |
| Function entries | yes | Probe: pidPID:[segment]:[function]:entry | dtrace -n 'pid$target:a.out:main:entry' -p PID |
| Function entry arguments | yes | Variables: arg0, arg1 .. argN, int64_t | dtrace -n 'pid$target:libc:close:entry { printf("%d", arg0); }' -p PID |
| Function returns | yes | Probe: pidPID:[segment]:[function]:return | dtrace -n 'pid$target:libc:fopen:return' -p PID |
| Function return value | yes | Variable: arg1, int64_t | dtrace -n 'pid$target:libc:fclose:return { trace(arg1); }' -p PID |
| Function instructions | yes | Probe: pidPID:[segment]:[function]:[offset] | dtrace -n 'pid$target:libc:strlen:' -c date |
| Stack traces | yes | Action: ustack() | dtrace -n 'pid$target:libc:fopen:entry { ustack() }' -p PID |
[edit] Capabilities: kernel
| Capability | Exists | Format | Example |
| Function entries | yes | Probe: fbt:[module]:[function]:entry | dtrace -n 'fbt:genunix:bdev_strategy:entry' |
| Function entry arguments | yes | Variables: args[0], args[1] .. args[N], casted | dtrace -n 'fbt::bdev_strategy:entry { trace(args[0]->b_bcount); }' |
| Function returns | yes | Probe: fbt:[module]:[function]:return | dtrace -n 'fbt:genunix:bdev_strategy:entry' |
| Function return value | yes | Variable: arg1, int64_t | dtrace -n 'fbt::bdev_strategy:return { trace(arg1); }' |
| Function instructions | no[1] | ||
| Stack traces | yes | Action: stack() | dtrace -n 'fbt:genunix:bdev_strategy:entry { stack() }' |
[1] not directly, although many indirect methods exist. See Limitations.
[edit] Java
The following is a quick reference for DTracing the Java programming language. See DTrace Topics: Java for further details.
[edit] Providers
| Provider | Relavence |
| hotspot | Tracing JVM events including Java method execution for JDK 1.6 |
| dvm/djvm | Tracing JVM events including Java method execution for JDK 1.4.2 and JDK 1.5 |
| profile | Sampling Java execution at fixed rates |
| syscall | Tracing system calls that Java code caused |
| pid | Tracing raw JVM operation |
[edit] Probes
| Probe | Description |
| hotspot*:::method-entry | A method begins (needs VM flag ExtendedDTraceProbes) |
| hotspot*:::method-returns | A method completes (needs VM flag ExtendedDTraceProbes) |
| hotspot*:::object-alloc | An object was created (needs VM flag ExtendedDTraceProbes) |
| hotspot*:::gc-begin | System wide GC begins |
| hotspot*:::gc-ends | System wide GC ended |
| hotspot*:::thread-start | A thread begins execution |
| hotspot*:::thread-stop | A thread completes |
This is a small selection from about 500 probes available.
[edit] Capabilities
| Capability | Exists | Format | Example |
| Function entries | yes | Probe: hotspot:::method-entry | # dtrace -qn 'hotspot*:::method-entry { printf("%s.%s\n",
stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }' |
| Function entry arguments | no[1] | ||
| Function returns | yes | Probe: hotspot:::method-return | # dtrace -qn 'hotspot*:::method-entry { printf("%s.%s\n",
stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }' |
| Function return value | no[1] | ||
| Function instructions | no[1] | ||
| Stack traces | yes | Action: jstack() | dtrace -n 'syscall::read:entry /pid == $target/ { jstack(); }' -p PID |
[1] this can probably be achieved with some effort using the pid provider to examine raw JVM execution.
[edit] JavaScript
The following is a quick reference for DTracing the JavaScript programming language. See DTrace Topics: JavaScript for further details.
[edit] Providers
| Provider | Relavence |
| mozilla | Tracing JavaScript events such as function calls and object allocation |
| profile | For rough analysis, such as looking for libmozjs in stack traces |
| syscall | Tracing system calls that JavaScript code caused |
| pid | Tracing raw JavaScript engine operation |
[edit] Probes
| Probe | Description |
| *mozilla*:::js_function-entry | A JavaScript function began |
| *mozilla*:::js_function-return | A JavaScript function completed |
| *mozilla*:::js_execute-start | JavaScript code began execution |
| *mozilla*:::js_execute-done | JavaScript code completed execution |
| *mozilla*:::js_object-create | A JavaScript object was created |
| *mozilla*:::js_object-create-start | A JavaScript object began creation |
| *mozilla*:::js_object-create-done | A JavaScript object completed creation |
| *mozilla*:::js_object-finalize | A JavaScript object destroyed (GC) |
[edit] Capabilities
| Capability | Exists | Format | Example |
| Function entries | yes | Probe: *mozilla*:::js_function-entry | # dtrace -n '*mozilla*:::js_function-entry { trace(copyinstr(arg2)); }' |
| Function entry arguments | no[1] | ||
| Function returns | yes | Probe: hotspot:::method-return | # dtrace -n '*mozilla*:::js_function-return { trace(copyinstr(arg2)); }' |
| Function return value | no[1] | ||
| Function instructions | no[1] | ||
| Stack traces | no[2] |
[1] this may be possible with some effort using the pid provider to examine raw JavaScript engine execution.
[2] same as [1], but even harder.
