DTrace Topics Errors
From Siwiki
Contents |
[edit] DTrace Topics: Errors
This article is about DTrace error messages, and is part of the DTrace Topics collection. A general understanding of DTrace is assumed knowledge, which can be studied from the DTrace Topics: Intro section.
DTrace is a dynamic troubleshooting and analysis tool first introduced in the Solaris 10 and OpenSolaris operating systems.
Completion 
Difficulty 
Audience All DTrace users
[edit] DTrace Messages
DTrace will print error or warning messages to STDERR in response to certain conditions. This page provides discussion on what each message means, and provides suggestions for avoiding them.
[edit] DTrace Errors
[edit] DTrace requires additional privileges
You must either be root or have additional privileges to be able to use DTrace. Those privileges are:
- dtrace_user - allows the use of profile, syscall and fasttrap providers, on processes that the user owns.
- dtrace_proc - allows the use of the pid provider on processes that the user owns.
- dtrace_kernel - allows most providers to probe everything, in read only mode.
Privileges can be added to a process (such as a user's shell) temporarily by using the ppriv(1) command. For example, to add dtrace_user to PID 1851,
ppriv -s A+dtrace_user 1851
usermod can be used to make this a permanent change to a user account. For example,
usermod -K defaultpriv=basic,dtrace_user brendan
[edit] invalid address (0x...) in action
# dtrace -n 'syscall::open:entry { trace(stringof(arg0)); }'
dtrace: description 'syscall::open:entry ' matched 1 probe
dtrace: error on enabled probe ID 1 (ID 6329: syscall::open:entry):
invalid address (0xd27f7a24) in action #1
dtrace: error on enabled probe ID 1 (ID 6329: syscall::open:entry):
invalid address (0xd27fbf38) in action #1
This error is caused when DTrace attempts to dereference a memory address which isn't mapped. In the above example, the arg0 variable for the open(2) syscall refers to a user-land address, however DTrace executes in the kernel address space; this example can be fixed by changing stringof to copyinstr. Listing remedies:
- Use either
copyin()orcopyinstr()to copy the data from user-land into the kernel. - Attempt to dereference on the return of a function, not the entry. On the entry, an address may be valid but not faulted in.
[edit] failed to create probe ... Not enough space
DTrace ran out of RAM when trying to create probes. This can happen if you attempt to probe far too many events. For example, here we leave fields blank in our probe description (wildcards), and so our probe description will attempt to match every instruction from every function of mozilla (which would be millions of probes),
# dtrace -ln 'pid$target:::' -p `pgrep mozilla-bin` dtrace: invalid probe specifier pid$target:::: failed to create probe in process 7424: Not enough space #
In this case, perhaps we meant to probe just function entries - pid$target:::entry, or perhaps instructions from just one library - pid$target:libaio::.
If you are using the pid provider you can increase the max number of probes by modifying the /kernel/drv/fasttrap.conf file to something similar to the following. (notice the fasttrap-max-probes line)
# # Copyright 2004 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # #ident "@(#)fasttrap.conf 1.2 04/11/07 SMI" name="fasttrap" parent="pseudo" instance=0; # fasttrap-max-probes is the limit on the number of tracepoints created # by DTrace's pid provider. This value should be increased if DTrace is # unable to create the desired probes because the limit has been hit. #fasttrap-max-probes=250000; fasttrap-max-probes=1000000; # fasttrap-hash-size determines the size of the hash table used to store # enabled DTrace pid provider tracepoints. If there are many enabled # tracepoints or many tracepoints hash to the same value, increasing this # variable can improve the performance of executing a traced instruction. #fasttrap-hash-size=16384;
After editing you may need to run "update_drv fasttrap" or reboot.
[edit] DTrace Warnings
[edit] drops on CPU #
dtrace: 864476 drops on CPU 0 dtrace: 2179050 drops on CPU 0 dtrace: 1343451 drops on CPU 0
The DTrace kernel buffer is overflowing due to output being generated too quickly for /usr/sbin/dtrace to read. This usually happens when your script would output hundreds of screens of text per second. Some remedies:
- Increase the switchrate of /usr/sbin/dtrace, so that rather than flushing the buffer at 1 Hertz (default), it is reading the buffer faster. At the command line this can be
-x switchrate=10hz. - Increase the size of the DTrace primary buffer. By default this is usually 4 Mbytes per CPU. At the command line it can be increased, eg
-b 8m. - Do you really want that much data to be output? Try to probe fewer events. Also, aggregations can be used so that DTrace can summarise the data and output the the final report, avoiding an output buffer overflow.
