SolarisInternals

Categories
OpenSolaris
directiostat
etruss
filebench
filestat
cpustat
mdb
::memstat
memtool
pmap
prstat
ptools
se toolkit
statit
taz
tnf
trapstat
vmstat/memstat
MemTool

The pointer to the package and README for the latest versions are available from the autoresponder memtool-request@sun.com  

Documentation

 

Examples

README



The most recent documentation discussing Solaris Virtual Memory can be
found in the Solaris Internals book co-authored with Jim Mauro:

	     http://www.solarisinternals.com


               MemTool 3.9.5beta Release Notes

                        --OOO--

04 Jun 03

System Requirements:	SPARC/Solaris 2.6
			SPARC/Solaris 7 
			SPARC/Solaris 8   
			SPARC/Solaris 9   

			x86  /Solaris 8  
			x86  /Solaris 9  

Download location: 
http://www.solarisinternals.com/si/downloads/_memtool

Please send any reports to Richard.McDougall@Eng.Sun.COM 

Important Note 
--------------

MemTool is not supported in anyway by Sun; use at your own
risk. MemTool loads a kernel module into the target system, and may
sometimes be incompatible with new Solaris patches. For this reason,
do not use MemTool on a production system.  

Solaris MemTool Features
------------------------

Over time, some of the MemTool functionality is being moved into
Solaris:

pmem:   Solaris 2.6 and later: pmap(1M)
prtmem: Solaris 9 and later: mdb(1M) ::memstat macro

Compatibility
-------------

MemTool 3.9.5beta is compatible with Solaris 2.6, 7, 8 and 9. Solaris 7
requires the following revisions or greater of the kernel patch to allow 
MemTool driver to load.

	  RELEASE     ARCH  PATCH     
	5.7  (on998 ) i386  106542-04  
	5.7  (on998 ) sparc 106541-04 


Alert - for old MemTool versions
--------------------------------

The older MemTool 3.7.3, is incompatible with the latest kernel patches. 
The latest MemTool should always be used on patched machines. Using the 
process memory features of the older MemTool 3.7.3 (memtool, memps, pmem) 
will cause the system to panic with the following patches or greater are 
installed: 

	  RELEASE     ARCH  PATCH     
	5.6  (on297 ) i386  105182-14  
	5.6  (on297 ) sparc 105181-14  
	5.7  (on998 ) i386  106542-04  
	5.7  (on998 ) sparc 106541-04 

Solaris pmap -x bug
-------------------

Solaris 2.6, 7 and 8 has a bug in pmap -x which will over-estimate the
amount of private memory for a process. The MemTool pmem command should
be used in place of pmap -x for accurate memory analysis. 

Until a patch for Solaris is available, a stand alone version of pmem that
works without MemTool can be used safely on machines which you don't want 
to load the complete MemTool package. It is available from:

ftp://playground.sun.com/pub/memtool/pmem.sparc
ftp://playground.sun.com/pub/memtool/pmem.i386

A Solaris patch will be available shortly. Details for the patch will be
noted here and on the MemTool alias.


Installation
------------

The RMCmemx.y.tar.Z file is a Solaris pkgadd file, compressed with 
"compress".

To install:-

# cd /tmp
# zcat RMCmemx.x.tar.Z |tar xvf -
# pkgadd -d .

Answer yes to "load driver now", if you want to load the driver and use 
MemTool without having to reboot. 

Answer yes to "loaded at boot" if you want to have the driver loaded at 
boot time.

If the driver is not loaded, then you will need to manually load it 
before running the tools. The driver can be loaded with the command:

# /opt/RMCmem/drv/bunyipload

Release Notes
-------------

3.9.4:
- Rebuilt to be compatible with latest Solaris 8 kernel patches
  (mem & memtool coredump; memps -m shows only a few entries)

3.9.3:
- Workaround for NFS panic/bug
- Provided missing Intel support

3.9.2:
- Solaris 9 FCS Support
- memps -m now includes cachelist files

3.8.3:
- memps displays no rows on Solaris 2.6

3.8.2:
- Fixes prtmem issues with Solaris 2.6

3.8.1:
- Solaris 8 FCS.
- MemTool gui only available Solaris 7 upwards
- Fixes private memory calculation bugs with shared memory. The pmem 
  command should be used instead of the pmap in Solaris 2.6, 7 and 8 FCS.
- memps provides a wide display

3.7.5:

- Solaris 2.5.1 & 2.5 no longer supported
- Tools updated to use libproc interfaces, which provides enhanced
  library naming for memtool.
- memps and memtool now show file system name when file name cannot
  be calculated
- prtmem enhanced to show more detail (See example below)
- pmem replaced by /usr/proc/bin/pmap
- prtswap utility added

3.7.4: (Internal only test release)

- Solaris patches introduced an interface change that will
  cause pmem to panic the system. Bug id 4178298 references the 
  problem. 3.7.4 will now only work if that patch is installed.
- Free memory now broken out into "cachefree and zerofree" in prtmem
- InUse column added to memps -m
- Shared memory identifier bugs fixed (shmid=0)
- Shared memory ID changed to reflect pmap -x 
- pmem now agrees with pmap -x

3.7.3:

- pmem bug fixed (Solaris bug 4164856), private column can sometimes
  report incorrect value. This was fixed in FCS Solaris 7.
- Code merge and common features between Solaris 2.6 and 7 drivers.
- 2.7 Systems or 2.6 systems with 105181-09 or later show executable
  code size in the prtmem and memps -m commands
- Support for Intel 2.6, 7, 8
- Untested support for Intel 2.5.1

3.7.2:

- Support for Intel Solaris 2.6

Overview
--------

Installed in the /opt/RMCmem directory are the drivers,
utilities and libraries for the package.

        bin/prtmem  - command line utility, displays system memory summary
        bin/prtswap - command line utility, displays swap summary
	bin/memps   - command line, ps of memory, and rolling sar
	bin/pmem    - command line, detailed single process memory display
	bin/mem     - curses based interface for process and files
	bin/memtool - GUI tool for analysing memory usage

The utilities are:-

prtmem
------

The prtmem utility shows a quick summary of memory usage. 

Total memory:             241 Megabytes	 (Total, minus page tables)
Kernel Memory:             21 Megabytes  (Kernel Memory)
Application:               31 Megabytes  (anonymous: heap & stack)
Executable & libs:         14 Megabytes  (files mapped 777)
File Cache:                11 Megabytes  (File cache in kernel)
Free, file cache:          88 Megabytes  (File cache on freelist)
Free, free:                75 Megabytes	 (Free, zeroed)


Total Memory - The total amount of physical memory in the system available
to the operating system at boot time. This figure is usually minus a few
megabytes used by startup.

Kernel Memory - The wired down memory allocated for kernel data structures
buffers and caches. This excludes the file system cache.

Application - Anonymous memory, used for heap, stack, copy on write, and
shared memory.

Executable and Libs - Memory used for files that are mapped with 
executable permission. e.g., binaries and shared libraries.

File Cache - The portion of the file system cache which is not on the
free lists.

Free, file cache - The portion of the file system cache which IS on the
free lists, and is counted in the free column reported by vmstat. This
memory is still valid file cache, but resides on the free lists.

Free, free - The amount of memory on the free lists that has no
identify; that is, it represents no file in the file system
cache.

pmem 
----

pmem is a utility for displaying the address space of a process.

Process address space is broken into several sections, some
which is shared and some which is not shared.

A typical  address space has an  executable, a heap space, some
shared libraries and it's stack:-

# pmem 25628

25628:  -ksh
Address   Kbytes Resident Shared Private Permissions       Mapped File
00010000     192     192     192       - read/exec         ksh
00040000       8       8       -       8 read/write/exec   ksh
00042000      32      32       -      32 read/write/exec    [ heap ]
FF180000     664     664     664       - read/exec         libc.so.1
FF236000      24      24       -      24 read/write/exec   libc.so.1
FF23C000       8       8       -       8 read/write/exec    [ anon ]
FF280000     552     552     552       - read/exec         libnsl.so.1
FF31A000      32      32       -      32 read/write/exec   libnsl.so.1
FF322000      32      16       -      16 read/write/exec    [ anon ]
FF340000      16      16      16       - read/exec         libc_psr.so.1
FF350000      16      16      16       - read/exec         libmp.so.2
FF364000       8       8       -       8 read/write/exec   libmp.so.2
FF380000       8       8       -       8 read/write/exec    [ anon ]
FF390000      40      40      40       - read/exec         libsocket.so.1
FF3AA000       8       8       -       8 read/write/exec   libsocket.so.1
FF3B0000       8       8       8       - read/exec         libdl.so.1
FF3C0000     128     128     128       - read/exec         ld.so.1
FF3E0000       8       8       -       8 read/write/exec   ld.so.1
FFBEC000      16      16       -      16 read/write/exec    [ stack ]
--------  ------  ------  ------  ------
total Kb    1800    1784    1616     168

Program text is the executable  component of a binary, which is
mapped into a process as read only.

Each binary also has a data section, which is mapped just above
the  text  portion.  The  data  section  contains  all  of  the
initialized  sections of an  application,  such as i=10 or char
*str={\"hello world\"}.

Because the data section can be shared amongst other  processes
to save memory, the segment is mapped shared, and any writes to
this  segment  cause  COW (copy on  writes)  which  creates  an
anonymous private page for each page that is writen to.

This means that the data  section of a  binary typically has  a 
shared portion an a non-shared portion.

The process heap  is  allocated  from anonymous  memory, and is 
always reported as 100% private.

Shared libraries appear  as text/heap/stack, and are mapped the
same way as the program binary.

Stacks appear at the end  of the address  space (sparc), or  at
the start of the address  space (i386). The  stack is allocated 
from anonymous memory, and is always reported as 100% private.

Shared memory is always reported as shared.

This is four basic  measures  for the  amount of memory in each
segment:-

Size - The amount of virtual memory space assigned to this 
segment.

Resident - The amount of a segment which is currently in memory, 
including the portions which are shared between other processes.

Shared - The  amount  of  this  segment  which  is  not  
allocated from anonymous  memory , and could potentially be 
shared with other processes in the system.

Private - The amount of anonymous memory for this segment. Heap 
space is always reported as 100% private, but only the anonymous 
copy-on-write portions of data segments are reported as private.

To  calculate  the  amount  of memory a group of  processes  is
using:
	- for each process of the same binary, count the 
	  private memory used for n-1 processes.
	- for each binary, count the total resident size
	  once.

For example, using the pmem data above, we can calculate that
20 ksh binaries would used 

	168Kbytes * 19 + 1784Kbytes = 3192 + 1784 Kbytes
				    = 4976Kbytes

memps
-----

A utility for displaying a ps of memory, for processes or files.


Usage:  memps                           -   Memory ps

        memps -p                        -   ps type process output

        memps -m                        -   ps type file output


        memps -p [-n n] [-o file] [t s] -   Sar type process output
                                            (-n n=number of processes)

        memps -m [-n n] [-o file] [t s] -   Sar type file output
                                            (-n n=number of files)


SunOS alvinn 5.5 Generic sun4u    11/12/95

   PID    Size     Res  Shared Private  Process                              
  6852  11888k   5268k   1216k   4052k  xemacs
 10352   9440k   4164k   1612k   2552k  /usr/dist/local/share/sunpro,v4.0/5.x-sp
  9815  11728k   2704k   1500k   1204k  /usr/dist/local/pkgs/sunpro,v4.0/5.x-spa
  3277   7944k   2884k   1692k   1192k  roam -fg black
  2935  18272k   1672k    776k    896k  /usr/doctools/frame4.0/bin/sunxm.s5.spar
 10290   5832k   2284k   1488k    796k  /usr/dist/local/pkgs/teamware/5.x-sparc/
  2520   6392k   2644k   1856k    788k  dtwm
  2521   6032k   2696k   1952k    744k  dtterm -session dta001SM -sdtserver
 10208  11160k   2480k   1880k    600k  netscape
  7318   8488k   2060k   1480k    580k  /usr/dist/local/pkgs/sunpro,v4.0/5.x-spa
  2512   3912k   1416k   1016k    400k  /usr/dt/bin/ttsession -s
  2522   5280k   2308k   1952k    356k  /usr/dt/bin/dtterm -session dta001Sb -C 
  7334   4472k   1780k   1448k    332k  /usr/dist/local/pkgs/sunpro,v4.0/5.x-spa
  2525   3592k   1584k   1284k    300k  perfbar -Wp 1192 3 -Ws 76 984 -WP 6 6 +W
  2524   3792k   1552k   1272k    280k  clock -Wp 1035 3 -Ws 150 164 -WP 6 6 +Wi
 14153   3376k    680k    428k    252k  /opt/RMCmem/bin/memps
  3601   5360k   2108k   1856k    252k  dtpad -server
  3294   9088k   1664k   1448k    216k  snapshot
  9921   7864k   1772k   1616k    156k  /home/rmc/src/bunyiptool/memtool
  2499   1976k    912k    772k    140k  /usr/dt/bin/dsdm
 14125   1480k    592k    468k    124k  vi README
 10342   2304k    888k    768k    120k  /usr/dist/pkgs/cam,v1.4/5bin.sun4/cam vi
 13950   1480k    580k    468k    112k  vi pkginfo pkgmap
  7333  21456k   1100k   1008k     92k  /usr/dist/local/pkgs/sunpro,v4.0/5.x-spa
  2477   1784k    840k    768k     72k  /usr/openwin/bin/fbconsole
 11839   1528k    748k    680k     68k  /bin/ksh


mem
---

A curses based top like display for process and file memory. 

Use LV or LP to switch between modes


+- MemTool 3.0.0 ---------- KEYS: U)pper H)elp R)efresh Q)uit -----------------+
|                                                                              |
| [CPU Usage] [Paging        ] [Page in/outs ][Memory   (k)] [Inode        ]   |
|  Idle   82   Scanr      0.0   Pgouts    0.0  Tot    23110   Namei      10    |
|  User   15   Pgfr       0.0   Ppgouts   0.0  Used   16616   Dnlc     100%    |
|  Sys     3   PageRt     Inf   Pgins     0.2  UFS    10010   Dirblk      0    |
|  Wait    0   Fr/Lts    32.5   Ppgins    0.2  Free     787   Ncsize   2181    |
+------------------------------------------------------------------------------+
|  VFS Memory Dump         |  All Files              |  Sorted By Size         |
+------------------------------------------------------------------------------+
|   PID    Size  Mapped  Shared Private  Process                               |
|                                                                              |
|  2935  18272k   6068k    788k   5280k  /usr/doctools/frame4.0/bin/sunxm.s5.s |
|  2656   9664k   4176k   1212k   2964k  xemacs                                |
|  3294   9088k   4396k   1560k   2836k  snapshot                              |
|  3277   6184k   2804k   1536k   1268k  roam -fg black                        |
|  2520   6240k   2616k   1820k    796k  dtwm                                  |
|  2521   5592k   2508k   1952k    556k  dtterm -session dta001SM -sdtserver   |
|  2522   5280k   2368k   1952k    416k  /usr/dt/bin/dtterm -session dta001Sb  |
|  2524   3792k   1728k   1364k    364k  clock -Wp 1035 3 -Ws 150 164 -WP 6 6  |
|  2512   3464k   1416k   1056k    360k  /usr/dt/bin/ttsession -s              |
|  3581   3432k    692k    368k    324k  ./mem                                 |
|  2525   3568k   1648k   1344k    304k  perfbar -Wp 1192 3 -Ws 76 984 -WP 6 6 |
|  2523   3592k   1664k   1364k    300k  clock -Wp 871 3 -Ws 150 164 -WP 6 6 + |
|  2499   1808k    864k    784k     80k  /usr/dt/bin/dsdm                      |
|  3137   1488k    684k    604k     80k  ftp adelaide.aus                      |
|  2527   1528k    744k    680k     64k  -ksh                                  |
|  2477   1784k    844k    780k     64k  /usr/openwin/bin/fbconsole            |
|  2944   1528k    744k    680k     64k  /bin/ksh                              |
|  2628   1552k    736k    680k     56k  /bin/ksh                              |
|  2531   1528k    728k    680k     48k  /bin/ksh                              |
|  2725   1384k    656k    608k     48k  rlogin dreadnought-185                |
|  2726   1384k    648k    608k     40k  rlogin dreadnought-185                |
|                                                                              |
+------------------------------------------------------------------------------+



memtool
-------

A Motif based GUI for analizing memory usage. Requires SUNWmfrun or CDE
to be installed.

NOTE: memtool may take several minutes to calculate the full process matrix
on older systems with large process tables. Use the filter options to select a
smaller subset of processes for analysis.






Browse the book online via Safari

Original Material © 2000-2005 Richard McDougall & Jim Mauro
Last Updated Dec 02 2005 00:21