Segmap tuning

From Siwiki

Abstract

To improve the performance of file system intensive workloads on Solaris/x64, increase the size of the segmap cache using the undocumented /etc/system tunable segmapsize.

Segmap Overview

When applications access a UFS, NFS, QFS, or VxFS file using read, write, and related system calls, the Solaris kernel maps portions of the file into pages of kernel virtual memory, and copies data between these pages and user space buffers.Solaris maintains a cache called segmap of recently mapped file pages, and subsequent read and write operations may find their data in segmap and avoid physical I/O and a minor page fault. ZFS does not use segmap.

The size of the segmap can be increased to improve performance of file system intensive workloads, using the /etc/system tunable segmap_percent, but this tunable is only recognized on Solaris/SPARC systems. Also, such tuning is no longer necessary for SPARC as of Solaris 10 3/05 (FCS), because the Kernel Physical Mapping (KPM) feature greatly reduced the cost of mapping and unmapping pages in segmap. Segmap is still used, and segmap_percent is still recognized, but it makes little difference in performance.

Segmap on Solaris/x64

Segmap tuning is still beneficial on the x64 architecture, at least until a later update of Solaris 10; more on this below. The default size on 64-bit kernels is 64 MB, which is too small for most servers. This can be increased using the undocumented segmapsize tunable, which has units of bytes, and can be modified using the eeprom(1M) command. For example, to set it to 1 GB, execute this command as root and reboot:

   # eeprom segmapsize=0x40000000

Start with the value that corresponds to 12% of physical memory, which matches the default size of the segmap on SPARC, and increase if necessary. The improvements in I/O performance will depend on application and configuration, but they can be dramatic. Bandwidth improved by 2X and system time was reduced by 5X in one test.

Segmap on Solaris/x86, 32-bit kernel

The segmapsize tunable is recognized and useful on Solaris/x86 32-bit kernels, but it cannot be greatly increased because the amount of memory that the kernel can access is limited by a 32-bit address, and this memory must serve many purposes. As of Solaris 10 3/05 (FCS), the default is 16 MB, and the maximum that can be set in /etc/system is 128 MB. This can be increased further using the eeprom(1M) command, but exercise caution, as this reduces the amount of virtual memory available to the kernel for other purposes.

Diagnosis

Your workload might benefit from increasing the segmap size if you observe these symptoms:

• high page-in rate, or high minor-fault rate
• moderate to high %sys time
• high cross-call rate

Page-ins and minor-faults are shown in the vmstat "mf" and "pi" columns, and cross calls are shown in the mpstat "xcal" column. In the example below, there are 160521 page-in/sec, and between 27000 and 72000 cross calls per second per CPU:

   % vmstat 
   kthr      memory            page            disk          faults      cpu
   r b w   swap  free  re  mf pi po fr de sr s0 s1 s2 s3   in   sy   cs us sy id
   0 0 0 72298480 62856612 40125 0 160521 0 0 0 0 0 0 0 0 624626 232 37059 0 36 64

   % mpstat
   CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
     0    0 618 72177 34970    0 5084    0  351 1027    0    10    0  41   0  59
     1    0 516 58554 35644    0 4598    0  294  943    0     4    0  37   0  63
     2    0 229 27586 37589    0 5004    1  117  662    0    70    0  46   0  54
     3    0 266 31936 36608    1 3523    1   89  598    0    38    0  38   0  62

Future Enhancement

The segmapsize tunable will not be documented, nor will its default value be increased, because it will have little effect on 64-bit kernels after the following CR is integrated into an update to Solaris 10:

6256083 Need a lightweight file page mapping mechanism to substitute segmap

This CR implements a new cache called VPM, implemented on top of KPM, that replaces segmap for most purposes for the UFS, NFS, TMPFS, and SPECFS file systems. Tuning the VPM cache size should not be necessary. Segmap is still used in limited circumstances.

The CR initially applies to x64 systems only, so segmap will still be the primary cache on SPARC for the time being. It will never be supported on 32-bit kernels due to the limited kernel virtual address space. It is expected to be available in a future Solaris 10 Update, so check the fixed CR list in the release notes when they are published. There are no plans to back port it to earlier Solaris versions.

ZFS, QFS, and VxFS are not affected by 6256083. QFS and VxFS will continue to use segmap on all architectures, so they will still benefit from segmap tuning on Solaris/x64/x86, though this could change in the future. ZFS implements its own buffer cache and does not use segmap.