To improve the performance of file system intensive workloads on Solaris/x64, increase the size of the segmap cache using the undocumented /etc/system tunable segmapsize.
When applications access a UFS, NFS, QFS, or VxFS file using read, write, and related system calls, the Solaris kernel maps portions of the file into pages of kernel virtual memory, and copies data between these pages and user space buffers.Solaris maintains a cache called segmap of recently mapped file pages, and subsequent read and write operations may find their data in segmap and avoid physical I/O and a minor page fault. ZFS does not use segmap.
The size of the segmap can be increased to improve performance of file system intensive workloads, using the /etc/system tunable segmap_percent, but this tunable is only recognized on Solaris/SPARC systems. Also, such tuning is no longer necessary for SPARC as of Solaris 10 3/05 (FCS), because the Kernel Physical Mapping (KPM) feature greatly reduced the cost of mapping and unmapping pages in segmap. Segmap is still used, and segmap_percent is still recognized, but it makes little difference in performance.
Segmap on Solaris/x64
Segmap tuning is still beneficial on the x64 architecture, at least until a later update of Solaris 10; more on this below. The default size on 64-bit kernels is 64 MB, which is too small for most servers. This can be increased using the undocumented segmapsize tunable, which has units of bytes, and can be modified using the eeprom(1M) command. For example, to set it to 1 GB, execute this command as root and reboot:
# eeprom segmapsize=0x40000000
Start with the value that corresponds to 12% of physical memory, which matches the default size of the segmap on SPARC, and increase if necessary. The improvements in I/O performance will depend on application and configuration, but they can be dramatic. Bandwidth improved by 2X and system time was reduced by 5X in one test.
Segmap on Solaris/x86, 32-bit kernel
The segmapsize tunable is recognized and useful on Solaris/x86 32-bit kernels, but it cannot be greatly increased because the amount of memory that the kernel can access is limited by a 32-bit address, and this memory must serve many purposes. As of Solaris 10 3/05 (FCS), the default is 16 MB, and the maximum that can be set in /etc/system is 128 MB. This can be increased further using the eeprom(1M) command, but exercise caution, as this reduces the amount of virtual memory available to the kernel for other purposes.
Your workload might benefit from increasing the segmap size if you observe these symptoms:
- high page-in rate, or high minor-fault rate
- moderate to high %sys time
- high cross-call rate
Page-ins and minor-faults are shown in the vmstat "mf" and "pi" columns, and cross calls are shown in the mpstat "xcal" column. In the example below, there are 160521 page-in/sec, and between 27000 and 72000 cross calls per second per CPU:
% vmstat kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id 0 0 0 72298480 62856612 40125 0 160521 0 0 0 0 0 0 0 0 624626 232 37059 0 36 64 % mpstat CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 618 72177 34970 0 5084 0 351 1027 0 10 0 41 0 59 1 0 516 58554 35644 0 4598 0 294 943 0 4 0 37 0 63 2 0 229 27586 37589 0 5004 1 117 662 0 70 0 46 0 54 3 0 266 31936 36608 1 3523 1 89 598 0 38 0 38 0 62
The segmapsize tunable was not documented, and its default value was unchanged, because it had little effect on 64-bit kernels once the following CR was integrated into Solaris 10 Update 6.
- 6256083 Need a lightweight file page mapping mechanism to substitute segmap
This CR implemented a new cache called VPM, built on top of KPM, that replaced segmap for most purposes in buffered filesystem operations. The new VPM implementation greatly improves the bandwidth and efficiency of buffered filesystem that use it. Tuning the VPM cache size should not be necessary. Segmap is still used in limited circumstances.
UFS, NFS, TMPFS, and SPECFS filesystems that ship with Solaris take advantage of VPM since Solaris 10, Update 6. Note that VPM is irrelevant to ZFS because ZFS implements its own buffer cache (the ARC) and does not use segmap. Sun's QFS filesystem leverages VPM beginning with the QFS 5.0 release.
The CR applied to 64-bit systems only. It will never be supported on 32-bit kernels due to the limited kernel virtual address space. There are no plans to back port it to earlier Solaris versions, but kernel patches can be applied to upgrade existing Solaris 10 systems.
The initial Solaris 10 patches listing this CR as fixed are: SPARC 137137-05 x64 137138-05
VxFS is not affected by the VPM enhancements, and continues to use segmap on all architectures. Therefore, it might benefit from traditional segmap tuning, though this could change in the future.