Unification of files and filesets

From Siwiki

Jump to: navigation, search

[edit] Changes coming soon to a FileBench repository near you.

The biggest change is to turn FileBench files into filesets with 1 entry. Basically this involves automatically creating a fileset with 1 entry whenever a define file command is found. This eliminates the need fileobj.c and fileobj.h files while still (almost) preserving the "file" semantics in the workload language. One small but significant difference is that the fileobj.c code created a file of size 0 when an unallocated file was defined (prealloc attribute missing), while unallocated fileset (and now also file) files are not allocated on disk at all. This broke two existing workload files, but they were easily fixed by adding the prealloc attribute to the file definition commands. The two that were modified are the singlestreamwrite.f and singlestreamwritedirect.f, both found in filebench/workloads. Any privately developed workloads that use files that don't have the prealloc attribute will need to be changed. If filesize is not specified it will default to zero length and give you the same behavior as before.

Another change was to fix the raw file feature so it actually worked, and limit it to use with a fileset that had been defined as a file (using the define file command).

In the current code base, files can be pre-allocated in parallel but filesets, and the files they consist of, cannot. In order to preserve the parallel pre-allocation feature of files, it was necessary to enable parallel fileset pre-allocation, which was extended to pre-allocate the files of each fileset in parallel as well, thus significantly improving the speed of pre-allocation on systems with multiple disk drives. Just add the paralloc attribute to your filesets to enable this feature.

These changes were simplified by combining the file setup code in to a common subroutine, which, by adding common I/O buffer allocation code addressed a desire to have the memory buffer usage of flowop reads and writes be consistent. Previously three flowops used shared thread memory, while the other four used private, per flowop buffers. In the new scheme, if thread memory has been specified (memsize > 0), then a random offset into thread memory is calculated and passed back to the calling routine. Otherwise, a private buffer is allocated or reused, with its location passed back. The private buffer is created to be large enough to hold iosize worth of bytes, which may be much larger than the old method of only allocating one megabyte. Thus, this change also addresses a bug in appendfile, which caused it to crash if the append size was set to larger than one megabyte.

If thread memory is in use, it will provide an error message if iosize is larger than the thread memory size, and which now applies to all seven I/O flowops. If thread memory is not in use (memsize unspecified or set to zero), then private buffers will be used and allocated with iosize bytes. Thus I/O size larger than one megabyte can be handled for all the I/O flowops. Thus I/O flowop behavior with respect to file and fileset selection and I/O buffer usage is now consistent across all I/O flowops.

Before this change the readwholefile and writewholefile flowops did multiple one megabyte transfers for files larger than one megabyte. This "feature" was not really documented (except by way of warning in the most recent documentation), and is not what would be expected, so has not been kept. The new behavior (which is mostly provided by the common code) is to do multiple iosize transfers as necessary to transfer the whole file, or a single transfer of the whole file if iosize is 0 (or not specified). The current workload files have been modified with the iosize attribute, which is assigned from a variable whose default value is one megabyte. Thus the supplied workloads will behave as before, but can be easily changed to transfer the files in any size chunks, including one of the full file size. Users who have created their own workloads with readwholefile or writewholefile flowops may want to make similar modifications.

Finally, the oltp.f workload file, which models a transaction processing system, was not doing synchronous log writes as is done by real TP systems. Apparently the dysnc attribute got lost somewhere along the way. So, the dysnc attribute was put back to make it behave correctly.

To summarize the changes that folks who have developed their own workloads will need to address, make sure that all define file commands include prealloc if the workload attempts to use the file before creating it with a createfile flowop. Also, if you may get slightly different results with workloads that use readwholefile or writewholefile, unless you added an iosize attribute and set it to 1m (one Megabyte).

Solaris Internals
Personal tools
The Books
The Ads