HDF5 properties and property lists make it possible to shape or modify an HDF5 file, group, dataset, attribute, committed datatype, or even an I/O stream, in a number of ways. For example, you can do any of the following:
|
The remaining sections in this chapter discuss the following topics:
The discussions and function listings in this chapter focus on general property operations, object and link properties, and related functions.
File, group, dataset, datatype, and attribute properties are discussed in the chapters devoted to those features, where that information will be most convenient to users. For example, the Datasets chapter discusses dataset creation property lists and functions, dataset access property lists and functions, and dataset transfer property lists and functions. This chapter does not duplicate those discussions.
Generic property operations are an advanced feature and are beyond the scope of this guide.
This chapter assumes an understanding of the following chapters of this HDF5 User’s Guide:
HDF5 property lists and the property list interface (H5P) provide a mechanism for storing characteristics of objects in an HDF5 file and economically passing them around in an HDF5 application. In this capacity, property lists significantly reduce the burden of additional function parameters throughout the HDF5 API. Another advantage of property lists is that features can often be added to HDF5 by adding only property list functions to the API; this is particularly true when all other requirements of the feature can be accomplished internally to the library.
For instance, a file creation operation needs to know several
things about a file, such as the size of the user-block
or the sizes of various file data structures.
Bundling this information as a property list simplifies the
interface by reducing the number of parameters to the function
H5Fcreate
.
As illustrated in the figure above, the HDF5 property environment is a three-level hierarchy:
The following subsections discuss property list classes, property lists, and properties in more detail.
Table 1: Property list classes in HDF5 | ||||
---|---|---|---|---|
Property List Class
|
|
For further discussion
|
||
File creation (FCPL) | H5P_FILE_CREATE |
See various sections of “The HDF5 File” chapter. | ||
File access (FAPL) | H5P_FILE_ACCESS |
|||
File mount (FMPL) | H5P_FILE_MOUNT |
Used only as H5P_DEFAULT
(see footnote 1).
|
||
Object creation (OCPL) | H5P_OBJECT_CREATE |
See table of object property functions below. | ||
Object copy (OCPYPL) | H5P_OBJECT_COPY |
|||
Group creation (GCPL) | H5P_GROUP_CREATE |
See “Programming Model” section of “HDF5 Groups” chapter. | ||
Group access (GAPL) | H5P_GROUP_ACCESS |
|||
Link creation (LCPL) | H5P_LINK_CREATE |
See examples in “Programming Model” section in this chapter and the table of link creation property functions below. | ||
Link access (LAPL) | H5P_LINK_ACCESS |
|||
Dataset creation (DCPL) | H5P_DATASET_CREATE |
See “Programming Model” section of “HDF5 Datasets” chapter. | ||
Dataset access (DAPL) | H5P_DATASET_ACCESS |
|||
Dataset transfer (DXPL) | H5P_DATASET_XFER |
|||
Datatype creation (TCPL) | H5P_DATATYPE_CREATE |
See various sections of “HDF5 Datatypes” chapter. | ||
Datatype access (TAPL) | H5P_DATATYPE_ACCESS |
|||
String creation (STRCPL) | H5P_STRING_CREATE |
See “Programming Model” sections of “HDF5 Datasets” and “HDF5 Datatypes” chapters. | ||
Attribute creation (ACPL) |
H5P_ATTRIBUTE_CREATE |
See “Working with Attributes” (or “Attribute Usage”) section of “HDF5 Attributes” chapter. | ||
The abbreviations to the right of each property list class name
in this table are widely used in both HDF5 programmer documentation
and HDF5 source code.
For example,
FCPL is file creation property list,
OCPL is object creation property list,
OCPYPL is object copy property list, and
STRCPL is string creation property list.
These abbreviations may appear in either uppercase or lowercase.
|
The “HDF5 property list class inheritance hierarchy” figure, immediately following, illustrates the inheritance hierarchy of HDF5’s property list classes. Properties are defined at the root of the HDF5 property environment (“Property List Class Root” in the figure below). Property list classes then inherit properties from that root, either directly or indirectly through a parent class. In every case, a property list class inherits only the properties relevant to its role. For example, the object creation property list class (OCPL) inherits all properties that are relevant to the creation of any object while the group creation property list class (GCPL) inherits only those properties that are relevant to group creation.
![]() |
Figure 2. HDF5 property list class inheritance hierarchy
Property list classes displayed above in black are directly accessible through the programming interface; the root of the property environment and the STRCPL and OCPL property list classes, in gray above, are not user-accessible. The red empty set symbol (∅) indicates that the file mount property list class (FMPL) is an empty class; that is, it has no settable properties (see footnote 1). Abbreviations used in this figure are defined in the preceding table, “Property list classes in HDF5”. |
Property lists are implemented as containers holding a collection of name/value pairs. Each pair specifies a property name and a value for the property. A property list usually contains information for one to many properties.
HDF5’s default property values are designed to be reasonable for general use cases. Therefore, an application can often use a property list without modification. On the other hand, adjusting property list settings is a routine action and there are many reasons for an application to do so.
A new property list may either be derived from a property list class or copied from an existing property list. When a property list is created from a property list class, it contains all the properties that are relevant to the class, with each property set to its default value. A new property list created by copying an existing property list will contain the same properties and property values as the original property list. In either case, the property values can be changed as needed through the HDF5 API.
Property lists can be freely reused to create consistency. For example, a single set of file, group, and dataset creation property lists might be created at the beginning of a project and used to create hundreds, thousands, even millions, of consistent files, file structures, and datasets over the project’s life. When such consistency is important to a project, this is an economical means of providing it.
Further examples include the following:
|
Purpose | |
Examples | |
Property List |
---|---|---|---|---|---|
Specify the driver to be used to open a file | A POSIX driver or an MPI IO driver | FAPL | |||
Specify filters to be applied to a dataset | Gzip compression or checksum evaluation | DCPL | |||
Specify whether to record key times associated with an object | Creation time and/or last-modified time | OCPL | |||
Specify the access mode for a file opened via an external link | Read-only or read-write | LAPL |
Each property is initialized with a default value.
For each property, there are one or more dedicated H5Pset_*
calls that can be used to change that value.
Creation, access, and transfer properties:
Properties fall into one of several major categories:
creation properties, access properties, and transfer properties.
Creation properties control permanent object characteristics. These characteristics must be established when an object is created, cannot change through the life of the object (they are immutable), and the property setting usually has a permanent presence in the file.
Examples of creation properties include:
The default for this dataset creation property
(H5Pset_layout
) is that a dataset is stored
in a contiguous block.
This works well for datasets with a known size limit that will
fit easily in system memory.
A chunked layout is important if a dataset is to be compressed, to enable extending the dataset’s size, or to enable caching during I/O.
A compact layout is suitable only for very small datasets because the raw data is stored in the object header.
This link creation property
(H5Pset_create_intermediate_group
)
enables an application to add an object in a file without
having to know that the group or group hierarchy containing
that object already exists.
With this property set, HDF5 automatically creates missing groups.
If this property is not set, an application must verify that each
group in the path exists, and create those that do not,
before creating the new object; if any group is missing,
the create operation will fail.
Certain file creation properties enable the application
to select one of several file layouts.
Examples of the available layouts include
a standard POSIX-compliant layout
(H5Pset_fapl_sec2
),
a family of files (H5Pset_fapl_family
),
and a split file layout that separates raw data and metadata
into separate files (H5Pset_fapl_split
).
These and other file layout options are discussed in the
“Storage Layouts and Drivers” section of the
“HDF5
File” chapter of this HDF5 User’s
Guide.
In settings where data integrity is vulnerable,
it may be desirable to set checksumming when datasets are created
(H5Pset_fletcher32
). A subsequent application will
then have a means to verify data integrity when reading the
dataset.
Access properties control transient object characteristics. These characteristics may change with the circumstances under which an object is accessed.
Examples of access properties include:
For example, a file might be created with the MPI I/O driver
(H5Pset_fapl_mpio
) during high-speed
data acquisition in a parallel computing environment.
The same file might later be analyzed in a serial computing
environment with I/O access handled through the serial
POSIX driver (H5Pset_fapl_sec2
).
Optimizations differ across computing environments and according to the needs of the task being performed, so are transient by nature.
Transfer properties apply only to datasets and control transient aspects of data I/O. These characteristics may change with the circumstances under which data is accessed.
Examples of dataset trasfer properties include:
If checksumming has been set on a dataset
(with H5Pset_fletcher32
,
in the dataset creation prpertiy list),
an application reading that dataset can choose whether
check for data integrity (H5Pset_edc_check
).
HDF5 provides several properties for tuning I/O of
chunked datasets in a parallel computing environment
(H5Pset_dxpl_mpio_chunk_opt
,
H5Pset_dxpl_mpio_chunk_opt_num
,
H5Pset_dxpl_mpio_chunk_opt_ratio
, and
H5Pget_mpio_actual_chunk_opt_mode
).
Optimal settings differ due to the characteristics of a computing environment and due to an application’s data access patterns; even when working with the same file, these settings might change for every application and every platform.
The programming model for HDF5 property lists is actually quite simple:
In some cases, you will not have to define property lists at all. If the default property settings are sufficient for your application, you can tell HDF5 to use the default property list.
The following sections
first discuss the use of default property lists,
then each step of the programming model,
and finally a few less frequently used property list operations.
10.3.1. Using Default Property Lists
Default property lists can simplify many routine HDF5 tasks because
you do not always have to create every property list you use.
An application that would be well-served by HDF5’s default
property settings can use the default property lists simply by
substituting the value H5P_DEFAULT
for a property list identifier. HDF5 will then apply the
default property list for the appropriate property list class.
For example, the function H5Dcreate2
calls for
a link creation property list,
a dataset creation property list, and
a dataset access property list.
If the default properties are suitable for a dataset,
this call can be made as
dset_id = H5Dcreate2( loc_id, name, dtype_id, space_id
;
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT );
Of course, you would not want to do this without considering where it is appropriate, as there may be unforeseen consequences. Consider, for example, the use of chunked datasets. Optimal chunking is quite dependent on the makeup of the dataset and the most common access patterns, both of which must be taken into account in setting up the size and shape of chunks.
H5Pcreate
.
A second dataset creation property list is then created
by copying the first one with H5Pcopy
.
dcplA_id = H5Pcreate (H5P_DATASET_CREATE)
;
H5P_DATASET_CREATE
.
The new dataset creation property list’s identifier
is returned in dcplA_id
and the
property list is initialized with default dataset creation
property values.
A list of valid classes appears above in “Table 1: Property List Classes in HDF5”.
dcplB_id = H5Pcopy (dcplA_id)
;
dcplB_id
,
is created as a copy of dcplA_id
and is
initialized with dataset creation property values currently
in dcplA_id
.
dcplA_id
and dcplB_id
are identical; they will both contain any modified property values
that were changed in dcplA_id
before dcplB_id
was created. They may, however, diverge as additional property values
are reset in each.
While we are creating property lists, let’s create a link creation property list; we will need this property list when the new dataset is linked into the file below:
lcplAB_id = H5Pcreate (H5P_LINK_CREATE)
;
Later in this section, the dataset creation property lists
dcplA_id
and dcplB_id
created in the
section above will be used respectively to create
chunked and contiguous datasets.
To set this up, we must set the layout property in each property list.
The following example sets
dcplA_id
for chunked datasets and
dcplB_id
for contiguous datasets:
error = H5Pset_layout (dcplA_id, H5D_CHUNKED)
;
error = H5Pset_layout (dcplB_id, H5D_CONTIGUOUS)
;
Since dcplA_id
specifies a chunked layout,
we must also set the number of dimensions and the size of the chunks.
The example below specifies that datasets created with
dcplA_id
will be 3-dimensional and
that the chunk size will be 100 in each dimension:
error = H5Pset_chunk (dcplA_id, 3, [100,100,100])
;
These datasets will be created with UTF-8 encoded names. To accomplish that, the following example sets the character encoding property in the link creation property list to create link names with UTF-8 encoding:
error = H5Pset_char_encoding (lcplAB_id, H5T_CSET_UTF8)
;
dcplA_id
can now be used to create chunked datasets
and dcplB_id
to create contiguous datasets.
And with the use of lcplAB_id
, they will be created
with UTF-8 encoded names.
Assume that the datatype dtypeAB
and the
dataspaces dspaceA
and dspaceB
have been defined
and that the location identifier locAB_id
specifies
the group AB
in the current HDF5 file.
We have already created the required link creation and dataset creation
property lists. For the sake of illustration, we assume that the
default dataset access property list meets our application requirements.
The following calls would create the datasets dsetA
and dsetB
in the group AB
.
The raw data in dsetA
will be contiguous
while dsetB
raw data will be chunked;
both datasets will have UTF-8 encoded link names:
dsetA_id = H5Dcreate2( locAB_id, dsetA, dtypeAB, dspaceA_id, lcplAB_id, dcplA_id, H5P_DEFAULT ); dsetB_id = H5Dcreate2( locAB_id, dsetB, dtypeAB, dspaceB_id, lcplAB_id, dcplB_id, H5P_DEFAULT );
hid_t
and include things like
file identifiers, often expressed as file_id
;
dataset identifiers, dset_id
; and
property list identifiers, plist_id
.
To reduce the risk of memory leaks, all of these identifiers must
be closed once they are no longer needed.
Property list identifiers are no exception to this rule,
and H5Pclose
is used for this purpose.
The calls immediately following would close the
property lists created and used in the examples above.
error = H5Pclose (dcplA_id)
;
error = H5Pclose (dcplB_id)
;
error = H5Pclose (lcplAB_id)
;
PList_Class = H5Pget_class (dcplA_id)
;
PList_Class
will contain the value H5P_DATASET_CREATE
indicating that dcplA_id
is a dataset creation property list.
But sometimes an application must do something that requires knowing
the creation property settings. HDF5 makes the acquisition of this
information fairly straight-forward;
for each property setting call, H5Pset_*
,
there is a corresponding H5Pget_*
call
to retrieve the property’s current setting.
Consider the following examples which illustrate the determination of dataset layout and chunking settings:
H5Dget_create_plist
will return a property list
identifier for the creation property list that was used
to create the dataset. Call it DCPL1_id
.H5Pset_layout
sets a dataset’s layout to be
compact, contiguous, or chunked.H5Pget_layout
called with DCPL1_id
will return the dataset’s layout, either
H5D_COMPACT
, H5D_CONTIGUOUS
,
or H5D_CHUNKED
.H5Pset_chunk
sets the rank of a dataset,
that is the number of dimensions it will have,
and the maximum size of each dimension.H5Pget_chunk
, also called with DCPL1_id
,
will return the rank of the dataset
and the maximum size of each dimension.H5Pget_
calls will return the property’s
default value.
One circumstance under which an application might need
to determine access property settings might be when a file or object
is already open but the application does not know the property list
settings.
In that case, the application can use the appropriate
get access property list call to retrieve a property list
identifier. For example, if the dataset dsetA
from the earlier examples is still open, the following call
would return an identifier for the dataset access property list
in use:
dsetA_dacpl_id = H5Dget_access_plist( dsetA_id );The application could then use the returned property list identifier to analyze the property settings.
Further discussion of HDF5’s generic property interface and user-defined properties and property lists is beyond the scope of this document.
General property functions, generic property functions and macros, property functions that are used with multiple types of objects, and object and link property functions are listed below.
Property list functions that apply to a specific type of object are listed in the chapter that discusses that object. For example, the Datasets chapter has two property list function listings: one for dataset creation property list functions and one for dataset access property list functions. As has been stated, this chapter is not intended to describe every property list function.
Function Listing 1. General property list functions (H5P) | ||
---|---|---|
C Function Fortran Subroutine |
Purpose | |
H5Pcreate
h5pcreate_f
|
Creates a new property list as an instance of a specified parent property list class. | |
H5Pcopy
h5pcopy_f
|
Creates a new property list by copying the specified existing property list. | |
H5Pget_class
h5pget_class_f
|
Retrieves the parent property list class of the specified property list. | |
H5Pclose
h5pclose_f
|
Closes the specified property list. | |
Object property functions can be used with several kinds of objects.
Function Listing 2. Object property functions (H5P) | ||
---|---|---|
C Function
Fortran Subroutine |
Purpose | |
Object Creation Properties | ||
H5Pget_attr_creation_order
h5pget_attr_creation_order_f
|
Retrieves tracking and indexing settings for attribute creation order. | |
H5Pget_attr_phase_change
h5pget_attr_phase_change_f
|
Retrieves attribute storage phase change thresholds. | |
H5Pget_obj_track_times
h5pget_obj_track_times_f
|
Determines whether times associated with an object are being recorded. | |
H5Pset_attr_creation_order
h5pset_attr_creation_order_f
|
Sets tracking and indexing of attribute creation order. | |
H5Pset_attr_phase_change
h5pset_attr_phase_change_f
|
Sets attribute storage phase change thresholds. | |
H5Pset_obj_track_times
h5pset_obj_track_times_f
|
Sets the recording of times associated with an object. | |
Object Copy Properties | ||
H5Padd_merge_committed_dtype_path
(none)
|
Adds a path to the list of paths that will be searched in the destination file for a matching committed datatype. | |
H5Pfree_merge_committed_dtype_paths
(none)
|
Clears the list of paths stored in an object copy property list. | |
H5Pget_copy_object
h5pget_copy_object_f
|
Retrieves the properties to be used when an object is copied. | |
H5Pget_mcdt_search_cb
(none)
|
Retrieves the callback function from the specified object copy property list. | |
H5Pset_copy_object
h5pset_copy_object_f
|
Sets the properties to be used when an object is copied. | |
H5Pset_mcdt_search_cb
(none)
|
Sets the callback function that H5Ocopy will invoke before searching the entire destination file for a matching committed datatype. | |
The following table lists link creation properties. Since the creation of a link is almost always a step in the creation of an object, these properties may also be set in group creation property lists, dataset creation property lists, datatype creation property lists, and the more generic object creation property lists. Some are also applicable to the attribute creation property lists.
Function Listing 3. Link creation property functions (H5P) | ||
---|---|---|
These properties can be used with any of the indicated property lists. | ||
C Function
Fortran Subroutine |
Purpose | |
H5Pget_char_encoding
h5pget_char_encoding_f
|
Queries the character encoding used to encode link or attribute names.
Any link, object, dataset, datatype, group, or attribute creation property list |
|
H5Pset_char_encoding
h5pset_char_encoding_f
|
Sets the character encoding used to encode link and attribute names.
Any link, object, dataset, datatype, group, or attribute creation property list |
|
H5Pget_create_intermediate_group
h5pget_create_intermediate_group_f
|
Queries setting for creation of intermediate groups.
Link creation property list, which in turn can be used in the create call for any dataset, datatype, or group |
|
H5Pset_create_intermediate_group
h5pset_create_intermediate_group_f
|
Specifies whether to create intermediate groups
when they do not already exist.
Link creation property list, which in turn can be used in the create call for any dataset, datatype, or group |
|
10.6. Additional Property List Resources
Property lists are ubiquitous in an HDF5 environment and are therefore
discussed in many places in HDF5 documentation.
The following sections and listings in the
HDF5 User’s Guide
are of particular interest:
All property list functions are described in the “H5P: Property List Interface” section of the HDF5 Reference Manual. The function index at the top of the page provides a categorized listing grouped by property list class. Those classes are listed below:
The in-memory file image feature of HDF5 uses property lists in a manner that differs substantially from their use elsewhere in HDF5. Those who plan to use in-memory file images must study “File Image Operations” (PDF) in the Advanced Topics in HDF5 collection.
H5P_FILE_MOUNT
is a valid HDF5 property list class,
no file mount properties are defined by the HDF5 Library.
References to a file mount property list should always be
expressed as H5P_DEFAULT
, meaning the
default file mount property list.
The following properties are file access properties but they are not transient; they have permanent and different effects on a file. They could be validly classified as file creation properties as they must be set at creation time to properly create the file. But they are access properties because they must also be set when a file is reopened to properly access the file.
|
Property | |
Related function |
---|---|---|---|
|
Family file driver | |
H5Pset_fapl_family |
|
Split file driver | |
H5Pset_fapl_split |
|
Core file driver |
|
H5Pset_fapl_core |
The following is a link creation property, but it is not relevant after an object has been created and is not retained in the file or object. | |||
|
Property | |
Related function |
|
Create missing intermediate groups | |
H5Pset_create_intermediate_groups |