3.1.14. MPI I/O for AMR

Currently, only the method='legacy' file format works with AMR, and then only for writing – restarts have not been implemented yet. A new method='snapshot' will be implemented for efficient saving of AMR data in a single data/run/NNNNN/snapshot.dat file per snapshot.

The structure of data on disk should be optimized for fast reading, and should have a format that is independent of the MPI geometry. This is achieved by (temporarily) numbering all patches in a rank 1...np, where np is the number of patches at the time of writing. Then such blocks from all ranks are arranged with each variable filling a contiguous piece in the snapshot.dat file; viz:

r1p1...r1pN, r2P1...r2pN, . . .,rMp1...rMpN

where M is the number of ranks and N is the number of patches per ranks (which in general is allowed to differ from patch to patch). A metadata file gives, for each sequence number, the offset into the file, and the previous task number::

seq  rank var task offset
 1    0    0   1    xx
 .
 N    0    0
 .    .
 N    M    0
 .    .    .
 N    M    V

A restart does not need to result in the same tasks residing on the same rank as before, but should achieve that whenever possible. After or while reading in the metadata, each rank reads in as many as its share of the load