Sparse Files Become Colossal with io.Copy()

The Issue

When copying sparse files using io.Copy(), they often inflate dramatically in size at the destination.

Understanding the Nature of Sparse Files

io.Copy() transfers raw bytes, which masks the presence of holes in a sparse file – the spaces where no data actually resides. This information is not accessible through standard syscalls like read(2). As such, io.Copy() is incapable of preserving the sparseness of files.

Tackling the Problem

To address this, we need to delve deeper using the syscall package and manual handling. Lseek(2) system calls, with special SEEK_HOLE and SEEK_DATA values, can be used to manipulate holes in files.

Implementation

1. Use syscall.Seek() to control the location within the file.
2. Determine the SEEK_HOLE and SEEK_DATA values specific to the target platform.
3. Read from data-containing regions and ignore hole regions.

Considerations for Sparse File Transfer

If you want to maintain the sparseness of a file during transfer, the situation is more intricate. Fallocate(2) can be employed to attempt hole creation on supported filesystems. However, some filesystems inherently lack hole support, like the FAT family.

Additional Notes

1. Verify if the source and destination directories reside on the same filesystem. If so, using syscall.Rename() or os.Rename() instead of copying can preserve the file's size and attributes.
2. Refer to Go issue #13548 for further insight into sparse file handling in tar files.