MinIO Disk Partitions and a Wrong Hypothesis

Partitioning and Formatting

First, find the disk. I prefer lsblk over fdisk -l because this shows the tree of disks, partitions, mount points, etc. at a glance. You’re looking for the drive with no mount points, such as /dev/sdb, which I’ll call /dev/sdX below.

lsblk
sudo fdisk -l /dev/sdX

Then create the partitions with fdisk.

sudo fdisk /dev/sdX

Inside fdisk, press g first to make a GPT partition table. The older MBR table only allows 4 primary partitions and can’t address more than 2 TiB, and any HDD you’d buy for object storage today is bigger than that. Then for each partition press n, Enter for the default number, Enter for the default first sector, and type a size such as +500G for the last sector. Press w at the end to write the table and exit.

Next, format every partition as XFS and give each one a label. The label is what we’ll mount by later, and XFS caps labels at 12 characters, so keep them short.

for i in {1..4}; do
  sudo mkfs.xfs -f -L MINIODISK$i /dev/sdX$i
done

Note the range is {1..4} with two dots. I had three in my own notes at first, and bash doesn’t expand that at all, it just passes along the literal path /dev/sdX{1...4}.

Mounting (and Keeping It Mounted)

Create the mount points and mount everything.

sudo mkdir -p /mnt/minio/disk{1..4}
for i in {1..4}; do
  sudo mount LABEL=MINIODISK$i /mnt/minio/disk$i
done

To survive a reboot, the mounts go in /etc/fstab. Don’t use /dev/sdX1 there, because device names can shuffle between boots if you add or remove a disk, and then the wrong disk shows up at the wrong path. MinIO’s docs recommend mounting by label over UUID, because if a disk dies, you format the replacement with the same label and fstab doesn’t change.¹

LABEL=MINIODISK1 /mnt/minio/disk1 xfs defaults,noatime 0 0
LABEL=MINIODISK2 /mnt/minio/disk2 xfs defaults,noatime 0 0
LABEL=MINIODISK3 /mnt/minio/disk3 xfs defaults,noatime 0 0
LABEL=MINIODISK4 /mnt/minio/disk4 xfs defaults,noatime 0 0

noatime stops the kernel from writing an access timestamp on every read, which MinIO has no use for.

Finally, test it before you reboot. Stop MinIO first, otherwise umount fails because the mounts are busy and mount -a has nothing to test. If mount -a complains, fix it now, because a broken fstab entry can drop the machine into emergency mode on the next boot and that’s a much worse time to find out.

sudo umount /mnt/minio/disk{1..4}
sudo mount -a
df -h | grep minio

One Disk, One Actuator

So all of this works, and then MinIO got slower. I did benchmark it, but I didn’t keep the numbers around. The one I do remember is the job reading from MinIO: about 10 minutes with the disk as a single partition, and about 17 with four.

The reason is mechanical. An HDD has a read/write head for each platter surface, but every head is mounted on a single actuator (a stack of arms that pivots together), so the heads can’t go to different places at the same time. To read anything, the whole actuator swings to the right track and then waits for the platter to spin the right sector underneath it, and on a 7200 RPM drive that costs several milliseconds every time. Some enterprise drives (Seagate’s MACH.2, for example) have two independent actuators, and those show up to the OS as two separate devices.² My drive wasn’t.

MinIO, of course, has no idea. It sees four mounts and treats them as four independent drives. Its erasure coding splits every object into data and parity shards and puts one shard on each drive in the set, so a single write became four writes to four different regions of the same platters. A big file the disk could have streamed in a single sequential pass turned into the actuator bouncing between partitions, and seeking is the thing an HDD is worst at.

It also didn’t buy me any safety. Erasure coding exists so you can lose a drive, but four partitions on the same disk die together.

What I’d do now is simple. With a single HDD, give MinIO the whole disk as a single XFS mount and accept that it has no redundancy beyond the disk itself (MinIO supports that single-drive setup fine). If you want speed or redundancy, buy more physical disks and use the steps above with one partition per disk. Labels have to be unique across every attached disk, so name them per disk (MINIODISK1 on the first disk, MINIODISK2 on the second, etc.). And if you’re on SSD or NVMe, there are no moving heads, parallel requests are cheap, and MinIO recommends flash anyway, but partitioning a single SSD still gets you zero redundancy.

1. MinIO’s own deployment docs use exactly this LABEL=... defaults,noatime style in their fstab example.↩︎

2. Tom’s Hardware has a good write-up on MACH.2 if you’re curious how that works.↩︎