Troubleshoot MariaDB Disk Full and Inode Exhaustion

A MariaDB disk full incident is not just a storage alert. The server may be unable to extend a tablespace, create a temporary file, rotate a binary log, write an error log, persist a transaction, or complete crash recovery. Applications can see failed writes while reads still work, replicas may stop, backups may become incomplete, and an automated restart can turn a degraded server into a full outage.

The dangerous part is that “disk” may refer to several different resources. The data directory can have free bytes while /tmp has none. A filesystem can report gigabytes available but have zero free inodes. A deleted multi-gigabyte log may still consume space because a process holds it open. A thin-provisioned volume can look healthy inside the guest while its backing pool is exhausted. Kubernetes can add ephemeral-storage eviction and persistent-volume limits to the same problem.

This guide presents an evidence-first recovery procedure for Linux and systemd environments. It identifies the affected write path, limits additional growth, frees space without deleting database internals, handles binary logs safely around replicas, validates InnoDB and replication after recovery, and builds capacity controls that prevent recurrence. Adapt every command to the actual MariaDB version, topology, storage layout, backup policy, and change process.

What MariaDB disk full actually means

MariaDB writes outside the obvious schema directories. A production server may depend on all of these locations:

MariaDB error 1021, ER_DISK_FULL, can say that the server is waiting for someone to free space. Other symptoms use operating-system wording such as No space left on device, errno: 28, Disk is full writing, Can't create/write to file, or a storage-engine-specific error. Do not assume every errno: 28 is byte exhaustion; inode exhaustion returns the same operating-system error.

Search intent and diagnosis goals

Someone searching for “MariaDB disk full” normally needs one of four outcomes:

1. Restore service without corrupting data.
2. Find what consumed the space.
3. Clean binary logs or temporary files safely.
4. Prevent the same outage with retention and capacity alerts.

The correct sequence is observe, contain, create headroom, recover, verify, prevent. Jumping directly to file deletion skips the information needed to distinguish a disposable artifact from part of the database recovery chain.

First response: contain growth and preserve evidence

Declare the write risk

Treat the node as storage-impaired. Pause deployments, schema migrations, bulk imports, reporting queries, and backup jobs that write to the affected storage. If the application supports a maintenance or read-only mode, use the application control plane rather than killing sessions blindly.

For a primary, coordinate traffic through the proxy or orchestrator. Do not promote a replica merely because the primary filesystem is full; first verify that the candidate is current, writable, and has adequate capacity. An unplanned promotion can create split brain or lose transactions.

Record the time and current state:

date -Is
hostnamectl
systemctl status mariadb --no-pager -l
journalctl -u mariadb --since '-30 min' --no-pager

Capture logs to a filesystem that still has space, or stream them to the incident terminal. Do not redirect evidence into the full mount.

Avoid restart loops

If MariaDB is still running, a restart usually makes recovery harder. The server may still serve reads and expose variables needed to locate files. Restarting requires PID/socket creation and may trigger crash recovery, which itself needs writable capacity.

If systemd is repeatedly restarting the service, stop the loop after collecting the current journal:

sudo systemctl stop mariadb
sudo systemctl reset-failed mariadb

Stopping a healthy-but-degraded primary has availability consequences. Use this only when the restart policy is amplifying failure, the server is already unusable, or the incident owner has chosen a controlled shutdown.

Step 1: identify every effective MariaDB path

When SQL remains available, query the running server rather than guessing from package defaults:

SHOW VARIABLES WHERE Variable_name IN (
  'datadir',
  'tmpdir',
  'log_bin',
  'log_bin_basename',
  'log_error',
  'relay_log',
  'slow_query_log_file',
  'general_log_file',
  'innodb_temp_data_file_path'
);

Also inspect configuration precedence:

mariadbd --print-defaults
mariadbd --help --verbose | sed -n '/Default options are read from/,/The following groups are read/p'

The service may supply additional options through an environment file or unit override:

systemctl cat mariadb
systemctl show mariadb -p ExecStart -p Environment -p EnvironmentFiles

Map paths to mounts. findmnt -T is more reliable than visually matching a long df listing:

findmnt -T /var/lib/mysql
findmnt -T /tmp
findmnt -T /var/log/mysql
df -hT /var/lib/mysql /tmp /var/log/mysql
df -i /var/lib/mysql /tmp /var/log/mysql

Replace these defaults with the discovered paths. A separate binlog or temporary volume must be checked separately.

Step 2: distinguish bytes, inodes, quotas, and hidden usage

Check byte and inode capacity

df -hT
df -i
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS

Important signals include:

• Use% at or near 100 percent for bytes.
• IUse% at 100 percent even though bytes remain.
• A read-only mount option after a filesystem or device error.
• An unexpected mount missing, leaving MariaDB to write into the underlying root filesystem.
• A small container overlay or ephemeral filesystem rather than the intended persistent volume.

Test path resolution without creating a file in the datadir:

namei -l /var/lib/mysql
findmnt -T /var/lib/mysql

Do not use touch inside the data directory as a casual permission test. Extra files complicate forensic review, and a successful tiny write does not prove MariaDB can extend a large tablespace or fsync it.

Find large consumers on the affected filesystem

Use du -x so the scan does not cross into other mounted filesystems:

sudo du -xhd1 /var/lib/mysql 2>/dev/null | sort -h
sudo du -xhd1 /var/log 2>/dev/null | sort -h
sudo find /var/lib/mysql -xdev -type f -printf '%st%pn' 2>/dev/null 
  | sort -n | tail -n 30

On a very busy or damaged filesystem, a full traversal adds I/O. Start with mount-level data and known directories, then deepen the scan deliberately.

Compare df and du. If df shows much more used space than du, look for deleted files held open:

sudo lsof +L1

A deleted log remains allocated until every process closes its file descriptor. Restarting the database solely to release an unrelated web-server log is unnecessary; rotate or restart the owning service. If MariaDB owns the deleted file, understand which log it is before using the appropriate FLUSH ... LOGS operation or a controlled restart.

Check quotas and thin provisioning

User, group, or project quotas can reject writes before the filesystem is globally full:

sudo quota -u mysql 2>/dev/null || true
sudo repquota -a 2>/dev/null || true

For LVM thin pools, inspect both data and metadata capacity:

sudo lvs -a -o lv_name,vg_name,lv_size,data_percent,metadata_percent,attr

Cloud block devices, SAN LUNs, ZFS pools, Ceph, and storage arrays need their own control-plane checks. Guest df alone cannot prove the backing storage is healthy.

Step 3: determine what grew

Database and table growth

When the server accepts queries, estimate schema allocation from metadata:

SELECT table_schema,
       ROUND(SUM(data_length + index_length) / 1024 / 1024 / 1024, 2) AS allocated_gib
FROM information_schema.tables
GROUP BY table_schema
ORDER BY SUM(data_length + index_length) DESC;

Then identify the largest tables:

SELECT table_schema, table_name, engine,
       ROUND((data_length + index_length) / 1024 / 1024 / 1024, 2) AS allocated_gib,
       table_rows
FROM information_schema.tables
ORDER BY (data_length + index_length) DESC
LIMIT 30;

These values are allocation estimates, not exact live-row bytes. InnoDB can retain free pages inside a tablespace. Never infer that an .ibd file is disposable because the table appears small.

Binary log growth

Check server-managed binlogs through SQL:

SHOW BINARY LOGS;
SHOW VARIABLES LIKE 'binlog_expire_logs_seconds';
SHOW VARIABLES LIKE 'expire_logs_days';

MariaDB 10.6 and later support binlog_expire_logs_seconds, which takes precedence when both expiry variables are nonzero. Expiration is not an immediate wall-clock deletion guarantee; log rotation participates in normal cleanup behavior.

On MariaDB 11.4 and later, max_binlog_total_size can limit total traditional binlog use, and binlog_disk_use reports usage when that limit is enabled. Version-check before relying on either variable:

SELECT VERSION();
SHOW VARIABLES LIKE 'max_binlog_total_size';
SHOW STATUS LIKE 'binlog_disk_use';

MariaDB 12.3 introduces an optional InnoDB-based binary log. Its storage and maintenance characteristics differ from traditional flat files, so identify binlog_storage_engine before applying a filesystem-oriented runbook.

Temporary and operational log growth

Large sorts, joins, ALTER TABLE, index creation, and reporting queries can spill to tmpdir or grow the InnoDB temporary tablespace. Check active statements and temporary-table counters:

SHOW FULL PROCESSLIST;
SHOW GLOBAL STATUS LIKE 'Created_tmp%';
SHOW VARIABLES LIKE 'tmpdir';
SHOW VARIABLES LIKE 'innodb_temp_data_file_path';

Also inspect whether general or slow logging was enabled unexpectedly:

SHOW VARIABLES WHERE Variable_name IN (
  'general_log', 'general_log_file',
  'slow_query_log', 'slow_query_log_file',
  'long_query_time', 'log_output'
);

Do not disable diagnostic logging reflexively during an incident. Confirm that it is the growth source and preserve the relevant evidence first.

Step 4: create emergency headroom safely

The immediate goal is enough free space for metadata writes, log rotation, and controlled recovery. It is not to redesign storage during the outage.

Prefer non-database artifacts

Candidates may include expired package caches, known disposable application caches, completed backup staging files already verified elsewhere, or rotated non-database logs covered by retention policy. Use the owning tool where possible:

journalctl --disk-usage
sudo logrotate -d /etc/logrotate.conf

logrotate -d is a dry run. Do not truncate active logs with shell redirection unless the incident procedure explicitly approves it. Truncation destroys evidence and can interact badly with file descriptors and rotation state.

Moving a file to another directory on the same filesystem does not free capacity. Moving it across filesystems requires enough destination space, preserves sensitive-data controls, and may create heavy I/O. Verify the mount boundary with findmnt -T before acting.

Never manually delete core MariaDB files

Do not remove any of these merely to gain space:

• InnoDB system or file-per-table tablespaces such as ibdata1 or .ibd files.
• Redo or undo files.
• aria_log_control or arbitrary Aria log files.
• Binary logs that are still registered in the binlog index.
• Relay logs while replication owns them.
• .frm, dictionary, system-table, encryption-key, or Galera state files.
• A temporary-looking file whose owner and lifecycle are unknown.

Filesystem deletion bypasses MariaDB metadata, replication safety, crash recovery, and backup/PITR policy. A server that starts after such deletion may be less recoverable than the stopped server.

Extend storage when that is the safest option

Adding capacity is often lower risk than deleting under pressure. The exact procedure depends on the volume manager and filesystem. Confirm that a snapshot, thin pool, cloud volume, partition, and filesystem all have compatible expansion steps. Never paste a generic resize2fs, xfs_growfs, or LVM command without identifying the actual stack.

After expansion, prove that the filesystem sees it:

findmnt -T /var/lib/mysql
df -hT /var/lib/mysql
df -i /var/lib/mysql

Safely purge MariaDB binary logs

Binary logs are a common large consumer, but they are also the replication stream and point-in-time recovery chain. Use SQL, not rm.

Inventory primary and replicas

On the primary:

SHOW BINARY LOGS;
SHOW MASTER STATUS;

On every replica, capture replication position and health:

SHOW REPLICA STATUSG

Older releases and installations may still use SHOW SLAVE STATUSG. Determine the earliest binlog any replica still needs. Include disconnected replicas, delayed replicas, backup consumers, CDC connectors, and recovery policy. MariaDB can prevent a purge for a connected replica that still needs a file, but it cannot protect a disconnected consumer whose requirement the primary cannot see.

Purge through MariaDB

Purge files strictly before a verified retained file:

PURGE BINARY LOGS TO 'mariadb-bin.000123';

The named file remains; earlier files are removed. A time-based purge is also available:

PURGE BINARY LOGS BEFORE '2026-08-18 00:00:00';

File-position purging is usually easier to audit against replica requirements. Do not run RESET MASTER as emergency cleanup: it removes the complete binlog set and resets indexing, which can break replication and point-in-time recovery.

After purging, confirm both MariaDB state and filesystem capacity:

SHOW BINARY LOGS;

df -hT /path/to/binlog

Handle inode exhaustion

An inode represents a filesystem object. Millions of tiny files can exhaust inodes while byte capacity looks healthy. Confirm with:

df -i /var/lib/mysql
sudo find /affected/mount -xdev -printf '%hn' 2>/dev/null 
  | sort | uniq -c | sort -n | tail -n 30

The directory-count scan can be expensive, so constrain it after identifying likely consumers. Common causes outside MariaDB include runaway session files, mail queues, application caches, container layers, and monitoring spool files.

Do not solve inode exhaustion by deleting unknown files from the datadir. Apply the retention mechanism of the actual producer. Longer term, reduce file churn, move the workload to an appropriate filesystem, or recreate the filesystem with a suitable inode design during a planned migration. Filesystem recreation is destructive and requires a validated backup and restore plan.

Recover MariaDB after capacity is restored

Recheck ownership, mounts, and configuration

An emergency remount or volume replacement may change ownership, labels, or mount timing:

findmnt -T /var/lib/mysql
namei -l /var/lib/mysql
sudo stat -c '%U:%G %a %n' /var/lib/mysql
sudo mariadbd --validate-config

Do not recursively chown a large datadir unless evidence proves ownership is wrong and the intended owner is known. It adds I/O and can alter files that deliberately have different permissions.

Start once and observe

sudo systemctl start mariadb
sudo journalctl -u mariadb -f

Use a second terminal for capacity:

watch -n 2 'df -h /var/lib/mysql; df -i /var/lib/mysql'

Crash recovery can consume additional space and I/O. If usage rises quickly, do not launch repeated restarts. Capture the first causal error and reassess headroom.

Verify database health

sudo systemctl is-active mariadb
mariadb-admin ping

Then validate server identity and write behavior using a dedicated health-check account and disposable test schema approved for the environment:

SELECT VERSION(), @@hostname, @@server_id, @@read_only;
SHOW ENGINE INNODB STATUSG

Avoid using an existing application table for a write test. A controlled transaction in a designated health schema can prove create, insert, commit, read, and cleanup without touching business data.

Review fresh error-log entries, not only the service exit code:

journalctl -u mariadb --since '-15 min' --no-pager

Verify replication, Galera, and backups

A recovered standalone process is not automatically a recovered database service.

For each replica, check both receiver and applier state, lag, last errors, and GTID or file position:

SHOW REPLICA STATUSG

If a required binlog was deleted outside MariaDB, do not fabricate a position. Reprovision from a known-consistent backup or use the topology’s documented recovery method.

For Galera, verify membership and readiness:

SHOW GLOBAL STATUS WHERE Variable_name IN (
  'wsrep_cluster_size',
  'wsrep_cluster_status',
  'wsrep_connected',
  'wsrep_ready',
  'wsrep_local_state_comment'
);

Do not bootstrap a new primary component solely because one node filled its disk. Cluster bootstrap decisions require authoritative membership and sequence-state analysis.

Run the next scheduled backup only after enough free capacity exists for both the backup process and normal database growth. Validate backup output, restoreability, and the surviving binlog range required for PITR. A “successful” backup job that wrote to a full target may be truncated or absent.

Kubernetes-specific checks

In Kubernetes, inspect the pod, PVC, events, and node ephemeral storage separately:

kubectl -n database get pod,pvc
kubectl -n database describe pod mariadb-0
kubectl -n database get events --sort-by=.lastTimestamp
kubectl -n database exec mariadb-0 -- df -hT
kubectl -n database exec mariadb-0 -- df -i

Replace namespace and pod names. A PVC can have room while the container writable layer or node ephemeral storage is exhausted. Conversely, increasing a PVC object does not guarantee that the storage class supports expansion or that the filesystem inside the pod has grown.

Do not delete a StatefulSet PVC as a cleanup action. PVC retention and reclamation policies can make that destructive. Follow the operator’s documented resize and recovery workflow, and verify application-level health after the storage layer reports success.

Common mistakes and why they fail

Deleting ibdata1, redo, or .ibd files

These are database structures, not cache. Deletion can make tables inaccessible, invalidate crash recovery, and turn a capacity incident into data loss.

Running rm mariadb-bin.*

Manual removal desynchronizes the binlog index and can strand replicas or invalidate PITR. PURGE BINARY LOGS performs server-aware maintenance.

Restarting until it works

Each attempt can repeat recovery, consume the last blocks, rotate evidence, and extend outage time. Create measured headroom and correct the cause before one observed start.

Trusting only df -h

It misses inode exhaustion, quotas, deleted-open files, read-only remounts, thin-pool exhaustion, separate tmpdir, and backing-storage limits.

Moving files within the same mount

Renaming changes directory entries, not allocated blocks. Confirm source and destination mounts before claiming space was freed.

Purging based only on replica lag seconds

Lag is not a complete statement of required binlog files, especially for stopped, delayed, or external consumers. Record actual positions and recovery requirements.

Returning all traffic immediately

Recovery may still be consuming capacity, replicas may be behind, and the original growth source may still run. Ramp traffic while monitoring bytes, inodes, write errors, latency, and replication.

Prevention and production best practices

Alert on both capacity and time to exhaustion

Monitor:

• Free bytes and percentage for every MariaDB write path.
• Free inodes and inode percentage.
• Growth rate and estimated time to exhaustion.
• Binlog and relay-log bytes.
• Datadir, temporary, backup, audit, general, slow, and error-log growth.
• LVM thin-pool data and metadata usage.
• Cloud-volume burst, IOPS, latency, and queue metrics.
• Kubernetes PVC, node ephemeral storage, and eviction signals.

Static thresholds alone are late for a fast import. Combine warning/critical floors with trend alerts, and reserve enough headroom for peak writes, online DDL, crash recovery, backups, and operational response.

Define a binlog retention policy

Retention must exceed the longest credible replica outage and satisfy PITR requirements. On supported releases, configure seconds explicitly:

[mariadb]
binlog_expire_logs_seconds = 1209600

This example is 14 days, not a universal recommendation. Choose the value from business recovery objectives, replica behavior, change windows, and storage capacity. Validate the effective value after restart and ensure rotation occurs as expected.

MariaDB 11.4+ environments can evaluate max_binlog_total_size, but a size cap must not undermine retention or disconnected replicas. Version-sensitive controls belong in tested configuration management, not an incident-time paste.

Separate workloads deliberately

Separate volumes for data, binlogs, temporary work, and backups can contain a growth event, but each new mount adds capacity, monitoring, backup, security, and startup dependencies. Separation without alerts merely creates more places to fill.

Keep backups off the production data filesystem. A backup intended to protect the database should not consume the last space the database needs to operate.

Control unbounded producers

• Keep the general query log off unless actively required and bounded.
• Rotate and retain slow/error/audit logs according to policy.
• Put lifecycle rules on backup staging and exports.
• Review application retention for event, session, and audit tables.
• Estimate temporary-space impact before large DDL or analytical queries.
• Test cleanup jobs and alert when they stop running.

Practice recovery

Run a non-production exercise that fills a dedicated test filesystem, triggers byte and inode alerts, pauses growth, frees approved artifacts, validates binlog positions, and restores service. Never fill a shared production filesystem for a drill. The exercise should prove the runbook, monitoring, escalation path, and restore chain.

Troubleshooting checklist

1. Timestamp the incident and preserve journal/error-log evidence.
2. Pause bulk writes, backups, DDL, and restart amplification.
3. Discover effective datadir, tmpdir, log, binlog, relay, and backup paths.
4. Map each path with findmnt -T.
5. Check bytes, inodes, mount flags, quotas, and backing storage.
6. Compare df with du and inspect deleted-open files.
7. Identify whether growth is data, binlog, relay, temp, logs, or backups.
8. Create headroom from approved non-database artifacts or expand storage.
9. Purge binlogs only through SQL after checking every consumer.
10. Start once, observe recovery and capacity, then validate SQL writes.
11. Verify replicas, Galera, backups, PITR continuity, and application health.
12. Add retention, capacity forecasts, and growth-source controls.

FAQ

Can I delete MariaDB binary logs with rm when the disk is full?

No. Inventory replica and PITR requirements, then use PURGE BINARY LOGS. Manual deletion can break the binlog index and downstream consumers.

Why does MariaDB report no space when df -h shows free space?

Check df -i, quotas, separate tmp/log mounts, deleted-open files, read-only mounts, thin pools, and container ephemeral storage. Byte capacity is only one limit.

Should I restart MariaDB after freeing a little space?

Not automatically. If it is running, verify whether operations resume first. If stopped, create sufficient recovery headroom, validate mounts/configuration, and perform one observed start.

Is ibtmp1 safe to delete?

Do not delete it while MariaDB is running. InnoDB temporary tablespaces have a managed lifecycle and are recreated after a graceful shutdown/start; use version-appropriate documented controls.

How much free disk should MariaDB keep?

There is no universal percentage. Reserve for peak growth, temp work, online DDL, binlogs, backups, and crash recovery, then alert on both a hard floor and time to exhaustion.

Can a full disk corrupt InnoDB?

InnoDB is designed around crash recovery, but failed writes, abrupt termination, hardware/filesystem errors, and unsafe manual deletion can still cause damage. Restore capacity, preserve logs, and verify engine health.

Why did deleting a large log not change df?

A running process may still hold the deleted inode open. Use lsof +L1, identify the owner, and rotate or restart that service through its supported procedure.

Does automatic binlog expiration guarantee immediate cleanup?

No. Expiry behavior involves log management and rotation, and retention must still protect replicas and PITR. Monitor actual files and disk use rather than trusting configuration alone.

Conclusion

A MariaDB disk full recovery succeeds when it protects data continuity, not merely when df shows a lower percentage. Preserve evidence, contain new writes, map every effective path, and distinguish byte exhaustion from inodes, quotas, open-deleted files, read-only mounts, and backing-storage limits. Free headroom from known non-database artifacts or expand storage; never delete tablespaces, redo, relay logs, or registered binlogs from the filesystem.

Once capacity returns, start MariaDB only when necessary and observe the attempt. Verify SQL reads and controlled writes, InnoDB, replication or Galera, backups, and point-in-time recovery continuity before restoring full traffic. Finally, convert the incident into byte and inode alerts, growth forecasting, tested retention, capacity reserves, and a practiced recovery procedure.

Suggested Internal Links

• Safely Recover MariaDB When the Service Will Not Start
• Troubleshoot MariaDB High Memory Usage and OOM Kills
• Manage MariaDB Binary Logs for Replication and PITR
• Back Up MariaDB with mariadb-backup
• Monitor MariaDB Performance and Capacity
• Run MariaDB on Kubernetes with Persistent Storage

Suggested External Sources

• MariaDB Error Code Reference
• Using and Maintaining the Binary Log
• SHOW BINARY LOGS
• MariaDB InnoDB Temporary Tablespaces
• MariaDB Error Log
• MariaDB InnoDB Troubleshooting