Safely Recover MariaDB When the Service Will Not Start

When a MariaDB service failed to start, the safest response is to preserve the reason before trying to make the error disappear. Startup can fail because of an unknown option, unreadable data directory, full filesystem, port conflict, second server process, missing plugin, expired TLS file, systemd sandbox, OOM kill, interrupted upgrade, crash recovery that exceeds a service timeout, or genuine data corruption. These causes need very different actions.

Repeated restarts are especially harmful. They rotate or overwrite evidence, repeat crash recovery, consume I/O, trigger client reconnect storms, and can prompt operators to delete ib_logfile0, aria_log_control, socket files, or InnoDB tablespaces. Those files are not generic cache. Removing them can destroy the recovery path or hide a second process that is still writing to the same data directory.

This guide provides an evidence-first Linux/systemd runbook. It identifies the effective service and binary, captures journal and error logs, validates option-file precedence, checks disk, paths, permissions, mandatory access control, sockets, ports, resources, upgrades, plugins, and crash recovery. It treats innodb_force_recovery as an emergency extraction mode, not a repair. Commands must be adapted to the actual distribution, MariaDB version, topology, and data paths.

Define the failure state precisely

Do not begin with “database is down.” Determine whether:

The unit is failed, inactive, activating, or repeatedly restarting.
mariadbd starts and exits immediately.
systemd kills a healthy but slow startup at its timeout.
The process runs but the socket or TCP listener is unavailable.
Crash recovery is progressing slowly.
The server accepts local admin sessions but application readiness fails.
A cluster node refuses to join while standalone InnoDB is healthy.
The wrong service unit or instance is being managed.

Capture service state:

systemctl status mariadb --no-pager --full
systemctl show mariadb 
  --property=ActiveState 
  --property=SubState 
  --property=Result 
  --property=ExecMainCode 
  --property=ExecMainStatus 
  --property=MainPID 
  --property=NRestarts

Package/unit names can be mariadb, mysql, or an instance template. List candidates without starting them:

systemctl list-unit-files | rg 'mariadb|mysql'
systemctl list-units --all | rg 'mariadb|mysql'

Confirm the exact intended instance before changing anything.

Stop a restart loop and application amplification

If systemd or an external supervisor continually restarts MariaDB, stop the loop after evidence capture begins:

systemctl stop mariadb
systemctl status mariadb --no-pager --full

Also pause Kubernetes controllers, Pacemaker, Galera automation, cloud agents, or custom watchdogs that own the process. Do not fight two supervisors.

Stop application reconnect storms at the proxy or workload layer. Thousands of immediate connection attempts can consume CPU and obscure startup readiness.

Do not disable the service permanently during an incident unless the recovery plan requires it. Record who stopped automation and how it will be restored.

Preserve journal and error-log evidence

Capture the current boot and recent time window:

install -d -m 0750 /var/lib/db-diagnostics/mariadb-startup

journalctl -u mariadb -b --no-pager 
  > /var/lib/db-diagnostics/mariadb-startup/journal-current-boot.txt

journalctl -u mariadb --since '-2 hours' --no-pager 
  > /var/lib/db-diagnostics/mariadb-startup/journal-recent.txt

journalctl -k --since '-2 hours' --no-pager 
  > /var/lib/db-diagnostics/mariadb-startup/kernel-recent.txt

These files can contain paths, SQL, hostnames, and security details. Restrict ownership and retention.

Inspect on screen:

journalctl -u mariadb -b -n 300 --no-pager --full
journalctl -k -b -n 300 --no-pager --full

Look for the first causal error, not only systemd's final “failed with result exit-code.” Later messages are often consequences.

Find the configured error destination from parsed options when possible:

mariadbd --print-defaults

If log_error is empty under systemd, messages may go to the journal. A relative error-log filename commonly resolves under datadir. Search narrowly rather than scanning every large filesystem:

find /var/log /var/lib/mysql -maxdepth 2 -type f 
  ( -name '*.err' -o -name '*mariadb*.log' -o -name '*mysql*.log' ) 
  -ls 2>/dev/null

Do not truncate or rotate the active evidence until it is copied.

Record the change timeline

Startup failures usually follow a change. Record:

MariaDB package upgrade or downgrade.
Kernel, systemd, container, or OS upgrade.
Configuration deployment.
Datadir, tmpdir, log, socket, or certificate move.
Restore, snapshot attach, or storage migration.
Plugin install/removal.
Permission, SELinux, AppArmor, or unit hardening change.
Unclean shutdown, OOM, power loss, or full disk.
Cluster/bootstrap action.

Inspect installed version without starting the server:

mariadbd --version
mariadb --version

On Debian/Ubuntu:

dpkg-query -W -f='${Package}t${Version}n' 
  'mariadb-*' 'mysql-*' 2>/dev/null

On RPM systems:

rpm -qa | rg -i 'mariadb|mysql'

Mixed server, plugin, and library packages can cause startup failures. Do not reinstall blindly; package scripts can initialize, upgrade, or alter files.

Inspect the effective systemd unit

Distribution units include environment, pre-start scripts, timeouts, sandboxing, resource limits, and drop-ins.

systemctl cat mariadb
systemctl show mariadb 
  --property=FragmentPath 
  --property=DropInPaths 
  --property=ExecStart 
  --property=Environment 
  --property=User 
  --property=Group 
  --property=TimeoutStartUSec 
  --property=ProtectHome 
  --property=ProtectSystem 
  --property=ReadWritePaths 
  --property=LimitNOFILE 
  --property=TasksMax

Do not edit the vendor unit under /usr/lib/systemd/system or /lib/systemd/system; package upgrades overwrite it. Use a reviewed drop-in:

systemctl edit mariadb

After changing a drop-in:

systemctl daemon-reload
systemd-analyze verify mariadb.service

systemd-analyze verify checks unit structure, not MariaDB configuration or data integrity.

MariaDB's packaged unit often restricts /home, /root, and /run/user through ProtectHome, and makes parts of the filesystem read-only through ProtectSystem. A custom datadir or log under those paths can fail despite Unix ownership. Add the narrow required path rather than disabling all hardening.

Determine option-file order and groups

Ask the installed binary which files and groups it reads:

mariadbd --help --verbose 2>/dev/null | 
  sed -n '/Default options are read from/,/The following groups are read/p'

Print effective options:

mariadbd --print-defaults

On supported clients:

my_print_defaults --mariadbd

The --mariadbd switch is available only from MariaDB 10.11.3. Older versions can use explicit groups such as:

my_print_defaults mysqld server mariadb mariadbd

Option-file order depends on packaging and build. Later occurrences can override earlier settings. Server groups may include version-specific names and galera; a setting placed under [client] does not configure mariadbd.

Search configuration without printing secrets from client groups:

rg -n --glob '*.cnf' --glob 'my.cnf' 
  '^s*(!include|!includedir|[|datadir|socket|port|log_error|plugin|innodb|aria|wsrep)' 
  /etc/mysql /etc/my.cnf /etc/my.cnf.d 2>/dev/null

Read every included file in actual order. Duplicate variables, stale .cnf backups inside an included directory, and wrong section headers are frequent causes.

Validate configuration without starting MariaDB

MariaDB provides:

mariadbd --validate-config

It reads configuration and command-line options, exits zero when validation succeeds, and nonzero on failure without starting the server. Capture stderr and exit status:

mariadbd --validate-config
printf 'exit_status=%sn' "$?"

Run in an environment that reflects the service where practical. The unit can pass extra arguments, environment variables, or defaults files. Validation also may encounter path permissions different for root and the mysql service user.

Test as the service identity only after checking the command and writable paths:

sudo -u mysql mariadbd --validate-config

Do not start mariadbd manually as root against the production datadir. It can create root-owned files and conflict with systemd.

Common validation errors:

Unknown or removed variable after upgrade.
Invalid numeric unit or enum value.
Option placed in the wrong group.
Duplicate/conflicting plugin load.
Typographic Unicode quote or hidden character.
Missing included directory or unreadable file.
Setting copied from MySQL or another MariaDB major version.

The loose- option prefix suppresses failure for unsupported options. It is useful for intentional mixed-version fleets, but adding it during an incident can silently disable a critical setting. Understand the variable before using it.

Compare configuration against the installed version

Version transitions remove or change variables. Query official documentation and release notes for every startup error. Examples include removed InnoDB variables, authentication/plugin names, Galera options, and changed defaults.

Do not delete every unknown line. Classify it:

Obsolete and safely removable.
Renamed or replaced with a new control.
Plugin option whose plugin package is missing.
Safety/durability setting that requires an equivalent.
Typo that never worked.

Make the smallest change in an owned override file. Preserve the original and record a diff:

cp --preserve=all /etc/mysql/mariadb.conf.d/99-production.cnf 
  /var/lib/db-diagnostics/mariadb-startup/99-production.cnf.before

Use apply_patch, configuration management, or an editor that creates an auditable change. Do not keep backup files ending in .cnf inside an auto-included directory; MariaDB may read them too.

Re-run --validate-config after each change.

Verify datadir, log, tmpdir, socket, and PID paths

Extract paths from effective options and unit configuration. Common values include:

datadir
innodb_log_group_home_dir
tmpdir
log_error
socket
pid_file
plugin_dir
ssl_ca / ssl_cert / ssl_key

Inspect paths without modifying them:

namei -l /var/lib/mysql
stat /var/lib/mysql
findmnt -T /var/lib/mysql
df -hT /var/lib/mysql
df -i /var/lib/mysql

Check mount read/write state:

findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS -T /var/lib/mysql

Check the configured tmp and log paths similarly. A full inode table can fail writes even when df -h shows free bytes.

Do not initialize a directory merely because it appears empty. First confirm whether the intended volume failed to mount. Running mariadb-install-db on the wrong empty mount point can create a new system database that hides the missing production volume.

Compare mount UUID/device and cloud volume identity against infrastructure inventory.

Check ownership and traversal permissions safely

The service user needs appropriate access to every parent directory and write access to required locations.

namei -om /var/lib/mysql
find /var/lib/mysql -maxdepth 1 -printf '%M %u %g %pn'

Do not run:

chmod -R 777 /var/lib/mysql

This exposes database files and does not fix SELinux, AppArmor, systemd, ACL, or mount restrictions.

Do not blindly run recursive chown. Some files or plugin key material may intentionally have different ownership, and a large datadir traversal is expensive. Compare with package policy and repair only the proven incorrect paths.

Check ACLs:

getfacl -p /var/lib/mysql 2>/dev/null

Confirm the service identity:

systemctl show mariadb --property=User --property=Group
id mysql

An NFS or network datadir adds UID mapping, locking, durability, and support constraints. MariaDB's data directory generally requires storage semantics validated for database use.

Diagnose SELinux and AppArmor denials

On SELinux systems:

getenforce
ausearch -m AVC,USER_AVC -ts recent 2>/dev/null
ls -Zd /var/lib/mysql /custom/mariadb 2>/dev/null

Use distribution-supported labeling, such as semanage fcontext plus restorecon, after reviewing the intended path. Do not rely on a temporary chcon that disappears after relabeling.

Do not disable SELinux globally as the fix. Permissive mode can help isolate a denial in a controlled window, but it weakens the whole host and some controls can still behave unexpectedly.

On AppArmor systems:

aa-status 2>/dev/null
journalctl -k --since '-30 minutes' --no-pager | rg -i 'apparmor|denied'

Extend the packaged MariaDB profile narrowly for custom paths. Reload and test the policy; do not disable the profile permanently merely because ownership looks correct.

Check disk space, inodes, and filesystem health

MariaDB needs space for redo, temporary work, logs, binlogs, table growth, DDL, and recovery.

df -hT /var/lib/mysql /var/log/mysql /tmp
df -i /var/lib/mysql /var/log/mysql /tmp

Replace paths with actual configuration. Check read-only remounts and kernel errors:

journalctl -k --since '-2 hours' --no-pager | 
  rg -i 'I/O error|read-only|filesystem|ext4|xfs|nvme|blk'

Do not delete unknown files from the datadir to free space. InnoDB, Aria, binary logs, relay logs, and temporary-looking files may be active or recovery-critical.

Safe emergency space actions depend on ownership:

Rotate/compress external application logs outside the datadir.
Expand the volume according to platform procedure.
Move approved unrelated artifacts.
Purge binary logs only through MariaDB after startup and replication/PITR verification.

Never shell-delete binary logs that MariaDB still indexes.

If the filesystem reports corruption or hardware errors, stop write attempts and involve storage/recovery owners. Database recovery cannot make faulty storage trustworthy.

Check port, socket, PID, and second-process conflicts

Inspect running processes:

pgrep -a mariadbd
pgrep -a mysqld

Inspect listeners:

ss -lntp | rg ':3306b'
ss -lxnp | rg 'mysql|mariadb|mysqld'

If the log says it cannot lock aria_log_control or ibdata1 with resource-unavailable error, MariaDB documentation says another server commonly uses the same datadir. Do not delete those files or bypass locking. Identify the other PID, its service owner, datadir, and client traffic.

A stale socket or PID file after a confirmed dead process is possible, but prove no process holds the datadir or listener before removal:

lsof /var/lib/mysql/aria_log_control /var/lib/mysql/ibdata1 2>/dev/null
fuser -v /var/lib/mysql/aria_log_control /var/lib/mysql/ibdata1 2>/dev/null

Do not kill a process until you establish whether it is the real production instance. Two supervisors may refer to one database.

For a port conflict, change or stop the unintended listener through its owner. Moving MariaDB to another port can break clients, grants through proxies, firewalls, monitoring, and replication.

Diagnose OOM and system resource limits

Search kernel evidence:

journalctl -k --since '-2 hours' --no-pager | rg -i 'oom|out of memory|killed process'

Inspect service limits:

systemctl show mariadb 
  --property=MemoryCurrent 
  --property=MemoryMax 
  --property=TasksCurrent 
  --property=TasksMax 
  --property=LimitNOFILE 
  --property=LimitNPROC

If OOM caused shutdown, correct buffer pool, connection concurrency, cgroup limit, or competing processes before restart. Crash recovery also needs memory.

“Too many open files,” thread creation failures, or allocation errors require matching MariaDB and systemd/kernel limits. Raising limits without a resource budget can move failure to OOM.

Check process and host memory:

free -h
vmstat 1 5

Do not disable swap or drop caches during recovery.

Let crash recovery progress when it is healthy

After an unclean shutdown, InnoDB reads redo and restores consistency. Large redo distance, slow storage, or rollback can make startup exceed normal time.

Follow logs:

journalctl -u mariadb -f

In another session, observe I/O and process state:

pidstat -d -p "$(pgrep -o mariadbd)" 1 10
iostat -xz 1 10

An advancing log sequence, changing recovery percentage, or sustained legitimate I/O can indicate progress. Repeatedly killing the process restarts work and can extend downtime.

MariaDB's systemd unit may have a startup timeout around 90 seconds on some distributions. If recovery is healthy but systemd kills it at that exact boundary, create a reviewed timeout drop-in based on the installed unit documentation rather than disabling supervision entirely.

Example:

[Service]
TimeoutStartSec=infinity

An infinite timeout can hide a true hang. Prefer a tested finite recovery objective where supported and maintain external progress monitoring.

Diagnose plugin and shared-library failures

Startup can fail when plugin_load, plugin_load_add, plugin_maturity, plugin_dir, or a plugin-specific option references a missing/incompatible module.

Inspect effective plugin settings:

mariadbd --print-defaults | tr ' ' 'n' | rg 'plugin|provider|wsrep'

Check package files and dependencies without loading them:

find /usr/lib /usr/lib64 -path '*mariadb*plugin*' -maxdepth 5 -type f 
  2>/dev/null | sort

Use ldd only on the exact trusted plugin file if logs report a library dependency:

ldd /usr/lib/mysql/plugin/example.so

Do not load unknown shared objects or weaken plugin maturity policy. Align server and plugin package versions. Disabling an encryption, key-management, audit, or Galera plugin may make data unreadable or violate controls; escalate before removal.

Check TLS and key-management startup dependencies

MariaDB can fail when configured certificate, private key, CA, key-management file, socket, vault, or KMS endpoint is missing or unreadable.

Inspect paths and metadata without printing key material:

namei -l /etc/mysql/tls/server-key.pem
stat /etc/mysql/tls/server-cert.pem /etc/mysql/tls/server-key.pem
openssl x509 -in /etc/mysql/tls/server-cert.pem 
  -noout -subject -issuer -dates

Never use cat on private keys in terminal logs. Validate key/certificate match in a secure procedure and correct permissions for the service user.

For encrypted tables/logs, losing the encryption key is a data-recovery event. Do not remove encryption options to force start; encrypted data will not become plaintext. Restore the approved key provider and audit access.

External KMS/Vault dependencies need DNS, network, credentials, time, and certificate checks. Preserve provider errors.

Handle upgrade and downgrade failures correctly

Confirm binary and datadir history. Starting an older MariaDB binary on a datadir modified by a newer major release is not a safe rollback. On-disk formats, redo, system tables, and defaults can differ.

Do not run mariadb-upgrade as a startup fix while the server is down. Official workflow runs it after the new server starts. It updates system tables and checks table compatibility; it requires backup and datadir write access.

After the server starts on the intended version:

mariadb-upgrade --check-if-upgrade-is-needed

Interpret exit codes according to the installed tool's documentation. When required:

mariadb-upgrade --verbose

Use secure client credentials/options. Do not add --force repeatedly without understanding why the upgrade state is rejected; it can run checks/DDL and consume resources.

Safe rollback from a major upgrade usually means restoring a pre-upgrade physical backup/snapshot into the old version, not reusing the forward-modified datadir.

Recognize an accidentally wrong or empty datadir

Symptoms include missing mysql system tables, request to initialize, unexpected new server UUID/state, or an empty directory after a mount failure.

Verify effective datadir, mount, and identity:

mariadbd --print-defaults | tr ' ' 'n' | rg '^--datadir'
findmnt -T /var/lib/mysql
ls -la /var/lib/mysql

Do not run:

mariadb-install-db

until you have proven this is a genuinely new instance. On an existing production server, initialization can create fresh system tables in the wrong location and complicate recovery.

Restore the expected volume/mount or correct the configuration. Validate ownership and security contexts after reattachment.

Use innodb_force_recovery only for emergency extraction

If logs show InnoDB corruption that prevents startup and verified backups are insufficient, innodb_force_recovery may allow read access long enough to dump data.

Critical rules:

It does not repair corruption.
Corrupted files remain corrupted.
Higher levels disable more recovery work and carry greater inconsistency risk.
It is not a production operating mode.
Begin at 1 and increase one level at a time only if required.
Avoid writes and DDL.
Preserve a raw storage snapshot/copy before experiments where feasible.

Add an emergency owned option:

[mariadb]
innodb_force_recovery = 1

Start under isolation from application traffic, inspect logs, and dump the most valuable data first:

mariadb-dump --single-transaction 
  --routines --events --triggers 
  --databases critical_db 
  > /secure-recovery/critical_db.sql

--single-transaction may not work normally at higher recovery levels or with corruption, and non-InnoDB objects have different consistency semantics. Validate dump exit status and restore it into a clean server.

Remove recovery mode before rebuilding or normal service. The safe destination is a newly initialized supported instance restored from validated backup/dumps, not continued use of corrupted files.

Do not run REPAIR TABLE on InnoDB expecting physical repair. It is not the recovery method for InnoDB corruption.

Decide when to restore instead of repairing in place

Restore is usually safer when:

Storage hardware/filesystem is unreliable.
InnoDB corruption affects core structures.
A failed downgrade modified files.
Encryption keys cannot be recovered.
Emergency dumps are incomplete or inconsistent.
Startup experiments have altered the datadir.
The recovery-time objective favors a known backup.

Validate:

Backup completion and checksum.
Restore rehearsal results.
Binary log/GTID position for point-in-time recovery.
Encryption keys.
Schema, routines, events, users/grants, and plugins.
Application consistency checks.
Replica/cluster rebuild plan.

Preserve the failed volume read-only for forensics until retention policy permits deletion. Do not overwrite the only copy while testing recovery.

Start once under observation after the fix

Re-run gates:

mariadbd --validate-config
df -hT /var/lib/mysql
df -i /var/lib/mysql
systemctl cat mariadb

Start:

systemctl reset-failed mariadb
systemctl start mariadb

Follow logs in parallel:

journalctl -u mariadb -f

Check state:

systemctl is-active mariadb
systemctl status mariadb --no-pager --full

Confirm listener and socket:

ss -lntp | rg ':3306b'
ss -lxnp | rg 'mysql|mariadb'

Do not expose application traffic until recovery completes and health checks pass.

Verify database and topology health

Connect through the secured admin path:

SELECT VERSION(), @@hostname, @@port, @@server_id, NOW(6);
SHOW ENGINE INNODB STATUSG

Check:

No ongoing crash recovery or rollback blocking readiness.
Expected databases and critical tables exist.
Read and controlled write path works.
Error log has no new corruption or permission errors.
Buffer pool, connections, memory, disk, and latency are stable.
Events and schedulers have correct state.
TLS and authentication work.
Backup/monitoring agents reconnect.

For replication:

SHOW ALL REPLICAS STATUSG

Use older syntax where required. Verify IO/SQL threads, errors, GTID, lag, and topology identity. Do not start replication automatically if data lineage is uncertain.

For Galera, inspect wsrep_cluster_status, wsrep_local_state_comment, cluster size, readiness, and bootstrap ownership. Never bootstrap an arbitrary node without identifying the most advanced safe state; that can create split brain or data loss.

Roll back the emergency changes

Remove temporary settings that weaken safety or alter behavior:

innodb_force_recovery
Increased systemd timeout if not approved permanently
Temporary SELinux/AppArmor diagnostic state
Disabled plugins or encryption controls
Alternate datadir/socket/port
Reduced buffer pool or connection limits
Application traffic blocks and supervisor changes

Re-run config validation and perform a planned restart if required. Verify each security control returns to enforcing mode.

Keep the minimal permanent fix in configuration management. Delete stray .cnf diagnostic files from included directories only after preserving them in the incident record.

Troubleshoot common startup messages

Unknown variable or unknown option

The option is misspelled, in the wrong group, removed, or belongs to a missing plugin/other product. Check exact version docs and replace it intentionally; do not blanket-prefix loose-.

Permission denied on datadir or log

Check every parent path, ownership, ACL, mount options, SELinux/AppArmor, and systemd sandbox. Avoid 777 and broad hardening disablement.

Can't lock aria_log_control or ibdata1

Assume another MariaDB process may use the datadir until proven otherwise. Find process/service ownership; never delete the locked control/tablespace file.

No space left on device

Check blocks and inodes for datadir, log, tmpdir, and mounts. Expand or remove approved unrelated data; do not shell-delete MariaDB files.

Address already in use

Identify listener with ss/lsof. Stop the unintended owner or restore intended port. Do not run two instances on one datadir.

Startup killed after about 90 seconds

Check whether systemd timeout ended a progressing crash recovery. Extend timeout through a reviewed drop-in and monitor progress; do not assume every 90-second failure is harmless.

Plugin cannot be loaded

Align plugin package, server version, dependencies, plugin_dir, ownership, and MAC policy. Never disable a key-management plugin without confirming data remains accessible.

Table mysql.* does not exist

Confirm correct datadir/mount and upgrade history before initializing. A missing mount often looks like an empty instance.

InnoDB reports corruption

Stop repeated starts, preserve storage, validate backups, and plan restore. Use force recovery only for read extraction at the lowest workable level.

Server starts manually but not through systemd

Compare user, environment, options, paths, sandbox, limits, and working directory. Do not keep running an unmanaged root-started process against production data.

Monitoring and prevention

Monitor and alert on:

Unit failures, restart count, and startup duration.
MariaDB error-log fatal messages.
Disk bytes, inodes, read-only mounts, and I/O errors.
OOM and cgroup events.
Certificate and encryption-key expiry/access.
Config validation in CI before deployment.
Package/plugin version consistency.
Backup and restore-test age.
Crash-recovery duration and redo/checkpoint health.
Replication/cluster readiness after restart.
Correct datadir mount identity.

Pre-deployment gates should run:

mariadbd --validate-config
systemd-analyze verify mariadb.service

in an environment matching the target package and unit. A syntax check does not validate disk capacity, permissions, data formats, plugins, or runtime workload, so maintain a staging restart test too.

Best practices checklist

Stop automated restart and reconnect loops.
Preserve journal, kernel, and error logs first.
Identify exact service, binary, version, and datadir.
Inspect effective unit, drop-ins, options, and include order.
Run mariadbd --validate-config before startup.
Check disk blocks, inodes, mounts, and filesystem health.
Verify path traversal, ACL, SELinux/AppArmor, and systemd sandbox.
Treat lock-file errors as possible second instances.
Never initialize an unexpectedly empty datadir.
Let healthy crash recovery progress and adjust timeout deliberately.
Run mariadb-upgrade only after the intended new server starts.
Never delete redo, control, or tablespace files casually.
Use force recovery only for emergency read extraction.
Prefer validated restore over unbounded in-place experiments.
Verify data, writes, authentication, replication, and cluster state.

FAQ

Where should I look when MariaDB will not start?

Start with systemctl status mariadb, the systemd journal, and MariaDB error log. The first fatal message usually identifies configuration, path, resource, plugin, or recovery failure.

How do I validate MariaDB configuration without starting it?

Run mariadbd --validate-config. It exits zero for valid parsed configuration and nonzero on failure. Also inspect unit arguments and option-file precedence.

Should I delete ib_logfile0 if MariaDB fails to start?

No. It contains InnoDB recovery information. Deleting it after an unclean shutdown can cause data loss. Diagnose the exact log error and follow version-specific recovery.

Why does MariaDB start manually but fail under systemd?

Systemd can use a different user, options, environment, limits, and filesystem sandbox. Compare the effective unit rather than running production manually as root.

Does innodb_force_recovery repair InnoDB?

No. It may allow read access to extract data while bypassing recovery work. Corruption remains, and higher levels carry greater consistency risk.

Should I run mariadb-upgrade when the service is down?

No. The normal tool connects after the new MariaDB server starts. Resolve startup first, then run the required backed-up upgrade procedure.

Why does MariaDB say it cannot lock ibdata1?

Another process commonly has the same datadir open. Identify it before action. Do not delete ibdata1 or bypass locking.

Can a full filesystem prevent MariaDB startup?

Yes. MariaDB may need log, redo, temporary, socket, PID, or recovery writes. Check both free bytes and inodes on every configured path.

Conclusion

When a MariaDB service failed to start, the error log and service environment are the recovery map. Stop restart amplification, preserve evidence, identify the exact binary/unit/datadir, validate configuration, and check storage, permissions, security policy, listeners, resources, plugins, and version history in a controlled order. The first causal error matters more than the final systemd status.

Crash recovery needs time and capacity; corruption needs a validated restore or cautious data extraction. Never delete redo, Aria control, or InnoDB tablespace files to make an error vanish, and never initialize an unexpectedly empty path. Start once after the cause is corrected, then prove data, writes, security, replication, cluster, and application readiness. Recovery is complete only when both the database and its operational safeguards are healthy again.