Service Management and pg_ctl

Start, stop, reload, and inspect the PostgreSQL server process — and diagnose, fix, and recover from a failed startup

A running PostgreSQL cluster is a collection of processes managed by the postmaster. Controlling those processes — starting, stopping, reloading, restarting — is done with either pg_ctl (the PostgreSQL-native utility) or the operating system's service manager (systemctl on systemd systems).\n\npg_ctl operates directly on PGDATA and the postmaster process. It is always available, regardless of whether the package manager has configured a service unit. systemctl is the right tool on production Linux systems managed by systemd because it handles service dependencies, log capture via journald, and automatic restart on failure.\n\nFor stopping the server, the mode matters. -m smart waits for every client to disconnect on its own — the gentlest option, but it can hang indefinitely. -m fast (the default) cancels in-flight queries and rolls back open transactions before shutting down cleanly. -m immediate kills processes without cleanup — the next startup will need crash recovery. Use immediate only in emergencies.\n\nThe real test of this lab is not running the commands — it is what happens when pg_ctl start simply fails. pg_ctl will tell you it failed, but never why. The log file always has the answer: a syntax error, a permissions problem, a port already in use. Reading the log, fixing the root cause, and confirming the server is genuinely healthy afterward — not just "started" — is the actual skill this lab builds.

pg_ctl Command Reference

On this cluster the student account does not own PGDATA, so use the as-postgres helper to run pg_ctl as the postgres OS user.

# Check whether the server is running
as-postgres pg_ctl status -D $PGDATA

# Apply config changes (SIGHUP) without downtime
as-postgres pg_ctl reload -D $PGDATA

# Restart the server (applies postmaster-context changes)
as-postgres pg_ctl restart -D $PGDATA

# Stop the server cleanly (cancel queries, rollback transactions)
as-postgres pg_ctl stop -D $PGDATA -m fast

# Start the server
as-postgres pg_ctl start -D $PGDATA -l $PGDATA/pg_log/startup.log

systemctl on Production Linux

On systemd-based production systems you usually use the OS service manager instead of pg_ctl directly:

# Check service status
systemctl status postgresql

# Reload (SIGHUP)
systemctl reload postgresql

# Restart
systemctl restart postgresql

# Stop / Start
systemctl stop postgresql
systemctl start postgresql

Reading the PostgreSQL Log

The server log records startup, shutdown, and errors. On this VM logging_collector is off and pg_ctl start writes startup output to the file given with -l.

# Tail the startup log written by pg_ctl start
tail -f /var/lib/postgresql/18/data/pg_log/startup.log

# Or use pg_log/ under PGDATA for any log files created there
tail -f $PGDATA/pg_log/*.log

Key log events to recognise:

database system is ready to accept connections — server started successfully
received fast shutdown request — pg_ctl stop -m fast was issued
database system was shut down at ... — clean shutdown completed
database system was not properly shut down — crash recovery will run

Diagnosing a Failed Start

When pg_ctl start fails, it tells you almost nothing on its own:

pg_ctl: could not start server
Examine the log output.

That instruction is the whole procedure. The three most common root causes, and the log line each one produces:

| Cause | What the log says | |---|---| | Syntax error in postgresql.conf or pg_hba.conf | FATAL: syntax error in file "..." line N, near token "..." | | Port already in use | FATAL: could not bind IPv4 address ... Address already in use | | Wrong ownership/permissions on PGDATA | FATAL: data directory "..." has wrong ownership |

The recovery procedure is always the same shape:

# 1. Attempt start — it fails
as-postgres pg_ctl start -D $PGDATA -l $PGDATA/pg_log/startup.log

# 2. Read the log for the exact error
tail -n 20 /var/lib/postgresql/18/data/pg_log/startup.log

# 3. Fix the root cause (edit the offending line, free the port, fix ownership)

# 4. Start again
as-postgres pg_ctl start -D $PGDATA -l $PGDATA/pg_log/startup.log

# 5. Confirm genuine health, not just a "server started" message
psql -U postgres -c "SELECT pg_postmaster_start_time();"

A "server started" message only means the postmaster process is up. Step 5 — a query actually returning a fresh timestamp — is what proves the cluster is accepting and answering connections.

pg_ctl

The PostgreSQL control utility for starting, stopping, reloading, and checking the status of a server cluster. It always requires -D <pgdata> to locate the cluster (or the PGDATA environment variable). pg_ctl reads the postmaster.pid file to find the running server's PID and send signals to it. If postmaster.pid does not exist, pg_ctl status reports "no server is running".

Fast vs Immediate Shutdown

pg_ctl stop -m fast (the default) sends SIGTERM to all backend processes, causing them to cancel in-flight queries and roll back open transactions, then exit. The postmaster then shuts down cleanly. -m immediate sends SIGQUIT, killing all processes immediately without any rollback or cleanup. The next startup will perform crash recovery from WAL. Use immediate only when fast mode is hanging and you need the server down urgently.

Diagnosing a Failed Start

pg_ctl start failing is one of the most common real-world incidents a DBA handles. pg_ctl itself only ever reports that the start failed — it never inspects postgresql.conf or pg_hba.conf, so it has no idea why. The log file is the only place that information exists: a malformed config line produces an exact "syntax error ... line N, near token ..." message, a port conflict produces "Address already in use", and a permissions problem produces "wrong ownership". The fix is always: read the log, address the specific cause it names, start again, then confirm health with a real query — not just the "server started" message, which only confirms the process launched.

📋 Check Server Status with pg_ctl

Run pg_ctl status to confirm the server is running. Because the student account cannot read PGDATA directly, use the as-postgres helper. pg_ctl reads postmaster.pid to find the server PID — if the file exists, the server is running.

as-postgres pg_ctl status -D /var/lib/postgresql/18/data

pg_ctl: server is running (PID: 87) /usr/bin/postgres "-D" "/var/lib/postgresql/18/data"

🔌 Connect as postgres

Connect to the postgres database as the postgres superuser.

psql -U postgres

SET psql (18.4) Type "help" for help. postgres=#

📁 Find the Log File

Query pg_settings for the log directory, log file name pattern, and data directory, then identify the current log file path.

SELECT name, setting FROM pg_settings WHERE name IN ('log_directory', 'log_filename', 'data_directory') ORDER BY name;

name | setting ----------------+------------------------------------------ data_directory | /var/lib/postgresql/18/data log_directory | log log_filename | postgresql-%Y-%m-%d_%H%M%S.log (3 rows)

🔄 Reload via pg_ctl

First exit psql so you are back at the shell prompt. Then send a SIGHUP to the server using pg_ctl reload. This is equivalent to SELECT pg_reload_conf() from SQL but works from the OS without a database connection.

To leave psql: press Ctrl+D (or type \q and press Enter).

\! as-postgres pg_ctl reload -D /var/lib/postgresql/18/data

server signaled

⏱️Record Start Time Before Restart

Record pg_postmaster_start_time() while the server is still up. You will compare this after the restart to confirm the server cycled.

SELECT pg_postmaster_start_time();

pg_postmaster_start_time -------------------------------------- 2026-06-26 16:02:25.102021+00 (1 row)

🛑 Stop the Cluster with -m smart

Stop the cluster with pg_ctl stop -m smart. Smart mode waits for every connected client to disconnect on its own before shutting down — the gentlest option, though it can wait indefinitely if a session never closes.

\q

as-postgres pg_ctl stop -D /var/lib/postgresql/18/data -m smart

waiting for server to shut down....done server stopped

💥 Introduce a Syntax Error

Simulate a botched manual edit: append an invalid line to postgresql.conf. The next startup will fail to parse the file — exactly the scenario you are called in to fix.

echo "shared_buffers = 256MB oops" | as-postgres tee -a /var/lib/postgresql/18/data/postgresql.conf

shared_buffers = 256MB oops

▶️ Attempt to Start — and Watch It Fail

Try to bring the cluster back up with pg_ctl start. pg_ctl will report the failure but deliberately will not tell you why — that detail lives only in the log.

as-postgres pg_ctl start -D /var/lib/postgresql/18/data -l /var/lib/postgresql/18/data/pg_log/startup.log

waiting for server to start....stopped waiting pg_ctl: could not start server Examine the log output.

🔍 Read the Log to Find the Exact Error

Tail the log file to see exactly what PostgreSQL logged when it tried, and failed, to read postgresql.conf.

tail -n 20 /var/lib/postgresql/18/data/pg_log/startup.log

2026-06-26 16:02:25 UTC LOG: starting PostgreSQL 18.4 on x86_64-pc-linux-gnu 2026-06-26 16:02:25 UTC FATAL: syntax error in file "/var/lib/postgresql/18/data/postgresql.conf" line 22, near token "oops" 2026-06-26 16:02:25 UTC LOG: database system is shut down

🛠️ Fix the Configuration Error

Remove the bad line from postgresql.conf, then confirm shared_buffers now appears with valid syntax only.

as-postgres sed -i "/oops/d" /var/lib/postgresql/18/data/postgresql.conf

as-postgres grep -n shared_buffers /var/lib/postgresql/18/data/postgresql.conf

4:shared_buffers = 16MB

✅ Start Successfully

Start the cluster again. With the syntax error gone, postgresql.conf should now parse cleanly and the postmaster should come up.

as-postgres pg_ctl start -D /var/lib/postgresql/18/data -l /var/lib/postgresql/18/data/pg_log/startup.log

waiting for server to start....done server started

🩺 Confirm the Cluster Reached a Healthy State

A successful start message is not the same as a healthy cluster. Reconnect to psql and confirm it is actually accepting connections with a fresh pg_postmaster_start_time().

psql -U postgres

SELECT pg_postmaster_start_time();

pg_postmaster_start_time -------------------------------------- 2026-06-26 16:02:25.102021+00 (1 row)

Lab 2.1.5 complete — Block 2.1 complete! You can now manage the PostgreSQL service, including recovering from a failed start:\n\n\n pg_ctl status : ✅ check if server is running\n pg_ctl reload : ✅ apply sighup changes from shell\n pg_ctl stop -m smart : ✅ wait for clients to disconnect\n pg_ctl stop -m fast : ✅ clean shutdown\n Failed start : ✅ diagnosed from the log, not guessed\n Syntax error fixed : ✅ cluster recovered cleanly\n pg_postmaster_start_time : ✅ confirm genuine health\n

Enable JavaScript to run the live terminal and track your progress.