PostgreSQL案例 on Last DBA

Case Study: Startup Failure and SysV Shared Memory

Mon, 09 Mar 2026 00:00:00 +0000

Problem Symptoms
#

The database instance’s RSS memory was maxed out, OOM messages appeared in the logs, and the instance died. We won’t analyze the OOM cause here.

But startup kept failing — 4 or 5 attempts according to the logs:

2026-02-12 09:15:21 CST::@:[578272]: FATAL: pre-existing shared memory block (key 2048, ID 1328250881) is still in use
2026-02-12 09:15:21 CST::@:[578272]: HINT: Terminate any old server processes associated with data directory "/data".
2026-02-12 09:15:21 CST::@:[578272]: LOG: database system is shut down
2026-02-12 09:21:03 CST::@:[658824]: FATAL: pre-existing shared memory block (key 2048, ID 1328250881) is still in use
2026-02-12 09:21:03 CST::@:[658824]: HINT: Terminate any old server processes associated with data directory "/data".
2026-02-12 09:21:03 CST::@:[658824]: LOG: database system is shut down
2026-02-12 09:31:12 CST::@:[794791]: LOG: redirecting log output to logging collector process
2026-02-12 09:31:12 CST::@:[794791]: HINT: Future log output will appear in directory "/data/pg_log".
2026-02-12 09:31:37 CST::@:[801049]: FATAL: lock file "postmaster.pid" already exists
2026-02-12 09:31:37 CST::@:[801049]: HINT: Is another postmaster (PID 794791) running in data directory "/data"?
2026-02-12 09:32:34 CST::@:[814396]: FATAL: lock file "postmaster.pid" already exists
2026-02-12 09:32:34 CST::@:[814396]: HINT: Is another postmaster (PID 794791) running in data directory "/data"?

Startup succeeded after the DBA ran ipcrm -m xxx before starting.

Although the issue was quickly resolved, many questions remained:

Why isn’t this scenario more common in practice?
The start.log shows two different error types — what operations and logic do they correspond to?
Can shared memory still exist even if the postmaster is gone?
How do you locate and clean up this shared memory segment?
PG has multiple shared memory segments — which one is this?
Besides ipcrm -m, are there other ways to get the instance started?

Error Analysis: `pre-existing shared memory block`
#

Three Types of Shared Memory
#

Normally, after PG starts, there are three shared memory segments.

Using the default shared_memory_type='mmap' without huge pages as an example:

## View PG's actual shared memory usage from its virtual memory map
cat /proc/`head -1 $PGDATA/postmaster.pid`/smaps | grep -E "\-s"

2b61b0563000-2b61b0564000 rw-s 00000000 00:04 116293664 /SYSV00001000 (deleted)
2b61b057f000-2b61b05b3000 rw-s 00000000 00:12 1501001168 /dev/shm/PostgreSQL.1193490778
2b61bbac2000-2b61fa67a000 rw-s 00000000 00:04 1500999610 /dev/zero (deleted)

From top to bottom, these are: the SysV shared memory used at startup, shared memory for parallel queries, and shared memory for shared_buffers.

If shared_buffers uses huge pages, or if the shared_memory_type is SysV instead of mmap, the output differs slightly.

Huge pages:

2aaaaac00000-2aba9ca00000 rw-s 00000000 00:0e 48453452 /anon_hugepage (deleted)
2b08f2eea000-2b08f2eeb000 rw-s 00000000 00:04 50692152 /SYSV00001000 (deleted)
2b08f2f05000-2b08f302d000 rw-s 00000000 00:12 48436142 /dev/shm/PostgreSQL.1345689218

shared_memory_type = ‘sysv’:

2b03b3ceb000-2b03b3d1f000 rw-s 00000000 00:12 1572332304 /dev/shm/PostgreSQL.2883611352
2b03bf0c2000-2b03fdc7a000 rw-s 00000000 00:04 143917075 /SYSV00001000 (deleted)

Summary:

PG Shared Memory Config	smaps Segments	shared_buffers smaps	sysv smaps
shared_memory_type=mmap, no huge pages	3 segments	/dev/zero	/SYSV00001000
shared_memory_type=sysv, no huge pages	2 segments	/SYSV00001000	/SYSV00001000
shared_memory_type=mmap, with huge pages	3 segments	/anon_hugepage	/SYSV00001000
shared_memory_type=sysv, with huge pages	not supported	not supported

Now the key question: when the error says pre-existing shared memory block, which shared memory segment is it talking about?

Source Code Analysis
#

Searching for the error message in the source quickly leads to the key location: src/backend/port/sysv_shmem.c

First, understand what the SysV shmem is for. From scattered README content:

We still require a SysV shmem block to
 * exist, though, because mmap'd shmem provides no way to find out how
 * many processes are attached, which we need for interlocking purposes.

 * As of PostgreSQL 9.3, we normally allocate only a very small amount of
 * System V shared memory, and only for the purposes of providing an
 * interlock to protect the data directory. The real shared memory block
 * is allocated using mmap(). This works around the problem that many
 * systems have very low limits on the amount of System V shared memory
 * that can be allocated. Even a limit of a few megabytes will be enough
 * to run many copies of PostgreSQL without needing to adjust system settings.

SysV shmem can determine whether shared memory is still attached; mmap cannot
This SysV shmem is used to protect the data directory; shared_buffers uses mmap (by default), not SysV
This SysV shmem segment is tiny (from the virtual addresses we can see it’s just 4K = 2b61b0563000-2b61b0564000)

Now look at the shm state enum:

typedef enum
{
	SHMSTATE_ANALYSIS_FAILURE,	/* unexpected failure to analyze the ID */
	SHMSTATE_ATTACHED,			/* pertinent to DataDir, has attached PIDs */
	SHMSTATE_ENOENT,			/* no segment of that ID */
	SHMSTATE_FOREIGN,			/* exists, but not pertinent to DataDir */
	SHMSTATE_UNATTACHED			/* pertinent to DataDir, no attached PIDs */
} IpcMemoryState;

The key states are ATTACHED, FOREIGN, and UNATTACHED.

The SysV shmem protects the data directory — the common scenario is ensuring the directory isn’t running two instances. Since it’s shared memory, weird scenarios could mean the segment doesn’t belong to this directory or this process (FOREIGN state). If the shared memory corresponds to the data directory but no processes are running, it should be UNATTACHED. With processes running, it’s ATTACHED.

Now look at the error thrown by PGSharedMemoryCreate:

PGShmemHeader *
PGSharedMemoryCreate(Size size,
					 PGShmemHeader **shim)
{...
 for (;;) // infinite loop
	{..
 shmid = shmget(NextShmemSegID, sizeof(PGShmemHeader), 0);// shmget to fetch the SysV shmem and return its shmid
		if (shmid < 0)
		{
			oldhdr = NULL;
			state = SHMSTATE_FOREIGN;
		}
		else
			state = PGSharedMemoryAttach(shmid, NULL, &oldhdr);// determine this shmem segment's state

 switch (state)// take different actions based on the shared memory state
		{
 ...// only showing 2 states here: attached and unattached
			case SHMSTATE_ATTACHED: // shm is attached — throw the error (this is the fault symptom we saw)
				ereport(FATAL,
						(errcode(ERRCODE_LOCK_FILE_EXISTS),
						 errmsg("pre-existing shared memory block (key %lu, ID %lu) is still in use",
								(unsigned long) NextShmemSegID,
								(unsigned long) shmid),
						 errhint("Terminate any old server processes associated with data directory \"%s\".",
								 DataDir)));
				break;
 ...
			case SHMSTATE_UNATTACHED:// shm is unattached

				/*
				 * The segment pertains to DataDir, and every process that had
				 * used it has died or detached. Zap it, if possible, and any
				 * associated dynamic shared memory segments, as well. This
				 * shouldn't fail, but if it does, assume the segment belongs
				 * to someone else after all, and try the next candidate.
				 * Otherwise, try again to create the segment. That may fail
				 * if some other process creates the same shmem key before we
				 * do, in which case we'll try the next key.
				 */
 // The segment belongs to the data directory, and no process still holds it
				if (oldhdr->dsm_control != 0)
					dsm_cleanup_using_control_segment(oldhdr->dsm_control);
				if (shmctl(shmid, IPC_RMID, NULL) < 0)
					NextShmemSegID++; // Note: ShmemSegID increments and retries
				break;
		}
 ...
 }
 ...
 }

When shmem is ATTACHED, it throws the error. When unattached, it loops infinitely, trying to clean up the segment and incrementing ShmemSegID to request a new one.

The first case corresponds to this fault
The second case corresponds to normal crash recovery (instance can still start after a crash)

SysV shmem
#

From PG10 onwards, the postmaster.pid and SysV shmem logic was significantly reworked and has been largely stable since. This article only covers the PG10+ logic.

pidfile.h:

#define LOCK_FILE_LINE_SHMEM_KEY	7

sysv_shmem.c, InternalIpcMemoryCreate():

	{
		char		line[64];

		sprintf(line, "%9lu %9lu",
				(unsigned long) memKey, (unsigned long) shmid);
		AddToDataDirLockFile(LOCK_FILE_LINE_SHMEM_KEY, line);
	}

From the source code, shmem info is saved on line 7 of postmaster.pid, containing the shmkey and shmid.

> cat postmaster.pid
242712
/data
1772698474
8531
/tmp
0.0.0.0
 4096 143917078 # <----here
ready

What Are shmkey and shmid?
#

In PG’s source, the call path is: InternalIpcMemoryCreate():

			shmid = shmget(memKey, 0, IPC_CREAT | IPC_EXCL | IPCProtection);

PG uses shmkey/memkey as a seed key to request shared memory from the kernel, which returns a unique identifier, shmid.

shmid is highly dependent on the server or rather the server’s memory state. For PG, when quickly restarting an instance, the shmid may be the same or +1 — this depends on Linux kernel internals. After a full server reboot, it’ll be completely different.

To aid understanding: whether the server reboots or not, shmkey/memkey can remain constant (since it’s user/PG input). But across a server reboot, even with the same shmkey, the returned shmid is very unlikely to be the same value.

How PG Obtains the shmkey
#

PGSharedMemoryCreate():

	/*
	 * We use the data directory's ID info (inode and device numbers) to
	 * positively identify shmem segments associated with this data dir, and
	 * also as seeds for searching for a free shmem key.
	 */
	if (stat(DataDir, &statbuf) < 0)
		ereport(FATAL,
				(errcode_for_file_access(),
				 errmsg("could not stat data directory \"%s\": %m",
						DataDir)));
...
	/*
	 * Loop till we find a free IPC key. Trust CreateDataDirLockFile() to
	 * ensure no more than one postmaster per data directory can enter this
	 * loop simultaneously. (CreateDataDirLockFile() does not entirely ensure
	 * that, but prefer fixing it over coping here.)
	 */
	NextShmemSegID = statbuf.st_ino;

	for (;;)
	{
		IpcMemoryId shmid;
		PGShmemHeader *oldhdr;
		IpcMemoryState state;

		/* Try to create new segment */
		memAddress = InternalIpcMemoryCreate(NextShmemSegID, sysvsize);
		if (memAddress)
			break;				/* successful create and attach */

		/* Check shared memory and possibly remove and recreate */

		/*
		 * shmget() failure is typically EACCES, hence SHMSTATE_FOREIGN.
		 * ENOENT, a narrow possibility, implies SHMSTATE_ENOENT, but one can
		 * safely treat SHMSTATE_ENOENT like SHMSTATE_FOREIGN.
		 */
		shmid = shmget(NextShmemSegID, sizeof(PGShmemHeader), 0);

PG calls stat() on the data directory, which returns the directory’s inode. PG directly uses datadir.inode as the shmkey.

In PG, the shmem key is tightly coupled to the data directory’s inode. Under normal circumstances, shmem key = datadir inode.

Verification example:

> ls -id $PGDATA
4096 /lzlcloud/pg8574/data
> cat postmaster.pid |head -7|tail -1
 4096 143917090

We can see datadir.inode = shmkey = 4096.

PG shmkey in Cloud Environments
#

Above I said generally shmkey = datadir.inode, but in cloud environments this is typically not the case.

Our cloud environment:

> ls -id /lzlcloud/pg8298/data
4096 /lzlcloud/pg8298/data
> ls -id /lzlcloud/pg8388/data
4096 /lzlcloud/pg8388/data
> ls -id /lzlcloud/pg8095/data
4096 /lzlcloud/pg8095/data

> cat /lzlcloud/pg8298/data/postmaster.pid|head -7|tail -1
 4096 971833391
> cat /lzlcloud/pg8388/data/postmaster.pid|head -7|tail -1
 4097 62128161
> cat /lzlcloud/pg8095/data/postmaster.pid|head -7|tail -1
 4098 143163441

The data disk directories all have inode 4096, but the shmkeys are 4096, 4097, 4098.

Why?

The inode issue relates to the filesystem:

Each filesystem has independent inodes
The filesystem reserves some inodes — the first few are unusable. Depending on mount options, our data disk’s real inodes start at 4096

So datadir.inode = 4096 is the default behavior of our cloud environment’s disk mounts. Other environments may differ — I haven’t analyzed those deeply. But with the same filesystem and mount approach for PG data directories, inode collisions are still possible.

The shmkey issue relates to PG’s source code, PGSharedMemoryCreate():

	for (;;)
	{
 ...
 NextShmemSegID = statbuf.st_ino;
 ...
		shmid = shmget(NextShmemSegID, sizeof(PGShmemHeader), 0);
		...
		switch (state)
		{
			case SHMSTATE_FOREIGN:
				NextShmemSegID++;
				break;

The initial shmkey = datadir.inode, but since the requested shmem might be FOREIGN (used by another process), PG increments shmkey by 1 and tries again.

For example, the instance with shmkey=4097 in postmaster.pid: at startup it tried shmkey=4096, but found that shmid’s memory segment was already in use by another instance (the one with shmkey=4096). So it used shmkey+1 to request a different shmid segment.

Similarly, the instance with shmkey=4098 had to increment twice to find a free shmkey-shmid pair.

shmid Relationships
#

The SysV shmid can be found in the startup error log, line 7 of postmaster.pid, and virtual memory smaps. It can be inspected via the ipcs command and cleaned up with ipcrm.

Example — note shmid=143917078 throughout:

Startup error log:

pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 16:02:19 CST::@:[262388]: FATAL: pre-existing shared memory block (key 4096, ID 143917078) is still in use

postmaster.pid line 7:

> cat postmaster.pid |head -7|tail -1
 4096 143917078

Virtual memory smaps:

cat /proc/`head -1 $PGDATA/postmaster.pid`/smaps | grep -E "\-s"
2ad2b5189000-2ad2b518a000 rw-s 00000000 00:04 143917078 /SYSV00001000 (deleted)

Inspecting and cleaning via SysV shmid:

ipcs -m -i 143917078 # cleanup: ipcrm -m shmid

Shared memory Segment shmid=143917078
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=242712 cpid=242712 nattch=10
att_time=Thu Mar 5 16:14:51 2026
det_time=Thu Mar 5 16:14:49 2026
change_time=Thu Mar 5 16:14:34 2026

Testing
#

Reproducing the Production Issue
#

Hold a backend process alive indefinitely, then kill -9 the postmaster:

> cat postmaster.pid
 4096 143917076

> ipcs -m -i 143917076 # shmem id
Shared memory Segment shmid=143917076
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=241567 cpid=64757 nattch=23

> kill -stop 107648 # any backend

> kill -9 64757 # postmaster or another process

> ipcs -m -i 143917076
Shared memory Segment shmid=143917076
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=252283 cpid=64757 nattch=1 # nattch != 0

> pg_ctl start -D $PGDATA
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 16:02:19 CST::@:[262388]: FATAL: pre-existing shared memory block (key 4096, ID 143917076) is still in use
2026-03-05 16:02:19 CST::@:[262388]: HINT: Terminate any old server processes associated with data directory "/data".
 stopped waiting
pg_ctl: could not start server

nattch=1 — the instance cannot start.

Normal Crash Recovery (Successful Startup)
#

Essentially, kill the instance and then start it:

> cat postmaster.pid
 4096 143917077

> ipcs -m -i 143917077 # shmem id
Shared memory Segment shmid=143917077
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=154800 cpid=134329 nattch=18

> kill -9 134329 # postmaster or another process

> cat postmaster.pid
 4096 143917077

> ipcs -m -i 143917077 # shmem id unchanged, segment still exists
Shared memory Segment shmid=143917077
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=169360 cpid=134329 nattch=0 # nattch=0

> ipcs -m -i 143917077 # shmem id unchanged, segment still exists

> pg_ctl start -D $PGDATA # startup succeeds
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 16:14:34 CST::@:[242712]: LOG: redirecting log output to logging collector process
2026-03-05 16:14:34 CST::@:[242712]: HINT: Future log output will appear in directory "/data/pg_log".
 done
server started

> ipcs -m -i 143917077 # residual shmem cleaned up during startup
ipcs: id 143917077 not found
> ipcs -m -i 143917078 # shmid incremented by 1 at startup
Shared memory Segment shmid=143917078
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=273571 cpid=242712 nattch=26

> cat postmaster.pid # shmkey unchanged, shmid +1
 4096 143917078

A normal kill -9 followed by startup works fine — the residual shmem is cleaned up during startup. shmkey stays the same because inode=4096 and shmkey=4096 wasn’t occupied. shmid+1 is Linux kernel behavior, at least indicating a different shared memory segment was used.

Holding a File Descriptor But Not shmem
#

Since startup is tied to the data directory inode, and inode is tied to shmem id, startup essentially checks whether the shmem is held by another process, not whether a file descriptor is still open. So let’s test with the logger process, which holds file descriptors but not shared memory:

$ cat /proc/77300/smaps | grep -E "\-s" # logger process — verify it has no shared memory
$ kill -stop 77300 # stop logger
$ kill -9 77076 # kill -9 pm
$ cat postmaster.pid # file still exists
77076
/lzlcloud/pg8531/data
1772700343
8531
/tmp
0.0.0.0
 4096 143917080
ready
$ ipcs -m -i 143917080 # shared memory still exists

Shared memory Segment shmid=143917080
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=77319 cpid=77076 nattch=0
att_time=Thu Mar 5 17:27:11 2026
det_time=Thu Mar 5 17:27:15 2026
change_time=Thu Mar 5 16:45:43 2026

$ ps -ef|grep 77300 # process still alive
postgres 77300 1 0 16:45 ? 00:00:00 postgresql: lzldb: logger
postgres 135246 46622 0 17:27 pts/1 00:00:00 grep --color=auto 77300
$ pg_ctl start -D $PGDATA # startup succeeds
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 17:27:55 CST::@:[140497]: LOG: redirecting log output to logging collector process
2026-03-05 17:27:55 CST::@:[140497]: HINT: Future log output will appear in directory "/data/pg_log".
 done
server started

The logger holds files in the data directory but is not associated with shared memory — it does not block startup.

Deleting postmaster.pid Then Failing to Start
#

Same procedure: hold a backend process, kill -9 the PM, delete postmaster.pid, attempt startup.

I’ll skip the full output — result: startup fails with:

waiting for server to start....2026-03-06 15:29:48 CST::@:[22475]: FATAL: pre-existing shared memory block (key 4098, ID 171868173) is still in use
2026-03-06 15:29:48 CST::@:[22475]: HINT: Terminate any old server processes associated with data directory "/data".
2026-03-06 15:29:48 CST::@:[22475]: LOG: database system is shut down

This shows: even with a zombie process holding shmem, deleting the postmaster.pid (which contains the shmid) doesn’t stop PG from finding the corresponding shmid.

Stop a Different Instance, Start the Current One
#

PG analyzes shmid from two sources to determine if it belongs to the current instance:

The shmid corresponding to datadir.inode as shmkey, or after shmkey++
The shmid stored in postmaster.pid

Even if postmaster.pid is deleted, PG can still tell whether shmem is held by another process. But we can exploit datadir.inode and shmkey++ behavior to get it started.

Since in our cloud environment all data directory inodes are 4096, and shmkeys differ due to the shmkey++ source logic, we can: start or stop a PG instance whose datadir.inode = 4096 to shift the current instance’s shmkey++ by one, obtaining a different shmid.

$ kill -stop 165245
$ kill -9 164411 # stop current instance, keep one of its backend processes alive

$ pg_ctl stop -D /pg8531/data # stop a different instance
waiting for server to shut down.... done
server stopped
$ pg_ctl start -D /pg8574/data # try starting the current instance — fails because postmaster.pid still exists
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 18:22:35 CST::@:[196209]: FATAL: pre-existing shared memory block (key 4097, ID 143917087) is still in use
2026-03-05 18:22:35 CST::@:[196209]: HINT: Terminate any old server processes associated with data directory "/pg8574/data".
 stopped waiting
pg_ctl: could not start server
Examine the log output.

$ mv /lzlcloud/pg8574/data/postmaster.pid{,.bak} # delete current instance's postmaster.pid
$ pg_ctl start -D /lzlcloud/pg8574/data # try again — succeeds
2026-03-05 18:23:09 CST::@:[207725]: LOG: redirecting log output to logging collector process
2026-03-05 18:23:09 CST::@:[207725]: HINT: Future log output will appear in directory "/lzlcloud/pg8574/data/pg_log".
 done
server started

$ ipcs -m -i 143917087 # the shmid's SysV segment is still held by our zombie process

Shared memory Segment shmid=143917087
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=196209 cpid=164411 nattch=1
att_time=Thu Mar 5 18:22:35 2026
det_time=Thu Mar 5 18:22:35 2026
change_time=Thu Mar 5 18:21:04 2026

Startup succeeds — the current instance requested a different shared memory segment. The old segment wasn’t cleaned up. This is the “hack” of stopping another instance to start the current one in a cloud environment.

A small prerequisite: the other instance must have not only inode = current instance inode, but also shmkey < current instance shmkey.

Error Analysis: `lock file "postmaster.pid" already exists`
#

This problem is much simpler than the shared memory one.

During startup, PG checks the lock file and its contained PID, in CreateLockFile():

		if (other_pid != my_pid && other_pid != my_p_pid &&
			other_pid != my_gp_pid)
		{
			if (kill(other_pid, 0) == 0 ||
				(errno != ESRCH && errno != EPERM))
			{
				/* lockfile belongs to a live process */
				ereport(FATAL,
						(errcode(ERRCODE_LOCK_FILE_EXISTS),
						 errmsg("lock file \"%s\" already exists",
								filename),
						 isDDLock ?
						 (encoded_pid < 0 ?
						 errhint("Is another postgres (PID %d) running in data directory \"%s\"?",
								 (int) other_pid, refName) :
						 errhint("Is another postmaster (PID %d) running in data directory \"%s\"?",
								 (int) other_pid, refName)) :
						 (encoded_pid < 0 ?
						 errhint("Is another postgres (PID %d) using socket file \"%s\"?",
								 (int) other_pid, refName) :
						 errhint("Is another postmaster (PID %d) using socket file \"%s\"?",
								 (int) other_pid, refName))));
			}
		}

Testing is even simpler — just start it a second time while already running:

$ pg_ctl start -D /pg8531/data
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-06 15:59:05 CST::@:[89145]: FATAL: lock file "postmaster.pid" already exists
2026-03-06 15:59:05 CST::@:[89145]: HINT: Is another postmaster (PID 255500) running in data directory "/pg8531/data"?
 stopped waiting
pg_ctl: could not start server
Examine the log output.

So the later errors in the fault’s start.log were because the instance was already running and someone tried starting it multiple more times.

Summary
#

When starting, PG first allocates a SysV shmem segment (not the mmap-based shared_buffers) to lock the data directory. The lock is obtained by using the data directory’s inode as the shmkey passed to shmget(), which returns a unique shmid. Since the requested shmem may already be in use by another process, PG increments shmkey++ in an infinite loop until it finds an unclaimed segment. postmaster.pid line 7 stores both the shmkey and shmid. In cloud environments, you’ll often see adjacent PG instances with incrementing shmkeys — this happens because the data disks are mounted identically and share the same starting inode, causing shmkey++ to kick in.

If a PG instance is killed unexpectedly, the shmem is not automatically cleaned up. Under normal conditions, no zombie process holds the shared memory, so startup cleans it up and proceeds normally. Under abnormal conditions, a zombie process still holds the shared memory — startup fails and manual intervention is required.

Recommended handling:

ipcrm -m (most recommended)
Use lsof to find the zombie process and kill it
Reboot the host

Not recommended but possible workarounds:

mv postmaster.pid + stop a different PG instance (where the other instance’s shmkey < current instance’s shmkey)
mv postmaster.pid + remount the data disk to change its inode

Finally, answering the opening questions:

Why isn’t this scenario more common in practice?

Abnormal instance crash + zombie processes still alive. Many crash scenarios leave no zombie processes, so startup just works.

The start.log shows two different error types — what do they correspond to?

The “shared memory in use” error means abnormal crash + zombie processes still exist. The “postmaster.pid already exists” error means the instance was started multiple times.

Can shared memory still exist if the postmaster is gone?

Yes, shared memory can persist when the postmaster is gone — PG processes don’t always cleanly exit or get cleaned up by the OS. However, if all processes are gone, the shared memory should not exist.

How do you locate and clean up this shared memory segment?

The shmid can be found in the startup error log (start.log). Clean it with ipcrm -m $shmid.

PG has multiple shared memory segments — which one is this?

The SysV shmem used to protect the data directory. It always exists. See the “Three Types of Shared Memory” section. It’s distinct from the mmap-based shared_buffers.

Can you find the corresponding shmem via inode or file?

Linux does not provide a userspace interface to find SysV shmem by inode or file (this statement is 100% AI-generated, cross-validated across multiple models). PG uses the data directory’s inode as a seed shmkey to request shared memory — it does not directly find shmem by inode. PG has its own mechanism for locating SysV shmem, but it’s not an absolute mapping; shmkey++ is a compromise startup logic for this reason.

PostgreSQL Operations Experience 2025

Sun, 11 Jan 2026 00:00:00 +0000

This is a technical operations summary, focused on being accessible and practical. It also serves as a periodic reflection on PostgreSQL database operations. Hope it helps fellow PGers.

Previous ops experience: PostgreSQL Operations Experience 2024. Note: this article does not repeat content from that one.

CPU
#

SQL performance problems are the most common root cause in PostgreSQL incident handling. This includes poor SQL performance, suboptimal indexing, sudden high concurrency, and execution plan regressions. For a database like PostgreSQL that lacks a robust plan-binding mechanism, having a DBA team to help design data models, access patterns, indexes, and tune execution plans is crucial — it can significantly reduce sudden CPU saturation incidents.

Execution Plans
#

Execution plan instability is an age-old problem with cost-based optimizers, and PostgreSQL is no exception.

Inaccurate DISTINCT Estimates
#

Case Study: From Inaccurate DISTINCT to DISTINCT Calculation Principles

The default maximum sample size is 30,000 rows. For tables exceeding this size, the estimated distinct count is likely to be low. Note: this assumes the data doesn’t have too many unique values.

Testing on a table with different sample sizes:

Table: reltuples=800 million, relpages=20 million, size=175GB, actual distinct on the target column: 100 million.

target statistics	pages sampling rate	tuples sampling rate	n_distinct	execution time
50	0.00075	0.00001875	60K	2s
100	0.0015	0.0000375	110K	5s
1000	0.015	0.000375	1.03M	58s
3000	0.045	0.001125	2.68M	3m01s
10000	0.15	0.00375	6.75M	7m21s

(target statistics max value: 10000)

Rough summary: n_distinct and analyze execution time grow proportionally with sample size.

n_distinct increases with sample size, while pages and tuples estimates remain consistently accurate.

Generic Plan Interference
#

PostgreSQL execution plans must account for generic plans. A generic plan is parameter-independent — it uses default values to compute cost, then compares against the first five custom plan costs; whichever is cheaper wins.

Case Study: Adding an Index Causes Performance Degradation and Generic Plans

I. Classification of generic plan estimation problems

Because of the 5-execution comparison mechanism, generic plan problems fall into two categories:

The first 5 SQL executions are not representative. Heavily dependent on data skew and whether the first 5 parameter values are representative.
The generic plan itself is flawed. Due to data skew or inability to accurately compute selectivity even with balanced data, the generic plan is inherently inefficient.

II. Solution reference

Generic plan problems often surface on partitioned tables. When the partition key is continuous, scanning all partitions should yield a selectivity of 1, but the generic plan estimates 0.05 — likely resulting in a “full index scan” scenario.

Consider these when optimizing:

Don’t create too many indexes that confuse the optimizer
Eliminate generic plan interference. Execute the prepared statement 6 times for real
Compare plans with session-level set plan_cache_mode='force_generic_plan'; or set plan_cache_mode='force_custom_plan';; or on PG 16+, use explain (GENERIC_PLAN) to compare

Syntax reference:

--prepare/execute
PREPARE sql1(text) AS
SELECT COUNT(*) FROM LZL where a=$1;

EXECUTE sql1('zzz'); --run 6 times first
EXPLAIN EXECUTE sql1('zzz');

select * from pg_prepared_statements --view prepared statement info, current session only

--Compare execution plans, set session parameter then EXPLAIN EXECUTE
set plan_cache_mode='force_generic_plan'
set plan_cache_mode='force_custom_plan'
--Directly view generic plan, 16+
explain (GENERIC_PLAN) xx 

LWLock:Lockmanager Caused by Row Locks
#

LWLock Lockmanager issues typically occur on partitioned tables under high concurrency with queries lacking partition keys. This year, a new scenario was discovered: Row Locks Causing LWLock:Lockmanager

This isn’t a major issue — blocking on concurrent updates to the same row is well known. I just hadn’t expected that updating the same row could also produce LWLock:Lockmanager. Not a particularly valuable case study, but when you see LWLock:Lockmanager as a wait event, consider row locks.

Idle Connections
#

PostgreSQL performance generally improves with each major release. PG 14 made significant optimizations to snapshot acquisition and backend transaction tracking, yielding noticeable improvements for high idle connection counts:

(https://techcommunity.microsoft.com/blog/adforpostgresql/improving-postgres-connection-scalability-snapshots/1806462)

However, this doesn’t mean you can ignore idle connections after PG 14. They still consume backend transaction maintenance overhead, cause context switches, fragment memory, etc. — the more idle connections, the worse the performance.

Typically, application connections have keepalive and pooling. Maintaining some idle connections avoids creating new connections for every request, which would be far more expensive. Small databases generally don’t need to worry much about connection counts (as long as they’re not absurd) — CPUs are cheap, the system isn’t critical, and scaling is easy. But large databases are different. CPU count is the hard limit; you can’t just add more. Large databases already have many idle connections; adding more doesn’t necessarily increase throughput — when CPU is already tight, it can backfire.

PG 15 benchmark experience: with 5K idle as baseline, increasing to 10K idle adds ~2-5 vCPU overhead for idle maintenance; 20K idle adds ~5-10 vCPU. Approximate.

Idle in Transaction
#

Last year I thoroughly criticized long transactions, because they impact PostgreSQL more severely than other databases (Oracle, MySQL, etc.). But this is manageable — with proper alerting and operations, long transactions are solvable.

When monitoring session states, you need to check them. active means running SQL, idle in transaction means in a transaction but not currently executing SQL. All pg_stat_activity states, PG 15:

Current overall state of this backend. Possible values are:

active: The backend is executing a query.
idle: The backend is waiting for a new client command.
idle in transaction: The backend is in a transaction, but is not currently executing a query.
idle in transaction (aborted): This state is similar to idle in transaction, except one of the statements in the transaction caused an error.
fastpath function call: The backend is executing a fast-path function.
disabled: This state is reported if track_activities is disabled in this backend.

Common states are: active, idle, idle in transaction, idle in transaction (aborted). A common misconception about idle in transaction: it only means no SQL is running right now and the transaction hasn’t committed — it does NOT mean the transaction has been idle for a long time. Don’t use xact_start + idle in transaction to judge how long a transaction has been idle. Use state_change + idle in transaction instead.

Memory
#

Memory issues are extremely tricky, and I handled many this year, finding some good solutions. But memory knowledge is broad — I’ll try to simplify as much as possible, going straight to symptoms, results, and solutions.

Memory Issues and Huge Pages
#

Classification of PostgreSQL memory problems:

Relevant wchan states for PG memory issues:

Huge pages are very effective against memory fragmentation and direct memory reclaim within cgroups.

Benchmark results for huge pages: https://docs.paic.com.cn/#/post/84479375

Theoretical benefits of huge pages:

Reduced TLB pressure
Reduced page table size in main memory
Huge pages are physically contiguous. Contiguous physical memory access is better than non-contiguous
With huge pages, pages are directly mapped without multi-level PTE entries

However, huge pages bring management challenges:

Must pre-allocate huge pages
Must calculate huge page size in advance to avoid memory waste

Memory knowledge is extensive. For more, refer to Advanced Linux Memory. Key takeaways:

Rule out OS-level issues before tackling PG instance-level issues
Huge pages have remarkable effects, but in rare cases they don’t help
Many people don’t monitor pgpgin/pgpgout/pgfree, or even pgscank/pgscand — they only look at CPU and memory usage. That’s insufficient for operating PostgreSQL.
Without good operational practices, PG memory can be very unstable

Notable Cgroup Knowledge
#

Cgroup knowledge is also extensive. Refer to earlier articles; here’s a quick summary.

Cgroup v1 has inherent flaws:

Does not account for cgroup page tables
Does not account for cgroup slab
Does not account for cgroup huge pages (huge pages are not charged, not just uncounted)
Does not account for cgroup async/sync page reclaim
Cgroup RSS and process RSS have inconsistent accounting methods
shmem accounting is messy

Unsolved Mysteries
#

Huge pages have solved many problems, but not all. The unsolved portion remains to be researched — hopefully clarified in 2026.

Pay Attention to the OS
#

Pay Attention to Everything OS
#

To operate open-source databases well, you need to understand the operating system.

(Source forgotten)

To operate PostgreSQL well, understanding OS principles is essential. PostgreSQL is built on top of the OS (especially Linux) — it uses whatever Linux provides. PostgreSQL is part of the Linux ecosystem. To truly understand how it works, understand the OS first.

Rule out OS-level issues before tackling PG instance-level issues.

(My own words)

I. CPU

Since PostgreSQL doesn’t use NUMA, whether on bare metal or cgroup/pod-managed CPU, you rarely need to dive into OS-level CPU internals. CPU issues can mostly be diagnosed from SQL or PG stack traces.

II. Memory

See the Memory section. Memory issues require OS-level investigation.

III. Processes

Inspecting PG process states from the OS is critical. You need to check D state, wchan, RSS, syscalls, at minimum.

IV. Host Status and Logs

Monitor host status — CPU, memory, IO, network, logs at the host level. Very important.

It’s hard to imagine that a vague network IO alert like “an I/O error occurred while sending to the backend” is related to underlying storage. Beyond /var/log/messages, PG itself shows nothing. (Of course, this error may have other causes — don’t misinterpret.)

V. Others

Uncategorized.

Physical Reads
#

PostgreSQL itself does not directly expose a “true physical disk read” metric. The various reads in pg_stat_* (e.g., pg_stat_database.blks_read) are reads from the OS cache.

So how do you monitor physical reads?

Reads or buffer allocation metrics are supplementary. The best approach is monitoring the OS itself.

The OS is PostgreSQL’s ecosystem. Never look at the database in isolation. Not being able to monitor physical reads at the database level is nothing to be ashamed of — as long as you have a solution.

Monitor iostat and other disk metrics. For cloud environments, OS-level observability is already mature — don’t waste cloud-native observability.

Autovacuum
#

SQL for monitoring autovacuum processes: sql autovacuum_queue_and_progress

Autovacuum Freeze on Large Databases
#

With properly configured parameters, monitoring, and alerting, autovacuum freeze requires little attention in most databases.

However, in databases with extremely high transaction throughput and very large data volumes, you still can’t ignore it. Autovacuum prevent wraparound may be running constantly. At minimum, watch these two points:

Age alerting: handle promptly and try to prevent the next alert. Don’t wait until the last moment to panic (acceleration options depend on version, e.g., INDEX_CLEANUP OFF, BUFFER_USAGE_LIMIT adjustments)
Impact on memory (especially cache). If autovacuum runs nonstop on a very large database, it impacts cache and memory

For principles and parameters, see this howtos diagram:

Large Tables That Won’t Finish Vacuuming
#

“Large tables” means hundreds of GB, typically with many indexes and dead tuples that prevent vacuum from completing.

The main bottleneck: (auto)vacuum cleans dead index tuples one by one per dead row. Large table (auto)vacuum is slow here — you’ll typically see many dead tuples on the table. Worse, (auto)vacuum may run slower than the rate of dead tuple generation — vacuum never finishes, infinite bloat.

Experience with large tables that can’t finish:

For the same table, dead tuple count is roughly proportional to execution time
From autovacuum log’s user time and elapsed time, you can observe CPU time and execution time, and roughly estimate delay sleep time
Disabling autovacuum cost-based delay can reduce execution time by ~3× (index-size dependent; based on a 200GB table with 280GB indexes)
Adjusting a table’s autovacuum cost-based delay means letting autovacuum rest less when processing that table — consuming more CPU and scan IO in a shorter time

How to accelerate?

Repack. Repack is a nuclear option — fast table rebuild for emergencies. But repack is a CLI tool; running it manually each time is cumbersome.
Tune autovacuum cost-based delay parameters. Either 1. Increase cost limit: alter table t1 SET (autovacuum_vacuum_cost_limit=1000);, or 2. Disable delay entirely: alter table t1 SET (autovacuum_vacuum_cost_delay=0);. Recommended only for tables that can’t keep up.
Drop unnecessary indexes. Scanning indexes and updating index entries takes the most time — dropping unnecessary indexes is effective.
Partitioned tables. Recommended partition size ≤10GB. Converting to partitioned tables is the best solution.
Drop updated_time column indexes to leverage HOT, reducing bloat rate.

Checkpoint and Bgwriter
#

The checkpointer not only creates checkpoints (affecting recovery time) but also flushes dirty buffers. The bgwriter only flushes dirty buffers. Starting from PG 17, some metrics moved to pg_stat_checkpointer. For PG ≤16, mainly look at pg_stat_bgwriter.

I. Checkpoint intervals

Metric checkpoints_timed: corresponds to checkpoint_timeout parameter
Metric checkpoints_req: corresponds to max_wal_size parameter

Recommend using checkpoint_timeout as the primary checkpoint interval. If checkpoints_req appears, increase max_wal_size and tune flush parameters accordingly. When FPIs are present, also check these two metrics.

II. Flush metrics

Metric buffers_checkpoint: dirty buffers flushed by checkpointer
Metric buffers_clean: dirty buffers flushed by bgwriter
Metric buffers_backend: dirty buffers flushed by backends — should be as close to zero as possible; occurrence means bgwriter isn’t aggressive enough
Metric buffers_backend_fsync: meaning unclear

The tuning goal is flush priority: bgwriter flush > checkpointer flush > backend flush

The checkpointer can flush as a side effect, but checkpointer flush speed is hard to control — it can cause IO spikes. So bgwriter flush priority should be higher than checkpointer. Backend flush is obviously worst — minimize it.

III. Bgwriter flush parameters

Bgwriter controls flush speed through a “write some, pause, write again” cycle:

Parameter bgwriter_delay: how long to pause
Parameter bgwriter_lru_maxpages: max pages to write per cycle
Parameter bgwriter_lru_multiplier: pages per cycle = (recent buffer allocation × lru_multiplier), capped at lru_maxpages
Parameter bgwriter_flush_after: fsync after writing this many buffers
Metric pg_buffers_alloc: represents shared memory buffer allocation (allocation means actual eviction occurred, somewhat indicative of pgpgin)
Metric maxwritten_clean: number of times bgwriter_lru_maxpages was reached

Default bgwriter flush logic: each cycle: flush (new buffer count × 2, max 100 dirty buffers), delay 200ms, fsync every 64 buffers flushed.

Per-cycle flush volume depends on recent buffer allocation and bgwriter_lru_multiplier. During peak times, buffer allocation is typically high, so it usually hits bgwriter_lru_maxpages. Thus: bgwriter_lru_maxpages caps peak flush volume; bgwriter_lru_multiplier prevents excessive flushing during off-peak times.

IV. Flush parameter reference

Default max bgwriter flush = 100 × 5 × 8KB = 3.9MB/s. The defaults are definitely too low. If tuning upward, adjust based on shared_buffers size and workload.

After all that theory, here’s a practical reference:

#Read/write ratio 2:8, high load
shared_buffers=40GB
checkpoint_timeout=20min;
max_wal_size=80GB
bgwriter_delay=20ms
bgwriter_lru_maxpages=1000
bgwriter_lru_multiplier=4

Adjust further as needed.

As for effects: from practical experience, don’t expect standalone bgwriter tuning to yield great results. Overly aggressive bgwriter tuning can even backfire.

So: If your database hasn’t been clearly diagnosed with checkpoint flush spikes or other flush issues, don’t touch this. Only recommended for core large databases with high concurrency, as a supplementary tuning strategy alongside other changes (migrations, shared_buffer adjustments, etc.).

V. Flush parameter summary

Bgwriter flushing can be summarized as “three hard’s”:

“Hard to understand, hard to tune, hard to see results.”

DB4AI
#

AI Task Scheduling Writes to Database
#

AI applications are widely deployed at the development level. One scenario: AI task invocations write to the database. Task invocations can spike instantly, and the database writes may lack concurrency control, causing CPU or other resource spikes.

This is a new database incident pattern in the AI era. Be careful.

Vector HNSW
#

Reference: https://postgresql.us/events/pgconfnyc2024/sessions/session/1862/slides/172/pgvector_best_practices_pgconfnyc2024.pdf

HNSW Index Build Acceleration
#

HNSW index builds can be extremely slow — millions of rows can take hours.

Factors affecting HNSW build speed include instance memory (and CPU) as well as index build parameters:

maintenance_work_mem=3g
max_parallel_maintenance_workers=2
m=12
ef_construction=100

Building HNSW indexes can be painful. Ways to accelerate:

Building the index before data load is an option. Though the total initial time is slower, developers may accept “a bit slower” but cannot accept “index building for 1 hour.”
Optimizing post-load index builds:
- SET maintenance_work_mem = '8GB'
- SET max_parallel_maintenance_workers = 8

Post-load index builds need attention to memory — strongly related to instance memory and free memory.

Note: maintenance_work_mem can protect OS memory. If maintenance_work_mem exceeds available OS memory and the table is large, the connection is terminated immediately (fast failure):
```
ERROR: 53200: could not resize shared memory segment "/PostgreSQL.1390017142" to 6439348672 bytes: Cannot allocate memory
LOCATION: dsm_impl_posix, dsm_impl.c:314
```

Note: if memory used during build exceeds maintenance_work_mem, an info notice appears (after some time):

NOTICE: 00000: hnsw graph no longer fits into maintenance_work_mem after 886990 tuples
DETAIL: Building will take significantly more time.
HINT: Increase maintenance_work_mem to speed up builds.
LOCATION: InsertTuple, hnswbuild.c:525

HNSW Index Query Performance
#

Query recall and performance need to be balanced via the ef_search parameter.

Besides ef_search, one more factor significantly impacts query speed: whether the HNSW index is cached in memory.

Index NOT in memory:

explain (analyze,buffers) SELECT image_id, applyNo, feature_vector <-> (select vectorsit
 FROM image_features_test2
 ORDER BY distance
 LIMIT 10;
 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=11852.80..11865.74 rows=10 width=35) (actual time=82193.073..82193.185 rows=10 loops=1)
 Buffers: shared hit=1796 read=9309
 I/O Timings: shared/local read=82108.559
 InitPlan 1 (returns $0)
 -> Limit (cost=0.00..0.02 rows=1 width=32) (actual time=0.008..0.009 rows=1 loops=1)
 Buffers: shared hit=1
 -> Seq Scan on test_0 (cost=0.00..23.60 rows=1360 width=32) (actual time=0.007..0.008 rows=1 loops=1)
 Buffers: shared hit=1
 -> Index Scan using idx_feature_hnsw on image_features_test2 (cost=11852.78..1292546.60 rows=989705 width=35) (actual time=82193.071..82193.179 rows=10 loops=1)
 Order By: (feature_vector <-> $0)
 Buffers: shared hit=1796 read=9309
 I/O Timings: shared/local read=82108.559
 Planning:
 Buffers: shared hit=1
 Planning Time: 0.130 ms
 Execution Time: 82193.279 ms

Index IN memory:

 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=11852.80..11865.74 rows=10 width=35) (actual time=20.240..20.350 rows=10 loops=1)
 Buffers: shared hit=11105
 InitPlan 1 (returns $0)
 -> Limit (cost=0.00..0.02 rows=1 width=32) (actual time=0.007..0.008 rows=1 loops=1)
 Buffers: shared hit=1
 -> Seq Scan on test_0 (cost=0.00..23.60 rows=1360 width=32) (actual time=0.007..0.007 rows=1 loops=1)
 Buffers: shared hit=1
 -> Index Scan using idx_feature_hnsw on image_features_test2 (cost=11852.78..1292546.60 rows=989705 width=35) (actual time=20.239..20.344 rows=10 loops=1)
 Order By: (feature_vector <-> $0)
 Buffers: shared hit=11105
 Planning:
 Buffers: shared hit=1
 Planning Time: 0.093 ms
 Execution Time: 20.392 ms

Same index, same execution plan — the performance difference between index-in-memory and index-not-in-memory is 82193.279 / 20.392 = 4000×!

This gap cannot be ignored. When monitoring HNSW index performance, always check whether the index is in memory. Reference SQL:

--Check if HNSW index is cached in shared buffers via pg_buffercache
SELECT c.relname, pg_size_pretty(count(*) * 8192) as buffered, round(100.0 * count(*) / (SELECT setting FROM pg_settings WHERE name='shared_buffers')::integer, 1) AS buffer_percent, round(100.0 * count(*) * 8192 / pg_table_size(c.oid), 1) AS percent_of_relation FROM pg_class c INNER JOIN pg_buffercache b ON b.relfilenode = c.relfilenode INNER JOIN pg_database d ON (b.reldatabase = d.oid AND d.datname = current_database()) GROUP BY c.oid, c.relname ORDER BY 3 DESC LIMIT 10; 

 relname | buffered | buffer_percent | percent_of_relation 
---------------------------------+------------+----------------+---------------------
 idx_feature_hnsw_1 | 2117 MB | 91.9 | 44.5
 idx_feature_hnsw | 78 MB | 3.4 | 2.0
 pg_inherits_parent_index | 8192 bytes | 0.0 | 100.0

Application Releases
#

DDL Tips
#

Online DDL tools like pg-osc and pg_migrate don’t support partitioned tables, and they have other issues — real-world use is difficult. So DDL tips are still useful: lowering lock levels, proactively identifying blocking, etc., to reduce DDL blocking and rewrite risks.

Key points for understanding this diagram:

Before changes:

Ensure no long transactions on the table — long transactions hold locks on tables persistently. Long transactions are a well-known hazard in PG; handle them first.
Ensure no autovacuum (to prevent wraparound) on the table — autovacuum generally doesn’t block SQL, except when running to prevent wraparound:

Autovacuum workers generally don’t block other commands. If a process attempts to acquire a lock that conflicts with the SHARE UPDATE EXCLUSIVE lock held by autovacuum, lock acquisition will interrupt the autovacuum. However, if the autovacuum is running to prevent transaction ID wraparound (i.e., the autovacuum query name in the pg_stat_activity view ends with (to prevent wraparound)), the autovacuum is not automatically interrupted.
lock_timeout=2000 — if a lock cannot be acquired within 2 seconds, bail out to avoid mass blocking.

Special cases for “small-to-large” type changes:

Small-to-large type changes generally don’t rewrite the table, but there are exceptions. Pay special attention to int → bigint (common for PK columns) and char(n) → char(m).
Partitioned table indexes. Small-to-large type changes on partitioned tables don’t rewrite the table, but they do rebuild indexes — and rebuilding indexes on partitioned tables is typically very slow, potentially causing prolonged level-8 lock blocking. This behavior is unique to partitioned tables.

Changing column types:

Almost always rewrites the table, except for equivalent types or small-to-large cases.

DDL lock-level reduction tips:

Use CIC (CREATE INDEX CONCURRENTLY) for indexes. If partitions don’t support it, do CIC on child tables (remember to attach the index).
CIC has multiple phases. Phases 2 and 3 acquire a SHARE lock, blocking DML. (Official docs only mention SHARE UPDATE EXCLUSIVE — CIC isn’t a simple explicit lock.)
Add primary keys with USING INDEX. For partitions, leverage “add PK on child table + add PK on parent can merge existing child PKs.”
Use VALIDATE CONSTRAINT for constraints.
PG <17 doesn’t support NOT NULL VALIDATE. Use CHECK(col1 IS NOT NULL) instead. This CHECK-to-NOT-NULL conversion won’t produce extra scans.
Adding a column with a volatile DEFAULT rewrites the table. Use the non-volatile-no-rewrite property: add the column first (no rewrite), then UPDATE legacy data as needed.
When attaching partitions, use CHECK constraints to reduce downtime, and use VALIDATE CONSTRAINT for the CHECK.
CREATE TABLE LIKE + ATTACH has much lower lock levels than PARTITION OF (though I still prefer PARTITION OF).

After changes:

Remember to collect statistics (needed in many scenarios).

Parallel Index Creation
#

In production, you may need to create indexes on very large tables that take a long time. Parallel index creation can shorten build time.

Parallel index creation on regular tables:

Parallel parameter: max_parallel_maintenance_workers

Prerequisites:

Enough workers: check max_parallel_workers, max_worker_processes
Increase maintenance_work_mem to GB scale

Notes:

Effective for B-tree and BRIN
maintenance_work_mem limits the entire utility command. Unlike parallel query, where resource limits are per worker process.

From test results, parallel index creation shows diminishing returns beyond 8 workers (this conclusion may not hold in all environments).

Parallel index creation on partitioned tables:

Recommend manual parallel creation across child partitions — run index creation on multiple partitions simultaneously rather than using native parallelism. This reduces multi-process coordination overhead.

Cached Plan Must Not Change Resource
#

After adding a new column the previous night, application connections started throwing errors the next morning: “cached plan must not change result type in PostgreSQL”

Reproduction:

create table a(b varchar(10));
PREPARE p1 (varchar) AS SELECT * FROM a WHERE b=$1;
ALTER TABLE a ALTER COLUMN b TYPE varchar(20);
EXECUTE p1 ('abcd');

ERROR: 0A000: cached plan must not change result type
LOCATION: RevalidateCachedQuery, plancache.c:718

Test environment solutions: DEALLOCATE ALL — actively discard prepared statements Or, DISCARD ALL — actively discard all session state

DEALLOCATE ALL; --DISCARD ALL
PREPARE p1 (varchar) AS SELECT * FROM a WHERE b=$1;
EXECUTE p1 ('abcd');

Production environment solutions:

Since the error occurs at the application layer, JDBC can handle DEALLOCATE ALL / DISCARD ALL, but the application may not have implemented this. Immediate production solutions:

Solutions (choose one):

Since connection pools like HikariCP have connection cycling and timeout mechanisms, killing idle sessions will gradually reduce errors.
Similarly, due to connection pool cycling, you can do nothing — as the pool gradually establishes new connections, the errors fade.
If business pressure is high enough, consider killing all application connections.
Rolling restart of the application.

Not recommended:

“Restart the application after every DDL.” It works but don’t recommend this as a standard practice.
autosave=conservative. It works but enables subtransactions. A savepoint is set for each query; rollback happens only for rare cases like ‘cached statement cannot change return type’ or ‘statement XXX is not valid,’ where the JDBC driver rolls back and retries.

JDBC configuration suggestions:

Configure automatic retry after transaction rollback: https://developer.aliyun.com/article/741750
Other JDBC config references: https://jdbc.postgresql.org/documentation/server-prepare/#corner-cases. Note: some suggestions are not suitable for production.

Physical Replication
#

Query Conflicts
#

Query conflicts are a notoriously frustrating feature that directly impacts the usability of PG standby queries. Query conflicts increase standby lag, yet long-running queries on the standby are logically reasonable. This forces PG administrators to balance between lag management and long-query management — a problem that doesn’t exist in other relational databases.

Hidden characteristics of query conflicts:

Even static tables can trigger query conflicts (see: From Static Table Query Conflicts to Their Principles). The conflict is a snapshot conflict, largely unrelated to table-level locks — snapshot conflicts are cross-table.
Long queries affect short queries. Once a long query pushes standby lag to max_standby_streaming_delay, even short queries get canceled.
Continuous short queries also cause query conflicts. For example, one short query hasn’t finished when the next starts — the two queries may be logically similar, and the startup process hasn’t had time to apply WAL. Both short queries hold the XID that needs to be applied. Check whether pg_stat_activity.backend_xmin is less than the XID the startup process is applying.

Recommended standby query practices:

Using RTO SLO to tune max_standby_streaming_delay is a good approach. When arguments lead nowhere, SLO-based IT management saves the day.
Separate short/fast business queries from long queries (data extraction, reporting) onto different standbys to reduce mutual interference.
Standby queries still need SQL optimization.
Standby WAL apply lag must be monitored.

Logical Replication
#

Logical replication has countless pitfalls. 2024 had many nasty cases; 2025 had some too, but less severe, mostly on older PG versions. Overall, logical replication on newer PG versions is trending toward stability.

Slow DDL/DCL Parsing on Older PG Versions
#

Case Study: GRANT and Walsender Stuck

On PG 13 and earlier, certain DDL/DCL statements parse slowly and may affect walsender lag. These include:

Batch GRANT (including grant all tables) + pathman extension installed (whether used or not)
Batch DDL/TRUNCATE/DCL/DROP PUBLICATION

Older PG + Multiple Replication Links + Flink
#

Flink requires one link per table. Since PostgreSQL walsenders re-decode independently, dozens of Flink links on one PG database are common — and hard to refactor.

On PG 11 and earlier, the walsender main loop calls PostmasterIsAlive(), causing poor loop performance. Starting from PG 12, WalSndLoop no longer polls PostmasterIsAlive() in the main loop; instead, status checks are placed inside WalSndWait, using event-based passive notification. This greatly reduces CPU contention.

If you have multiple Flink links on an older PG version, upgrading can alleviate certain walsender resource contention issues, including:

May resolve the problem where walsender startup resource contention prevents the database from coming up for a long time
May resolve upstream heavy data changes (including DDL rewrites) causing runtime walsender log decoding CPU saturation

Older PG Cannot Auto-Sync New Partitions
#

On older PG versions with declarative partitioning, note that you can only publish child tables individually. PG ≥13 supports publishing by parent table. Below that, you must configure sync per partition child table name:

Allow partitioned tables to be logically replicated via publications (Amit Langote) § §

Previously, partitions had to be replicated individually. Now a partitioned table can be published explicitly, causing all its partitions to be published automatically. Addition/removal of a partition causes it to be likewise added to or removed from the publication. The CREATE PUBLICATION option publish_via_partition_root controls whether changes to partitions are published as their own changes or their parent’s.

In other words, if this partitioned table is an upstream for sync, every time a new partition is added, you must adapt the sync tool to publish it — otherwise, new partition data won’t sync.

Migration and Upgrades
#

Xinchuang Migration and glibc Upgrades
#

Whether it’s Xinchuang (domestic tech migration) or Linux OS version upgrades, glibc upgrades may be involved — and glibc upgrades can be extremely painful. PG sorting was entirely OS-dependent before PG 17.

PostgreSQL cannot detect compatibility issues from glibc upgrades. Every minor version of GNU C library makes locale changes. The most problematic version in practice is glibc 2.28, because 2.28 upgraded to a major Unicode 9.0.0 release (has been updated to a new upstream version from ISO which is in sync with Unicode 9.0.0).

Collations come in many types, and many environments use linguistic sorting (e.g., en_US.utf8), which is the most version-sensitive. Collation changes most commonly cause database crashes during index scans, but also uncommon issues like duplicate primary keys, data landing in wrong partitions, inconsistent merge join results, etc.

Fortunately, PG 17 provides a very safe locale provider: builtin, no longer dependent on OS-provided glibc, ICU, etc. Example:

initdb --locale-provider=builtin --bultin-locale=C.UTF-8 dbname1

However,

builtin is great but arrived too late. Converting existing production instances to builtin collation is no small task. Moreover, Xinchuang migrations or OS upgrades may not mandate database upgrades.

During Xinchuang migration, the target host’s glibc version is typically higher than the old Intel server’s — likely crossing version 2.28. Combined with tight deadlines, KPI pressure, staffing shortages, and large databases, physical migration is unavoidable. So physical Xinchuang migration must account for glibc version and collation-induced anomalies.

What can you do after physical migration?

I. Official required steps

Check indexes, rebuild those clearly problematic
REFRESH DATABASE COLLATION VERSION
Check dependent objects
REFRESH COLLATION VERSION

II. Unofficial “dark arts” approaches

I don’t have a complete solution, just ideas:

Handle partitioned table data landing in wrong partitions
- Partition key is int/bigint/float: unrelated to collation, don’t worry
- Partition key is timestamp: don’t worry; if varchar or other character types: evaluate
- Partition key is character type: refer to “a” vs “-” sort order (pgconf Collation Challenges Sorting It Out). But note:
  - If querying data, don’t query from the parent table — may crash or return nothing
  - No simple detection method
Handle primary key / unique key conflicts
Handle FDW sort range anomalies
Unknown issues

Reference: collation

Smooth Major Version Upgrades
#

https://gitlab.com/postgres-ai/postgresql-consulting/postgres-howtos/-/blob/main/0077_zero_downtime_major_upgrade.md?ref_type=heads

https://www.postgresql.eu/events/pgconfeu2023/sessions/session/4791/slides/439/2023.pgconf.eu%20Zero%20Downtime%20PostgreSQL%20Upgrades.pdf

Common major version upgrade approaches:

pg_upgrade in-place upgrade. Not recommended — may blow up in place.
pg_dump: suitable for small databases, longer maintenance windows.
Logical sync + switchover (pub/sub, pg_logical, DTS, etc.): suitable for small databases, shorter windows.
Physical forward sync + logical reverse sync: suitable for large databases, not-too-short windows.
Physical replication full sync + logical incremental sync + switchover: suitable for large databases, extremely short windows.

Syncing full data via logical replication can be extremely slow. In-place upgrade of a new standby carries uncertainty and upgrade time, plus the need for reverse logical sync. “Smooth major version upgrade” is essentially “physical replication full sync + logical incremental sync + switchover.”

Key technique: the primary creates a slot and returns an LSN. The new standby uses recovery_target_lsn to recover to that LSN, then logical sync begins.

Approximate workflow:

Pre-checks. Multi-database (consider applying one slot LSN for all), extensions, pathman, triggers, foreign keys, unlogged tables, crontab, etc.
Physical sync. Old and new version software, compare and backup conf files, pg_basebackup to build new standby on old version.
Logical sync prep 1. Primary keys and replica identity, create publication; prohibit application DDL/DCL.
Restore new standby to target LSN. Stop new standby; create slot on old primary and record LSN; start new standby with target LSN.
New standby major version upgrade. Upgrade, handle issues, switch environment variables.
Logical sync prep 2. Disable triggers, foreign keys, jobs, extensions, etc.
Logical sync. Create subscription with specified slot, copy_data=false.
Post logical sync. Check for index corruption, check logs for errors and fix, rebuild remote standbys.
Switchover. Stop application; advance sequences, enable foreign keys, triggers, jobs, etc.
Switchover. Build reverse link (old primary subscribes).
Switchover. Application cutover.

The smooth major upgrade approach is smooth for the business but complex for the DBA. It combines all the drawbacks of logical and physical migration — quite painful to execute. The steps above are already simplified. This approach consumes DBA manpower; consider it only for the most critical databases.

Partitioned Table Management
#

PostgreSQL partitioned tables are very flexible, lack built-in interval partitioning, and have varied behavior across versions — making partition management problems an annual occurrence. I believe many PG DBAs still worry about new partition issues.

My observations on partition management and usage issues:

Not using declarative partitioning. Older versions still use pathman partitioning or inheritance-based partitioning, or continue using them even after upgrading. Declarative partitioning was introduced in PG 10. Due to early version limitations, recommend only using declarative partitioning from at least PG 12 onward to reduce environmental complexity.
Developers building child table indexes/primary keys directly. Creating indexes/PKs directly on child tables via SQL rather than through parent table inheritance means the next developer writing SQL may forget. This leads not only to parent-child inconsistency but also child-child inconsistency, eventually making the partition structure unrecognizable.
No new partition management strategy. Forgetting to create new partitions or using a DEFAULT partition. Typically, developers create partitions for a few years ahead; next time, the developers may have moved on, and no one manages new partition creation. This is a ticking time bomb, or data lands in the DEFAULT partition, defeating the purpose of partitioning.
Lack of DBA management. Yes, DBA! PG partitioned table knowledge is extensive (see PostgreSQL Partitioned Tables). How to build management strategies and implement them in your environment requires proactive DBA involvement. This may be the most important factor.

My partition management goals (from Case Study: 2026-01-01 Partition Data Update Failure):

Use the parent table structure as the canonical structure — the parent table faces developers; it should have primary keys, indexes, and replica identity (unless the PG version doesn’t support it).
Keep parent and child tables consistent. Use PARTITION OF when creating new partitions (yes, I don’t recommend ATTACH).
Keep child tables consistent with each other.
Create new partitions in advance. Partition data volume should not be too large.
DEFAULT partitions are not recommended. If created, must monitor writes to them.
Queries on frequently accessed tables must include the partition key for partition pruning. Otherwise, convert to a regular table.

Observability
#

The official documentation clearly explains database, table, index, SQL, flush, and other metrics.

A few metrics deserve special attention — not only are they unclearly explained, but they’re frequently used and have a learning curve.

buffers_alloc, blks_read
#

pg_stat_bgwriter.buffers_alloc: Number of buffers allocated — shared memory eviction volume.
pg_stat_database.blks_read: OS cache reads.

(buffers_alloc may appear in different views across PG versions, but the meaning is the same.)

pg_stat_bgwriter.buffers_alloc is the shared memory buffer allocation count (called buffer allocation in the source). It represents shared memory eviction volume — newly started databases typically have higher values. When observing shared memory busyness, buffer allocation may be better than hit ratio — high hit ratios can be inflated by frequent small-table access, while allocation represents actual eviction.

buffers_alloc counts buffers allocated after reading from cache and loading into a new shared buffer — somewhat representative of OS cache reads too? But in practice, buffers_alloc and blks_read have similar meanings yet can differ significantly in value. Why? Unclear, pending research.

Source: numBufferAllocs

tup_fetched, tup_returned
#

These are metrics in pg_stat_database:

tup_fetched: Number of rows ultimately returned from index scans, after removing filtered rows, dead tuples, and invisible rows. Result-oriented.
tup_returned: Number of rows fetched from the table during index scans, regardless of filter conditions, dead tuples, or visibility. Process-oriented.

Thus, tup_returned is typically much higher than tup_fetched. An abnormally high tup_returned suggests optimization opportunity — after all, many rows were accessed but few returned to the client.

idx_tup_fetch, idx_tup_read
#

These are metrics in pg_stat_all_indexes:

idx_tup_read: Number of index entries accessed (counted from the index side), includes bitmap scans.
idx_tup_fetch: Number of rows ultimately returned from index scans (counted from the table side), excludes bitmap scans.

Madness.

One thing to remember: xx_tup_fetch refers to the final rows returned after index access + table fetch — result-oriented.

References
#

postgres-ai howtos

Best practices for using pgvector

Case Study: 2026-01-01 Partition Data Update Failure

PostgreSQL Partitioned Tables

Case Study: From Inaccurate DISTINCT to DISTINCT Calculation Principles

Case Study: Adding an Index Causes Performance Degradation and Generic Plans

From Static Table Query Conflicts to Their Principles

Control File Parameters and Primary-Standby Parameter Mismatch

https://liuzhilong.blog.csdn.net/article/details/130783036

https://techcommunity.microsoft.com/blog/adforpostgresql/improving-postgres-connection-scalability-snapshots/1806462

https://www.postgresql.org/docs/17/sql-prepare.html

https://www.postgresql.org/docs/17/sql-deallocate.html

https://www.postgresql.org/docs/release/13.0/

https://jdbc.postgresql.org/documentation/use/

https://jdbc.postgresql.org/documentation/server-prepare/#server-prepared-statements

https://www.postgresql.eu/events/pgconfeu2023/sessions/session/4791/slides/439/2023.pgconf.eu%20Zero%20Downtime%20PostgreSQL%20Upgrades.pdf

Thanks to Master Gao for the 2025 battles.

Case: Partition Data UPDATE Failure on 2026-01-01

Sun, 04 Jan 2026 00:00:00 +0000

Symptoms
#

On December 30, business errors were reported — data could not be updated:

ERROR: 55000: cannot update table "tablzl_202601" because it does not have a replica identity and publishes updates
HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
LOCATION: CheckCmdReplicaIdentity, execReplication.c:575

Temporary Recovery
#

The error message was clear: no replica identity. The table was a partitioned table and a 2026 partition, so I immediately suspected the new partition lacked a primary key. (A new table’s replica identity defaults to default, which only uses a primary key as the replica identity. Without a primary key, updates are impossible.)

Further investigation revealed: the parent table had no primary key or indexes, child partitions from 2025 and earlier had both primary keys and indexes, but 2026 and later child partitions had neither — and all child partitions were published. Roughly:

p_parent -- no PK, no indexes
p_child_202511 -- has PK, has indexes, published
p_child_202512 -- has PK, has indexes, published
p_child_202601 -- no PK, no indexes, published
p_child_202602 -- no PK, no indexes, published

Since the parent table had nothing, a partition of child would also have nothing — you must manually create the primary key and indexes for each child partition. So the new partition creation was problematic; the old partitions presumably had them added after creation.

Additionally, publishing partitioned tables via the parent was only supported starting from PG13. Previously, you couldn’t publish via the parent — only via child tables. This database was on PG11.

Allow partitioned tables to be logically replicated via publications (Amit Langote) § §

Previously, partitions had to be replicated individually. Now a partitioned table can be published explicitly, causing all its partitions to be published automatically. Addition/removal of a partition causes it to be likewise added to or removed from the publication. The CREATE PUBLICATION option publish_via_partition_root controls whether changes to partitions are published as their own changes or their parent’s.

After the initial diagnosis and given the urgency, there were three ways to temporarily resolve:

Add primary keys to the 2026 partitions
Set replica identity full on the 2026 partitions
Remove the 2026 partitions from the publication

Since recovery time was about the same for all options, we chose adding primary keys — the lowest operational cost — to at least stop the business errors.

Root Cause Analysis
#

The issue seems clear: “no replica identity + published + no primary key” prevents updates. But several questions still needed answers.

Question 1: Why does the UPDATE fail even though there’s no 202601 data at all (the new partition has zero rows)?
#

The SQL text was:

UPDATE tablzl_202601
 SET idid = $1,...
 date_updated = now()
 WHERE mykey = $4

The partition key for tablzl_202601 is created_date. The SQL WHERE clause didn’t include the partition key, so when attempting to update the 202601 partition, it found no primary key and errored out.

As for whether row existence or replica identity is checked first, we can see from ExecSimpleRelationUpdate. This function has changed very little across PG versions:

/*
 * Find the searchslot tuple and update it with data in the slot,
 * update the indexes, and execute any constraints and per-row triggers.
 *
 * Caller is responsible for opening the indexes.
 */
void
ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate,
					 TupleTableSlot *searchslot, TupleTableSlot *slot)
{
...
	CheckCmdReplicaIdentity(rel, CMD_UPDATE); // check replica identity

	/* BEFORE ROW UPDATE Triggers */
	if (resultRelInfo->ri_TrigDesc &&
		resultRelInfo->ri_TrigDesc->trig_update_before_row)
	{
		slot = ExecBRUpdateTriggers(estate, epqstate, resultRelInfo,
									&searchslot->tts_tuple->t_self,
									NULL, slot);

		if (slot == NULL)		/* "do nothing" */
			skip_tuple = true;
	}

	if (!skip_tuple)
	{
		List	 *recheckIndexes = NIL;

		/* Check the constraints of the tuple */
		if (rel->rd_att->constr)
			ExecConstraints(resultRelInfo, slot, estate);
		if (resultRelInfo->ri_PartitionCheck)
			ExecPartitionCheck(resultRelInfo, slot, estate, true);

		/* Materialize slot into a tuple that we can scribble upon. */
		tuple = ExecMaterializeSlot(slot);

		/* OK, update the tuple and index entries for it */
		simple_heap_update(rel, &searchslot->tts_tuple->t_self,
						 slot->tts_tuple);

		if (resultRelInfo->ri_NumIndices > 0 &&
			!HeapTupleIsHeapOnly(slot->tts_tuple))
			recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
												 estate, false, NULL,
												 NIL);

		/* AFTER ROW UPDATE Triggers */
		ExecARUpdateTriggers(estate, resultRelInfo,
							 &searchslot->tts_tuple->t_self,
							 NULL, tuple, recheckIndexes, NULL);

		list_free(recheckIndexes);
	}
}

ExecSimpleRelationUpdate flow:

Check replica identity
BEFORE ROW UPDATE triggers
Check constraints (both non-partition and partition constraints)
Update the row
Insert index entries
AFTER ROW UPDATE triggers

So PG’s logic checks replica identity first, before row updates and everything else.

Even though the SQL didn’t include the partition key, would adding it trigger partition pruning? The answer is: maybe not.

Partition pruning improvements across versions:

PG10 introduced declarative partitioning. There was no enable_partition_pruning parameter; pruning was done at planning time via constraint_exclusion. So PG10 had no query-execution-time pruning.
PG11 added runtime partition pruning: Allow partition elimination during query execution (David Rowley, Beena Emerson). But it only supports pruning with bound variables, not non-immutable functions (including now()).
PG14 added final pruning: This wins in UPDATEs on partitioned tables when only some of the partitions will actually receive updates. i.e., supports pruning with non-immutable functions.

Since PG11 doesn’t support now() pruning, adding a now() condition to the business SQL wouldn’t trigger pruning — the error would still occur. However, if the business passed a bound variable, pruning would trigger and the error wouldn’t appear. Note: “the error wouldn’t appear” means updating 202512 data wouldn’t error out on the 202601 partition; updating 202601 data would still fail regardless.

Question 2: The partition was created on 2025-12-26, so why was the problem only discovered on December 30?
#

This is even simpler: “no replica identity + published + no primary key” is an AND condition.

Although the new partitions were created early, they were published on the evening of December 29 at 20:47:

cat postgresql-12-29.csv.bak |grep "alter publication"
2025-12-29 20:48:07.730 CST,"userlzlreplication","lzldb",xxx"statement: alter publication publzl add table ""public"".""tablzl_202601"", ""public"".""tablzl_202602"",...

The first error appeared on December 29 at 22:26, about 1.5 hours later:

 cat postgresql-12-29.csv.bak |grep "REPLICA IDENTITY"
2025-12-29 22:26:01.404 CST,"userlzlreplication","lzldb",375121,xxx,"cannot update table ""tablzl_202601"" because it does not have a replica identity and publishes updates",,"To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.",,,,"UPDATE tablzl

Summary
#

Root cause overview: The parent table had no primary key, so partition of child partitions naturally also had none. Old child partitions had their primary keys added manually; new child partitions did not, resulting in the 202601 partition lacking a primary key. Logical replication relies on the primary key (default replica identity) for synchronization. Without replica identity, changes can’t be sent downstream, and UPDATE/DELETE statements on published tables cannot execute. In PG11, an UPDATE SQL that does include the partition key condition may still visit the new partition.

A stroke of luck: Due to various factors, this problem was discovered early in this particular database. We had a one-day buffer on December 31 to fix all database instances, ensuring at least that January 1 new partition data updates wouldn’t error out. Otherwise, on January 1, 2026, multiple systems would have likely gone up in flames.

Temporary measures (pick one):

Add primary keys to 2026 partitions
Set replica identity full on 2026 partitions
Remove 2026 partitions from the publication

For replication pipeline optimization:

Tables without primary keys should be detected proactively, otherwise publishing them could cause business-side UPDATE failures

For partition management strategy:

PG’s partitioned tables are highly flexible, and developers generally don’t know how to create partitions correctly. Combined with significant new partitioning features across roughly PG10-15, and the lack of INTERVAL partitioning in PG, partitioned tables can end up a mess. Standardized management of partitioned tables is thus critical. For partition table features and operational tips, see: PostgreSQL Partitioned Tables

As for management tools, I’ll skip those.

Management goals:

Use the parent table structure as the standard: the parent table, being developer-facing, should have primary keys, indexes, and replica identity (unless the PG version doesn’t support it)
Keep parent and child tables consistent; use partition of to create new partitions (yes, I don’t recommend attach)
Keep child tables consistent with each other
Create new partitions in advance; partition data volumes should not be excessive
Default partitions are not recommended; if created, their writes must be monitored
Frequently accessed tables must have partition keys in their SQL queries and use partition pruning; otherwise, convert them to regular tables

References
#

https://www.postgresql.org/docs/release/10.0/

https://www.postgresql.org/docs/release/11.0/

https://www.postgresql.org/docs/release/12.0/

https://www.postgresql.org/docs/release/13.0/

https://www.postgresql.org/docs/release/14.0/

src/backend/executor/execReplication.c

PostgreSQL Partitioned Tables

Case Study: Row Locks and LWLock LockManager

Sun, 21 Dec 2025 00:00:00 +0000

Symptoms
#

The database showed a large number of row locks and a smaller number of LWLock LockManager waits. CPU was maxed out and active sessions spiked. The blocking PID associated with the locks kept changing, with no obvious long-transaction blocker. (Imagine high CPU and active sessions.)

The SQL corresponding to the large number of locks was as follows:

UPDATE lzl_record SET rc_lzl1= rc_lzl1 + $1, pc_lzl2 = pc_lzl2 + $2, rc_lzl3 = rc_lzl3 + $3 where lzl_id = $4

Analysis
#

No Increase in SQL Concurrency Observed
#

From the correlation between hits and CPU, we can analyze from the SQL hit perspective. That UPDATE SQL accounted for about 80% of activity. The SQL’s execution count had not changed, but blks hit was clearly abnormal.

We also analyzed metadata access — within snapshots, no metadata tables showed unusually high access.

From the symptom analysis, neither SQL concurrency increase nor metadata anomalies were apparent. The reason for the SQL hit increase wasn’t obvious at this point.

LWLock LockManager Analysis
#

Since the SQL itself is simple — the lzl_id field in the lzl_record table is a unique field, meaning the update is done by unique key.

In addition to the large number of explicit locks, the wait events at the scene also included LWLock LockManager.

However, the table is a regular table (not partitioned), with only 4 or 5 indexes on it.

LWLock LockManager is related to not using the fast path. Simple queries and DML can use the fast path:

Weak relation locks. SELECT, INSERT, UPDATE, and DELETE must acquire a lock on every relation they operate on, as well as various system catalogs that can be used internally. Many DML operations can proceed in parallel against the same table at the same time; only DDL operations such as CLUSTER, ALTER TABLE, or DROP – or explicit user action such as LOCK TABLE – will create lock conflicts with the “weak” locks (AccessShareLock, RowShareLock, RowExclusiveLock) acquired by DML operations.

So a SELECT/DML accessing no more than 16 relations (including indexes) should be able to use the fast path, and there shouldn’t be much LWLock LockManager.

However, DML certainly can’t simply use the fast path — fast path handles lock operations entirely locally, but DML must check whether other sessions hold locks on the row and needs to access shared memory. Combined with the fact that this SQL updates by unique field yet still encounters row locks, it must be updating the same row.

From the logs, we could see instances of updating the same row — one row had tens of thousands of lock-waiting updates.

Benchmark Testing
#

Benchmarking Same-Row Updates to Reproduce LWLock LockManager
#

Given that row locks definitely can’t rely solely on the fast path, and knowing that LWLock LockManager degrades database performance, we benchmarked different scenarios.

#prompt
Give me a pgbench benchmark script
Table structure: primary key, unique field + unique index, other fields
Update: update by unique field

Benchmark repeated updates on the same row (repeated row-lock updates)
Benchmark random updates on different rows (no row-lock updates)

Script omitted. Environment: 20 cores, 96GB RAM.

pgbench commands:

pgbench -h localhost -p $PGPORT -d lzldb -U dbmgr -f update_same_unique_key.sql -c 200 -j 32 -T 600 -r -S
pgbench -h localhost -p $PGPORT -d lzldb -U dbmgr -f update_random_unique_key.sql -c 200 -j 32 -T 600 -r -S

Wait events during the benchmark:

 -- Update same row, 2 typical samples
 usename | state | wait_event | wait_event_type | cnt 
----------+--------+---------------------+-----------------+-----
 dbmgr | active | LockManager | LWLock | 105
 dbmgr | active | transactionid | Lock | 61
 dbmgr | active | tuple | Lock | 25
 dbmgr | active | [null] | [null] | 8
 dbmgr | active | WALSync | IO | 1
 
 usename | state | wait_event | wait_event_type | cnt 
----------+--------+---------------------+-----------------+-----
 dbmgr | active | transactionid | Lock | 180
 dbmgr | active | LockManager | LWLock | 18
 dbmgr | active | tuple | Lock | 1
 dbmgr | active | WALSync | IO | 1

-- Update different rows, 2 typical samples
usename | state | wait_event | wait_event_type | cnt 
----------+---------------------+---------------------+-----------------+-----
 dbmgr | active | [null] | [null] | 106
 dbmgr | idle | ClientRead | Client | 34
 dbmgr | idle in transaction | ClientRead | Client | 25
 dbmgr | active | WALWrite | LWLock | 21
 dbmgr | active | BufferMapping | LWLock | 7
 dbmgr | idle in transaction | [null] | [null] | 4
 dbmgr | idle in transaction | WALWrite | LWLock | 2
 
 usename | state | wait_event | wait_event_type | cnt 
----------+---------------------+---------------------+-----------------+-----
 dbmgr | active | [null] | [null] | 117
 dbmgr | idle | ClientRead | Client | 42
 dbmgr | idle in transaction | ClientRead | Client | 24
 dbmgr | active | WALWrite | LWLock | 12
 dbmgr | active | XactGroupUpdate | IPC | 1
 dbmgr | active | WALSync | IO | 1
 dbmgr | active | XactSLRU | LWLock | 1
 dbmgr | active | BufferContent | LWLock | 1
 dbmgr | active | ClientRead | Client | 1

From the wait events, the difference is clear: updating the same row produces LWLock LockManager, sometimes at a high proportion. Updating different rows mostly just waits on CPU. Scenario 1 matches the production situation.

A Brief Analysis of Row Locks and Fast Path
#

The lmgr README’s explanation of the fast path:

Fast Path Locking
-----------------

Fast path locking is a special purpose mechanism designed to reduce the
overhead of taking and releasing certain types of locks which are taken
and released very frequently but rarely conflict. Currently, this includes
two categories of locks:

(1) Weak relation locks. SELECT, INSERT, UPDATE, and DELETE must acquire a
lock on every relation they operate on, as well as various system catalogs
that can be used internally. Many DML operations can proceed in parallel
against the same table at the same time; only DDL operations such as
CLUSTER, ALTER TABLE, or DROP -- or explicit user action such as LOCK TABLE
-- will create lock conflicts with the "weak" locks (AccessShareLock,
RowShareLock, RowExclusiveLock) acquired by DML operations.

Conditions for locks that can use the fast path, from lmgr/lock.c:

/*
 * The fast-path lock mechanism is concerned only with relation locks on
 * unshared relations by backends bound to a database. The fast-path
 * mechanism exists mostly to accelerate acquisition and release of locks
 * that rarely conflict. Because ShareUpdateExclusiveLock is
 * self-conflicting, it can't use the fast-path mechanism; but it also does
 * not conflict with any of the locks that do, so we can ignore it completely.
 */
#define EligibleForRelationFastPath(locktag, mode) \
	((locktag)->locktag_lockmethodid == DEFAULT_LOCKMETHOD && \
	(locktag)->locktag_type == LOCKTAG_RELATION && \
	(locktag)->locktag_field1 == MyDatabaseId && \
	MyDatabaseId != InvalidOid && \
	(mode) < ShareUpdateExclusiveLock)

SELECT/DML can use the fast path, but only for locktype=relation.

Let’s look at the actual lock situation when there’s a row lock:

-- Session 1
begin;
update lzl1 set b='zzz' where a=1;

-- Session 2
begin;
update lzl1 set b='zzz' where a=1;
-- waiting


-- Session 3
 select * from pg_locks where pid<>(select pg_backend_pid()) order by pid,locktype;
 locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath 
---------------+----------+----------+--------+--------+------------+---------------+---------+--------+----------+--------------------+--------+------------------+---------+----------
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 5/4791 | 220559 | RowExclusiveLock | t | t
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706189 | [null] | [null] | [null] | 5/4791 | 220559 | ExclusiveLock | t | f
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706190 | [null] | [null] | [null] | 5/4791 | 220559 | ExclusiveLock | t | f
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706187 | [null] | [null] | [null] | 5/4791 | 220559 | ShareLock | f | f
 tuple | 4267681 | 5290151 | 0 | 1 | [null] | [null] | [null] | [null] | [null] | 5/4791 | 220559 | ExclusiveLock | t | f
 virtualxid | [null] | [null] | [null] | [null] | 5/4791 | [null] | [null] | [null] | [null] | 5/4791 | 220559 | ExclusiveLock | t | t
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 7/562 | 253641 | RowExclusiveLock | t | t
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706187 | [null] | [null] | [null] | 7/562 | 253641 | ExclusiveLock | t | f
 virtualxid | [null] | [null] | [null] | [null] | 7/562 | [null] | [null] | [null] | [null] | 7/562 | 253641 | ExclusiveLock | t | t

PG’s row lock implementation is quite complex — it involves not only tuple locks, but also transactionid and relation locks. Among these, only locktype=relation and virtualxid can use the fast path; all others cannot.

Compare with the no-row-lock case:

-- Session 1
begin;
update lzl1 set b='zzz' where a=1;

-- Session 2
begin;
update lzl1 set b='zzz' where a=2;
-- waiting

select * from pg_locks where pid<>(select pg_backend_pid()) order by pid,locktype;
 locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath 
---------------+----------+----------+--------+--------+------------+---------------+---------+--------+----------+--------------------+--------+------------------+---------+----------
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 5/4792 | 220559 | RowExclusiveLock | t | t
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 5/4792 | 220559 | AccessShareLock | t | t
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706214 | [null] | [null] | [null] | 5/4792 | 220559 | ExclusiveLock | t | f
 virtualxid | [null] | [null] | [null] | [null] | 5/4792 | [null] | [null] | [null] | [null] | 5/4792 | 220559 | ExclusiveLock | t | t
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 7/563 | 253641 | AccessShareLock | t | t
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 7/563 | 253641 | RowExclusiveLock | t | t
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706212 | [null] | [null] | [null] | 7/563 | 253641 | ExclusiveLock | t | f
 virtualxid | [null] | [null] | [null] | [null] | 7/563 | [null] | [null] | [null] | [null] | 7/563 | 253641 | ExclusiveLock | t | t

There are only 2-3 fewer fastpath=f entries. The transactionid locks held by both sessions definitely can’t use the fast path.

Summary of conditions for using the fast-path lock mechanism (all must be met):

Lock level <= 3, i.e., SELECT/DML statements
locktype=relation. PG’s row locks also require at least transactionid and tuple locks, so these two can’t use the fast path
Fewer than 16 relations accessed (typically exceeded only with full partition access on partitioned tables)

Conclusion
#

Is the row lock the cause or the effect? Is it a row lock problem, or did database performance degrade causing SQL to run slower and produce row locks?

Row lock is the cause. The SQL execution count didn’t change, but the SQL parameters shifted from scattered to concentrated — i.e., updates to the same row noticeably increased. From the benchmark data, updating the same row produces row lock and LWLock LockManager waits.

SQL execution count didn’t increase — did SQL performance degrade?

SQL performance did degrade, but the index was definitely not chosen incorrectly — it was simply because the same row was being updated repeatedly.

Solution:

From the business side, the SQL was tied to a certain API endpoint: after being called, it updates the call count into the table. If the same endpoint is called repeatedly, it’s possible to repeatedly update the same row. Therefore, reducing repeated calls to the same endpoint, or batching the database updates into fewer, larger batches, is expected to mitigate this problem.

Case: From Inaccurate DISTINCT to the Principles of DISTINCT Estimation

Sun, 19 Oct 2025 00:00:00 +0000

Problem Description
#

The n_distinct statistic was severely inaccurate.

This problem appeared across multiple databases. For example:

A table with 200 million rows and a true DISTINCT count of 8 million had a statistics DISTINCT value of only 40,000.

Analysis
#

Sampling Model
#

The default default_statistics_target=100 means 30,000 rows are sampled from 30,000 pages.

analyze verbose tablzl1;
INFO: 00000: analyzing "public.tablzl1"
LOCATION: do_analyze_rel, analyze.c:332
INFO: 00000: "tablzl1": scanned 30000 of 22963751 pages, containing 1061942 live rows and 3953 dead rows; 30000 rows in sample, 812872389 estimated total rows
LOCATION: acquire_sample_rows, analyze.c:1340

Note “scanned 30000” and “30000 rows in sample”.

DISTINCT Estimation Algorithm
#

The DISTINCT estimation algorithm in analyze.c:

			/*----------
			 * Estimate the number of distinct values using the estimator
			 * proposed by Haas and Stokes in IBM Research Report RJ 10025:
			 *		n*d / (n - f1 + f1*n/N)
			 * where f1 is the number of distinct values that occurred
			 * exactly once in our sample of n rows (from a total of N),
			 * and d is the total number of distinct values in the sample.
			 * This is their Duj1 estimator; the other estimators they
			 * recommend are considerably more complex, and are numerically
			 * very unstable when n is much smaller than N.
			 *
			 * In this calculation, we consider only non-nulls. We used to
			 * include rows with null values in the n and N counts, but that
			 * leads to inaccurate answers in columns with many nulls, and
			 * it's intuitively bogus anyway considering the desired result is
			 * the number of distinct non-null values.
			 *
			 * We assume (not very reliably!) that all the multiply-occurring
			 * values are reflected in the final track[] list, and the other
			 * nonnull values all appeared but once. (XXX this usually
			 * results in a drastic overestimate of ndistinct. Can we do
			 * any better?)
			 *----------
			 */
			int			f1 = nonnull_cnt - summultiple;
			int			d = f1 + nmultiple;
			double		n = samplerows - null_cnt;
			double		N = totalrows * (1.0 - stats->stanullfrac);
			double		stadistinct;

n*d / (n - f1 + f1*n/N)

n = number of sample rows (rows scanned)
d = number of distinct values found in the sample
f1 = number of values appearing exactly once in the sample
N = total number of rows in the table

Algorithm paper: https://hugepdf.com/download/download-extended-version-of-this-paper_pdf

The paper is rather dense, so let’s work through some assumptions to understand this DISTINCT algorithm:

Assume all values appear exactly once, and the table is large (n « N), so f1 = d, n/N ≈ 0

d*d / (d - d + d*0) = d²/0 — this would evaluate to -1.

Assume all values appear exactly once, and the table is small (n = N), so f1 = d, n/N = 1

n*d / (n - d + d*1) = d — the sampled distinct count, which equals the number of sampled rows.

Assume no values appear exactly once in the sample, i.e., f1 = 0

n*d / (n - f1 + f1*n/N) = n*d / n = d — just the distinct count in the sample.

If a column is populated by inserting several rows of the same value, then several rows of another value, like:

11, 2, 2, 2, 2, 3, 3, 3, …

3.1 Small table, all 30,000 rows sampled, true distinct = 10,000 (assumed): estimated distinct = d = 10,000

3.2 Large table, sample contains both repeating values and singletons (some repeating values only have one row captured), i.e., n = 30,000, n/N ≈ 0

n*d / (n - f1 + f1*n/N) = n*d / (n - f1) = 30000*d/(30000-f1) — the larger the distinct count in the sample, the larger the estimated distinct; the larger the number of singletons, the larger the estimated distinct.

Summary:

DISTINCT estimation is directly related to the distinct count and singleton count in the sample
If the singleton count = 0, then larger samples yield larger estimated distinct values

Verification
#

Since the default maximum sample size is 30,000 rows, for tables larger than this, the estimator is likely to underestimate DISTINCT. Note: the data should not have too many unique values.

Testing a table with different sample sizes:

Table: reltuples = 800 million, relpages = 20 million, size = 175GB, true column distinct = 100 million

target statistics	pages sampling ratio (approx)	tuples sampling ratio (approx)	n_distinct	execution time
50	0.00075	0.00001875	60k	2s
100	0.0015	0.0000375	110k	5s
1000	0.015	0.000375	1.03M	58s
3000	0.045	0.001125	2.68M	3min 1s
10000	0.15	0.00375	6.75M	7min 21s

(maximum target statistics is 10000)

A rough conclusion: n_distinct and ANALYZE execution time grow proportionally with the sample size.

n_distinct grows with sample size, while pages and tuples estimates remain consistently accurate.

Solution
#

For extremely large tables, consider partitioning or optimizing based on actual SQL patterns.

You can also adjust the statistics target. The default default_statistics_target=100 means 30,000 rows from 30,000 pages.

Temporary fix:

set default_statistics_target=3000;
analyze tab1;

Long-term fix:

alter table tab1 alter column col1 set STATISTICS 3000;

Notes:

Column-level statistics target has the highest priority, overriding default_statistics_target
Maximum statistics target is 10000
The table’s sampling target is determined by the maximum column target:

	/*
	 * Determine how many rows we need to sample, using the worst case from
	 * all analyzable columns. We use a lower bound of 100 rows to avoid
	 * possible overflow in Vitter's algorithm. (Note: that will also be the
	 * target in the corner case where there are no analyzable columns.)
	 */
	targrows = 100;
	for (i = 0; i < attr_cnt; i++)
	{
		if (targrows < vacattrstats[i]->minrows)
			targrows = vacattrstats[i]->minrows;
	}
	for (ind = 0; ind < nindexes; ind++)
	{
		AnlIndexData *thisdata = &indexdata[ind];

		for (i = 0; i < thisdata->attr_cnt; i++)
		{
			if (targrows < thisdata->vacattrstats[i]->minrows)
				targrows = thisdata->vacattrstats[i]->minrows;
		}
	}

If ANALYZE collects more or fewer rows than expected, check pg_statistic for per-column stattarget settings:

select attrelid::regclass,attname,attstattarget from pg_attribute where attrelid = 'tab1'::regclass and attstattarget not in (-1,0);

Summary
#

For large tables where columns are non-unique but have high distinct counts (a realistic scenario), the sampling algorithm underestimates the DISTINCT value, and this is positively correlated with the sampling ratio. The default sampling ratio is too small for large tables. You can increase it, but even the maximum is not that large.

Case Study: Performance Degradation After Adding an Index and the Generic Plan

Sat, 13 Sep 2025 00:00:00 +0000

Problem Description
#

An index was added the night before, and the next morning the CPU was maxed out. The problematic SQL was easy to locate — just one query. The SQL was running for over 30 seconds, but the day before it only took about 3 seconds, so we needed to examine the before-and-after execution plan changes.

Only the key parts of the execution plan are shown below.

Execution plan before adding the index:

 -> Nested Loop (cost=19.92..2259694.20 rows=265822 width=33)
 -> Index Scan using uk_lzl_task on lzl_task t (cost=0.29..20007.99 rows=195 width=24)
 Filter: ((created_by)::text = 'LIUZHILONG62'::text)
 -> Append (cost=19.63..11337.15 rows=14842 width=57)
 -> Bitmap Heap Scan on lzl_202501 cc_1 (cost=19.63..3053.69 rows=1467 width=66)
 Recheck Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Bitmap Index Scan on lzl_202501_task_no_idx (cost=0.00..19.27 rows=1594 width=0)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 -> Bitmap Heap Scan on lzl_202502 cc_2 (cost=21.67..3066.85 rows=1604 width=66)
 Recheck Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Bitmap Index Scan on lzl_202502_task_no_idx (cost=0.00..21.27 rows=1605 width=0)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 -> Index Scan using lzl_202503_task_no_idx on lzl_202503 cc_3 (cost=0.43..1362.61 rows=1637 width=57)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202504_task_no_idx on lzl_202504 cc_4 (cost=0.43..604.64 rows=1795 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202505_task_no_idx on lzl_202505 cc_5 (cost=0.43..445.30 rows=1450 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202506_task_no_idx on lzl_202506 cc_6 (cost=0.43..583.94 rows=1675 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202507_task_no_idx on lzl_202507 cc_7 (cost=0.43..633.45 rows=1973 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202508_task_no_idx on lzl_202508 cc_8 (cost=0.43..619.43 rows=1720 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202509_task_no_idx on lzl_202509 cc_9 (cost=0.42..893.03 rows=1521 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))

The created_date time range searches for data within 1 year. The index added the night before was on created_date.

Execution plan after adding the index:

 -> Hash Join (cost=63.37..23740.82 rows=191 width=33)
 Hash Cond: ((cc.task_no)::text = (t.task_no)::text)
 -> Append (cost=0.00..23376.98 rows=114435 width=58)
 Subplans Removed: 28
 -> Index Scan using idx_lzltab_202501_created_date on lzltab_202501 cc_1 (cost=0.43..1450.59 rows=8958 width=66)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202502_created_date on lzltab_202502 cc_2 (cost=0.43..1822.73 rows=7405 width=66)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202503_created_date on lzltab_202503 cc_3 (cost=0.43..1430.03 rows=7917 width=57)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202504_created_date on lzltab_202504 cc_4 (cost=0.43..2412.44 rows=11041 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202505_created_date on lzltab_202505 cc_5 (cost=0.43..2260.73 rows=13381 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202506_created_date on lzltab_202506 cc_6 (cost=0.43..3930.10 rows=17832 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202507_created_date on lzltab_202507 cc_7 (cost=0.43..3878.77 rows=21786 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202508_created_date on lzltab_202508 cc_8 (cost=0.43..4736.72 rows=22033 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202509_created_date on lzltab_202509 cc_9 (cost=0.42..627.09 rows=1893 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Hash (cost=63.03..63.03 rows=27 width=24)
 -> Bitmap Heap Scan on ai_outbound_call_task t (cost=2.99..63.03 rows=27 width=24)
 Recheck Cond: ((created_by)::text = ($3)::text)
 -> Bitmap Index Scan on idx_ai_call_task_c (cost=0.00..2.99 rows=27 width=0)
 Index Cond: ((created_by)::text = ($3)::text)

The new execution plan switched from using the task_no index to using the created_date index, and changed from a Nested Loop to a Hash Join. The cost dropped from 2,259,694 to 23,740 — a 100x reduction. However, the actual execution time increased by roughly 10x.

Problem Diagnosis
#

Let’s work through three questions to analyze and diagnose the issue:

Why did the optimizer suggest the created_date index?
Why did it end up using the new index?
Why is the estimated row count very small even though the actual execution time is very long?

Why Did the Optimizer Suggest the created_date Index?
#

If we directly substitute the parameters from the PostgreSQL log into the SQL text, the execution plan is actually the good one — the one that runs in 3 seconds using the task_no index. The optimization engineer also ran it this way and found it to be fine. But in production, this wasn’t the execution plan that was used.

Even when we force PostgreSQL not to use the task_no index, the optimizer chooses a sequential scan rather than the created_date index:

 Hash Cond: (((cc.task_no)::text || ''::text) = (t.task_no)::text)
 -> Append (cost=0.00..2794425.58 rows=22238757 width=57)
 -> Seq Scan on lzltab_202501 cc_1 (cost=0.00..193060.05 rows=1585238 width=66)
 Filter: ((created_date > '2025-01-08 11:00:00'::timestamp without time zone) AND (created_date < '2025-09-04 08:31:43'::timestamp without time zone))
 -> Seq Scan on lzltab_202502 cc_2 (cost=0.00..178567.54 rows=1480969 width=66)
 Filter: ((created_date > '2025-01-08 11:00:00'::timestamp without time zone) AND (created_date < '2025-09-04 08:31:43'::timestamp without time zone))
 -> Seq Scan on lzltab_202503 cc_3 (cost=0.00..191073.34 rows=1583356 width=57)

This is very strange: no matter how we ran it ourselves, we couldn’t get it to use the bad created_date index. So how did production end up using it?

The answer lies in bind variables — it was likely a generic plan.

Characteristics of the generic plan:

When plan_cache_mode = auto, PostgreSQL compares the generic plan cost against the average cost of the first five hard parses (custom plans). If the generic plan has a lower cost, it is used and subsequent executions skip hard parsing; otherwise, every execution undergoes hard parsing (see the source function choose_custom_plan).
What the generic plan looks like has nothing to do with the actual bind variable values.

This is easy to reproduce using bind variables via PREPARE/EXECUTE:

PREPARE sql1(timestamp without time zone,timestamp without time zone,text) AS
SELECT COUNT(*)
xxxxxxx...;

=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 367.220 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 254.386 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 235.343 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 234.110 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 233.570 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 70678.344 ms (01:10.678) -- 6th execution is significantly slower


=# select * from pg_prepared_statements\gx -- pg14 supports pg_prepared_statements
generic_plans | 1
custom_plans | 5

The first 5 hard parses (custom plans) all executed quickly. The 6th execution used the generic plan, which used the created_date index — this was the exact production failure plan, which was extremely slow.

So while the optimization suggestion to use the created_date index was somewhat problematic, when you substituted bind variables with actual values and ran EXPLAIN, the execution plan was correct. In production, however, the application used bind variables, and the generic plan kicked in — causing the failure.

Why Is the Estimated Row Count Small But the Actual Execution Time Very Long?
#

The failing execution plan has a problem: the estimated cost is too small, and the estimated rows are too few.

 -> Index Scan using idx_lzltab_202501_created_date on lzltab_202501 cc_1 (cost=0.43..1450.59 rows=8958 width=66)
 Index Cond: ((created_date > $1) AND (created_date < $2))

From a business logic perspective, this looks abnormal. The created_date condition spans multiple partitions, and since created_date is the partition key, WHERE created_date >= xx AND <= yy must be contiguous. The selectivity on a sub-partition should always be 1, meaning rows should equal the sub-partition row count — several million, not several thousand.

At first I thought it was a statistics issue, but the statistics were fairly accurate — the historical partition data for 202501 hadn’t changed.

Since this is a generic plan issue, we need to examine the generic plan cost estimation by reading the source code. Cost estimation is more complex, but rows estimation is relatively easier to understand and locate.

static double
calc_rangesel(TypeCacheEntry *typcache, VariableStatData *vardata,
			 const RangeType *constval, Oid operator)
{
...
		else
		{
			/* with any other operator, empty Op non-empty matches nothing */
			selec = (1.0 - empty_frac) * hist_selec;
		}
	}

	/* all range operators are strict */
	selec *= (1.0 - null_frac);

range_select = (1 - null_frac) * histogram_selectivity. The range histogram selectivity looks at the histogram buckets hit by the range plus any matching MCV entries. However, we don’t need to compute all this for this case.

Because the generic plan does not look at the histogram:

/*
 * rangesel -- restriction selectivity for range operators
 */
Datum
rangesel(PG_FUNCTION_ARGS)
{
...
	/*
	 * If we got a valid constant on one side of the operator, proceed to
	 * estimate using statistics. Otherwise punt and return a default constant
	 * estimate. Note that calc_rangesel need not handle
	 * OID_RANGE_ELEM_CONTAINED_OP.
	 */
	if (constrange)
		selec = calc_rangesel(typcache, &vardata, constrange, operator);
	else
		selec = default_range_selectivity(operator);
...
}

calc_rangesel is the selectivity calculation function that takes constant values (used above). The else branch calls default_range_selectivity, which does not pass any constants.

/*
 * Returns a default selectivity estimate for given operator, when we don't
 * have statistics or cannot use them for some reason.
 */
static double
default_range_selectivity(Oid operator)
{
	switch (operator)
	{
 ...
		case OID_RANGE_CONTAINS_ELEM_OP:
		case OID_RANGE_ELEM_CONTAINED_OP:

			/*
			 * "range @> elem" is more or less identical to a scalar
			 * inequality "A >= b AND A <= c".
			 */
			return DEFAULT_RANGE_INEQ_SEL;
	}
}

The default range selectivity define:

/* default selectivity estimate for range inequalities "A > b AND A < c" */
#define DEFAULT_RANGE_INEQ_SEL	0.005

Let’s verify this against the production row estimate:

 select reltuples::bigint*0.005 from pg_class where relname='lzltab_202501'\gx
-[ RECORD 1 ]------
?column? | 8958.350

This matches the actual estimated rows of 8958:

idx_lzltab_202501_created_date on lzltab_202501 cc_1 (cost=0.43..1450.59 rows=8958 width=66)

So the new execution plan’s inaccurate estimate is because the generic plan uses a default selectivity of 0.005.

Summary
#

Why Does the Generic Plan Exist, and the Problem with Soft Parsing
#

It’s easier to think of the generic plan as a “DEFAULT estimate plan.”

Why does the generic plan always seem to have problems?

Let’s trace the reasoning chain:

The generic plan exists to reduce hard parsing, i.e., to enable soft parsing.
If we don’t hard-parse every execution, we can reuse an execution plan without passing specific parameter values.
If we don’t pass parameters and directly use an execution plan, that plan must be generated in advance.

Ways to generate an execution plan in advance:

A parameter-less execution plan (the generic plan)
Reuse an execution plan generated from the first few executions with parameters (PostgreSQL doesn’t have this)

If we use a generic plan, it can be inaccurate, for example:

1. Data skew (e.g., a particular MCV has a very high frequency, like WHERE a = 1 but a = 1 appears extremely often). This heavily depends on what the parameter value actually is, but the generic plan receives no parameters, so the plan cannot be accurate.
1. Evenly distributed data where selectivity still cannot be accurately calculated (e.g., WHERE a > $1 AND a < $2). Without knowing the range, no one can compute the selectivity. The generic plan receives no parameters, so the plan cannot be accurate.

If we reused plans from the first few parameterized executions (which PostgreSQL doesn’t do), they could also be inaccurate:

Data skew: the first few parameter values may not be representative, and they would heavily influence what the subsequent fixed plan looks like.

Categories of Generic Plan Estimation Problems
#

Because the comparison requires 5 custom plans first, generic plan problems can be divided into two categories:

The first 5 SQL executions are not representative. This is closely tied to the first 5 execution plans and depends on data skew and whether the first 5 parameter values are representative.
The generic plan itself is problematic. Due to data skew or the inability to accurately compute selectivity for evenly distributed data, the generic plan itself is inefficient.

Optimization Recommendations
#

Based on this case, generic plan issues can appear on partitioned tables. The partition key is contiguous, and selectivity when scanning all partitions should be 1, but the generic plan uses 0.005, which can easily lead to a “full index scan” scenario.

So during optimization, we need to consider more:

Avoid creating too many indexes that confuse the optimizer.
Eliminate generic plan interference. Use EXECUTE to truly run the query 6 times.
At the session level, set plan_cache_mode = 'force_generic_plan' or set plan_cache_mode = 'force_custom_plan' to compare execution plans. Or, on pg16+, use EXPLAIN (GENERIC_PLAN) to compare.

Syntax reference:

--prepare/excute
PREPARE sql1(text) AS
SELECT COUNT(*) FROM LZL where a=$1;

EXECUTE sql1('zzz'); -- run 6 times first
EXPLAIN EXECUTE sql1('zzz');

select * from pg_prepared_statements -- view prepared statement info, current session only

-- Compare execution plans by setting session parameters before EXPLAIN EXECUTE
set plan_cache_mode='force_generic_plan'
set plan_cache_mode='force_custom_plan'
-- Directly view generic plan, pg16+
explain (GENERIC_PLAN) xx 

Case: GRANT Authorization Causes Walsender to Hang

Thu, 26 Jun 2025 00:00:00 +0000

Symptoms
#

The walsender’s LSN stopped advancing. The stack trace showed it was stuck in pathman’s invalidate_psin_entries_using_relid, with the relid constantly changing and the walsender CPU pegged at 100%.

pstack 121327
#0 hash_seq_search (status=status@entry=0x7fffaadf8330) at dynahash.c:1441
#1 0x00002ba3b40ec728 in invalidate_psin_entries_using_relid (relid=relid@entry=42319501) at src/relation_info.c:251
#2 0x00002ba3b40ecb3d in forget_status_of_relation (relid=relid@entry=42319501) at src/relation_info.c:232
#3 0x00002ba3b40fcc96 in pathman_relcache_hook (arg=<optimized out>, relid=42319501) at src/hooks.c:934
#4 0x000000000087168a in LocalExecuteInvalidationMessage (msg=0x3a391c8) at inval.c:595
#5 0x000000000071d50e in ReorderBufferExecuteInvalidations (rb=0x1b63ff8, txn=0x1be5f58, txn=0x1be5f58) at reorderbuffer.c:2238
#6 ReorderBufferCommit (rb=0x1b63ff8, xid=xid@entry=4285897514, commit_lsn=405674661986920, end_lsn=<optimized out>, commit_time=commit_time@entry=799377897828299, origin_id=origin_id@entry=0, origin_lsn=origin_lsn@entry=0) at reorderbuffer.c:1819
#7 0x0000000000712d18 in DecodeCommit (xid=4285897514, parsed=0x7fffaadf8630, buf=0x7fffaadf87f0, ctx=0x1a359e8) at decode.c:637
#8 DecodeXactOp (ctx=0x1a359e8, buf=buf@entry=0x7fffaadf87f0) at decode.c:245
#9 0x00000000007130b2 in LogicalDecodingProcessRecord (ctx=0x1a359e8, record=0x1a35c80) at decode.c:114
#10 0x0000000000733662 in XLogSendLogical () at walsender.c:2885
#11 0x0000000000735942 in WalSndLoop (send_data=send_data@entry=0x733620 <XLogSendLogical>) at walsender.c:2287
#12 0x0000000000736692 in StartLogicalReplication (cmd=0x1846c68) at walsender.c:1213
#13 exec_replication_command (cmd_string=cmd_string@entry=0x181a288 "START_REPLICATION SLOT \"lzl_logical_rep\" LOGICAL 170F5/7C3EAE78 (\"proto_version\" '1', \"publication_names\" 'lzl_logical_rep')") at walsender.c:1640
#14 0x0000000000774e91 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x1866478, dbname=0x18662b8 "lzldb", username=<optimized out>) at postgres.c:4325
#15 0x0000000000485989 in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4526
#16 BackendStartup (port=0x18635b0) at postmaster.c:4210
#17 ServerLoop () at postmaster.c:1739
#18 0x0000000000702f08 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x1814da0) at postmaster.c:1412
#19 0x000000000048660a in main (argc=3, argv=0x1814da0) at main.c:210

## Second execution, same stack, different relid
pstack 121327
#0 hash_seq_search (status=status@entry=0x7fffaadf8330) at dynahash.c:1441
#1 0x00002ba3b40ec728 in invalidate_psin_entries_using_relid (relid=relid@entry=26560221) at src/relation_info.c:251
#2 0x00002ba3b40ecb3d in forget_status_of_relation (relid=relid@entry=26560221) at src/relation_info.c:232
#3 0x00002ba3b40fcc96 in pathman_relcache_hook (arg=<optimized out>, relid=26560221) at src/hooks.c:934
#4 0x000000000087168a in LocalExecuteInvalidationMessage (msg=0x39f1f68) at inval.c:595
...

Analysis
#

The changing relid showed that the walsender was still running, not dead. The LSN was not advancing, so we analyzed the LSN position to see what the transaction was doing.

If the slot information was still available, we could look up the restart LSN via the slot view to find the WAL position. If not, we could use the LSN from the stack trace to identify the WAL log.

Using pg_waldump to inspect WAL log entries, filtering by xid:

rmgr: Heap len (rec/tot): 961/ 961, tx: 4285897514, lsn: 170F5/7DFE3470, prev 170F5/7DFE3430, desc: UPDATE+INIT off 2 xmax 4285897514 flags 0x00 ; new off 1 xmax 0, blkref #0: rel 1663/17662/1259 blk 8443, blkref #1: rel 1663/17662/1259 blk 7327

...
rmgr: Transaction len (rec/tot): 1778325/1778325, tx: 4285897514, lsn: 170F5/7E1F4268, prev 170F5/7E1F4220, desc: COMMIT 2025-05-01 09:24:57.828299 CST; inval msgs: catcache 22 catcache 22 catcache 22 catcache 22 catcache 50 catcache 49 catcache 50 catcache 49 catcache 50 catcache 49 catcache 50 catcache 49 catcache 50 catcache 49 catcache 50 
...
 relcache 48813261 relcache 48813255 relcache 51030741 relcache 48813252 relcache 50737247 relcache 48813246 relcache 48813243 relcache 48813237 relcache 50737241 relcache 48813234 relcache 48813224 relcache 49379811 relcache 48813216 relcache 48813210 relcache 45452775

The transaction for rel 1663/17662/1259 had 180,000 records. The last record was inval msgs: ~70,000 catcache entries and ~30,000 relcache entries.

rel 1663/17662/1259 is pg_class. Querying by xmin reveals the affected tables and commit time:

select xmin,xmax,pg_xact_commit_timestamp(xmin),relname from pg_class where xmin='4285897514'::xid order by relname desc ;
 xmin | xmax | pg_xact_commit_timestamp | relname 
------------+------+-------------------------------+---------------------------------------------
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | v$session
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | tmp_20230801_id_seq
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | tmp_20230801
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | test_param
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | test_20240105
 ...
 
 select count(*) from pg_class where xmin='4285897514'::xid ;
 count 
-------
 18523

 select count(*) from pg_class ;
 count 
--------
 139138

Checking the pglog by timestamp:

2025-05-01 09:24:59.837 CST,"postgres","lzldb",61418,"[local]",6812cd65.efea,3,"DO",2025-05-01 09:24:53 CST,549/0,0,LOG,00000,"duration: 6036.275 ms statement: 
...
 EXECUTE 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO r_lzldbdata_qry';
...
 END;
 $$",,,,,,,,,"psql","client backend"

We can basically confirm that the GRANT operation was the culprit. GRANT updates relacl in pg_class, and at least 18,000 relations had their permissions updated. Updates to pg_class trigger invalidation messages, and the massive number of invalidation messages were being processed slowly in the walsender process.

Reproduction
#

-- Create a logical replication slot, any kind will do
select pg_create_logical_replication_slot('logical_test','test_decoding');
pg_recvlogical -h 127.0.0.1 -p 7997 -d lzldb -U repuser --slot=logical_test --start -f recv.sql &

-- Create many tables
DO $$
BEGIN
 FOR i IN 1..20000 LOOP
 EXECUTE format(
 'CREATE TABLE IF NOT EXISTS table_%s ( 
 col1 varchar(10)
 )', 
 lpad(i::text, 5, '0') -- Generate 5-digit numbered table names
 );
 END LOOP;
END $$;

-- Single GRANT
grant select on all tables in schema public to r_lzldb_qry;

-- Perfectly reproduced
postgres@lzlhost:~/lzl/grant]$ pstack 172862
#0 hash_seq_search (status=status@entry=0x7ffd664be280) at dynahash.c:1444
#1 0x00002ad31235e728 in invalidate_psin_entries_using_relid (relid=relid@entry=1002857) at src/relation_info.c:251
#2 0x00002ad31235eb3d in forget_status_of_relation (relid=relid@entry=1002857) at src/relation_info.c:232
#3 0x00002ad31236ec96 in pathman_relcache_hook (arg=<optimized out>, relid=1002857) at src/hooks.c:934
#4 0x000000000087168a in LocalExecuteInvalidationMessage (msg=0x2ad3c3f61a88) at inval.c:595
#5 0x000000000071d50e in ReorderBufferExecuteInvalidations (rb=0x17e5698, txn=0x180d698, txn=0x180d698) at reorderbuffer.c:2238


[postgres@lzlhost:~/lzl/grant]$ pstack 172862
#0 0x0000000000891d0c in hash_seq_search (status=status@entry=0x7ffd664be280) at dynahash.c:1441
#1 0x00002ad31235e728 in invalidate_psin_entries_using_relid (relid=relid@entry=1011110) at src/relation_info.c:251
#2 0x00002ad31235eb3d in forget_status_of_relation (relid=relid@entry=1011110) at src/relation_info.c:232
#3 0x00002ad31236ec96 in pathman_relcache_hook (arg=<optimized out>, relid=1011110) at src/hooks.c:934


-- relid keeps changing
-- CPU pegged at 100%:
ps -eo pid,%cpu,%mem|grep 172862
172862 99.3 0.0

-- Takes about 2 hours to catch up

Accelerating Walsender by Removing Pathman
#

Since the database wasn’t actually using pathman partitioned tables but had the extension installed, we tried bypassing the pathman hook to speed up walsender processing.

drop extension pg_pathman;
grant update on all tables in schema public to r_lzldb_upd;

[postgres@lzlhost~/lzl/grant]$ pstack 133460
#0 hash_seq_search (status=status@entry=0x7ffe292d5c90) at dynahash.c:1418
#1 0x000000000087f228 in RelfilenodeMapInvalidateCallback (arg=<optimized out>, relid=1034036) at relfilenodemap.c:64
#2 0x000000000087168a in LocalExecuteInvalidationMessage (msg=0x2b9699795768) at inval.c:595
#3 0x000000000071d50e in ReorderBufferExecuteInvalidations (rb=0x195a358, txn=0x1a6ff38, txn=0x1a6ff38) at reorderbuffer.c:2238
#4 ReorderBufferCommit (rb=0x195a358, xid=xid@entry=328684387, commit_lsn=8016890875224, end_lsn=<optimized out>, commit_time=commit_time@entry=799851538975691, origin_id=origin_id@entry=0, origin_lsn=origin_lsn@entry=0) at reorderbuffer.c:1819

## Completed within 20 seconds

Even without commenting out pg_pathman from shared_preload_libraries, there was a dramatic improvement — walsender went from 2 hours to 20 seconds.

This seemed odd at first — without commenting shared_preload_libraries, the hook should still run. Source analysis revealed the reason: the very first step of the hook checks for the pathman config table; if it doesn’t exist, it skips pathman’s invalidation logic entirely, so execution completes quickly:

/*
 * Invalidate PartRelationInfo cache entry if needed.
 */
void
pathman_relcache_hook(Datum arg, Oid relid)
{
	Oid pathman_config_relid;

	/* See cook_partitioning_expression() */
	if (!pathman_hooks_enabled)
		return;

	if (!IsPathmanReady())
		return;

...

	/*
	 * Invalidation event for PATHMAN_CONFIG table (probably DROP EXTENSION).
	 * Digging catalogs here is expensive and probably illegal, so we take
	 * cached relid. It is possible that we don't know it atm (e.g. pathman
	 * was disabled). However, in this case caches must have been cleaned
	 * on disable, and there is no DROP-specific additional actions.
	 */
	pathman_config_relid = get_pathman_config_relid(true);
	if (relid == pathman_config_relid)
	{
		delay_pathman_shutdown();
	}

	/* Invalidation event for some user table */
	else if (relid >= FirstNormalObjectId)
	{
		/* Invalidate PartBoundInfo entry if needed */
		forget_bounds_of_rel(relid);

		/* Invalidate PartStatusInfo entry if needed */
		forget_status_of_relation(relid);

		/* Invalidate PartParentInfo entry if needed */
		forget_parent_of_partition(relid);
	}
}

get_pathman_config_relid fetches the pathman_config table. drop extension pg_pathman removes the pathman_config table from the database, so the source code never enters the forget_* logic.

There are other ways to accelerate walsender processing: setting pg_pathman.enable=off causes IsPathmanReady() to return false and bail out immediately. Or, most directly, comment out pg_pathman from shared_preload_libraries and restart the instance (this is instance-level, not database-level).

Improvements in PG14
#

PG14.0 release notes:

Allow logical decoding to more efficiently process cache invalidation messages (Dilip Kumar) This allows logical decoding to work efficiently in presence of a large amount of DDL.

https://www.postgresql.org/docs/release/14.0/

Patch:

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d7eb52d71

Comment from PG14’s ReorderBufferAddInvalidations:

We require to record it in form of the change so that we can execute only the required invalidations instead of executing all the invalidations on each CommandId increment.

Comparing PG14 vs PG13, ReorderBufferCommit underwent a major rewrite.

In PG13, transaction processing logic was directly in the ReorderBufferCommit function:

				case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
					Assert(change->data.command_id != InvalidCommandId);

					if (command_id < change->data.command_id)
					{
						command_id = change->data.command_id;

						if (!snapshot_now->copied)
						{
							/* we don't use the global one anymore */
							snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
																txn, command_id);
						}

						snapshot_now->curcid = command_id;

						TeardownHistoricSnapshot(false);
						SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);

						/*
						 * Every time the CommandId is incremented, we could
						 * see new catalog contents, so execute all
						 * invalidations.
						 */
						ReorderBufferExecuteInvalidations(rb, txn);
					}

In PG14, the main logic moved to ReorderBufferReplay -> ReorderBufferProcessTXN.

ReorderBufferProcessTXN introduced a new case REORDER_BUFFER_CHANGE_INVALIDATION branch to execute invalidations from the reorder buffer:

				case REORDER_BUFFER_CHANGE_INVALIDATION:
					/* Execute the invalidation messages locally */
					ReorderBufferExecuteInvalidations(
												 change->data.inval.ninvalidations,
												 change->data.inval.invalidations);
					break;

The logic after ReorderBufferExecuteInvalidations is largely the same. The main differences between PG13 and PG14’s ReorderBufferCommit:

ReorderBufferCommit is no longer the primary transaction processing function; the call stack is deeper
A new case REORDER_BUFFER_CHANGE_INVALIDATION branch was added, separated from REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, to handle invalidations independently
The per-command_id invalidation processing logic was removed

Root Cause and Solutions
#

The root cause of the walsender hang was a bulk GRANT operation that updated many rows in pg_class, triggering a massive number of invalidation messages. A statement like GRANT privs ON ALL TABLES IN SCHEMA public TO role1 executes as multiple commands within a single transaction in PostgreSQL. In PG13, logical replication processes invalidation messages per-command, invoking each hook’s inval hash table processing. In this scenario, pathman’s hook was particularly slow at processing the inval hash table, causing replication lag.

Conditions for pathman-induced slowness (all must apply):

PG13 or earlier
Bulk GRANT
pathman extension installed (whether used or not)
Logical replication slot active

Even after removing pathman, significant CPU time was still spent in functions like RelfilenodeMapInvalidateCallback. In PG13 testing, the processing time difference between with and without pathman was hours vs. minutes.

Other untested but community-mentioned scenarios (all must apply):

PG13 or earlier
Bulk DDL / TRUNCATE / DCL / DROP PUBLICATION
Logical replication slot active

Short-term fix: If pathman tables are not in use, drop the extension or unload the pathman shared library; restart the replication slot.

Long-term fix: Upgrade to PG14+ (tested — extremely fast with no lag).

#

References
#

https://www.postgresql.org/message-id/flat/17716-1fe42e7b44fc2f25%40postgresql.org

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d7eb52d71

PostgreSQL Ops Experience 2024

Wed, 08 Jan 2025 00:00:00 +0000

This article focuses on common PostgreSQL operations issues — rare edge cases that surface once every two or three years are out of scope.

It’s primarily a technical ops summary, aiming for clarity and quick applicability. Deep dives at the source-code level are deliberately avoided.

SQL Performance & Execution Plans
#

Sudden Execution Plan Changes
#

PostgreSQL does not support optimizer hints natively, and the community has made it clear it never will. The PG community’s stance is roughly: “Our optimizer is perfect. If the current plan isn’t good enough, it’s because the developer doesn’t understand optimization.”

Regardless of what the PG community thinks, sudden execution plan regressions happen all the time in production, and we don’t have the rich, native plan-binding mechanisms that Oracle provides. This is a real challenge for production operations. For example: one morning, a sensitive query suddenly changes its plan, runtime jumps from 0.1s to 1s, and due to some concurrency the database CPU gets hammered — the business notices immediately. Without plan-binding tools, our only two rapid recovery options are: 1) collect statistics, or 2) scale up CPU.

A question about rapid recovery: does collecting statistics always help? A good DBA can identify where the optimizer went wrong, but can’t instantly conjure up a complete correct plan — especially for complex queries. Collecting statistics essentially hands the optimization problem back to the optimizer, trusting it to get it right. While this sounds a bit shaky, in PostgreSQL it actually works most of the time. (For scenarios where collecting stats is known to be useless, see the “ORDER BY LIMIT Problem” section.)

Why do execution plans suddenly change and regress?

Plans are cost-based, costs rely on statistics, and statistics are always lagging
Sufficiently complex SQL has a huge number of possible execution paths, and the optimizer picks the lowest-cost one
PG exposes many optimizer parameters to tune for local hardware (e.g., seq_page_cost, effective_cache_size). These can nudge the optimizer’s preferences but are very low-level. While there’s theoretical tuning headroom, changing them has system-wide effects. After go-live, adjusting these is extremely high-risk. The very existence of these parameters hints that no plan can be 100% perfect, because the optimizer’s reasoning depends on its environment

Even mighty Oracle, with its arsenal of plan-stabilization features, can’t guarantee 100% problem-free SQL — because SQL, data, statistics, bind variables, etc. are all dynamic.

For PG users, we’re not there yet, but we can work on making plans more stable:

Don’t join too many tables. More tables mean more possible plans — to the point where PG GEQO stops enumerating all plans, reducing the chance of finding the optimal one
Don’t write overly complex SQL. Keep in mind SQL may come from ORM frameworks rather than hand-written queries. Framework-generated SQL is often optimized for a goal with little regard for brevity or readability, making it very hard to tune
Don’t create indexes indiscriminately — have a clear goal. Random indexes confuse the optimizer
Tune per-table statistics collection thresholds via autovacuum_analyze_scale_factor (see “Delayed Statistics Collection”)
Use pg_hint_plan to give the optimizer hints

pg_hint_plan
#

pg_hint_plan is a third-party extension that uses hints to guide the optimizer toward the correct plan.

What pg_hint_plan supports:

Specifying scan methods (e.g., index scan), join methods (NL/HASH/MERGE), join order, memoize, estimated row counts, parallelism, and GUC parameters
Binding hints to SQL via hint_plan.hints without modifying the application SQL text

pg_hint_plan limitations:

Usage restrictions with subqueries, foreign tables, CTEs, views, PL/pgSQL, etc.
compute_query_id treats hints as comments and ignores them
Unknown bugs

While this extension is actively maintained, I haven’t found large-scale production deployment cases yet. We’ve also encountered issues in limited production use where hints don’t take effect — possibly related to JDBC plan caching — but it’s hard to draw firm conclusions.

In short: pg_hint_plan is a good tool, but large-scale production deployment is still TBD. I recommend waiting and watching. You can trial it, but don’t become dependent on it.

Delayed Statistics Collection
#

Statistics are the foundation of SQL optimization. PG statistics aren’t particularly complex, but many people still don’t fully understand them.

The three key views for PG statistics: pg_class, pg_stat_all_tables, pg_stats

-- pg_class: pages and tuples
select relname,relpages,reltuples::bigint from pg_class where relname='lzlpg'\gx
-[ RECORD 1 ]------
relname | lzlpg
relpages | 187501
reltuples | 6000032

-- pg_stat_all_tables: live tuples, dead tuples, last analyze time
 select relname,n_live_tup,n_dead_tup,last_analyze,last_autoanalyze from pg_stat_all_tables where relname='lzlpg'\gx
-[ RECORD 1 ]----+------------------------------
relname | lzlpg
n_live_tup | 6000032
n_dead_tup | 0
last_analyze | 2025-01-04 15:54:44.553057+08
last_autoanalyze | [null]

-- pg_stats: per-column statistics — understand every field
 select * from pg_stats where tablename='lzlpg' and attname='a'\gx
-[ RECORD 1 ]----------+-------
schemaname | public
tablename | lzlpg
attname | a
inherited | f
null_frac | 0
avg_width | 70
n_distinct | -1
most_common_vals | [null]
most_common_freqs | [null]
histogram_bounds | [null]
correlation | [null]
most_common_elems | [null]
most_common_elem_freqs | [null]
elem_count_histogram | [null]

Stale statistics are very likely to cause execution plan changes and SQL performance issues. Check last_autovacuum and last_autoanalyze in pg_stat_all_tables to determine if collection is lagging.

Why tune it? Because the default autovacuum_analyze_scale_factor is 0.1, meaning statistics are only collected when data changes exceed 10%. For a 1-billion-row table, that’s 100 million rows — possibly far too infrequent.

Evaluate whether to tune per-table autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor based on: whether it’s a core business table, number of joins, query complexity, access frequency, month-boundary issues, data skew, etc. The goal: increase collection frequency to reduce plan-regression risk without wasting resources on excessive vacuuming.

What value should you set? An example:

For a monthly table (or monthly partition) with queries hitting the current day’s data: with autovacuum_analyze_scale_factor = 0.1, the table gets analyzed almost daily for the first ~10 days, but may skip analysis around day 12. At that point statistics can cross a boundary and plans may degrade. To ensure analysis continues through days 10–31 of the month, set autovacuum_analyze_scale_factor below 0.03. I recommend autovacuum_analyze_scale_factor = 0.02.

Parameter tuning reference (consider your table’s data model!):

Parameter	Default	Recommended
`autovacuum_vacuum_scale_factor`	`0.2`	`0.04`
`autovacuum_analyze_scale_factor`	`0.1`	`0.02`

The Optimizer May Choose a Non-Primary-Key Index
#

Intuitively, a primary key should have the best selectivity, but the optimizer may still choose something else.

-- Reproduction commands
create table t1(a char(1000) primary key,b char(1000));
insert into t1 select md5(g::text),md5(g::text) from generate_series(1,10000) g;
create index idxa on t1(a);
create index idxb on t1(b);
analyze t1;

explain (analyze,buffers) select * from t1 where a='qwer' and b='qwer';
explain (analyze,buffers) select * from t1 where a='qwer' and b||''='qwer';

-- Columns a and b have identical selectivity, but the optimizer picks the regular index, not the PK
explain (analyze,buffers) select * from t1 where a='qwer' and b='qwer';
 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Index Scan using idxb on t1 (cost=0.41..5.43 rows=1 width=2008) (actual time=0.045..0.046 rows=0 loops=1)
 Index Cond: (b = 'qwer'::bpchar)
 Filter: (a = 'qwer'::bpchar)
 Buffers: shared hit=3

-- Force the PK path — cost is only marginally higher
explain (analyze,buffers) select * from t1 where a='qwer' and b||''='qwer';
 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Index Scan using idxa on t1 (cost=0.41..5.44 rows=1 width=2008) (actual time=0.079..0.079 rows=0 loops=1)
 Index Cond: (a = 'qwer'::bpchar)`
 Filter: (((b)::text || ''::text) = 'qwer'::text)
 Buffers: shared read=3

Even though columns a and b have the same type and selectivity, the optimizer picks the regular index over the PK. The PK path costs 0.01 more.

Why does this matter?

With the current data distribution, picking the regular index is harmless. But once data changes, the two index plans can diverge dramatically:

alter table t1 set (autovacuum_enabled ='off');
insert into t1 select md5(g::text),'repeat' from generate_series(20001,30000) g;

-- b='repeat' has terrible selectivity, but the b index is still chosen
explain (analyze,buffers) select * from t1 where a='qwer' and b='repeat';
 QUERY PLAN
--------------------------------------------------------------------------------------------------------------
 Index Scan using idxb on t1 (cost=0.41..5.43 rows=1 width=2008) (actual time=15.823..15.824 rows=0 loops=1)
 Index Cond: (b = 'repeat'::bpchar)
 Filter: (a = 'qwer'::bpchar)
 Rows Removed by Filter: 10000
 Buffers: shared hit=2511

-- Compare with the PK plan
 explain (analyze,buffers) select * from t1 where a='qwer' and b||''='repeat';
 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Index Scan using idxa on t1 (cost=0.41..5.44 rows=1 width=2008) (actual time=0.041..0.041 rows=0 loops=1)
 Index Cond: (a = 'qwer'::bpchar)
 Filter: (((b)::text || ''::text) = 'repeat'::text)
 Buffers: shared hit=3

Even with poor real selectivity, the optimizer sticks with the regular index — but efficiency is far worse (shared hit=2511 vs. shared hit=3). For latency-sensitive queries or larger data volumes, this becomes a real production problem.

Solutions:

Manually collect statistics; increase collection frequency
Use pg_hint_plan
Rewrite the SQL to prevent it from using the regular index

The ORDER BY LIMIT Problem
#

ORDER BY with LIMIT is a well-known issue with plenty of write-ups and case studies online (see my post ORDER BY LIMIT 10 Is Slower Than ORDER BY LIMIT 100).

The root cause: the optimizer currently can’t estimate where data sits in the table relative to the index order. If matching rows happen to be near the end of the table, the scan reads far more data than expected before returning the LIMIT rows. Note this isn’t limited to ORDER BY + LIMIT — any operation involving sorted output + LIMIT can hit it: GROUP BY + LIMIT, DISTINCT + LIMIT, merge joins, etc.

Solutions:

Rewrite the SQL: add an expression to prevent using the sort-column index (including PK), e.g., order by ''||col1 limit xxx
Create a composite index: a composite index on (sort_column + index_column) may be chosen by the optimizer and is generally more efficient than an index on the sort column alone. This approach doesn’t require changing the SQL

Table Bloat
#

Something Blocking Dead Tuple Cleanup
#

Putting aside autovacuum configuration issues and edge cases, the common blockers are:

Long-running transactions. Note: a long transaction on a different table also blocks dead-tuple reclamation. Read-only queries cause this too.
Replication slots. Lagging or defunct replication slots cause this.

Both are relatively easy to solve: 1) terminate the long-transaction session, 2) drop the replication slot, or have the consumer analyze why consumption is so slow.

High-Concurrency UPDATE Causing Table Bloat
#

Unlike something blocking vacuum, this is about dead tuples being generated faster than vacuum can clean them up. Typically, such tables show high pg_stat_all_tables.n_tup_upd. If table bloat requires repack, assess whether write volume is high enough to make repeated manual repack a losing game. In that case, tune the table/index fillfactor.

For the underlying principles, see this post From Painfully Slow Unique Index Scans to Index Bloat. I’ll summarize the conclusions here:

fillfactor basics:

fillfactor acts as a high-water mark for tables or indexes. During INSERT, once a page reaches its fillfactor line, new rows go to the next page. The purpose is to reserve space for UPDATEs so they don’t constantly seek new pages.

While both tables and indexes have fillfactor with the same goal (accommodating UPDATEs), the details differ significantly:

Tables: If a page still has free space, an UPDATE can stay within the same page — no new page needed, no need to find another page with space. More importantly, thanks to PG’s HOT (Heap-Only Tuple) feature, in-page updates don’t touch indexes, naturally slowing index bloat
Indexes: Different rows or out-of-page updates of the same row generate new index entries. Reserving space in index pages via fillfactor greatly reduces index page splits

Of course, fillfactor settings are tightly coupled with the workload. If data is append-only like logs with zero updates, fillfactor=100 for both tables and indexes is perfectly fine. But most business tables see updates, so fillfactor shouldn’t be 100. With frequent UPDATEs, it should be even lower.

Yet PG’s defaults are:

Table default: fillfactor=100
Index default: fillfactor=90

Recommended settings:

alter table lzlpg set (fillfactor=60);
alter index lzlpg_pkey set (fillfactor=70);
-- These commands only affect new pages; existing pages need repack

-- Repack:
1. Check for long transactions; resolve them first
2. nohup pg_repack -d lzldb --table lzlpg -p 6666 -no-kill-backend > pgrepack_lzlpg_log.log 2>&1 &

Long Transaction Problems
#

Long transactions don’t have a huge amount of theory behind them — monitor and handle promptly — but they absolutely deserve their own section.

Long transactions cause many problems:

Unreleased locks → application blocking
WAL not recycled → disk alerts
Dead tuples not cleaned → SQL performance degradation
Various other bizarre performance issues linked to long transactions
…

Long transactions in PostgreSQL are far more damaging than in Oracle or MySQL. They must be strictly managed.

Subtransaction Problems
#

“Subtransactions are basically cursed. Rip em out.”

Subtransactions cause many problems and are a frequent pain point in the industry.

Industry experience reports:

Waiting for Postgres 17: Configurable SLRU cache sizes for increased performance

Subtransactions-overflow-and-the-performance-cliff

Why we spent the last month eliminating PostgreSQL subtransactions

Where subtransactions come from:

PL/pgSQL functions containing a block with an exception clause
savepoints
JDBC + autosave=always (default autosave=never)
ODBC

Note: OGG uses an ODBC driver, and ODBC cannot disable subtransactions.

GaussDB’s ODBC can disable subtransactions via ForExtensionConnector.

So we can advise applications to keep subtransactions under 64, but we can’t easily advise against using OGG, since migrating off Oracle often depends on OGG-based data sync tools.

Subtransaction problem scenarios and symptoms:

1(+) long transaction + subtransaction overflow + high concurrency → severe performance drop
Subtransaction overflow (64+) → noticeable performance dip
Subtransaction overflow (64+) + multixact → severe performance drop
1(+) long transaction + 1(+) subtransaction → severe query performance drop on read replicas

Improvements in PG17:

SLRU manages transaction relationships for clog, multixact, subtrans, etc. in shared memory. Relevant source definitions:

/* Number of SLRU buffers to use for subtrans */
#define NUM_SUBTRANS_BUFFERS	32 // 32 SLRU pages in shared memory

/*
 * Each backend advertises up to PGPROC_MAX_CACHED_SUBXIDS TransactionIds
 * for non-aborted subtransactions of its current top transaction. These
 * have to be treated as running XIDs by other backends.
 *
 * We also keep track of whether the cache overflowed (ie, the transaction has
 * generated at least one subtransaction that didn't fit in the cache).
 * If none of the caches have overflowed, we can assume that an XID that's not
 * listed anywhere in the PGPROC array is not a running transaction. Else we
 * have to look at pg_subtrans.
 */
#define PGPROC_MAX_CACHED_SUBXIDS 64	// Overflow at 64+, per backend

PG17 SLRU improvements: New GUC parameter to configure SLRU slot count; split the existing single centralized SLRU lock into multiple bank locks.

Improvement effect:

(https://www.pgevents.ca/events/pgconfdev2024/sessions/session/53/slides/27/SLRU%20Performance%20Issues.pdf)

Subtransaction handling strategies:

Dev standards: Don’t use savepoints; consider ON CONFLICT for write conflicts
Dev standards: Don’t use exception blocks
Dev standards: Ensure JDBC does not have autosave=always enabled
Monitoring: Targeted monitoring of pg_stat_slru
Monitoring: Targeted monitoring of SAVEPOINT and EXCEPTION
CDC standards: Use ODBC (and OGG or other ODBC-based tools) with care; split transactions, cap subtransactions per large transaction at 50K
Upgrade: Move to PG17

Concurrency & Performance
#

Snapshot and Concurrency Parameter Tuning
#

Parameter	Type	Default	Recommended	Requires Restart
`old_snapshot_threshold`	cpu	-1 (community)	-1	Yes
`max_parallel_workers_per_gather`	cpu	2	0	No

old_snapshot_threshold easily causes performance problems when enabled — there’s plenty of material online. Even though it requires a restart, I strongly recommend keeping it disabled.

max_parallel_workers_per_gather auto-enables parallelism for large queries, but parallelism of 2 won’t give a proportional 2x speedup. This parameter is best used in specific scenarios, like explicitly setting parallel workers for batch jobs. Since no restart is needed, it’s a quick change.

Will disabling old_snapshot_threshold cause problems?

No. This parameter exists to limit long transactions — which do damage performance in PG — but the parameter itself causes performance issues, defeating the purpose.

Long transactions can be handled via several mechanisms:

Long transaction monitoring. This is the most important, and monitoring is fairly mature.
Set statement_timeout (default 0)
Set transaction_timeout (default 0, available in PG17+)
Set lock_timeout (default 0; recommended at session level for DDL)
Set idle_in_transaction_session_timeout (default 0; we set it to 2h)
Set idle_session_timeout (default 0; not relevant here)

High-Concurrency Commits Causing LWLOCK:WALWrite
#

Case Study: Intermittent Slow INSERT … VALUES

Key takeaways:

There’s only one IO:WALWrite, but there can be dozens of LWLOCK:WALWrite waiters
You can’t directly see the LWLOCK blocking chain, but from the source code we know LWLOCK:WALWrite is waiting on IO:WALWrite
In high-concurrency small-transaction scenarios, increasing WAL buffer size theoretically doesn’t help much

What problems does this cause?

Concurrent writes block, write latency increases, active sessions may spike
High-concurrency small transactions can’t saturate disk IO

Solutions:

Distribute concurrent writes across time
Batch commits at the application level
Analyze and try to reduce FPI (see FPI section)
Group commit (TBD)

WAL & Latency
#

FPI and Checkpoint Parameters
#

PG generates WAL FPI (Full Page Images) the first time a page is touched after a checkpoint. So more frequent checkpoints → higher probability of FPI.

Checkpoint frequency is controlled by two parameters:

checkpoint_timeout
max_wal_size

Principle:

(Egor Rogov, PostgreSQL 14 Internals)

max_wal_size defaults to 1GB, which is too small for high-load databases. Generally, you should increase this parameter to reduce FPI.

checkpoint_timeout defaults to 5 minutes, which seems reasonable.

FPI and Random Writes
#

Even with longer checkpoint intervals, FPI problems may persist. Check whether the workload involves UUID-based random writes. You may need to switch to sequences or another UUID scheme.

Finding the specific index:

Check if FPI is severe

--stats=record is handy

pg_waldump -z --stats=record 00000001000001860000001B

Sort which relations have the most FPWs

pg_waldump 00000001000001860000001B|grep FPW|awk -F ':' '{print $7}'|awk '{print $2}'|sort -n|uniq -c |sort -r|head -10

Logical Replication & Replication Slots
#

Logical replication has many issues and is a key optimization area for the community — nearly every major version brings significant improvements.

Logical Replication and Replication Slots Basics

Spill Problem
#

Analysis of PG Startup Logic and Spill-Induced Slow Startup

Spill key takeaways:

Spill occurs when logical decoding can’t fit transaction data in memory, so it writes to disk. Spill files contain transaction information
Each walsender has independent decoding, so each logical replication subscriber has its own spill
Large transactions produce large spill files, typically few in number
Subtransaction spill produces one spill file per subtransaction

Versions:

PG12 and earlier: hard-coded 4096 changes
PG13 added logical_decoding_work_mem to adjust memory and reduce spill probability
PG14+ supports streaming replication
Streaming also requires certain conditions to trigger, so even with streaming, spilling can still occur
PG17 added debug_logical_replication_streaming to force streaming

WALSender Blocking Shutdown
#

PG Shutdown Logic and WALSender Blocking Shutdown Analysis

In reality, any process that doesn’t exit can block shutdown. The question is which ones are most likely to cause trouble. From the shutdown code flow, archiver and walsender are frequent blockers because during shutdown they attempt a final archive or log transmission.

If shutdown is stuck on walsender, try kill (not kill -9) — the checkpoint hasn’t finished yet, and a forced shutdown leaves an inconsistent state. Even for forced shutdown, prefer pg_ctl stop -D $PGDATA -m i over raw kill -9
If shutdown is stuck on archiver, kill -9 is fine — the checkpoint is already complete and the database is in a consistent state

Partitioned Tables
#

Partitioned Table Basics

PG’s partitioned tables have unique characteristics that developers generally don’t fully understand without study, leading to many pitfalls.

Index Mismatch Between Parent and Child Partitions
#

Due to non-standard partition creation, many indexes are created directly on child tables (which should not be done), and the “create index on all children + attach” workflow is skipped. The result: the parent table has no index or no effective index. Since the parent has no data, this doesn’t directly impact queries — but when new partitions are created, they only inherit the parent’s indexes, so new child tables end up missing indexes.

Fixing parent-table missing indexes is fairly straightforward: see The Correct Way to Create Partition Indexes

-- Create an invalid index ONLY on the parent. Fast, but blocks subsequent DML — watch for long transactions
CREATE INDEX IDX_DATECREATED ON ONLY lzlpartition1(date_created);
-- Create the index CONCURRENTLY on each child partition. Slow, but doesn't block DML — watch for long DML transactions that could cause the operation to fail
create index concurrently idx_datecreated_202302 on lzlpartition1_202302(date_created);
-- Attach all indexes. Fast, no business blocking
 ALTER INDEX idx_datecreated ATTACH PARTITION idx_datecreated_202302;

Fixing a missing primary key on the parent is harder: see Adding Primary Keys and Unique Indexes to Partitioned Tables

Adding a primary key on the parent acquires AccessExclusiveLock, blocking everything. Creating an index on a partitioned table is slow, and the PK then causes further blocking. There’s currently no low-impact way to add a PK on a partitioned table. Workarounds: “attach a unique index + NOT NULL constraint”, schedule extended downtime for the partition table while the index builds, or use a third-party sync tool to populate a new table that already has the PK.

Abusing the DEFAULT Partition
#

Default Partition Overgrowth Causing Prolonged Blocking During CREATE TABLE ... PARTITION OF

The root cause is simple: when adding a new partition, the DDL must validate that data in the DEFAULT partition doesn’t conflict with the new partition’s range. This scans a large amount of data in the DEFAULT partition, and the new partition creation never completes. Blocking then cascades — business queries and writes stall.

DEFAULT partition abuse is a widespread problem! The community PG doesn’t provide interval partitioning. If a developer forgets to create a partition, data silently lands in DEFAULT with no error or alert. Day after day, the DEFAULT partition grows enormous — and then the next schema change causes an outage.

You can’t leave an oversized DEFAULT partition as-is forever. Even though ATTACH can avoid the blocking problem, you still need to defuse this bomb eventually.

DEFAULT partition data handling — Plan 1:

DETACH the default partition, create proper partitions, then re-insert DEFAULT data into the partitioned table
If needed, after detach and creating proper partitions, create an empty DEFAULT partition to maintain business continuity
Note: DETACH (unlike ATTACH) requires an AccessExclusiveLock on the parent. PG14 supports DETACH CONCURRENTLY, but not for DEFAULT partitions

DEFAULT partition data handling — Plan 2:

DETACH the default partition, create proper partitions, then ATTACH the detached DEFAULT table as a regular child partition — careful with range boundaries
If needed, after detach and creating proper partitions, create an empty DEFAULT partition to maintain business continuity
Note: DETACH (unlike ATTACH) requires an AccessExclusiveLock on the parent. PG14 supports DETACH CONCURRENTLY, but not for DEFAULT partitions

DEFAULT partition data handling — Plan 3:

Create a new table, sync all data via DTS
Rename tables

Plan 3 looks the crudest, but it’s the one I personally recommend most. If you have 5 instances to fix, a surgical approach is fine. If you have 200 instances, the labor cost makes DTS the practical winner.

Missing SELECT Privileges on Partitions Causing Abnormal Plans
#

If a user lacks SELECT privilege on a child partition, their queries can’t access that partition’s statistics, leading to bad execution plans. Partitions created via CREATE TABLE ... PARTITION OF normally don’t carry SELECT grants — but data is accessible through the parent — so this is a widespread issue.

Solutions:

Have the cloud platform handle it automatically
Enforce dev standards requiring SELECT grants on child partitions

High-Concurrency Full Partition Scans and LWLock:lockmanager
#

This is another very common problem!

I recommend reading the AWS documentation, which explains it clearly: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/wait-event.lw-lock-manager.html

Symptoms:

Spiking active sessions
Severe LWLock:lockmanager wait events
Database performance cliff

Trigger conditions:

Query scans multiple partitions
That query has high concurrency

Key takeaways:

The fastpath lock mechanism is designed for quick access to “weak locks”, improving database concurrency
fastpath works for lock levels ≤ 3 — i.e., SELECT, SELECT FOR xxx, and DML (lock modes below ShareUpdateExclusiveLock — levels 1, 2, 3 can use fastpath). It’s meant to benefit normal operations
FP_LOCK_SLOTS_PER_BACKEND: a local process holds at most 16 fastpath locks; beyond that, it must acquire locks in shared memory, producing LWLock:lockmanager
Not just tables — every accessed index also acquires a lock
This problem isn’t tightly coupled to partition count — even a modest number of partitions can trigger LWLock:lockmanager and degrade performance

Let’s calculate: with a partitioned table having 1 primary key and 2 regular indexes, how many partitions exhaust the fastpath?

16 / (3 indexes + 1 table) - 1 parent = 3 child partitions

Yes — a full scan across just 3 partitions can already trigger LWLock:lockmanager waits.

For a regular table, 16 indexes would similarly exhaust fastpath.

Solutions:

For not-too-large tables, merge partitions into a regular table
Add partition key filter conditions to queries
Reduce indexes (not very practical, since partition count alone can exceed 16)

The hard part:

In Oracle-to-PG migrations, Oracle supports global indexes, so primary keys and unique indexes don’t need to include the partition key. In PG, they must include the partition key.

PK example:

idxlzl(primarykey) --oracle
idxlzl(primarykey,partitionkey) --pg

A common query pattern:

select col from tlzl where primarykey=12345;

Should you push the application to add a partition filter here? It’s a tough sell. The resistance is: “I already passed the primary key — what more do you want? If I knew everything, why would I query the database?”

In this case, the only recommendation is to convert the partitioned table to a regular table. I haven’t found a better solution.

Memory
#

Excessive Objects Leading to Oversized relcache
#

Key takeaways:

relcache stores relation metadata: OID, pg_class info, partitions, subtransactions, row-level security policies, statistics, index metadata, access methods, etc.
Each session has its own (rel)cache for system catalog data (metadata, etc.)
Normally this cache is small. When the catalog is huge and a session accesses all of it, the cache can become very large
Cache management is simple: no eviction mechanism, no limit (though there are invalidation messages)
Closing the session releases the cache

Solutions:

Reduce the number of objects — especially check whether partition child tables are excessive
Set aggressive connection-pool disconnection parameters so business connections recycle more frequently

Memory Fragmentation
#

Recommended commands:

cat /proc/meminfo|grep whatyouneed
cat /proc/buddyinfo

## cgroup memory
/opt/cgtools/cginfo -t perf -s mem

# Pay attention to pgscand/s (direct memory reclaim) — values in the tens of thousands indicate a problem
sar -B -s "08:00:00" -e "09:00:00"
# min_free_kbytes setting:
cat /proc/sys/vm/min_free_kbytes
# Total physical memory usage of all processes:
grep Pss /proc/[1-9]*/smaps | awk '{total+=$2}; END {printf "%d kB\n", total }'
# PSS memory for a specific process:
cat /proc/90875/smaps |grep Pss |awk '{sum+=$2 };END {print sum/1024}'
# RSS memory for a specific process:
cat /proc/68729/smaps |grep Rss |awk '{sum+=$2 };END {print sum/1024}'
# Private memory for a specific process:
cat /proc/90875/smaps|sed '/zero/,/VmFlags/d' |grep Private |awk '{sum+=$2 };END {print sum/1024}'

min_free_kbytes:

(https://vivani.net/2022/06/14/linux-kernel-tuning-page-allocation-failure/)

When free memory is low, the kswapd daemon is woken to free pages:

pages_low: when free pages fall below pages_low, buddy allocator wakes kswapd and the kernel begins swapping pages to disk
pages_min: when free pages reach pages_min, reclamation pressure is high — the zone urgently needs free pages. The allocator performs synchronous kswapd work, sometimes called direct reclaim
pages_high: once kswapd is awake and freeing pages, the kernel considers the zone “balanced” only when free pages reach pages_high. At pages_high, kswapd goes back to sleep. Free pages above pages_high means the zone is in an ideal state

vm.min_free_kbytes (the pages_min watermark) is an extremely important OS parameter. Too low a value prevents effective memory reclamation, potentially causing system crashes and service interruptions. Too high a value increases reclaim activity, causing allocation delays that can immediately trigger OOM.

Optimization results:

After increasing min_free_kbytes + deploying off-peak drop-cache jobs, problems have decreased significantly.

Why increase min_free_kbytes?

This is used to force the Linux VM to keep a minimum number of kilobytes free. The VM uses this number to compute a watermark[WMARK_MIN] value for each lowmem zone in the system. Each lowmem zone gets a number of reserved free pages based proportionally on its size.

Source: kernel.org docs

The point of raising min_free_kbytes isn’t to raise the min watermark and trigger direct reclaim more often — it’s because the low watermark couldn’t be tuned before Linux 7. The only way to raise low proportionally was to raise min, making asynchronous reclaim trigger earlier and giving direct reclaim a buffer window.

Red Hat 8 added two memory parameters to improve reclaim: watermark_scale_factor can raise watermarks without touching min_free_kbytes.

Recommend enabling huge pages:

Huge pages perform better when PG requests contiguous memory
Huge pages also help reduce page cache size
shared_buffers can use huge pages; requires Huge_pages=on and OS-level huge pages enabled
Instances with huge pages enabled in production show better performance and fewer problems
AWS huge pages standard: enabled by default for all instance classes except certain test tiers, and cannot be disabled

Huge_pages parameter is turned on by default for all DB instance classes other than t3.medium, db.t3.large, db.t4g.medium, db.t4g.large instance classes. You can’t change the huge_pages parameter value or turn off this feature in the supported instance classes of Aurora PostgreSQL.

cgroup and Host Memory Mismatch
#

When cgroup memory hits its limit, kswapd prioritizes reclaiming pages within the cgroup. With cloud VM instance types and cgroup configurations, the host may have free memory above watermarks while the cgroup is under pressure. The host-level pages_low doesn’t trigger asynchronous reclaim for either host or cgroup memory. Eventually, direct reclaim fires to satisfy the cgroup’s DB memory demand.

The root cause: cgroups lack independent free-page memory management.

The only fix: increase the cgroup memory limit, overcommitting the host more aggressively so the host reaches pages_low sooner.

shared_buffer and pagecache
#

PG uses a double-buffer mechanism — no direct IO yet.

Double buffer: DB shared_buffers (one layer of shared memory) + OS pagecache (another layer). In real deployments, pagecache is typically far larger than shared_buffers. And pagecache counts against cgroup mem but isn’t reflected in cgroup memory monitoring…

Bottom line: leave plenty of memory for pagecache. Don’t make shared_buffers excessively large (20GB seems sufficient for most cases). Only increase it if you clearly observe buffer-mapping-related wait events.

work_mem Cannot Cap Hash Join / Hash Aggregate Memory
#

hash_mem_multiplier limits memory for hash-based operations (hash join, hash agg, etc.), capping at hash_mem_multiplier * work_mem. The default is 2.

Before PG13, work_mem was tunable, but there was no way to limit how many hash operations a single query could use. PG13 added this multiplier. In other words, pre-13, it was very hard to cap hash-table memory.

In a PG12- production environment, I found a single session consuming 300GB of memory — the culprit was the lack of hash-table limits combined with a plan that incorrectly used hash tables.

Other Issues
#

Exclusive Backup and Startup Issues
#

Normally, when the database stops and restarts, the startup position comes from pg_controldata’s LSN. But if there’s a backup_label file in PGDATA, the startup LSN is read from backup_label.

What problems does this cause?

Disk snapshots taken directly on the data directory may include the label file. If the database is large and the backup took a long time, restart can be very slow
Bigger problem: after a production shutdown from certain causes, restart takes forever. The root cause is the startup LSN coming from the backup rather than controldata

Version changes:

PG13:

pg_start_backup() pg_stop_backup()

Supports exclusive and non-exclusive modes; exclusive is the default. Exclusive mode creates backup_label in the data directory at start and cleans it at stop. Non-exclusive mode doesn’t create the label at start; it returns the label info at stop.

PG15:

pg_backup_start() pg_backup_stop()

Function names changed, and exclusive backup mode was removed. No backup_label is written at backup start; instead it’s written to the backup area at backup stop.

pg_stat_activity Unqueryable
#

Symptom:

pg_stat_activity hangs and can’t be queried.

pstack at the time:

#0 pgstat_read_current_status () at pgstat.c:3642
#1 0x0000000000727181 in pgstat_read_current_status () at pgstat.c:2788
#2 pgstat_fetch_stat_numbackends () at pgstat.c:2789
#3 0x000000000083f2ee in pg_stat_get_activity (fcinfo=0x25c2d98) at pgstatfuncs.c:575
#4 0x000000000065058f in ExecMakeTableFunctionResult (setexpr=0x25b1d28, econtext=0x25b1c48, argContext=<optimized out>, expectedDesc=0x2545218, randomAccess=false) at execSRF.c:234
#5 0x00000000006609dc in FunctionNext (node=node@entry=0x25b1b38) at nodeFunctionscan.c:94
#6 0x000000000065110c in ExecScanFetch (recheckMtd=0x660700 <FunctionRecheck>, accessMtd=0x660720 <FunctionNext>, node=0x25b1b38) at execScan.c:133

Analysis:

The code location is clear — stuck in an infinite loop after st_changecount becomes odd.

Triggers: OOM (reproducible), abnormal backend exit (possible), terminate (maybe). None of these guarantee the issue, though.

Community thread didn’t reach a conclusion. Currently the trigger probability appears low.

Solution: restart the database.

Connection and Connection Pooling Issues
#

IO Error Messages
#

IO errors typically mean the application is using a connection that’s already been closed. This happens often, and diagnosing it is difficult because the entire chain involves many components and broad domain knowledge. Here’s a brief summary.

Known active-disconnection scenarios:

hikari maxLifetime

Symptom: session lifetime matches the parameter. Possible cause: the application holds an explicit transaction with an uncommitted SELECT, the pool closes the session, and the app gets io error; could not rollback or similar.

pg.datasouce.maxLifetime

druid timeout

Symptom: connection drops after SQL execution exceeds 20s.

spring.datasource.dynamic.druid.socketTimeout=20000
spring.datasource.dynamic.druid.connectTimeout=20000
Change to:
spring.datasource.socketTimeout=3600000
spring.datasource.connectTimeout=3600000

Application Horizontal Scaling vs. Database Connection Limits
#

Horizontal application scaling meets PG connection bottlenecks:

HikariCP is now Spring Boot’s default connection pool. With the proliferation of Spring Boot and microservices, HikariCP usage is widespread. Every pod scaled out increases database connection count. The maximumPoolSize stays the same per pod, but more nodes mean more total connections. From existing node count, added node count, and current total connections, you can proportionally calculate how many idle connections will be added.

Applications can scale horizontally without state, but databases cannot. PG’s connection limit is max_connections. Unchecked application scaling can saturate idle connections. Tuning max_connections is painful because it requires a database restart.

PG connection upper limit:

Also, even with unlimited horizontal scaling, max_connections should adjust with instance class — but there’s a real ceiling. In any database, idle connections degrade performance as they increase.

Refer to AWS’s approach: max_connections is tied to instance class, with a maximum of 5000, LEAST({DBInstanceClassMemory/9531392}, 5000). This reduces manual connection ops and provides a reasonable ceiling.

PG Shutdown Logic and Walsender Blocking Shutdown Analysis

Sat, 04 Jan 2025 00:00:00 +0000

Walsender Blocking Shutdown Symptoms
#

Production shutdown log output:

2024-12-06 17:00:02.036 CST,,,447560,,65693cde.6d448,1320,,2023-12-01 09:54:38 CST,,0,LOG,00000,"received fast shutdown request",,,,,,,,,"","postmaster"
2024-12-06 17:00:02.295 CST,,,447560,,65693cde.6d448,1322,,2023-12-01 09:54:38 CST,,0,LOG,00000,"background worker ""logical replication launcher"" (PID 448996) exited with exit code 1",,,,,,,,,"","postmaster"
2024-12-06 17:00:10.627 CST,,,448990,,65693ce0.6d9de,213833,,2023-12-01 09:54:40 CST,,0,LOG,00000,"checkpoint complete: wrote 426844 buffers (5.1%); 0 WAL file(s) added, 0 removed, 5 recycled; write=91.427 s, sync=0.055 s, total=91.508 s; sync files=761, longest=0.028 s, average=0.001 s; distance=2197531 kB, estimate=2680783 kB",,,,,,,,,"","checkpointer"
2024-12-06 17:00:10.628 CST,,,448990,,65693ce0.6d9de,213834,,2023-12-01 09:54:40 CST,,0,LOG,00000,"shutting down",,,,,,,,,"","checkpointer"
...
--checkpointer finished checkpoint and is in shutting down state, pm has not exited

--160s later pm receives immediate shutdown, triggered by health check script
2024-12-06 17:02:43.348 CST,,,447560,,65693cde.6d448,1323,,2023-12-01 09:54:38 CST,,0,LOG,00000,"received immediate shutdown request",,,,,,,,,"","postmaster"
2024-12-06 17:02:43.370 CST,"logicaluser","lzldb",283840,"10.33.77.159:39865",6751a2dc.454c0,7,"idle",2024-12-05 20:55:56 CST,89/847309655,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"
2024-12-06 17:02:43.370 CST,"logicaluser","lzldb",157641,"10.33.77.159:39407",67408354.267c9,7,"idle",2024-11-22 21:12:52 CST,9/3193590104,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"
2024-12-06 17:02:43.370 CST,"logicaluser","lzldb",157916,"10.33.77.159:57038",67408356.268dc,7,"idle",2024-11-22 21:12:54 CST,115/3293293502,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"
2024-12-06 17:02:43.370 CST,"repuser","",164392,"30.151.40.19:41641",66b25869.28228,3,"streaming 42D3B/1732C5F0",2024-08-07 01:07:53 CST,296/0,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"standby_6666","walsender"
2024-12-06 17:02:43.371 CST,,,447560,,65693cde.6d448,1324,,2023-12-01 09:54:38 CST,,0,LOG,00000,"archiver process (PID 448994) exited with exit code 2",,,,,,,,,"","postmaster"
2024-12-06 17:02:43.371 CST,"logicaluser","lzldb",57755,"10.33.77.159:38918",67125534.e19b,7,"idle",2024-10-18 20:31:48 CST,243/902018192,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"
2024-12-06 17:02:43.372 CST,"logicaluser","lzldb",157915,"10.33.77.159:43433",67408356.268db,7,"idle",2024-11-22 21:12:54 CST,60/3248014863,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"


--pm finished shutting down
2024-12-06 17:02:57.534 CST,,,447560,,65693cde.6d448,1325,,2023-12-01 09:54:38 CST,,0,LOG,00000,"database system is shut down",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.536 CST,,,211844,,6752bdf3.33b84,1,,2024-12-06 17:03:47 CST,,0,LOG,00000,"ending log output to stderr",,"Future log output will go to log destination ""csvlog"".",,,,,,,"","postmaster"

17:00:02 postmaster receives fast shutdown

17:00:10 checkpoint completed, checkpointer stopped

17:02:43 postmaster receives immediate shutdown

17:02:43 1 physical and 5 logical replication walsenders stopped

17:02:57 postmaster stopped

17:03:49 postmaster receives startup task

From the above, it’s clear that walsender was blocking the shutdown.

Shutdown and Signals
#

Before diving into source code, we need to understand signals and signal registration in PG.

Common Signals in PG
#

OS signals:

$ kill -l
 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
...

Common signals used in PG:

-1 or -SIGHUP: Hangup signal. In PG, typically tells the process to reload configuration.
-2 or -SIGINT: Interrupt signal (usually Ctrl+C). In PG, usually corresponds to cancel command.
-3 or -SIGQUIT: In PG, usually means forced exit (die).
-9 or -SIGKILL: Unconditional termination signal.
-15 or -SIGTERM: Termination signal, the signal used by pg_terminate_backend. In PG, usually means graceful exit.
-10 or -SIGUSR1: Custom signal.
-12 or -SIGUSR2: Custom signal.
-17 or SIGCHLD: Signal used by the pm process. When a child process exits, pm receives this signal to trigger child process reaping.

The specific meaning of signals registered by each type of PG process can be found by reading the respective process source code.

Shutdown Defined by pg_ctl
#

There are several ways to shut down a PG database. At the bottom level, they all boil down to sending a signal to the postmaster process.

signal	pg_ctl	Meaning
`SIGTERM`	Smart Shutdown	Disallow new connections, but allow existing sessions to finish their work normally. Only shuts down after all sessions terminate.
`SIGINT`	Fast Shutdown	Server disallows new connections and sends SIGTERM to all existing child processes, aborting current transactions and exiting quickly. Waits for almost all child processes (some are not needed) to exit, then shuts down.
`SIGQUIT`	Immediate Shutdown	Sends SIGQUIT to all child processes and waits for them to terminate. If any child process has not terminated within 5 seconds, they are sent SIGKILL.

Note: pg_ctl has no parameter for sending SIGKILL (kill -9), but you can send SIGKILL directly to pm — though it’s definitely not recommended. When sending SIGKILL to pm, pm won’t do any cleanup of child processes, shared memory, or semaphores. Since SIGQUIT to pm has fallback logic for SIGKILL-ing child processes, SIGQUIT to pm basically guarantees pm will stop.

In the source code, there are only 3 shutdown states, corresponding to shutdown modes:

/* Startup/shutdown state */
#define			NoShutdown		0
#define			SmartShutdown	1
#define			FastShutdown	2
#define			ImmediateShutdown	3

These states appear frequently in shutdown routine source code, generally checked via the Shutdown variable:

Shutdown >= FastShutdown

pm Signals
#

When pm receives the corresponding signal, it handles it accordingly:

void
PostmasterMain(int argc, char *argv[])
{...
	pqsignal_pm(SIGHUP, SIGHUP_handler);	/* reread config file and have
											 * children do same */
	pqsignal_pm(SIGINT, pmdie); /* send SIGTERM and shut down */
	pqsignal_pm(SIGQUIT, pmdie);	/* send SIGQUIT and die */
	pqsignal_pm(SIGTERM, pmdie);	/* wait for children and shut down */
	pqsignal_pm(SIGALRM, SIG_IGN);	/* ignored */
	pqsignal_pm(SIGPIPE, SIG_IGN);	/* ignored */
	pqsignal_pm(SIGUSR1, sigusr1_handler);	/* message from child process */
	pqsignal_pm(SIGUSR2, dummy_handler);	/* unused, reserve for children */
	pqsignal_pm(SIGCHLD, reaper);	/* handle child termination */

pmdie: The three shutdown signals call the pmdie function. pmdie is the key shutdown function, analyzed in detail below.
reaper: During shutdown, handles child process exit cleanup. When a child process exits, it sends SIGCHLD to pm, which enters reaper to clean up the child. Each child process cleanup has its own logic — for instance, normal exit of the checkpointer process checks whether archiver and walsender have completed their respective tasks.
sigusr1, sigusr2: sigusr1_handler is the standard routine for SIGUSR1. Each child process handles SIGUSR1 differently. SIGUSR2 is entirely custom per child process; some child processes don’t even register this signal.

Walsender Signals
#

When a child process is forked, it first registers signals.

WalSndSignals registers signals for the walsender process:

/* Set up signal handlers */
void
WalSndSignals(void)
{
	/* Set up signal handlers */
	pqsignal(SIGHUP, SignalHandlerForConfigReload);
	pqsignal(SIGINT, StatementCancelHandler);	/* query cancel */
	pqsignal(SIGTERM, die);		/* request shutdown */
	pqsignal(SIGQUIT, quickdie);	/* hard crash time */
	InitializeTimeouts();		/* establishes SIGALRM handler */
	pqsignal(SIGPIPE, SIG_IGN);
	pqsignal(SIGUSR1, procsignal_sigusr1_handler);
	pqsignal(SIGUSR2, WalSndLastCycleHandler);	/* request a last cycle and
												 * shutdown */
}

Note SIGUSR1 and SIGUSR2.

Checkpointer Signals
#

CheckpointerMain registers checkpointer signals:

void
CheckpointerMain(void)
{
...
 //checkpointer blocks SIGTERM, the actual stop signal is SIGUSR2
	pqsignal(SIGHUP, SignalHandlerForConfigReload);
	pqsignal(SIGINT, ReqCheckpointHandler); /* request checkpoint */
	pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
	pqsignal(SIGQUIT, SignalHandlerForCrashExit);
	pqsignal(SIGALRM, SIG_IGN);
	pqsignal(SIGPIPE, SIG_IGN);
	pqsignal(SIGUSR1, procsignal_sigusr1_handler);
	pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);

Note SIGUSR1 and SIGUSR2, and also note that checkpointer does not register SIGTERM.

Shutdown Source Code Analysis
#

pm Signal Handling and State Machine
#

The pmdie function handles different postmaster signals, including SIGCHLD sent by child processes to pm and shutdown signals sent by pg_ctl. The main logic of pm signal handling is converting the signal into a pmState state machine state transition, then entering PostmasterStateMachine for processing.

pmdie:

/*
 * pmdie -- signal handler for processing various postmaster signals.
 */
static void
pmdie(SIGNAL_ARGS)
{
	int			save_errno = errno;

...

	switch (postgres_signal_arg)
	{
		case SIGTERM://Smart Shutdown
			...
			if (pmState == PM_RUN)
				connsAllowed = ALLOW_SUPERUSER_CONNS;
			...
 //smart shutdown does not process pmstate, hands directly to state machine
			//at this point normal pmState = PM_RUN
			PostmasterStateMachine();
			break;

		case SIGINT://Fast Shutdown
			...
			else if (pmState == PM_RUN ||
					 pmState == PM_HOT_STANDBY)
			{
				/* Report that we're about to zap live client sessions */
				ereport(LOG,
						(errmsg("aborting any active transactions")));
				pmState = PM_STOP_BACKENDS;
			}

			//Fast Shutdown transitions pmstate to PM_STOP_BACKENDS
 //then hands to state machine
			PostmasterStateMachine();
			break;

		case SIGQUIT://Immediate Shutdown
			...

			TerminateChildren(SIGQUIT);//abort all children with SIGQUIT, wait for them to exit
			pmState = PM_WAIT_BACKENDS;

			/* set stopwatch for them to die */
			AbortStartTime = time(NULL);

 //Immediate Shutdown transitions pmstate to PM_WAIT_BACKENDS
 //process children before entering state machine
 //first interrupt children with SIGQUIT, wait for them to exit
 //then use SIGKILL on remaining children
 //finally non-consistent exit
			PostmasterStateMachine();
			break;
	}
...
}

Before entering the state machine handler, let’s look at the postmaster states:

typedef enum
{
	PM_INIT,					/* postmaster starting */
	PM_STARTUP,					/* waiting for startup subprocess */
	PM_RECOVERY,				/* in archive recovery mode */
	PM_HOT_STANDBY,				/* in hot standby mode */
	PM_RUN,						/* normal "database is alive" state */
	PM_STOP_BACKENDS,			/* need to stop remaining backends */
	PM_WAIT_BACKENDS,			/* waiting for live backends to exit */
	PM_SHUTDOWN,				/* waiting for checkpointer to do shutdown
								 * ckpt */
	PM_SHUTDOWN_2,				/* waiting for archiver and walsenders to
								 * finish */
	PM_WAIT_DEAD_END,			/* waiting for dead_end children to exit */
	PM_NO_CHILDREN				/* all important children have exited */
} PMState;

Since shutdown normally happens from the running state, we only need to focus on states at PM_RUN and below.

PostmasterStateMachine execution has a sequential logic:

/*
 * Advance the postmaster's state machine and take actions as appropriate
 *
 * This is common code for pmdie(), reaper() and sigusr1_handler(), which
 * receive the signals that might mean we need to change state.
 */
static void
PostmasterStateMachine(void)
{
	//smart shutdown, pmState should be PM_RUN at this point
	if (pmState == PM_RUN || pmState == PM_HOT_STANDBY)
	{
	...

		if (connsAllowed == ALLOW_NO_CONNS)
		{
 //After all normal backends exit, transition pmState to PM_STOP_BACKENDS
			if (CountChildren(BACKEND_TYPE_NORMAL) == 0)
				pmState = PM_STOP_BACKENDS;
		}
	}

 //PM_STOP_BACKENDS stops some core child processes, some will continue running
 //autovacuum, bgwriter, walwriter, startup, walreceiver will stop
 //walsender, checkpointer, archiver, stats, and syslogger will keep running
 //smart shutdown later phase enters this logic, fast shutdown enters directly
	if (pmState == PM_STOP_BACKENDS)
	{
		...		
		//Note this line about walsender!
		/* Signal all backend children except walsenders */
		SignalSomeChildren(SIGTERM,
						 BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND);
		/* and the autovac launcher too */
		if (AutoVacPID != 0)
			signal_child(AutoVacPID, SIGTERM);
		/* and the bgwriter too */
		if (BgWriterPID != 0)
			signal_child(BgWriterPID, SIGTERM);
		/* and the walwriter too */
		if (WalWriterPID != 0)
			signal_child(WalWriterPID, SIGTERM);
		/* If we're in recovery, also stop startup and walreceiver procs */
		if (StartupPID != 0)
			signal_child(StartupPID, SIGTERM);
		if (WalReceiverPID != 0)
			signal_child(WalReceiverPID, SIGTERM);
		/* checkpointer, archiver, stats, and syslogger may continue for now */

 //Transition pmState from PM_STOP_BACKENDS to PM_WAIT_BACKEND
 //PM_WAIT_BACKEND means waiting for backends to exit
		pmState = PM_WAIT_BACKENDS;
	}

	/*
	 * If we are in a state-machine state that implies waiting for backends to
	 * exit, see if they're all gone, and change state if so.
	 */
 //
 //smart shutdown, fast shutdown later phase enters this logic
 //immediate shutdown when entering state machine, directly enters this logic
	if (pmState == PM_WAIT_BACKENDS)
	{
 //During crash recovery and immediate shutdown, checkpointer needs proper exit
 //archiver, stats, and syslogger don't need handling since they don't touch shared memory
 //Walsenders also don't need handling; they exit after checkpoint record is written, just like archiver
		if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
			StartupPID == 0 &&
			WalReceiverPID == 0 &&
			BgWriterPID == 0 &&
			(CheckpointerPID == 0 ||
			 (!FatalError && Shutdown < ImmediateShutdown)) &&
			WalWriterPID == 0 &&
			AutoVacPID == 0)
		{
			if (Shutdown >= ImmediateShutdown || FatalError)
			{
				//ImmediateShutdown waits for dead end processes to finish
				pmState = PM_WAIT_DEAD_END;

				/*
				 * We already SIGQUIT'd the archiver and stats processes, if
				 * any, when we started immediate shutdown or entered
				 * FatalError state.
				 */
			}
			else
			{
 //smart, fast shutdown goes here
 //regular child processes have all exited, now notify checkpointer to do shutdown checkpoint
				Assert(Shutdown > NoShutdown);
				//If checkpointer process doesn't exist, start one
				if (CheckpointerPID == 0)
					CheckpointerPID = StartCheckpointer();
				/* And tell it to shut down */
				if (CheckpointerPID != 0)
				{
 //Send SIGUSR2 to Checkpointer
 //pmState = PM_SHUTDOWN
					signal_child(CheckpointerPID, SIGUSR2);
					pmState = PM_SHUTDOWN;
				}
				else
				{
 //Failing to start Checkpointer is a serious problem
					FatalError = true;
					pmState = PM_WAIT_DEAD_END;

					/* Kill the walsenders, archiver and stats collector too */ 
 //Comment says kill walsender, but it actually doesn't; at least not via SIGQUIT
					SignalChildren(SIGQUIT);
					if (PgArchPID != 0)
						signal_child(PgArchPID, SIGQUIT);
					if (PgStatPID != 0)
						signal_child(PgStatPID, SIGQUIT);
				}
			}
		}
	}
	
 //The pmdie function and state machine function won't create PM_SHUTDOWN_2 state, but reaper will
 //When reaper handles checkpointer exit, it sets pmState = PM_SHUTDOWN_2; at the end of reaper, it enters the state machine function, which is here
	if (pmState == PM_SHUTDOWN_2)
	{
		/*
		 * PM_SHUTDOWN_2 state ends when there's no other children than
		 * dead_end children left. There shouldn't be any regular backends
		 * left by now anyway; what we're really waiting for is walsenders and
		 * archiver.
		 */
 //PM_SHUTDOWN_2 essentially waits for walsender and archiver
 //only changes pmState
		if (PgArchPID == 0 && CountChildren(BACKEND_TYPE_ALL) == 0)
		{
			pmState = PM_WAIT_DEAD_END;
		}
	}

	if (pmState == PM_WAIT_DEAD_END)
	{
 //PM_WAIT_DEAD_END means BackendList is completely empty
		if (dlist_is_empty(&BackendList) &&
			PgArchPID == 0 && PgStatPID == 0)
		{
			/* These other guys should be dead already */
			Assert(StartupPID == 0);
			Assert(WalReceiverPID == 0);
			Assert(BgWriterPID == 0);
			Assert(CheckpointerPID == 0);
			Assert(WalWriterPID == 0);
			Assert(AutoVacPID == 0);
			/* syslogger is not considered here */
			pmState = PM_NO_CHILDREN;
		}
	}

 //PM_NO_CHILDREN is the last shutdown state, meaning normal shutdown can proceed
	if (Shutdown > NoShutdown && pmState == PM_NO_CHILDREN)
	{
		if (FatalError)
		{
			ereport(LOG, (errmsg("abnormal database system shutdown")));
 //Abnormal pm exit
			ExitPostmaster(1);
		}
		...

		 	//Normal pm exit
			ExitPostmaster(0);
		}
	}
	...
}

reaper is the process reaping function. When a child process exits, it sends SIGCHLD to pm, and pm cleans up the process via the reaper function. Each process type — backend, startup, checkpointer, etc. — has its own cleanup flow.

Here we only look at checkpointer cleanup. Also, reaper has no cleanup logic for walsender:

		if (pid == CheckpointerPID)
		{
			CheckpointerPID = 0;
 //Checkpointer exited normally, and pmState is PM_SHUTDOWN: waiting for checkpoint completion
			if (EXIT_STATUS_0(exitstatus) && pmState == PM_SHUTDOWN)
			{
				/*
				 * OK, we saw normal exit of the checkpointer after it's been
				 * told to shut down. We expect that it wrote a shutdown
				 * checkpoint. (If for some reason it didn't, recovery will
				 * occur on next postmaster start.)
				 *
				 * At this point we should have no normal backend children
				 * left (else we'd not be in PM_SHUTDOWN state) but we might
				 * have dead_end children to wait for.
				 *
				 * If we have an archiver subprocess, tell it to do a last
				 * archive cycle and quit. Likewise, if we have walsender
				 * processes, tell them to send any remaining WAL and quit.
				 */
				Assert(Shutdown > NoShutdown);

				//Wake archiver for the last time
				if (PgArchPID != 0)
					signal_child(PgArchPID, SIGUSR2); //pgarch SIGUSR2=pgarch_waken_stop

 //Wake walsender for the last time
				SignalChildren(SIGUSR2);//walsender SIGUSR2=WalSndLastCycleHandler
 
 //Here PM_SHUTDOWN_2 is set
 //At this point Checkpointer has exited normally; we should wait for pgarch and walsender to finish their last task
 //This is PM_SHUTDOWN_2 state
 pmState = PM_SHUTDOWN_2;

				...
			}
			else
			{
 //checkpointer abnormal exit is considered a crash
				HandleChildCrash(pid, exitstatus,
								 _("checkpointer process"));
			}

			continue;
		}
...
	//At the end reaper still enters the state machine function
	PostmasterStateMachine();

...
}

Checkpointer and Walsender Process Exit
#

Checkpointer main loop handling requests and shutdown:

void
CheckpointerMain(void)
{
	/*
	 * Loop forever
	 */
	for (;;)
	{
		bool		do_checkpoint = false;
		int			flags = 0;
		pg_time_t	now;
		int			elapsed_secs;
		int			cur_timeout;

		/* Clear any already-pending wakeups */
		ResetLatch(MyLatch);

		/*
		 * Process any requests or signals received recently.
		 */
 //Process recent sync requests and signals
		AbsorbSyncRequests();
		HandleCheckpointerInterrupts();

Checkpointer shutdown function:

/*
 * Process any new interrupts.
 */
static void
HandleCheckpointerInterrupts(void)
{
 ...
	if (ShutdownRequestPending)
	{
		/*
		 * From here on, elog(ERROR) should end with exit(1), not send control
		 * back to the sigsetjmp block above
		 */
		ExitOnAnyError = true;
		
		ShutdownXLOG(0, 0);//This writes the shutdown checkpoint
		
		proc_exit(0);//Normal exit code 0
	}
}

Checkpointer exit needs to wait for ShutdownXLOG to complete.

ShutdownXLOG:

/*
 * This must be called ONCE during postmaster or standalone-backend shutdown
 */
void
ShutdownXLOG(int code, Datum arg)
{
...

	//Here's the checkpointer "shutting down" log, usually always seen
	ereport(IsPostmasterEnvironment ? LOG : NOTICE,
			(errmsg("shutting down")));

	/*
	 * Signal walsenders to move to stopping state.
	 */
 //Initialize walsender stopping
	WalSndInitStopping();

 //Wait for all walsenders to be in stopping state
	WalSndWaitStopping();

	if (RecoveryInProgress())
		CreateRestartPoint(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_IMMEDIATE);
	else
	{
		/*
		 * If archiving is enabled, rotate the last XLOG file so that all the
		 * remaining records are archived (postmaster wakes up the archiver
		 * process one more time at the end of shutdown). The checkpoint
		 * record will go to the next XLOG file and won't be archived (yet).
		 */
		if (XLogArchivingActive() && XLogArchiveCommandSet())
			RequestXLogSwitch(false);
		//This is the shutdown checkpoint creation function
		CreateCheckPoint(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_IMMEDIATE);
	}
	ShutdownCLOG();
	ShutdownCommitTs();
	ShutdownSUBTRANS();
	ShutdownMultiXact();
}

Checkpointer notifies all walsenders to begin stopping:

/*
 * Signal all walsenders to move to stopping state.
 *
 * This will trigger walsenders to move to a state where no further WAL can be
 * generated. See this file's header for details.
 */
void
WalSndInitStopping(void)
{
	int			i;

	for (i = 0; i < max_wal_senders; i++)
	{
		WalSnd	 *walsnd = &WalSndCtl->walsnds[i];
		pid_t		pid;

		SpinLockAcquire(&walsnd->mutex);
		pid = walsnd->pid;
		SpinLockRelease(&walsnd->mutex);

		if (pid == 0)
			continue;

		SendProcSignal(pid, PROCSIG_WALSND_INIT_STOPPING, InvalidBackendId);
	}
}

Walsender receives the signal via the SendProcSignal function, with signal SIGUSR1:

/*
 * SendProcSignal
 *		Send a signal to a Postgres process
 *
 * Providing backendId is optional, but it will speed up the operation.
 *
 * On success (a signal was sent), zero is returned.
 * On error, -1 is returned, and errno is set (typically to ESRCH or EPERM).
 *
 * Not to be confused with ProcSendSignal
 */
int
SendProcSignal(pid_t pid, ProcSignalReason reason, BackendId backendId)
{
	else
	{
		/*
		 * BackendId not provided, so search the array using pid. We search
		 * the array back to front so as to reduce search overhead. Passing
		 * InvalidBackendId means that the target is most likely an auxiliary
		 * process, which will have a slot near the end of the array.
		 */
		int			i;

		for (i = NumProcSignalSlots - 1; i >= 0; i--)
		{
			slot = &ProcSignal->psh_slot[i];

			if (slot->pss_pid == pid)
			{
				/* the above note about race conditions applies here too */

				/* Atomically set the proper flag */
				slot->pss_signalFlags[reason] = true;
				/* Send signal */
				return kill(pid, SIGUSR1);
			}
		}
	}

	errno = ESRCH;
	return -1;
}

Walsender’s SIGUSR1 registration:

	pqsignal(SIGUSR1, procsignal_sigusr1_handler);
	pqsignal(SIGUSR2, WalSndLastCycleHandler);	/* request a last cycle and
												 * shutdown */

sigusr1 classifies handling by signal reason:

/*
 * procsignal_sigusr1_handler - handle SIGUSR1 signal.
 */
void
procsignal_sigusr1_handler(SIGNAL_ARGS)
{
...
	if (CheckProcSignal(PROCSIG_WALSND_INIT_STOPPING))
		HandleWalSndInitStopping();
...
}

The handler for PROCSIG_WALSND_INIT_STOPPING is HandleWalSndInitStopping:

/*
 * Handle PROCSIG_WALSND_INIT_STOPPING signal.
 */
void
HandleWalSndInitStopping(void)
{
	Assert(am_walsender);

	/*
	 * If replication has not yet started, die like with SIGTERM. If
	 * replication is active, only set a flag and wake up the main loop. It
	 * will send any outstanding WAL, wait for it to be replicated to the
	 * standby, and then exit gracefully.
	 */
	if (!replication_active)
		kill(MyProcPid, SIGTERM);
	else
		got_STOPPING = true;//If walsender is active, initstopping just sets a flag for the main loop to handle
}

The “main loop” mentioned in the comment is somewhat ambiguous. Walsender has a main loop ServerLoop, but in reality only the loop in WalSndWaitForWal has checks for got_STOPPING.

The WalSndWaitForWal function is the main loop for walsender waiting for new WAL records. Since WAL records are initially generated in memory, walwriter flushes them based on certain conditions, not all the time. WalSndWaitForWal compares the currently sent LSN with the flushed LSN to determine whether new WAL needs to be sent. In other words, unflushed WAL is not transmitted; only flushed WAL is passed downstream.

WalSndWaitForWal code segment about stopping:

/*
 * Wait till WAL < loc is flushed to disk so it can be safely sent to client.
 *
 * Returns end LSN of flushed WAL. Normally this will be >= loc, but
 * if we detect a shutdown request (either from postmaster or client)
 * we will return early, so caller must always check.
 */
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
 ...
	for (;;)
	{
 ...
 //After receiving got_STOPPING, do one flush of WAL
 //This is necessary! Because walwriter may have already shut down at this point, WAL may not be flushed yet
		if (got_STOPPING)
			XLogBackgroundFlush();

		/* Update our idea of the currently flushed position. */
		if (!RecoveryInProgress())
			RecentFlushPtr = GetFlushRecPtr();
		else
			RecentFlushPtr = GetXLogReplayRecPtr(NULL);

 //Break out of the for loop
 //After getting new RecentFlushPtr, still need to send
		if (got_STOPPING)
			break;
 ...
	}
	/* reactivate latch so WalSndLoop knows to continue */
	SetLatch(MyLatch);
	return RecentFlushPtr;
}

Back to walsender main loop: WalSndLoop(XLogSendLogical):

/* Main loop of walsender process that streams the WAL over Copy messages. */
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
...
	for (;;)
	{
		/* Clear any already-pending wakeups */
		ResetLatch(MyLatch);
		...

		//Process replies from downstream
		ProcessRepliesIfAny();

		/*
		 * If we have received CopyDone from the client, sent CopyDone
		 * ourselves, and the output buffer is empty, it's time to exit
		 * streaming.
		 */
 //Exit loop when streaming is done
		if (streamingDoneReceiving && streamingDoneSending &&
			!pq_is_send_pending())
			break;

 //If output buffer has pending data, send it
		if (!pq_is_send_pending())
			send_data();
		else
			WalSndCaughtUp = false;

		/* Try to flush pending output to the client */
		if (pq_flush_if_writable() != 0)
			WalSndShutdown();//Downstream not writable, downstream closed, normal walsender shutdown, exit code 0

		/* If nothing remains to be sent right now ... */
		if (WalSndCaughtUp && !pq_is_send_pending())
		{
			/*
			 * If we're in catchup state, move to streaming. This is an
			 * important state change for users to know about, since before
			 * this point data loss might occur if the primary dies and we
			 * need to failover to the standby. The state change is also
			 * important for synchronous replication, since commits that
			 * started to wait at that point might wait for some time.
			 */
 //Data transmission is done, but commit info still needs to be sent
			if (MyWalSnd->state == WALSNDSTATE_CATCHUP)
			{
				ereport(DEBUG1,
						(errmsg("\"%s\" has now caught up with upstream server",
								application_name)));
				WalSndSetState(WALSNDSTATE_STREAMING);
			}

 //Received SIGUSR2, meaning shutdown checkpoint is done.
 //Send the shutdown checkpoint record, wait for completion, then exit
			if (got_SIGUSR2)
				WalSndDone(send_data);//exit code 0
		}
		
		...
	}
}

Let’s return to checkpointer’s ShutdownXLOG logic. The above only analyzed WalSndInitStopping(). After this signal is sent to walsender, WalSndWaitStopping executes to wait for walsender.

As long as any walsender hasn’t exited, this is an infinite loop that won’t return:

/*
 * Wait that all the WAL senders have quit or reached the stopping state. This
 * is used by the checkpointer to control when the shutdown checkpoint can
 * safely be performed.
 */
void
WalSndWaitStopping(void)
{
	for (;;)
	{
		int			i;
		bool		all_stopped = true;

		for (i = 0; i < max_wal_senders; i++)
		{
			WalSnd	 *walsnd = &WalSndCtl->walsnds[i];

			SpinLockAcquire(&walsnd->mutex);

			if (walsnd->pid == 0)
			{
				SpinLockRelease(&walsnd->mutex);
				continue;
			}

			if (walsnd->state != WALSNDSTATE_STOPPING)
			{
				all_stopped = false;
				SpinLockRelease(&walsnd->mutex);
				break;
			}
			SpinLockRelease(&walsnd->mutex);
		}

		/* safe to leave if confirmation is done for all WAL senders */
		if (all_stopped)
			return;

		pg_usleep(10000L);		/* wait for 10 msec */
	}
}

Finally, combined with the comments in walsender.c:

 * If the server is shut down, checkpointer sends us
 * PROCSIG_WALSND_INIT_STOPPING after all regular backends have exited. If
 * the backend is idle or runs an SQL query this causes the backend to
 * shutdown, if logical replication is in progress all existing WAL records
 * are processed followed by a shutdown. Otherwise this causes the walsender
 * to switch to the "stopping" state. In this state, the walsender will reject
 * any further replication commands. The checkpointer begins the shutdown
 * checkpoint once all walsenders are confirmed as stopping. When the shutdown
 * checkpoint finishes, the postmaster sends us SIGUSR2. This instructs
 * walsender to send any outstanding WAL, including the shutdown checkpoint
 * record, wait for it to be replicated to the standby, and then exit.

After all regular backends have exited, checkpointer sends PROCSIG_WALSND_INIT_STOPPING to walsenders
Walsender may enter the stopping state
Only after all walsenders enter stopping state does checkpointer perform the shutdown checkpoint
After the shutdown checkpoint completes, pm sends SIGUSR2 to walsender, which sends any remaining WAL including the shutdown checkpoint record itself, waits for standby to complete, then exits

Shutdown Flow Diagram
#

After going through the source code, it felt like I understood but also didn’t — needed a shutdown flowchart to clarify.

Summary of the fast shutdown flow:

(High resolution: https://www.processon.com/view/link/6778a73a04a8344b9502637a)

PG manages shutdown logic through signals, per-process main loops, PM state machine, and the pmdie process reaping function
Also note: signals themselves are asynchronous. If you need to wait for the result of signal processing in a target process, you typically need other synchronization mechanisms (pipes, semaphores, shared memory, etc.). PG mainly relies on process dependencies and whether processes exit normally to determine if signals were properly handled.
pgarch and walsender are treated as the same type of process, handled differently from others (walwriter, bgwriter). pgarch and walsender need to do an additional “last task”. The signal for the “last task” is typically defined as SIGUSR2.
Checkpointer’s normal exit depends on pgarch and walsender exiting normally.
pgarch’s last task is the final archive. So archiving can affect shutdown.
Walsender’s second-to-last task is delivering the final WAL, and its last task is delivering the checkpoint shutdown info. These tasks require downstream reply messages, so walsender can affect shutdown.

Test Reproduction
#

Test: Reproducing Walsender Blocking Shutdown
#

After fast stop shutdown, walsender can block the shutdown.

Tested various scenarios to reproduce walsender blocking shutdown. Currently, the following conditions together make it easier to trigger abnormal shutdown:

One walsender for publication/subscription
One walsender for DTS
Large number of subtransactions causing replication slot spill

This three-in-one scenario doesn’t represent the only scenario; it’s just one that was easier to reproduce after testing many.

--Reproduction commands (not extremely stable reproduction)
1.Create table
--pg
create table lzlpg(id bigserial primary key,a char(2000),b char(2000),c char(2000));
--oracle
create table lzl.lzloracle(id number primary key ,a char(2000),b char(2000),c char(2000)) tablespace FADATA;
2.Set up 2 logical replication links (1 pub/sub, 1 DTS to oracle)

3.Reduce logical_decoding_work_mem
logical_decoding_work_mem=1MB

4.Write large amounts of data (recommended: subtransaction spill)
--Insert one row at a time, each insert as a subtransaction
echo "begin;">subtx.sql
for i in {1..500000}
do
 echo "savepoint p$i;">>subtx.sql
 echo "insert into lzlpg(column1,column2,column3) select 'a','b','c';">>subtx.sql
done

nohup psql -d lzl -f subtx.sql &

5.Stop the database before writing completes
pg_ctl stop -D $PGDATA -m fast

At this point, with fast shutdown, the database is in an incomplete shutdown state:

~/lzl/slot]$ ps -axjf|grep 110402
150696 64964 64961 146782 pts/42 64961 S+ 6001 0:00 \_ grep --color=auto 110402
 1 110402 110402 110402 ? -1 Ss 6001 0:00 /myhost/postgres/base/rasesql1.5.6/bin/postgres -D /myhost/pg8094/data
110402 110599 110599 110599 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: logger 
110402 117803 117803 117803 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: checkpointer 
110402 117807 117807 117807 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: stats collector 
110402 118563 118563 118563 ? -1 Rs 6001 3:29 \_ postgres: lzlpg: walsender lzl 127.0.0.1(62971) idle
110402 222918 222918 222918 ? -1 Rs 6001 2:59 \_ postgres: lzlpg: walsender dtssync 30.181.46.203(57218) idle

Walsender, checkpointer, postmaster are all still there; logger and stats haven’t exited either.

The control file state is in production: meaning running in production, indicating the local shutdown checkpoint by checkpointer didn’t complete:

~/lzl/slot]$ pg_controldata|grep -i state
Database cluster state: in production

Checkpointer stack:

pstack 117803
#0 0x00002b879fe0b983 in __select_nocancel () from /lib64/libc.so.6
#1 0x00000000008fd04a in pg_usleep (microsec=microsec@entry=10000) at pgsleep.c:56
#2 0x00000000007610c8 in WalSndWaitStopping () at walsender.c:3209
#3 0x000000000051fa86 in ShutdownXLOG (code=code@entry=0, arg=arg@entry=0) at xlog.c:8596
#4 0x00000000007215ff in HandleCheckpointerInterrupts () at checkpointer.c:566
#5 CheckpointerMain () at checkpointer.c:343
...

At this point, checkpointer is stuck in WalSndWaitStopping, meaning checkpointer is waiting for walsender processes to enter stopping state.

Walsender stack at this point:

#0 0x00000000007484fb in ReorderBufferLargestTXN (rb=<optimized out>) at reorderbuffer.c:2345
#1 ReorderBufferCheckMemoryLimit (rb=0x2b8808b94118) at reorderbuffer.c:2390
#2 ReorderBufferQueueChange (rb=0x2b8808b94118, xid=<optimized out>, lsn=1676456602544, change=change@entry=0x2b87a229f408) at reorderbuffer.c:649
#3 0x000000000073ec99 in DecodeTruncate (buf=<optimized out>, buf=<optimized out>, ctx=<optimized out>) at decode.c:872
#4 DecodeHeapOp (buf=0x7ffda7d35180, ctx=0x2b87a224b118) at decode.c:455
#5 LogicalDecodingProcessRecord (ctx=0x2b87a224b118, record=<optimized out>) at decode.c:126
#6 0x000000000075f502 in XLogSendLogical () at walsender.c:2886
#7 0x0000000000761822 in WalSndLoop (send_data=send_data@entry=0x75f4c0 <XLogSendLogical>) at walsender.c:2287
...

Walsender is stuck in the transaction spill function. (Why it’s stuck is still unclear!!!)

Checkpointer process is blocked in WalSndWaitStopping:

/*
 * Wait that all the WAL senders have quit or reached the stopping state. This
 * is used by the checkpointer to control when the shutdown checkpoint can
 * safely be performed.
 */
void
WalSndWaitStopping(void)
{
	for (;;)
	{
		int			i;
		bool		all_stopped = true;

		for (i = 0; i < max_wal_senders; i++)
		{
			WalSnd	 *walsnd = &WalSndCtl->walsnds[i];

			SpinLockAcquire(&walsnd->mutex);

			if (walsnd->pid == 0)
			{
				SpinLockRelease(&walsnd->mutex);
				continue;
			}

			if (walsnd->state != WALSNDSTATE_STOPPING)
			{
				all_stopped = false;
				SpinLockRelease(&walsnd->mutex);
				break;
			}
			SpinLockRelease(&walsnd->mutex);
		}

		/* safe to leave if confirmation is done for all WAL senders */
		if (all_stopped)
			return;

		pg_usleep(10000L);		/* wait for 10 msec */
	}
}

From the code and stack, it’s clear the condition walsnd->state != WALSNDSTATE_STOPPING is hit, causing the infinite loop.

Test: Handling the Mid-Shutdown State
#

The above is an awkward mid-shutdown state. Besides kill -9, there are other better ways to achieve consistent shutdown:

Solution 1: Shut down the downstream process
Solution 2: Send SIGTERM to walsender

Solution 1 test:

When the downstream exits, walsender will also exit:

static void
ProcessRepliesIfAny(void)
{...
				/*
				 * 'X' means that the standby is closing down the socket.
				 */
			case 'X':
				proc_exit(0);

For pub/sub, execute the following on the subscriber side; even if the upstream is in mid-shutdown state, this will cause walsender to exit:

\c lzldb
alter SUBSCRIPTION sub_lzl disable;

However, this depends on the downstream’s own handling; we can’t always quickly shut down the downstream receiver process of DTS and other sync tools.

Solution 2 test:

Since walsender registers the SIGTERM signal, and the select pg_terminate_backend($walsender_pid) command run while the database is running also sends SIGTERM to walsender, theoretically just sending SIGTERM to walsender should handle this, without needing kill -9.

Command:

kill -SIGTERM 62834
#same as kill -15 62834
#same as kill 62834

After normal kill, pm and all other processes exit completely.

Check the control file and WAL log to confirm consistent shutdown:

pg_controldata database state changed from in production to shut down — consistent shutdown:

$ pg_controldata|grep -i state
Database cluster state: shut down

The last record in the WAL log is CHECKPOINT_SHUTDOWN:

pg_waldump 000000010000018600000012|tail -1
pg_waldump: fatal: error in WAL record at 186/915D7920: invalid record length at 186/915D7998: wanted 24, got 0
rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 186/915D7920, prev 186/915D78A8, desc: CHECKPOINT_SHUTDOWN redo 186/915D7920; tli 1; prev tli 1; fpw true; xid 0:13431045; oid 3808147; multi 3; offset 6; oldest xid 485 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 494/13431044; oldest running xid 0; shutdown

Test: Reproducing Only Primary Having CHECKPOINT_SHUTDOWN
#

A phenomenon in the production environment was that the local WAL had a shutdown checkpoint but the standby didn’t. In production, an immediate stop was performed during mid-shutdown, and then startup failed.

At the time, the last 2 WAL records on primary and standby looked something like:

#Primary WAL:
CHECKPOINT_ONLINE
CHECKPOINT_SHUTDOWN
#Standby WAL:
CHECKPOINT_ONLINE

Reproduction commands:

## 1. First reproduce walsender blocking shutdown
(skipped)
## 2. Check the last WAL record
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 188/307ABE00, prev 188/307ABDC8, desc: RUNNING_XACTS nextXid 13432445 latestCompletedXid 13432444 oldestRunningXid 13432445
## 3. pg_ctl stop -D $PGDATA -m i
## 4. Check last WAL record
Unchanged, same as 2
## 5. pg_ctl start -D $PGDATA
## 6. Check last two WAL records
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 188/307ABE00, prev 188/307ABDC8, desc: RUNNING_XACTS nextXid 13432445 latestCompletedXid 13432444 oldestRunningXid 13432445 #same as 2
rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 188/307ABE38, prev 188/307ABE00, desc: CHECKPOINT_SHUTDOWN redo 188/307ABE38; tli 1; prev tli 1; fpw true; xid 0:13432445; oid 3832732; multi 3; offset 6; oldest xid 485 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 494/13432444; oldest running xid 0; shutdown #CHECKPOINT_SHUTDOWN appears

From this reproduction, CHECKPOINT_SHUTDOWN is actually done during startup!

This matches the production sequence: 1. fast shutdown didn’t complete 2. immediate shutdown 3. startup failed.

Question 1: When during startup is CHECKPOINT_SHUTDOWN done?

Question 2: When is CHECKPOINT_ONLINE triggered? From reproduction appearances, occasionally fast shutdown results in the last WAL record being CHECKPOINT_ONLINE.

Question 1 analysis:

Doing a shutdown checkpoint at startup easily suggests the startup process. Since we’ve previously analyzed the startup process flow, we can directly locate the function StartupXLOG:

/*
 * This must be called ONCE during postmaster or standalone-backend startup
 */
void
StartupXLOG(void)
{...
	if (InRecovery) //Since it was a shutdown stop, instance recovery is needed
	{
		/*
		 * Perform a checkpoint to update all our recovery activity to disk.
		 *
		 * Note that we write a shutdown checkpoint rather than an on-line
		 * one. This is not particularly critical, but since we may be
		 * assigning a new TLI, using a shutdown checkpoint allows us to have
		 * the rule that TLI only changes in shutdown checkpoints, which
		 * allows some extra error checking in xlog_redo.
		 *
		 * In fast promotion, only create a lightweight end-of-recovery record
		 * instead of a full checkpoint. A checkpoint is requested later,
		 * after we're fully out of recovery mode and already accepting
		 * queries.
		 */
		if (bgwriterLaunched) //This if is clearly for standby streaming replication
		{...
		}
		else //Primary startup goes here
			CreateCheckPoint(CHECKPOINT_END_OF_RECOVERY | CHECKPOINT_IMMEDIATE);
	}

Doing a shutdown checkpoint is intentional, mainly for TLI logic code robustness
Whenever it’s not a consistent shutdown, a shutdown checkpoint is performed during startup

So, doing -m i forced shutdown and then starting up will also produce CHECKPOINT_SHUTDOWN — self-tested.

Question 2 analysis:

Tested multiple times, occasionally seen. Speculation: it just happened that before shutdown, checkpoint conditions were met and an online checkpoint was triggered — pure coincidence.

Considering that after a failed database shutdown, whether it’s a script, HA, or manual intervention, forced shutdown may be done, it’s recommended to do at least one checkpoint before shutdown.

Test: Impact of Archiving on Shutdown
#

While analyzing the shutdown code, I also found that after the checkpointer process exits, reaper for checkpointer sends SIGUSR2 to pgarch for its last archive and exit:

static void
reaper(SIGNAL_ARGS)
{...
		if (pid == CheckpointerPID)
		{
			CheckpointerPID = 0;
			if (EXIT_STATUS_0(exitstatus) && pmState == PM_SHUTDOWN)
			{...

				/* Waken archiver for the last time */
				if (PgArchPID != 0)
					signal_child(PgArchPID, SIGUSR2);
 ...
			}
...

And pm’s exit depends on all processes except syslogger having exited:

	if (pmState == PM_WAIT_DEAD_END)
	{
		if (dlist_is_empty(&BackendList) &&
			PgArchPID == 0 && PgStatPID == 0)
		{
			/* These other guys should be dead already */
			Assert(StartupPID == 0);
			Assert(WalReceiverPID == 0);
			Assert(BgWriterPID == 0);
			Assert(CheckpointerPID == 0);
			Assert(WalWriterPID == 0);
			Assert(AutoVacPID == 0);
			/* syslogger is not considered here */
			pmState = PM_NO_CHILDREN;
		}
	}

So in production, slow archiving was also found to affect shutdown.

Reproduction commands:

#Configure archiving
archive_mode = on
archive_command = '/bin/false ;sleep 1000'#Set archiving to always fail with sleep to bypass NUM_ARCHIVE_RETRIES logic

#Shutdown
pg_ctl stop -D $PGDATA -m fast

Processes after shutdown:

$ ps -axjf|grep 61470
88406 88405 68705 pts/48 88405 S+ 6001 0:00 \_ grep --color=auto 61470
61470 61470 61470 ? -1 Ss 6001 0:00 /myhost/postgres/base/rasesql1.5.6/bin/postgres -D /myhost/pg8094/data
61772 61772 61772 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: logger 
63880 63880 63880 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: archiver archiving 000000010000018800000007

Since the checkpointer here has already fully stopped, the database is in a consistent state, so using kill -9 on archiver is fine.

One-Sentence Summary
#

Q1: Why didn’t shutdown complete?

Walsender blocked shutdown. Checkpointer sent SIGUSR1 to walsender and infinitely waited for all walsender processes to enter stopping state; checkpointer got stuck at this step.

The shutdown eventually completed due to -m i forced shutdown.

Q2: Is there a graceful way to shut down from the mid-shutdown state caused by walsender blocking?

Yes. Send SIGTERM (i.e. kill, or kill -15, kill -SIGTERM) to all walsenders. Afterwards, checkpointer and postmaster will complete a clean shutdown.

Walsender registers the SIGTERM signal at startup, and testing shows no scenario where it can’t be handled.

SIGTERM is also the signal sent by pg_terminate_backend(pid), and it’s the command that should be executed to stop walsender during a standard shutdown.

Q3: Why did primary and standby differ by exactly one shutdown checkpoint?

3.1 Explanation for both primary and standby having CHECKPOINT_ONLINE:

The primary triggering CHECKPOINT_ONLINE was purely coincidental
Since the physical walsender was still there, this WAL record was transmitted to the standby

3.2 Explanation for only primary having CHECKPOINT_SHUTDOWN:

This CHECKPOINT_SHUTDOWN was done during primary startup
Since the primary hadn’t fully started, this WAL record wasn’t transmitted to the standby

Q4: Why does archiver block shutdown?

When reaping the checkpointer process, pm tells archiver to do one last archive, and pm depends on all processes except syslogger having exited. So if the last archive is slow or has issues, it blocks shutdown. Archive failure won’t — the archiver process exits quickly on failure.

Q5: Which processes can block shutdown?

Actually, any process not exiting can block shutdown. The question is which ones are more likely to cause trouble. From the shutdown code flow, archiver and walsender commonly block shutdown because they perform a last archive or log transmission during the shutdown phase.

References
#

https://www.postgresql.org/docs/current/server-shutdown.html https://wiki.postgresql.org/wiki/Signals postgres.c postmaster.c walsender.c xlog.c checkpointer.c startup.c pgarch.c

PG Startup Logic and Spill-Caused Slow Startup Analysis

Sat, 04 Jan 2025 00:00:00 +0000

Problem Symptom — Slow Startup
#

Version: PG 13.2

Database startup was slow. The startup process was reading spill files, and the filenames kept changing. Checking the spill files was also very slow — ls -l eventually showed 8 million spill files.

Why Tens of Millions of Spill Files?
#

WAL Segment and LSN Meaning
#

LSN
#

LSN is a 64-bit bigint. An LSN actually looks like 42D3B/1732C540 (hex). Before the slash / is the 32-bit logical log number, and after the / are 32 bits split into segment number + block number + intra-block offset. These 4 parts are:

32 bits	8 bits	11 bits	13 bits
Logical log number	Log segment number	Block number	Intra-block offset

Intra-block offset 8192 = 2^13

Block number = 16M (default WAL segment size) / 8192

WAL Segment
#

A WAL filename consists of 3 groups of hex digits.

Taking the 8k WAL file 0000000300042D3B00000002 as example:

32 bits	32 bits	32 bits
timeline	Logical log number	Log segment number
00000003	00042D3B	00000002

It can be seen that an LSN can locate a WAL filename and the offset position within the file.

Among these, the part before the LSN slash / is the logical log number, and the 8-bit log segment number after the slash / will be used below.

Spill Filename Conversion
#

Replication slot name: logical_ex2209_rep

Spill filename: xid-407989064-lsn-42D1E-20000000.spill

42D1E is not a complete LSN and cannot be directly used with pg_walfile_name to locate a WAL filename. 42D1E is a logical log number. If we directly filter WAL files containing 42D1E in the name, we find 16 WAL files.

Can we locate the WAL log segment number from the number 20000000 to pinpoint the exact file?

Spill filename generation:

/*
 * Given a replication slot, transaction ID and segment number, fill in the
 * corresponding spill file into 'path', which is a caller-owned buffer of size
 * at least MAXPGPATH.
 */
static void
ReorderBufferSerializedPath(char *path, ReplicationSlot *slot, TransactionId xid,
 XLogSegNo segno)
{
 XLogRecPtr recptr;

 XLogSegNoOffsetToRecPtr(segno, 0, wal_segment_size, recptr);

 snprintf(path, MAXPGPATH, "pg_replslot/%s/xid-%u-lsn-%X-%X.spill",
 NameStr(MyReplicationSlot->data.name),
 xid,
 (uint32) (recptr >> 32), (uint32) recptr);
}

The pg_replslot/%s and xid-%u-lsn parts are easy to understand — just the replication slot name and xid. The recptr needs a closer look at its definition:

/*
 * Pointer to a location in the XLOG. These pointers are 64 bits wide,
 * because we don't want them ever to overflow.
 */
typedef uint64 XLogRecPtr;

XLogSegNoOffsetToRecPtr calculates the LSN from the WAL log segment number, segment size, and offset:

#define XLogSegNoOffsetToRecPtr(segno, offset, wal_segsz_bytes, dest) \
 (dest) = (segno) * (wal_segsz_bytes) + (offset)

XLogRecPtr is the LSN! So:

(uint32) (recptr >> 32) takes the first 32 bits of LSN, (uint32) recptr) takes the last 32 bits.

The first 32 bits of LSN is what we saw as the first half of LSN, lsn-42D1E. The last 32 bits of LSN actually contain more information; here we only need the first few bits of the last 32 bits — the segment number.

Since the passed-in offset=0 and we also have segno, we don’t actually need the intra-block offset information to calculate the dest value. The real value of wal_segsz_bytes is 128M = 128*1024*1024. Converting the formula in XLogSegNoOffsetToRecPtr:

segno= dest/(128*1024*1024)
-- Convert hex 20000000
segno= x'20000000'::int/(128*1024*1024)
segno= 4

From this formula we can derive the log segment number segno, which lets us locate the WAL file number.

So the spill filename xid-407989064-lsn-42D1E-20000000.spill corresponds to the WAL file:

Logical log number=42D1E, segment number=04:

ls 42D1E*04
0000000200042D1E00000004

pg_waldump shows xid 407989064 inside.

In practice, the WAL size is also fixed after instance creation, i.e. (128*1024*1024) is a constant, so segno is absolutely correlated with (uint32) recptr, but not equal to it. This means that switching to a new WAL log file creates a new spill file.

Summary of spill file generation rules:

Same transaction id: if it spans multiple WAL files, it produces multiple spills. E.g., a large transaction without subtransactions spanning 3 WAL files produces 3 spill files.
Different transaction ids produce different spills. E.g., 10 million subtransactions produce 10 million spill files.

Spill filename structure xid-407989064-lsn-42D1E-20000000.spill:

xid	First 32 bits of LSN; i.e., WAL logical log number	Converted from WAL log segment number; not equal to segment number
xid-407989064	lsn-42D1E	20000000

## Recovered environment

[postgres]$ ll |head -100
total 40000276
-rw------- 1 postgres postgres 184 Dec 6 15:20 state
-rw------- 1 postgres postgres 196 Dec 6 13:25 xid-407989064-lsn-42D1E-0.spill
-rw------- 1 postgres postgres 208 Dec 6 13:25 xid-407989064-lsn-42D1E-20000000.spill
...
-rw------- 1 postgres postgres 540 Dec 6 16:44 xid-407989064-lsn-42D2A-D0000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989065-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989066-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989068-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989070-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989072-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989076-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989079-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989080-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989082-lsn-42D1D-C8000000.spill

[postgres@lzlhost /myhost/liuzhilong/pg_replslot/logical_ex9e15_rep]$ ll |awk '{print $9}'|awk -F '-' '{print $2}'|sort|uniq -c|wc -l
10000003
[postgres@lzlhost /myhost/liuzhilong/pg_replslot/logical_ex9e15_rep]$ ll |wc -l
10000070

So in the actual environment we saw 10,000,070 files, with 10,000,003 distinct xids among them — meaning 1 parent transaction spanning about 70 WAL files, with this parent transaction having 10 million subtransactions.

Replication Slot Spill Testing
#

--Pub/sub replication link setup
logical_decoding_work_mem = 64MB #pg_ctl reload
wal_segment_size =128 MB

--source
CREATE TABLE replication_table (
 id BIGSERIAL PRIMARY KEY,
 column1 char(2000),
 column2 char(2000),
 column3 char(2000)
);
create publication pub_test for table replication_table ;

--dest
CREATE TABLE replication_table (
 id BIGSERIAL PRIMARY KEY,
 column1 char(2000),
 column2 char(2000),
 column3 char(2000)
);

CREATE SUBSCRIPTION sub_test
CONNECTION 'host=127.0.0.1 port=8094 dbname=lzl user=lzl password=qwer'
PUBLICATION pub_test;

--source
select * from pg_replication_slots;

Large Transaction, No Subtransactions, Replicated Table Spill Test
#

--Create a large transaction, don't commit yet
begin;
insert into replication_table(column1,column2,column3) 
select 'a','b','c' from generate_series(1,1000000) g;

--Replication slot spill
 ll
total 331924
-rw------- 1 postgres postgres 184 Dec 9 20:22 state
-rw------- 1 postgres postgres 88226964 Dec 9 20:22 xid-5074343-lsn-163-38000000.spill
-rw------- 1 postgres postgres 119698488 Dec 9 20:22 xid-5074343-lsn-163-40000000.spill

After the large transaction commits, wait for consumption until replication lag is 0, and the spill files disappear:

M=# select pid,usename,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,reply_time from pg_stat_replication;
 pid | usename | sent_lsn | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | reply_time 
--------+---------+--------------+--------------+--------------+--------------+-----------+-----------+------------+------------------------------
 163525 | lzl | 163/4996E1C8 | 163/4996E1C8 | 163/4996E1C8 | 163/4996E1C8 | [null] | [null] | [null] | 2024-12-09 20:25:35.14769+08
(1 row)

M=# select pid,usename,pg_wal_lsn_diff(pg_current_wal_lsn(),sent_lsn) diff_sent_mb,pg_wal_lsn_diff(pg_current_wal_lsn(),write_lsn) diff_write_mb,pg_wal_lsn_diff(pg_current_wal_lsn(),flush_lsn) diff_flush_mb,pg_wal_lsn_diff(pg_current_wal_lsn(),replay_lsn) diff_replay_mb,pg_walfile_name_offset(sent_lsn) sentoffset,pg_walfile_name_offset(write_lsn) writeoffset,pg_walfile_name_offset(flush_lsn) flush_lsn from pg_stat_replication;
 pid | usename | diff_sent_mb | diff_write_mb | diff_flush_mb | diff_replay_mb | sentoffset | writeoffset | flush_lsn 
--------+---------+--------------+---------------+---------------+----------------+-------------------------------------+-------------------------------------+-------------------------------
 163525 | lzl | 0 | 0 | 0 | 0 | (000000010000016300000009,26665416) | (000000010000016300000009,26665416) | (000000


[/mypg/pg8094/data/pg_replslot/sub_test]$ ll
total 357392
-rw------- 1 postgres postgres 184 Dec 9 20:23 state
-rw------- 1 postgres postgres 88226964 Dec 9 20:22 xid-5074343-lsn-163-38000000.spill
-rw------- 1 postgres postgres 137696328 Dec 9 20:23 xid-5074343-lsn-163-40000000.spill
-rw------- 1 postgres postgres 26076708 Dec 9 20:23 xid-5074343-lsn-163-48000000.spill
[/mypg/pg8094/data/pg_replslot/sub_test]$ ll
total 4
-rw------- 1 postgres postgres 184 Dec 9 20:25 state2666
(1 row)

Large Transaction, No Subtransactions, Non-Replicated Table Spill Test
#

--source: create an unrelated table for writing data
CREATE TABLE no_replication_table (
 id BIGSERIAL PRIMARY KEY,
 column1 char(2000),
 column2 char(2000),
 column3 char(2000)
);

--Create a large transaction, don't commit yet
begin;
insert into no_replication_table(column1,column2,column3) 
select 'a','b','c' from generate_series(1,1000000) g;

--Spill
[postgres@lzldb:MYINST:8094 /mypg/pg8094/data/pg_replslot/sub_test]$ ll
total 357492
-rw------- 1 postgres postgres 184 Dec 9 20:09 state
-rw------- 1 postgres postgres 107511456 Dec 9 20:08 xid-5074106-lsn-163-28000000.spill
-rw------- 1 postgres postgres 137698804 Dec 9 20:09 xid-5074106-lsn-163-30000000.spill
-rw------- 1 postgres postgres 4308444 Dec 9 20:09 xid-5074106-lsn-163-38000000.spill

Large Transaction, Subtransactions, Non-Replicated Table Spill Test
#

## One insert per row, each insert as one subtransaction
echo "begin;">subtx.sql
for i in {1..1000000}
do
 echo "savepoint p$i;">>subtx.sql
 echo "insert into no_replication_table(column1,column2,column3) select 'a','b','c';">>subtx.sql
done


nohup psql -d lzl -f subtx.sql &

#During execution, observed 800k+ spill files
[/myhost/pg8094/data/pg_replslot/sub_test]$ ll |wc -l
823749
[/myhost/pg8094/data/pg_replslot/sub_test]$ ll |head -10
total 1099532
-rw------- 1 postgres postgres 184 Dec 9 21:10 state
-rw------- 1 postgres postgres 1236 Dec 9 21:10 xid-5519686-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519687-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519688-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519689-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519690-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519691-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519692-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519693-lsn-163-70000000.spill

Analysis of Slow Database Startup
#

Startup Process Startup Flow Analysis
#

Here we parse the startup flow frame by frame using the call stack:

11: main: Nothing to say.

10: PostmasterMain:

Before the main loop, it first calls the startup flow StartupPID = StartupDataBase(); which essentially calls StartChildProcess(StartupProcess):

#define StartupDataBase()		StartChildProcess(StartupProcess)

9: StartChildProcess: Forks a process. This process is the auxiliary process for starting postmaster; normal child process startup goes through this logic, forking at this step. The input AuxProcType=StartupProcess.

8: AuxiliaryProcessMain:

Since MyAuxProcType=StartupProcess, it goes through the StartupProcessMain flow, which is different from child processes like walsender, walwriter, bgwriter. The startup process itself exists for crash recovery WAL reading, but it does many other things:

	switch (MyAuxProcType)
	{
		case CheckerProcess:
			/* don't set signals, they're useless here */
			CheckerModeMain();
			proc_exit(1);		/* should never return */

		case BootstrapProcess:

			/*
			 * There was a brief instant during which mode was Normal; this is
			 * okay. We need to be in bootstrap mode during BootStrapXLOG for
			 * the sake of multixact initialization.
			 */
			SetProcessingMode(BootstrapProcessing);
			bootstrap_signals();
			BootStrapXLOG();
			BootstrapModeMain();
			proc_exit(1);		/* should never return */

		case StartupProcess: //Here here here here
			/* don't set signals, startup process has its own agenda */
			StartupProcessMain();
			proc_exit(1);		/* should never return */

		case BgWriterProcess:
			/* don't set signals, bgwriter has its own agenda */
			BackgroundWriterMain();
			proc_exit(1);		/* should never return */

		case CheckpointerProcess:
			/* don't set signals, checkpointer has its own agenda */
			CheckpointerMain();
			proc_exit(1);		/* should never return */

		case WalWriterProcess:
			/* don't set signals, walwriter has its own agenda */
			InitXLOGAccess();
			WalWriterMain();
			proc_exit(1);		/* should never return */

		case WalReceiverProcess:
			/* don't set signals, walreceiver has its own agenda */
			WalReceiverMain();
			proc_exit(1);		/* should never return */

		default:
			elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
			proc_exit(1);
	}

7: StartupProcessMain: Mainly to call StartupXLOG().

6: StartupXLOG:

Function comment:

This must be called ONCE during postmaster or standalone-backend startup

StartupXLOG is always called by postmaster regardless, whether crash shutdown or consistent shutdown:

	switch (ControlFile->state)
	{
 ...
			case DB_IN_PRODUCTION:
			ereport(LOG,
					(errmsg("database system was interrupted; last known up at %s",
							str_time(ControlFile->time))));
			break;

This matches the log output. Here’s the shutdown and startup log:

2024-12-06 17:02:57.534 CST,,,447560,,65693cde.6d448,1325,,2023-12-01 09:54:38 CST,,0,LOG,00000,"database system is shut down",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.536 CST,,,211844,,6752bdf3.33b84,1,,2024-12-06 17:03:47 CST,,0,LOG,00000,"ending log output to stderr",,"Future log output will go to log destination ""csvlog"".",,,,,,,"","postmaster"
2024-12-06 17:03:49.536 CST,,,211844,,6752bdf3.33b84,2,,2024-12-06 17:03:47 CST,,0,LOG,00000,"starting PostgreSQL 13.2 (RaseSQL 1.3) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39.0.1), 64-bit",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.537 CST,,,211844,,6752bdf3.33b84,3,,2024-12-06 17:03:47 CST,,0,LOG,00000,"listening on IPv4 address ""0.0.0.0"", port 7284",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.539 CST,,,211844,,6752bdf3.33b84,4,,2024-12-06 17:03:47 CST,,0,LOG,00000,"listening on Unix socket ""/tmp/.s.PGSQL.7284""",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.557 CST,,,211995,,6752bdf5.33c1b,1,,2024-12-06 17:03:49 CST,,0,LOG,00000,"database system was interrupted; last known up at 2024-12-06 17:00:10 CST",,,,,,,,,"","startup"

So, after shutdown, the control file recorded the database state as in production:

Database cluster state: in production

The in production state means the database is running, not a normal shutdown state — indicating that at the time of shutdown, it was not a consistent shutdown.

Continuing with the key code about fsync:

	/*----------
	 * If we previously crashed, perform a couple of actions:
	 *
	 * - The pg_wal directory may still include some temporary WAL segments
	 * used when creating a new segment, so perform some clean up to not
	 * bloat this path. This is done first as there is no point to sync
	 * this temporary data.
	 *
	 * - There might be data which we had written, intending to fsync it, but
	 * which we had not actually fsync'd yet. Therefore, a power failure in
	 * the near future might cause earlier unflushed writes to be lost, even
	 * though more recent data written to disk from here on would be
	 * persisted. To avoid that, fsync the entire data directory.
	 */
	if (ControlFile->state != DB_SHUTDOWNED &&
		ControlFile->state != DB_SHUTDOWNED_IN_RECOVERY)
	{
		RemoveTempXlogFiles();
		SyncDataDirectory();
	}

Here, because the control file state is not a normal shutdown, it enters the if-block and calls SyncDataDirectory() for fsync persistence.

StartupXLOG does many many things. Among those related to spill, besides SyncDataDirectory(), there’s also StartupReorderBuffer():

	/*
	 * Initialize replication slots, before there's a chance to remove
	 * required resources.
	 */
	StartupReplicationSlots();

	/*
	 * Startup logical state, needs to be setup now so we have proper data
	 * during crash recovery.
	 */
	StartupReorderBuffer();

StartupReorderBuffer is also called. It calls ReorderBufferCleanupSerializedTXNs to clean up spill files in all slot directories (but does not delete directories or state files):

/*
 * Delete all data spilled to disk after we've restarted/crashed. It will be
 * recreated when the respective slots are reused.
 */
void
StartupReorderBuffer(void)
{
	DIR		 *logical_dir;
	struct dirent *logical_de;

	logical_dir = AllocateDir("pg_replslot");
	while ((logical_de = ReadDir(logical_dir, "pg_replslot")) != NULL)
	{
		if (strcmp(logical_de->d_name, ".") == 0 ||
			strcmp(logical_de->d_name, "..") == 0)
			continue;

		/* if it cannot be a slot, skip the directory */
		if (!ReplicationSlotValidateName(logical_de->d_name, DEBUG2))
			continue;

		/*
		 * ok, has to be a surviving logical slot, iterate and delete
		 * everything starting with xid-*
		 */
		ReorderBufferCleanupSerializedTXNs(logical_de->d_name);
	}
	FreeDir(logical_dir);
}

5: SyncDataDirectory:

The function comment is very important:

/*
 * Issue fsync recursively on PGDATA and all its contents.
 *
 * We fsync regular files and directories wherever they are, but we
 * follow symlinks only for pg_wal and immediately under pg_tblspc.
 * Other symlinks are presumed to point at files we're not responsible
 * for fsyncing, and might not have privileges to write at all.
 *
 * Errors are logged but not considered fatal; that's because this is used
 * only during database startup, to deal with the possibility that there are
 * issued-but-unsynced writes pending against the data directory. We want to
 * ensure that such writes reach disk before anything that's done in the new
 * run. However, aborting on error would result in failure to start for
 * harmless cases such as read-only files in the data directory, and that's
 * not good either.
 *
 * Note that if we previously crashed due to a PANIC on fsync(), we'll be
 * rewriting all changes again during recovery.
 *
 * Note we assume we're chdir'd into PGDATA to begin with.
 */

fsync all data directory files to persist them
This action only happens during the startup phase
This action ensures the data directory is fully persistent before the database starts running

The body of SyncDataDirectory recursively walks directories and fsyncs (with some special handling for symlinks):

	walkdir(".", datadir_fsync_fname, false, LOG);
	if (xlog_is_symlink)
		walkdir("pg_wal", datadir_fsync_fname, false, LOG);
	walkdir("pg_tblspc", datadir_fsync_fname, true, LOG);

4: walkdir: Recurse to .

3: walkdir: Recurse to ./pg_replslot

2: walkdir: Recurse to ./pg_replslot/slotname

1: lstat: C library call. walkdir not only does fsync (via the callback datadir_fsync_fname), the walkdir function body also does lstat to get file info such as inode, file size, last modification time, etc. — similar to the Linux stat command.

0: _lxstat: C library call.

Startup logic summary:

PG starts an auxiliary process startup to help with startup. Unlike common child processes (walwriter, bgwriter, checkpointer, etc.), it’s always started during the startup process and does many things.
StartupXLOG is always called during startup, whether or not the database was consistently shut down.
Only in a non-normal shutdown state does SyncDataDirectory get triggered.
SyncDataDirectory fsyncs all data files for persistence and checks stat info for all data files.
fsync ensures data file consistency before startup; stat is probably to verify files are normal and readable (before the startup process starts, only the readability of the datadir directory was verified).
Regardless of shutdown state, StartupReorderBuffer is always called and cleans up spill files for all replication slots.

When Is the Ready State?
#

After the startup process finishes its work, the database is not yet in ready state. When the pmState state machine changes state, the reaper process reaping function is called. The reaper function itself does some recovery or startup work after a child process exits. The pmState state machine records the state as PM_STARTUP, which controls the startup/shutdown state.

Last steps of PostmasterMain:

	StartupPID = StartupDataBase();
	Assert(StartupPID != 0);
	StartupStatus = STARTUP_RUNNING;
	pmState = PM_STARTUP; //State machine changes state

	/* Some workers may be scheduled to start now */
	maybe_start_bgworkers();

	status = ServerLoop();

	/*
	 * ServerLoop probably shouldn't ever return, but if it does, close down.
	 */
	ExitPostmaster(status != STATUS_OK);

	abort();					/* not reached */
}

The core startup flow of PostmasterMain goes to reaper to handle the normal exit of the startup process.

PMState comment:

/*
 * We use a simple state machine to control startup, shutdown, and
 * crash recovery (which is rather like shutdown followed by startup).
 *
 * After doing all the postmaster initialization work, we enter PM_STARTUP
 * state and the startup process is launched. The startup process begins by
 * reading the control file and other preliminary initialization steps.
 * In a normal startup, or after crash recovery, the startup process exits
 * with exit code 0 and we switch to PM_RUN state. 

PMState is passed and processed via signals. After the startup process exits, reaper is activated to reap the process.

reaper function handling the startup child process’s normal exit:

		if (pid == StartupPID)
		{
			StartupPID = 0;
...

			/*
			 * Startup succeeded, commence normal operations
			 */
			StartupStatus = STARTUP_NOT_RUNNING; //Transition from STARTUP_RUNNING to STARTUP_NOT_RUNNING
			FatalError = false; //After none of the above ifs are hit, it's not fatal
			AbortStartTime = 0;
			ReachedNormalRunning = true;
			pmState = PM_RUN; //State machine transitions from PM_STARTUP to PM_RUN
			connsAllowed = ALLOW_ALL_CONNS;

			/*
			 * Crank up the background tasks, if we didn't do that already
			 * when we entered consistent recovery state. It doesn't matter
			 * if this fails, we'll just try again later.
			 */
 //Below: starting core child processes
			if (CheckpointerPID == 0)
				CheckpointerPID = StartCheckpointer();
			if (BgWriterPID == 0)
				BgWriterPID = StartBackgroundWriter();
			if (WalWriterPID == 0)
				WalWriterPID = StartWalWriter();

			/*
			 * Likewise, start other special children as needed. In a restart
			 * situation, some of them may be alive already.
			 */
 //Below: starting non-core child processes
			if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
				AutoVacPID = StartAutoVacLauncher();
			if (PgArchStartupAllowed() && PgArchPID == 0)
				PgArchPID = pgarch_start();
			if (PgStatPID == 0)
				PgStatPID = pgstat_start();

			/* workers may be scheduled to start now */
			maybe_start_bgworkers();
		 //At this point it's officially ready to accept connections
			/* at this point we are really open for business */
			ereport(LOG,
					(errmsg("database system is ready to accept connections")));

			/* Report status */
			AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_READY);
#ifdef USE_SYSTEMD
			sd_notify(0, "READY=1");
#endif

			continue;
		}

The “database system is ready to accept connections” message is right here.

Checkpointer, bgwriter, walwriter, autovacuum, arch (if present), stats — all these processes need to be started. At this stage, launching these processes doesn’t have to return success; they can be retried later in ServerLoop or on the next reaper execution. Only the startup process must start and complete all related tasks in one shot:

	if (pid < 0)
	{
		/* in parent, fork failed */
		int			save_errno = errno;

		errno = save_errno;
		switch (type)
		{
			case StartupProcess:
				ereport(LOG,
						(errmsg("could not fork startup process: %m")));
				break;
			case BgWriterProcess:
				ereport(LOG,
						(errmsg("could not fork background writer process: %m")));
				break;
			case CheckpointerProcess:
				ereport(LOG,
						(errmsg("could not fork checkpointer process: %m")));
				break;
			case WalWriterProcess:
				ereport(LOG,
						(errmsg("could not fork WAL writer process: %m")));
				break;
			case WalReceiverProcess:
				ereport(LOG,
						(errmsg("could not fork WAL receiver process: %m")));
				break;
			default:
				ereport(LOG,
						(errmsg("could not fork process: %m")));
				break;
		}

		/*
		 * fork failure is fatal during startup, but there's no need to choke
		 * immediately if starting other child types fails.
		 */
		if (type == StartupProcess)
			ExitPostmaster(1);
		return 0;
	}

Spill File Generation Logic Across Versions
#

Spill in all versions spills the largest transaction. Here we focus on when spilling happens.

PG12: pg12 hard-codes 4096 changes:

static const Size max_changes_in_memory = 4096;

/*
 * Check whether the transaction tx should spill its data to disk.
 */
static void
ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
	/*
	 * TODO: improve accounting so we cheaply can take subtransactions into
	 * account here.
	 */
	if (txn->nentries_mem >= max_changes_in_memory)
	{
		ReorderBufferSerializeTXN(rb, txn);
		Assert(txn->nentries_mem == 0);
	}
}

PG13: Spills when exceeding logical_decoding_work_mem memory size:

static void
ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
...
	while (rb->size >= logical_decoding_work_mem * 1024L)
	{
		/*
		 * Pick the largest transaction (or subtransaction) and evict it from
		 * memory by serializing it to disk.
		 */
		txn = ReorderBufferLargestTXN(rb);

		ReorderBufferSerializeTXN(rb, txn);
...
}

PG14: Adds streaming transfer ReorderBufferStreamTXN:

static void
ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
...
	while (rb->size >= logical_decoding_work_mem * 1024L)
	{
		/*
		 * Pick the largest transaction (or subtransaction) and evict it from
		 * memory by streaming, if possible. Otherwise, spill to disk.
		 */
		if (ReorderBufferCanStartStreaming(rb) &&
			(txn = ReorderBufferLargestTopTXN(rb)) != NULL)
		{...
			ReorderBufferStreamTXN(rb, txn);
		}
		else
		{...
			ReorderBufferSerializeTXN(rb, txn);
		}
...
}

Although PG14 has streaming replication, triggering it requires certain conditions:

/* Returns true, if the streaming can be started now, false, otherwise. */
static inline bool
ReorderBufferCanStartStreaming(ReorderBuffer *rb)
{
	LogicalDecodingContext *ctx = rb->private_data;
	SnapBuild *builder = ctx->snapshot_builder;

	/* We can't start streaming unless a consistent state is reached. */
	if (SnapBuildCurrentState(builder) < SNAPBUILD_CONSISTENT)
		return false;

	/*
	 * We can't start streaming immediately even if the streaming is enabled
	 * because we previously decoded this transaction and now just are
	 * restarting.
	 */
	if (ReorderBufferCanStream(rb) &&
		!SnapBuildXactNeedsSkip(builder, ctx->reader->EndRecPtr))
		return true;

	return false;
}

	/*
	 * Found a point after SNAPBUILD_FULL_SNAPSHOT where all transactions that
	 * were running at that point finished. Till we reach that we hold off
	 * calling any commit callbacks.
	 */
	SNAPBUILD_CONSISTENT = 2

Additional streaming trigger conditions:

Condition 1: All transactions covered by the snapshot have completed (presumably committed or rolled back)
Condition 2: The context is private data (does this mean two links to one table won’t trigger streaming?)
Condition 3: Transactions in the snapshot are not skippable (probably some special transactions can be skipped)

PG15: Similar to 14, just cleaner functions with less nesting.

PG16: About the same.

PG17: About the same, adds DEBUG_LOGICAL_REP_STREAMING_IMMEDIATE to force streaming.

Key points to remember:

PG12 and earlier: hard-coded 4096 changes
PG13: adds logical_decoding_work_mem parameter, allowing memory size adjustment to reduce spill probability
PG14 and later: supports streaming replication
Triggering streaming also requires certain conditions, so even with streaming, spills can still happen
PG17: adds debug_logical_replication_streaming parameter to force streaming

Spill File Cleanup Logic
#

Startup-time spill cleanup is just one scenario. There’s also walsender startup cleanup and drop slot cleanup.

Walsender Startup Cleanup
#

ReorderBufferCleanupSerializedTXNs is called during database startup (before walsender has started) and during walsender startup (while the database is running). Note these are different scenarios, though they call the same function. From the function comment, it’s meant to “remove leftover serialized reorder buffers” — i.e., clean up spill files.

/*
 * Remove any leftover serialized reorder buffers from a slot directory after a
 * prior crash or decoding session exit.
 */
static void
ReorderBufferCleanupSerializedTXNs(const char *slotname)
{
	DIR		 *spill_dir;
	struct dirent *spill_de;
	struct stat statbuf;
	char		path[MAXPGPATH * 2 + 12];

	sprintf(path, "pg_replslot/%s", slotname);

	/* we're only handling directories here, skip if it's not ours */
	if (lstat(path, &statbuf) == 0 && !S_ISDIR(statbuf.st_mode))
		return;

	spill_dir = AllocateDir(path);
	while ((spill_de = ReadDirExtended(spill_dir, path, INFO)) != NULL)
	{
		/* only look at names that can be ours */
 //Only compare first 3 characters
		if (strncmp(spill_de->d_name, "xid", 3) == 0)
		{
			snprintf(path, sizeof(path),
					 "pg_replslot/%s/%s", slotname,
					 spill_de->d_name);

			if (unlink(path) != 0)
				ereport(ERROR,
						(errcode_for_file_access(),
						 errmsg("could not remove file \"%s\" during removal of pg_replslot/%s/xid*: %m",
								path, slotname)));
		}
	}
	FreeDir(spill_dir);
}

Two things to note about the above cleanup logic:

Cleans files whose names start with “xid”. Obviously, the state file is not cleaned.
Uses unlink to clean, one file at a time. This can help us devise a faster startup scheme.

Database Startup Cleanup
#

During database startup, a startup process is forked to clean slots. The cleanup function is the same one walsender calls: ReorderBufferCleanupSerializedTXNs.

One more difference: after walsender restarts, it only cleans spills for the current slot with the same name; whereas during database startup, all slot spills are cleaned sequentially.

Database startup process, while-loop sequential cleanup logic:

void
StartupReorderBuffer(void)
{
	DIR		 *logical_dir;
	struct dirent *logical_de;

	logical_dir = AllocateDir("pg_replslot");
	while ((logical_de = ReadDir(logical_dir, "pg_replslot")) != NULL)
	{	//Exclude . and ..
		if (strcmp(logical_de->d_name, ".") == 0 ||
			strcmp(logical_de->d_name, "..") == 0)
			continue;
		//Validate slot name
		/* if it cannot be a slot, skip the directory */
		if (!ReplicationSlotValidateName(logical_de->d_name, DEBUG2))
			continue;

		/*
		 * ok, has to be a surviving logical slot, iterate and delete
		 * everything starting with xid-*
		 */
		ReorderBufferCleanupSerializedTXNs(logical_de->d_name);
	}
	FreeDir(logical_dir);
}

The while loop calls ReorderBufferCleanupSerializedTXNs, and after that, the logic is the same as walsender startup cleanup.

Manual Cleanup via pg_drop_replication_slot
#

The drop slot cleanup logic is different from the automatic spill file cleanup — it does not call ReorderBufferCleanupSerializedTXNs.

Drop slot flow:

pg_drop_replication_slot(PG_FUNCTION_ARGS) -> ReplicationSlotDrop(const char *name, bool nowait) -> ReplicationSlotDropAcquired(void) -> ReplicationSlotDropPtr

ReplicationSlotDropPtr’s slot cleanup logic is also interesting:

/*
 * Permanently drop the replication slot which will be released by the point
 * this function returns.
 */
static void
ReplicationSlotDropPtr(ReplicationSlot *slot)
{
	char		path[MAXPGPATH];
	char		tmppath[MAXPGPATH];

	/*
	 * If some other backend ran this code concurrently with us, we might try
	 * to delete a slot with a certain name while someone else was trying to
	 * create a slot with the same name.
	 */
	LWLockAcquire(ReplicationSlotAllocationLock, LW_EXCLUSIVE);

	/* Generate pathnames. */
	sprintf(path, "pg_replslot/%s", NameStr(slot->data.name));
	sprintf(tmppath, "pg_replslot/%s.tmp", NameStr(slot->data.name));

	/*
	 * Rename the slot directory on disk, so that we'll no longer recognize
	 * this as a valid slot. Note that if this fails, we've got to mark the
	 * slot inactive before bailing out. If we're dropping an ephemeral or a
	 * temporary slot, we better never fail hard as the caller won't expect
	 * the slot to survive and this might get called during error handling.
	 */
	if (rename(path, tmppath) == 0) //rename file
	{
		/*
		 * We need to fsync() the directory we just renamed and its parent to
		 * make sure that our changes are on disk in a crash-safe fashion. If
		 * fsync() fails, we can't be sure whether the changes are on disk or
		 * not. For now, we handle that by panicking;
		 * StartupReplicationSlots() will try to straighten it out after
		 * restart.
		 */
 //fsync persistence
		START_CRIT_SECTION();
		fsync_fname(tmppath, true);
		fsync_fname("pg_replslot", true);
		END_CRIT_SECTION();
	}
...

	/*
	 * If removing the directory fails, the worst thing that will happen is
	 * that the user won't be able to create a new slot with the same name
	 * until the next server restart. We warn about it, but that's all.
	 */
 
	if (!rmtree(tmppath, true))
		ereport(WARNING,
				(errmsg("could not remove directory \"%s\"", tmppath)));
	/*
	 * We release this at the very end, so that nobody starts trying to create
	 * a slot while we're still cleaning up the detritus of the old one.
	 */
	LWLockRelease(ReplicationSlotAllocationLock);
}

Drop slot doesn’t directly unlink files in the slot directory. Instead, it first renames the slotname/ directory to slotname.tmp/, then unlinks the files inside, and finally removes the slotname.tmp/ directory itself.

In this, rmtree also loops to unlink files.

Accelerated Startup Scheme After Replication Slot Spill
#

Deleting 10 million spill files is obviously very slow, but directly moving the directory (mv) is extremely fast. However, direct mv requires attention to the name after the move and the state file, as well as knowing which source code step the mv bypasses.

mv Naming Notes
#

Since it was an abnormal shutdown, the startup process will execute SyncDataDirectory to fsync and stat all data files — this is hard to bypass. After SyncDataDirectory completes, it starts handling replication slots. When handling slots, it calls StartupReorderBuffer() -> ReorderBufferCleanupSerializedTXNs to fully clean up spill files.

Before entering cleanup, ReplicationSlotValidateName validates the slot name. We can exploit ReplicationSlotValidateName to trick the startup process into skipping the ReorderBufferCleanupSerializedTXNs process.

ReplicationSlotValidateName rules:

bool
ReplicationSlotValidateName(const char *name, int elevel)
{
...

	for (cp = name; *cp; cp++)
	{ //Key rule here
		if (!((*cp >= 'a' && *cp <= 'z')
			 || (*cp >= '0' && *cp <= '9')
			 || (*cp == '_')))
		{
			ereport(elevel,
					(errcode(ERRCODE_INVALID_NAME),
					 errmsg("replication slot name \"%s\" contains invalid character",
							name),
					 errhint("Replication slot names may only contain lower case letters, numbers, and the underscore character.")));
			return false;
		}
	}
	return true;
}

Valid slot names only contain a-z, 0-9, _.

So when renaming, it’s recommended to add a dot .:

Recommended: slotname.bak, slotname.20241215, etc.
Not recommended: slotnamebackup, slotname20241215, slotname_bak, etc.
Not recommended: .tmp suffix — slot names with .tmp have special meaning.

After renaming, you need to create the directory and copy the state file, otherwise the slot will behave strangely on startup (e.g., duplicate slot names, auto-generated slot names, inability to delete slots, downstream unable to start the replication link, etc.).

Recommended mv operations summarized:

cd pg_replslot
mv slotname slotname.bak 
mkdir slotname
cp slotname.bak/state slotname/

Startup Time Comparison
#

Compare startup speed across different source code flows to see if manual mv/rm acceleration is actually meaningful.

Reference source logic principles:

Normal shutdown: goes through fsync and stat
Abnormal shutdown: goes through fsync and stat
Valid mv: rename slot directory to .bak, skip unlink
Invalid mv: rename slot directory to _bak, spill files start with xid, goes through unlink

Since actual spill files would be too slow, I manually created fake slot directories and spill files: 50 slots total, 400k spills per slot, 20 million spills total, to test startup time (using cp directory is much faster than cp or dd files).

#	Test Plan	Startup Time
1	Normal shutdown; no fsync/stat, no unlink	0.1 seconds
2	Normal shutdown, invalid mv; no fsync/stat, unlink	11 min 41 sec
3	Abnormal shutdown, valid mv; fsync/stat, no unlink	4 min 35 sec
4	Abnormal shutdown, invalid mv; fsync/stat, unlink	32 min 2 sec
5	Abnormal shutdown, rm (create slot dir, keep state)	13 min 4 sec

Comparing plans 3 and 5, theoretically in the scenario at hand, a valid mv could achieve startup in about 4 minutes, while rm would take about 13 minutes. (This is a rough comparison; the recovery environment already showed some differences.)

PostgreSQL Case Study: Analysis of Abnormally Long Planning Time

Wed, 21 Aug 2024 00:00:00 +0000

Problem Analysis Overview
#

The database kept OOMing. Analysis revealed the issue was in query plan generation: planning time ~1 second, planning shared hits ~1 million. After thorough investigation, the root cause was identified as bloat in the statistics base table pg_statistic. On the first SQL execution of a session — due to a CatCacheMiss — the backend accessed and cached an excessive amount of dead tuple data from pg_statistic. Application connections always spawned new sessions, and the combined memory usage across multiple backends was too large, leading to OOM.

Below is the detailed analysis process.

Problem Symptoms
#

A certain database kept OOMing and restarting. After investigation, we found that while the number of concurrent sessions wasn’t high, each session’s memory footprint was quite large. The total memory exceeded the cgroup memory limit, causing OOM.

We could preliminarily rule out the following causes:

Not caused by excessive metadata. Too many objects (typically too many partitions) would cause sessions to cache excessive metadata. This database didn’t have that many objects.
Not caused by SQL execution plan issues. Sorting/hash operations might use too much memory. This database didn’t fit that scenario — the SQL in question was a simple sequential scan.

During the investigation, we discovered that any simple SQL query in this database took a very long time to execute, and Planning Buffers showed about 1 million hits:

 explain (analyze,buffers,timing) select * from lzlinfo limit 1;
 QUERY PLAN 
--------------------------------------------------------------------------------------------------------------------
 Limit (cost=0.00..1.02 rows=1 width=71) (actual time=0.011..0.012 rows=1 loops=1)
 Buffers: shared hit=1
 -> Seq Scan on lzlinfo (cost=0.00..480.73 rows=473 width=71) (actual time=0.010..0.010 rows=1 loops=1)
 Buffers: shared hit=1
 Planning:
 Buffers: shared hit=1127312 -- Abnormal planning shared hit
 Planning Time: 947.038 ms -- Abnormal planning time
 Execution Time: 0.035 ms
(8 rows)

Running the same SQL a second time, the planning time was normal.

Problem Investigation Process
#

Printing Execution Plan Statistics
#

We enabled logging for each phase of the execution plan:

set log_parser_stats =on;
set log_planner_stats =on;
set log_executor_stats =on;

Then ran the SQL. The log output was as follows:

2024-08-13 10:02:33.936 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,13,"idle",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"PARSER STATISTICS","! system usage stats:
! 0.000046 s user, 0.000046 s system, 0.000091 s elapsed
! [0.001661 s user, 0.001661 s system total]
! 4660 kB max resident size
! 0/0 [0/8] filesystem blocks in/out
! 0/36 [0/996] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/0 [5/0] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
2024-08-13 10:02:33.938 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,14,"EXPLAIN",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"PARSE ANALYSIS STATISTICS","! system usage stats:
! 0.001459 s user, 0.000000 s system, 0.001464 s elapsed
! [0.003146 s user, 0.001687 s system total]
! 5972 kB max resident size
! 0/0 [0/8] filesystem blocks in/out
! 0/325 [0/1324] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/0 [5/0] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
2024-08-13 10:02:33.938 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,15,"EXPLAIN",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"REWRITER STATISTICS","! system usage stats:
! 0.000001 s user, 0.000000 s system, 0.000001 s elapsed
! [0.003177 s user, 0.001687 s system total]
! 5972 kB max resident size
! 0/0 [0/8] filesystem blocks in/out
! 0/0 [0/1324] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/0 [5/0] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
2024-08-13 10:02:34.644 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,16,"EXPLAIN",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"PLANNER STATISTICS","! system usage stats:
! 0.539964 s user, 0.164083 s system, 0.705718 s elapsed
! [0.543248 s user, 0.165770 s system total]
! 745072 kB max resident size -- Abnormal point
! 0/0 [0/8] filesystem blocks in/out
! 0/184803 [0/186157] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/1 [5/1] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
2024-08-13 10:02:34.644 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,17,"EXPLAIN",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"EXECUTOR STATISTICS","! system usage stats:
! 0.540248 s user, 0.164170 s system, 0.706088 s elapsed
! [0.543532 s user, 0.165857 s system total]
! 745596 kB max resident size
! 0/0 [0/8] filesystem blocks in/out
! 0/184898 [0/186252] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/1 [5/1] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
"

During the planner phase, memory usage skyrocketed and elapsed time also spiked. This pinpointed the issue to the planner phase within the overall planning stage. There wasn’t much else actionable from the stats.

strace Tracing
#

strace -p 76419

strace: Process 76419 attached
epoll_wait(4, [{EPOLLIN, {u32=15422552, u64=15422552}}], 1, -1) = 1
recvfrom(9, "Q\0\0\0\262explain (analyze,buffers) s"..., 8192, 0, NULL, NULL) = 179
lseek(5, 0, SEEK_END) = 8192
brk(NULL) = 0xfed000
brk(0x100e000) = 0x100e000
brk(NULL) = 0x100e000
brk(NULL) = 0x100e000
brk(0x1007000) = 0x1007000
brk(NULL) = 0x1007000
mmap(NULL, 270336, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b7806b0c000
open("base/17076/16678", O_RDWR) = 7
lseek(7, 0, SEEK_END) = 0
open("base/17076/46160", O_RDWR) = 12
lseek(12, 0, SEEK_END) = 7667712
open("base/17076/46168", O_RDWR) = 13
lseek(13, 0, SEEK_END) = 188416
open("base/17076/46170", O_RDWR) = 14
lseek(14, 0, SEEK_END) = 188416
mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b78c1b36000
brk(NULL) = 0x1007000
brk(0x102c000) = 0x102c000
brk(NULL) = 0x102c000
brk(NULL) = 0x102c000
brk(0x1025000) = 0x1025000
brk(NULL) = 0x1025000
lseek(12, 0, SEEK_END) = 7667712
open("pg_stat_tmp/pgss_query_texts.stat", O_RDWR|O_CREAT, 0600) = 15
pwrite64(15, "explain (analyze,buffers) select"..., 172, 93934) = 172
pwrite64(15, "\0", 1, 94106) = 1
close(15) = 0
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\10\1\0\0\264B\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 264, 0, NULL, 0) = 264
sendto(8, "\2\0\0\0\10\1\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 264, 0, NULL, 0) = 264
sendto(8, "\16\0\0\0H\0\0\0\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 72, 0, NULL, 0) = 72
sendto(9, "T\0\0\0#\0\1QUERY PLAN\0\0\0\0\0\0\0\0\0\0\31\377\377\377\377"..., 826, 0, NULL, 0) = 826
recvfrom(9, 0xd2b4e0, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(4, 

Although there were many shared hits, strace didn’t reveal much. strace showed the session only opened 4 data files. Using fd and oid2name to look up the data files, they turned out to be: the table, two indexes on the table, and pathman_config:

From database "lzldb":
 Filenode Table Name
--------------------------------------
 46170 ix_name
 46168 pk_lzlinfo
 46160 lzlinfo
 16678 pathman_config

These objects are not large, so it didn’t look like oversized tables (or indexes) were the cause.

perf
#

(No screenshot — use your imagination.)

The perf flame graph showed ~40% of the time spent on the heap_hot_search_buffer stack.

gdb
#

Using heap_hot_search_buffer as a clue, after multiple gdb sessions, we set the following breakpoints to investigate:

b relation_open
b get_relation_info
b RelationCacheInvalidateEntry 
b get_relname_relid
b AcceptInvalidationMessages
b RelationClearRelation
b pg_hint_plan_planner
b heap_hot_search_buffer

When breakpoints first hit, there was a lot of noise — they were normal logic. But later, after execution reached a certain point, only heap_hot_search_buffer kept hitting:

Breakpoint 15, heap_hot_search_buffer (tid=tid@entry=0x2313c60, relation=0x2b2141663910, buffer=17045, snapshot=snapshot@entry=0x228a058, heapTuple=heapTuple@entry=0x23273d0, 
 all_dead=all_dead@entry=0x7ffce272e28f, first_call=true) at heapam.c:1503
1503 in heapam.c
(gdb) 
Continuing.
...

Breakpoint 15, heap_hot_search_buffer (tid=tid@entry=0x2313c60, relation=0x2b2141663910, buffer=96708, snapshot=snapshot@entry=0x228a058, heapTuple=heapTuple@entry=0x23273d0, 
 all_dead=all_dead@entry=0x7ffce272e28f, first_call=true) at heapam.c:1503
1503 in heapam.c

Most arguments passed to heap_hot_search_buffer remained unchanged — including the addresses of relation and heapTuple — only the buffer parameter changed, indicating it was scanning the same relation.

heapTuple contained table OID information. Let’s print it:

(gdb) p *heapTuple
$46 = {
 t_len = 968, 
 t_self = {
 ip_blkid = {
 bi_hi = 0, 
 bi_lo = 7211
 }, 
 ip_posid = 5
 }, 
 t_tableOid = 2619, -- This is useful
 t_data = 0x2b2155fced00

heap_hot_search_buffer was called with OID=2619. Looking up 2619 in pg_class, it’s pg_statistic:

select oid,relname from pg_class where oid in (2619)
 oid | relname 
-------+----------------------------------
 2619 | pg_statistic

Accessing the statistics base table is expected — PG needs statistics to estimate costs when generating candidate execution plans.

pg_statistic Bloat
#

Now that we’ve pinpointed pg_statistic, let’s check its condition:

> \dt+ pg_statistic
 List of relations
 Schema | Name | Type | Owner | Persistence | Size | Description 
------------+--------------+-------+----------+-------------+---------+-------------
 pg_catalog | pg_statistic | table | postgres | permanent | 1036 MB | 


> select * from pg_class where relname='pg_statistic'\gx
-[ RECORD 1 ]-------+------------------------------------------------
oid | 2619
relname | pg_statistic
relnamespace | 11
reltype | 12016
reloftype | 0
relowner | 10
relam | 2
relfilenode | 2619
reltablespace | 0
relpages | 132481
reltuples | 4655

pg_statistic is 1GB — certainly oversized. 132,481 blocks but only 4,655 rows — this is clearly table bloat. But even with bloat, does accessing statistics really require caching the entire pg_statistic table? Logically, no — you only need the statistics for the specific table. And indeed, PG accesses pg_statistic through its primary key index pg_statistic_relid_att_inh_index. From the call stack below, we can see the composite primary key fields being passed:

bt
...
#6 0x000000000086edbc in SearchCatCacheMiss (cache=cache@entry=0x226ba80, nkeys=nkeys@entry=3, hashValue=hashValue@entry=853716409, hashIndex=hashIndex@entry=57, v1=v1@entry=18767, v2=v2@entry=1, 
 v3=v3@entry=0, v4=v4@entry=0) at catcache.c:1368
#7 0x000000000086fa82 in SearchCatCacheInternal (v4=0, v3=<optimized out>, v2=<optimized out>, v1=<optimized out>, nkeys=3, cache=0x226ba80) at catcache.c:1299
#8 SearchCatCache3 (cache=0x226ba80, v1=v1@entry=18767, v2=v2@entry=1, v3=v3@entry=0) at catcache.c:1183
#9 0x0000000000880d70 in SearchSysCache3 (cacheId=cacheId@entry=58, key1=key1@entry=18767, key2=key2@entry=1, key3=key3@entry=0) at syscache.c:1145
#10 0x0000000000874092 in get_attavgwidth (relid=relid@entry=18767, attnum=1) at lsyscache.c:2991
#11 0x00000000006a2d46 in set_rel_width (root=root@entry=0x2326600, rel=rel@entry=0x21e8418) at costsize.c:5516 
...

The call passes relid=relid@entry=18767, attnum=1:

 select ctid,starelid,staattnum from pg_statistic where starelid=18767;
 ctid | starelid | staattnum 
------------+----------+-----------
 (132657,6) | 18767 | 1
 (132657,7) | 18767 | 2
 (132657,8) | 18767 | 3
 (132657,9) | 18767 | 4
 (132658,1) | 18767 | 5
 (132658,2) | 18767 | 6
 (132658,3) | 18767 | 7
 (132658,4) | 18767 | 8
 (132658,5) | 18767 | 9
 (132658,6) | 18767 | 10
 -- lzlinfo has 10 columns total, each with a staattnum entry

From the ctid, we can see this data actually lives in just 2 blocks.

Now let’s access pg_statistic via the composite primary key index. Even with data in only 2 blocks, it took 1 second to access with ~1 million (1,141,568) shared hits:

 explain (analyze,buffers,timing,verbose) select ctid,starelid from pg_statistic where starelid=18767;
 QUERY PLAN 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Index Scan using pg_statistic_relid_att_inh_index on pg_catalog.pg_statistic (cost=0.41..103.31 rows=23 width=10) (actual time=105.416..1035.723 rows=10 loops=1)
 Output: ctid, starelid
 Index Cond: (pg_statistic.starelid = '18767'::oid)
 Buffers: shared hit=1141568 -- Abnormal
 Planning:
 Buffers: shared hit=8
 Planning Time: 0.102 ms
 Execution Time: 1035.802 ms

Accessing 10 rows in pg_statistic via the index resulted in ~1M shared hits — roughly matching the ~1M planning shared hits from the original SQL. (Note: Planning Time here is minimal, meaning the issue is not in plan generation per se, but in the data access during planning.)

Index Dead Tuples
#

If vacuum hasn’t truly “run properly”, index dead tuples still point to dead heap tuples.

Refer to: From Very Slow Unique Index Scans to Index Bloat

autovacuum Not Reclaiming Dead Tuples
#

With such severe table bloat, shouldn’t autovacuum have reclaimed it?

select * from pg_stat_all_tables where relname='pg_statistic'\gx
-[ RECORD 1 ]-------+------------------------------
relid | 2619
schemaname | pg_catalog
relname | pg_statistic
seq_scan | 1 	 -- Very few sequential scans on pg_statistic
seq_tup_read | 4655
idx_scan | 28715508 -- Many index scans on pg_statistic
idx_tup_fetch | 25150245
n_tup_ins | 46
n_tup_upd | 1292143 -- Lots of updates
n_tup_del | 14
n_tup_hot_upd | 138448
n_live_tup | 4655
n_dead_tup | 1496776
n_mod_since_analyze | 1292203
n_ins_since_vacuum | 0
last_vacuum | [null]
last_autovacuum | 2024-08-16 20:34:15.045022+08 -- Note: autovacuum timestamp is recent
last_analyze | [null]
last_autoanalyze | [null]
vacuum_count | 0
autovacuum_count | 144170
analyze_count | 0
autoanalyze_count | 0

Actually, autovacuum was constantly running on pg_statistic, but the worker process may not have been visible because it finished quickly (having nothing to actually reclaim) and went back to naptime:

show autovacuum_naptime ;
 autovacuum_naptime 
--------------------
 1min

It naps every 1 minute, and the logs show autovacuum info printed every 1 minute as well:

2024-08-16 21:05:15.267 CST,,,41080,,66bf4e87.a078,1,,2024-08-16 21:05:11 CST,27/166839,0,LOG,00000,"automatic vacuum of table ""lzldb.pg_catalog.pg_statistic"": index scans: 0
pages: 0 removed, 132685 remain, 1 skipped due to pins, 0 skipped frozen
tuples: 0 removed, 1501745 remain, 1497090 are dead but not yet removable, oldest xmin: 119329380
buffer usage: 265443 hits, 0 misses, 0 dirtied
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s
system usage: CPU: user: 0.53 s, system: 0.17 s, elapsed: 3.38 s
WAL usage: 1 records, 0 full page images, 233 bytes",,,,,,,,,"","autovacuum worker"
2024-08-16 21:05:17.474 CST,,,41080,,66bf4e87.a078,2,,2024-08-16 21:05:11 CST,27/166844,136438968,LOG,00000,"automatic analyze of table ""lzldb.public.lzlinfo"" system usage: CPU: user: 2.02 s, system: 0.00 s, elapsed: 2.08 s",,,,,,,,,"","autovacuum worker"
"

1497090 are dead but not yet removable — although autovacuum was triggered, it didn’t reclaim any dead tuples at all. 1,497,090 dead tuples remained uncleaned.

Investigating who held oldest xmin: 119329380, we quickly identified a replication slot:

select * from pg_replication_slots;
 slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn | wal_status | safe_wal_size 
-----------------+----------+-----------+--------+----------+-----------+--------+------------+--------+--------------+--------------+---------------------+------------+---------------
 slotslotlostname | pgoutput | logical | 17076 | lzldb | f | f | [null] | [null] | 119329380 | 3F9/105A4970 | 3F9/105F8778 | extended | [null]

The slot’s catalog_xmin=119329380 matched the vacuum’s oldest xmin: 119329380.

active=f indicated that the replication link was already broken.

Fixing the Problem
#

Drop the replication slot:

select pg_drop_replication_slot('slotslotlostname');
 pg_drop_replication_slot 
--------------------------

Then manually vacuum or wait 1 minute for autovacuum.

Finally, open a brand-new session to verify the fix:

## psql
psql (13.2)
Type "help" for help.

> \c lzldb
You are now connected to database "lzldb" as user "postgres".
> explain (analyze,buffers,timing) select * from lzlinfo limit 1;
 QUERY PLAN 
---------------------------------------------------------------------------------------------------------------------
 Limit (cost=0.00..8.04 rows=1 width=71) (actual time=0.023..0.025 rows=1 loops=1)
 Buffers: shared hit=1
 -> Seq Scan on lzlinfo (cost=0.00..3802.73 rows=473 width=71) (actual time=0.018..0.018 rows=1 loops=1)
 Buffers: shared hit=1
 Planning:
 Buffers: shared hit=2578
 Planning Time: 9.605 ms
 Execution Time: 0.098 ms

Planning time dropped from ~1 second to ~10 ms, and planning shared hits dropped from ~1M to ~2K. The problem was basically resolved.

Case Summary
#

The replication link broke and the replication slot wasn’t cleaned up in time, leading to bloat in the pg_statistic statistics base table. This caused each backend to be very slow when loading statistics for the first time and to read excessive pages into its local cache. Each backend’s cache exceeded normal levels (~2GB), and with multiple backends this led to OOM.

The problem itself is simple — it was just the investigation that was convoluted. In short: bloat in the base table pg_statistic caused excessive data access during the plan generation phase. Metadata base table bloat can cause other tricky problems too — until next time.

A Classic Case of Long Transaction, Table Bloat, and LIMIT Issues

Mon, 12 Aug 2024 00:00:00 +0000

Slow Primary Key Update — Problem Analysis
#

A simple primary key update took over 1 second to execute. Due to high concurrency, the CPU was completely maxed out:

2024-04-01 10:19:36.084 CST,"lzlopr","lzl",158751,"10.33.78.149:51502",66055a6b.26c1f,172,"UPDATE",2024-03-28 19:54:19 CST,528/19816630,970251337,LOG,00000,"duration: 1218.688 ms plan:
Query Text: update table_a set (omitted...）=$6 where column_id =$7
Update on table_a (cost=0.40..5.49 rows=1 width=2774)
-> Index Scan using pk_id on table_a (cost=0.40..5.49 rows=1 width=2774)
 Index Cond: ((column_id)::text = $7)",,,,,,,,,"PostgreSQL JDBC Driver","client backend"

The SQL itself is very simple — an update with a condition on the primary key. Looking at the execution plan, it used the pk_id primary key index, so there was no problem with the plan itself; the issue wasn’t a plan change.

Let’s rewrite the SQL (since it’s an UPDATE) and use explain (analyze,buffers) to compare the execution cost:

=> explain (analyze,buffers) select * from table_a where column_id='d4f713370e584820a9b15e2218ea436a';
 QUERY PLAN 
---------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on table_a (cost=2.91..5.42 rows=1 width=1156) (actual time=55.052..123.354 rows=1 loops=1)
 Recheck Cond: ((column_id)::text = 'd4f713370e584820a9b15e2218ea436a'::text)
 Heap Blocks: exact=1
 Buffers: shared hit=13870
 -> Bitmap Index Scan on pk_id (cost=0.00..2.91 rows=1 width=0) (actual time=3.464..3.465 rows=13866 loops=1)
 Index Cond: ((column_id)::text = 'd4f713370e584820a9b15e2218ea436a'::text)
 Buffers: shared hit=24
Planning:
 Buffers: shared hit=4261
Planning Time: 11.028 ms
Execution Time: 123.567 ms
(11 rows)

The actual execution plan is fine, but shared hit=13870 is clearly way too high. Normally, a primary key lookup shouldn’t scan that many pages. This strongly suggests table bloat.

Checking table bloat:

-- Table size \dt
Size | 525 MB

-- Actual row count
count | 827

-- Dead tuples from pg_stat_all_tables
n_live_tup | 786
n_dead_tup | 657604

Only ~800 live tuples but 650K dead tuples! This explains why the primary key scan visited so many pages. But why weren’t the dead tuples reclaimed?

When a table exceeds the default 20% modification threshold, autovacuum triggers vacuum to reclaim space. We can see in the logs that autovacuum was indeed being triggered:

2024-04-01 14:13:46.649 CST,,,14081,,660a5099.3701,1,,2024-04-01 14:13:45 CST,259/17828993,0,LOG,00000,"automatic vacuum of table ""lzl.public.table_a"": index scans: 0
2024-04-01 14:13:47.801 CST,,,14081,,660a5099.3701,2,,2024-04-01 14:13:45 CST,259/17828994,971045014,LOG,00000,"automatic analyze of table ""lzl.public.table_a"" system usage: CPU: user: 0.08 s, system: 0.01 s, elapsed: 1.15 s",,,,,,,,,"","autovacuum worker"
2024-04-01 14:14:46.673 CST,,,26136,,660a50d5.6618,1,,2024-04-01 14:14:45 CST,259/17829090,0,LOG,00000,"automatic vacuum of table ""lzl.public.table_a"": index scans: 0
2024-04-01 14:14:47.833 CST,,,26136,,660a50d5.6618,2,,2024-04-01 14:14:45 CST,259/17829091,971049759,LOG,00000,"automatic analyze of table ""lzl.public.table_a"" system usage: CPU: user: 0.08 s, system: 0.03 s, elapsed: 1.15 s",,,,,,,,,"","autovacuum worker"
2024-04-01 14:15:46.680 CST,,,40743,,660a5111.9f27,1,,2024-04-01 14:15:45 CST,259/17829164,0,LOG,00000,"automatic vacuum of table ""lzl.public.table_a"": index scans: 0
2024-04-01 14:15:47.849 CST,,,40743,,660a5111.9f27,2,,2024-04-01 14:15:45 CST,259/17829165,971055464,LOG,00000,"automatic analyze of table ""lzl.public.table_a"" system usage: CPU: user: 0.08 s, system: 0.03 s, elapsed: 1.16 s",,,,,,,,,"","autovacuum worker"
2024-04-01 14:16:46.677 CST,,,52599,,660a514d.cd77,1,,2024-04-01 14:16:45 CST,259/17829263,0,LOG,00000,"automatic vacuum of table ""lzl.public.table_a"": index scans: 0
2024-04-01 14:16:47.844 CST,,,52599,,660a514d.cd77,2,,2024-04-01 14:16:45 CST,259/17829264,971061382,LOG,00000,"automatic analyze of table ""lzl.public.table_a"" system usage: CPU: user: 0.08 s, system: 0.03 s, elapsed: 1.16 s",,,,,,,,,"","autovacuum worker"
2024-04-01 14:17:46.699 CST,,,64858,,660a5189.fd5a,1,,2024-04-01 14:17:45 CST,234/16589539,0,LOG,00000,"automatic vacuum of table ""lzl.public.table_a"": index scans: 0
2024-04-01 14:17:47.851 CST,,,64858,,660a5189.fd5a,2,,2024-04-01 14:17:45 CST,234/16589540,971066091,LOG,00000,"automatic analyze of table ""lzl.public.table_a"" system usage: CPU: user: 0.09 s, system: 0.02 s, elapsed: 1.15 s",,,,,,,,,"","autovacuum worker"
2024-04-01 14:18:46.703 CST,,,78112,,660a51c5.13120,1,,2024-04-01 14:18:45 CST,259/17829409,0,LOG,00000,"automatic vacuum of table ""lzl.public.table_a"": index scans: 0
2024-04-01 14:18:47.854 CST,,,78112,,660a51c5.13120,2,,2024-04-01 14:18:45 CST,259/17829410,971070390,LOG,00000,"automatic analyze of table ""lzl.public.table_a"" system usage: CPU: user: 0.09 s, system: 0.02 s, elapsed: 1.15 s",,,,,,,,,"","autovacuum worker"		

Not only was it triggered, but the interval was exactly 1 minute. The default autovacuum_naptime is 1 minute:

>= show autovacuum_naptime ;
autovacuum_naptime 
--------------------
1min
(1 row)

We can conclude:

autovacuum was successfully triggered
Dead tuples either couldn’t be reclaimed fast enough — the dead tuples generated within 1 minute exceeded 20% (maybe 1 minute is too long); or they weren’t being reclaimed at all, guaranteeing the next autovacuum trigger

Let’s look at the detailed autovacuum output:

2024-04-01 10:22:44.648 CST,,,16827,,660a1a73.41bb,1,,2024-04-01 10:22:43 CST,170/16910186,0,LOG,00000,"automatic vacuum of table ""lzl.public.table_a"": index scans: 0
pages: 0 removed, 48745 remain, 6 skipped due to pins, 0 skipped frozen
tuples: 0 removed, 744488 remain, 743666 are dead but not yet removable, oldest xmin: 969118077
buffer usage: 97603 hits, 0 misses, 5 dirtied
avg read rate: 0.000 MB/s, avg write rate: 0.028 MB/s
system usage: CPU: user: 0.21 s, system: 0.22 s, elapsed: 1.41 s
WAL usage: 4 records, 3 full page images, 5129 bytes",,,,,,,,,"","autovacuum worker"

autovacuum triggered but reclaimed nothing: tuples: 0 removed, 744488 remain, 743666 are dead but not yet removable, oldest xmin: 969118077. oldest xmin represents the oldest transaction in the database — meaning there’s a long-running transaction. This is easy to find:

>= select pid,usename,xact_start,state_change,wait_event,state,query from pg_stat_activity where state<>'idle' order by xact_start ;
pid | usename | xact_start | state_change | wait_event | state | 
--------+------------+-------------------------------+-------------------------------+---------------------+---------------------+------------------------------------------------------------------------------
164658 | phbdspsqp | 2024-04-01 08:36:57.275408+08 | 2024-04-01 08:36:57.299609+08 | DataFileRead | active | SELECT "minval","maxval" FROM (select min(ID) as minval,max(TRACK

The long transaction was a SQL that had been running since around 8 AM that morning, for several hours. Even though it wasn’t on the same table, being the oldest xmin it still had an impact.

At this point the root cause is identified:

Table A had frequent updates, high bloat risk
A long transaction on table B prevented dead tuple reclamation on table A
Table A’s update statements scanned excessive pages

Solution:

Kill the long transaction: select pg_terminate_backend(164658)
Manually vacuum or wait 1 minute (or less) for automatic vacuum: vacuum table_a

After both steps were completed, checking dead tuples:

n_live_tup | 707
n_dead_tup | 298

650K dead tuples have been cleaned up.

Checking the execution plan again:

=> explain (analyze,buffers) select * from table_a where column_id='d4f713370e584820a9b15e2218ea436a';
 QUERY PLAN 
-------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using pk_id on table_a (cost=0.40..5.44 rows=1 width=621) (actual time=0.026..0.029 rows=1 loops=1)
 Index Cond: ((column_id)::text = 'd4f713370e584820a9b15e2218ea436a'::text)
 Buffers: shared hit=6
Planning Time: 0.057 ms
Execution Time: 0.043 ms

Shared hits down to just 6 — issue resolved.

Additionally, vacuum only reclaims dead tuples but does not shrink the table — the table remains the same size. Space can only be returned to the OS when new data reuses those pages, or through a repack/table rebuild:

Size | 525 MB

Bonus SQL Optimization — ORDER BY LIMIT
#

That long-running transaction SQL also had its own problems… The business reported it ran fast a few days ago but took several hours today:

explain select min(ID) as minval,max(ID) as maxval from table_b where time_at >= to_timestamp('2024-03-30 00:00:00','yyyy-MM-dd HH24:mi:ss');
 QUERY PLAN 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
Result (cost=4298.54..4298.55 rows=1 width=64)
 InitPlan 1 (returns $0)
 -> Limit (cost=0.70..2149.27 rows=1 width=32)
 -> Index Scan using pk_b on table_b (cost=0.70..1181490202.27 rows=549896 width=32)
 Index Cond: ((ID)::text IS NOT NULL)
 Filter: (time_at >= to_timestamp('2024-03-30 00:00:00'::text, 'yyyy-MM-dd HH24:mi:ss'::text))
 InitPlan 2 (returns $1)
 -> Limit (cost=0.70..2149.27 rows=1 width=32)
 -> Index Scan Backward using pk_b on table_b table_b_1 (cost=0.70..1181490202.27 rows=549896 width=32)
 Index Cond: ((ID)::text IS NOT NULL)
 Filter: (time_at >= to_timestamp('2024-03-30 00:00:00'::text, 'yyyy-MM-dd HH24:mi:ss'::text))

The SQL is also simple — only one condition on a time column, with decent selectivity. However, this SQL did not use the time_at index but instead used the ID primary key index. This is the same LIMIT problem. Running ANALYZE is useless here — it’s better to rewrite the SQL.

After rewriting, the result came back instantly:

explain select min(ID||'') as minval,max(ID||'') as maxval from table_b where time_at >= to_timestamp('2024-03-30 00:00:00','yyyy-MM-dd HH24:mi:ss')

 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=1201418.86..1201418.87 rows=1 width=64)
 -> Index Scan using idx_time_at on table_b (cost=0.57..1195919.90 rows=549896 width=33)
 Index Cond: (time_at >= to_timestamp('2024-03-30 00:00:00'::text, 'yyyy-MM-dd HH24:mi:ss'::text))

This isn’t really an execution plan regression, because the plan didn’t change. A few days ago it had the same plan but ran fast — the reason is tied to data distribution and the LIMIT mechanism: when data is quickly found, it returns immediately (which is why the optimizer chose the primary key index); when it’s “unlucky” and the matching data is far away, it takes a very long time.

Summary
#

A classic case:

A small table with frequent updates
A long transaction preventing dead tuple reclamation
The long transaction itself was caused by an index selection problem due to sorting and LIMIT operations (ORDER BY, MAX/MIN, GROUP can all trigger this)

One incident, three classic PostgreSQL knowledge points — quite representative.

Analyzing a 5MB SQL That Consumed 70GB of Memory

Mon, 12 Aug 2024 00:00:00 +0000

Process Memory Analysis
#

"WAL writer process (PID 66902) was terminated by signal 6: Aborted",,,,,,,,,"","postmaster"

The log shows postmaster process 66902 was killed.

Checking OS-level process memory: since top doesn’t show PPID and ps doesn’t show USS, we need both:

USER PID PPID PRI %CPU %MEM VSZ RSS WCHAN S STARTED TIME COMMAND
postgres 211276 66478 19 8.7 10.6 57488380 56389972 - R 17:13:03 00:02:47 postgres: BIND
postgres 211277 66478 19 7.8 9.6 52294700 51127480 - R 17:13:03 00:02:31 postgres: BIND
postgres 222749 66478 19 22.7 9.3 51320000 49073368 - R 17:35:33 00:02:09 postgres: BIND
postgres 39513 66478 19 2.9 6.8 38651084 36354736 ep_poll S 16:13:03 00:02:43 postgres: idle

Using PPID to identify high-memory backend processes. Let’s examine process 211276:

[postgres@lzl]$ zcat /osw/oswtop/toposw.dat.gz |grep 211276

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
211276 postgres 20 0 3271756 1.1g 1.1g S 7.3 0.2 0:03.93 postgres
211276 postgres 20 0 3291784 1.3g 1.2g R 96.4 0.2 0:11.87 postgres
211276 postgres 20 0 7369628 6.0g 2.1g R 100.0 1.2 0:46.58 postgres
211276 postgres 20 0 17.0g 15.9g 2.1g R 100.0 3.2 1:16.70 postgres
211276 postgres 20 0 28.8g 27.7g 2.1g R 100.0 5.5 1:46.82 postgres
211276 postgres 20 0 41.4g 40.4g 2.1g R 100.0 8.0 2:16.99 postgres
211276 postgres 20 0 54.7g 53.7g 2.1g R 88.8 10.7 2:47.60 postgres
211276 postgres 20 0 66.5g 64.9g 2.1g R 34.7 12.9 3:22.76 postgres
211276 postgres 20 0 71.0g 68.2g 2.1g R 99.1 13.6 3:52.94 postgres
211276 postgres 20 0 74.9g 71.2g 2.1g R 100.0 14.2 4:23.05 postgres
211276 postgres 20 0 0 0 0 R 100.0 0.0 4:45.65 postgres

We can estimate private memory via RES - SHR = USS. Process 211276’s memory ballooned from ~1GB to ~70GB within minutes, then crashed. All memory growth was private process memory.

SQL Analysis
#

The PostgreSQL log shows a 5MB SQL containing 5,000+ UNION ALLs and 30,000+ bind variables.

The execution plan is over 70,000 lines long:

Append (cost=218196.51..218216.28 rows=1318 width=1628)
 InitPlan 1 (returns $0)
 -> Index Scan using table1 on table1nfo (cost=0.29..5.31 rows=1 width=40)
 Index Cond: ((col1)::text = 'xxx'::text)
 Filter: ((colcolcol)::text = 'xxx'::text)
 InitPlan 2 (returns $1)
 -> Index Scan using table1 on table1nfo table1nfo_1 (cost=0.29..5.31 rows=1 width=40)
 Index Cond: ((col1)::text = 'xxx'::text)
 Filter: ((colcolcol)::text = 'xxx'::text)
 ...
 InitPlan 10544 (returns $10543)
 -> Aggregate (cost=5.58..5.59 rows=1 width=32)
 -> Index Scan using table2 on table2col t_1317 (cost=0.56..5.58 rows=1 width=19)
 Index Cond: ((ididid)::text = 'xxx'::text)
 Filter: ((idididid)::text = '1'::text)

The plan structure is simple: ~10,000 sub-plans fetching data, then an Append to combine results.

This SQL monstrosity pushed a single backend process to 70GB. The root cause is clear: reduce the UNION ALLs and the problem goes away (which is indeed what happened). But if we dig deeper, many interesting questions arise:

Why did a 5MB SQL consume 70GB of memory?
Is the data itself related to memory usage? Was it caused by returning too many rows?
Is the memory from parsing cache or plan cache?
Why didn’t work_mem limit the operation memory, even though it’s set to a reasonable value?

Initial Analysis
#

A 5MB SQL cached in a backend would at minimum contain: metadata, parsed SQL, and plan cache information.

We’ve seen cases before where metadata cache (relcache) for hundreds of thousands of tables/partitions caused huge backend memory. But this database has few tables, so relcache can be preliminarily ruled out (later confirmed by memory dump).

Parsed SQL data shouldn’t be that large — a 5MB SQL parsed shouldn’t produce 70GB.

work_mem limitations and more:

work_mem only limits per-operation memory for sort and hash operations. This creates the “multiple sort/hash” problem: a single SQL with many sorts can use work_mem × N. PG 13 introduced hash_mem_multiplier to cap hash usage within one statement. But what about sorts? Currently no multiplier for sorts, though in practice it’s less of a problem — statements with dozens of sort nodes are rare, as they carry high cost, and the optimizer tends to place sorts late in the plan.

Here, work_mem is 128MB and the instance is PG 13+ with hash_mem_multiplier=1, so mass hash memory consumption can be ruled out. Furthermore, the execution plan above has zero sort or hash operations, confirming this is not a sort/hash issue.

So the earlier question: “Why didn’t work_mem limit operation memory?”

Because the SQL only has UNION ALL — no sort or hash operations at all. work_mem does not constrain memory here.

Other plan nodes:

No matter what, work_mem only (!) limits sort/hash. There are dozens of plan node types — are the rest all unconstrained?

Reproduction and Deep Analysis
#

Empty Table Reproduction
#

--Create empty table
 create table lzl1(col1 varchar(1));
--Query with many UNION ALLs
select col1 from lzl1 union all
select col1 from lzl1 union all 
...(5000 UNION ALLs, SQL size 150KB)
select col1 from lzl1

(Too many UNION ALLs may exceed max_stack_depth)

An empty table + many UNION ALLs immediately reproduces the memory spike. Moreover, after the SQL completes, the backend memory is reclaimed.

Since this is an empty table (0KB data file), we can rule out data as the cause. So: “Is the data itself related to memory? Was it caused by returning too many rows?” — No, data is not the main factor.

Strace System Call Analysis
#

While executing the SQL, capture system calls with strace -p:

 strace -p 198337 > strace.198337 2>&1

Quick primer on relevant Linux syscalls:

epoll_wait: Wait for an event. Idle processes sit in this state.
recvfrom: Receive a message from a socket.
getrusage: Get resource usage.
brk: Program break. Increasing it allocates memory to the process; decreasing it deallocates. malloc ultimately calls brk.
lseek: Reposition file offset.
write: Write to a file descriptor. Does not guarantee disk write.
sendto: Send a message on a socket.

Syscalls like lseek, write, sendto include fd (file descriptor) information:

lseek(37, 0, SEEK_END) = 0

/proc/[pid]/fd caches the process’s file descriptors. We can map an fd back to a relation — fd 37 is table lzl1:

[postgres@lzl]$ cd /proc/198337/fd
[postgres@lzl]$ ll 37
lrwx------ 1 postgres postgres 64 Jan 26 22:59 37 -> /pgdata/lzl/data13/base/16385/16386
[postgres@lzl]$ oid2name -d lzldb -f 16386
From database "lzldb":
 Filenode Table Name
----------------------
 16386 lzl1

The strace output is dense but structurally simple:

strace: Process 198337 attached
epoll_wait(4, [{EPOLLIN, {u32=44314568, u64=44314568}}], 1, -1) = 1
## step1
recvfrom(9, "Q\0\2p\372select col1 from lzl1 union"..., 8192, 0, NULL, NULL) = 8192
recvfrom(9, " all\nselect col1 from lzl1 union"..., 8192, 0, NULL, NULL) = 8192
recvfrom(9, " all\nselect col1 from lzl1 union"..., 8192, 0, NULL, NULL) = 8192
...
recvfrom(9, " all\nselect col1 from lzl1 union"..., 8192, 0, NULL, NULL) = 8192
recvfrom(9, " all\nselect col1 from lzl1 union"..., 8192, 0, NULL, NULL) = 4347
## step2
brk(NULL) = 0x34d5000
brk(0x3cd5000) = 0x3cd5000
brk(NULL) = 0x3cd5000
...
brk(NULL) = 0x88cd6000
brk(0x894d6000) = 0x894d6000
## step3
lseek(37, 0, SEEK_END) = 0
lseek(37, 0, SEEK_END) = 0
...
lseek(37, 0, SEEK_END) = 0
## step4
brk(NULL) = 0x89cd6000
brk(0x8a4d6000) = 0x8a4d6000
brk(NULL) = 0x8a4d6000
...
brk(NULL) = 0x8a516000
brk(0x8a556000) = 0x8a556000
## step5
write(2, "2024-01-26 23:08:01.800 CST [198"..., 165521) = 165521
brk(NULL) = 0x8a556000
brk(0x8a57d000) = 0x8a57d000
brk(NULL) = 0x8a57d000
brk(0x8a59f000) = 0x8a59f000
...
brk(NULL) = 0x8d449000
brk(0x8d46b000) = 0x8d46b000
brk(NULL) = 0x8d46b000
brk(0x8d48d000) = 0x8d48d000
#step6
lseek(37, 0, SEEK_END) = 0
lseek(37, 0, SEEK_END) = 0
...
lseek(37, 0, SEEK_END) = 0
#step7
brk(NULL) = 0x8dcb1000
brk(NULL) = 0x8dcb1000
brk(0x8c179000) = 0x8c179000
brk(NULL) = 0x8c179000
brk(NULL) = 0x8c179000
brk(NULL) = 0x8c179000
brk(0x8a526000) = 0x8a526000
...
brk(0x34d5000) = 0x34d5000
brk(NULL) = 0x34d5000
#step8
sendto(8, "\2\0\0\0\230\0\0\0\1@\0\0\1\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 152, 0, NULL, 0) = 152
sendto(9, "T\0\0\0\35\0\1col1\0\0\0\0\0\0\0\0\0\4\23\377\377\0\0\0\5\0\0C\0"..., 50, 0, NULL, 0) = 50
#step9
recvfrom(9, 0xddcf60, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(4, strace: Process 198337 detached
 <detached ...> 

Receive the UNION ALL SQL from fd=9 socket
brk allocates memory: process memory grows from 0x34d5000 (54MB) to 0x894d6000 (2.1GB)
lseek on table lzl1
Memory grows ~4MB
write to fd=2 (log file); memory grows ~48MB
lseek on table lzl1
Memory peaks at 0x8dcb1000 (2.1GB), then brk releases memory back down to 0x34d5000 (54MB) — exactly matching the start
Send result via socket
Receive empty message from fd=9

The strace doesn’t reveal much beyond the OS allocating and releasing memory for the process.

Memory Dump Analysis
#

pmap of the process during the memory spike:

[postgres@lzl pg_log]$ pmap -x 76207
76207: postgres: postgres lzldb [local] SELECT 
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 7984 2192 0 r-x-- postgres
0000000000dcc000 4 4 4 r---- postgres
0000000000dcd000 60 60 60 rw--- postgres
0000000000ddc000 200 60 60 rw--- [ anon ]
0000000001e49000 264 224 224 rw--- [ anon ]
0000000001e8b000 1812380 1804400 1804400 rw--- [ anon ]
...
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------- ------- ------- 
total kB 2089384 1810232 1807384

pmap doesn’t label the segments, but we can see the largest segment starts at address 0x1e49000. Checking smaps for more detail:

[postgres@lzl 76207]$ cat smaps |grep 1e49000 -A 30
01e49000-01e8b000 rw-p 00000000 00:00 0 [heap]
Size: 264 kB
...
01e8b000-70872000 rw-p 00000000 00:00 0 [heap]
Size: 1812380 kB
Rss: 1804400 kB
Pss: 1804400 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 1804400 kB
Referenced: 1804400 kB
Anonymous: 1804400 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB

Heap segment. PSS (private memory): 1.8GB!

(I tried using gdb to dump the 0x1e8b000-0x70872000 segment but it failed — not sure why. Suggestions welcome!)

Using gcore for a rough dump:

[postgres@lzl lzl]$ gcore -o /pgdata/lzl/gcore.dump 76207
[postgres@lzl lzl]$ strings gcore.dump.76207> text.dump.76207
[postgres@lzl lzl]$ ll -h
-rw-r----- 1 postgres postgres 2.0G Jan 26 17:29 gcore.dump.76207
-rw-r----- 1 postgres postgres 5.2M Jan 26 17:30 text.dump.76207

2GB virtual memory allocated, 1.8GB physical memory occupied — but only 5.2MB of actual data stored!

A rough hexdump reveals many memory holes:

[postgres@lzl lzl]$ hexdump -C gcore.dump.76207 |head -10000 |grep "00 00 00 00 00 00 00 00"|wc -l
3690

log_planner_stats and Other Info
#

To verify whether the plan cache is the culprit, enable logging for parse, planner, and executor phases:

 log_parser_stats = on
 log_planner_stats = on
 log_executor_stats = on

The logs show the parse phase uses little memory, while the planner consumes significantly more.

Planner stats log:

2024-01-26 18:01:41.592 CST [208503] LOG: PLANNER STATISTICS
2024-01-26 18:01:41.592 CST [208503] DETAIL: ! system usage stats:
 ! 0.048955 s user, 0.004996 s system, 0.054077 s elapsed
 ! [11.208034 s user, 1.313838 s system total]
 ! 2255352 kB max resident size
 ! 0/0 [0/352] filesystem blocks in/out
 ! 0/1315 [0/563859] page faults/reclaims, 0 [0] swaps
 ! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
 ! 0/0 [1/16] voluntary/involuntary context switches

2GB max resident size — consistent with the RES observed from the OS. This answers: “Is the memory from parsing cache or plan cache?” — The planner phase consumes the memory.

Inspecting TopMemoryContext
#

PostgreSQL manages backend private memory through MemoryContext. We can dump TopMemoryContext via gdb:

TopMemoryContext: 101488 total in 6 blocks; 48464 free (28 chunks); 53024 used
 pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 used
 TopTransactionContext: 8192 total in 1 blocks; 7720 free (0 chunks); 472 used
 TableSpace cache: 8192 total in 1 blocks; 2048 free (0 chunks); 6144 used
 RowDescriptionContext: 8192 total in 1 blocks; 6880 free (0 chunks); 1312 used
 MessageContext: 1854981336 total in 235 blocks; 7911304 free (9 chunks); 1847070032 used
...
Grand total: 1856104056 bytes in 431 blocks; 8226712 free (179 chunks); 1847877344 used

MessageContext accounts for 1.8GB — the largest consumer.

From src/backend/utils/mmgr/README:

MessageContext — this context holds the current command message from the frontend, as well as any derived storage that need only live as long as the current message (for example, in simple-Query mode the parse and plan trees can live here). This context will be reset, and any children deleted, at the top of each cycle of the outer loop of PostgresMain. This is kept separate from per-transaction and per-portal contexts because a query string might need to live either a longer or shorter time than any single transaction or portal.

When creating a prepared statement, the parse and plan trees will be built in a temporary context that’s a child of MessageContext.

MessageContext caches messages from the frontend, including derived parse and plan tree data.
Parse and plan trees are children of MessageContext — when MessageContext is reclaimed, parse and plan trees are reclaimed too.

This also explains the private memory reclamation: the plan tree data produced during the planner phase is a child of MessageContext. Once results are returned, MessageContext is reset and all children are freed. This matches the strace observation where memory after release matches memory before allocation exactly.

Summary
#

Answering the final question: “Why did a 5MB SQL consume 70GB of memory?”

The overwhelming majority of memory was consumed during plan creation. The planner allocated enormous amounts of memory. work_mem and hash_mem_multiplier can only constrain sort and hash operations — they cannot limit other memory operations during planning. The plan tree itself isn’t that large, but the allocation process creates massive memory holes: megabyte-scale data (metadata, parse tree, plan tree, etc.) ends up stored in gigabyte-scale memory regions.

These SQL, parse tree, and plan tree structures are all cached in MessageContext and its children. Once the result is sent back to the client, all memory from this phase is reclaimed.

Case Study: Analyzing Occasional Slow INSERT VALUES

Mon, 12 Aug 2024 00:00:00 +0000

The business team reported that INSERT VALUES occasionally became slow. By the time I checked the active sessions, the slow write problem had already subsided.

Later, I discovered that the slow write problem lasted less than half a minute, with INSERT VALUES taking 1-2 seconds. I wrote a script to capture active session information and managed to get the session data:

 wait_event | count
---------------------+-------
 [null] | 11
 WALRead | 1
 DataFileRead | 2
 BgWriterMain | 1
 WALWrite | 40
 AutoVacuumMain | 1
 ClientRead | 385
 LogicalLauncherMain | 1

The most abnormal wait event was WALWrite with 40 sessions.

Two of the WALWrite-waiting sessions looked like this:

 pid | usename | xact_start | state_change | wait_event | wait_event_type | state | partofquery
-------+----------+-------------------------------+-------------------------------+---------------+-----------------+--------+--------------------------------------------------------------
 144955 | lzluser11 | 2024-05-23 07:58:27.516574+08 | 2024-05-23 07:58:27.516588+08 | WALWrite | LWLock | active | insert into table1( 
 179869 | lzluser11 | 2024-05-23 07:58:28.116371+08 | 2024-05-23 07:58:28.116386+08 | WALWrite | IO | active | insert into table1( 

Let’s search the source code for WALWrite-related content:

 * WALWriteLock: must be held to write WAL buffers to disk (XLogWrite or
 * XLogFlush).

 /*
 * LWLockAcquireOrWait - Acquire lock, or wait until it's free
 *
 * The semantics of this function are a bit funky. If the lock is currently
 * free, it is acquired in the given mode, and the function returns true. If
 * the lock isn't immediately free, the function waits until it is released
 * and returns false, but does not acquire the lock.
 *
 * This is currently used for WALWriteLock: when a backend flushes the WAL,
 * holding WALWriteLock, it can flush the commit records of many other
 * backends as a side-effect. Those other backends need to wait until the
 * flush finishes, but don't need to acquire the lock anymore. They can just
 * wake up, observe that their records have already been flushed, and return.
 */

When WAL is written from WAL buffers to disk, the WALWriteLock must be held.

When a backend flushes WAL while holding WALWriteLock, it can also flush the commit records of other backends. Those other backends need to wait for this flush to finish, but they don’t need to acquire the lock afterward. If their WAL has been flushed, they can return directly (rather than flushing WAL again).

XLogFlush is extremely important. The key code in XLogFlush is in the for loop:

/*
 * Ensure that all XLOG data through the given position is flushed to disk.
 *
 * NOTE: this differs from XLogWrite mainly in that the WALWriteLock is not
 * already held, and we try to avoid acquiring it if possible.
 */
void
XLogFlush(XLogRecPtr record)
{
...

	/*
	 * Now wait until we get the write lock, or someone else does the flush
	 * for us.
	 */
	for (;;)
	{
		XLogRecPtr	insertpos;

		/* read LogwrtResult and update local state */
		SpinLockAcquire(&XLogCtl->info_lck);
		if (WriteRqstPtr < XLogCtl->LogwrtRqst.Write)
			WriteRqstPtr = XLogCtl->LogwrtRqst.Write;
		LogwrtResult = XLogCtl->LogwrtResult;
		SpinLockRelease(&XLogCtl->info_lck);

		/* done already? */
		if (record <= LogwrtResult.Flush)
			break;

		/*
		 * Before actually performing the write, wait for all in-flight
		 * insertions to the pages we're about to write to finish.
		 */
		insertpos = WaitXLogInsertionsToFinish(WriteRqstPtr);

		/*
		 * Try to get the write lock. If we can't get it immediately, wait
		 * until it's released, and recheck if we still need to do the flush
		 * or if the backend that held the lock did it for us already. This
		 * helps to maintain a good rate of group committing when the system
		 * is bottlenecked by the speed of fsyncing.
		 */
		if (!LWLockAcquireOrWait(WALWriteLock, LW_EXCLUSIVE))
		{
			/*
			 * The lock is now free, but we didn't acquire it yet. Before we
			 * do, loop back to check if someone else flushed the record for
			 * us already.
			 */
			continue;
		}

		/* Got the lock; recheck whether request is satisfied */
		LogwrtResult = XLogCtl->LogwrtResult;
		if (record <= LogwrtResult.Flush)
		{
			LWLockRelease(WALWriteLock);
			break;
		}

		/*
		 * Sleep before flush! By adding a delay here, we may give further
		 * backends the opportunity to join the backlog of group commit
		 * followers; this can significantly improve transaction throughput,
		 * at the risk of increasing transaction latency.
		 *
		 * We do not sleep if enableFsync is not turned on, nor if there are
		 * fewer than CommitSiblings other backends with active transactions.
		 */
		if (CommitDelay > 0 && enableFsync &&
			MinimumActiveBackends(CommitSiblings))
		{
			pg_usleep(CommitDelay);

			/*
			 * Re-check how far we can now flush the WAL. It's generally not
			 * safe to call WaitXLogInsertionsToFinish while holding
			 * WALWriteLock, because an in-progress insertion might need to
			 * also grab WALWriteLock to make progress. But we know that all
			 * the insertions up to insertpos have already finished, because
			 * that's what the earlier WaitXLogInsertionsToFinish() returned.
			 * We're only calling it again to allow insertpos to be moved
			 * further forward, not to actually wait for anyone.
			 */
			insertpos = WaitXLogInsertionsToFinish(insertpos);
		}

		/* try to write/flush later additions to XLOG as well */
		WriteRqst.Write = insertpos;
		WriteRqst.Flush = insertpos;

		XLogWrite(WriteRqst, false);

		LWLockRelease(WALWriteLock);
		/* done */
		break;
	}
...
}

The XLogFlush function is the main function for flushing dirty WAL:

Check if the dirty WAL that needs to be flushed has already been flushed by someone else. If so, return directly.
Try to acquire the lock WALWriteLock in exclusive mode, retrying continuously until the lock is acquired.
Once the lock is acquired, check again if the dirty WAL that needs to be flushed has already been flushed by someone else. If so, release WALWriteLock and return (during the lock acquisition wait, someone else might have flushed the dirty WAL — if so, there’s nothing to do).
Wait for commit_delay milliseconds, and if the number of concurrent committing transactions exceeds commit_siblings, update the WAL write position to form a group commit. This step currently doesn’t apply because CommitDelay defaults to 0, effectively meaning group commit is not enabled.
Call XLogWrite to write the log, release WALWriteLock after completion.

XLogFlush for flushing dirty WAL needs to check whether the currently requested dirty WAL has already been written. If not, it will hold WALWriteLock until the XLogWrite function completes writing the log. XLogWrite is the specific function for writing WAL, such as writing to which position on which page.

Returning to the wait events from active sessions, the IO:WALWrite wait is relatively easy to understand, but how do we confirm whether LWLock:WALWrite is a problem?

From the XLogFlush function logic, we know that WALWriteLock is an exclusive LWLock that PostgreSQL acquires when writing dirty WAL (this makes sense — WAL commit information is written sequentially and can only be written in exclusive mode; you can’t let whoever writes fastest write first, as that could easily corrupt data). It’s a serialized write of WAL commit information.

Understanding this part of the logic, looking back at pg_stat_activity, we can see that there was only 1 IO:WALWrite, while there were dozens of LWLock:WALWrite waits.

Although we can’t directly see the LWLock blocking chain, we can infer from the source code that LWLock:WALWrite is waiting on IO:WALWrite.

The official documentation has a section about XLogFlush and adjusting WAL buffers:

Normally, WAL buffers should be written and flushed by an XLogFlush request, which is made, for the most part, at transaction commit time to ensure that transaction records are flushed to permanent storage. On systems with high WAL output, XLogFlush requests might not occur often enough to prevent XLogInsertRecord from having to do writes. On such systems one should increase the number of WAL buffers by modifying the wal_buffers parameter. When full_page_writes is set and the system is very busy, setting wal_buffers higher will help smooth response times during the period immediately following each checkpoint.

Under normal circumstances, WAL buffers are flushed by XLogFlush, for example during transaction commit to write WAL logs to disk. If the WAL log volume is large but XLogFlush is not triggered frequently enough (meaning mostly large transactions), XLogInsertRecord needs to write uncommitted WAL records — meaning the WAL buffer is insufficient. In this case, increasing wal_buffers may slightly help with system response time.

There are two commonly used internal WAL functions: XLogInsertRecord and XLogFlush. XLogInsertRecord is used to place a new record into the WAL buffers in shared memory. If there is no space for the new record, XLogInsertRecord will have to write (move to kernel cache) a few filled WAL buffers

Combined with a description from the XLogInsertRecord function:

	 * We have now done all the preparatory work we can without holding a
	 * lock or modifying shared state. From here on, inserting the new WAL
	 * record to the shared WAL buffer cache is a two-step process:
	 *
	 * 1. Reserve the right amount of space from the WAL. The current head of
	 *	 reserved space is kept in Insert->CurrBytePos, and is protected by
	 *	 insertpos_lck.
	 *
	 * 2. Copy the record to the reserved WAL space. This involves finding the
	 *	 correct WAL buffer containing the reserved space, and copying the
	 *	 record in place. This can be done concurrently in multiple processes.

The XLogInsertRecord function is used to place new WAL records into the WAL buffer:

Writing requires reserving a certain amount of space.
Copy the WAL record to the reserved WAL space (presumably the reserved space in the WAL buffer). Multiple processes can execute this in parallel.

Copying WAL records to the WAL buffer can be done in parallel. This is unlikely to be a bottleneck since it’s an in-memory copy with parallelism.

But XLogFlush is different — it holds an exclusive LWLock throughout the write. So, in scenarios with high concurrency and small transactions, increasing WAL buffers theoretically won’t be very effective.

At this point, we can rule out wal_buffers memory tuning and focus our attention on I/O. Looking at the I/O-related wait counts in pg_stat_activity:

DataFileRead	4
DataFileExtend	1
WALWrite		1
WALRead			1

The INSERT VALUES slowness lasted less than a minute and was not normally present. However, looking at the normal session information, I/O class WALWrite waits were almost always there:

 pid | usename | xact_start | state_change | wait_event | wait_event_type | state | partofquery
-------+----------+-------------------------------+-------------------------------+---------------+-----------------+--------+--------------------------------------------------------------
 72668 | lzluser11 | 2024-05-23 09:32:20.828394+08 | 2024-05-23 09:32:20.82841+08 | WALWrite | IO | active | insert into table1( +
 77215 | lzluser11 | 2024-05-23 09:33:01.342541+08 | 2024-05-23 09:33:01.342552+08 | WALWrite | IO | active | insert into table1 +
 94904 | lzluser11 | 2024-05-23 09:34:32.442309+08 | 2024-05-23 09:34:32.442323+08 | WALWrite | IO | active | insert into table1 +
 88024 | lzluser11 | 2024-05-23 09:36:28.779086+08 | 2024-05-23 09:36:28.779311+08 | WALWrite | IO | active | UPDATE table2 SET +
 103236 | lzluser11 | 2024-05-23 09:37:04.144283+08 | 2024-05-23 09:37:04.144302+08 | WALWrite | IO | active | insert into table1 +
 47342 | lzluser11 | 2024-05-23 09:37:09.192683+08 | 2024-05-23 09:37:09.192699+08 | WALWrite | IO | active | insert into table1 +
 75399 | lzluser11 | 2024-05-23 09:45:30.743023+08 | 2024-05-23 09:45:30.743024+08 | WALWrite | IO | active | update table1 +
 221993 | lzluser11 | 2024-05-23 09:46:16.184532+08 | 2024-05-23 09:46:16.184541+08 | WALWrite | IO | active | insert into table1 

However, checking the I/O performance at that time, writing 15 MB/s was not high — in fact, it was relatively low compared to other time periods, and w_await was also very low:

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-322 0.00 0.00 187.00 1515.00 3572.00 15344.00 22.23 2.05 1.20 9.39 0.18 0.15 25.70

There was no strong evidence pointing to a storage performance issue.

At present, it appears to be transient lock contention during concurrent INSERT VALUES small transactions when flushing WAL. We can rule out the following options:

Concurrent small transactions — no need to adjust WAL buffers
WAL log volume is not large — no need to enable log compression
Not many FPIs (Full Page Images) — no need to adjust checkpoint
I/O pressure is not high — no need to improve I/O performance

At minimum, the following optimizations can be made:

Enable database group commit (can be deferred if concerned about risk; testing required)
Batch multiple INSERT VALUES statements at the application level to reduce WALWriteLock contention

Case Study: Logical Replication Deadlocks Checkpoint, Walsender, and Backup

Mon, 12 Aug 2024 00:00:00 +0000

Problem Symptoms
#

The backup process (pg_start_backup()) was blocked by the checkpointer, and the checkpointer was blocked by the logical replication walsender. The database was still serving queries, but backup, checkpoint, and logical replication were all completely hung.

Two processes in pg_stat_activity showed an unusual wait event: replication_slot_io.

[postgres@hostlzl:6666/postgres][04-08.16:50:28]=> select * from pg_stat_activity where pid=173038 \gx
-[ RECORD 1 ]----+------------------------------
datid | 17630
datname | lzldb
pid | 173038
usesysid | 35157
usename | repuser
application_name | PostgreSQL JDBC Driver
client_addr | 30.88.75.58
client_hostname | [null]
client_port | 37623
backend_start | 2024-04-02 11:40:07.75022+08
xact_start | [null]
query_start | [null]
state_change | 2024-04-02 11:40:07.764475+08
wait_event_type | LWLock
wait_event | replication_slot_io
state | active
backend_xid | [null]
backend_xmin | [null]
query | 
backend_type | walsender

Time: 6.658 ms
[postgres@hostlzl:6666/postgres][04-08.16:50:34]=> select * from pg_stat_activity where pid=12729\gx
-[ RECORD 1 ]----+------------------------------
datid | [null]
datname | [null]
pid | 12729
usesysid | [null]
usename | [null]
application_name | 
client_addr | [null]
client_hostname | [null]
client_port | [null]
backend_start | 2024-04-02 11:23:03.343116+08
xact_start | [null]
query_start | [null]
state_change | [null]
wait_event_type | LWLock
wait_event | replication_slot_io
state | [null]
backend_xid | [null]
backend_xmin | [null]
query | 
backend_type | checkpointer

One walsender and one checkpointer. Both were started on April 2. Let’s check the walsender 173038 logs:

--repuser log
2024-04-02 11:40:07.750 CST,,,173038,"30.88.75.58:37623",660b7e17.2a3ee,1,"",2024-04-02 11:40:07 CST,,0,LOG,00000,"connection received: host=30.88.75.58 port=37623",,,,,,,,,""
2024-04-02 11:40:07.756 CST,"repuser","lzldb",173038,"30.88.75.58:37623",660b7e17.2a3ee,2,"authentication",2024-04-02 11:40:07 CST,32/30,0,LOG,00000,"replication connection authorized: user=repuser",,,,,,,,,""
2024-04-02 11:40:07.765 CST,"repuser","lzldb",173038,"30.88.75.58:37623",660b7e17.2a3ee,3,"idle",2024-04-02 11:40:07 CST,32/0,0,LOG,00000,"starting logical decoding for slot ""pg_lzldb_lzldb_ora_pgdb_pgdb""","Streaming transactions committing after 4263/42E6EF88, reading WAL from 4263/41DAEBD0.",,,,,,,,"PostgreSQL JDBC Driver"
2024-04-02 11:40:07.765 CST,"repuser","lzldb",173038,"30.88.75.58:37623",660b7e17.2a3ee,4,"idle",2024-04-02 11:40:07 CST,32/0,0,LOG,00000,"logical decoding found consistent point at 4263/41DAEBD0","There are no running transactions.",,,,,,,,"PostgreSQL JDBC Driver"

Walsender 173038 only shows startup information. After that, no more log output — it likely hung from the very start.

Scrolling back a bit, we can find an earlier walsender for the same replication slot (different PID, same slot name):

--84918 earlier startup logs
2024-04-02 11:30:07.498 CST,,,84918,"30.88.75.58:54898",660b7bbf.14bb6,1,"",2024-04-02 11:30:07 CST,,0,LOG,00000,"connection received: host=30.88.75.58 port=54898",,,,,,,,,""
2024-04-02 11:30:07.504 CST,"repuser","lzldb",84918,"30.88.75.58:54898",660b7bbf.14bb6,2,"authentication",2024-04-02 11:30:07 CST,30/3,0,LOG,00000,"replication connection authorized: user=repuser",,,,,,,,,""
2024-04-02 11:30:07.514 CST,"repuser","lzldb",84918,"30.88.75.58:54898",660b7bbf.14bb6,3,"idle",2024-04-02 11:30:07 CST,30/0,0,LOG,00000,"starting logical decoding for slot ""pg_lzldb_lzldb_ora_pgdb_pgdb""","Streaming transactions committing after 4263/41DADE38, reading WAL from 4263/358F1340.",,,,,,,,"PostgreSQL JDBC Driver"
2024-04-02 11:30:07.516 CST,"repuser","lzldb",84918,"30.88.75.58:54898",660b7bbf.14bb6,4,"idle",2024-04-02 11:30:07 CST,30/0,0,LOG,00000,"logical decoding found consistent point at 4263/358F1340","There are no running transactions.",,,,,,,,"PostgreSQL JDBC Driver"

2024-04-02 11:36:07.061 CST,"repuser","lzldb",86630,"30.88.75.58:45227",660b7bca.15266,5,"idle",2024-04-02 11:30:18 CST,30/0,0,ERROR,XX000,"could not write to file ""pg_replslot/pg_lzldb_lzldb_ora_pgdb_pgdb/state.tmp"": Cannot allocate memory",,,,,,,,,"PostgreSQL JDBC Driver"
2024-04-02 11:36:40.151 CST,"repuser","lzldb",86630,"30.88.75.58:45227",660b7bca.15266,6,"idle",2024-04-02 11:30:18 CST,,0,LOG,00000,"disconnection: session time: 0:06:21.760 user=repuser database=lzldb host=30.88.75.58 port=45227",,,,,,,,,"PostgreSQL JDBC Driver"

This replication slot was also started at 11:30:07. Six minutes later, it failed to write state.tmp due to memory exhaustion.

The checkpointer process 12729 also reported the same state.tmp error — "pg_replslot/pg_lzldb_lzldb_ora_pgdb_pgdb/state.tmp"": File exists". This error appeared ~30 seconds after the replication slot error:

--checkpoint log
2024-04-02 11:36:39.925 CST,,,12729,,660b7a17.31b9,4,,2024-04-02 11:23:03 CST,,0,LOG,58P02,"could not create file ""pg_replslot/pg_lzldb_lzldb_ora_pgdb_pgdb/state.tmp"": File exists",,,,,,,,,""
2024-04-02 11:36:40.151 CST,,,12729,,660b7a17.31b9,5,,2024-04-02 11:23:03 CST,,0,LOG,00000,"checkpoint complete: wrote 334 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.108 s, sync=0.082 s, total=217.083 s; sync files=139, longest=0.004 s, average=0.000 s; distance=2295 kB, estimate=2295 kB",,,,,,,,,""
2024-04-02 11:48:03.414 CST,,,12729,,660b7a17.31b9,6,,2024-04-02 11:23:03 CST,,0,LOG,00000,"checkpoint starting: time",,,,,,,,,""

After this, the checkpointer produced no more log output — it hung, just like the walsender.

Searching for pg_replslot/pg_lzldb_lzldb_ora_pgdb_pgdb/state.tmp"": File exists" quickly leads to a community thread: https://www.postgresql.org/message-id/14b3454f-2d68-c637-68e4-2b42ff976168%40postgrespro.ru

The actual fix landed in PG 12.3:

Ensure that a replication slot’s io_in_progress_lock is released in failure code paths (Pavan Deolasee) This could result in a walsender later becoming stuck waiting for the lock.

Deep Dive
#

We found the bug, but several questions remain:

Why did the walsender and checkpointer hang?
Who is blocking whom — the walsender or the checkpointer?
How was this triggered?
What are the solutions?

Source Code Analysis
#

Current version: 11.5.

Pstack of both processes:

[postgres@hostlzl:lzldb:6666: /pg/pg6666/data/pg_log]$ pstack 173038 ##walsender
#0 0x00002b9eec171a0b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00002b9eec171a9f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00002b9eec171b3b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00000000006b2512 in PGSemaphoreLock (sema=0x2b9ef5fdb0b8) at pg_sema.c:316
#4 0x000000000071e94c in LWLockAcquire (lock=lock@entry=0x2babd8cee5b8, mode=mode@entry=LW_EXCLUSIVE) at lwlock.c:1243
#5 0x00000000006ef7cb in SaveSlotToPath (slot=0x2babd8cee500, dir=dir@entry=0x7ffcaffd79f0 "pg_replslot/pg_lzldb_lzldb_ora_pgdb_pgdb", elevel=elevel@entry=20) at slot.c:1249
#6 0x00000000006f0515 in ReplicationSlotSave () at slot.c:653
#7 0x00000000006d75d8 in LogicalConfirmReceivedLocation (lsn=<optimized out>) at logical.c:1049
#8 0x00000000006d774d in LogicalIncreaseXminForSlot (current_lsn=current_lsn@entry=72994075200640, xmin=xmin@entry=1241611955) at logical.c:914
#9 0x00000000006e0fb3 in SnapBuildProcessRunningXacts (builder=builder@entry=0x23146c0, lsn=72994075200640, running=running@entry=0x22e8978) at snapbuild.c:1146
#10 0x00000000006d484c in DecodeStandbyOp (buf=0x7ffcaffd7eb0, buf=0x7ffcaffd7eb0, ctx=0x2209820) at decode.c:318
#11 LogicalDecodingProcessRecord (ctx=0x2209820, record=<optimized out>) at decode.c:121
#12 0x00000000006e50e0 in XLogSendLogical () at walsender.c:2799
#13 0x00000000006e7122 in WalSndLoop (send_data=send_data@entry=0x6e5080 <XLogSendLogical>) at walsender.c:2162
#14 0x00000000006e7d91 in StartLogicalReplication (cmd=0x22eedd8) at walsender.c:1109
#15 exec_replication_command (cmd_string=cmd_string@entry=0x2210c48 "START_REPLICATION SLOT pg_lzldb_lzldb_ora_pgdb_pgdb LOGICAL 4263/42E6EF88 (\"add-tables\" 'public.acr_finance_coa_partition_17_01,public.acr_finance_coa_partition_17_02,public.acr_finance_coa_part"...) at walsender.c:1541
#16 0x000000000072c899 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x2216f78, dbname=0x2216c98 "lzldb", username=<optimized out>) at postgres.c:4178
#17 0x000000000047e481 in BackendRun (port=0x20eda0) at postmaster.c:4358
#18 BackendStartup (port=0x20eda0) at postmaster.c:4030
#19 ServerLoop () at postmaster.c:1707
#20 0x00000000006c4359 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x21dbe90) at postmaster.c:1380
#21 0x000000000047eefb in main (argc=3, argv=0x21dbe90) at main.c:228


[postgres@hostlzl:lzldb:6666: /pg/pg6666/data/pg_wal]$ pstack 12729 ##checkpointer
#0 0x00002b9eec171a0b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00002b9eec171a9f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00002b9eec171b3b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00000000006b2512 in PGSemaphoreLock (sema=0x2b9ef5fdcd38) at pg_sema.c:316
#4 0x000000000071e94c in LWLockAcquire (lock=lock@entry=0x2babd8cee5b8, mode=mode@entry=LW_EXCLUSIVE) at lwlock.c:1243
#5 0x00000000006ef7cb in SaveSlotToPath (slot=slot@entry=0x2babd8cee500, dir=dir@entry=0x7ffcaffd6ee0 "pg_replslot/pg_lzldb_lzldb_ora_pgdb_pgdb", elevel=elevel@entry=15) at slot.c:1249
#6 0x00000000006f11a7 in CheckPointReplicationSlots () at slot.c:1100
#7 0x00000000004f674f in CheckPointGuts (checkPointRedo=72994093982360, flags=flags@entry=128) at xlog.c:9146
#8 0x00000000004fcc77 in CreateCheckPoint (flags=flags@entry=128) at xlog.c:8937
#9 0x00000000006b8312 in CheckpointerMain () at checkpointer.c:491
#10 0x000000000050ba15 in AuxiliaryProcessMain (argc=argc@entry=2, argv=argv@entry=0x7ffcaffd7540) at bootstrap.c:451
#11 0x00000000006c1cb9 in StartChildProcess (type=CheckpointerProcess) at postmaster.c:5337
#12 0x00000000006c2f5a in reaper (postgres_signal_arg=<optimized out>) at postmaster.c:2867
#13 <signal handler called>
#14 0x00002b9eed5ba783 in __select_nocancel () from /lib64/libc.so.6
#15 0x000000000047db38 in ServerLoop () at postmaster.c:1671
#16 0x00000000006c4359 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x21dbe90) at postmaster.c:1380
#17 0x000000000047eefb in main (argc=3, argv=0x21dbe90) at main.c:228

The key observation is the LWLockAcquire frame. Both the walsender and the checkpointer are trying to acquire the same LWLOCK address in exclusive mode: lock=lock@entry=0x2babd8cee5b8, mode=mode@entry=LW_EXCLUSIVE — waiting indefinitely.

The function right above LWLockAcquire is SaveSlotToPath.

Looking at the source in src/backend/replication/slot.c, the critical function SaveSlotToPath:

//SaveSlotToPath stores slot state
static void
SaveSlotToPath(ReplicationSlot *slot, const char *dir, int elevel)
{	//11.5 code
	char		tmppath[MAXPGPATH];
	char		path[MAXPGPATH];
	int			fd;
	ReplicationSlotOnDisk cp;
	bool		was_dirty;
...
	/* and don't do anything if there's nothing to write */
	if (!was_dirty)
		return;
	//Acquire LWLock in exclusive mode at function entry
	LWLockAcquire(&slot->io_in_progress_lock, LW_EXCLUSIVE);

...
	//Note the fd logic — the error matches the second walsender error
	fd = OpenTransientFile(tmppath, O_CREAT | O_EXCL | O_WRONLY | PG_BINARY);
	if (fd < 0)
	{
		ereport(elevel,
				(errcode_for_file_access(),
				 errmsg("could not create file \"%s\": %m",
						tmppath)));
		return;
...
	//The logic for writing to fd — the error matches the first walsender error
	if ((write(fd, &cp, sizeof(cp))) != sizeof(cp))
	{
		int			save_errno = errno;

		pgstat_report_wait_end();
		CloseTransientFile(fd);

		/* if write didn't set errno, assume problem is no disk space */
		errno = save_errno ? save_errno : ENOSPC;
		ereport(elevel,
				(errcode_for_file_access(),
				 errmsg("could not write to file \"%s\": %m",
						tmppath)));
		return;
	}
...
	LWLockRelease(&slot->io_in_progress_lock);	//Release LWLock at end of function
	}

SaveSlotToPath acquires LWLockAcquire on the slot’s io_in_progress_lock in LW_EXCLUSIVE mode — very similar to the wait event name: io_in_progress_lock ↔ replication_slot_io.

At the end of the function, LWLockRelease releases the lock.

But in both if branches, there is no LWLockRelease — the function just returns directly!

The PostgreSQL log shows “could not create file” for tmppath, meaning the code hit one of those two if branches — either the write to state.tmp failed branch or the create state.tmp failed branch.

Let’s reconstruct the timeline from the logs:

11:36:07: Logical replication first error — “could not write to file … state.tmp”. Replication link dies.
11:36:39: Checkpointer error — “could not create file … state.tmp”. One second later, checkpoint “completes” with 0 dirty buffers, 0 WAL.
11:40:07: Logical replication starts again. No more output.
11:48:03: Checkpointer triggers start again. No more output.

Important: the first and second logical replication connections belong to different walsender PIDs; the first and second checkpoint entries belong to the same checkpointer PID.

Fault mechanism reconstructed:

Logical replication walsender, due to memory pressure, fails to write state.tmp, leaving a residual state.tmp file behind.
The checkpointer, encountering the residual state.tmp, enters the if (fd < 0) branch in SaveSlotToPath after acquiring the LWLock in exclusive mode — and returns without releasing the LWLock.
A new walsender starts for logical replication and tries to acquire the LWLock at the top of SaveSlotToPath — waits indefinitely.
The checkpointer triggers a new checkpoint and also tries to acquire the LWLock at the top of SaveSlotToPath — waits indefinitely.

With the mechanism clear, the answers follow:

Why did the walsender and checkpointer hang? Residual state.tmp. The checkpointer held the LWLock without releasing it. Both walsender and checkpointer wait indefinitely.
Who blocks whom? The checkpointer blocks the walsender.
How was it triggered? The previous walsender exhausted memory, leaving an uncleaned state.tmp.
Solutions? Force restart the database.

Reproduction
#

For background on PostgreSQL logical replication, refer to: PG inner workings: Logical Replication. Key commands:

select pg_create_logical_replication_slot('logical_test','test_decoding');
pg_recvlogical -h 127.0.0.1 -p 5558 -d lzldb -U lzl --slot=logical_test --start -f recv.sql &

The slot and replication link are ready:

postgres=# select pid,usename,xact_start,state_change,wait_event,state,query from pg_stat_activity where state<>'idle' order by xact_start ;
 pid | usename | xact_start | state_change | wait_event | state | query 
 
-------+----------+-------------------------------+-------------------------------+---------------------+--------+----------------------------------------------------------------------------------------------
--------------------------------------
 59916 | postgres | 2024-04-08 21:14:32.015534+08 | 2024-04-08 21:14:32.015545+08 | | active | select pid,usename,xact_start,state_change,wait_event,state,query from pg_stat_activity wher
e state<>'idle' order by xact_start ;
 59791 | lzl | | 2024-04-08 21:14:19.566112+08 | WalSenderWaitForWAL | active | SELECT pg_catalog.set_config('search_path', '', false)

postgres=# select pid,usename,application_name,backend_start,state,pg_walfile_name_offset(sent_lsn) sentoffset,pg_walfile_name_offset(write_lsn) writeoffset,pg_walfile_name_offset(flush_lsn) flushoffset from pg_stat_replication;
 pid | usename | application_name | backend_start | state | sentoffset | writeoffset | flushoffset 
-------+---------+------------------+------------------------------+-----------+------------------------------------+------------------------------------+------------------------------------
 59791 | lzl | pg_recvlogical | 2024-04-08 21:14:19.56364+08 | streaming | (000000010000000000000001,6612032) | (000000010000000000000001,6612032) | (000000010000000000000001,6612032)

Since the problem is caused by state.tmp, just touch it under pg_replslot:

[postgres@testhost logical_test]$ pwd
/pgdata/lzl/data11/pg_replslot/logical_test

pg_recvlogical immediately errors:

 pg_recvlogical: unexpected termination of replication stream: ERROR: could not create file "pg_replslot/logical_test/state.tmp": File exists

Manual CHECKPOINT hangs:

lzldb=# checkpoint;

--hang

Now check the walsender and session states:

postgres=> select * from pg_stat_activity ;
 datid | datname | pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start 
 | state_change | wait_event_type | wait_event | state | backend_xid | backend_xmin | query | backend_type 
-------+----------+-------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+------------------------------
-+-------------------------------+-----------------+---------------------+--------+-------------+--------------+--------------------------------------------------------+------------------------------
... 
 | | Activity | LogicalLauncherMain | | | | | logical replication launcher
 | 2024-04-08 21:25:55.058523+08 | | | active | | | checkpoint; | client backend
 16384 | lzldb | 77638 | 16385 | lzl | pg_recvlogical | 127.0.0.1 | | 56928 | 2024-04-08 21:25:17.495833+08 | | 2024-04-08 21:25:17.497754+08
 | 2024-04-08 21:25:17.498329+08 | LWLock | replication_slot_io | active | | | SELECT pg_catalog.set_config('search_path', '', false) | walsender
 | | LWLock | replication_slot_io | | | | | checkpointer

Perfectly reproduced — two replication_slot_io wait events.

PG 12.3 Code Fix
#

//Here showing 15.3, which has an extra save_errno vs 12.3
static void
SaveSlotToPath(ReplicationSlot *slot, const char *dir, int elevel)
{	
	fd = OpenTransientFile(tmppath, O_CREAT | O_EXCL | O_WRONLY | PG_BINARY);
	if (fd < 0)
	{
		/*
		 * If not an ERROR, then release the lock before returning. In case
		 * of an ERROR, the error recovery path automatically releases the
		 * lock, but no harm in explicitly releasing even in that case. Note
		 * that LWLockRelease() could affect errno.
		 */
		int			save_errno = errno;

		LWLockRelease(&slot->io_in_progress_lock);
		errno = save_errno;
		ereport(elevel,
				(errcode_for_file_access(),
				 errmsg("could not create file \"%s\": %m",
						tmppath)));
		return;
	}
...
LWLockRelease(&slot->io_in_progress_lock);

}	

In every if branch, LWLockRelease is called before returning. This eliminates the logical vulnerability where the LWLock is not released in certain code paths. The code is clearly more robust.

Solution Analysis
#

Deleting state.tmp won’t help — the LWLock is already held; the file was just the trigger.
Restarting the replication link or killing the downstream won’t help — the checkpointer is the one holding the LWLock.
The checkpointer cannot be killed directly. The only solution in this state is a force restart to perform instance recovery. A normal shutdown is impossible because CHECKPOINT is blocked.
The ultimate fix: upgrade to PG 12.3 or later.

(I also tried using gdb to call LWLockRelease with the LWLock address from pstack — it crashed the test instance immediately. Not recommended.)

Summary
#

Logical replication is one of the most significant feature enhancements in recent PostgreSQL releases. Early versions did have many issues and pitfalls. PostgreSQL’s ambitious logical replication approach shows genuine innovation, and the community continuously refines and strengthens it — nearly every minor release includes many logical replication updates. This case is a real-world example: the logical replication code is clearly becoming more robust.

Logical replication has a lot of depth. Recommended reading: PG Inner Workings: Logical Replication

Case Study: Predicate Out-of-Bounds and Prepared Statement Issues in PostgreSQL

Mon, 12 Aug 2024 00:00:00 +0000

The Phenomenon
#

Case: The execution plan changed and chose the wrong index, causing SQL performance to degrade from milliseconds to seconds. After collecting statistics, the business SQL was still slow. Ultimately, the problem was resolved by dropping the DAILY_DATE time index and creating a composite index on (DAILY_DATE, A_ID).

Questions:

Why did the optimizer choose the DAILY_DATE index instead of the more selective A_ID index?
Why did collecting statistics have no effect?

Stale Statistics
#

-- Simplified SQL
select
 *
 from tablzl
 where A_ID = $1
 AND IS_DELETE = 'N'
 AND DAILY_DATE = to_date($2, 'yyyyMMdd')
 and PARTITION_KEY >= $3 
 and PARTITION_KEY <= $4

The optimizer chose the DAILY_DATE index instead of the more selective A_ID index:

Append (cost=0.44..8.83 rows=2 width=204)
 -> Index Scan using tablzl_p202401_DAILY_DATE_idx on tablzl_p202401 tablzl_1 (cost=0.44..5.47 rows=1 width=203)
 Index Cond: (DAILY_DATE = to_date('20240223'::text, 'yyyyMMdd'::text))
 Filter: ((partition_key >= 202401) AND (partition_key <= 202402) AND ((A_ID)::text = 'ID1234567890987654321'::text) AND ((is_delete)::text = 'N'::text))
 -> Index Scan using tablzl_p202402_DAILY_DATE_idx on tablzl_p202402 tablzl_2 (cost=0.44..3.35 rows=1 width=204)
 Index Cond: (DAILY_DATE = to_date('20240223'::text, 'yyyyMMdd'::text))
 Filter: ((partition_key >= 202401) AND (partition_key <= 202402) AND ((A_ID)::text = 'ID1234567890987654321'::text) AND ((is_delete)::text = 'N'::text))

For the p202401 partition, whether it uses the DAILY_DATE or A_ID index doesn’t make much difference, because the January partition has no data for February 23.
For the p202402 partition, whether it uses the DAILY_DATE or A_ID index makes a huge difference. Using the DAILY_DATE index, its estimated cost is 3.35 with rows=1, but in reality there are millions of rows, causing it to run for 2 seconds.

The statistics for p202402 contain MCV (Most Common Values):

= select * from pg_stats where tablename='tablzl_p202402' and attname='DAILY_DATE' \gx
most_common_vals | {2024-02-21,2024-02-20,2024-02-22,2024-02-10,2024-02-15,2024-02-19,2024-02-16,2024-02-18,2024-02-17,2024-02-14,2024-02-11,2024-02-07,2024-02-12,2024-02-06,2024-02-08,2024-.
 |.02-09,2024-02-03,2024-02-05,2024-02-01,2024-02-02,2024-01-31,2024-02-13,2024-02-04}
most_common_freqs | {0.0481,0.047766667,0.0466,0.0449,0.0441,0.043833334,0.043733332,0.043466665,0.043133333,0.043066666,0.042366665,0.041866668,0.041366667,0.041366667,0.039766666,0.0394,0.039333332,0..
 |.038766667,0.03863333,0.0381,0.038066667,0.037966665,0.037566666,0.036733333}

Calculate the sum of MCV frequencies:

= select 0.0481+0.047766667+0.0466+0.0449+0.0441+0.043833334+0.043733332+0.043466665+0.043133333+0.043066666+0.042366665+0.041866668+0.041366667+0.041366667+0.039766666+0.0394+0.039333332+0.038766667+0.03863333+0.0381+0.038066667+0.037966665+0.037566666+0.036733333;
 ?column? 
-------------
 0.999999990

It’s exactly 1, meaning the planner estimates that days 1-22 represent all the data in this partition, and day 23 should have 0 rows. So when estimating rows for day 23 data, the planner assumes rows=1, and thus chooses the DAILY_DATE index. In reality, day 23 had millions of rows.

Essentially, this is a problem caused by stale statistics. Why were the first 22 days fine, and why didn’t day 23 trigger automatic collection?

= select relname,reloptions from pg_class where relname='tablzl';
 relname | reloptions 
----------------------------+------------
 tablzl | [null]

= show autovacuum_analyze_scale_factor;
 autovacuum_analyze_scale_factor 
---------------------------------
 0.1

The trigger threshold defaults to 0.1 — auto-ANALYZE only triggers when data changes reach 1/10. This is a monthly partition, with data inserted and updated daily. Early in the month, writing 2 million rows per day would trigger multiple ANALYZEs (the threshold of 50 can be ignored), but at month end, for example on day 23, writing 2 million rows would not trigger ANALYZE because only 1/23 of the data changed. In this scenario, data was also updated after insertion — 2 million inserts and 2 million updates — so the data change on day 23 was about 1/11, just barely not triggering ANALYZE. This also explains why the first 20 days ran stably.

Additionally, since the data change threshold is a ratio, as long as the daily data change volume is relatively uniform, this month-end statistics inaccuracy problem will always occur!

Execution Plan Caching
#

Since this was a stale statistics problem, manually collecting statistics should have resolved it. In practice, however, after collection, the business SQL was still slow.

After running ANALYZE, manual EXPLAIN ANALYZE showed the correct execution plan.

This indicated that ANALYZE should have helped, but it didn’t affect the business sessions. Since the SQL execution used long-lived sessions, I suspected that the JDBC driver was using prepared statements to cache execution plans (JDBC PreparedStatement).

In PostgreSQL 13 (RasesQL 1.3), collecting statistics does not invalidate prepared statements; re-parsing only happens by reconnecting the session.

Prepared statements generate a generic execution plan. Due to inaccurate statistics, the generic execution plan, like the parameter-specific execution plan, could choose the wrong index.

Characteristics of Prepared Statements
#

psql supports prepared statements, controlled by the plan_cache_mode parameter:

auto: default, uses the five-execution mechanism
force_custom_plan: always performs hard parsing, generating a custom plan
force_generic_plan: always uses the generic plan with bound variables

Syntax:

PREPARE plan1(text,integer) AS select * from tlzl1 where id=$1 and month=$2;
EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','11');

deallocate plan1|all; -- invalidates the prepared statement; disconnecting also works

View: (basically useless since it’s local — you can’t see anything in production)

select * from pg_prepared_statements;

How Generic Plans Are Generated
#

Normally, a prepared statement can generate a generic plan after running 5 times. There are many demonstrations online, so I won’t demonstrate the normal case here. Below are the “magical” phenomena I observed during testing:

-- Prepare data
create table tlzl1(id varchar(50),month int);
INSERT INTO tlzl1 SELECT md5(g::text),EXTRACT(month FROM g)
FROM generate_series('2023-01-01'::date, '2023-11-30'::date, '1 minute') as g;
create index idx_id on tlzl1(id);
create index idx_month on tlzl1(month);
analyze tlzl;

-- Execute prepared statement
PREPARE plan1(text,integer) AS 
select * from tlzl1 where id=$1 and month=$2;
EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','11');
explain analyze EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','11');

Note that only data before December was inserted — December has no data. At this point, querying December data can use the month index:

=# explain analyze EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','12');
 QUERY PLAN 
------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_month on tlzl1 (cost=0.42..2.94 rows=1 width=37) (actual time=0.035..0.036 rows=0 loops=1)
 Index Cond: (month = 12)
 Filter: ((id)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 Planning Time: 0.170 ms
 Execution Time: 0.058 ms
(5 rows)

Time: 0.551 ms
=# explain analyze EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','12');
 QUERY PLAN 
------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_month on tlzl1 (cost=0.42..2.94 rows=1 width=37) (actual time=0.021..0.021 rows=0 loops=1)
 Index Cond: (month = 12)
 Filter: ((id)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 Planning Time: 0.168 ms
 Execution Time: 0.046 ms
(5 rows)

Time: 0.488 ms
=# explain analyze EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','12');
 QUERY PLAN 
------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_month on tlzl1 (cost=0.42..2.94 rows=1 width=37) (actual time=0.017..0.018 rows=0 loops=1)
 Index Cond: (month = 12)
 Filter: ((id)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 Planning Time: 0.157 ms
 Execution Time: 0.040 ms
(5 rows)

Time: 0.419 ms
=# explain analyze EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','12');
 QUERY PLAN 
------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_month on tlzl1 (cost=0.42..2.94 rows=1 width=37) (actual time=0.019..0.020 rows=0 loops=1)
 Index Cond: (month = 12)
 Filter: ((id)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 Planning Time: 0.160 ms
 Execution Time: 0.044 ms
(5 rows)

Time: 0.479 ms
=# explain analyze EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','12');
 QUERY PLAN 
------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_month on tlzl1 (cost=0.42..2.94 rows=1 width=37) (actual time=0.018..0.018 rows=0 loops=1)
 Index Cond: (month = 12)
 Filter: ((id)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 Planning Time: 0.155 ms
 Execution Time: 0.041 ms
(5 rows)

Time: 0.426 ms
-- Sixth execution
=# explain analyze EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','12');
 QUERY PLAN 
---------------------------------------------------------------------------------------------------------------
 Index Scan using idx_id on tlzl1 (cost=0.42..5.44 rows=1 width=37) (actual time=0.044..0.045 rows=0 loops=1)
 Index Cond: ((id)::text = $1)
 Filter: (month = $2)
 Rows Removed by Filter: 1
 Planning Time: 0.023 ms
 Execution Time: 0.079 ms
(6 rows)

On the sixth execution, the generic plan was bound — but it wasn’t the same plan as the first five executions; it used the id index. If id had even higher cardinality, you could also observe cases where the generic plan simply couldn’t be bound (not shown here).

Let’s look at the source code:

choose_custom_plan:

static bool

choose_custom_plan(CachedPlanSource *plansource, ParamListInfo boundParams)

{
...
	/* Generate custom plans until we have done at least 5 (arbitrary) */
	if (plansource->num_custom_plans < 5)
		return true;

	avg_custom_cost = plansource->total_custom_cost / plansource->num_custom_plans;

	/*
	 * Prefer generic plan if it's less expensive than the average custom
	 * plan. (Because we include a charge for cost of planning in the
	 * custom-plan costs, this means the generic plan only has to be less
	 * expensive than the execution cost plus replan cost of the custom
	 * plans.)
	 *
	 * Note that if generic_cost is -1 (indicating we've not yet determined
	 * the generic plan cost), we'll always prefer generic at this point.
	 */
	if (plansource->generic_cost < avg_custom_cost)
		return false;

	return true;
}		

As long as the generic plan’s cost is less than the average cost of the first 5 custom plans, the generic plan is used.

While the 5-execution mechanism is well-known, it’s important to note how the generic plan is generated. On the 5th execution, there is no generic plan yet (initially, generic_cost=-1), so it directly goes to the !customplan logic in GetCachedPlan:

CachedPlan *
GetCachedPlan(CachedPlanSource *plansource, ParamListInfo boundParams,
			 bool useResOwner, QueryEnvironment *queryEnv)
{
...
	customplan = choose_custom_plan(plansource, boundParams);

	if (!customplan)
	{
		if (CheckCachedPlan(plansource))
		{
			/* We want a generic plan, and we already have a valid one */
			plan = plansource->gplan;
			Assert(plan->magic == CACHEDPLAN_MAGIC);
		}
		else
		{
			/* Build a new generic plan */
			plan = BuildCachedPlan(plansource, qlist, NULL, queryEnv);
			/* Just make real sure plansource->gplan is clear */
			ReleaseGenericPlan(plansource);
			/* Link the new generic plan into the plansource */
			plansource->gplan = plan;
			plan->refcount++;
			/* Immediately reparent into appropriate context */
			if (plansource->is_saved)
			{
				/* saved plans all live under CacheMemoryContext */
				MemoryContextSetParent(plan->context, CacheMemoryContext);
				plan->is_saved = true;
			}
			else
			{
				/* otherwise, it should be a sibling of the plansource */
				MemoryContextSetParent(plan->context,
									 MemoryContextGetParent(plansource->context));
			}
			/* Update generic_cost whenever we make a new generic plan */
			plansource->generic_cost = cached_plan_cost(plan, false);

			/*
			 * If, based on the now-known value of generic_cost, we'd not have
			 * chosen to use a generic plan, then forget it and make a custom
			 * plan. This is a bit of a wart but is necessary to avoid a
			 * glitch in behavior when the custom plans are consistently big
			 * winners; at some point we'll experiment with a generic plan and
			 * find it's a loser, but we don't want to actually execute that
			 * plan.
			 */
			customplan = choose_custom_plan(plansource, boundParams);

			/*
			 * If we choose to plan again, we need to re-copy the query_list,
			 * since the planner probably scribbled on it. We can force
			 * BuildCachedPlan to do that by passing NIL.
			 */
			qlist = NIL;
		}
	}
	...
	return plan;
}	

In the !customplan logic, if a generic plan already exists, use it directly. If not, generate one via BuildCachedPlan, which is the main logic for generating plans — converting a query tree into a plan tree.

What about parameters? As the comments explain, pass NULL when there are no parameters to enter the plan generation logic:

To build a generic, parameter-value-independent plan, pass NULL for
 * boundParams. To build a custom plan, pass the actual parameter values via
 * boundParams

What execution plan does the optimizer prefer when NULL is passed? This part of the code logic is somewhat complex. From the optimizer’s perspective, there may be multiple plans to choose from, but one must be selected as the generic plan.

And that selected generic plan is what gets compared against the cost of the first 5 plans.

Why didn’t repeatedly executing a lower-cost plan produce the desired generic plan?

What the generic plan looks like has nothing to do with the first five execution plans — the first five only determine whether this generic plan gets bound.

From an optimizer design perspective, generic plans are meant to reduce parsing time and improve SQL execution efficiency, suitable for many small queries. The problem is that generic plans themselves are crude, and PostgreSQL introduced the five-execution mechanism precisely to reduce the likelihood of a generic plan being terrible.

Even with the five-execution mechanism, the reasons a bad generic plan still gets bound are:

Generic plans are plans too, and they can inherently be bad
Statistics are inaccurate, so the generic plan’s cost estimate is very low
The first five executions had low selectivity (or other factors) causing high custom plan costs

Prepared Statement Invalidation
#

Besides DDL, DEALLOCATE, and disconnecting sessions, collecting statistics can also invalidate prepared statements — but this is a PostgreSQL 14 feature.

PostgreSQL 13:

PostgreSQL will force re-analysis and re-planning of the statement before using it whenever database objects used in the statement have undergone definitional (DDL) changes since the previous use of the prepared statement

PostgreSQL 14:

PostgreSQL will force re-analysis and re-planning of the statement before using it whenever database objects used in the statement have undergone definitional (DDL) changes or their planner statistics have been updated since the previous use of the prepared statement

Test confirming that in PostgreSQL 13, collecting statistics does not invalidate prepared statements:

= explain analyze EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','11');
 QUERY PLAN 
---------------------------------------------------------------------------------------------------------------
 Index Scan using idx_id on tlzl1 (cost=0.42..5.44 rows=1 width=37) (actual time=0.033..0.033 rows=0 loops=1)
 Index Cond: ((id)::text = $1)
 Filter: (month = $2)
 Rows Removed by Filter: 1
 Planning Time: 0.098 ms
 Execution Time: 0.050 ms
(6 rows)

= select * from pg_prepared_statements;
 name | statement | prepare_time | parameter_types | from_sql 
-------+-----------------------------------------------+-------------------------------+-----------------+----------
 plan1 | PREPARE plan1(text,integer) AS +| 2024-02-29 14:27:59.966733+08 | {text,integer} | t
 | select * from tlzl1 where id=$1 and month=$2; | | | 
(1 row)

= analyze tlzl1;
ANALYZE
= select * from pg_prepared_statements;
 name | statement | prepare_time | parameter_types | from_sql 
-------+-----------------------------------------------+-------------------------------+-----------------+----------
 plan1 | PREPARE plan1(text,integer) AS +| 2024-02-29 14:27:59.966733+08 | {text,integer} | t
 | select * from tlzl1 where id=$1 and month=$2; | | | 
(1 row)

= explain analyze EXECUTE plan1('256ac66bb53d31bc6124294238d6410c','11');
 QUERY PLAN 
---------------------------------------------------------------------------------------------------------------
 Index Scan using idx_id on tlzl1 (cost=0.42..5.44 rows=1 width=37) (actual time=0.051..0.052 rows=0 loops=1)
 Index Cond: ((id)::text = $1)
 Filter: (month = $2)
 Rows Removed by Filter: 1
 Planning Time: 0.022 ms
 Execution Time: 0.098 ms
(6 rows)

JDBC Prepared Statements
#

Prepared statements are not unique to PostgreSQL — other databases also have similar pre-parsing features. For example, Oracle can achieve similar functionality.

JDBC itself can call the database’s pre-parsing interface and directly use prepared statements.

Example JDBC configuration:

String sql = "select * from people where id=?";

PreparedStatement preparedStatement =
 connection.prepareStatement(sql);

Recommendations
#

Reduce the table-level autovacuum_analyze_scale_factor to 0.02 (why 0.02? Because 0.02 < 1/31). Since data is written and queried simultaneously, manual collection timing is hard to get right; reducing autovacuum_analyze_scale_factor can only mitigate this problem.
Consider removing the PREPARE setting in JDBC, or set force_custom_plan.
Adjust the SQL logic.
Adjust indexes: 4.1 Remove unnecessary time indexes; 4.2 Rebuild the index that gets chosen after predicate out-of-bounds as a composite index that includes the id field (a good suggestion).
Emergency procedure: If business performance doesn’t recover after statistics collection, and you’ve confirmed the execution plan has changed via manual EXPLAIN, consider killing sessions (for versions before 13).

Finally, predicate out-of-bounds problems exist in essentially all databases, especially on time-based fields. There is currently no simple yet perfectly effective solution. Oracle’s SPM (SQL Plan Management) gains another point in my favorability…

Incorrect Execution Plan Caused by Partition Permission Issues

Mon, 12 Aug 2024 00:00:00 +0000

Problem Overview
#

Last night, the business team updated a SQL query. Previously, the query ran very fast without the DATE_CREATED field (the partition key). After the release, the partition field was added to reduce the number of partitions accessed. However, after adding it, the UPDATE execution actually became slower.

Before:

 update TABLE_RECORD
 set IS_DELETED = '1', DATE_UPDATED = LOCALTIMESTAMP(0)
 WHERE APPL_NO = $1
 AND IS_DELETED = '0'

After:

 update TABLE_RECORD
 set IS_DELETED = '1', DATE_UPDATED = LOCALTIMESTAMP(0)
 WHERE APPL_NO = $1
 AND IS_DELETED = '0'
 AND DATE_CREATED > now() - interval '31' day
 AND DATE_CREATED < now()

Before the release, access time was in milliseconds. After the release, access time was 10 seconds. The SQL runs frequently, and the business found this unacceptable.

Problem Analysis
#

The Execution Plan Appeared Correct
#

Table structure:

## \d+ TABLE_RECORD
 Partitioned table "public.TABLE_RECORD"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
--------------------------------+-----------------------------+-----------+----------+---------------------------------------------------+----------+--------------+--------------------------
 id_TABLE_RECORD | character varying(32) | | not null | nextval('seq_TABLE_RECORD'::regclass) | extended | | 
 appl_no | character varying(100) | | | | extended | | 
 r_appl_no | character varying(100) | | | | extended | | 
...
 created_by | character varying(100) | | not null | 'sys'::character varying | extended | | 
 date_created | timestamp without time zone | | not null | now() | plain | | 
 updated_by | character varying(100) | | not null | 'sys'::character varying | extended | | 
 date_updated | timestamp without time zone | | not null | now() | plain | | 
Partition key: RANGE (date_created)
Indexes:
 "date_TABLE_RECORD" btree (date_created)
 "idx_dateupdated" btree (date_updated)
 "idx_applnodeleted" btree (appl_no, is_deleted)
 "nk_TABLE_RECORD" btree (appl_no)
Partitions: TABLE_RECORD_202211 FOR VALUES FROM ('2022-11-01 00:00:00') TO ('2022-12-01 00:00:00'),
...
 TABLE_RECORD_202303 FOR VALUES FROM ('2023-03-01 00:00:00') TO ('2023-04-01 00:00:00'),
 TABLE_RECORD_202304 FOR VALUES FROM ('2023-04-01 00:00:00') TO ('2023-05-01 00:00:00'),
 TABLE_RECORD_202305 FOR VALUES FROM ('2023-05-01 00:00:00') TO ('2023-06-01 00:00:00'),
 TABLE_RECORD_202306 FOR VALUES FROM ('2023-06-01 00:00:00') TO ('2023-07-01 00:00:00'),
 ...
 TABLE_RECORD_202512 FOR VALUES FROM ('2025-12-01 00:00:00') TO ('2026-01-01 00:00:00'),
 TABLE_RECORD_other DEFAULT

This SQL would access partitions from the last 2 months, both of which contained data. The above UPDATE would only update one row.

At first, analyzing the problem was very confusing because when we ran EXPLAIN, the execution plan looked fine.

EXPLAIN partition scan info:

 -> Index Scan using TABLE_RECORD_202302_date_created_idx on TABLE_RECORD_202302 TABLE_RECORD_4 (cost=0.44..5.47 rows=1 width=485)
 Index Cond: ((date_created > (now() - '31 days'::interval day)) AND (date_created < now()))
 Filter: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 -> Index Scan using TABLE_RECORD_202303_date_created_idx on TABLE_RECORD_202303 TABLE_RECORD_5 (cost=0.44..5.47 rows=1 width=482)
 Index Cond: ((date_created > (now() - '31 days'::interval day)) AND (date_created < now()))
 Filter: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 -> Index Scan using TABLE_RECORD_202304_date_created_idx on TABLE_RECORD_202304 TABLE_RECORD_6 (cost=0.44..5.47 rows=1 width=481)
 Index Cond: ((date_created > (now() - '31 days'::interval day)) AND (date_created < now()))
 Filter: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 -> Index Scan using idx_applnodeleted_25 on TABLE_RECORD_202305 TABLE_RECORD_7 (cost=0.43..30.49 rows=1 width=483)
 Index Cond: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 Filter: ((date_created < now()) AND (date_created > (now() - '31 days'::interval day)))
 -> Index Scan using idx_applnodeleted_14 on TABLE_RECORD_202306 TABLE_RECORD_8 (cost=0.56..45.11 rows=18 width=485)
 Index Cond: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 Filter: ((date_created < now()) AND (date_created > (now() - '31 days'::interval day)))
 -> Index Scan using idx_applnodeleted_38 on TABLE_RECORD_202307 TABLE_RECORD_9 (cost=0.14..5.17 rows=1 width=3502)
 Index Cond: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 Filter: ((date_created < now()) AND (date_created > (now() - '31 days'::interval day)))
 -> Index Scan using idx_applnodeleted_1 on TABLE_RECORD_202308 TABLE_RECORD_10 (cost=0.14..5.17 rows=1 width=3502)

Partition data distribution:

 select count(*),tableoid::regclass from TABLE_RECORD group by 2;
 count | tableoid 
-------+---------------------------------
| TABLE_RECORD_202303
| TABLE_RECORD_202211
| TABLE_RECORD_202306
| TABLE_RECORD_202305
| TABLE_RECORD_202304
| TABLE_RECORD_202212
| TABLE_RECORD_202301
| TABLE_RECORD_202302

The execution plan appeared to access different indexes for different partitions:

date_TABLE_RECORD: index on the partition key
idx_applnodeleted: composite index on appl_no, is_deleted

In reality, the SQL could prune partitions using the DATE_CREATED (last 31 days) field. But if it used the index on that field, there would be no selectivity at all. The composite index idx_applnodeleted on appl_no, is_deleted had much better selectivity within partitions, so the correct execution plan should choose the idx_applnodeleted composite index.

The EXPLAIN plan above is not the actual execution plan, but we can see that the May and June partitions did use the correct index — the appl_no, is_deleted composite index.

To view the actual execution plan, we need to execute the SQL. So we changed the UPDATE to a SELECT:

 # explain (analyze,buffers,timing,verbose) select count(*) from TABLE_RECORD WHERE APPL_NO = 'LZLMATH20230132302302' AND IS_DELETED = '0' AND DATE_CREATED > now() - interval '31' day AND DATE_CREATED < now() ;
 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate (cost=266.09..266.10 rows=1 width=8) (actual time=0.565..0.566 rows=1 loops=1)
 Output: count(*)
 Buffers: shared hit=48
 -> Append (cost=0.14..265.95 rows=56 width=0) (actual time=0.388..0.558 rows=1 loops=1)
 Buffers: shared hit=48
 Subplans Removed: 37
 -> Index Scan using idx_applnodeleted_25 on public.TABLE_RECORD_202305 TABLE_RECORD_1 (cost=0.43..30.39 rows=2 width=0) (actual time=0.059..0.059 rows=0 loops=1)
 Index Cond: (((TABLE_RECORD_1.appl_no)::text = 'LZLMATH20230132302302'::text) AND ((TABLE_RECORD_1.is_deleted)::text = '0'::text))
 Filter: ((TABLE_RECORD_1.date_created < now()) AND (TABLE_RECORD_1.date_created > (now() - '31 days'::interval day)))
 Buffers: shared hit=3
 -> Index Scan using idx_applnodeleted_14 on public.TABLE_RECORD_202306 TABLE_RECORD_2 (cost=0.56..42.52 rows=17 width=0) (actual time=0.328..0.498 rows=1 loops=1)
 Index Cond: (((TABLE_RECORD_2.appl_no)::text = 'LZLMATH20230132302302'::text) AND ((TABLE_RECORD_2.is_deleted)::text = '0'::text))
 Filter: ((TABLE_RECORD_2.date_created < now()) AND (TABLE_RECORD_2.date_created > (now() - '31 days'::interval day)))
 Buffers: shared hit=45
 Planning:
 Buffers: shared hit=5867
 Planning Time: 17.195 ms
 Execution Time: 0.654 ms
(18 rows)

The SELECT only accessed the May and June partitions, indicating partition pruning worked correctly. Both partitions used the idx_applnodeleted index, so index selection was also correct.

Direct execution of the SELECT statement returned results in milliseconds:

## select count(*) from TABLE_RECORD WHERE APPL_NO = 'LZLMATH20230132302302' AND IS_DELETED = '0' AND DATE_CREATED > now() - interval '31' day AND DATE_CREATED < now() ;
 count 
-------
 1
(1 row)

Time: 4.946 ms

At this point in the analysis, the execution plan appeared normal and execution time appeared normal.

The Business SQL Was Still Slow
#

However, slow SQL still appeared in the PostgreSQL logs — the UPDATE took 10 seconds:

2023-06-29 11:06:45.077 CST,"lzldbopr","lzldb",116286,"30.88.78.90:51871",649cdebf.1c63e,7,"UPDATE",2023-06-29 09:30:39 CST,759/12440291,4002354803,LOG,00000,"duration: 10287.105 ms " plan:
Query Text: update TABLE_RECORD
 set IS_DELETED = '1', DATE_UPDATED = LOCALTIMESTAMP(0)
 WHERE APPL_NO = $1
 AND IS_DELETED = '0'
 AND DATE_CREATED > now() - interval '31' day
 AND DATE_CREATED < now()
Update on TABLE_RECORD (cost=0.14..203.79 rows=39 width=2960)
 Update on TABLE_RECORD_202211 TABLE_RECORD_1
...
 -> Index Scan using TABLE_RECORD_202304_date_created_idx on TABLE_RECORD_202304 TABLE_RECORD_6 (cost=0.44..5.47 rows=1 width=481)
 Index Cond: ((date_created > (now() - '31 days'::interval day)) AND (date_created < now()))
 Filter: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 -> Index Scan using TABLE_RECORD_202305_date_created_idx on TABLE_RECORD_202305 TABLE_RECORD_7 (cost=0.44..5.47 rows=1 width=483)
 Index Cond: ((date_created > (now() - '31 days'::interval day)) AND (date_created < now()))
 Filter: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 -> Index Scan using TABLE_RECORD_202306_date_created_idx on TABLE_RECORD_202306 TABLE_RECORD_8 (cost=0.44..5.47 rows=1 width=485)
 Index Cond: ((date_created > (now() - '31 days'::interval day)) AND (date_created < now()))
 Filter: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 -> Index Scan using idx_applnodeleted_38 on TABLE_RECORD_202307 TABLE_RECORD_9 (cost=0.14..5.17 rows=1 width=3502)
 Index Cond: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 Filter: ((date_created < now()) AND (date_created > (now() - '31 days'::interval day)))
...

The May and June partitions were still using the date_created index on the partition key. The execution plan estimated only 1 row, but in reality these two partitions each had millions of rows.

This was very confusing — the optimizer itself could choose a better index, and EXPLAIN showed it going to that index, but the business SQL simply wasn’t using the correct index.

Updating Statistics
#

Since this was a PostgreSQL execution plan issue, the first thought was to collect statistics.

After the problem occurred, we collected statistics for both the parent partitioned table and child partitions. Concerned that sessions might have cached the execution plan (plan_cache_mode=auto), we killed all sessions that connected before the statistics collection.

The logs still showed the SQL taking 10 seconds, indicating it wasn’t a statistics issue.

At this point the problem remained unsolved. We seemed to have exhausted all options.

Root Cause
#

Earlier, when analyzing execution plans, the DBA’s EXPLAIN output differed from the application’s execution plan. However, we had been executing everything as the PostgreSQL superuser. We switched to the application user and ran EXPLAIN again — the execution plan matched what was in the logs!

Since we had previously encountered issues with native partitioned table permissions causing abnormal execution plans, we immediately checked partition permissions.

Parent table permissions:

## \dp+ TABLE_RECORD
 Access privileges
 Schema | Name | Type | Access privileges | Column privileges | Policies 
--------+--------------------------+-------------------+-------------------------------------+-------------------+----------
 public | TABLE_RECORD | partitioned table | lzldbdata=arwdDxt/lzldbdata +| | 
 | | | r_lzldbdata_qry=r/lzldbdata +| | 
 | | | r_lzldbdata_dml=arwd/lzldbdata +| | 
(1 row)

Child partition permissions:

## \dp+ TABLE_RECORD_202505
 Access privileges
 Schema | Name | Type | Access privileges | Column privileges | Policies 
--------+---------------------------------+-------+------------------------------------+-------------------+----------
 public | TABLE_RECORD_202505 | table | lzldbdata=arwdDxt/lzldbdata +| | 

The partition permissions were missing the r_lzldbdata_dml role, which is granted to the business user.

We immediately granted the permissions, and the problem was resolved:

grant select,update,delete,insert on TABLE_RECORD_202305 to r_lzldbdata_dml;
grant select,update,delete,insert on TABLE_RECORD_202306 to r_lzldbdata_dml;

After switching to the opr user again and running EXPLAIN, the execution plan was correct — the May and June partitions used the proper index:

\c - lzldbopr

 -> Index Scan using TABLE_RECORD_202303_date_created_idx on TABLE_RECORD_202303 TABLE_RECORD_5 (cost=0.44..5.47 rows=1 width=482)
 Index Cond: ((date_created > (now() - '31 days'::interval day)) AND (date_created < now()))
 Filter: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 -> Index Scan using TABLE_RECORD_202304_date_created_idx on TABLE_RECORD_202304 TABLE_RECORD_6 (cost=0.44..5.47 rows=1 width=481)
 Index Cond: ((date_created > (now() - '31 days'::interval day)) AND (date_created < now()))
 Filter: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 -> Index Scan using idx_applnodeleted_25 on TABLE_RECORD_202305 TABLE_RECORD_7 (cost=0.43..30.39 rows=1 width=483)
 Index Cond: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 Filter: ((date_created < now()) AND (date_created > (now() - '31 days'::interval day)))
 -> Index Scan using idx_applnodeleted_14 on TABLE_RECORD_202306 TABLE_RECORD_8 (cost=0.56..42.57 rows=17 width=485)
 Index Cond: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 Filter: ((date_created < now()) AND (date_created > (now() - '31 days'::interval day)))
 -> Index Scan using idx_applnodeleted_38 on TABLE_RECORD_202307 TABLE_RECORD_9 (cost=0.14..5.17 rows=1 width=3502)
 Index Cond: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 Filter: ((date_created < now()) AND (date_created > (now() - '31 days'::interval day)))
 -> Index Scan using idx_applnodeleted_1 on TABLE_RECORD_202308 TABLE_RECORD_10 (cost=0.14..5.17 rows=1 width=3502)
 Index Cond: (((appl_no)::text = 'LZLMATH20230132302302'::text) AND ((is_deleted)::text = '0'::text))
 Filter: ((date_created < now()) AND (date_created > (now() - '31 days'::interval day)))

No more slow UPDATE statements were observed in the PostgreSQL logs.

Testing (Not Reproduced)
#

Initial table creation script:

-- Switch to non-superuser
\c - lzldbdata
-- create table
CREATE TABLE PUBLIC.LZLPARTITION
(
 APPL_NO varchar(100) NULL,
	IS_DELETED varchar(8) NULL, 
 DATE_CREATED timestamp NOT NULL DEFAULT now(),
 DATE_UPDATED timestamp NOT NULL DEFAULT now()
) PARTITION BY RANGE(DATE_CREATED);

-- indexes
create index DATE_LZLPARTITION on PUBLIC.LZLPARTITION (DATE_CREATED);
create index NK_LZLPARTITION on PUBLIC.LZLPARTITION (APPL_NO);


-- privs
GRANT SELECT ON TABLE public.LZLPARTITION TO r_lzldbdata_qry;
GRANT SELECT,INSERT,UPDATE,DELETE ON TABLE public.LZLPARTITION TO r_lzldbdata_dml;

-- partition
create table LZLPARTITION_202301 partition of LZLPARTITION for values from ('2023-01-01 00:00:00') to ('2023-02-01 00:00:00');
create table LZLPARTITION_202302 partition of LZLPARTITION for values from ('2023-02-01 00:00:00') to ('2023-03-01 00:00:00');
create table LZLPARTITION_202303 partition of LZLPARTITION for values from ('2023-03-01 00:00:00') to ('2023-04-01 00:00:00');
create table LZLPARTITION_202304 partition of LZLPARTITION for values from ('2023-04-01 00:00:00') to ('2023-05-01 00:00:00');
create table LZLPARTITION_202305 partition of LZLPARTITION for values from ('2023-05-01 00:00:00') to ('2023-06-01 00:00:00');
create table LZLPARTITION_202306 partition of LZLPARTITION for values from ('2023-06-01 00:00:00') to ('2023-07-01 00:00:00');

Generate data:

insert
 into
 public.LZLPARTITION
select
 n + 10,
 'N',
 to_char(to_date('2023-01-01', 'YYYY-MM-DD') + ('' || n || ' minute') ::interval, 'YYYY-MM-DD')::"date",
 now()
from
 generate_series(0, 300000) n

Data distribution:

select count(*),tableoid::regclass from lzlpartition group by 2;
 count | tableoid 
-------+---------------------
| lzlpartition_202301
| lzlpartition_202302
| lzlpartition_202303
| lzlpartition_202304
| lzlpartition_202305
| lzlpartition_202306
| lzlpartition_202307

Permissions not inherited:

## \dp+ lzlpartition
 Access privileges
 Schema | Name | Type | Access privileges | Column privileges | Policies 
--------+--------------+-------------------+-------------------------------------+-------------------+----------
 public | lzlpartition | partitioned table | lzldbdata=arwdDxt/lzldbdata +| | 
 | | | r_lzldbdata_qry=r/lzldbdata +| | 
 | | | r_lzldbdata_dml=arwd/lzldbdata | | 


## \dp+ lzlpartition_202306
 Access privileges
 Schema | Name | Type | Access privileges | Column privileges | Policies 
--------+---------------------+-------+-------------------+-------------------+----------
 public | lzlpartition_202306 | table | | | 

Execution plan (correct):

explain select count(*) from lzlpartition WHERE APPL_NO = '217450' AND IS_DELETED = 'N' AND DATE_CREATED > now() - interval '31' day AND DATE_CREATED < now();
 QUERY PLAN 
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate (cost=36.76..36.77 rows=1 width=8)
 -> Append (cost=0.29..36.74 rows=7 width=0)
 Subplans Removed: 5
 -> Index Scan using lzlpartition_202305_appl_no_idx on lzlpartition_202305 lzlpartition_1 (cost=0.15..5.19 rows=1 width=0)
 Index Cond: ((appl_no)::text = '217450'::text)
 Filter: (((is_deleted)::text = '0'::text) AND (date_created < now()) AND (date_created > (now() - '31 days'::interval day)))
 -> Index Scan using lzlpartition_202306_appl_no_idx on lzlpartition_202306 lzlpartition_2 (cost=0.15..5.19 rows=1 width=0)
 Index Cond: ((appl_no)::text = '217450'::text)
 Filter: (((is_deleted)::text = '0'::text) AND (date_created < now()) AND (date_created > (now() - '31 days'::interval day)))

The permissions were still not inherited. In fact, we tested on other PostgreSQL versions and observed the same behavior — it seems to be a general behavior.

However, even so, we couldn’t reproduce the issue. The test results used the correct index, unlike the production environment which used the wrong index.

Summary
#

Since we had collected statistics and killed sessions, it shouldn’t have been a cached execution plan issue. After executing GRANT, the partition execution plan immediately became correct (even granting just one partition fixed that specific partition), so we are fairly confident that the partition permission issue caused the abnormal partition execution plan.

The analysis and resolution process can be summarized as follows:

Switch to the application user to view the execution plan. Using the superuser to view execution plans is a common practice, but the plan seen from the superuser may not be correct.
Permissions on child partitions of partitioned tables. The root cause is that permissions on child partitions of PostgreSQL partitioned tables were inconsistent with the parent table, causing the execution plan to be abnormal. In other words, permission issues affected PostgreSQL’s execution plan.
This issue is difficult to reproduce and occurs very, very rarely.
Permission-caused execution plan anomalies are extremely subtle and hard to diagnose.

Two questions worth deeper discussion:

Permission issues shouldn’t affect execution plans. Why do permissions affect execution plans?
Child partition permissions are inconsistent with parent table permissions. Why don’t child partitions fully inherit parent table permissions?

A bug report has been submitted to see what the official team says.

ORDER BY LIMIT 10 Slower Than ORDER BY LIMIT 100

Mon, 12 Aug 2024 00:00:00 +0000

Problem Analysis
#

When executing SQL in a PostgreSQL database, ORDER BY LIMIT 10 runs slower than ORDER BY LIMIT 100.

Execution Plan Analysis
#

 SELECT
			*,
 (select cl.ITEM_DESC from tablelzl2 cl where item_name='name' and cl.ITEM_NO='abcdefg') AS "item"
 FROM
 tablelzl1 RI
 WHERE RI.column1='AAAA'
		AND RI.column2 = 'applyno20231112'
 ORDER BY
 RI.column3 DESC limit 10

 Limit (cost=0.43..1522.66 rows=10 width=990)
 -> Index Scan Backward using idx_tablelzl1_column3 on tablelzl1 ri (cost=0.43..158007.45 rows=1038 width=990)
 Filter: (((column1)::text = 'AAAA'::text) AND ((column2)::text = 'applyno20231112'::text))
 SubPlan 1
 -> Index Scan using uk_tablelzl2_ii on tablelzl2 cl (cost=0.27..5.29 rows=1 width=18)
 Index Cond: (((item_no)::text = 'manualSign'::text) AND ((item_name)::text = (ri.manual_sign)::text))

The main table does not use the column2 index. Instead it uses an Index Scan Backward on the column3 sort index. The scan cost for the index is very high, yet the final cost looks low. Actual execution takes 9 seconds.

Changing LIMIT 10 to LIMIT 100 yields a normal execution plan:

 SELECT
			*,
 (select cl.ITEM_DESC from tablelzl2 cl where cl.ITEM_NAME = RI.MANUAL_SIGN AND cl.ITEM_NO='manualSign') AS "manualSign"
 FROM
 tablelzl1 RI
 WHERE RI.column1='AAAA'
		AND RI.column2 = 'applyno20231112'
 ORDER BY
 RI.column3 DESC limit 100 

 QUERY PLAN 
-----------------------------------------------------------------------------------------------------------------------
Limit (cost=2632.28..3162.78 rows=100 width=990)
 -> Result (cost=2632.28..8138.87 rows=1038 width=990)
 -> Sort (cost=2632.28..2634.87 rows=1038 width=474)
 Sort Key: ri.column3 DESC
 -> Index Scan using idx_cri_column2 on tablelzl1 ri (cost=0.43..2592.61 rows=1038 width=474)
 Index Cond: ((column2)::text = 'applyno20231112'::text)
 Filter: ((column1)::text = 'AAAA'::text)
 SubPlan 1
 -> Index Scan using uk_tablelzl2_ii on tablelzl2 cl (cost=0.27..5.29 rows=1 width=18)
 Index Cond: (((item_no)::text = 'manualSign'::text) AND ((item_name)::text = (ri.manual_sign)::text))
(10 rows)

The subquery plan remains unchanged. The main table now uses the column2 single-column index, fetches rows, sorts, then applies LIMIT — execution is extremely fast.

This is not just about LIMIT values — changing only the column2 value in the original SQL can also produce a normal plan. In practice, only a few specific column2 values trigger the abnormal plan.

Execution plan comparison:

column2 is a filter column, column3 is a sort column. The two plans choose different indexes:

Abnormal LIMIT 10 plan: Backward scan sort-column index → fetch rows → limit. No extra sort needed; scanning backward, it can stop as soon as it finds enough rows matching the LIMIT. The estimated cost of scanning the sort-column index is very high, but the top-level LIMIT cost estimate is very low.
Normal LIMIT 100 plan: Access filter-column index → fetch rows → sort by sort column → limit. Because sorting is required, all matching index entries must be retrieved. The filter-column index scan itself has a low cost estimate.

So the key issue is: the optimizer underestimates the cost of a partial backward scan on the sort index.

Actual Execution
#

Let’s look at explain (analyze,buffers):

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=0.43..1521.93 rows=10 width=990) (actual time=23.311..8122.516 rows=10 loops=1)
 Buffers: shared hit=861100 read=42985 dirtied=7
 I/O Timings: read=6741.003
 -> Index Scan Backward using idx_tablelzl1_column3 on tablelzl1 ri (cost=0.43..157932.45 rows=1038 width=990) (actual time=23.309..8122.505 rows=10 loops=1)
 Filter: (((column1)::text = 'AAAA'::text) AND ((column2)::text = 'applyno20231112'::text))
 Rows Removed by Filter: 1521796
 Buffers: shared hit=861100 read=42985 dirtied=7
 I/O Timings: read=6741.003
 SubPlan 1
 -> Index Scan using uk_tablelzl2_ii on tablelzl2 cl (cost=0.27..5.29 rows=1 width=18) (actual time=0.005..0.005 rows=0 loops=10)
 Index Cond: (((item_no)::text = 'manualSign'::text) AND ((item_name)::text = (ri.manual_sign)::text))
 Buffers: shared hit=6
 Planning:
 Buffers: shared hit=121 read=28
 I/O Timings: read=1.476
 Planning Time: 2.314 ms
 Execution Time: 8122.658 ms

 Limit (cost=2632.28..3162.78 rows=100 width=990) (actual time=150.101..150.122 rows=14 loops=1)
 Buffers: shared hit=700 read=274
 I/O Timings: read=146.903
 -> Result (cost=2632.28..8138.87 rows=1038 width=990) (actual time=150.100..150.119 rows=14 loops=1)
 Buffers: shared hit=700 read=274
 I/O Timings: read=146.903
 -> Sort (cost=2632.28..2634.87 rows=1038 width=474) (actual time=150.072..150.073 rows=14 loops=1)
 Sort Key: ri.column3 DESC
 Sort Method: quicksort Memory: 30kB
 Buffers: shared hit=694 read=274
 I/O Timings: read=146.903
 -> Index Scan using idx_cri_column2 on tablelzl1 ri (cost=0.43..2592.61 rows=1038 width=474) (actual time=0.418..149.973 rows=14 loops=1)
 Index Cond: ((column2)::text = 'applyno20231112'::text)
 Filter: ((column1)::text = 'AAAA'::text)
 Rows Removed by Filter: 1218
 Buffers: shared hit=691 read=274
 I/O Timings: read=146.903
 SubPlan 1
 -> Index Scan using uk_tablelzl2_ii on tablelzl2 cl (cost=0.27..5.29 rows=1 width=18) (actual time=0.002..0.002 rows=0 loops=14)
 Index Cond: (((item_no)::text = 'manualSign'::text) AND ((item_name)::text = (ri.manual_sign)::text))
 Buffers: shared hit=6
 Planning Time: 0.334 ms
 Execution Time: 150.257 ms

The LIMIT 10 plan executes in 8 seconds: shared hit=861,100, disk read=42,985, 1,521,796 rows removed by filter.

The LIMIT 100 plan executes in 0.15 seconds: shared hit=694, read=274, 1,218 rows removed.

The LIMIT 10 plan is clearly abnormal — it reads far too many rows before finding qualifying ones, which is why the query is slow.

Statistics Analysis
#

The estimated cost is low, but the actual scan touches many index rows. First, check whether the statistics are accurate.

Table statistics:

[postgres@cnsz381785:7169/(rasesql)phmamp][10-30.15:01:26]M=# select relpages,reltuples::bigint from pg_class where relname='tablelzl1';
 relpages | reltuples 
----------+-----------
 91172 | 2280874 -- roughly matches actual count

Column statistics:

[phmampopr@cnsz381785:7169/(rasesql)phmamp][10-27.17:08:48]M=> select * from pg_stats where tablename='tablelzl1' and attname='column2';
-[ RECORD 1 ]----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
schemaname | public
tablename | tablelzl1
attname | column2
inherited | f
null_frac | 0
avg_width | 18
n_distinct | -0.11990886
most_common_vals | {applyno20231112,DY20190723006650,DY20200102012899,DY20180827000557,DY20190524001304,DY20190529001885,DY20190728002359}
most_common_freqs | {0.0005,0.00026666667,0.00023333334,0.0002,0.0002,0.0002,0.0002}
histogram_bounds | {CULZF0000121605605,DSNEW0000126854232,DSNEW0000137652871,DY20160516001057,DY20161104005509,DY20170306002677,DY20170703010428,DY20170928013517,DY20180410007383,DY20180615002936,DY20180
correlation | 0.3131596
most_common_elems | [null]
most_common_elem_freqs | [null]
elem_count_histogram | [null]

The value applyno20231112 happens to be the top most_common_vals, with an estimated frequency of 0.0005. Multiplying: 2,280,874 × 0.0005 = 1,140, which is close to the real count of 1,232.

[postgres@cnsz381785:7169/(rasesql)phmamp][10-30.15:05:28]M=# select count(*) from tablelzl1 where column2 = 'applyno20231112';
 count 
-------
 1232

Statistics are accurate. Running ANALYZE to recollect statistics would not fix this.

The Effect of Uneven Data Distribution
#

Using the current statistics, the estimated number of matching rows is ~1,140. On average, finding the first matching row through the sort-column index would require scanning 2,280,874 / 1,140 ≈ 2,000 index entries. For 10 rows, about 20,000 entries; for 100 rows, about 200,000 entries.

Let’s disable sort and force the LIMIT 100 statement to use the sort-column index:

M=# set enable_sort=off;
SET

--limit 100 execution plan
 Limit (cost=0.43..15222.69 rows=100 width=990)
 -> Index Scan Backward using idx_tablelzl1_column3 on tablelzl1 ri (cost=0.43..158007.45 rows=1038 width=990)
 Filter: (((column1)::text = 'AAAA'::text) AND ((column2)::text = 'applyno20231112'::text))
 SubPlan 1
 -> Index Scan using uk_tablelzl2_ii on tablelzl2 cl (cost=0.27..5.29 rows=1 width=18)
 Index Cond: (((item_no)::text = 'manualSign'::text) AND ((item_name)::text = (ri.manual_sign)::text))

When LIMIT 10 becomes LIMIT 100, the cost jumps from 1522.66 to 15222.69 — roughly a ×10 multiplication. The LIMIT 100 cost of 15222.69 now exceeds the filter-column index plan’s cost of 3162.78, so the optimizer switches indexes.

The above estimates all assume data is evenly scattered across the sort-column index. In reality, the data could be at the very end (backward scan finds it quickly), or all concentrated in the first few leaf pages (requiring nearly a full index scan + fetch), making the cost extremely high.

The correlation between the two columns — how the data is distributed across the index — determines whether using the sort-column index is efficient.

Let’s look at how many rows were actually scanned:

 -> Index Scan Backward using idx_tablelzl1_column3 on tablelzl1 ri (cost=0.43..157932.45 rows=1038 width=990) (actual time=23.309..8122.505 rows=10 loops=1)
 Filter: (((column1)::text = 'AAAA'::text) AND ((column2)::text = 'applyno20231112'::text))
 Rows Removed by Filter: 1521796

In reality, about 1,521,796 rows were scanned to find just 10 matching rows. The estimate was 20,000 — a 76× discrepancy!

Trigger Conditions
#

Must involve WHERE + ORDER BY + LIMIT clauses
Both the sort column and filter column must have indexes
The LIMIT value is typically not very large
Uneven data distribution

Solution
#

Rewrite the SQL: add an expression to prevent the ORDER BY column from using its index.

 SELECT
			*,
 (select cl.ITEM_DESC from tablelzl2 cl where cl.ITEM_NAME = RI.MANUAL_SIGN AND cl.ITEM_NO='manualSign') AS "manualSign"
 FROM
 tablelzl1 RI
 WHERE RI.column1='AAAA'
		AND RI.column2 = 'applyno20231112'
 ORDER BY
 RI.column3 +'0' DESC limit 10

How Oracle Handles This
#

Cost Estimation Differences in Execution Plans
#

From the analysis above, the PostgreSQL execution plan’s cost looks unbalanced — the upper-level cost is lower than the inner-level cost, unlike Oracle’s hierarchical accumulation.

Let’s run an experiment: a table containing only rows where colname='x', comparing how PostgreSQL and Oracle calculate costs:

[postgres@cnsz381785:7169/(rasesql)dbmgr][10-31.14:32:19]M=# explain select * from testlzl where col1='x' limit 1;
 QUERY PLAN 
-----------------------------------------------------------------------
 Limit (cost=0.00..0.02 rows=1 width=2)
 -> Seq Scan on testlzl (cost=0.00..17747.20 rows=1048576 width=2)
 Filter: ((col1)::text = 'x'::text)

[postgres@cnsz381785:7169/(rasesql)dbmgr][10-31.14:32:30]M=# explain select * from testlzl where col1='xx' limit 1;
 QUERY PLAN 
-----------------------------------------------------------------
 Limit (cost=0.00..17747.20 rows=1 width=2)
 -> Seq Scan on testlzl (cost=0.00..17747.20 rows=1 width=2)
 Filter: ((col1)::text = 'xx'::text)

When col1='x', the row is found immediately, but the LIMIT cost is not pushed down into the seq scan cost — the total cost is 17747.20, the same as scanning the whole table. The LIMIT cost is not pushed into the inner node’s cost, but the rows estimate is.

Now let’s see how Oracle handles the same case:

SYS@t8icss1> select * from dbmgr.testlzl where a='x' and rownum<=1;

1 row selected.


Execution Plan
----------------------------------------------------------
Plan hash value: 2045386539

------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 2 | 2 (0)| 00:00:01 |
|* 1 | COUNT STOPKEY | | | | | |
|* 2 | TABLE ACCESS FULL| TESTLZL | 1 | 2 | 2 (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 1 - filter(ROWNUM<=1)
 2 - filter("A"='x')

SYS@t8icss1> select * from dbmgr.testlzl where a='xx' and rownum<=1;

no rows selected


Execution Plan
----------------------------------------------------------
Plan hash value: 2045386539

------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 2 | 302 (2)| 00:00:01 |
|* 1 | COUNT STOPKEY | | | | | |
|* 2 | TABLE ACCESS FULL| TESTLZL | 1 | 2 | 302 (2)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 1 - filter(ROWNUM<=1)
 2 - filter("A"='xx')

In Oracle, when a='x' is found immediately, the STOPKEY cost is pushed into the inner node — cost is only 2. When the data doesn’t exist (a='xx'), the full scan cost is 302.

This is an important difference between Oracle and PostgreSQL cost calculation:

In Oracle, the outer node cost is always ≥ the inner node cost; in PostgreSQL, this is not guaranteed.
Oracle’s inner node cost incorporates outer operators (e.g., STOPKEY); PostgreSQL does not — it gives the full cost of the child path.

Oracle and Uneven Data Distribution
#

Knowing the principle, we can reproduce the issue by placing data at the beginning of the sort index:

create table tlzl(a char(100) not null,b char(100) not null);
--Insert bulk data
begin
for i in 1..100000 loop
insert into tlzl values('test','test');
end loop;
end;
/
--Insert special data
insert into tlzl values('aaaa','aaaa');
insert into tlzl values('zzzz','zzzz');

--Create indexes
create index idx_a on tlzl(a);
create index idx_b on tlzl(b);
--Collect statistics
EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>'SYS',TABNAME=>'TLZL',estimate_percent => 10, degree=>1,METHOD_OPT=>'FOR ALL COLUMNS SIZE AUTO',cascade=>true);

select * from (select /*+ index(tlzl idx_a)*/* from tlzl where b='aaaa' order by a) where rownum<=1; 
select * from (select /*+ index(tlzl idx_a)*/* from tlzl where b='zzzz' order by a) where rownum<=1; 

SYS@t8icss1> select * from (select /*+ index(tlzl idx_a)*/* from tlzl where b='aaaa' order by a) where rownum<=1; 

Execution Plan
----------------------------------------------------------
Plan hash value: 3674066029

---------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 204 | 2210 (1)| 00:00:01 |
|* 1 | COUNT STOPKEY | | | | | |
| 2 | VIEW | | 1 | 204 | 2210 (1)| 00:00:01 |
|* 3 | TABLE ACCESS BY INDEX ROWID| TLZL | 1 | 202 | 2210 (1)| 00:00:01 |
| 4 | INDEX FULL SCAN | IDX_A | 98830 | | 779 (1)| 00:00:01 |
---------------------------------------------------------------------------------------

SYS@t8icss1> select * from (select /*+ index(tlzl idx_a)*/* from tlzl where b='zzzz' order by a) where rownum<=1; 

Execution Plan
----------------------------------------------------------
Plan hash value: 3674066029

---------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 204 | 2210 (1)| 00:00:01 |
|* 1 | COUNT STOPKEY | | | | | |
| 2 | VIEW | | 1 | 204 | 2210 (1)| 00:00:01 |
|* 3 | TABLE ACCESS BY INDEX ROWID| TLZL | 1 | 202 | 2210 (1)| 00:00:01 |
| 4 | INDEX FULL SCAN | IDX_A | 98830 | | 779 (1)| 00:00:01 |
---------------------------------------------------------------------------------------

Oracle’s optimizer has the same limitation — it doesn’t know where the data actually sits within the index. Whether the data is at the first or last position in the index, the estimated cost is the same.

However, Oracle provides more tools to address this: extended statistics, Automatic Column Group Detection, plan baselines, etc.

References
#

http://www.postgres.cn/v2/news/viewone/1/717 https://oracle-base.com/articles/12c/automatic-column-group-detection-extended-statistics-12cr1

PG Error: attempted to delete invisible tuple

Mon, 12 Aug 2024 00:00:00 +0000

Problem Description
#

PostgreSQL DELETE was failing with attempted to delete invisible tuple, but SELECT with the same conditions worked fine.

delete from lzltab1;
select count(*) from lzltab1;

Results of full-table delete and full-table select:

M=# delete from lzltab1;
ERROR: 55000: attempted to delete invisible tuple
LOCATION: heap_delete, heapam.c:2500
Time: 511.050 ms

M=# select count(*) from lzltab1;
 count 
--------
 231187

DELETE found an invisible tuple, but SELECT was fine.

This seemed very strange at first. PG visibility is determined by the tuple’s xmin, xmax, cid and the snapshot’s xmin, xmax, xip_list. Although the transaction state and timing of the tuple deletion can affect visibility, if the table data is stable (no ongoing DML), any subsequent snapshot should yield a stable visibility set. There shouldn’t be a case where the current transaction’s visibility differs from others — DML transaction tuple visibility should be consistent. In other words, in this scenario, the SELECT snapshot and DELETE snapshot shouldn’t produce different results.

Analysis
#

Finding the Source Code
#

Note the error location: heapam.c:2500

Find the source at src/backend/access/heap/heapam.c.

Line 2500 is blank; nearby code is:

	/*
	 * Before locking the buffer, pin the visibility map page if it appears to
	 * be necessary. Since we haven't got the lock yet, someone else might be
	 * in the middle of changing this, so we'll need to recheck after we have
	 * the lock.
	 */

	if (PageIsAllVisible(page))
		visibilitymap_pin(relation, block, &vmbuffer);

	LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);

From the source, it’s trying to acquire a lock on the VM, so the problem appears related to the VM file.

The VM File
#

What is the VM file?

The VM (Visibility Map) file exists to reduce the time vacuum spends scanning pages. If a page doesn’t need vacuuming, it can be skipped, greatly reducing the time spent finding pages that need cleaning. This is the original purpose of the VM file. (It’s also sometimes used by index-only scans, but that doesn’t apply here since we’re doing a sequential scan.)

The VM file stores two pieces of information:

Whether all tuples on a page are visible. This means the page has no dead tuples needing vacuum.
Whether all tuples on a page are frozen. This means vacuum freeze doesn’t need to visit this page.

The VM helps vacuum find dead tuples while reducing the number of pages scanned. For example, in the diagram above (interdb ftw!), the first page contains no dead tuples, so vacuum can skip it.

Finding the VM File

Every table has a Visibility Map (VM) file (indexes don’t have VM files), stored alongside the table file. If a table’s filenode is 12345, its VM file is 12345_vm.

First, cd to the data directory:

M=# show data_directory;
 data_directory 
----------------------
 /pg/pg6666/data

Find the file storage location using the database OID and table OID:

=# select oid,datname from pg_database where datname='sdp';
-------+----------------------
 oid | datname
 17075 | sdp
=# select oid,relname from pg_class where relname='lzltab1'; 
-------+----------------------
 17362 | lzltab1

Or:

# select pg_relation_filepath('lzltab1');
 pg_relation_filepath 
----------------------
 base/17075/17362

Find the data file and VM:

$ cd /pg/pg6666/data/base/17075
$ ll 17362*
-rw------- 1 postgres postgres 86761472 Jun 15 17:43 17362
-rw------- 1 postgres postgres 40960 Jun 9 21:09 17362_fsm
-rw------- 1 postgres postgres 8192 Nov 14 2022 17362_vm

The pg_visibility Extension
#

pg_visibility provides page-level visibility information by inspecting VM files, and can detect VM corruption. Since the VM stores “are all tuples on this page visible; are all tuples on this page frozen” information, pg_visibility can identify which pages are all-frozen and which are all-visible.

pg_visibility extension reference: https://www.postgresql.org/docs/current/pgvisibility.html

Useful pg_visibility Functions
#

pg_visibility_map_summary(): Shows the count of all-visible and all-frozen pages in the VM.

pg_check_frozen(): Returns rows where a tuple is not frozen but its page is marked all-frozen in the VM. If this function returns results, the VM file is corrupt.

pg_check_visible(): Returns rows where a tuple is not visible but its page is marked all-visible in the VM. If this function returns results, the VM file is corrupt.

pg_truncate_visibility_map(): Clears the VM file. After clearing, the next vacuum on the table will scan all pages and rebuild the VM.

Repairing the VM File
#

Check for VM corruption:

M=# select pg_visibility_map_summary('lzltab1');
 pg_visibility_map_summary 
---------------------------
 (472,0)

472 all-visible pages, 0 all-frozen pages.

M=# select pg_check_frozen('lzltab1');
 pg_check_frozen 
-----------------
(0 rows)
M=# select pg_check_visible('lzltab1');
 pg_check_visible 
------------------
 (6839,1)
 (6839,2)
 ...
 (7296,15)
(1423 rows)

pg_check_visible() returning results means the VM is corrupted.

Now use pg_truncate_visibility_map() to clear the VM:

M=# select pg_truncate_visibility_map('lzltab1');
 pg_truncate_visibility_map 
----------------------------

On disk, you can see the VM was cleared:

 ll 17362*
-rw------- 1 postgres postgres 86761472 Jun 27 10:39 17362
-rw------- 1 postgres postgres 40960 Jun 9 21:09 17362_fsm
-rw------- 1 postgres postgres 0 Jun 27 18:18 17362_vm

Now verify by vacuuming the table to regenerate the VM file and check it’s not corrupted:

M=# vacuum lzltab1;
VACUUM
Time: 3692.402 ms (00:03.692)
M=# \q
$ ll 17362*
-rw------- 1 postgres postgres 86761472 Jun 28 03:37 17362
-rw------- 1 postgres postgres 40960 Jun 9 21:09 17362_fsm
-rw------- 1 postgres postgres 8192 Jun 28 10:21 17362_vm

After manual vacuum, the VM was regenerated correctly:

M=# select pg_check_visible('lzltab1');
 pg_check_visible 
------------------
(0 rows)

M=# select pg_check_frozen('lzltab1');
 pg_check_frozen 
-----------------
(0 rows)

Both checks return empty — VM file is healthy. Repair complete.

Finally, re-run the SQL:

## delete from lzltab1;
DELETE 229766

DELETE executes normally. Problem resolved.

Checking the Entire Database for VM Corruption
#

Although we fixed one corrupted VM file, we should check the entire database for other VM corruption (requires the pg_visibility extension installed):

SELECT oid::regclass AS relname
FROM pg_class
WHERE relkind IN ('r', 'm', 't') AND (
 EXISTS (SELECT * FROM pg_check_visible(oid))
 OR EXISTS (SELECT * FROM pg_check_frozen(oid)));

If results are returned, there’s VM corruption. Use pg_truncate_visibility_map() to clear the VM, then vacuum to regenerate it, as shown above.

For versions before 9.6 (which lack the pg_visibility extension), you’d need to stop the database, manually delete the VM files, restart, then vacuum to regenerate them.

Why Does VM Corruption Happen?
#

We traced the issue step by step to VM file corruption, but why did it corrupt?

PostgreSQL bugs. PG has had some bugs causing VM corruption (see Visibility Map Problems wiki), but these were all before PG 9.6.1.
Operating system or hardware issues.

Our version was PG13, so the cause can only be broadly attributed to OS or hardware problems.

Why Did SELECT Succeed But DELETE Fail?
#

A full-table SELECT working while a full-table DELETE errors out seems bizarre. The root cause is VM file corruption.

As mentioned, the VM file exists to speed up vacuum. Even though we weren’t running vacuum, the VM file still needs to be updated — DML operations always update (or at least check) the VM, while SELECT does not change VM state. So in this case, SELECT executed normally, but DELETE errored during VM processing.

In our case, DELETE scanned the VM and found pages marked all-visible, but the VM was wrong — those pages still contained invisible tuples. This is exactly the attempted to delete invisible tuple error. Invisible tuples may have already been deleted, and trying to delete them again naturally errors out, violating transaction visibility rules.

Additionally, index-only scans also use the VM file, so they would also be affected. However, this case involved a sequential scan, so SELECT was unaffected.

VM Corruption Causing Incorrect Index-Only Scan Results
#

As mentioned earlier, besides vacuum, index-only scans also use the VM file. Even though our case didn’t involve index-only scans, let’s dig deeper for completeness.

What Is an Index-Only Scan?
#

As the name suggests, an index-only scan accesses only the index structure to get results, without touching the table. Almost all relational databases support index-only scans because B+tree index structures store key values — if the query only needs key values, an index-only scan is possible.

However, PostgreSQL’s transaction implementation differs significantly from other databases (Oracle, MySQL), giving its index-only scans some unique characteristics.

PostgreSQL checks tuple visibility via xmin, xmax, and other information in tuple headers, but indexes don’t contain this information. This means PG’s index-only scans must visit data blocks to check visibility. This is where the VM comes in: since the VM stores all-visible and all-frozen information, pages marked as such don’t need visibility checks — the VM has already confirmed their visibility.

Another interdb diagram (interdb ftw!). When a query looks up tuples with keys 18 and 19: the page containing key=18 is marked all-visible in the VM, so accessing this tuple only requires the index page and VM file. The page containing key=19 is not marked all-visible, so the index-only scan still needs to visit the data page to check visibility.

Index-Only Scan Returning Incorrect Results
#

Because index-only scans consult the VM, and a corrupted VM stores wrong information — e.g., a page’s tuples aren’t all visible (some may have been deleted), but the page is still marked all-visible — the index-only scan skips the data page visibility check and directly returns index key values that should be invisible.

You can set enable_indexonlyscan=off to disable index-only scans and guarantee correct results. Or, as shown above, repair the VM file — which is probably the better choice.

Summary
#

The journey had some twists: at first glance the error seemed like a transaction visibility rule problem, which would have been serious — but it was actually much simpler.

We traced the attempted to delete invisible tuple error to the source code, identified it as a VM issue, used the pg_visibility extension to detect and fix the VM corruption, resolved the DELETE error, and finally explored the relationship between index-only scans and the VM.

Key takeaways:

The pg_visibility extension can read, check, and clear VM files
Without VM information, vacuum will generate a new VM
DML reads/updates VM files; SELECT does not (non-index-only-scan)
The VM file exists to improve vacuum efficiency, and sometimes index-only scan efficiency
The attempted to delete invisible tuple error warrants checking the VM file for corruption
VM file corruption can cause DML failures and incorrect index-only scan results

References
#

https://www.postgresql.org/docs/13/pgvisibility.html

https://wiki.postgresql.org/wiki/Visibility_Map_Problems

https://www.interdb.jp/pg/pgsql06.html

https://www.interdb.jp/pg/pgsql07.html

The Table I Wanted to Query Was Not in the Execution Plan

Mon, 12 Aug 2024 00:00:00 +0000

Problem: The Queried Table Did Not Appear in the Execution Plan
#

SQL:

SELECT
 *
FROM
 (
 SELECT
 A.column1 as "column1",
 -- many A columns omitted in between
 A.column99 as "column99"
 from
 table_a A
 left join (
 SELECT
 lzl_id
 from
 table_a AA
 inner join table_b BB ON AA.lzl_key = BB.lzl_id
 where
 AA.column_code = '1'
 GROUP BY
 lzl_id
 ) B ON B.lzl_id = A.lzl_key
 where
 A.flagflagflag = '1'
 AND A.typetypetype = '2'
 ) TEMP
limit
 100
offset
 1000

Execution plan:

 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=2.84..5.68 rows=1 width=1105) (actual time=0.038..0.039 rows=0 loops=1)
 Buffers: shared hit=2
 -> Seq Scan on table_a a (cost=0.00..2.84 rows=1 width=1105) (actual time=0.036..0.037 rows=0 loops=1)
 Filter: (((flagflagflag)::text = '1'::text) AND ((typetypetype)::text = '2'::text))
 Rows Removed by Filter: 38
 Buffers: shared hit=2
 Planning Time: 0.184 ms
 Execution Time: 0.066 ms

As you can see, the SQL itself is fairly complex. Logically, the SQL queries 3 tables / accesses 2 tables total. I can understand table_a appearing in the execution plan, but table_b, which needed to be queried, wasn’t in the execution plan at all! The execution plan was simply a sequential scan of table_a.

The Analytical Journey
#

In the middle of the analysis, I actually considered many possibilities, but the most likely one was logical optimization — that is, the PostgreSQL optimizer determined that table_b didn’t need to be queried.

Observing the SQL, I noticed that the final query only selected columns from table_a, without any columns from table_b. Adding any column from the intermediate table B made the SQL execution plan appear “normal” — it accessed table_b:

explain SELECT
 *
FROM
 (
 SELECT
 A.column1 as "column1",
 -- many A columns omitted in between
 A.column99 as "column99",
 B.lzl_id -- added a column from intermediate table B
 from
 table_a A
 left join (
 SELECT
 lzl_id
 from
 table_a AA
 inner join table_b BB ON AA.lzl_key = BB.lzl_id
 where
 AA.column_code = '1'
 GROUP BY
 lzl_id
 ) B ON B.lzl_id = A.lzl_key
 where
 A.flagflagflag = '1'
 AND A.typetypetype = '2'
 ) TEMP
limit
 100
offset
 1000

---------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=14.69..17.67 rows=1 width=1113)
 -> Nested Loop Left Join (cost=11.72..14.69 rows=1 width=1113)
 Join Filter: (bb.lzl_id = a.lzl_key)
 -> Seq Scan on table_a a (cost=0.00..2.84 rows=1 width=1113)
 Filter: (((flagflagflag)::text = '1'::text) AND ((typetypetype)::text = '2'::text))
 -> Group (cost=11.72..11.74 rows=5 width=8)
 Group Key: bb.lzl_id
 -> Sort (cost=11.72..11.73 rows=5 width=8)
 Sort Key: bb.lzl_id
 -> Nested Loop (cost=0.15..11.66 rows=5 width=8)
 -> Seq Scan on table_a aa (cost=0.00..2.70 rows=1 width=8)
 Filter: ((company_code)::text = '1'::text)
 -> Index Only Scan using idx_table_b_lzl_id on table_b bb (cost=0.15..8.83 rows=13 width=8)
 Index Cond: (lzl_id = aa.lzl_key)

This seems related to LEFT JOIN, but a quick thought makes it seem incorrect — after all, the results from the right table should affect the final query result, so the right table shouldn’t be skipped. Let’s try a simple LEFT JOIN:

explain select lzlleft.a from lzlleft left join lzlright on lzlleft.a=lzlright.a;
 QUERY PLAN 
--------------------------------------------------------------------
 Hash Left Join (cost=1.04..15.47 rows=320 width=4)
 Hash Cond: (lzlleft.a = lzlright.a)
 -> Seq Scan on lzlleft (cost=0.00..13.20 rows=320 width=4)
 -> Hash (cost=1.02..1.02 rows=2 width=4)
 -> Seq Scan on lzlright (cost=0.00..1.02 rows=2 width=4)

The right table is scanned. But, in intermediate table B, there’s the keyword GROUP BY. If we remove GROUP BY, then table_b is accessed regardless of whether we query columns from B.

Let’s add a GROUP BY in our test table and see the result:

> select * from lzlleft;
 a | b 
---+-----
 1 | zzz
(1 row)

Time: 0.259 ms
> select * from lzlright;
 a | b 
---+-------
 1 | qwer
 1 | poiuy 
 
> select lzlright.b from lzlleft full join lzlright on lzlleft.b=lzlright.b group by lzlright.b;
 b 
--------
 [null]
 poiuy
 qwer
(3 rows)

This is where I realized that the result set from GROUP BY must have a certain property — uniqueness.

Let’s add GROUP BY in the test table:

 explain select lzlleft.a from lzlleft left join (select a from lzlright group by a) c on lzlleft.a=c.a;
 QUERY PLAN 
----------------------------------------------------------
 Seq Scan on lzlleft (cost=0.00..13.20 rows=320 width=4)

The right table is not queried!

Based on the principle of right-table uniqueness, we can also have some fun variations:

-- distinct ensures right-table uniqueness
> explain select lzlleft.a from lzlleft left join (select distinct a from lzlright) c on lzlleft.a=c.a;
 QUERY PLAN 
----------------------------------------------------------
 Seq Scan on lzlleft (cost=0.00..13.20 rows=320 width=4) 

-- unique index ensures right-table uniqueness, even with just select a from lzlright
> explain select lzlleft.a from lzlleft left join (select a from lzlright) c on lzlleft.a=c.a;
 QUERY PLAN 
-----------------------------------------------------------------------
 Hash Left Join (cost=17.20..49.12 rows=512 width=4)
 Hash Cond: (lzlleft.a = lzlright.a)
 -> Seq Scan on lzlleft (cost=0.00..13.20 rows=320 width=4)
 -> Hash (cost=13.20..13.20 rows=320 width=4)
 -> Seq Scan on lzlright (cost=0.00..13.20 rows=320 width=4)
(5 rows)

Time: 0.510 ms
> create unique index idx_right on lzlright(a);
CREATE INDEX
Time: 3.576 ms
> explain select lzlleft.a from lzlleft left join (select a from lzlright) c on lzlleft.a=c.a;
 QUERY PLAN 
----------------------------------------------------------
 Seq Scan on lzlleft (cost=0.00..13.20 rows=320 width=4)
(1 row)

Here’s a summary of the analysis: when the right table’s data is unique and only the left table’s data is being queried, there’s no need to actually access the right table. So this is not a bug, but a feature of the PostgreSQL optimizer — and it makes logical sense.

Source Code Analysis
#

No source code analysis this time~

The optimizer source code is just too difficult. I only looked at some optimizer source code comments. Search for the keyword unique-ify, and you’ll find this:

 * Also, this routine and others in this module accept the special JoinTypes
 * JOIN_UNIQUE_OUTER and JOIN_UNIQUE_INNER to indicate that we should
 * unique-ify the outer or inner relation and then apply a regular inner
 * join. These values are not allowed to propagate outside this module,
 * however. Path cost estimation code may need to recognize that it's
 * dealing with such a case --- the combination of nominal jointype INNER
 * with sjinfo->jointype == JOIN_SEMI indicates that. 

Special JoinTypes: JOIN_UNIQUE_INNER and JOIN_UNIQUE_OUTER — they try to unique-ify the outer and inner relations and then treat them as an inner join. Path cost estimation needs to consider this scenario.

Comparison with Oracle and MySQL Optimizers
#

Let’s compare whether Oracle and MySQL optimizers have similar logical optimization improvements.

-- Oracle
create table lzlleft(a number);
create table lzlright(a number);
 select lzlleft.a from lzlleft left join (select distinct a from lzlright) c on lzlleft.a=c.a;

-- GROUP BY uniqueness
SQL> select lzlleft.a from lzlleft left join (select a from lzlright group by a) c on lzlleft.a=c.a; 

no rows selected


Execution Plan
----------------------------------------------------------
Plan hash value: 3533354041

---------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 26 | 5 (20)| 00:00:01 |
|* 1 | HASH JOIN OUTER | | 1 | 26 | 5 (20)| 00:00:01 |
| 2 | TABLE ACCESS FULL | LZLLEFT | 1 | 13 | 2 (0)| 00:00:01 |
| 3 | VIEW | | 1 | 13 | 3 (34)| 00:00:01 |
| 4 | HASH GROUP BY | | 1 | 13 | 3 (34)| 00:00:01 |
| 5 | TABLE ACCESS FULL| LZLRIGHT | 1 | 13 | 2 (0)| 00:00:01 |
---------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 1 - access("LZLLEFT"."A"="C"."A"(+))

-- DISTINCT uniqueness
SQL> select lzlleft.a from lzlleft left join (select distinct a from lzlright) c on lzlleft.a=c.a;

no rows selected


Execution Plan
----------------------------------------------------------
Plan hash value: 3859658234

---------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 26 | 5 (20)| 00:00:01 |
|* 1 | HASH JOIN OUTER | | 1 | 26 | 5 (20)| 00:00:01 |
| 2 | TABLE ACCESS FULL | LZLLEFT | 1 | 13 | 2 (0)| 00:00:01 |
| 3 | VIEW | | 1 | 13 | 3 (34)| 00:00:01 |
| 4 | HASH UNIQUE | | 1 | 13 | 3 (34)| 00:00:01 |
| 5 | TABLE ACCESS FULL| LZLRIGHT | 1 | 13 | 2 (0)| 00:00:01 |
---------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 1 - access("LZLLEFT"."A"="C"."A"(+))

-- MySQL
create table lzlleft(a int primary key);
create table lzlright(a int primary key);

-- GROUP BY uniqueness
explain select lzlleft.a from lzlleft left join (select a from lzlright group by a) c on lzlleft.a=c.a; 
+----+-------------+------------+------------+-------+---------------+-------------+---------+-----------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+---------------+-------------+---------+-----------------+------+----------+-------------+
| 1 | PRIMARY | lzlleft | NULL | index | NULL | PRIMARY | 4 | NULL | 1 | 100.00 | Using index |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 4 | lzldb.lzlleft.a | 2 | 100.00 | Using index |
| 2 | DERIVED | lzlright | NULL | index | PRIMARY | PRIMARY | 4 | NULL | 1 | 100.00 | Using index |
+----+-------------+------------+------------+-------+---------------+-------------+---------+-----------------+------+----------+-------------+

-- DISTINCT uniqueness
explain select lzlleft.a from lzlleft left join (select distinct a from lzlright) c on lzlleft.a=c.a;
+----+-------------+------------+------------+-------+---------------+-------------+---------+-----------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+---------------+-------------+---------+-----------------+------+----------+-------------+
| 1 | PRIMARY | lzlleft | NULL | index | NULL | PRIMARY | 4 | NULL | 1 | 100.00 | Using index |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 4 | lzldb.lzlleft.a | 2 | 100.00 | Using index |
| 2 | DERIVED | lzlright | NULL | index | PRIMARY | PRIMARY | 4 | NULL | 1 | 100.00 | Using index |
+----+-------------+------------+------------+-------+---------------+-------------+---------+-----------------+------+----------+-------------+

In summary, neither Oracle nor MySQL performs the optimization of eliminating the right table in a LEFT JOIN when only left-table columns are queried and the right table is unique — they both access the right table.

The PostgreSQL optimizer really has some impressive tricks.

Too Many Range Table Entries Even with Not-That-Many Partitions

Mon, 12 Aug 2024 00:00:00 +0000

Problem Description
#

PostgreSQL UPDATE statement throws error: too many range table entries

Original SQL:

with t as (select id from LZLTAB where id=8723 limit 100 )
update	LZLTAB set
	STATUS = '00',
	FILE_ID = null,
	DATE_UPDATED = localtimestamp(0)
where id in (select	id from t)

If we rewrite UPDATE as SELECT, it succeeds:

with t as (select	id from	LZLTAB where	id=8723 limit 100 )
select * from 	LZLTAB where	id in (select id	from t)
 id | date_created 
------+----------------------------+...
 8723 | 2023-06-21 18:02:21.161687 
(1 row)	

Primary key and partitions — 400 partitions total:

Partition key: RANGE (partition_key)
Indexes:
 "pk_lzl" PRIMARY KEY, btree (id, partition_key)
...
Partitions: lzl_p20230601 FOR VALUES FROM ('20230601') TO ('20230602'),
 lzl_p20230602 FOR VALUES FROM ('20230602') TO ('20230603'),
 lzl_p20230603 FOR VALUES FROM ('20230603') TO ('20230604')

The SQL logic has many optimization opportunities, but we won’t discuss those here. The focus is on why UPDATE fails and why SELECT and UPDATE behave differently.

EXPLAIN UPDATE throws this error:

explain with t as (selec tid from LZLTAB where id=8723 limit 100 )
update LZLTAB set
STATUS = '00',
FILE_ID = null,
DATE_UPDATED = localtimestamp(0)
where id in (select id from t);

ERROR: 54000: too many range table entries
LOCATION: add_rte_to_flat_rtable, setrefs.c:451
Time: 18341.171 ms (00:18.341)

EXPLAIN took 18 seconds, then threw the error.

Source Code Analysis
#

The error directly points to the source location: LOCATION: add_rte_to_flat_rtable, setrefs.c:451

Find the source at src/backend/optimizer/plan/setrefs.c.

The comment explains that setrefs.c handles post-processing of a completed plan tree:

/*
 *Post-processing of a completed plan tree: fix references to subplan
 *	 vars, compute regproc values for operators, etc
 */

Find the function at line 451:

/*
 * Add (a copy of) the given RTE to the final rangetable
 *
 * In the flat rangetable, we zero out substructure pointers that are not
 * needed by the executor; this reduces the storage space and copying cost
 * for cached plans. We keep only the ctename, alias and eref Alias fields,
 * which are needed by EXPLAIN, and the selectedCols, insertedCols,
 * updatedCols, and extraUpdatedCols bitmaps, which are needed for
 * executor-startup permissions checking and for trigger event checking.
 */
static void
add_rte_to_flat_rtable(PlannerGlobal *glob, RangeTblEntry *rte)
{
...
	/*
	 * Check for RT index overflow; it's very unlikely, but if it did happen,
	 * the executor would get confused by varnos that match the special varno
	 * values.
	 */
	if (IS_SPECIAL_VARNO(list_length(glob->finalrtable)))
		ereport(ERROR,
				(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
				 errmsg("too many range table entries")));
...
}

errmsg() is at line 451. From the comments, add_rte_to_flat_rtable() is related to RTE. What is RTE? We’ll analyze below.

The error check uses IS_SPECIAL_VARNO(). Searching for this macro in src/include/nodes/primnodes.h:

/*
 * Var - expression node representing a variable (ie, a table column)
 *
 * In the parser and planner, varno and varattno identify the semantic
 * referent, which is a base-relation column unless the reference is to a join
 * USING column that isn't semantically equivalent to either join input column
 * (because it is a FULL join or the input column requires a type coercion).
 * In those cases varno and varattno refer to the JOIN RTE. (Early in the
 * planner, we replace such join references by the implied expression; but up
 * till then we want join reference Vars to keep their original identity for
 * query-printing purposes.)
 ...
 */
#define INNER_VAR		65000	/* reference to inner subplan */
#define OUTER_VAR		65001	/* reference to outer subplan */
#define INDEX_VAR		65002	/* reference to index column */

#define IS_SPECIAL_VARNO(varno)		((varno) >= INNER_VAR)

The comment above is a bit dense, but one phrase is key: In those cases varno and varattno refer to the JOIN RTE. varno is related to RTE.

When varno>=65000, the error is thrown. (We won’t go into the differences between INNER_VAR, OUTER_VAR, and INDEX_VAR here since their values are close and don’t affect the analysis.)

What is RTE?

Descriptions of RTE (rangetable or RangeTblEntry) can be found throughout the execution plan source code, and the error is clear: ERROR: 54000: too many range table entries — it’s about RTE. So what is RTE?

In src/include/nodes/parsenodes.h, there’s a description of RTE:

/*--------------------
 * RangeTblEntry -
 *	 A range table is a List of RangeTblEntry nodes.
 *
 *	 A range table entry may represent a plain relation, a sub-select in
 *	 FROM, or the result of a JOIN clause. (Only explicit JOIN syntax
 *	 produces an RTE, not the implicit join resulting from multiple FROM
 *	 items. This is because we only need the RTE to deal with SQL features
 *	 like outer joins and join-output-column aliasing.) Other special
 *	 RTE types also exist, as indicated by RTEKind.
 *
 *	 Note that we consider RTE_RELATION to cover anything that has a pg_class
 *	 entry. relkind distinguishes the sub-cases.
 */

Simply put, an RTE is a “table” in the execution plan — it can be a concrete table or a generated “table” like a subquery, join result, etc. The RTE limit of 65000 means too many RTEs were generated in the execution plan.

Viewing the UPDATE Execution Plan
#

Since we now know what RTE is, looking at the SQL execution plan may help. But since the original SQL (400 partitions) couldn’t generate an execution plan, let’s create a 30-partition table and hopefully EXPLAIN it to observe the plan.

30-partition table with the same UPDATE statement:

explain with t as (select id from lzl where id=8723 limit 100 )
update lzl set
STATUS = '00',
FILE_ID = null,
DATE_UPDATED = localtimestamp(0)
where id in ( select id from t);

Generated execution plan:

 QUERY PLAN 
-----------------------------------------------------------------------------------------------------------------------------------------
 Update on lzl (cost=155.48..4980.00 rows=600 width=3042)
 Update on lzl_p20230601 lzl_1
 Update on lzl_p20230602 lzl_2
	...
 Update on lzl_p20230630 lzl_30
 -> Hash Semi Join (cost=155.48..166.00 rows=20 width=3042)
 Hash Cond: (lzl_1.id = t.id)
 -> Seq Scan on lzl_p20230601 lzl_1 (cost=0.00..10.20 rows=20 width=2912)
 -> Hash (cost=155.10..155.10 rows=30 width=40)
 -> Subquery Scan on t (cost=0.14..155.10 rows=30 width=40)
 -> Limit (cost=0.14..154.80 rows=30 width=8)
 -> Append (cost=0.14..154.80 rows=30 width=8)
 -> Index Only Scan using lzl_p20230601_pkey on lzl_p20230601 lzl_32 (cost=0.14..5.16 rows=1 width=8)
 Index Cond: (id = 8723)
 -> Index Only Scan using lzl_p20230602_pkey on lzl_p20230602 lzl_33 (cost=0.14..5.16 rows=1 width=8)
 Index Cond: (id = 8723)
 ...
 -> Index Only Scan using lzl_p20230630_pkey on lzl_p20230630 lzl_61 (cost=0.14..5.16 rows=1 width=8)
 Index Cond: (id = 8723)
		...
 -> Hash Semi Join (cost=155.48..166.00 rows=20 width=3042)
 Hash Cond: (lzl_30.id = t_29.id)
 -> Seq Scan on lzl_p20230630 lzl_30 (cost=0.00..10.20 rows=20 width=2912)
 -> Hash (cost=155.10..155.10 rows=30 width=40)
 -> Subquery Scan on t_29 (cost=0.14..155.10 rows=30 width=40)
 -> Limit (cost=0.14..154.80 rows=30 width=8)
 -> Append (cost=0.14..154.80 rows=30 width=8)
 -> Index Only Scan using lzl_p20230601_pkey on lzl_p20230601 lzl_931 (cost=0.14..5.16 rows=1 width=8)
 Index Cond: (id = 8723)
 -> Index Only Scan using lzl_p20230602_pkey on lzl_p20230602 lzl_932 (cost=0.14..5.16 rows=1 width=8)
 Index Cond: (id = 8723)
								...
 -> Index Only Scan using lzl_p20230630_pkey on lzl_p20230630 lzl_960 (cost=0.14..5.16 rows=1 width=8)
 Index Cond: (id = 8723)
(2041 rows)

The execution plan is extremely long — 2041 rows in total. This plan is very inefficient: every time a partition is updated, the predicate conditions are run against the partitioned table all over again. Since the SQL lacks a partition key, each run scans all partitions. For a 30-partition table, each partition is scanned 30 times, totaling 900 partition scans.

From the execution plan, we can see that initially 30 RTEs were allocated for UPDATE up to lzl_30. Then each hash match per partition scan also allocated 30 RTEs — for example, the hash under lzl_1 has partition scans from lzl_32 to lzl_61. Why 32 instead of 31? Because the entire partition scan is a subquery and also an RTE, named t (and t, t1-t_29), totaling 30. So the total RTEs generated in the plan are 30+30+30×30=960.

Looking at the SELECT execution plan, it’s very different from UPDATE:

explain with t as (select id from lzl where id=8723 limit 100 )
select STATUS ,FILE_ID ,DATE_UPDATED from lzl where id in ( select id from t);

 Hash Semi Join (cost=155.48..467.05 rows=90 width=98)
 Hash Cond: (lzl.id = lzl_31.id)
 -> Append (cost=0.00..309.00 rows=600 width=106)
 -> Seq Scan on lzl_p20230601 lzl_1 (cost=0.00..10.20 rows=20 width=106)
 -> Seq Scan on lzl_p20230602 lzl_2 (cost=0.00..10.20 rows=20 width=106)
...
 -> Seq Scan on lzl_p20230630 lzl_30 (cost=0.00..10.20 rows=20 width=106)
 -> Hash (cost=155.10..155.10 rows=30 width=8)
 -> Limit (cost=0.14..154.80 rows=30 width=8)
 -> Append (cost=0.14..154.80 rows=30 width=8)
 -> Index Only Scan using lzl_p20230601_pkey on lzl_p20230601 lzl_32 (cost=0.14..5.16 rows=1 width=8)
 Index Cond: (id = 8723)
 -> Index Only Scan using lzl_p20230602_pkey on lzl_p20230602 lzl_33 (cost=0.14..5.16 rows=1 width=8)
 Index Cond: (id = 8723)
...
 -> Index Only Scan using lzl_p20230630_pkey on lzl_p20230630 lzl_61 (cost=0.14..5.16 rows=1 width=8)
 Index Cond: (id = 8723)
(96 rows)

No repeated (Cartesian product-style) table access — RTEs only go up to 61. This is also why SELECT succeeds on 400 partitions, because 400×400 accesses is simply too many.

So regarding the original SQL where UPDATE fails and SELECT succeeds, we can conclude:

For 400 partitions with SELECT, the execution plan has 801 RTEs, which doesn’t exceed INNER_VAR (65000), so it can generate a plan and execute.
For 400 partitions with UPDATE, the execution plan has 160,160,400 RTEs, far exceeding INNER_VAR (65000), so the plan cannot be generated and throws the RTE overflow error.

The cause is mostly analyzed, but the significant difference between SELECT and UPDATE plans is still puzzling. Let’s compare Oracle and MySQL execution plans horizontally.

Oracle Behavior
#

Oracle partitioned table with local index:

CREATE TABLE lzl (
 id number NOT NULL,
 partition_key number DEFAULT 0 NOT NULL,
...
)
PARTITION BY RANGE (partition_key)
(
PARTITION lzl_p20230601 VALUES LESS THAN ('20230602'),
PARTITION lzl_p20230602 VALUES LESS THAN ('20230603'),
...
PARTITION lzl_p20230630 VALUES LESS THAN ('20230631'));

create index PKLZL on lzl(id, partition_key) local;
alter table lzl add constraint pklzl primary key (id, partition_key) using index pklzl;

Execution plan:

with t as (select id from lzl where id=8723 and rownum<= 100 )
select STATUS ,FILE_ID ,DATE_UPDATED from lzl where id in ( select id from t)

update lzl set
STATUS = '00',
FILE_ID = null,
DATE_UPDATED = sysdate
where id in (select id from lzl where id=8723 and rownum<= 100)

In Oracle, both SELECT and UPDATE use NESTED LOOP, accessing all partitions (PARTITION RANGE ALL). So in Oracle, regardless of SELECT or UPDATE, table t is the driving table. Because of IN, results are sorted and deduplicated. So Oracle’s plan is not 30×30 accesses but depends on the result set size in the driving table — n rows means n×30 partition accesses. Since driving table t has minimal data, this plan is fine.

MySQL Behavior
#

Since MySQL only supports local indexes, just create the primary key directly:

CREATE TABLE test (
 id bigint NOT NULL,
 date_created timestamp ,
...
)
PARTITION BY RANGE (partition_key) 
(
PARTITION lzl_p20230601 VALUES LESS THAN (20230602),
PARTITION lzl_p20230602 VALUES LESS THAN (20230603),
...
PARTITION lzl_p20230630 VALUES LESS THAN (20230631));

alter table lzl add primary key pklzl(id,partition_key);

MySQL starting from 5.7 shows which partitions are scanned in the execution plan (version 8.0 here).

SELECT plan:

> explain with t as (select id from lzl where id=8723 limit 100 )
 -> select STATUS ,FILE_ID ,DATE_UPDATED from lzl where id in ( select id from t);
+----+-------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+---------------+---------+---------+-------+------+----------+-----------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+---------------+---------+---------+-------+------+----------+-----------------+
| 1 | PRIMARY | <derived3> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 100.00 | Start temporary |
| 1 | PRIMARY | lzl | lzl_p20230601,lzl_p20230602,lzl_p20230603,lzl_p20230604,lzl_p20230605,lzl_p20230606,lzl_p20230607,lzl_p20230608,lzl_p20230609,lzl_p20230610,lzl_p20230611,lzl_p20230612,lzl_p20230613,lzl_p20230614,lzl_p20230615,lzl_p20230616,lzl_p20230617,lzl_p20230618,lzl_p20230619,lzl_p20230620,lzl_p20230621,lzl_p20230622,lzl_p20230623,lzl_p20230624,lzl_p20230625,lzl_p20230626,lzl_p20230627,lzl_p20230628,lzl_p20230629,lzl_p20230630 | ref | PRIMARY | PRIMARY | 8 | t.id | 1 | 100.00 | End temporary |
| 3 | DERIVED | lzl | lzl_p20230601,lzl_p20230602,lzl_p20230603,lzl_p20230604,lzl_p20230605,lzl_p20230606,lzl_p20230607,lzl_p20230608,lzl_p20230609,lzl_p20230610,lzl_p20230611,lzl_p20230612,lzl_p20230613,lzl_p20230614,lzl_p20230615,lzl_p20230616,lzl_p20230617,lzl_p20230618,lzl_p20230619,lzl_p20230620,lzl_p20230621,lzl_p20230622,lzl_p20230623,lzl_p20230624,lzl_p20230625,lzl_p20230626,lzl_p20230627,lzl_p20230628,lzl_p20230629,lzl_p20230630 | ref | PRIMARY | PRIMARY | 8 | const | 1 | 100.00 | Using index |

UPDATE plan:

> explain with t as (select id from lzl where id=8723 limit 100 )
 -> update lzl set
 -> STATUS = '00',
 -> FILE_ID = null,
 -> DATE_UPDATED = localtimestamp(0) where id in ( select id from t);
+----+-------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+---------------+---------+---------+-------+------+----------+-----------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+---------------+---------+---------+-------+------+----------+-----------------+
| 1 | PRIMARY | <derived3> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 100.00 | Start temporary |
| 1 | UPDATE | lzl | lzl_p20230601,lzl_p20230602,lzl_p20230603,lzl_p20230604,lzl_p20230605,lzl_p20230606,lzl_p20230607,lzl_p20230608,lzl_p20230609,lzl_p20230610,lzl_p20230611,lzl_p20230612,lzl_p20230613,lzl_p20230614,lzl_p20230615,lzl_p20230616,lzl_p20230617,lzl_p20230618,lzl_p20230619,lzl_p20230620,lzl_p20230621,lzl_p20230622,lzl_p20230623,lzl_p20230624,lzl_p20230625,lzl_p20230626,lzl_p20230627,lzl_p20230628,lzl_p20230629,lzl_p20230630 | ref | PRIMARY | PRIMARY | 8 | t.id | 1 | 100.00 | End temporary |
| 3 | DERIVED | lzl | lzl_p20230601,lzl_p20230602,lzl_p20230603,lzl_p20230604,lzl_p20230605,lzl_p20230606,lzl_p20230607,lzl_p20230608,lzl_p20230609,lzl_p20230610,lzl_p20230611,lzl_p20230612,lzl_p20230613,lzl_p20230614,lzl_p20230615,lzl_p20230616,lzl_p20230617,lzl_p20230618,lzl_p20230619,lzl_p20230620,lzl_p20230621,lzl_p20230622,lzl_p20230623,lzl_p20230624,lzl_p20230625,lzl_p20230626,lzl_p20230627,lzl_p20230628,lzl_p20230629,lzl_p20230630 | ref | PRIMARY | PRIMARY | 8 | const | 1 | 100.00 | Using index |

MySQL’s two execution plans are identical. However, the driving table selection could be better — const should be the driving table to reduce scan count.

Bug?
#

Bug Description
#

https://postgrespro.com/list/thread-id/2482006

This bug is easy to find via the error. It was submitted by digoal (德哥) back in 2020, followed by discussion between two source code experts. The discussion is lengthy, but to summarize: PG does not support unlimited partitions, which is understandable in the real world — too many partitions can cause rapid performance degradation. However, the community still felt the limit needed adjustment and discussed the INNER_VAR, Var.varno values in the source code.

Misleading Nature
#

The bug title is somewhat misleading: BUG #16302: too many range table entries - when count partition table(65538 childs)

The bug seems to say the number of partition child tables can’t exceed 65,538. The discussion also mentions PG can handle up to 64K relations in a query — a query cannot have more than 64K relations.

This is odd because our table has 400 partitions and still throws the error. In fact, both descriptions above are not entirely accurate. The 64K limit refers to the “tables” in the execution plan, which doesn’t exactly equal real tables. Of course, if tables or partitions exceed this count, there will be problems. But even without exceeding 64K, issues can arise, as in our case with only 400 partitions.

Fix
#

The bug was submitted for version 12.2; our environment is 13.2.

This bug is fixed in PG15. The source in src/include/nodes/primnodes.h is different:

#define INNER_VAR		(-1)	/* reference to inner subplan */
#define OUTER_VAR		(-2)	/* reference to outer subplan */
#define INDEX_VAR		(-3)	/* reference to index column */
#define ROWID_VAR		(-4)	/* row identity column during planning */

#define IS_SPECIAL_VARNO(varno)		((int) (varno) < 0)

As discussed in the community, PG15 not only changed VAR values to negative numbers but also converted varno to 32-bit (4 billion), compared to the previous 16-bit (65,536).

And in the function that previously threw the error, add_rte_to_flat_rtable() in src/backend/optimizer/plan/setrefs.c, the error code has been completely removed! The entire PG15 source code no longer contains too many range table entries!

Summary
#

PG still has room for improvement in partitioned table optimization. PG treats child partitions as regular tables, unlike Oracle and MySQL. Oracle treats child partitions as segments distinct from tables. This causes PG to output the access method for every partition in the execution plan (when pruning doesn’t occur), making plans extremely long when there are many partitions. Oracle just writes PARTITION RANGE ALL. MySQL also prints all partitions but doesn’t treat each partition’s access as a subquery, reducing plan complexity.
Even when partitions haven’t reached 64K, you can still get too many range table entries. This limit is actually on execution plan RTE count, not partition count (though if partition count reaches this number, RTE count will too, as mentioned — PG prints access methods for all partitions).
The too many range table entries error is resolved in PG15.
For versions below 15, don’t create too many partitions! You can also leverage partition pruning to reduce accessed partitions — in this case, simply adding a partition key condition to the WHERE clause would work.

Why Is 'partition of' Slow When There's No Blocking?

Mon, 12 Aug 2024 00:00:00 +0000

Analyzing Slow `CREATE TABLE.. PARTITION OF` Statements
#

2024-05-16 22:02:59.063 CST,"user1","dblzl",125889,"30.88.79.3:37423",66461213.1ebc1,2,"authentication",2024-05-16 22:02:59 CST,34/41364668,0,LOG,00000,"connection authorized: user=user1 database=dblzl",,,,,,,,,"","client backend"
2024-05-16 22:02:59.079 CST,"user1","dblzl",125889,"30.88.79.3:37423",66461213.1ebc1,3,"idle",2024-05-16 22:02:59 CST,34/41364669,0,LOG,00000,"statement: -- a86fae372f73414bbe1af18213a47beb
/*a86fae372f73414bbe1af18213a47beb */
create table if not exists table1_partition_p2406 partition of table1 for values from ('2024-06-01 00:00:00') to ('2024-07-01 00:00:00'); ",,,,,,,,,"","client backend"
...
2024-05-16 22:38:28.555 CST,"user1","dblzl",125889,"30.88.79.3:37423",66461213.1ebc1,4,"CREATE TABLE",2024-05-16 22:02:59 CST,34/0,0,LOG,00000,"duration: 2129483.549 ms",,,,,,,,,"","client backend"

The user ‘user1’ connected to the database at 22:02:59 and immediately executed a create table.. partition of.. statement, which didn’t complete until 22:38:28. The logs in between are omitted — there was a lot of session blocking information, with session 125889 as the blocking source.

Blocked sessions looked like:

process 33569 still waiting for RowExclusiveLock on relation 53733 of database 17073 after 1000.048 ms","Process holding the lock: 125889. Wait queue: 33569.

When PARTITION OF adds a partition, it acquires an AccessExclusiveLock (level 8) on the parent table, which blocks all operations on the partitioned table. Normally, adding a partition via PARTITION OF is very fast, and the lock is released immediately. However, if there’s a long-running transaction on the partitioned table, the level 8 lock on the parent table must wait, causing subsequent blocking.

(Stolen from my own diagram):

However, in this case there was no long transaction on the table, yet PARTITION OF took 35 minutes.

From historical process information, this process was in D state (uninterruptible sleep), which was suspicious. Initially, I suspected memory or disk issues, but after investigation, everything was normal.

However, this problem was easy to reproduce — running create table partition of directly in a simulation environment was very slow. pg_stat_activity showed the statement waiting on IO:

wait_event_type | IO
wait_event | DataFileRead
state | active
query | create table xxx partition of xx for values from ('2025-05-01 00:00:00') to ('2025-06-01 00:00:00');

strace tracing revealed the process was heavily reading one file:

pread64(53, "\22\2\0\0\220w\321>\0\0\5\0\24\0018\1\0 \4 \0\0\0\0\200\237\0\1\310\236p\1"..., 8192, 863485952) = 8192

Using file descriptor 53, we identified the file:

[/proc/356174/fd] ll |grep 53
lrwx------ 1 postgres postgres 64 May 17 15:34 53 -> /lzl/pglzl/data/base/17076/25883

oid2name -d lzldb -f 25883
From database "lzldb":
 Filenode Table Name
-----------------------------------------------
 25883 table_partition_default

Finally located: the table table_partition_default:

=> \d+ table_partition_default
...
Partition of: table_partition_default DEFAULT
Partition constraint: (NOT ((date_created IS NOT NULL) AND ((date_created < '2022-05-01 00:00:00'::timestamp without time zone) OR ((date_created >= '2022-05-01 00:00:00'::timestamp without time zone) AND (da


=> \dt+ table_partition_default
 List of relations
 Schema | Name | Type | Owner | Persistence | Size | Description 
--------+------------------------------------+-------+------------+-------------+-------+-------------
 public | table_partition_default | table | user1 | permanent | 50 GB | 
(1 row)

It was the default partition table, with tens of GB of data. Oracle DBAs might find this unfamiliar — PG’s default partition receives data that doesn’t fall into any defined partition range. The default partition ensures data is still accepted even if no matching range is defined.

If data exists in the default partition and a new partition needs to cover that range, what happens? It directly throws an error:

=> create table if not exists table_partition_pxxxx partition of table_partition for values from ('2023-01-12 00:00:00') to ('2023-01-13 00:00:00');
ERROR: 23514: updated partition constraint for default partition "table_partition_default" would be violated by some row
SCHEMA NAME: public
TABLE NAME: table_partition_default
LOCATION: check_default_partition_contents, partbounds.c:3227

As you can see, when adding a child partition, the default partition’s partition constraint is automatically modified. The default partition constraint check is essentially validating the default partition’s data against the new partition’s range.

At this point, the cause is clear:

When adding a new child partition to a partitioned table, the partition creation statement needs to validate data in the default partition to ensure the new partition’s data range doesn’t conflict with existing default partition data. This caused CREATE TABLE PARTITION OF to read a massive amount of default partition data, preventing the new partition from being created. The blocking then cascaded, making business data unqueryable and unwritable.

Summary and Recommendations
#

PostgreSQL partitioned tables are becoming increasingly common. Maintaining partitions requires attention to many details. I recommend reading PostgreSQL Partitioned Tables, which covers almost everything.

In this case, the key to resolution is the data in the default partition. Before refactoring the default partition, do not use PARTITION OF to create child partitions.

Default partition refactoring plan:

Detach the default child partition, then properly create child partitions, and reinsert the default table data back into the partitioned table.
If necessary, after detaching and creating proper child partitions, create an empty default partition to maintain business data continuity.
Note that detach differs from attach — detach requires a level 8 lock on the parent table. PG14 supports DETACH CONCURRENTLY.

If you don’t refactor the default partition, check the current data range in the default partition. Using ATTACH to add child partitions will be slow, but won’t block reads and writes.

Finally, a review of best practices for adding partitions:

PARTITION OF requires a level 8 lock on the parent table, which carries risk. The recommended approach is to use ATTACH to add new child partitions (partition indexes can be handled similarly). This does not block reads and writes, has no business impact, and can be done online.

The correct approach for adding new partitions:

CREATE TABLE lzlpartition1_202303
 (LIKE lzlpartition1 INCLUDING DEFAULTS INCLUDING CONSTRAINTS);
alter table LZLPARTITION1 attach partition LZLPARTITION1_202303 for values from ('2023-03-01 00:00:00') to ('2023-04-01 00:00:00');

If the new partition already has data, ATTACH may still be slow. You can optimize by pre-creating constraints:

The correct approach for adding a partition that already has data:

-- Reduce verbose DDL by using LIKE
CREATE TABLE lzlpartition1_202303
 (LIKE lzlpartition1 INCLUDING DEFAULTS INCLUDING CONSTRAINTS);
-- Skip this step if no data exists. Add a CHECK constraint referencing other partitions' Partition constraint to reduce ATTACH constraint validation time.
alter table lzlpartition1_202303 add constraint chk_202303 CHECK ((date_created IS NOT NULL) AND (date_created >= '2023-03-01 00:00:00'::timestamp without time zone) AND (date_created < '2023-04-01 00:00:00'::timestamp without time zone));
-- Add partition via ATTACH
alter table LZLPARTITION1 attach partition LZLPARTITION1_202303 for values from ('2023-03-01 00:00:00') to ('2023-04-01 00:00:00');
-- Optional. Before transactions occur on the new partition, drop the extra CHECK constraint
alter table lzlpartition1_202303 drop constraint chk_202303;

PostgreSQL案例 on Last DBA

Case Study: Startup Failure and SysV Shared Memory

Problem Symptoms #

Error Analysis: pre-existing shared memory block #

Three Types of Shared Memory #

Source Code Analysis #

SysV shmem #

What Are shmkey and shmid? #

How PG Obtains the shmkey #

PG shmkey in Cloud Environments #

shmid Relationships #

Testing #

Reproducing the Production Issue #

Normal Crash Recovery (Successful Startup) #

Holding a File Descriptor But Not shmem #

Deleting postmaster.pid Then Failing to Start #

Stop a Different Instance, Start the Current One #

Error Analysis: lock file "postmaster.pid" already exists #

Summary #

PostgreSQL Operations Experience 2025

CPU #

Execution Plans #

Inaccurate DISTINCT Estimates #

Generic Plan Interference #

LWLock:Lockmanager Caused by Row Locks #

Idle Connections #

Idle in Transaction #

Memory #

Memory Issues and Huge Pages #

Notable Cgroup Knowledge #

Unsolved Mysteries #

Pay Attention to the OS #

Pay Attention to Everything OS #

Physical Reads #

Autovacuum #

Autovacuum Freeze on Large Databases #

Large Tables That Won’t Finish Vacuuming #

Checkpoint and Bgwriter #

DB4AI #

AI Task Scheduling Writes to Database #

Vector HNSW #

HNSW Index Build Acceleration #

HNSW Index Query Performance #

Application Releases #

DDL Tips #

Parallel Index Creation #

Cached Plan Must Not Change Resource #

Physical Replication #

Query Conflicts #

Logical Replication #

Slow DDL/DCL Parsing on Older PG Versions #

Older PG + Multiple Replication Links + Flink #

Older PG Cannot Auto-Sync New Partitions #

Migration and Upgrades #

Xinchuang Migration and glibc Upgrades #

Smooth Major Version Upgrades #

Partitioned Table Management #

Observability #

buffers_alloc, blks_read #

tup_fetched, tup_returned #

idx_tup_fetch, idx_tup_read #

References #

Case: Partition Data UPDATE Failure on 2026-01-01

Symptoms #

Temporary Recovery #

Root Cause Analysis #

Question 1: Why does the UPDATE fail even though there’s no 202601 data at all (the new partition has zero rows)? #

Question 2: The partition was created on 2025-12-26, so why was the problem only discovered on December 30? #

Summary #

References #

Case Study: Row Locks and LWLock LockManager

Symptoms #

Analysis #

No Increase in SQL Concurrency Observed #

LWLock LockManager Analysis #

Benchmark Testing #

Benchmarking Same-Row Updates to Reproduce LWLock LockManager #

A Brief Analysis of Row Locks and Fast Path #

Conclusion #

Case: From Inaccurate DISTINCT to the Principles of DISTINCT Estimation

Problem Symptoms
#

Error Analysis: `pre-existing shared memory block`
#

Three Types of Shared Memory
#

Source Code Analysis
#

SysV shmem
#

What Are shmkey and shmid?
#

How PG Obtains the shmkey
#

PG shmkey in Cloud Environments
#

shmid Relationships
#

Testing
#

Reproducing the Production Issue
#

Normal Crash Recovery (Successful Startup)
#

Holding a File Descriptor But Not shmem
#

Deleting postmaster.pid Then Failing to Start
#

Stop a Different Instance, Start the Current One
#

Error Analysis: `lock file "postmaster.pid" already exists`
#

Summary
#

CPU
#

Execution Plans
#

Inaccurate DISTINCT Estimates
#

Generic Plan Interference
#

LWLock:Lockmanager Caused by Row Locks
#

Idle Connections
#

Idle in Transaction
#

Memory
#

Memory Issues and Huge Pages
#

Notable Cgroup Knowledge
#

Unsolved Mysteries
#

Pay Attention to the OS
#

Pay Attention to Everything OS
#

Physical Reads
#

Autovacuum
#

Autovacuum Freeze on Large Databases
#

Large Tables That Won’t Finish Vacuuming
#

Checkpoint and Bgwriter
#

DB4AI
#

AI Task Scheduling Writes to Database
#

Vector HNSW
#

HNSW Index Build Acceleration
#

HNSW Index Query Performance
#

Application Releases
#

DDL Tips
#

Parallel Index Creation
#

Cached Plan Must Not Change Resource
#

Physical Replication
#

Query Conflicts
#

Logical Replication
#

Slow DDL/DCL Parsing on Older PG Versions
#

Older PG + Multiple Replication Links + Flink
#

Older PG Cannot Auto-Sync New Partitions
#

Migration and Upgrades
#

Xinchuang Migration and glibc Upgrades
#

Smooth Major Version Upgrades
#

Partitioned Table Management
#

Observability
#

buffers_alloc, blks_read
#

tup_fetched, tup_returned
#

idx_tup_fetch, idx_tup_read
#

References
#

Symptoms
#

Temporary Recovery
#

Root Cause Analysis
#

Question 1: Why does the UPDATE fail even though there’s no 202601 data at all (the new partition has zero rows)?
#

Question 2: The partition was created on 2025-12-26, so why was the problem only discovered on December 30?
#

Summary
#

References
#

Symptoms
#

Analysis
#

No Increase in SQL Concurrency Observed
#

LWLock LockManager Analysis
#

Benchmark Testing
#

Benchmarking Same-Row Updates to Reproduce LWLock LockManager
#

A Brief Analysis of Row Locks and Fast Path
#

Conclusion
#

Problem Description
#

Analysis
#

Sampling Model
#

DISTINCT Estimation Algorithm
#

Verification
#

Solution
#