PostgreSQL内功修炼 on Last DBA

UUID v4 and v7: Collision Incidents and Performance Benchmarks

Fri, 29 May 2026 00:00:00 +0000

Source material: HN UUID v4 Collision Thread, dev.to UUID Benchmark

AI-generated ratio: 99%

TL;DR
#

UUID v4 collided — someone on HackerNews actually hit a real collision. The root cause was a software stack bug, not math. v4 and v7 have no fundamental difference in collision safety. The real difference is index performance: v7 is time-ordered, B-tree is more compact, writes are 35% faster, indexes are 22% smaller. Your UUID v4 is probably fine, but if you care about index performance, switching to v7 is a cheap win.

The UUID v4 Collision Incident
#

A HackerNews thread blew up — Ask HN: We just had an actual UUID v4 collision…, 479 upvotes, 347 comments.

The OP’s own words:

I know what you’re thinking… and I still can’t believe it, but… This morning, our database flagged a duplicate UUID (v4).

It wasn’t a double-insert bug. The code didn’t write it twice. Only ~15,000 rows in the table, using npm’s uuid package uuidv4(), and two rows created at different times collided on the same UUID:

b6133fd6-70fe-4fe3-bed6-8ca8fc9386cd

What’s the probability of a UUID v4 collision? 122 random bits, 2^122 ≈ 5.3×10^36 possibilities. With 15,000 records, collision probability is roughly 2×10^-29. Theoretically “impossible.”

But it happened.

Cause 1: Unreliable entropy sources
#

HN’s top-voted comment (jandrewrogers):

UUIDv4 security depends on high-quality entropy sources. Hardware defects, software bugs, and misunderstandings of “high-quality entropy” all break this assumption. Detecting entropy source failures is expensive, so nobody checks — until a collision happens.

UUID v4 is explicitly banned in high-reliability systems because entropy source quality cannot be verified.

Cause 2: Known npm uuid package bugs
#

The npm uuid package README itself warns:

This module may generate duplicate UUIDs when run in clients with deterministic random number generators, such as Googlebot crawlers.

More seriously, its internal rng() function has global mutable state. One commenter pointed out: calling rng() and sending the result effectively overwrites someone else’s random number, and you can predict it.

Related commit: 91805f665c

Community advice: use Node.js built-in crypto.randomUUID(), not the npm uuid package.

Cause 3: Linux kernel /dev/random race condition
#

Another comment:

I encountered duplicate UUIDs during soak testing of a distributed system. After extensive debugging, I found it was a Linux kernel race condition bug — on multi-processor systems, two processes simultaneously reading /dev/random could, with extremely low probability (~one in a million), get the same bytes.

Cause 4: Go UUID library not checking return values
#

Early Go UUID libraries called random functions without checking the return value length. “Request N bytes, got 3 bytes back” never happened on most hardware, so nobody checked — until production, where it generated thousands of duplicate UUIDs.

Cause 5: Historical AMD CPU RNG defects
#

Certain AMD CPUs had built-in random number generator issues. VM environments can also “virtualize away” entropy — both time sources and entropy sources can degrade inside VMs.

v4 and v7 have no fundamental difference in collision safety. The difference is in the first 48 bits — v4 is random, v7 is a timestamp. You’re unlikely to encounter timestamp source issues, and random source issues are equally rare. The HN thread is an interesting edge case. Knowing that a tiny number of people hit it is enough — you don’t need to distrust the UUID v4 in your own systems.

When choosing v4 vs v7, what you should really look at isn’t collisions — it’s index performance.

UUID v7 Performance Comparison in PG 16
#

UUID v7 has one concrete advantage over v4 in PostgreSQL: temporal clustering, more B-tree-friendly. v4 can bloat and v7 can bloat too — the difference is simply that v7’s first 48 bits are time-ordered, so inserts concentrate on the right side of the B-tree, reducing page splits.

Umang Sinha’s benchmark ran a rigorous comparison on a PG 16 Docker container (8 cores, 16GB, NVMe).

Test Conditions
#

CREATE TABLE uuid_v4_test (id UUID PRIMARY KEY, payload TEXT);
CREATE TABLE uuid_v7_test (id UUID PRIMARY KEY, payload TEXT);

Parameter	Value
Data volume	10 million rows per table
Batch size	10,000 rows per batch
Client	Go + pq driver
UUID generation	Pre-generated in memory, not timed

Performance Results
#

Metric	UUID v4	UUID v7	Improvement
Write 10M rows	5 min 35 sec	3 min 38 sec	35% faster
Table + index total size	3618 MB	3443 MB	5% smaller
B-tree index size	776 MB	602 MB	22% smaller
Point lookup	0.167 ms	0.038 ms	4.4x faster
Range scan	8.283 ms	3.791 ms	2.2x faster

Why Such a Big Difference
#

UUID v4 is fully random. Newly inserted UUIDs scatter randomly across the B-tree index, causing massive page splits and severe index fragmentation. UUID v7 has a millisecond-precision timestamp in the first 48 bits, so newly generated UUIDs are naturally ordered — writes cluster on the right side of the B-tree, page splits drop dramatically, and the index is much more compact.

The 22% smaller index isn’t magic — it’s reduced fragmentation. Point lookups being 4x faster isn’t surprising either — fewer B-tree levels, higher cache hit rates.

Summary
#

UUID v4 and v7 are identical in collision safety — both depend on entropy source quality, one fills the first 48 bits with random numbers, the other with a timestamp. Collisions are edge cases that a tiny number of people hit in specific environments. Your environment is probably fine — that basic judgment doesn’t change.

What you really should think about is index performance. v7’s temporal property makes B-trees more compact, with measured results of 35% faster writes, 22% smaller indexes, and 2-4x faster queries. If your system writes UUIDs at high volume, switching to v7 saves meaningful storage and CPU.

PG 18 will natively support gen_uuid_v7(). For now, generate UUIDs at the application layer. Whichever version you use, always add a UNIQUE constraint.

This article was originally published in Chinese on lastdba.com.

From collation mismatch Exception to Its Principles

Sat, 13 Dec 2025 00:00:00 +0000

Problem Phenomenon
#

After physical migration to Xinchuang, occasional errors appear in the pg log, version pg15:

WARNING: 01000: collation "zh_CN.utf8" has version mismatch
DETAIL: The collation in the database was created using version 2.17, but the operating system provides version 2.28.
HINT: Rebuild all objects affected by this collation and run ALTER COLLATION pg_catalog."zh_CN.utf8" REFRESH VERSION, or build RaseSQL with the right library version.
LOCATION: pg_newlocale_from_collation, pg_locale.c:1660

Context: During the physical switch, invalid index rebuilding and refresh database collation version were performed.

Although the libc version was upgraded after physical migration, indexes were rebuilt and are now valid, and the collation version in the database is already consistent with the OS libc.

So,

Why is the error reported?

Where is the error triggered?

What is the impact of the error?

How to resolve it?

Problem Analysis
#

Why is the error reported?
#

The collation inside the database mainly involves 3 aspects: database, columns, and indexes. The first two use default collation, and the index collation is the real collation.

First, check the database collation. All databases use en_US.UTF8, and refresh database collation has already been done, so the “collation "zh_CN.utf8" has version mismatch” error should not be thrown at the database layer.

Then check columns without specially specified default collation:

select attrelid,attname,attcollation from pg_attribute where attcollation not in (0,100,950,951);
 attrelid | attname | attcollation 
----------+---------+--------------
(0 rows)

0 means no collation, default oid=100, C oid=950, POSIX oid=951; “zh_CN.utf8” definitely won’t be any of these four.

Finally, check indexes without specially specified collation:

select * from (select indexrelid ,unnest(indcollation) coll from pg_index) i where coll not in (0,100,950,951);
 indexrelid | coll 
------------+------
(0 rows)

Having ruled out database, columns, and indexes, only one situation remains: the application layer specifies a sort rule:

select col1 from (values ('a'), ('A'), ('啊'), ('阿')) AS l(col1) order by col1 collate "zh_CN.utf8";
WARNING: 01000: collation "zh_CN.utf8" has version mismatch
DETAIL: The collation in the database was created using version 2.17, but the operating system provides version 2.28.
HINT: Rebuild all objects affected by this collation and run ALTER COLLATION pg_catalog."zh_CN.utf8" REFRESH VERSION, or build RaseSQL with the right library version.
LOCATION: pg_newlocale_from_collation, pg_locale.c:1660
 col1 
------
 阿
 啊
 a
 A

This zh_CN.utf8 version is inconsistent with the actual one:

 select collname,collversion,pg_collation_actual_version(oid) from pg_collation where collname ='zh_CN.utf8';
 collname | collversion | pg_collation_actual_version 
------------+-------------+-----------------------------
 zh_CN.utf8 | 2.17 | 2.28

Not only zh_CN.utf8 is different, all are different (except a few collations without version concept).

So it’s very likely that the application itself specified a sort rule “zh_CN.utf8”, but the coll version in the database is inconsistent with the OS, which triggered the error.

Source Code Understanding
#

The error message makes it easy to locate the source code position. Two main functions are of interest: pg_newlocale_from_collation and CheckMyDatabase.

`pg_newlocale_from_collation` Caching and Checking `pg_collation`
#

pg_newlocale_from_collation was introduced in pg10.

/*
 * Create a locale_t from a collation OID. Results are cached for the
 * lifetime of the backend. Thus, do not free the result with freelocale().
 *
 * As a special optimization, the default/database collation returns 0.
 * Callers should then revert to the non-locale_t-enabled code path.
 * In fact, they shouldn't call this function at all when they are dealing
 * with the default locale. That can save quite a bit in hotspots.
 * Also, callers should avoid calling this before going down a C/POSIX
 * fastpath, because such a fastpath should work even on platforms without
 * locale_t support in the C library.
 *
 * For simplicity, we always generate COLLATE + CTYPE even though we
 * might only need one of them. Since this is called only once per session,
 * it shouldn't cost much.
 */
/* locale_t means non-ICU. This function caches a locale_t type collation OID for the backend
* the default/database collation returns 0. "default" means using the database's collation
*/
pg_locale_t
pg_newlocale_from_collation(Oid collid) // Note: passes in collation oid, not fetching all pg_collation
{
...
	/* Return 0 for "default" collation, just in case caller forgets */
	if (collid == DEFAULT_COLLATION_OID) // Three special collations:
		return (pg_locale_t) 0; // default oid=100, C oid=950, POSIX oid=951

...
	if (cache_entry->locale == 0)
	{
...
		collversion = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_collversion,
									 &isnull); // Get version from pg_collation data dictionary
		if (!isnull)
		{
...
			actual_versionstr = get_collation_actual_version(collform->collprovider, collcollate); // Get actual version via get_collation_actual_version
...
			collversionstr = TextDatumGetCString(collversion);

			if (strcmp(actual_versionstr, collversionstr) != 0) // Compare data dictionary version and actual version, throw error if different
				ereport(WARNING,
						(errmsg("collation \"%s\" has version mismatch",
								NameStr(collform->collname)),
						 errdetail("The collation in the database was created using version %s, "
								 "but the operating system provides version %s.",
								 collversionstr, actual_versionstr),
						 errhint("Rebuild all objects affected by this collation and run "
								 "ALTER COLLATION %s REFRESH VERSION, "
								 "or build PostgreSQL with the right library version.",
								 quote_qualified_identifier(get_namespace_name(collform->collnamespace),
															NameStr(collform->collname)))));
		}
...
	return cache_entry->locale;
}

The main check is: through the coll oid, check whether the version in the pg_collation data dictionary is consistent with the actual version; if inconsistent, throw an error.

`CheckMyDatabase` Caching and Checking `pg_database`
#

CheckMyDatabase has existed for a long time, performing many database-side checks. However, pg15 added logic for checking the database version.

/*
 * CheckMyDatabase -- fetch information from the pg_database entry for our DB
 */
static void
CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connections)
{
...
	/* Fetch our pg_database row normally, via syscache */
	tup = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
...
	default_locale.provider = dbform->datlocprovider; // default is the db's

	/*
	 * Default locale is currently always deterministic. Nondeterministic
	 * locales currently don't support pattern matching, which would break a
	 * lot of things if applied globally.
	 */
	default_locale.deterministic = true; // byte-order sensitive

	/*
	 * Check collation version. See similar code in
	 * pg_newlocale_from_collation(). Note that here we warn instead of error
	 * in any case, so that we don't prevent connecting.
	 */
	datum = SysCacheGetAttr(DATABASEOID, tup, Anum_pg_database_datcollversion,
							&isnull); // Get datcollversion from pg_database
	if (!isnull)
	{
		char	 *actual_versionstr;
		char	 *collversionstr;

		collversionstr = TextDatumGetCString(datum);

		actual_versionstr = get_collation_actual_version(dbform->datlocprovider, dbform->datlocprovider == COLLPROVIDER_ICU ? iculocale : collate); // Get actual version via get_collation_actual_version
...
		else if (strcmp(actual_versionstr, collversionstr) != 0) // Compare db datcollversion and actual version, throw warning if not equal
			ereport(WARNING,
					(errmsg("database \"%s\" has a collation version mismatch",
							name),
					 errdetail("The database was created using collation version %s, "
							 "but the operating system provides version %s.",
							 collversionstr, actual_versionstr),
					 errhint("Rebuild all objects in this database that use the default collation and run "
							 "ALTER DATABASE %s REFRESH COLLATION VERSION, "
							 "or build PostgreSQL with the right library version.",
							 quote_identifier(name))));
	}
...
}

The CheckMyDatabase function compares the datcollversion in the pg_database data dictionary with the actual version.

Function Differences
#

In pg14 and before, there was only 1 collation comparison logic: when a session first caches the corresponding collation, it calls pg_newlocale_from_collation to access the version of the corresponding collation in the pg_collation data dictionary and compare it with the real version.
In PG15 and later, because the datcollversion field was added to the pg_database table, a new logic for checking db collation version was added: when a session first accesses the db in pg_database, it calls CheckMyDatabase to check the datcollversion of the corresponding database in pg_database and compare it with the real version.

Why Are There Fewer Errors After Only Refreshing the Database?
#

After refreshing the database collation version, the warning about inconsistent pg_database coll version won’t be triggered, but it still cannot rule out the situation where pg_collation’s coll version is inconsistent. Why are there so many fewer errors after only refreshing the database? Could it be that pg_collation’s coll version simply won’t be loaded?

select c.coll,count(*) from (select unnest(indcollation) coll from pg_index ) c group by c.coll;
 coll | count 
------+-------
 950 | 37 --C
 0 | 2841 --No collation
 100 | 723 --default

In real environments, default is the most used. Generally, no one specifies a collation; if not specified it’s default, and default is the database’s default collation.

Here we need to revisit the pg_newlocale_from_collation function. The function starts like this:

pg_locale_t
pg_newlocale_from_collation(Oid collid)
{
	collation_cache_entry *cache_entry;

	/* Callers must pass a valid OID */
	Assert(OidIsValid(collid));

	/* Return 0 for "default" collation, just in case caller forgets */
	if (collid == DEFAULT_COLLATION_OID)
		return (pg_locale_t) 0;
...

When collid==DEFAULT_COLLATION_OID==100, it directly returns without executing the real version check below, so it won’t throw a warning. This logic is reasonable because the db coll version has already been verified when logging into the database; if there’s a problem, a warning must have already been thrown at the session layer.

Furthermore, even if a possible value like collid=37 is passed in, the corresponding C also has no version concept.

Therefore, after refreshing the database, in the vast majority of scenarios, as long as the database’s internal sorting is used (not expression sorting or specified index sorting), no error will be thrown.

Testing
#

Here we only test whether there is a refresh warning, not testing index corruption or database crashes.

# Check libc version
getconf GNU_LIBC_VERSION

Source host version glibc 2.17
Target host glibc 2.28
pg version pg15+

Test: Refresh db without refreshing pg_collation, only db coll version changes
#

select datname,datlocprovider,datcollate,datctype,datcollversion from pg_database 
 datname | datlocprovider | datcollate | datctype | datcollversion 
------------+----------------+-------------+-------------+----------------
 lzldb | c | en_US.UTF-8 | en_US.UTF-8 | 2.17
 
select collname,collprovider,collversion,pg_collation_actual_version(oid) from pg_collation where collname ~ 'en_US.utf8';
 collname | collprovider | collversion | pg_collation_actual_version 
------------+--------------+-------------+-----------------------------
 en_US.utf8 | c | 2.17 | 2.28

alter database lzldb refresh collation version;
NOTICE: 00000: changing version from 2.17 to 2.28
LOCATION: AlterDatabaseRefreshColl, dbcommands.c:2399
ALTER DATABASE

Check pg_collation and pg_database again:

 collname | collprovider | collversion | pg_collation_actual_version 
------------+--------------+-------------+-----------------------------
 en_US.utf8 | c | 2.17 | 2.28

 datname | datlocprovider | datcollate | datctype | datcollversion 
------------+----------------+-------------+-------------+----------------
 lzldb | c | en_US.UTF-8 | en_US.UTF-8 | 2.28

Consistent with the official documentation description: refresh database collation version only refreshes the db’s default collation; pg_collation itself won’t change.

Test: Refresh db without refreshing pg_collation, specifying expression sort reports warning
#

As analyzed at the beginning, expression sorting will report a warning, omitted.

Test: Refresh db without refreshing pg_collation, creating a new index with specified collation reports warning
#

Test 1: Specify collation when creating index

 collname | collversion | pg_collation_actual_version 
------------+-------------+-----------------------------
 zh_CN.utf8 | 2.17 | 2.28
 
 > create index idx11 on tt(a collate "zh_CN.utf8");
WARNING: 01000: collation "zh_CN.utf8" has version mismatch
DETAIL: The collation in the database was created using version 2.17, but the operating system provides version 2.28.
HINT: Rebuild all objects affected by this collation and run ALTER COLLATION pg_catalog."zh_CN.utf8" REFRESH VERSION, or build PostgreSQL with the right library version.
LOCATION: pg_newlocale_from_collation, pg_locale.c:1664
CREATE INDEX

Test 2: Specify column default collation when creating table, don’t specify when creating index

\c lzldb -- Reconnect a session
You are now connected to database "lzldb" as user "postgres".
create table ttt(a varchar(10) collate "zh_CN.utf8");
CREATE TABLE

> create index idxttt on ttt(a);
WARNING: 01000: collation "zh_CN.utf8" has version mismatch
DETAIL: The collation in the database was created using version 2.17, but the operating system provides version 2.28.
HINT: Rebuild all objects affected by this collation and run ALTER COLLATION pg_catalog."zh_CN.utf8" REFRESH VERSION, or build PostgreSQL with the right library version.
LOCATION: pg_newlocale_from_collation, pg_locale.c:1664
CREATE INDEX
Time: 7.904 ms

Column default collation and index specification of collation are essentially the same thing, both for specifying the index’s collation. Both can report warnings.

Test: Refresh db without refreshing pg_collation, existing index with specified collation does not report warning
#

Scenario: The original database already has an index specifying collation zh_CN.utf8, different from the db. Refreshing the db won’t catch it. But after migrating to a new database, the vendor’s coll version definitely changed.

select collname,collprovider,collversion,pg_collation_actual_version(oid) from pg_collation where collname ~ 'zh_CN.utf8';
 collname | collprovider | collversion | pg_collation_actual_version 
------------+--------------+-------------+-----------------------------
 zh_CN.utf8 | c | 2.17 | 2.28

Without using expression sorting, the index can be used, but index sorting cannot be used:

> set enable_seqscan =off;
SET
> EXPLAIN ANALYZE SELECT a FROM tt ORDER BY a LIMIT 1000;
 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=6667.80..6670.30 rows=1000 width=33) (actual time=44.928..45.145 rows=1000 loops=1)
 -> Sort (cost=6667.80..6892.81 rows=90004 width=33) (actual time=44.926..45.021 rows=1000 loops=1)
 Sort Key: a
 Sort Method: top-N heapsort Memory: 127kB
 -> Index Only Scan using idxtt on tt (cost=0.42..1732.98 rows=90004 width=33) (actual time=0.029..15.434 rows=90004 loops=1)
 Heap Fetches: 4

Existing indexes with specified collation do not report warnings when used.

Summary of This Problem
#

The refresh database and refresh collation warnings are session-level. In each session, for each database or each collation, it only reports once.

Only refreshing the database very likely won’t report warnings again, but there are situations where creating an index with a specified collation or running SQL with specified expression collation may still report warnings.

The coll version in the data dictionary is only for tracking whether the collation provider version has changed at the database layer. Imagine if there were no coll version in the data dictionary - the database might not even be able to return a warning saying “your sort rule provider has upgraded its version, your data sorting might have problems, you need to check it” (and of course it’s not just about sorting).

Solutions for This Problem
#

Corrupt indexes have already been rebuilt, the database has been refreshed, only collation hasn’t been refreshed. The inconsistency of coll version in the data dictionary is not a big problem, it’s just a warning. As for other hidden and strange pitfalls, refer to the more section.

Solution for this problem:

Step 1: Check if there are still dependencies

SELECT pg_describe_object(refclassid, refobjid, refobjsubid) AS "Collation",
 pg_describe_object(classid, objid, objsubid) AS "Object"
 FROM pg_depend d JOIN pg_collation c
 ON refclassid = 'pg_collation'::regclass AND refobjid = c.oid
 WHERE c.collversion <> pg_collation_actual_version(c.oid)
 ORDER BY 1, 2;

If there are returns, it’s best to rebuild the dependent objects; if not, follow step 2:

Solution 1: Do nothing. If there aren’t many warnings, leaving them alone is fine.
Solution 2: Only refresh collation zh_CN.UTF8. Fix one as it comes.
Solution 3: Refresh all collations. Even if the application incrementally uses expressions or index-specified collation, no warnings will be reported.

More
#

Key Summary of glibc Upgrade Related Issues
#

Locale is a very tricky area, and glibc upgrades cause many collation-related problems. Referencing reference materials, here’s a summary of some important points:

pg_collation is obtained from the OS command locale -a; the provider is basically glibc, so you need to look at the glibc version.

In pg_collation, “C” and “posix” have collprovider c, which looks the same as “C.UTF8” etc., but they’re not. “C.UTF8”’s provider is glibc, has a version, generally Unicode codepoint sorting or Unicode semantic sorting; “C” and “POSIX” are equivalent, the most basic locale defined by the POSIX standard, implemented by libc, not in locale -a, has no version, sorts directly by byte order.

Root cause of collation problems: The database requires that locale definitions never change during the database lifecycle, but OS vendors, especially the GNU C library, make changes to locale in every minor version, and this is legitimate.

GNU C library makes changes to locale in every minor version. The version most prone to problems in reality is glibc 2.28, because 2.28 upgraded the major version unicode 9.0.0 (has been updated to a new upstream version from ISO which is in sync with Unicode 9.0.0).

pg has no way to detect compatibility issues caused by glibc upgrades. Index corruption checking is not an all-check, and indexes are only one aspect. After physical replication or upgrade, even if indexes are rebuilt, you cannot rule out the possibility that the database crashes one day due to collation version issues.

Data anomalies include: duplicate primary keys, sort-dependent constraints, range partition table data written to wrong partitions, mergejoin and other sort operations, etc.

Character types depend on collation. Data types that don’t depend on collation:

bytea
tsvector gin indexes
pg_trgm indexes
numeric data types: int, bigint, numeric, float, …
custom data types like geometry (PostGIS)
timestamp

ASCII sorting is relatively common but doesn’t conform to human understanding, i.e., not semantic. Semantically conforming international sorting standards are generally Unicode standards.

Unicode-based sorting rules are divided into 2 types: codepoint sorting, UCA (Unicode Collation Algorithm).

UCA is based on DUCET (Default Unicode Collation Element Table). The DUCET table itself may have sorting changes between different versions. For example, en_US.UTF8 is UCA sorting, equivalent to semantic sorting; version upgrades will change sorting rules. C.UTF8 is codepoint sorting; once codepoints are confirmed they won’t change, and sorting rules won’t change.

PG 17+ provides a very safe locale provider method: builtin, no longer depending on OS-provided glibc, ICU and other providers. Example enable command:

initdb --locale-provider=builtin --bultin-locale=C.UTF-8 dbname1

17 only supports C, C.UTF-8. C is byte-order sorting (approximately ASCII sorting), C.UTF-8 is Unicode codepoint sorting; 18 adds one more PG_UNICODE_FAST, also Unicode codepoint sorting, with slight differences from C.UTF-8.

Because the database must maintain stable sorting, custom application sorting can only be pushed to the application layer. For example, expression sorting is semantically clear and doesn’t affect the database’s own choice of collation. If one day pg also supports built-in en_US.utf8, then we can consider built-in semantic sorting.

During Xinchuang migration, the glibc version of Xinchuang hosts is generally higher than old Intel server glibc versions, likely crossing the 2.28 version. Combined with tight deadlines, KPI pressure, insufficient manpower, and large databases, physical migration is unavoidable. So Xinchuang physical migration needs to pay attention to glibc versions and many anomalies caused by collation.

What to Do After Physical Migration
#

Assuming the database is en_US.utf8, provider c, and physical migration across libc versions has already been done, the following operations should be performed:

I. Official Required Solution

At minimum, rebuild problematic indexes. Install the amcheck extension and use the bt_index_check function:

SELECT bt_index_check('idx1'::regclass, true);

Refresh database version (pg15+):

ALTER DATABASE name REFRESH COLLATION VERSION

Check if there are other dependent objects. If there are, handle them accordingly:

SELECT pg_describe_object(refclassid, refobjid, refobjsubid) AS "Collation",
 pg_describe_object(classid, objid, objsubid) AS "Object"
 FROM pg_depend d JOIN pg_collation c
 ON refclassid = 'pg_collation'::regclass AND refobjid = c.oid
 WHERE c.collversion <> pg_collation_actual_version(c.oid)
 ORDER BY 1, 2;

After handling, then:

Refresh collation version (pg10+):

ALTER COLLATION name REFRESH VERSION

II. Unofficial Workaround Solutions

I haven’t made a complete solution here, just some thoughts.

Handling partition table data written to wrong partition:

Partition key is int/bigint/float, no relation to collation, can be ignored.

Partition key is time partition, if timestamp, can be ignored. If varchar or other character types, depends on the situation.

Partition key is character type, refer to “a” and “-” sorting (pgconf Collation Challenges Sorting It Out). But note the following points:

If querying data, don’t query from the parent table; it might crash or fail to return results.
There’s no simple detection solution.

Handling primary key/unique key conflicts.
Handling fdw sort range anomaly issues.
Unknown problems.

ref
#

https://wiki.postgresql.org/wiki/Locale_data_changes

https://wiki.postgresql.org/wiki/Collations

pgconf Collation Challenges Sorting It Out

PFCONF Collations from A to Z

http://www.unicode.org/reports/tr10/tr10-34.html

https://sourceware.org/glibc/wiki/Release/2.28

https://www.postgresql.org/docs/18/sql-altercollation.html

https://www.postgresql.org/docs/18/sql-alterdatabase.html

https://www.postgresql.org/docs/17/locale.html#LOCALE-PROVIDERS

A Brief Review of Logical Replication in Oracle, MySQL, and PostgreSQL

Sun, 30 Nov 2025 00:00:00 +0000

PostgreSQL Logical Replication
#

（https://www.pgconf.asia/JA/2017/wp-content/uploads/sites/2/2017/12/D2-A7-EN.pdf）

PostgreSQL places all logical decoding related matters entirely within the database’s replication slots for management — an all-inclusive approach. Early versions had somewhat limited logical replication support, but in recent major versions, logical replication has been one of the primary functional improvements.

Advantages of the PG approach:

Very flexible: it exposes the logical decoding interface to users, with multiple types of decoding methods available.
Users can subscribe to only the data they need based on their requirements.

Disadvantages of the PG approach:

The number of concepts to learn and the learning cost are relatively higher compared to MySQL. Just the basic concepts — publication, subscription, walsender, replication slots, output plugins, etc. — I believe many people haven’t fully grasped their definitions and relationships.
Does the hardest work and takes the hardest hits. All logical decoding problems are exposed within the database: WAL backlog, large transactions, long transactions, reorder transaction sorting, privilege issues, streaming transmission — these are all problems PG has to deal with.

MySQL’s binlog
#

(https://blog.fasterinfo.top/6243.html)

MySQL places all decoded logical data locally — in binlog files. The approach is simple. MySQL’s binlog is roughly equivalent to PostgreSQL with full-table logical replication enabled and written locally.

Advantages of the MySQL approach:

Simple and straightforward: MySQL doesn’t expose the logical decoding interface directly to users. Instead, it provides already-decoded files directly to users, who don’t need to care about how parsing works — just read the binlog files.
Mature ecosystem. I personally believe MySQL’s mature ecosystem is closely tied to binlog. During the internet era, PG’s logical replication was still weak, while binlog was extremely simple. Downstream parsing of binlog to put data onto other platforms became a common pattern.

Disadvantages of the MySQL approach:

All data must be decoded; no customizable subscription. Poor flexibility.
Two-phase commit. Because MySQL’s primary-standby replication heavily depends on binlog, binlog data must be fully flushed to binlog files at commit time. A single commit must write two (or two kinds of) logs — binlog and redolog. Dual log writes are one of MySQL’s eternal pain points.

Oracle Logical Replication
#

（https://www.oracle-scn.com/oracle-goldengate-integrated-capture/）

Oracle itself does have logical Data Guard functionality, but virtually no one uses it. Here we’ll only discuss LogMiner. The Oracle database itself provides an interface like LogMiner for parsing logs (e.g., OGG integrated capture mode), but has zero replication link management itself — it relies on third-party tools to create and manage replication links.

Advantages of the Oracle approach:

Only provides a parsing interface, no replication link management. For the database itself, this is very hassle-free.
Pay and you get a solution. Just buy the powerful OGG directly. Don’t say Oracle hasn’t provided a logical replication solution — we not only have one, it’s powerful and highly recognized.

Disadvantages of the Oracle approach:

Relies on third-party software to manage replication links.

In summary, PG’s logical replication is an all-in-one, do-everything approach — very much in the open-source, technical spirit. MySQL’s approach is simple, crude, but effective — somewhat “one-step-to-finish.” Oracle’s approach is: provide an interface and leave everything else to third parties, but from the customer’s perspective, there is a mature solution available.

Query Conflicts: From a Static Table Conflict to Its Root Cause

Sat, 13 Sep 2025 00:00:00 +0000

Problem Symptoms
#

The Symptom
#

A static historical table with no updates whatsoever — yet queries on the same-city standby consistently hit query conflicts:

ERROR: 40001: canceling statement due to conflict with recovery
DETAIL: User query might have needed to see row versions that must be removed.
LOCATION: ProcessInterrupts, postgres.c:3197
Time: 30534.973 ms (00:30.535)

Why a Query Conflict on a Static Table Matters
#

My understanding was that a static table should never experience conflicts (this understanding was wrong — I’ll explain later).

The official documentation lists the conflict cases:

Access Exclusive locks taken on the primary server, including both explicit LOCK commands and various DDL actions, conflict with table accesses in standby queries.
Dropping a tablespace on the primary conflicts with standby queries using that tablespace for temporary work files.
Dropping a database on the primary conflicts with sessions connected to that database on the standby.
Application of a vacuum cleanup record from WAL conflicts with standby transactions whose snapshots can still “see” any of the rows to be removed.
Application of a vacuum cleanup record from WAL conflicts with queries accessing the target page on the standby, whether or not the data to be removed is visible.

LOCK, DDL, drop tablespace, drop database — definitely none of those.

Vacuum — none either, confirmed by pg_stat_all_tables.last_autovacuum and WAL vacuum records.

The official documentation’s explanation stops there. I carefully verified that none of the above applied.

Extrapolating from existing knowledge, perhaps other scenarios could kill the xmin held by a standby query’s snapshot. For example, in-page pruning removes xmin from rows on a page — if the standby query’s snapshot still depends on those xmins, theoretically a conflict could occur. But a page belongs to a specific table, and querying only one table holds only snapshots and xmins on that table. So, theoretically, in-page pruning on table A should not cause a query conflict on table B (this understanding was also wrong — I’ll explain later).

PG’s official documentation on query conflict scenarios is fairly vague and doesn’t explain well why a static table can experience conflicts. Even combining it with my own extrapolations, there shouldn’t be a conflict. But I noticed this pattern seemed to exist on many instances, so it was worth investigating.

Root Cause Analysis
#

Since the startup process kills the query, checking the startup process’s pstack should reveal the conflict function:

$ pstack 212012
#0 0x00002b283f63d783 in __select_nocancel () from /lib64/libc.so.6
#1 0x00000000008fcf5a in pg_usleep (microsec=<optimized out>) at pgsleep.c:56
#2 0x0000000000787905 in WaitExceedsMaxStandbyDelay (wait_event_info=134217762) at standby.c:208
#3 ResolveRecoveryConflictWithVirtualXIDs (waitlist=0x2398a50, reason=reason@entry=PROCSIG_RECOVERY_CONFLICT_SNAPSHOT, wait_event_info=wait_event_info@entry=134217762, report_waiting=report_waiting@entry=true) at standby.c:276
#4 0x0000000000787b33 in ResolveRecoveryConflictWithVirtualXIDs (report_waiting=true, wait_event_info=134217762, reason=PROCSIG_RECOVERY_CONFLICT_SNAPSHOT, waitlist=<optimized out>) at standby.c:333
#5 ResolveRecoveryConflictWithSnapshot (latestRemovedXid=<optimized out>, node=...) at standby.c:329
#6 0x00000000004c8ffe in heap_xlog_clean (record=0x2366978) at heapam.c:7764
#7 heap2_redo (record=0x2366978) at heapam.c:8917
#8 0x0000000000519e55 in StartupXLOG () at xlog.c:7411
#9 0x000000000072f211 in StartupProcessMain () at startup.c:204
#10 0x00000000005286b1 in AuxiliaryProcessMain (argc=argc@entry=2, argv=argv@entry=0x7ffeb7e39d70) at bootstrap.c:450
#11 0x000000000072c369 in StartChildProcess (type=StartupProcess) at postmaster.c:5494
#12 0x000000000072eb54 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x232edb0) at postmaster.c:1407
#13 0x00000000004892cf in main (argc=3, argv=0x232edb0) at main.c:210

XLOG_HEAP2_CLEAN
#

void
heap2_redo(XLogReaderState *record)
{
	uint8		info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;

	switch (info & XLOG_HEAP_OPMASK)
	{
		case XLOG_HEAP2_CLEAN:
			heap_xlog_clean(record);
			break;

Only when the redo is XLOG_HEAP2_CLEAN does it enter the next function heap_xlog_clean.

PG 18 no longer has XLOG_HEAP2_CLEAN (it was actually removed around PG15 — this article only looks at versions 13 and 18), but the define can still be found in heapam_xlog.h:

//pg13
#define XLOG_HEAP2_CLEAN		0x10
#define XLOG_HEAP2_FREEZE_PAGE	0x20
#define XLOG_HEAP2_CLEANUP_INFO 0x30

//pg18
 * There's no difference between XLOG_HEAP2_PRUNE_ON_ACCESS,
 * XLOG_HEAP2_PRUNE_VACUUM_SCAN and XLOG_HEAP2_PRUNE_VACUUM_CLEANUP records.
 * They have separate opcodes just for debugging and analysis purposes, to
 * indicate why the WAL record was emitted.
 */
#define XLOG_HEAP2_PRUNE_ON_ACCESS		0x10
#define XLOG_HEAP2_PRUNE_VACUUM_SCAN	0x20
#define XLOG_HEAP2_PRUNE_VACUUM_CLEANUP	0x30

I pulled out PG18’s source because PG13 (our production version) has zero explanation for these CLEAN xl_info macros, making them hard to understand. Since PG18 renamed the macros to something more intuitive and added comments, we can use PG18’s source to understand PG13’s — to figure out what this WAL record does.

All three opcodes are fundamentally PRUNE-related WAL records. From the names, PRUNE_ON_ACCESS looks like pruning triggered by access, while the other two are tied to VACUUM operations.

When checking with pg_waldump, rmgr: Heap2 CLEAN remxid records appear every few seconds, with highly varied filenodes and no relation to the static table:

$ pg_waldump 00000001000012FE00000001 |tail -200|egrep -i heap2
pg_waldump: fatal: error in WAL record at 12FE/F34F138: invalid resource manager ID 50 at 12FE/F34F168
rmgr: Heap2 len (rec/tot): 61/ 3520, tx: 0, lsn: 12FE/0F346ED0, prev 12FE/0F346EA0, desc: CLEAN remxid 1983744188, blkref #0: rel 1663/88121/1083807 blk 617606 FPW
rmgr: Heap2 len (rec/tot): 66/ 66, tx: 0, lsn: 12FE/0F34BC60, prev 12FE/0F34BC30, desc: CLEAN remxid 1984090598, blkref #0: rel 1663/88121/504681 blk 1447147

This matches our symptom pattern: no vacuum activity, but PRUNE is happening, leading into heap_xlog_clean → ResolveRecoveryConflictWithSnapshot and the rest of the conflict machinery.

The PRUNE action producing rmgr: Heap2 CLEAN remxid WAL records will be demonstrated later via testing.

Let’s finish the source code analysis first.

ResolveRecoveryConflictWithSnapshot
#

void
ResolveRecoveryConflictWithSnapshot(TransactionId latestRemovedXid, RelFileNode node)
{
	VirtualTransactionId *backends;

	/*
	 * If we get passed InvalidTransactionId then we do nothing (no conflict).
	 *
	 * This can happen when replaying already-applied WAL records after a
	 * standby crash or restart, or when replaying an XLOG_HEAP2_VISIBLE
	 * record that marks as frozen a page which was already all-visible. It's
	 * also quite common with records generated during index deletion
	 * (original execution of the deletion can reason that a recovery conflict
	 * which is sufficient for the deletion operation must take place before
	 * replay of the deletion record itself).
	 */
	if (!TransactionIdIsValid(latestRemovedXid))
		return;

	backends = GetConflictingVirtualXIDs(latestRemovedXid,
										 node.dbNode);

	ResolveRecoveryConflictWithVirtualXIDs(backends,
										 PROCSIG_RECOVERY_CONFLICT_SNAPSHOT,
										 WAIT_EVENT_RECOVERY_CONFLICT_SNAPSHOT,
										 true);
}

There are several types of query conflicts. ResolveRecoveryConflictWithSnapshot lives up to its name — it’s a snapshot conflict.

GetConflictingVirtualXIDs finds which backends conflict with the snapshot. ResolveRecoveryConflictWithVirtualXIDs handles the actual conflict resolution and timeout.

GetConflictingVirtualXIDs
#

GetConflictingVirtualXIDs is the key function that determines whether a backend’s virtual transaction ID triggers a query conflict. It requires a bit of brainpower.

Prerequisite knowledge for understanding this function:

limitXmin is latestRemovedXid — the CLEAN remxid from WAL, the xid that needs to be cleaned up (I read remxid as “remove xid”). /*limitXmin is supplied as either latestRemovedXid, or InvalidTransactionId*/
PGPROC contains current process info: backend id, database id, lock info, and much more
PGXACT contains the transaction info for the snapshot held by the current process. It’s lighter — the key field is xmin, the lowest xid the current process considers still running
C’s || rule: if either operand is true (non-zero), the result is true (1)
TransactionIdIsValid means xid != 0 — 0 is meaningless

Key function GetConflictingVirtualXIDs explained:

VirtualTransactionId *
GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
	...
	for (index = 0; index < arrayP->numProcs; index++) // iterate all local processes
	{
		int			pgprocno = arrayP->pgprocnos[index];
		PGPROC	 *proc = &allProcs[pgprocno]; // process's PGPROC
		PGXACT	 *pgxact = &allPgXact[pgprocno]; // process's PGXACT

		/* Exclude prepared transactions */
		if (proc->pid == 0) // prepared transactions have no owning process — can't handle
			continue;

		if (!OidIsValid(dbOid) || // global tables have dbOid=0 which is invalid — satisfies condition
			proc->databaseId == dbOid) // only process current database. Cross-db is different — no transaction conflict at all.
		{
			/* Fetch xmin just once - can't change on us, but good coding */
			TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin); // pgxact->xmin is the minimum xid of transactions held by this process. UINT32_ACCESS_ONCE is just for atomic access protection — the xmin logic is unchanged

			/*
			 * We ignore an invalid pxmin because this means that backend has
			 * no snapshot currently. We hold a Share lock to avoid contention
			 * with users taking snapshots. That is not a problem because the
			 * current xmin is always at least one higher than the latest
			 * removed xid, so any new snapshot would never conflict with the
			 * test here.
			 */
			if (!TransactionIdIsValid(limitXmin) || // limitXmin=0 possible? At least latestRemovedXid can't be — I can't think of a scenario where WAL would log an invalid xid
				(TransactionIdIsValid(pxmin) && !TransactionIdFollows(pxmin, limitXmin))) // TransactionIdIsValid(pxmin) is also not really needed. !TransactionIdFollows(pxmin, limitXmin) means pxmin <= limitXmin
			{
				VirtualTransactionId vxid;

				GET_VXID_FROM_PGPROC(vxid, *proc);
				if (VirtualTransactionIdIsValid(vxid))
					vxids[count++] = vxid;
			}
		}
	}

The critical line is !TransactionIdFollows(pxmin, limitXmin).

So the core logic for determining query conflicts is:

The primary’s cleaned remxid >= the standby query’s snapshot-held minimum xid → conflict.
Only kills queries in the current database; global system tables (no database) are killed indiscriminately.

This means: even if the pruned table on the primary has nothing to do with the table being queried on the standby, a conflict CAN occur!!!

In-Page Pruning
#

Now that the conflict logic is clear, we still need to understand where the WAL CLEAN records come from. That requires looking at how PRUNE is triggered.

From README.HOT on when pruning and defragmentation occur — “When can/should we prune or defragment?”:

The currently planned heuristic is to prune and defrag when first accessing a page that potentially has prunable tuples

Prune and defragment are indeed two distinct concepts, but they often happen together.

Prune: updating line pointers to shorten HOT chains, but doesn’t free space
Defragment: reclaiming space from dead line pointers and tuples after pruning

We cannot prune or defragment unless we can get a “buffer cleanup lock” on the target page; otherwise, pruning might destroy line pointers that other backends have live references to, and defragmenting might move tuples that other backends have live pointers to

The page must be under a “buffer cleanup lock” for prune or defragment to occur.

The worst-case consequence of this is only that an UPDATE cannot be made HOT but has to link to a new tuple version placed on some other page, for lack of centralized space on the original page.

A typical scenario: a HOT update spills to another page (easy to test).

space reclamation happens during tuple retrieval when the page is nearly full (<10% free) and a buffer cleanup lock can be acquired. This means that UPDATE, DELETE, and SELECT can trigger space reclamation, but often not during INSERT … VALUES because it does not retrieve a row.

SELECT/UPDATE/DELETE that scan rows can trigger space reclamation. INSERT typically won’t, since it doesn’t retrieve rows.

Clearly, after prune or defragment, the corresponding xids should be reclaimed. From the README we can see that HOT updates can reproduce prune/defragment, generating CLEAN WAL records. See [Test: Pure UPDATE Produces In-Page Pruning](## Test: Pure UPDATE Produces In-Page Pruning).

Testing
#

The tests below only observe whether conflicts occur, whether CLEAN WAL records appear, or whether page line pointers are updated — without distinguishing prune vs. defragment. In many cases both are triggered together; distinguishing them is tedious and maybe best left for later. The focus here is whether CLEAN WAL records appear.

Helper SQL:

--sql for test
--heap_page_items
select t_ctid,lp,
case lp_flags
when 0 then '0:LP_UNUSED'
when 1 then 'LP_NORMAL'
when 2 then 'LP_REDIRECT'
when 3 then 'LP_DEAD'
end as lp_flags,
t_xmin,t_xmax,t_field3 as t_cid, raw_flags, info.combined_flags,substring(t_data,0,40)
from heap_page_items(get_raw_page('lzl',0)) item,
LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2) info
order by lp;

--heap header
select * from page_header(get_raw_page('lzl',0));

--bt_page_items
SELECT itemoffset, ctid, itemlen, nulls, vars, dead, htid, tids[0:2] AS some_tids FROM bt_page_items('idxlzl',1);

--create table
create table lzl(a char(2000));
create index idxlzl on lzl(a);
insert into lzl values('z');
update lzl set a=md5(random()::text); -- non-hot
update lzl set a='z'; -- hot

--force index scan
set enable_seqscan =off;
set enable_indexonlyscan=off;

--open an RR transaction to hold a snapshot for observation
BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;

Test: Cross-Table Query Conflict
#

primary	standby
create table lzl(a bigint primary key);
insert into lzl values(1);
	BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
	select 1;
update lzl set a=2;
	no blocking
vacuum lzl;
	#3 ResolveRecoveryConflictWithVirtualXIDs (waitlist=0x277c340, reason=reason@entry=PROCSIG_RECOVERY_CONFLICT_SNAPSHOT, wait_event_info=wait_event_info@entry=134217762, report_waiting=report_waiting@entry=true) at standby.c:276 #4 0x0000000000787b33 in ResolveRecoveryConflictWithVirtualXIDs (report_waiting=true, wait_event_info=134217762, reason=PROCSIG_RECOVERY_CONFLICT_SNAPSHOT, waitlist=) at standby.c:333 #5 ResolveRecoveryConflictWithSnapshot (latestRemovedXid=, node=…) at standby.c:329 #6 0x00000000004c8ffe in heap_xlog_clean (record=0x273a258) at heapam.c:7764

Conclusion: As long as a query exists, it has a snapshot, and a snapshot has a snapshot xmin. Even if the queried table is completely unrelated, a query conflict CAN occur.

Test: Vacuum Produces In-Page Pruning
#

Pruning occurs, conflicts occur. Example omitted — not relevant to this case.

Test: UPDATE Produces In-Page Pruning
#

--HOT, off-page update triggers defragment
--An 8k heap page stores 4-2xx rows. Here we size rows so 4 fit and remain HOT — the next update spills off-page
create table lzl(a char(2000));
create table idxlzl on lzl(a);
insert into lzl values('z');
update lzl set a='z'; --hot
update lzl set a='z'; --hot
update lzl set a='z'; --hot

--heap page: 4 rows, all HOT:
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags |
--------+----+-----------+----------+----------+-------+----------------------------------------------------------------------------------------------------------+----------------+----------
 (0,2) | 1 | LP_NORMAL | 34954161 | 34954162 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_HOT_UPDATED} | {} | \x501f000
 (0,3) | 2 | LP_NORMAL | 34954162 | 34954163 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f000
 (0,4) | 3 | LP_NORMAL | 34954163 | 34954164 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f000
 (0,4) | 4 | LP_NORMAL | 34954164 | 0 | 0 | {HEAP_HASVARWIDTH,HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f000
(4 rows)

--index: only one entry:
 itemoffset | ctid | itemlen | nulls | vars | dead | htid | some_tids
------------+-------+---------+-------+------+------+-------+-----------
 1 | (0,1) | 48 | f | t | f | (0,1) | [null]

--One more update triggers off-page update
update lzl set a='z'; --page full, can't HOT

--HOT chain changed. LP changed
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags |
--------+----+-------------+----------+----------+--------+--------------------------------------------------------------------------------------+----------------+---------------------------
 [null] | 1 | LP_REDIRECT | [null] | [null] | [null] | [null] | [null] | [null]
 (0,2) | 2 | LP_NORMAL | 34954165 | 0 | 0 | {HEAP_HASVARWIDTH,HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020202020
 [null] | 3 | 0:LP_UNUSED | [null] | [null] | [null] | [null] | [null] | [null]
 (0,2) | 4 | LP_NORMAL | 34954164 | 34954165 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020202020
(4 rows)

--index: still only one entry, unchanged:
 itemoffset | ctid | itemlen | nulls | vars | dead | htid | some_tids
------------+-------+---------+-------+------+------+-------+-----------
 1 | (0,1) | 48 | f | t | f | (0,1) | [null]

The next update doesn’t go to a new page — instead, in-page pruning happens first, freeing space on the same page, so the row is written locally. This saves a page access.

WAL produces CLEAN remxid, confirming that a query conflict can occur:

rmgr: Heap2 len (rec/tot): 62/ 62, tx: 0, lsn: 3DB/F8017348, prev 3DB/F8017310, desc: CLEAN remxid 34954177, blkref #0: rel 1663/5893914/5893920 blk 0
rmgr: Heap len (rec/tot): 2070/ 2070, tx: 34954178, lsn: 3DB/F8017388, prev 3DB/F8017348, desc: HOT_UPDATE off 4 xmax 34954178 flags 0x10 ; new off 2 xmax 0, blkref #0: rel 1663/5893914/5893920 blk 0

Conclusion: UPDATE statements can produce in-page pruning and can cause query conflicts.

Test: Hint-Bit Writeback Producing In-Page Pruning?
#

primary	standby
wal_log_hints=on
truncate table lzl;
insert into lzl values(‘z’);
	BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
	select * from lzl;
delete from lzl where a=‘z’;
checkpoint;
select * from lzl;
–WAL contains FPI_FOR_HINT
	–no query conflict

Standby pageinspect:

 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags | substring
--------+----+-----------+----------+----------+-------+------------------------------------------------------------------------------+----------------+-------------------------------------------------
 (0,1) | 1 | LP_NORMAL | 34954229 | 34954230 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_KEYS_UPDATED} | {} | \x501f00007a202020202020202020202020202020202020
(1 row)

Conclusion: WAL log hints only sync hint bits and don’t affect xmin/xmax. No CLEAN or similar records are produced, so hint-bit writeback does NOT cause query conflicts.

Test: SELECT Produces In-Page Pruning
#

SELECT normally doesn’t cause pruning, but it does when the page is nearly full: https://www.modb.pro/db/1683648157451362304

Testing pruning on a full page:

-- Same table as before, 4 HOT rows, nearly full
insert into lzl values('z');
update lzl set a='z';
update lzl set a='z';
update lzl set a='z';

--page at this point:
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags |
--------+----+-----------+----------+----------+-------+----------------------------------------------------------------------------------------------------------+----------------+---------------------
 (0,2) | 1 | LP_NORMAL | 34954232 | 34954233 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_HOT_UPDATED} | {} | \x501f00007a20202020
 (0,3) | 2 | LP_NORMAL | 34954233 | 34954234 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020
 (0,4) | 3 | LP_NORMAL | 34954234 | 34954235 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020
 (0,4) | 4 | LP_NORMAL | 34954235 | 0 | 0 | {HEAP_HASVARWIDTH,HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020
(4 rows)

-- A SELECT
select * from lzl;

--page now shows in-page pruning:
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags | sub
--------+----+-------------+----------+--------+--------+---------------------------------------------------------------------------------------+----------------+---------------------------------------
 [null] | 1 | LP_REDIRECT | [null] | [null] | [null] | [null] | [null] | [null]
 [null] | 2 | 0:LP_UNUSED | [null] | [null] | [null] | [null] | [null] | [null]
 [null] | 3 | 0:LP_UNUSED | [null] | [null] | [null] | [null] | [null] | [null]
 (0,4) | 4 | LP_NORMAL | 34954235 | 0 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020202020202020202020

Conclusion: SELECT can produce in-page pruning and can cause query conflicts.

Test: Shared Table Cross-Database Query Conflict
#

Shared tables are global. Earlier in GetConflictingVirtualXIDs we saw that global tables are killed indiscriminately. Let’s test.

Shared table info:

Source definition: IsSharedRelation
Source check: shared ? InvalidOid : MyDatabaseId;
Table: pg_class.relisshared
Directory: global/

Querying pg_class.relisshared directly is easier:

select relname,relkind,relisshared from pg_class where relisshared is true and relkind='r';
 relname | relkind | relisshared
-----------------------+---------+-------------
 pg_authid | r | t
 pg_subscription | r | t
 pg_database | r | t
 pg_db_role_setting | r | t
 pg_tablespace | r | t
 pg_auth_members | r | t
 pg_shdepend | r | t
 pg_shdescription | r | t
 pg_replication_origin | r | t
 pg_shseclabel | r | t

pg_authid stores role/user info. Testing with a password change:

--Test: on the primary, in a non-business database
create user lzl;
alter user lzl with password '1'; --run several times

CLEAN remxid appears:

rmgr: Heap len (rec/tot): 76/ 76, tx: 34954264, lsn: 3DB/F808D0F8, prev 3DB/F808D0B8, desc: HOT_UPDATE off 67 xmax 34954264 flags 0x20 ; new off 66 xmax 0, blkref #0: rel 1664/0/1260 blk 0
rmgr: Transaction len (rec/tot): 82/ 82, tx: 34954264, lsn: 3DB/F808D148, prev 3DB/F808D0F8, desc: COMMIT 2025-09-12 14:40:56.680782 CST; inval msgs: catcache 11 catcache 10
rmgr: Heap2 len (rec/tot): 60/ 60, tx: 0, lsn: 3DB/F808D1A0, prev 3DB/F808D148, desc: CLEAN remxid 34954264, blkref #0: rel 1664/0/1260 blk 0
rmgr: Heap2 len (rec/tot): 60/ 60, tx: 34954265, lsn: 3DB/F808D1E0,

The standby business database’s select 1 query was killed.

Conclusion: Shared tables can cause cross-database query conflicts.

That said, these shared system tables rarely see heavy updates in normal operations.

Conclusions
#

Developer Perspective
#

Query conflicts can be completely unrelated to the table being queried — meaning a fully static table CAN experience conflicts.

Cross-database means different business domains and data. Cross-database does NOT cause query conflicts. The one exception is shared tables, but these are just a handful of system tables that rarely see updates.

For developers, focus on:

Retry on failure: Standby queries can be killed — retrying is essential, and retries may succeed
Query duration: Longer queries are more likely to be killed
Alternative standbys: Consider using a different standby with lower disaster-recovery requirements

Operations Perspective
#

Since query conflicts can come from “all directions,” a simple long-running single-table query can be killed by in-page pruning on a completely different, frequently-updated table. You can increase max_standby_streaming_delay to reduce conflict probability.

However, max_standby_streaming_delay trades off against WAL apply — a longer delay means WAL application is paused. This parameter’s value directly represents the maximum possible standby replication lag (it can’t cap lag from network or other factors).

Query freshness: Prolonged WAL apply pauses mean the standby data lags significantly (WAL may already be on the standby’s disk), affecting data freshness requirements for other standby queries.
RTO: If the primary suffers a disaster and failover is needed, the standby must apply accumulated WAL. If apply delay stretches to hours, it may violate minute-level RTO SLAs.

So tuning max_standby_streaming_delay is a delicate exercise requiring consideration of the standby’s role, query freshness requirements, and even geography.

PostgreSQL DDL Pitfalls and Clever Solutions

Sat, 19 Jul 2025 00:00:00 +0000

Save it, use it freely, no need to ask.

May be updated, may not be.

Feedback welcome — pick it apart if you can.

This article was originally published in Chinese on lastdba.com.

Linux Memory Advanced

Thu, 19 Jun 2025 00:00:00 +0000

(For memory basics, refer to Linux Memory Analysis; this article covers memory knowledge above that foundation)

Memory Basic Concepts
#

buddy
#

The process of buddy system allocating and merging pages is omitted.

Easily overlooked knowledge points:

The prerequisite for buddy merging two blocks of the same size is that their physical addresses are contiguous
The merge algorithm is iterative: after merging at the current level, it will automatically attempt to merge larger blocks. This means compactd is not strictly required for merging

page table & PTE
#

page table and PTE are actually two different concepts, and they are easily confused because both generally refer to page tables. Below is relevant knowledge about page table and PTE[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]

PTE stores the physical address of the page frame
“page table” and “Page Table” are different concepts: “page table” refers to the pages that maintain the mapping between linear addresses and physical addresses, while “Page Table” refers to pages in the upper-level page table
pte_t, pmd_t, pud_t, pgd_t describe Page Table Entry, Page Middle Directory entry, Page Upper Directory entry, and Page Global Directory entry respectively
PTE is Page Table Entry

If you only look at the size of the pagetable used by the MMU to cache virtual-to-physical memory mappings, confusing pagetable with PTE doesn’t make much difference. However, if you need to go deep into page table directories, you need to separate the two concepts.

TLB
#

Each level of the page table is stored in memory. To complete a single virtual-to-physical address translation, all four page tables corresponding to the current virtual address must be found. This means a single memory IO requires looking up the page table in memory 4 times just for virtual-to-physical address translation. Translation Lookaside Buffers (TLB) are caches specifically designed to accelerate virtual-to-physical address translation.

Regarding the TLB’s location, it is usually in the L1 cache (some say it’s in registers or L2, which likely depends on the CPU architecture; for now, just consider it as CPU cache, distinct from main memory)¹:

In modern processors, the L1 cache is typically divided into multiple parts, including data cache dTLB and instruction cache iTLB. Frequently modifying page tables leads to increased main memory accesses, causing the CPU to frequently flush the TLB cache[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]. The TLB also has a finite size; improving TLB hit rate can reduce accesses to the main memory pagetable. Using huge pages can reduce PTEs by three orders of magnitude, greatly reducing TLB misses.[^ 《深入理解Linux进程和内存》 (Understanding Linux Processes and Memory)].

TLB information:

#cpuid -l
 L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax):
 L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx):
...
 L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax):

Observing TLB hit rate:

perf stat -e dTLB-loads,dTLB-load-misses,iTLB-loads,iTLB-load-misses -I 10000 -p $PM_PID 

During memory reclamation, TLB misses do increase, but it’s hard to establish a causal relationship. TLB miss is just one observation metric for the MMU — TLB is part of MMU.

Reverse Mapping
#

The general principles of PFRA (Page Frame Reclaiming Algorithm)[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]:

First, release “harmless” pages. Start by reclaiming harmless pages in the pagecache — pages not occupied by any process
All pages of user-mode processes are candidates for reclamation. FRPA will gradually deprive user-mode pages with longer sleep times of their page frames
Cancel the mapping of all page table entries for a shared page frame, then reclaim that shared page frame
Only reclaim “unused” pages

One of PFRA’s goals is to be able to release shared page frames. The process of quickly locating all page table entries pointing to the same page frame is called reverse mapping.

Reverse mappings for shared

Anonymous pages
File-mapping pages

Basic tricks of page frame reclaiming

LRU lists
Free cheapest pages first
Unmap all at once
Etc²

Huge Pages
#

Enabling huge pages provides certain performance improvements for specific application workloads. In PostgreSQL, enabling huge pages on large-memory instances also offers some performance gains and even some stability benefits.

Why are huge pages better?³:

Reduced TLB pressure
Reduced pagetable size in main memory
Huge pages are physically contiguous. Contiguous physical memory access is better than non-contiguous physical memory access
When using these kinds of larger pages, higher level pages can directly map them, with no need to use lower level page entries[^ kernel.org,mm pagetables]

However, using huge pages brings management challenges:

Huge pages need to be pre-allocated
Huge page size must be calculated in advance to avoid memory waste

Two ways for processes to use huge pages:

The first is by using shmget() to setup a shared region backed by huge pages
the second is the call mmap() on a file opened in the huge page filesystem

C Library and System Calls
#

The middle layer between kernel space and user space is the system call layer. Application Programming Interfaces (APIs) and system calls are different. Applications call APIs implemented in user space to program, rather than directly executing system calls. In the UNIX world, the most common system call layer is the POSIX standard (Portable Operation System Interface of UNIX). The POSIX standard targets APIs, not system calls. The Linux operating system’s API is typically provided in the form of C standard libraries, such as libc. The C standard library provides implementations for most POSIX APIs.[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)]

C app->C lib->system calls->OS->hardware⁴:

Common C library and system calls:

malloc,free=>C lib

mmap、brk、munmap=>system calls

Page Fault Exception
#

Page fault exceptions (or page fault interrupts) need to distinguish two cases: exceptions caused by programming errors; and physical page allocation behavior triggered by using virtual address space where physical page frames haven’t been allocated yet.[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]

Exceptional page fault: Segment Fault — each virtual memory area has associated permissions. If a process accesses a memory area outside its valid range, or illegally accesses a memory area, or accesses a memory area in an incorrect manner, the processor reports a page fault exception. In severe cases, it reports a “Segment Fault” and terminates the process[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)].
Normal page fault: System calls like mmap and brk manage virtual memory; they don’t directly allocate physical memory. Virtual memory system call functions only establish the process address space. Virtual memory is visible in user space, but no mapping between virtual memory and physical memory has been established. When a process accesses virtual memory where no mapping has been established, a page fault interrupt is triggered.[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)]

Page faults are also divided into two types:

minor fault: the page fault was handled without blocking the current process, and a page frame was allocated
major fault: the page fault forced the current process to sleep (likely because filling the page frame with data from disk took time). A page fault that blocks the current process is a major fault[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]

Copy-On-Write (COW)
#

When the fork system call is executed, the child process and parent process have independent process address spaces but share physical memory resources, including process context, process stack, memory information, file descriptors, directories, resource limits, etc. Only the parent process’s page table needs to be copied to the child process. At this point, sharing is read-only. When writing is needed (when running their respective tasks), data is copied, giving the parent and child processes their own copies.[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)]

For PostgreSQL’s multi-process model, fork itself isn’t heavy — you may only need to worry about page tables — but the various tasks that come after fork will trigger copy-on-write to create the child process’s own resource copies.

Note the distinction between copy-on-write and page fault exceptions: copy-on-write refers to resources not being allocated to the child process at fork time; page fault exceptions refer to physical memory allocation occurring for this process, unrelated to fork.

mmap, brk & Shared Memory Mapping Area, Heap Area
#

The functions and memory address regions used by mmap and brk are different:

mmap is used to manage shared memory, corresponding to the shared memory mapping area
brk is used to manage private memory, corresponding to the heap area

Linear address region functions:

mmap: The mapping area expands top-down. The mmap mapping area and heap expand toward each other until they exhaust the remaining space in the virtual address space. This structure facilitates the C runtime library’s use of the mmap mapping area and heap for memory allocation.
Stack: Stores local variables and function parameters during program execution, grows from high addresses to low addresses
Heap: Dynamic memory allocation area, managed through functions like malloc, new, free, and delete
BSS (Uninitialized Variables): Stores uninitialized global variables and static variables
Data: Stores global variables and static variables with predefined values in source code
Text (Code): Stores read-only program execution code, i.e., machine instructions.

Shared memory mapping area and heap area⁵:

Real postmaster heap and shared memory mapping:

cat /proc/1063005/smaps |grep -E "\-s|heap"
022a4000-022ee000 rw-p 00000000 00:00 0 [heap]
7fef6019e000-7fef601a5000 rw-s 00000000 00:17 21 /dev/shm/PostgreSQL.1291978332
7fef601a5000-7fef6098b000 rw-s 00000000 00:01 1052 /dev/zero (deleted) #this is shared buffers
7fef6e238000-7fef6e239000 rw-s 00000000 00:01 10 /SYSV0011f702 (deleted)

You can see the heap and shared memory area addresses roughly match.

VM
#

Linux kernel virtual memory subsystem

Directory: cd /proc/sys/vm/

compact
#

concept & param
#

Memory compaction is a mechanism in the Linux kernel for solving memory fragmentation problems. It improves the allocation and compaction efficiency of large contiguous memory pages by merging free physical pages.

Parameter	Function	Default/Range
`compact_memory`	Manually trigger a global memory compaction operation	Write 1 to trigger
`compaction_proactiveness`	Controls the frequency of proactive compaction	Parameter available since 4.x. 0-100 (default 20)
`compact_unevictable_allowed`	Whether to allow compaction of unreclaimable pages (e.g., `mlock` locked memory)	Parameter available since 4.x. 0 (disable) or 1 (allow)
`defrag_mode`	Controls the trigger strategy for memory defragmentation	Parameter available since 4.x. 0-3. 0 disables automatic compaction; 1 defers passive compaction. Default in 3.10 is 1
`extfrag_threshold`	Threshold for triggering compaction when large memory blocks are insufficient	0-1000 (default 500)

There are 3 compaction modes (depending on kernel version support):

Passive compaction: extfrag_threshold addresses “already occurred” fragmentation problems — triggered when a process requests large memory blocks and finds them insufficient.
Proactive compaction: compaction_proactiveness proactively controls compaction aggressiveness, optimizing “not yet occurred” but potential fragmentation risks.
Manual compaction: compact_memory.

extfrag_threshold is the Linux kernel parameter controlling passive compaction. When the kernel fails to allocate high-order contiguous physical memory (e.g., huge pages), it determines the failure cause via the fragmentation index:

-1: Allocation succeeded (watermark satisfied)
0: Failed due to insufficient memory
1000: Failed due to fragmentation

View specific values via /sys/kernel/debug/extfrag/extfrag_index. The output is a floating-point number (e.g., 0.854), but the actual range is magnified 1000x, so 0.854 corresponds to an actual value of 854:

cat /sys/kernel/debug/extfrag/extfrag_index |grep Normal
Node 0, zone Normal -1.000 -1.000 -1.000 -1.000 -1.000 -1.000 -1.000 -1.000 -1.000 0.995 0.998 

If extfrag_threshold=600, compaction is triggered when the fragmentation index > 600. extfrag_index is quite useful and can assist buddy in observing fragmentation issues.

dirty
#

concept & param
#

Dirty page flushing is somewhat similar to memory reclamation and is also divided into asynchronous and synchronous:

Asynchronous flushing: performed by background threads like pdflush/flush/kdmflush; application writes are not affected
Synchronous flushing: directly blocks the application process; the process that initiated the write operation flushes the dirty pages itself

Parameter Name	Description	Default
dirty_background_bytes	Background async flush threshold, in bytes	0 (disabled)
dirty_background_ratio	Background async flush threshold, as percentage	10%
dirty_bytes	Synchronous flush threshold, in bytes	0 (disabled)
dirty_ratio	Synchronous flush threshold, as percentage	20-40%
dirty_expire_centisecs	Maximum lifetime of dirty pages in memory	3000 (30s)
dirty_writeback_centisecs	Frequency of kernel periodic dirty page state checks	500 (5s)

xxx_bytes and xxx_ratio parameters are mutually exclusive.

Example parameters and flowchart:

dirty_background_bytes 0
dirty_background_ratio 10
dirty_bytes 0
dirty_ratio 40
dirty_expire_centisecs 3000
dirty_writeback_centisecs 500

%% Dirty page flushing flow diagram integrating time parameters
graph TD
 A[App writes generate dirty pages] --> B{Check interval reached?<br>dirty_writeback_centisecs every 5s}
 B -- No --> D{Expired dirty pages exist?<br> dirty_expire_centisecs>30s}
 B -- Yes --> C{Dirty page threshold check}
 C --> E[Dirty page ratio? dirty_background_ratio>10% ]
 C --> F[Dirty page ratio? dirty_ratio> 40%]
 E -- Trigger --> G[Background async flush]
 F -- Trigger --> H[Synchronous flush]
 D -- Trigger --> G
 G --> I[Dirty pages written to disk]
 H --> I[Dirty pages written to disk] 
 I --> J[Free memory space]

The configuration principles for dirty page flush parameters are basically the same as PostgreSQL dirty page flush parameters. Setting them too low causes overly frequent flushing — the same dirty page may be written to disk multiple times, wasting IO. Setting them too high may cause IO storms.

Observing Dirty Pages
#

Monitoring dirty pages:

ps -eo pid,%cpu,%mem,wchan,args,state|grep kdmflush|grep -E -w -v "S" #Observe async flush process state
cat /proc/vmstat| grep -E -w "nr_dirty|nr_writeback" #vmstat dirty, page count
cat /proc/meminfo |grep -i dirty #meminfo dirty, KB

Testing dirty pages with dd:

 grep -E "nr_dirty_threshold|nr_dirty_background_threshold" /proc/vmstat | awk '{printf "%s: %.2fGB\n", $1, ($2*4)/1048576}'
nr_dirty_threshold: 141.28GB
nr_dirty_background_threshold: 35.32GB

dd if=/dev/zero of=testfile bs=8k count=128000 # cache io

Failed test (same result after multiple tests):

No RUNNING kdmflush process observed
Dirty pages were flushed before reaching 35GB or 30S threshold

Timestamp	nr_dirty	nr_dirty(GB)	Trend Simulation
17:00:18	2,757	0.01052	▍
17:00:19	336,199	1.282	████▌
17:00:25	1,984,867	7.574	██████████████▍
17:00:32	4,252,177	16.22	████████████████████
17:00:33	3,699,227	14.11	█████████████████▊
17:00:38	170,865	0.652	▎
17:00:46	2,865,814	10.93	█████████▋
17:00:54	4,721,827	18.01	██████████████████████
17:00:55	3,876,509	14.79	██████████████████
17:01:03	835,097	3.186	██▊

os dirty != pg dirty
#

With pg fsync=on, data writes go through the OS pagecache before specific blocks are written to disk. PostgreSQL has its own dirty pages, and the OS also has dirty pages. What’s the relationship between the two?

## Observation commands
cat /proc/meminfo |grep -E -w "Dirty" # OS dirty pages

select isdirty,pinning_backends,count(*) from pg_buffercache where isdirty is true group by isdirty,pinning_backends; # PG dirty pages

checkpoint;
begin;
--Observe
insert into tlzl select generate_series(1,1000000);
--Observe
commit;
--Observe
checkpoint;
--Observe

Test results:

stage	dirty in pg	OS dirty
Clean state	0	0.02-2M fluctuating
After insert completion	200M	Rose to 1.7G, then dropped to 20KB
After commit	200M	0.02-2M fluctuating
After checkpoint flush	0	0.02-2M fluctuating

When the insert data size is increased, OS dirty rises during insert, rising to the GB level and then fluctuating.

PG dirty has some relation to OS dirty but they’re not entirely correlated. When PG inserts data, OS dirty does rise, but after the OS flushes its own dirty pages, PG’s dirty pages remain dirty. Preliminary judgment: dirty pages in shared memory are unrelated to OS dirty. It’s yet to be determined whether the OS dirty increase comes from PG’s private memory dirty pages.

swappiness
#

Controls the kernel’s bias toward reclaiming memory from the anonymous memory pool or the page cache. Essentially, it controls whether swapping anonymous pages or reclaiming file pages imposes a lower cost for the upper-layer application. For example, for compute-oriented applications using more dynamic allocation or private memory, a lower swappiness should be set; for data-dependent applications, a higher swappiness should be set to reduce the impact of flushing file pages on data access. However, all of this depends on the efficiency of swap IO and filesystem IO⁶. It all sounds ideal, but when swapping occurs, it very likely means performance degradation.

swappiness=0
#

When swappiness=0, the kernel will only swap when memory reaches the high watermark⁷. The specific strategy also relates to the kernel version and NUMA. What can be confirmed is that swappiness=0 does not mean swap is disabled — swapoff -a is what disables the swap functionality.

#Check if swap is enabled
swapon --show
free -h |grep Swap
cat /proc/swaps
grep -E 'swap|none' /etc/fstab
cat /proc/meminfo|grep Swap

#Monitor whether swapping is occurring
cat /proc/vmstat|grep swp
sar -W 1

inconsistent swap behavior
#

The OS-level /proc/sys/vm/swappiness has little-to-no effect on the swap behavior of cgroups v1 systems (has little-to-no effect on the swap). This issue can lead to inconsistent swap behavior⁸.

Occurrence conditions (all must be true):

vm.swappiness != cgroups memory.swappiness
cgroups v1

Cause:

systemd creates cgroups early during startup, before sysctl.service loads /etc/sysctl.conf. vm.swappiness cannot constrain cgroup memory.swappiness. The issue is: when the OS swap behavior and cgroup behavior differ, which one takes effect?

Solutions:

for cgroup v1, set vm.swappiness = all cgroups memory.swappiness
for cgroup v1, many solutions available, see https://access.redhat.com/solutions/6785021
Use cgroup v2. v2 adds the vm.force_cgroup_v2_swappiness parameter, which disables cgroup’s memory.swappiness

memory overcommitment
#

concept & param
#

Linux does not reserve physical memory for every virtual address; instead, it allocates memory only when actually needed. Overcommitment can limit the total virtual memory size that all processes can request. When the requested memory exceeds the defined physical memory size, it’s called overcommit.

There are three overcommit policy parameters: overcommit_memory, overcommit_ratio/overcommit_kbytes

The overcommit_memory parameter controls the overcommitment policy:

0 (default): Heuristic overcommitment policy, allows slight overcommit. CommitLimit = physical memory + swap.
1: No overcommit check
2: Strict limit, prohibits exceeding CommitLimit

graph TD
 A[Memory allocation request] --> B{Overcommit mode}
 B -->|Mode 0: Heuristic| C["Allow moderate virtual memory overcommit"]
 B -->|Mode 1: Unlimited| D["Virtual memory commits unconstrained"]
 B -->|Mode 2: Strict| E["Virtual memory total ≤ CommitLimit"]
 C --> F[Allocate physical pages on demand at runtime]
 D --> G[May exhaust physical memory + Swap]
 E --> H[Enforce virtual memory total control]

When overcommit_memory=2, only one of the overcommit_ratio and overcommit_kbytes parameters takes effect. The CommitLimit is calculated as follows: $$ CommitLimit = (RAM - huge page memory) × \frac{overcommit_ratio}{100} + SwapTotal $$ or $$ CommitLimit = (RAM - huge page memory) + overcommit_kbytes + SwapTotal $$ Interesting overcommit accounting⁹ — mmap, brk, fork are all accounted for, which clearly affects PostgreSQL:

Status
------
o	We account mmap memory mappings
o	We account mprotect changes in commit
o	We account mremap changes in size
o	We account brk
o	We account munmap
o	We report the commit status in /proc
o	Account and check on fork
o	Review stack handling/building on exec
o	SHMfs accounting
o	Implement actual limit enforcement

Reserve Memory and Overcommit
#

user_reserve_kbytes: When overcommit_memory=2, physical memory reserved for ordinary user processes. When system memory is severely insufficient, it ensures ordinary users can still perform basic operations (like starting new processes, handling memory allocation requests). Default is min(3% of the current process size, 128M). When set to 0, a single process can allocate (all free memory - admin_reserve_kbytes)

admin_reserve_kbytes: Physical memory reserved for users with CAP_SYS_ADMIN privileges (typically root user), ensuring admin recovery capability — reserved physical memory ensuring the system administrator can log in and execute commands. Default is min(3% memory, 8MB). When using strict overcommit mode, it’s best to increase this parameter.

$ cat user_reserve_kbytes 
131072
$ cat admin_reserve_kbytes 
8192

Observing Overcommit
#

grep -E 'CommitLimit|Committed_AS' /proc/meminfo
sar -r 1

$ grep -E 'CommitLimit|Committed_AS' /proc/meminfo
CommitLimit: 203103492 kB
Committed_AS: 252170700 kB

$ sar -r 1
07:32:35 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
07:32:37 PM 25472180 370249056 93.56 14588 274485956 252242936 62.91 233866528 103568816 12924
07:32:38 PM 25471904 370249332 93.56 14588 274487888 252242740 62.91 233851748 103570136 11180

Metric meanings:

meminfo CommitLimit: CommitLimit calculated from physical memory, Swap, and overcommit parameters
meminfo Committed_AS: Total virtual memory currently requested by all processes
sar -r kbcommit = Committed_AS
sar -r %commit = kbcommit / total physical memory

smaps or status can also show total requested virtual memory, but directly summing smaps/status total virtual memory double-counts shared library files and mapped files (like mmap), while Committed_AS only counts memory requested via mmap, brk, fork, etc., and does not double-count shared memory. The two have different calculation scopes. For total virtual memory, just look at Committed_AS or kbcommit.

watermark
#

Parameter Name	Description	Introduced	Default	Unit/Range
min_free_kbytes	Defines the minimum free memory the system reserves, directly affecting the watermarks `watermark[min]` calculation, ensuring the system retains enough memory for critical operations when memory is tight	Early kernel versions		KB
watermark_scale_factor	Globally adjusts the memory watermark gap (`high-low` and `low-min`)	Linux kernel 4.x (exact minor version unknown)	`10` (0.1% physical memory)	Max `3000` (30% physical memory)
watermark_boost_factor	Temporarily raises the high watermark (`high`), triggering aggressive memory reclamation to reduce fragmentation	Linux kernel 4.x (exact minor version unknown)	`15000` (i.e., 1.5x original high watermark)

min_free_kbytes
#

## Calculate total min and other values from zoneinfo
cat /proc/zoneinfo | grep -E -w "min|low|high"|grep -E -v "high:"| awk '
/min/ { total_min += $2 }
/low/ { total_low += $2 }
/high/ { total_high += $2 }
END {
 printf "Total min: %d KB\nTotal low: %d KB\nTotal high: %d KB\n",
 total_min * 4, total_low * 4, total_high * 4;
}'
Total min: 15828844 KB
Total low: 19786048 KB
Total high: 23743260 KB

#Current system min value
cat min_free_kbytes
15828849

Because there are other zones, the total min across all zones is approximately equal to min_free_kbytes. The Normal zone’s min is definitely slightly smaller than min_free_kbytes; you only need to focus on the Normal zone:

## Normal zone min, low, high settings; page=4k
cat /proc/zoneinfo | grep -A 50 Normal | grep -E "min|low|high"
 min 3931615
 low 4914518
 high 5897422

Before Linux kernel 4.6, min, low, and high had a fixed ratio, and you could only change low and high values by setting min_free_kbytes. min:low:high = 1:1.25:1.5.

Problems with the fixed ratio:

Ideally, you’d want to raise low to more proactively trigger kswapd async reclamation and lower min to reduce direct reclaim. Before 4.6, you could only indirectly adjust low/high by adjusting min, using min to adjust kswapd’s delta working buffer. For example:

	kswapd async reclamation working buffer (low-min)	kswapd async reclamation workload (high-low)
min=1GB, low=1.25GB, high=1.5GB	0.25GB	0.25GB
min=10GB, low=12.5GB, high=15GB	2.5GB	2.5GB

Raising min is done to raise low and high.

An excessively low min value causes kswapd to not have time to asynchronously reclaim more memory before direct reclaim triggers. An excessively high min not only wastes memory but also causes more frequent reclamation activity, resulting in higher sys CPU usage. The default difference between low and min in Linux indeed seems a bit small.

watermark_scale_factor
#

Wouldn’t it be great if you could directly adjust min, low, and high? Sorry, the Linux kernel doesn’t support that (Android has extra_free_kbytes). But…

Since Linux kernel 4.x, the watermark_scale_factor parameter was added, allowing adjustment of the ratios between parameters — the ratio is no longer fixed. Its default value is 10, corresponding to 0.1% of memory (10/10000), with a maximum of 3000. When set to 1000, it means the difference between “low” and “min”, and between “high” and “low”, will both be 10% of memory size (1000/10000).

0.1% is clearly too small — for 1TB of memory, the scale is only 1GB.

watermark_boost_factor
#

watermark_boost_factor is used to optimize external memory fragmentation. It temporarily raises the zone’s watermark, i.e., zone->watermark_boost, thereby raising the zone’s high watermark (WMARK_HIGH). This allows kswapd to reclaim more memory, making it easier for the memory compaction module (compactd kernel thread) to merge large blocks of contiguous physical memory. The default value of watermark_boost_factor is 15000, meaning the original high watermark is temporarily raised to 150%. Setting this to 0 disables the mechanism for temporarily raising zone watermarks¹⁰

oom
#

The OOM Killer is a kernel module, not a process.

Parameter Name	Description	Default
panic_on_oom	Controls system behavior when OOM occurs: 0: Don’t trigger panic, start OOM Killer 1: Trigger panic and halt 2: Trigger panic then attempt memory release	0
oom_kill_allocating_task	Whether to preferentially kill the process that triggered OOM (rather than traversing the process tree to select the optimal target): 0: Disabled 1: Enabled	0
oom_dump_tasks	Whether to dump all task information when OOM occurs (for post-mortem analysis): 0: Disabled 1: Enabled	1

oom_score
#

When OOM occurs, the system needs to decide which process to kill based on the OOM score. Each user process has 3 OOM score interface files:

-rw-r--r-- 1 postgres postgres 0 May 24 16:39 /proc/63766/oom_adj
-r--r--r-- 1 postgres postgres 0 May 24 16:39 /proc/63766/oom_score
-rw-r--r-- 1 postgres postgres 0 May 24 16:39 /proc/63766/oom_score_adj

oom_score is a dynamically calculated OOM score by the system, influenced at least by:

Many child processes: +points
Long-running: -points
Low nice value: +points (nice value represents process CPU time slice priority. Lower nice values mean higher priority, more CPU time slice allocation)
Direct hardware access: -points¹¹

In addition to the Linux-calculated OOM score, adjustments (adj) can be manually applied. oom_adj is from earlier Linux kernel versions; it’s best to adjust OOM scores through the oom_score_adj interface file.

Parameter/File	Purpose	Example Values
oom_score	Kernel-calculated raw score (dynamic)	0~1000
oom_score_adj	User-defined adjustment value, directly affects final score	-1000~1000; -1000 equivalent to disabling OOM
oom_adj (legacy)	Legacy adjustment parameter, range -17~15	-17~15

lowmem_reserve_ratio
#

Besides min_free_kbytes, there’s another minimum memory reserve parameter that can cause process memory allocation failures, but their functions differ significantly.

lowmem_reserve_ratio is a key kernel parameter used to protect low-end memory (DMA, DMA32) from being excessively consumed by high-end memory allocation requests. lowmem_reserve_ratio is just a coefficient, not a directly usable number; the kernel calculates the reserved page count for each zone.

#Default values below
cat /proc/sys/vm/lowmem_reserve_ratio 
256 256 32

Memory zones are ordered by priority from low to high: DMA → DMA32 → Normal → HighMem. Allocation requests from higher-priority zones can “borrow” memory from lower-priority zones, but must reserve a certain proportion of memory for use by the lower-priority zones.

cat /proc/zoneinfo |grep -Ew "Node 0|protection|free"
Node 0, zone DMA
 pages free 3976
 protection: (0, 2484, 386430, 386430)
Node 0, zone DMA32
 pages free 415741
 protection: (0, 0, 383946, 383946)
Node 0, zone Normal
 pages free 5658528
 protection: (0, 0, 0, 0)

For example, DMA’s protection indicates:

0: Allocation from this zone, no cross-zone allocation restrictions
2484: Pages DMA reserves for DMA32 zone allocations
386430: Pages DMA reserves for Normal zone allocations
386430: Reserved extension field, meaningless in this context

Based on these settings:

When DMA32 zone requests memory from DMA zone, 3976 > 2484, it may succeed
When Normal zone requests memory from DMA zone, 3976 < 386430, it will not succeed
Requests from lower zones to higher zones are not subject to this restriction

misc
#

A few more related parameters; those with less relevance are not listed:

Parameter	Purpose
nr_hugepages	Number of huge pages
~~nr_overcommit_hugepages~~	Overcommit of huge pages; The maximum is nr_hugepages + nr_overcommit_hugepages
~~nr_hugepages_mempolicy~~	NUMA-localized huge page allocation
~~hugetlb_shm_group~~	Shared memory permission control
~~hugetlb_optimize_vmemmap~~	Restructure huge page metadata management model, reducing memory usage of huge page metadata (struct page). Supported since Linux kernel 5.13
max_map_count	Limits the maximum number of memory mapping regions (VMA) a single process can have, default 65530
zone_reclaim_mode	Memory reclamation policy under NUMA, e.g., allocating memory from other nodes
stat_interval	VM stat refresh frequency, default 1 second
vfs_cache_pressure	Parameter for VFS (Virtual File System) cache reclamation pressure, mainly affecting the aggressiveness of kernel reclaiming dentry and inode caches
page-cluster	Swap readahead, swaps multiple pages to swap partition at once. Default 3, i.e., 8 pages at once

OS Memory Observation and Calculation
#

/proc/meminfo, /proc/vmstat, /proc/zoneinfo all contain memory information, much of it duplicative. I won’t list the differences — a glance tells you what’s what.

free available Calculation (Unfinished)
#

General direction: (NR_FREE_PAGES + NR_FILE_PAGES - NR_SHMEM + NR_SWAP_PAGES + NR_SLBA_RECLAIMABLE - TOTALRESERVE_PAGES - root reserved memory)

The kernel has its own estimated available memory. Directly calculating the available value using a formula is difficult to get exactly right:

## Not very accurate, don't use
cat /proc/meminfo |grep -Ew "MemFree|Active\(file\)|Inactive\(file\)|SwapFree|SReclaimable|nr_shmem|Shmem" |awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2} NR==4 {d=$2} NR==5 {e=$2} NR==6 {f=$2 ;print (a+b+c+d-e+f)}' ;
cat /proc/meminfo |grep -Ew "MemAvailable";

inactive_anon + active_anon != anon
#

Why?

Primary: Shmem separately counts shared memory pages. nr_anon_pages does not include shared memory pages, while nr_inactive_anon and nr_active_anon include anonymous shared memory pages
Secondary: anon includes some Unevictable pages (Mlocked is a subset of Unevictable)
Other minor statistical differences have little impact

A rough but relatively accurate formula: nr_inactive_anon + nr_active_anon + nr_unevictable - nr_shmem

## Applicable under huge pages; not applicable under NUMA
## /proc/meminfo, /proc/zoneinfo, /proc/vmstat can all be used for calculation

#/proc/vmstat
echo -n "anon_computed : ";cat /proc/vmstat|egrep -w "nr_inactive_anon|nr_active_anon|nr_unevictable|nr_shmem"| awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2} NR==4 {d=$2; print (a+b+c-d)}' ;\
echo -n "anon_real : ";cat /proc/vmstat|egrep -w "nr_anon_pages"|awk '{print $2}'
anon_computed : 15776924
anon_real : 15772671

##/proc/zoneinfo Normal
echo -n "anon_normal_computed : "; cat /proc/zoneinfo |grep Normal -A 50|egrep -w "nr_inactive_anon|nr_active_anon|nr_unevictable|nr_shmem"| awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2} NR==4 {d=$2; print (a+b+c-d)}' ;\
echo -n "anon_normal_real : "; cat /proc/zoneinfo |grep Normal -A 50|egrep -w "nr_anon_pages"|awk '{print $2}'
anon_normal_computed : 15711170
anon_normal_real : 15707402

cache Calculation
#

The buff/cache shown in the free command can be calculated from file pages or cache itself:

echo -n "filepage+shmem: ";cat /proc/meminfo |grep -Ew "Buffers|Active\(file\)|Inactive\(file\)|Shmem|SReclaimable"| awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2} NR==4 {d=$2} NR==5 {e=$2 ;print (a+b+c+d+e)}';\
echo -n "cached: ";cat /proc/meminfo |grep -Ew "Buffers|Cached|SReclaimable" | awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2 ;print (a+b+c)}';\
free -k;

#Execution results:
filepage+shmem: 289417584
cached: 289419156
 total used free shared buff/cache available
Mem: 395721236 79633516 26668564 84704912 289419156 178501152
Swap: 5242876 0 5242876

Controversy: Does shmem Count as cache?
#

Clearly, the calculation above includes shmem in cache. Theoretically, shmem shouldn’t be part of cache.

In fact, the kernel community has discussed thisWhy is Shmem included in Cached in /proc/meminfo?, wanting to remove shared memory from cache:

> -	cached = global_node_page_state(NR_FILE_PAGES) -
> -			total_swapcache_pages() - i.bufferram;
> +	cached = global_node_page_state(NR_FILE_PAGES) -
> +			total_swapcache_pages()
> +			- i.bufferram - i.sharedram;

But modifying this involves forward compatibility concerns. The question comes down to: which is more important — forward compatibility or improving the accuracy of a piece of information?

Currently, there’s no good resolution; that’s the status quo.

The email thread also discusses some interesting things:

Another point of view is that everything in tmpfs is part of the page
cache and can be written out to swap

- Dirty: total amount of RAM used to buffer data to be written on
permanent storage (dirty). Gets converted to Cached when write is
complete. (Actually I would call this "Buffers" but Dirty is okay, too.)
- Cached: total amount of RAM used to improve *performance* that can be
*immediately dropped* without any data-loss – note that this includes
all untouched RAM backed by swap.
- Shared: total amount of RAM shared between multiple process that
cannot be freed even if any single process gets killed. (If this is even
possible to know - note that this would *only* contain COW pages in
practice. We already have Committed_AS which is about as good for real
world heuristics.)

cache does not include dirty pages, and can be directly dropped without data loss
tmpfs is swapout

Shared memory appears to be swapout, which is clearly different from cache pages that can be directly dropped. PostgreSQL’s shared memory clearly cannot be directly dropped.

So for PostgreSQL, the fact that cache contains shared memory is quite important — don’t assume by default that it doesn’t.

Memory Page Statistics Often Don’t Add Up
#

When calculating memory pages, some calculations don’t add up. Summary of reasons:

shmem is counted in cache
Cannot see file-mapped and anonymous-mapped pages within shmem
nr_anon_pages does not include shared memory pages, while nr_inactive_anon and nr_active_anon include anonymous shared memory pages
VM and cgroup have slightly different statistical scopes

cgroup v1
#

cgroup Memory Management
#

cgroup can observe and limit the usage of anonymous pages, file pages, swap cache, and kernel memory. Each memcg has its own independent LRU; there is no concept of a GLOBAL LRU.

cgroup memory management differs from cgroup CPU management. A task can request lots of CPU work; reaching the cgroup CPU limit can extend execution time to handle it. However, the memory a task occupies is working memory — a task uses the same physical memory.

Key differences between cgroup CPU and memory management:

Memory must be managed through reuse and reclamation; a task’s working memory is truly occupied and cannot be used by other tasks. CPU is managed through time allocation; other tasks or cgroups can use it.
Memory needs to be instantly available; CPU works through time slicing — time can be dispersed.
CPU control’s core is time allocation; Memory Control’s core is page counting.

The core of the design is a counter called the page_counter. The page_counter tracks the current memory usage and limit of the group of processes associated with the controller

Memory Control’s core is page counting, meaning it’s not that physical pages are statically assigned. The memory allocated this time, when released back to free after use, most likely won’t be the same physical page next time¹².

Physical pages know which cgroup they belong to:

				+--------------------+
				| mem_cgroup |
				| (page_counter) |
				+--------------------+
				 / ^ \
				/ | \
 +---------------+ | +---------------+
 | mm_struct | |.... | mm_struct |
 | | | | |
 +---------------+ | +---------------+
 |
 + --------------+
 |
 +---------------+ +------+--------+
 | page +----------> page_cgroup|
 | | | |
 +---------------+ +---------------+

mm_struct represents virtual memory. Each virtual memory knows which cgroup it belongs to; each physical page can point to page_cgroup, meaning it knows which cgroup this physical memory belongs to¹².

cgroup Parameters and Metrics
#

cgroup uses interface files for configuration and viewing memory usage.

Directory: cd /sys/fs/cgroup/memory/xxx/

Kernel memory and mem+swap can have separate settings or usage viewing:

memory.kmem.xxx #kernel mem
memory.memsw.xxx #mem+swap

Below, we only look at mem-related items.

Interface files can be divided into three categories:

Read-only — show usage, permissions: -r--r--r--
Read-write — control parameters, permissions: -rw-r--r--
Other — special settings, permissions: other

Specific meanings are as follows, with important parameters highlighted:

Type	Interface File	Meaning
Read-only	`memory.numa_stat`	NUMA-dimensional memory stats
Read-only	`memory.stat`	Important, the primary memory usage interface file with many metrics; analyzed separately below
Read-only	`memory.usage_in_bytes`	usage_in_bytes is affected by the method and doesn’t show ’exact’ value of memory. Not recommended for viewing cgroup memory usage
Read-only	`memory.failcnt`	Number of times memory usage exceeded `memory.limit_in_bytes`, cumulative
Read-write	`cgroup.clone_children`	Controls whether child cgroups inherit parent configuration
Read-write	`cgroup.procs`	Used to manage process groups (process IDs, PIDs) in the current cgroup. For multi-process PostgreSQL, this means writing all PG processes, including management processes and backends, into the `procs` file
Read-write	`tasks`	Used to manage threads (thread IDs, TIDs) in the current cgroup. When writing a process PID to `cgroup.procs`, all its thread TIDs are automatically added to `tasks`
Read-write	`notify_on_release`	Controls whether a release operation is triggered when the last task (process or thread) in the cgroup exits. Would only be enabled for container management; traditional cgroup management keeps it disabled by default. Cgroups should be preserved after database restart
Read-write	memory.move_charge_at_immigrate	Deprecated in v2. Charge attribution rules when migrating cgroups
Read-write	memory.use_hierarchy	Whether parent cgroup limits child cgroups
Read-write	memory.limit_in_bytes	cgroup memory upper limit
Read-write	memory.soft_limit_in_bytes	Reclaim the portion exceeding the soft limit
Read-write	memory.max_usage_in_bytes	cgroup usage peak, an observation metric
Read-write	memory.oom_control	oom_kill_disable 1 — disable OOM under_oom 0 — whether currently in OOM state
Read-write	memory.swappiness	cgroup-level swappiness
Other	memory.force_empty	Write only; writing `0` forces release of all cgroup memory
Other	cgroup.event_control	Event notification interface, listens for memory pressure events, requires programming. Often used with memory.pressure_level
Other	memory.pressure_level	Memory pressure notification level

Using a PG instance to explain the meaning of various metrics in memory.stat.

This PG instance is configured as:

shared_memory_type=mmap
shared_buffers=64GB
approximately 800 clients, running

cat memory.stat

cache 345587761152 						 #page cache!!!
rss 27332608 #Anonymous and swap cache memory size. Note: differs from OS process RSS; clearly doesn't include PG shared memory
rss_huge 0 #of bytes of anonymous transparent hugepages. Note: anonymous huge pages
mapped_file 61491769344 #File shared memory size; includes PG shared memory here
swap 0 #On swap partition
pgpgin 389395357 #rss+cache charged pages
pgpgout 305016672 #rss+cache uncharged pages
pgfault 1954040341 #Omitted
pgmajfault 17 #Omitted
inactive_anon 165728256 #anonymous and swap cache memory on inactive LRU
active_anon 61549518848 #anonymous and swap cache memory on active LRU list
inactive_file 138240962560 #file-backed on inactive LRU list
active_file 145658613760 #file-backed memory on active LRU list
unevictable 0 #Unreclaimable memory
hierarchical_memory_limit 408021893120 #
hierarchical_memsw_limit 9223372036854771712 #
total_xxx #hierarchical 

Roughly (ignoring swap), cache+rss = inactive_anon+active_anon+inactive_file+active_file.

These values are quite convoluted. cache+rss doesn’t have a straightforward correspondence with [in]active_anon/file, and mapped_file (shared memory) is hard to categorize, making it easy to get confused. Combining various documentation and testing, I hand-rolled the following script:

#cginfo_lzl
echo -n "shared_mem_mapped : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "mapped_file"| awk '{print $2 / 1024 / 1024 /1024 }' ;\
echo -n "shared_mem_anon : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "rss|inactive_anon|active_anon"| awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2; print (b + c -a)/1024/1024/1024}' ;\
echo -n "pagecache : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "cache"| awk '{print $2 / 1024 / 1024 /1024 }' ;\
echo -n "pagecache_cache-share : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "cache|mapped_file"| awk 'NR==1 {a=$2} NR==2 {b=$2; print (a - b)/1024/1024/1024}';\\n
echo -n "file_total : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "inactive_file|active_file"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "anon_total : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "inactive_anon|active_anon"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_used_rss+map : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "rss|mapped_file"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_mem_file+rss+map : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "inactive_file|active_file|rss|mapped_file"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_mem_rss+cache : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "rss|cache"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_mem_anon+file : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "inactive_file|active_file|inactive_anon|active_anon"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_memsw : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "rss|cache|swap"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "hard_limit : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.limit_in_bytes| awk '{print $1 / 1024 / 1024 /1024 }'

#Database with shared_buffers=2GB
shared_mem_mapped : 1.69063
shared_mem_anon : 1.69828
pagecache : 5.94717
pagecache_cache-share : 4.25654
file_cache : 4.24889
anon_cache : 3.23096
total_used_rss+map : 3.2233
total_mem_file+rss+map : 7.47219
total_mem_rss+cache : 7.47984
total_mem_anon+file : 7.47984
total_memsw : 7.47984
hard_limit : 8

Differences Between cgroup RSS and Process RSS
#

#shared_buffers= 64GB, all PG process RSS sorted
ps -eo pid,ppid,rss,args |grep `cat $PGDATA/postmaster.pid|head -1`|sort -k3 -rn
 97632 97627 61103720 postgres: lzlinst: checkpointer 
 97633 97627 59045152 postgres: lzlinst: background writer 
 97627 1 2322820 /paic/postgres/base/11.3/bin/postgres -D /paic/pg6888/data
 97637 97627 85116 postgres: lzlinst: pgsentinel 
 97697 97627 19620 postgres: lzlinst: dbmgr users [local] idle
 97634 97627 17932 postgres: lzlinst: walwriter 
250063 97627 14508 postgres: lzlinst: dbmon postgres [local] idle
 97636 97627 13220 postgres: lzlinst: stats collector 
248777 97627 11576 postgres: lzlinst: dbmon postgres [local] idle
 97635 97627 2980 postgres: lzlinst: autovacuum launcher 
 97638 97627 2376 postgres: lzlinst: logical replication launcher 
 97630 97627 1592 postgres: lzlinst: logger 
250185 39130 972 grep --color=auto 97627

Generally, the PG processes with the highest RSS values are checkpointer and bgwriter, because RSS represents actual memory used, including shared memory, and these two processes that flush shared buffer dirty pages occupy the most. Backends with excessive data queries may also have higher RSS values, but this is usually caused by data extracts or slow full-scan queries.

Why is postmaster’s RSS so small? Because postmaster itself doesn’t need to do much shared_buffer operations; it only needs to open up the shared memory virtual address space and fork it for other processes to use.

PM’s child processes have the same shared memory address but not necessarily the same RSS:

$ cat /proc/97632/smaps |grep -A 3 "zero" #checkpointer
2b4fd87cf000-2b60a2143000 rw-s 00000000 00:04 15925397 /dev/zero (deleted)
Size: 70411728 kB
Rss: 61087812 kB
Pss: 31429895 kB
$ cat /proc/97633/smaps |grep -A 3 "zero" #bgwriter
2b4fd87cf000-2b60a2143000 rw-s 00000000 00:04 15925397 /dev/zero (deleted)
Size: 70411728 kB
Rss: 59043388 kB
Pss: 29394787 kB
$ cat /proc/97627/smaps |grep -A 3 "zero" #postmaster
2b4fd87cf000-2b60a2143000 rw-s 00000000 00:04 15925397 /dev/zero (deleted)
Size: 70411728 kB
Rss: 2318408 kB
Pss: 1741764 kB

Above, checkpointer and bgwriter occupy the most RSS, and most of their RSS is shared memory. These two processes almost evenly split the entire actually-used shared memory, while postmaster doesn’t use much. PM and all its forked child processes have the same shared memory virtual address.

But cgroup RSS is only a few tens of MB, far less than process RSS:

cat /sys/fs/cgroup/memory/lzlinst/memory.stat |egrep -w "rss|mapped_file"
rss 88997888
mapped_file 52963262464

You can see that PG shared memory is not in the cgroup stat RSS. cgroup RSS doesn’t count file pages or shared file pages.

linux kernel¹²:

Only anonymous and swap cache memory is listed as part of ‘rss’ stat. This should not be confused with the true ‘resident set size’ or the amount of physical memory used by the cgroup.

Process vs. cgroup memory statistics differences¹³:

Memory	Single Process	Process `cgroup(memcg)`
`cache`	None	`PageCache`
`mapped_file`	None	`file_rss + shmem_rss`
`RSS`	`anon_rss + file_rss ＋ shmem_rss`	`anon_rss`

For PostgreSQL, the RSS in stat does not include file map shared memory. The PG official documentation describes mmap as anonymous shared memory:

Possible values are mmap (for anonymous shared memory allocated using mmap), sysv (for System V shared memory allocated via shmget)

cgroup counts PG mmap memory as mapped_file.

Observing sysv and huge page scenarios, summary of PG’s memory.stat metrics:

RSS in stat does not include file map shared memory. Observation shows that regardless of mmap or sysv, RSS does not contain PG shared memory
Similarly, rss_huge also does not include file map shared huge page memory. Observation shows that even with huge pages enabled, stat does not contain PG shared memory
Without huge pages, PG shared memory (mmap or sysv) is all counted under memory.stat mapped_file; with huge pages, it’s in none of the stat metrics, including rss_huge

Where Exactly Is mapped_file?
#

mapped_file is in cache, and also in inactive_anon+active_anon
mapped_file can also be anonymous; both mmap and sysv are counted here

#Database with shared_buffers=2GB
shared_mem_mapped : 1.69063
shared_mem_anon : 1.69828
pagecache : 5.94717
pagecache_cache-share : 4.25654
file_cache : 4.24889
anon_cache : 3.23096
total_used_rss+map : 3.2233
total_mem_file+rss+map : 7.47219
total_mem_rss+cache : 7.47984
total_mem_anon+file : 7.47984
total_memsw : 7.47984
hard_limit : 8

soft_limit_in_bytes
#

Soft limit (memory.soft_limit_in_bytes) is a non-enforced constraint in cgroup memory management. When a cgroup’s memory usage exceeds the soft limit, the system does not immediately force memory reclamation. Instead, it will preferentially reclaim the excess memory of that cgroup when global memory pressure is high (e.g., when overall system free memory is insufficient).

Trigger condition: Global memory pressure (e.g., insufficient system free memory).
Call path: kswapd → balance_pgdat → check cgroup soft limits → trigger reclamation.
Reclamation target: Preferentially reclaim memory pages from cgroups exceeding their soft limits.

+-------------------+ Global memory pressure detection +-------------------+
| kswapd thread | ------------------------------------> | balance_pgdat |
+-------------------+ +-------------------+
 |
 | Traverse memory zones and check
 v
 +---------------------------+
 | Check each cgroup's soft |
 | limit usage |
 +---------------------------+
 |
 | Trigger reclamation for over-limit cgroups
 v
 +---------------------------+
 | Page reclamation (LRU list |
 | scanning, etc.) |
 +---------------------------+

The soft_limit_in_bytes mechanism is very similar to high. In v2, soft_limit_in_bytes has been deprecated, replaced by three new parameters: min, low, and high.

Impact of Overselling on pagecache
#

To be discussed later

cg oom
#

Normally, if sharedbuffer = 1/4 of cg mem, then without counting private memory, pagecache can reach up to 3/4 of cg mem. Generally, normal business private memory usage won’t be very high. If cg mem is full, memory can be reclaimed from cg pagecache (this is direct memory reclamation; AliOS has implemented async background reclamation: Memcg Background Async Reclamation). So the best way to test cg oom is to use sessions that consume lots of private memory rather than stress testing.

Test case:

#Observe score
-r--r--r-- 1 postgres postgres 0 May 24 16:39 /proc/63766/oom_score
rss # whichever command you like

## A SQL that can consume lots of private memory, many union alls create many plan nodes
psql -d lzldb -tX -c "create table lzl1(col1 varchar(1));"
psql -tX -c "\o sqltext.sql" -c "
SELECT 'select col1 from lzl1' || ' union all'
FROM generate_series(1, 100000)
UNION ALL
SELECT 'select col1 from lzl1;'
FROM generate_series(1, 1);
" 

#Adjust stack parameter otherwise SQL will be aborted
psql -d lzldb -c "set max_stack_depth=1024000" -f sqltext.sql

cg oom off:

wchan shows OOM information, even an oom score, but the process won’t be killed by the OOM killer

## vm oom enabled; 0: don't trigger panic, start OOM Killer
$ cat /proc/sys/vm/panic_on_oom 
0

## cg oom disabled; 1: disable oom
$ cat /sys/fs/cgroup/memory/$PGNAME/memory.oom_control
oom_kill_disable 1
under_oom 0

$ ps -eo user,ppid,pid,state,%cpu,%mem,stime,wchan:14,args,rss,vsz,sig_block |grep `head -1 $PGDATA/postmaster.pid` |grep -v grep 
postgres 19005 870 D 0.0 0.0 10:54 mem_cgroup_oom postgres: pg3ymhp2: lzluser 7216 2807460 0000000000000000
postgres 19005 3417 S 0.0 0.0 10:55 pipe_wait postgres: pg3ymhp2: lzluser 22944 2808540 0000000000000000
postgres 19005 13069 D 0.0 0.0 11:10 mem_cgroup_oom postgres: pg3ymhp2: lzluser 11944 2808348 0000000000000000
postgres 19005 13104 D 0.0 0.0 11:10 mem_cgroup_oom postgres: pg3ymhp2: lzluser 12224 2808348 0000000000000000
postgres 19005 14352 D 0.0 0.0 11:10 mem_cgroup_oom postgres: pg3ymhp2: lzluser 11680 2808348 0000000000000000

cat /sys/fs/cgroup/memory/$PGNAME/memory.oom_control
oom_kill_disable 1
under_oom 1

cat /proc/97994/oom_score
11

shared_mem_mapped : 2.00019
shared_mem_anon : 2.0023
pagecache : 2.0023
pagecache_cache-share : 0.00211334
file_cache : 0
anon_cache : 8
total_used_rss+map : 7.99789
total_mem_file+rss+map : 7.99789
total_mem_rss+cache : 8
total_mem_anon+file : 8
total_memsw : 8
hard_limit : 8

Currently, it appears that PG processes may also crash when unable to allocate memory. For example, if walwriter crashes, it can cause all other processes to crash.

cg oom on:

User processes are killed due to high OOM score, sent kill -9. Most PG processes crash; postmaster reset_shared() then automatically restarts other processes. Both message and dmesg show out-of-memory information:

#cg oom enabled
oom_kill_disable 0

pg log:
2025-05-29 19:10:45.945 CST,,,198877,,6838374d.308dd,4,,2025-05-29 18:30:37 CST,,0,LOG,00000,"server process (PID 236413) was terminated by signal 9: Killed","Failed process was running: select col1 from lzl1 union all

message:
May 29 19:10:45 lzlhost kernel: Memory cgroup stats for /t1lzldb: cache:8392988KB rss:8384228KB rss_huge:0KB mapped_file:7458316KB swap:0KB inactive_anon:1310184KB active_anon:15467032KB inactive_file:0KB active_file:0KB unevictable:0KB
May 29 19:10:45 lzlhost kernel: Memory cgroup out of memory: Kill process 236413 (postgres) score 497 or sacrifice child

dmesg:
[Thu May 29 18:26:27 2025] Memory cgroup stats for /t1lzldb: cache:8392988KB rss:8384228KB rss_huge:0KB mapped_file:7458316KB swap:0KB inactive_anon:1310184KB active_anon:15467032KB inactive_file:0KB active_file:0KB unevictable:0KB
[Thu May 29 18:26:27 2025] Memory cgroup out of memory: Kill process 236413 (postgres) score 497 or sacrifice child
[Thu May 29 18:26:27 2025] Killed process 236413 (postgres) total-vm:18828736kB, anon-rss:8328252kB, file-rss:2328kB, shmem-rss:1832kB

Management differences between cg oom on and off for PG databases:

on: cg oom killer will kill processes with high OOM score, typically user processes
off: cg oom killer won’t start. PG processes will hang — they may recover on their own, but PG’s critical processes (like walwriter) might crash due to insufficient memory, and the instance may still go down.

Note: this is cg oom, not vm oom. System-level vm oom is determined by the system-level vm overcommit mechanism.

cg v1 Problems
#

No cg pagetable statistics
No cg slab statistics
No cg hugepage statistics (hugepages are not charged, not just not counted)
No cg async/sync page reclamation statistics
cg RSS and process RSS have different statistical scopes
shmem statistics are messy

What’s New in V2
#

V2 Officially released in Linux 4.5 (March 2016)¹⁴.

cgroup v2 memory management improvements and changes:¹⁵

cg mem interface file	vs v1	Meaning
memory.current	Reworked	Current memory usage. Removes the less useful usage_in_bytes
memory.min	New	Different from VM’s min/low/high. VM watermarks are about remaining OS memory; cg v2 watermarks are about cg memory used. memory.min is a hard memory protection value, default 0. Even when the system has no reclaimable memory, memory at or below this boundary won’t be reclaimed¹⁶
memory.low	New	Best-effort memory protection value, default 0. System preferentially reclaims memory from unprotected cgroups. If still insufficient, reclaims memory between memory.min and memory.low.
memory.high	New	Memory reclamation warning threshold, default max. When cgroup memory usage reaches high, triggers synchronous memory reclamation for this cgroup and children, trying to keep memory below high
memory.max	Reworked	Equivalent to memory.limit_in_bytes
memory.reclaim	Reworked	Active reclamation interface file. v1 only had memory.force_empty for forced clearing
memory.peak	Reworked	Equivalent to max_usage_in_bytes; exceeding peak triggers cg oom killer
memory.oom.group	New	Controls whether cg OOM killer terminates the entire cgroup (1) or just a single process (0). Default 0. If oom_score_adj=-1000, process won’t be killed
memory.events	New	Reports memory-related events
memory.stat	Reworked	Many changes, analyzed separately
memory.zswap.current, memory.zswap.max, memory.zswap.writeback	New	Zswap is a compressed swap mechanism in the Linux kernel. Through compressing memory pages awaiting swap, it reduces disk I/O operations, improving system performance. Its core idea is to compress swap data that would have been written to disk and temporarily store it in memory, only writing data to physical swap devices (like swap partitions or files) when necessary
soft_limit_in_bytes	Removed
memory.oom_control	Removed	This means v2 cannot directly disable cg oom killer; however, fine-grained memory management can be achieved through min/low/high settings and event memory notifications

v2 cg mem management advantages:

Compared to v1, v2 has simpler and clearer hierarchical management
v1 only had OOM kill or freeze; v2 has more means to control memory size (such as memory.min/low/high)
v2 makes it easier to control burst loads¹⁷
Removes the interface file for directly disabling cg oom killer
Adds memory_hugetlb_accounting

memory.stat:

Parameter	Meaning	v1 Counterpart
anon	Anonymous pages	active_anon+inactive_anon
file	File pages, including tmpfs	active_file+inactive_file
kernel (npn)	Total kernel memory, including kernel_stack, pagetables, percpu, vmalloc, slab, and other kernel memory usage.	New
kernel_stack	Memory occupied by kernel stacks.	New
pagetables	page tables	New
sec_pagetables	Secondary page tables, suitable for VMs, GPU devices, network acceleration cards, and other hardware resource isolation scenarios	New
percpu (npn)	Memory size used for per-cpu kernel data structures	New
sock (npn)	network transmission buffers	New
vmalloc (npn)	vmalloc	New
shmem	Including tmpfs, shm, shared anonymous mmap	New
zswap	Memory consumed by zswap compression itself	New
zswapped	Amount of user memory zswapped	New
file_mapped	mmap() size	Somewhat similar to v1 mapped_file, though mapped_file includes tmpfs, shm
file_dirty		Same as v1 dirty
file_writeback		Same as v1 writeback
swapcached		Same as v1 swapcached
anon_thp	Anonymous pages in transparent huge pages	New
file_thp	File pages in transparent huge pages	New
shmem_thp	Transparent huge pages for shm, tmpfs, anonymous mmap	New
inactive_anon, active_anon, inactive_file, active_file, unevictable		Same as v1
slab_reclaimable	As the name suggests	New
slab_unreclaimable	As the name suggests	New
slab (npn)	As the name suggests	New
workingset_refault_anon, workingset_refault_file, workingset_activate_anon, workingset_activate_file, workingset_restore_anon, workingset_restore_file, workingset_nodereclaim	Refaulted page statistics	New
pswpin (npn)	swap in	Same as v1 pgpgin
pswpout (npn)	swap out	Same as v1 pgpgout
pgscan (npn)	scanned pages (in an inactive LRU list)	New
pgsteal (npn)	Reclaimed memory	New
pgscan_kswapd (npn)	As the name suggests	New
pgscan_direct (npn)	As the name suggests	New
pgscan_khugepaged (npn)	Pages scanned by the transparent huge page daemon	New
pgscan_proactive (npn)	Pages scanned proactively	New
pgsteal_kswapd (npn), pgsteal_direct (npn), pgsteal_khugepaged (npn), pgsteal_proactive (npn)	As the name suggests; pgsteal\* corresponds to pgscan\*	New
pgfault (npn)	As the name suggests	Same as v1 pgfault
pgmajfault (npn)	As the name suggests	Same as v1 pgmajfault
pgrefill (npn)	Pages scanned in active LRU	New
pgactivate (npn)	Pages moved to active LRU	New
pgdeactivate (npn)	Pages moved to inactive LRU	New
pglazyfree (npn)	Pages whose release is deferred when under memory pressure	New
pglazyfreed (npn)	Reclaimed lazyfree pages	New
swpin_zero,swpout_zero	zero-filled pages; during Swap In, when the kernel detects page content is all zeros (Zero-filled), marks the page as “zero page” in metadata, skipping disk I/O	New
zswpin,zswpout,zswpwb	zswap-related pages	New
thp_fault_alloc (npn), thp_collapse_alloc (npn), thp_swpout (npn), thp_swpout_fallback (npn)	Transparent huge page-related pages	New
numa_pages_migrated (npn), numa_pte_updates (npn), numa_hint_faults (npn)	NUMA-related pages; also memory.numa_stat exists	New
pgdemote_kswapd, pgdemote_direct, pgdemote_khugepaged, pgdemote_proactive	Unclear what demote means	New
hugetlb	Huge pages	New

v2 cg mem observation advantages:

Adds slab, pagetable, pgscank/pgscand/pgsteal, and huge page info — none of which v1 had
More observation metrics related to specific features, such as sock, vmalloc, transparent huge pages, zswap compression interactions, swap_zero zero-fill interactions, etc.
Shared memory shmem and file_mapped metrics are separated

wchan
#

Waiting Channel, name of the kernel function in which the process is sleeping

Generally, you should check the wchan of processes in D state to see what kernel function the process is waiting on.

-: Running tasks will display a dash (’-’) in this column

poll_schedule_timeout: Common for PM, usually in running state

zz ***Fri May 2 04:50:10 CST 2025
postgres 141378 1 19 0.5 0.4 70585180 2322876 poll_schedule_timeout S 21:06:18 00:02:40 /paic/postgres/base/11.3/bin/postgres -D /paic/pg6888/data
zzz ***Fri May 2 04:50:43 CST 2025
postgres 141378 1 19 0.5 0.4 70585180 2322876 - R 21:06:18 00:02:42 /paic/postgres/base/11.3/bin/postgres -D /paic/pg6888/data

futex_wait_queue_me: Common for SLEEP processes. Occasionally D state

postgres 455358 141378 19 4.7 1.0 70590684 5349576 futex_wait_queue_me S 03:01:12 00:02:47 postgres: t1lzldb: lzl test3 30.181.32.3(39801) COMMIT

hugetlb_fault: Only seen when huge pages are first loaded and load starts up

do_last: Function in the VFS (Virtual File System) path resolution logic, responsible for handling the last component of a file path (such as filename or symbolic link) and triggering actual file operations

lock_page_killable: Lock a physical memory page in an interruptible manner. “Interruptible” means the process is allowed to respond to fatal signals like SIGKILL while waiting for the page lock

rpc_wait_bit_killable: This function relates to the Remote Procedure Call (RPC) mechanism, used in the kernel to wait for changes to certain bit flags

wait_on_page_bit: Wait for changes to page flag states (e.g., PG_locked, PG_writeback)

blkdev_issue_flush: Block device layer cache flush function. Possible call chain: user calls fsync() → file system (e.g., ext4) submits relevant dirty pages to the block device layer → calls blkdev_issue_flush() to ensure device cache is flushed

on_proc_exit: Register cleanup functions for process exit

ima_file_check: Belongs to the IMA (Integrity Measurement Architecture) subsystem, used to verify file integrity during file access; typically involved with open() calls

flush_work: Wait for task completion

call_rwsem_down_write_failed: When attempting to acquire a write lock (down_write()) fails, this function handles write lock contention and waiting logic. It uses spin or sleep mechanisms to make the current process wait for lock release (rwsem: read-write semaphore)

get_request: Appears when iowait is high. Gets a free request structure (struct request) from the block device request queue. If the queue is full (device processing speed insufficient), the thread waits until a request is available

lookup_slow: Slow path for VFS (Virtual File System) path resolution

/**
 * lookup_fast - do fast lockless (but racy) lookup of a dentry
 * @nd: current nameidata
 *
 * Do a fast, but racy lookup in the dcache for the given dentry, and
 * revalidate it. Returns a valid dentry pointer or NULL if one wasn't
 * found. On error, an ERR_PTR will be returned.
 */
 static struct dentry *lookup_fast(struct nameidata *nd)

/* Fast lookup failed, do it the slow way */
static struct dentry *__lookup_slow(const struct qstr *name,
				 struct dentry *dir,
				 unsigned int flags)

static struct dentry *lookup_slow(const struct qstr *name,
				 struct dentry *dir,
				 unsigned int flags)
{
	struct inode *inode = dir->d_inode;
	struct dentry *res;
	inode_lock_shared(inode);
	res = __lookup_slow(name, dir, flags);
	inode_unlock_shared(inode);
	return res;
}

lookup_fast and lookup_slow both search for dentries and return them. lookup_fast searches in the dentry cache; if it fails, lookup_slow is used.

Stress testing with huge pages enabled, no direct memory reclamation, the following events occurred:

lock_page: Appears when iowait is high. When the kernel attempts to lock a memory page, if the page is already locked by another thread/process, the current thread enters a waiting state.

vx_svar_sleep_unlock, vx_ilock, vx_bc_biowait, vx_dio_physio, vx_rwsleep_lock:

vx is a journaling file system developed by Veritas (now owned by Symantec and subsequently spun off as Veritas Technologies), designed for high-performance, high-availability large-scale data storage, primarily targeting enterprise application scenarios. Like xfs and ext4, it is a type of file system.

pipe_wait: When a process attempts to read from or write to a pipe, if the pipe buffer is full (write operation) or empty (read operation), the current thread enters sleep state, waiting for buffer state changes

pipe_write: Entry function for pipe write operations. When the buffer is full, the thread sleeps in this function, waiting for writable space

congestion_wait: When the block device I/O queue is congested (e.g., request queue full or device processing delayed), the kernel uses this function to briefly sleep the thread

wait_iff_congested: Checks whether the block device queue is congested and enters brief sleep if so. Similar to congestion_wait but more lightweight, typically used in memory reclamation or dirty page writeback paths

mem_cgroup_oom_synchronize: When usage_in_bytes reaches limit_in_bytes, marks oom_control.under_oom=1. Whether the OOM killer kernel module is activated depends on oom_control.oom_kill_disable

mem_cgroup_oom: Same as mem_cgroup_oom_synchronize

rmap_walk
#

One of PFRA’s goals is to reclaim shared page frames. To achieve this, the Linux 2.6 kernel can quickly locate all page table entries pointing to the same page frame — this process is called reverse mapping[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)].

When a page frame already referenced by one process is inserted into another process’s page table entries (fork), rmap_walk should also occur

zcat hostlzl_ps_25.04.08.0900.dat.gz|egrep "\-D /dirlzl/pg5998/data|zzz"|less
zzz ***Tue Apr 8 09:10:50 CST 2025
postgres 209987 1 19 0.2 0.5 70247548 2117844 poll_schedule_timeout S 22:17:21 00:01:56 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
zzz ***Tue Apr 8 09:11:20 CST 2025
postgres 209987 1 19 0.2 0.5 70247548 2117844 poll_schedule_timeout S 22:17:21 00:01:56 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
zzz ***Tue Apr 8 09:13:08 CST 2025
postgres 209987 1 19 0.2 0.5 70247548 2117844 - D 22:17:21 00:01:57 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
postgres 225076 209987 19 1.6 0.0 70247548 1720 rmap_walk D 09:11:51 00:00:01 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
postgres 224924 209987 19 0.7 0.0 70247548 1728 rmap_walk D 09:11:46 00:00:00 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
postgres 224817 209987 19 0.5 0.0 70247548 1720 try_to_unmap_file D 09:11:44 00:00:00 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
zzz ***Tue Apr 8 09:19:16 CST 2025
postgres 209987 1 19 0.3 0.5 70247548 2117884 poll_schedule_timeout S 22:17:21 00:02:00 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
postgres 250875 209987 19 0.0 0.0 70247548 2208 - R 09:19:17 00:00:00 /dirlzl/postgres/base/postgressqlbin/postgresdb -D /dirlzl/pg5998/data
zzz ***Tue Apr 8 09:19:48 CST 2025
postgres 209987 1 19 0.3 0.5 70247548 2117884 poll_schedule_timeout S 22:17:21 00:02:01 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data

try_to_unmap_file
#

The try_to_unmap_file() function calls try_to_unmap_cluster(), and try_to_unmap_cluster() scans all page table entries corresponding to linear addresses in that linear region, attempting to clear them[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]. try_to_unmap_file() performs reverse mapping of mapped pages. Note: reverse mapping means finding all VMAs through the page table and reclaiming shared physical page frames.

page_referenced
#

referenced and active are used to control page activity level and are used in page reclamation. When refcount=0, it indicates free pages or pages about to be released[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)].

In kernel.org doc’s Object-Based Reverse Mapping, there is a description of the page_referenced() function³:

page_referenced() which checks all PTEs that map a page to see if the page has been referenced recently

page_referenced() calls page_referenced_obj() which is the top level function for finding all PTEs within VMAs that map the page.

If a page is mapped and it is referenced through the mapping, index hash table, this bit is set. It is used during page replacement for moving the page around the LRU lists

In short, page_referenced() finds all PTEs’ VMAs that map a page through the page frame. This is also a reverse mapping process.

Linux introduced two page flags, PG_active and PG_referenced, to identify the activity level of pages, thereby deciding how to move pages between two lists (active LRU and inactive LRU).

PG_active is used to indicate whether the page is currently active — if this bit is set, the page is active. PG_referenced is used to indicate whether the page has been accessed recently — each time the page is accessed, this bit is set.

page_referenced(): When the operating system performs page reclamation, each time a page is scanned, this function is called to set the page’s PG_referenced bit. If a page’s PG_referenced bit is set but the page is not accessed again within a certain time, its PG_referenced bit will be cleared.¹⁸.

Memory Observation Metrics
#

View basic memory settings:

Observe memory metrics:

Some Questions
#

Do kswapd and Direct Memory Reclamation Execute Together?
#

Yes. If it’s watermark-triggered memory reclamation, pgscand is often accompanied by pgscank; the reverse is not necessarily true. If both pgscank and pgscand are frequent, consider adjusting memory reclamation watermarks, increasing the delta to prevent it from being quickly breached.

However, there’s another case: when fragmentation rate is high and free memory is still plentiful, blocking memory compaction may be directly triggered with pgscand but no pgscank at all. In this case, adjusting watermarks won’t help. Consider enabling huge page memory and increasing shared buffer hit rate to reduce frequent pagecache allocation that fragments memory.

Impact of Oversized pagetable on Memory Reclamation
#

An oversized pagetable increases the cost and time of reverse mapping. During direct memory reclamation, reverse mapping is needed to find all processes’ virtual address spaces (VMAs), then cancel the VMA page table mappings of all processes. This means: the more processes, the larger the pagetable, and the slower the memory reclamation.

The more PostgreSQL processes, the larger the pagetable; the larger shared buffer, the larger the pagetable.

Enabling huge page memory can reduce pagetable size by 500x (4k=>2M), not only freeing up memory but also improving memory reclamation efficiency.

How Large Should shared buffers Be?
#

sharedbuffers = 1/4 cgmem seems to have become an industry standard, but the actual situation is far more complex. Theoretically, reducing sharedbuffers a bit can increase pagecache a bit, actually slightly increasing total cache size. Increasing sharedbuffers a bit slightly reduces total cache size but improves sharedbuffer hit rate somewhat. Clearly, making sharedbuffers too large is bad, and making it too small is also bad. If sharedbuffers is too small, PG’s own working memory becomes too small, effectively offloading memory management to the OS — OS pagecache reclamation will also affect performance. If sharedbuffers is too large, not only is pagecache squeezed, but PG’s dirty page flushing impact must also be considered, especially for write-heavy scenarios where corresponding bgwriter parameters need adjustment.

From rough stress testing:

Without huge pages, shared buffers = min(1/4 MEM, 20GB)
With huge pages, shared buffers = min(1/4 MEM, 60GB)

Is the Difference Between Processes and Threads Really Not That Big?
#

Any Linux kernel material will say that the difference between processes and threads is not significant. Whether creating a process or a thread, the kernel uses the same function, kernel_clone, to implement it. The only difference lies in the parameters passed. The fork and clone system calls are roughly the same[^ 《深入理解Linux进程和内存》 (Understanding Linux Processes and Memory)]:

Dimension	Process	Thread
childID	Each process has an independent `pid` (process ID)	Each thread has a `tid` (thread ID), but the thread’s `pid` is the same as its process’s `pid`.
Address Space	Each process has an independent address space (`mm_struct`), including memory, stack, etc.	Threads share the address space of their process; all threads’ `mm_struct` points to the same address space.
File System	Each process has its own `fs_struct`, including file descriptors, mount points, etc.	Threads share their process’s `fs_struct`; all threads’ file descriptors and mount points are the same as the process.

Compared to processes, threads are only slightly “lighter”. Overall, the similarities between processes and threads outweigh their differences.

However, when the number of processes increases, the difference becomes significant, especially for multi-process applications like PostgreSQL:

Each process has its own VMA, so more address spaces need to be maintained
Each process has its own pagetable, so pagetables consume more memory
Multiple processes increase TLB flush overhead, while threads do not
Process switching requires more context switch overhead, while threads do not
Inter-process communication (IPC) is less efficient, while threads can directly share memory without IPC communication issues

You could say: processes and threads don’t differ much at creation time, but multi-process management and multi-thread management differ greatly.

Why Does the Standby Have PG-Level Dirty Pages?
#

The standby’s WAL replay mechanism itself generates dirty pages, and the standby also flushes dirty pages. You can view standby dirty pages through pg_buffercache. The standby’s dirty pages are different from the primary’s — standby dirty data is also just regular relations. You can also observe that the standby’s checkpoint/bgwriter/backend dirty flushing is different from the primary’s.

Why Is File Cache Higher on Some Databases and Lower on Others?
#

Generally, databases with high data dispersion have more file cache. Simple slow SQL queries are unlikely to maintain high file cache levels long-term. A slow SQL query accessing lots of data might briefly raise filecache, but after a while, these file pages’ reference count drops, becoming inactive file pages, and memory can reclaim this portion. However, frequent data dispersion — such as when an index’s correlation approaches 0 (like a UUID primary key) — results in decent SQL performance but high reads, potentially generating frequent physical IO and loading too many pages into filecache. Even changes in business patterns can cause a large amount of shared buffer swapping in and out, significantly impacting performance.

PG Processes and Shared Memory Mapping
#

#Without huge pages: /dev/zero (deleted)
cat /proc/102208/smaps |egrep "rw\-s" -A 1
2aefd8901000-2aefd8902000 rw-s 00000000 00:04 1202061313 /SYSV00001000 (deleted)
Size: 4 kB
--
2aefd8918000-2aefd898f000 rw-s 00000000 00:13 4084862058 /dev/shm/PostgreSQL.1008001451
Size: 476 kB
--
2aefe2605000-2b00ad129000 rw-s 00000000 00:04 4084864418 /dev/zero (deleted)

#With huge pages: /anon_hugepage (deleted)
cat /proc/29091/smaps |egrep "rw\-s" -A 1
2aaaaac00000-2ac3a2c00000 rw-s 00000000 00:0e 215471503 /anon_hugepage (deleted)
Size: 104726528 kB
--
2b48dfe93000-2b48dfe94000 rw-s 00000000 00:04 88604727 /SYSV00001000 (deleted)
Size: 4 kB
--
2b48dfeab000-2b48dff22000 rw-s 00000000 00:12 215515747 /dev/shm/PostgreSQL.1123685558
Size: 476 kB

Child process page tables are all copied from the parent process; parent and child processes therefore share the same page frames[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]. So whether it’s the postmaster or backend processes (any process forked from postmaster), they all map the same shared memory address in their virtual memory — their addresses and Size in smaps are equal.

Why Do All PG Processes Have /dev/zero as the Largest Segment in Virtual Memory?
#

There are two main ways to implement anonymous page mapping with mmap: one is by setting the MAP_ANONYMOUS flag with fd=-1, and the other is by opening the /dev/zero device file and passing the resulting file descriptor to mmap. These two methods are functionally equivalent.

PG shared buffers use the /dev/zero device mapping to implement anonymous shared pages, which is why you typically see PG processes having a large proportion of their virtual memory address space as /dev/zero.

References
#

[Understanding the Linux Kernel]: Understanding the Linux Kernel: Memory Addressing, Memory Management, Address Space Management, Page Frame Reclamation

[Understanding Linux Processes and Memory]: Understanding Linux Processes and Memory: CPU Hardware Principles, Process and Thread Comparison

[Running Linux Kernel: Beginner’s Guide 2nd Edition]: Running Linux Kernel: Beginner’s Guide 2nd Edition: System Calls, Memory Management

PostgreSQL CLOG Files and Standby Synchronization Analysis

Tue, 03 Sep 2024 00:00:00 +0000

Among all relational databases, PostgreSQL’s CLOG is a very special type of log. CLOG’s existence is inseparable from PostgreSQL’s MVCC mechanism. Some basic knowledge about transaction IDs and CLOG won’t be covered in this article. If interested, please refer to CLOG and Hint Bits. This article focuses on the structure of CLOG files, manually locating transaction states, and the CLOG WAL log synchronization mechanism, to further understand PostgreSQL’s CLOG.

CLOG Segment
#

CLOG Directory
#

To distinguish from regular logs, PostgreSQL 10 renamed the CLOG and WAL directories ¹:

pg9.6	pg10
pg_clog	pg_xact
pg_xlog	pg_wal

Don’t get confused — I was also troubled by pg_xlog and pg_xact for a while…

CLOG Segment Name
#

CLOG is also managed by SLRU, and CLOG file naming is also in slru.c:

#define SlruFileName(ctl, path, seg) \
	snprintf(path, MAXPGPATH, "%s/%04X", (ctl)->Dir, seg)

%04X means hexadecimal (X), width of 4, zero-padded on the left (04). Example CLOG filenames:

[pg_xact]$ ll
-rw------- 1 postgres postgres 262144 Aug 15 16:29 03C0
-rw------- 1 postgres postgres 262144 Aug 19 23:04 03C1
...

TransactionID and CLOG Location Conversion
#

CLOG only stores transaction ID status, not the transaction ID itself. Through the TransactionID itself, you can directly locate the CLOG file and the position within the file. Before that, we need to understand some fundamentals.

Transaction States Stored in CLOG
#

There are only 4 transaction states:

typedef int XidStatus;

#define TRANSACTION_STATUS_IN_PROGRESS		0x00
#define TRANSACTION_STATUS_COMMITTED		0x01
#define TRANSACTION_STATUS_ABORTED		0x02
#define TRANSACTION_STATUS_SUB_COMMITTED	0x03

Transaction states are only: in progress, committed, aborted, subtransaction committed. Note that transaction IDs don’t have an “not started” state — as soon as a transaction ID is allocated in the database, that transaction has definitely already started. Conversely, transaction IDs not yet allocated in the database (actually a few — see the extend CLOG section below) correspond to in_progress status in CLOG. Four transaction states actually only need 2 bits to store. So 1 byte (8 bits) can store 4 transaction states, and 1 page (8k) can hold 8KB*4=32768 transaction states. These are all defined in the source code:

 * Defines for CLOG page sizes. A page is the same BLCKSZ as is used
 * everywhere else in Postgres.
// CLOG page size = BLCKSZ = 8k (default)
#define CLOG_BITS_PER_XACT	2 							 // One transaction state occupies 2 bits
#define CLOG_XACTS_PER_BYTE 4 							 // 1 byte can hold 4 transaction states
#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE) // 1 page can hold 32768 transaction states, 8KB*4=32768
#define CLOG_XACT_BITMASK ((1 << CLOG_BITS_PER_XACT) - 1) // Transaction status bitmask = ((1<<2)-1) = 3, expressed in binary as 11

#define SLRU_PAGES_PER_SEGMENT	32 // 1 segment has 32 pages

Summary:

1 CLOG segment has 32 pages
1 CLOG page is 8k (typically)
1 byte has 4 transaction states
1 transaction state occupies 2 bits

CLOG Segment/Page/Byte Conversion
#

Finding which CLOG segment a transaction ID corresponds to is not easy — it’s hidden in the comments:

 * Note: because TransactionIds are 32 bits and wrap around at 0xFFFFFFFF,
 * CLOG page numbering also wraps around at 0xFFFFFFFF/CLOG_XACTS_PER_PAGE,
 * and CLOG segment numbering at
 * 0xFFFFFFFF/CLOG_XACTS_PER_PAGE/SLRU_PAGES_PER_SEGMENT
// segment number = xid/CLOG_XACTS_PER_PAGE/SLRU_PAGES_PER_SEGMENT = xid/32768/32 // Which CLOG segment the transaction ID corresponds to, xid/32768/32, needs to be converted to hex

Mapping transaction ID to page, byte, etc. is clearer ²:

#define TransactionIdToPage(xid)	((xid) / (TransactionId) CLOG_XACTS_PER_PAGE) // Which CLOG page the transaction ID corresponds to, xid/32768
#define TransactionIdToPgIndex(xid) ((xid) % (TransactionId) CLOG_XACTS_PER_PAGE) // The offset within the above page, xid%32768
#define TransactionIdToByte(xid)	(TransactionIdToPgIndex(xid) / CLOG_XACTS_PER_BYTE) // Which byte in the page the transaction ID corresponds to, (xid%32768)/4
#define TransactionIdToBIndex(xid)	((xid) % (TransactionId) CLOG_XACTS_PER_BYTE)		// Which bit index in the above byte (note: bit index, not the bit itself), xid%4

Generally (with 8k BLCKSZ), 1 CLOG segment has 32 pages; 1 CLOG segment has 328k bytes, i.e., CLOG file size is fixed at 256K; 1 CLOG segment can hold 432*8k transaction states.

[pg_xact]$ ll # 256k CLOG segment
-rw------- 1 postgres postgres 262144 Aug 15 16:29 03C0
-rw------- 1 postgres postgres 262144 Aug 19 23:04 03C1
...

CLOG Bit Conversion
#

The functions for setting CLOG bits and getting CLOG bits (corresponding to TransactionIdSetStatusBit and TransactionIdGetStatus) both have the following code to obtain which two bits in the CLOG the transaction ID corresponds to:

	int			bshift = TransactionIdToBIndex(xid) * CLOG_BITS_PER_XACT;
	char	 *byteptr;
...
	byteptr = XactCtl->shared->page_buffer[slotno] + byteno;
	curval = (*byteptr >> bshift) & CLOG_XACT_BITMASK;

bshift represents the right-shift position, where TransactionIdToBIndex=xid%4, CLOG_BITS_PER_XACT=2, CLOG_XACT_BITMASK=3 (binary: 11). The key code for getting CLOG bits curval = (*byteptr >> bshift) & CLOG_XACT_BITMASK can be understood in two parts:

*byteptr >> bshift means right-shifting the pointer by 0, 2, 4, or 6 bits
& CLOG_XACT_BITMASK is simply taking the last two bits after the right shift (00&11=00, 01&11=01, 10&11=10, 11&11=11)

So, calculating the position of a transaction ID’s state within a byte:

xid%4=0: takes bits 7 and 8
xid%4=1: takes bits 5 and 6
xid%4=2: takes bits 3 and 4
xid%4=3: takes bits 1 and 2

Note: the transaction ID state’s bit positions within a byte are taken in reverse order, not sequentially forward. Byte and page positions are taken in sequential increasing order.

Manually Calculating Transaction ID Position in CLOG File
#

If we want to manually locate a transaction in CLOG using hexdump, we need to calculate three elements: <CLOG segment number, offset within segment in bytes, offset on byte in bit index>. (This references the approach in “PostgreSQL Database Kernel Analysis” but with some differences ³)

Before calculating, you also need to understand:

CLOG segment file numbers are in hexadecimal
hexdump is in hexadecimal, each line holds 16 bytes, i.e., each line holds 16*CLOG_XACTS_PER_BYTE=16*4=64 transaction states
hexdump -s xxx is in byte units

The following SQL can calculate the position of a transaction ID in CLOG:

-- CLOG segment number
-- %4294967296 represents transaction ID wraparound, /(8192*4*32) represents the maximum number of transactions a segment file can contain, to_hex converts to hex for filename, lpad left-pads to 4 digits
select lpad(upper(to_hex(txid_current()%4294967296/(8192*4*32))),4,'0') as clog_segmentno;

-- Offset within segment in bytes
-- %4294967296 represents transaction ID wraparound, %(8192*32*4) takes the remaining transaction IDs, /4 converts to byte units
select txid_current()%4294967296%(8192*32*4)/4 as in_clog_offset_bytes;

-- Offset on byte in bit index
-- %4294967296 represents transaction ID wraparound, %4 takes the bit index within the byte
select txid_current()%4294967296%4 as in_byte_offset_bitindex;


-- Or a single SQL
select lpad(upper(to_hex(txid_current()%4294967296/(8192*4*32))),4,'0') as clog_segmentno,txid_current()%4294967296%(8192*32*4)/4 as in_clog_offset_bytes,txid_current()%4294967296%4 as in_byte_offset_bitindex;

Practical simulation — computing a transaction ID’s state in CLOG:

begin;
 select lpad(upper(to_hex(txid_current()%4294967296/(8192*4*32))),4,'0') as clog_segmentno,txid_current()%4294967296%(8192*32*4)/4 as in_clog_offset_bytes,txid_current()%4294967296%4 as in_byte_offset_bitindex;
 clog_segmentno | in_clog_offset_bytes | in_byte_offset_bitindex 
----------------+----------------------+-------------------------
 0002 | 63196 | 3
rollback;
checkpoint; 

Rollback is used to roll back the transaction, mainly for easier observation, since most transactions are committed. Checkpoint is to ensure the CLOG page is flushed — otherwise the CLOG page might still be in the CLOG buffer and not yet written to the CLOG segment file.

cd pg_xact/
 hexdump -C 0002 -s 63196 -n 1 -v
0000f6dc 95 |.|
0000f6dd

-- Convert hex to binary
> select 'x96'::bit(8);
 bit 
----------
 10010110

When xid%4=3, take bits 1 and 2. So the bit value for this rolled-back transaction is 10, where 10 represents TRANSACTION_STATUS_ABORTED.

Why CLOG Usually Contains Many 55s and U’s?
#

In a typical transactional database CLOG file, a direct hexdump looks like this:

hexdump -C 0001 -v|head -10
00000000 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000010 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000020 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000030 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000040 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000050 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000060 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000070 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000080 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000090 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|

Because the committed transaction state = 01 = TRANSACTION_STATUS_COMMITTED. When 4 consecutive transactions in a byte are all committed, it becomes 01010101.

Binary: 01010101, hex: 55
Hex 55 in ASCII is ‘U’, so when visually examining CLOG files you can generally see many U’s
Occasionally some bytes are not 55 or U because in production environments some transactions occasionally haven’t completed or use subtransactions. The committed state of subtransactions in CLOG is 0x03.

Shared CLOG Buffer
#

The number of CLOG shared buffers is easy to understand:

/*
 * Number of shared CLOG buffers.
 *
 * On larger multi-processor systems, it is possible to have many CLOG page
 * requests in flight at one time which could lead to disk access for CLOG
 * page if the required page is not found in memory. Testing revealed that we
 * can get the best performance by having 128 CLOG buffers, more than that it
 * doesn't improve performance.
 *
 * Unconditionally keeping the number of CLOG buffers to 128 did not seem like
 * a good idea, because it would increase the minimum amount of shared memory
 * required to start, which could be a problem for people running very small
 * configurations. The following formula seems to represent a reasonable
 * compromise: people with very low values for shared_buffers will get fewer
 * CLOG buffers as well, and everyone else will get 128.
 */
Size
CLOGShmemBuffers(void)
{
	return Min(128, Max(4, NBuffers / 512));
}

Translation: Testing has shown that 128 CLOG buffers provide the best performance — more than that doesn’t improve performance. However, because some database configurations are too small, 128 CLOG buffers seems a bit large, so it takes 1/512 of the shared_buffers count. In other words: Number of CLOG buffers = 1/512 shared_buffer, minimum is 4, maximum is 128. Note: these are all buffer counts, not sizes!

How large is a single buffer? CLOG buffer is managed by SLRU, and each SLRU page is 8k:

A page is the same BLCKSZ as is used everywhere

We can glimpse the size of shared CLOG buffer from the perspective of CLOG SLRU initialization:

/*
 * Initialization of shared memory for CLOG
 */
Size
CLOGShmemSize(void)
{
	return SimpleLruShmemSize(CLOGShmemBuffers(), CLOG_LSNS_PER_PAGE);
}

The passed CLOGShmemBuffers() is 4~128, and the passed CLOG_LSNS_PER_PAGE = 1024 bytes (with 8k pages). SimpleLruShmemSize initializes SLRU shared memory:

Size
SimpleLruShmemSize(int nslots, int nlsns)
{
	Size		sz;

	/* we assume nslots isn't so large as to risk overflow */
	sz = MAXALIGN(sizeof(SlruSharedData));
	sz += MAXALIGN(nslots * sizeof(char *));	/* page_buffer[] */
	sz += MAXALIGN(nslots * sizeof(SlruPageStatus));	/* page_status[] */
	sz += MAXALIGN(nslots * sizeof(bool));	/* page_dirty[] */
	sz += MAXALIGN(nslots * sizeof(int));	/* page_number[] */
	sz += MAXALIGN(nslots * sizeof(int));	/* page_lru_count[] */
	sz += MAXALIGN(nslots * sizeof(LWLockPadded));	/* buffer_locks[] */

	if (nlsns > 0)
		sz += MAXALIGN(nslots * nlsns * sizeof(XLogRecPtr));	/* group_lsn[] */

	return BUFFERALIGN(sz) + BLCKSZ * nslots;
}

SLRU uses some arrays to store SLRU metadata and control information. The sz size is all roughly data type * buffer count, and these are generally not very large. The main initialized memory is BLCKSZ * nslots, i.e., 8k * (4~128) = (32k~1M). So we can roughly estimate that the shared CLOG buffer size is around 1M.

CLOG WAL: Types, Writing, and Redo
#

When writing CLOG, is CLOG WAL log also written? If so, wouldn’t that mean lost CLOG could be restored by reapplying WAL logs to recover transaction states? Let’s explore the CLOG WAL writing and redo source code with these questions in mind.

Extend CLOG
#

ZeroCLOGPage writes WAL. ZeroCLOGPage(pageno, true) is actually only called by ExtendCLOG:

/*
 * Make sure that CLOG has room for a newly-allocated XID.
 *
 * NB: this is called while holding XidGenLock. We want it to be very fast
 * most of the time; even when it's not so fast, no actual I/O need happen
 * unless we're forced to write out a dirty clog or xlog page to make room
 * in shared memory.
 */
void
ExtendCLOG(TransactionId newestXact)
{
	int			pageno;

	/*
	 * No work except at first XID of a page. But beware: just after
	 * wraparound, the first XID of page zero is FirstNormalTransactionId.
	 */
	if (TransactionIdToPgIndex(newestXact) != 0 &&
		!TransactionIdEquals(newestXact, FirstNormalTransactionId))
		return;

	pageno = TransactionIdToPage(newestXact); // CLOG page number converted from TransactionId

	LWLockAcquire(XactSLRULock, LW_EXCLUSIVE);

	/* Zero the page and make an XLOG entry about it */
	ZeroCLOGPage(pageno, true);

	LWLockRelease(XactSLRULock);
}

ZeroCLOGPage mainly calls WriteZeroPageXlogRec:

/*
 * Write a ZEROPAGE xlog record
 */
static void
WriteZeroPageXlogRec(int pageno)
{
	XLogBeginInsert();
	XLogRegisterData((char *) (&pageno), sizeof(int));
	(void) XLogInsert(RM_CLOG_ID, CLOG_ZEROPAGE);
}

WriteZeroPageXlogRec is writing a WAL record, with type “RM_CLOG_ID, CLOG_ZEROPAGE”. Using waldump, you can view CLOG_ZEROPAGE. Its proportion is generally very small:

pg_waldump -z 000000010000056B00000018 --stat=record
Type N (%) Record size (%) FPI size (%) Combined size (%)
---- - --- ----------- --- -------- --- ------------- ---
...
CLOG/ZEROPAGE 1 ( 0.00) 30 ( 0.00) 0 ( 0.00) 30 ( 0.00)
...

Extending CLOG page is always in page units. In fact, at the end of a CLOG segment you can easily see 00s:

hexdump 03C2
0000000 5555 5555 5555 5555 5555 5555 5555 5555
*
001bb30 5555 5555 0055 0000 0000 0000 0000 0000
001bb40 0000 0000 0000 0000 0000 0000 0000 0000 
* ## The end of the CLOG file is all zeros
001c000

Truncate CLOG
#

Besides extending CLOG, there’s also truncating CLOG. Truncate CLOG is called during vacuum. When called, it writes a truncate CLOG WAL record and flushes the WAL record to disk:

/*
 * Remove all CLOG segments before the one holding the passed transaction ID
 *
 * Before removing any CLOG data, we must flush XLOG to disk, to ensure
 * that any recently-emitted FREEZE_PAGE records have reached disk; otherwise
 * a crash and restart might leave us with some unfrozen tuples referencing
 * removed CLOG data. We choose to emit a special TRUNCATE XLOG record too.
 * Replaying the deletion from XLOG is not critical, since the files could
 * just as well be removed later, but doing so prevents a long-running hot
 * standby server from acquiring an unreasonably bloated CLOG directory.
 *
 * Since CLOG segments hold a large number of transactions, the opportunity to
 * actually remove a segment is fairly rare, and so it seems best not to do
 * the XLOG flush unless we have confirmed that there is a removable segment.
 */
void
TruncateCLOG(TransactionId oldestXact, Oid oldestxid_datoid)
{
	int			cutoffPage;

	/*
	 * The cutoff point is the start of the segment containing oldestXact. We
	 * pass the *page* containing oldestXact to SimpleLruTruncate.
	 */
	// What's written to WAL is the CLOG position, which is the CLOG page number converted from oldestXact
	cutoffPage = TransactionIdToPage(oldestXact); 

.....
	/*
	 * Write XLOG record and flush XLOG to disk. We record the oldest xid
	 * we're keeping information about here so we can ensure that it's always
	 * ahead of clog truncation in case we crash, and so a standby finds out
	 * the new valid xid before the next checkpoint.
	 */
	// WriteTruncateXlogRec writes the corresponding WAL record and flushes it to disk
	WriteTruncateXlogRec(cutoffPage, oldestXact, oldestxid_datoid);
	
	// After WAL is written, actually execute the CLOG segment truncation
	/* Now we can remove the old CLOG segment(s) */
	SimpleLruTruncate(XactCtl, cutoffPage);
}

WriteTruncateXlogRec writes a WAL record with RMGR as RM_CLOG_ID and info as CLOG_TRUNCATE:

/*
 * Write a TRUNCATE xlog record
 *
 * We must flush the xlog record to disk before returning --- see notes
 * in TruncateCLOG().
 */
static void
WriteTruncateXlogRec(int pageno, TransactionId oldestXact, Oid oldestXactDb)
{
	XLogRecPtr	recptr;
	xl_clog_truncate xlrec;

	xlrec.pageno = pageno;
	xlrec.oldestXact = oldestXact;
	xlrec.oldestXactDb = oldestXactDb;

	XLogBeginInsert();
	XLogRegisterData((char *) (&xlrec), sizeof(xl_clog_truncate));
	recptr = XLogInsert(RM_CLOG_ID, CLOG_TRUNCATE);
	XLogFlush(recptr);
}

After generating CLOG WAL records, the redo recovery routine is also needed:

/*
 * CLOG resource manager's routines
 */
void
clog_redo(XLogReaderState *record)
{
...
	// When redo info type is CLOG_ZEROPAGE, place the read redo information in memory, then write to the CLOG page file
	if (info == CLOG_ZEROPAGE)
	{
		int			pageno;
		int			slotno;

		memcpy(&pageno, XLogRecGetData(record), sizeof(int));

		LWLockAcquire(XactSLRULock, LW_EXCLUSIVE);

		slotno = ZeroCLOGPage(pageno, false);
		SimpleLruWritePage(XactCtl, slotno); 
		Assert(!XactCtl->shared->page_dirty[slotno]);

		LWLockRelease(XactSLRULock);
	}
	// When redo info type is CLOG_TRUNCATE, place the read redo information in memory, confirm the page is deletable (write page if not), then truncate the segment
	else if (info == CLOG_TRUNCATE)
	{
		xl_clog_truncate xlrec;

		memcpy(&xlrec, XLogRecGetData(record), sizeof(xl_clog_truncate));

		/*
		 * During XLOG replay, latest_page_number isn't set up yet; insert a
		 * suitable value to bypass the sanity test in SimpleLruTruncate.
		 */
		XactCtl->shared->latest_page_number = xlrec.pageno;

		AdvanceOldestClogXid(xlrec.oldestXact);

		SimpleLruTruncate(XactCtl, xlrec.pageno);
	}
	else
		elog(PANIC, "clog_redo: unknown op code %u", info);
}

What the CLOG redo routine does:

When redo info type is CLOG_ZEROPAGE: finds a suitable slot (evict if necessary), performs writability checks based on the read redo information (actually the CLOG page number), then writes the page to the CLOG file
When redo info type is CLOG_TRUNCATE: based on the read redo information (actually the CLOG page number), confirms the page is deletable (write page if not available), then truncates the CLOG segment

CLOG Synchronization Summary
#

CLOG has only two types of WAL logs, neither containing transaction status information. They are only triggered when extending CLOG pages and truncating CLOG segments, and the written WAL record is just a CLOG page number. CLOG’s WAL log RMGR type has only one: RM_CLOG_ID. This type has only two info codes: CLOG_ZEROPAGE, CLOG_TRUNCATE.

/* XLOG stuff */
#define CLOG_ZEROPAGE 0x00
#define CLOG_TRUNCATE 0x10

CLOG WAL synchronization summary: The standby database is essentially not synchronizing CLOG information — it’s only synchronizing some CLOG file expansion and deletion information.

However, the standby’s CLOG file clearly does have status information, and the standby obviously needs this information for visibility checking. How is the transaction status in CLOG synchronized?

Transaction ID WAL: Types, Writing, and Redo
#

The WAL for rmgr=CLOG doesn’t contain transaction status. Does the standby not synchronize CLOG transaction information? No — WAL logs do contain transaction ID status information, and CLOG is also updated:

-- Roll back a transaction, commit a transaction
> begin;
BEGIN
> select txid_current();
 txid_current 
--------------
 1817254
(1 row)

> rollback;
ROLLBACK
> begin;
BEGIN
> select txid_current();
 txid_current 
--------------
 1817258
(1 row)

> commit;
COMMIT
> checkpoint;
CHECKPOINT

-- pg_waldump to view transaction ID status in logs
[datalzl/pg_wal]$ pg_waldump ../../pg_wal/000000010000007300000008|grep -E "1817254|1817258"
rmgr: Transaction len (rec/tot): 34/ 34, tx: 1817254, lsn: 73/400ED210, prev 73/400ED1E0, desc: ABORT 2024-08-01 14:41:26.017612 CST
rmgr: Transaction len (rec/tot): 46/ 46, tx: 1817258, lsn: 73/400EEB08, prev 73/400EEAD8, desc: COMMIT 2024-08-01 14:41:37.042545 CST
pg_waldump: fatal: error in WAL record at 73/400F7C78: invalid record length at 73/400F7F88: wanted 24, got 0

The WAL records the status of transaction IDs (1817254, 1817258), recorded as ABORT and COMMIT respectively; rmgr is Transaction. Transaction ID status is in WAL logs, but does PostgreSQL write it to the standby’s CLOG? Obviously, we need to find this redo information. Based on previous experience, clog_redo represents the WAL redo source code for rmgr=CLOG. Searching the source for _redo should find the WAL redo source code for rmgr=Transaction. Searching… in xact.c we find the function xact_redo, which mainly calls xact_redo_commit and xact_redo_abort, clearly corresponding to WAL log application logic for committed and rolled-back transactions respectively.

void
xact_redo(XLogReaderState *record)
{
	uint8		info = XLogRecGetInfo(record) & XLOG_XACT_OPMASK;

	/* Backup blocks are not used in xact records */
	Assert(!XLogRecHasAnyBlockRefs(record));

	if (info == XLOG_XACT_COMMIT)
	{
	...
		xact_redo_commit(&parsed, XLogRecGetXid(record),
						 record->EndRecPtr, XLogRecGetOrigin(record));
	}
...
	else if (info == XLOG_XACT_ABORT)
	{
	...
		xact_redo_abort(&parsed, XLogRecGetXid(record));
	}
...
	}
	else
		elog(PANIC, "xact_redo: unknown op code %u", info);
}

Taking commit as an example:

static void
xact_redo_commit(xl_xact_parsed_commit *parsed,
				 TransactionId xid,
				 XLogRecPtr lsn,
				 RepOriginId origin_id)
{
...

	if (standbyState == STANDBY_DISABLED)
	{
		/*
		 * Mark the transaction committed in pg_xact.
		 */
		TransactionIdCommitTree(xid, parsed->nsubxacts, parsed->subxacts);
	}
	else // standby logic
	{
	...
		/*
		 * Mark the transaction committed in pg_xact. We use async commit
		 * protocol during recovery to provide information on database
		 * consistency for when users try to set hint bits. It is important
		 * that we do not set hint bits until the minRecoveryPoint is past
		 * this commit record. This ensures that if we crash we don't see hint
		 * bits set on changes made by transactions that haven't yet
		 * recovered. It's unlikely but it's good to be safe.
		 */
		// Mark transaction committed in pg_xact
		TransactionIdAsyncCommitTree(xid, parsed->nsubxacts, parsed->subxacts, lsn);

...
}

It looks like TransactionIdAsyncCommitTree is the function we’re looking for that writes to CLOG.

To verify the redo logic for transaction commit information in WAL, let’s set three breakpoints on the standby’s startup process, then execute begin;select txid_current();commit; on the source database to commit a transaction, and see if the standby’s startup process hits the functions we want to see when doing redo:

(gdb) bt
#0 TransactionIdAsyncCommitTree (xid=xid@entry=1818665, nxids=0, xids=0x0, lsn=lsn@entry=495398394064) at transam.c:274
#1 0x000000000050c139 in xact_redo_commit (parsed=parsed@entry=0x7ffda52c0fc0, xid=1818665, lsn=495398394064, origin_id=<optimized out>) at xact.c:5805
#2 0x000000000050ffa3 in xact_redo (record=0x2b5ff2434038) at xact.c:5962
#3 0x0000000000519ea5 in StartupXLOG () at xlog.c:7411
#4 0x000000000072f301 in StartupProcessMain () at startup.c:204
#5 0x0000000000528701 in AuxiliaryProcessMain (argc=argc@entry=2, argv=argv@entry=0x7ffda52c6ef0) at bootstrap.c:450
#6 0x000000000072c459 in StartChildProcess (type=StartupProcess) at postmaster.c:5494
#7 0x000000000072ec44 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x2b5ff242d1c0) at postmaster.c:1407
#8 0x000000000048931f in main (argc=3, argv=0x2b5ff242d1c0) at main.c:210
(gdb) info b
Num Type Disp Enb Address What
1 breakpoint keep y 0x000000000050c060 in xact_redo_commit at xact.c:5753
 breakpoint already hit 43 times
2 breakpoint keep y 0x0000000000508190 in TransactionIdCommitTree at transam.c:262
3 breakpoint keep y 0x00000000005081a0 in TransactionIdAsyncCommitTree at transam.c:274
 breakpoint already hit 1 time

The breakpoint TransactionIdAsyncCommitTree is hit, and xid=1818665, which is the transaction ID just committed on the source database. This confirms the code logic we visually traced is correct. So, the standby database’s CLOG transaction ID status is synchronized by WAL with rmgr=Transaction.

Summary
#

CLOG only stores transaction ID status, not the transaction ID itself
Transaction status in CLOG files can be manually located via the transaction ID
WAL for rmgr=CLOG only extends and cleans up CLOG files, it does not update transaction status
WAL for rmgr=Transaction updates CLOG transaction status

References
#

“Quickly Mastering PostgreSQL Version New Features”, p24 ↩︎
Yan Shuli, PostgreSQL CLOG Analysis https://www.modb.pro/db/606433 ↩︎
“PostgreSQL Database Kernel Analysis”, Chapter 7, p380-390 ↩︎

PostgreSQL Logical Replication

Tue, 13 Aug 2024 00:00:00 +0000

What is Logical Replication
#

PostgreSQL logical replication is based on logical decoding, which parses WAL log streams into a specified format for output. The subscriber node receives the parsed data and applies it.

Logical replication differs from streaming replication (physical replication) which is based on instance-level primary-standby where the physical structures are identical. Logical replication can selectively replicate at the table level. Logical Replication in official documentation specifically refers to the “publish-subscribe” model. In fact, many tools can use logical decoding for heterogeneous database data synchronization.

pg9.4’s pglogical plugin can support logical replication (https://github.com/2ndQuadrant/pglogical), and pg10 onwards natively supports logical replication.

Logical replication can be used for database upgrades, heterogeneous data migration, table-level data synchronization links, subscribing to data streams, etc.

Logical Decoding
#

Logical decoding can parse table data changes in WAL logs into row data streams or SQL text. These row data streams or SQL text can be consumed by other types of databases or software. The specific parsing format is determined by the output plugin.

Replication Slots
#

In logical replication, a replication slot represents a data change stream. Like physical replication slots, logical replication slots also ensure that after an abnormal replication interruption, the related WAL logs are not deleted, so that WAL log parsing can continue after replication reconnects. A database can have multiple replication slots. Each replication slot has only one output plugin, and each replication slot represents one replication link. Replication slots are essentially used to manage replication links. Unlike streaming replication which can function without replication slots, logical replication must have replication slots.

Output Plugin
#

The output plugin converts WAL log information into the format required by the replication slot. PostgreSQL has some built-in output plugins and additional ones can be added through plugins. Each logical replication slot has an output plugin for WAL-related parsing work.

Output plugins use callback functions to manage parsing. For example, OUTPUT_PLUGIN_BINARY_OUTPUT and OUTPUT_PLUGIN_TEXTUAL_OUTPUT are used to set whether the out_type is binary or text. There are also callback functions to notify the plugin of transaction data changes and sort transactions. Callback functions of course don’t need to be used manually; some built-in output plugins are already packaged.

Each output plugin has some different parsing behaviors and output formats.

Several Common Output Plugins
#

test_decoding: This is a sample output plugin, essentially the raw form of an output plugin. Official documentation says it’s a template, but it can still parse. This output plugin comes with PostgreSQL but needs to be compiled in contrib.

pgoutput: The default output plugin for the publish-subscribe model. In publish-subscribe, the walsender process uses this output plugin to logically decode WAL logs.

decoder_raw: Parses into SQL text format. This is not included with PostgreSQL; compile it yourself: https://github.com/michaelpq/pg_plugins/tree/main/decoder_raw

wal2json: This output plugin converts WAL log information into JSON format.

Other output plugins can be referenced at: https://wiki.postgresql.org/wiki/Logical_Decoding_Plugins

Some domestic vendors have also made their own output plugins.

Relationship between several output plugins and logical replication plugins:

pgoutput, test_decoding, and wal2json have been introduced above.

pglogical was the predecessor of pglogical replication in pg9.4.

BDR was developed by 2ndQuadrant, supporting bidirectional replication and DDL synchronization with more powerful features. BDR 3.0 onwards became closed-source.

Functions and Tools for Manually Receiving Parsed Data
#

pg_logical_slot_get_changes(): Displays parsed data and consumes it.

pg_logical_slot_peek_changes(): Displays parsed data without consuming it.

pg_recvlogical: A tool included with PostgreSQL that can consume data within a replication slot, equivalent to the downstream of logical replication. The corresponding physical WAL receiving tool is pg_receivewal.

Logical Decoding Test 1: Observing data parsing with 2 different output plugins
#

-- Create two logical replication slots using logical_test and logical_raw respectively

lzldb=# select pg_create_logical_replication_slot('logical_test','test_decoding');

 pg_create_logical_replication_slot 

------------------------------------

 (logical_test,0/1756F50)

lzldb=# select pg_create_logical_replication_slot('logical_raw','decoder_raw');

 pg_create_logical_replication_slot 

------------------------------------

 (logical_raw,0/1756F88)

-- Only the upstream is created, slot is in f state

lzldb=# select * from pg_replication_slots;

 slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn | wal_status | safe_wal_size 

--------------+---------------+-----------+--------+----------+-----------+--------+------------+------+--------------+-------------+---------------------+------------+---------------

 logical_test | test_decoding | logical | 16385 | lzldb | f | f | | | 558 | 0/1766878 | 0/17668B0 | reserved | 

 logical_raw | decoder_raw | logical | 16385 | lzldb | f | f | | | 557 | 0/1756F50 | 0/1756F88 | reserved | 

-- Create a table

lzldb=# create table tdecoder222(a int,b varchar(10));

CREATE TABLE

-- Attempt to get this DDL

lzldb=# SELECT * FROM pg_logical_slot_get_changes('logical_raw', NULL, NULL, 'include-xids', '0');

ERROR: option "include-xids" = "0" is unknown

CONTEXT: slot "logical_raw", output plugin "decoder_raw", in the startup callback

lzldb=# SELECT * FROM pg_logical_slot_get_changes('logical_test', NULL, NULL, 'include-xids', '0');

 lsn | xid | data 

-----------+-----+--------

 0/17669C8 | 558 | BEGIN

 0/1776778 | 558 | COMMIT

-- We can see that decoder_raw didn't parse the DDL at all, and logical_test only got the DDL transaction without the DDL statement itself, essentially not parsing the DDL

-- Insert a row

lzldb=# insert into tdecoder222 values(1,'lzl');

INSERT 0 1

lzldb=# select * from pg_logical_slot_peek_changes('logical_test',null,null);

 lsn | xid | data 

-----------+-----+---------------------------------------------------------------------------

 0/1776890 | 560 | BEGIN 560

 0/1776890 | 560 | table public.tdecoder222: INSERT: a[integer]:1 b[character varying]:'lzl'

 0/1776900 | 560 | COMMIT 560

lzldb=# select * from pg_logical_slot_peek_changes('logical_raw',null,null);

 lsn | xid | data 

-----------+-----+----------------------------------------------------------

 0/1776890 | 560 | INSERT INTO public.tdecoder222 (a, b) VALUES (1, 'lzl');

-- test_decoding parsed the transaction

-- decoder_raw parsed the transaction into SQL statements

This test allows us to conclude:

Replication slots in f state still parse, waiting for downstream consumption
Each output plugin has some different parsing behaviors and output formats

Logical Decoding Test 2: Using pg_recvlogical to receive logically decoded data, simulating a logical replication link
#

-- Configure passwordless login

[pg@lzl ~]$ vi .pgpass

[pg@lzl ~]$ cat .pgpass

lzl:5410:lzldb:pg:pg

[pg@lzl ~]$ chmod 0600 .pgpass

-- Start pg_recvlogical

[pg@lzl ~]$ pg_recvlogical -h lzl -p 5410 -d lzldb -U pg --slot=logical_raw --start -f recv.sql &

[pg@lzl ~]$ ps -ef|grep recv|grep -v grep

pg 7747 7355 0 21:40 pts/3 00:00:00 pg_recvlogical -h lzl -p 5410 -d lzldb -U pg --slot=logical_raw --start -f recv.sql

lzldb=# insert into tdecoder222 values(2,'qwe');

INSERT 0 1

lzldb=# update tdecoder222 set b='asd' where a=2;

UPDATE 1

[pg@lzl ~]$ tail -2f recv.sql

INSERT INTO public.tdecoder222 (a, b) VALUES (2, 'qwe');

-- update was not correctly parsed

-- Add a primary key to the table

lzldb=# alter table tdecoder222 add primary key(a);

ALTER TABLE

lzldb=# insert into tdecoder222 values(100,'lzl1');

INSERT 0 1

lzldb=# insert into tdecoder222 values(200,'lzl2');

INSERT 0 1

lzldb=# update tdecoder222 set b='lzlupdate' where a=200;

UPDATE 1

[pg@lzl ~]$ tail -3f recv.sql

INSERT INTO public.tdecoder222 (a, b) VALUES (100, 'lzl1');

INSERT INTO public.tdecoder222 (a, b) VALUES (200, 'lzl2');

UPDATE public.tdecoder222 SET a = 200, b = 'lzlupdate' WHERE a = 200;

– After adding a primary key, update was correctly parsed by decoder_raw – Without a primary key, it won’t be correctly parsed. This is related to replica identity, which will be introduced later.

Prerequisites for Logical Replication
#

1. Parameters
#

1.1 Basic Required Parameters

wal_level. Takes effect after restart, default is replica. The wal_level parameter must be logical. logical does not change WAL to logical; it means that on top of supporting physical replication (replica), the necessary information for logical decoding is added. Since pg9.6, there are only minimal, replica, and logical, with information content increasing successively.
max_replication_slots. Takes effect after restart, default value below pg9.6 is 0, pg10 and above is 10. 10 is generally sufficient. Like physical replication, logical replication generally also uses replication slots. PostgreSQL backups and physical replication can both occupy replication slot counts.

1.2 Source-side Required Parameters

max_wal_senders. Takes effect after restart, default 10. Sender process count limit. The publisher’s sender transmits the parsed logs. Generally, one logical replication slot corresponds to one sender and one worker. This is similar to physical replication, where one physical replication slot corresponds to one sender and one receiver.

1.3 Target-side Required Parameters

max_worker_processes. Takes effect after restart, default 8. Worker process count limit. Parallel processes (parallel queries, parallel statistics collection, etc., limited by max_parallel_workers), logical replication worker processes (max_logical_replication_workers), and some other programs that need to fork workers are all related to this parameter. It should be set to max_parallel_workers + logical replication apply workers + other background workers.
max_logical_replication_workers. Takes effect after restart, default 4. Logical replication worker process count, including logical replication apply worker processes and table sync worker processes.
max_sync_workers_per_subscription. Takes effect after reload, default 2. Sync worker processes when adding new tables to logical replication. Currently, one table has only one parallel.
The above three parameters are tiered: max_sync_workers_per_subscription < max_logical_replication_workers < max_worker_processes. In short, there must be workers available.

2. Permissions
#

Replication user permissions. Logical replication users need replication privileges.

ALTER ROLE <usename> WITH REPLICATION;

HBA access restrictions, allowing downstream to access the database using the replication user.

host lzldb user1 172.17.100.150/32 md5

For the publish-subscribe model, CREATE permission on the database or superuser permission is needed.

When creating a publication, for table only, at least the table owner with CREATE permission is needed. All other publications require superuser.

When creating a subscription, superuser is required.

grant create on database lzl1db to owner1; or

alter user replicate1 superuser;

Additionally, read or write permissions on tables during replication are also necessary.

Logical Synchronization Between PostgreSQL Instances — Publish and Subscribe
#

PostgreSQL’s built-in logical replication is based on the publish-subscribe model. The publish-subscribe model does not parse into SQL for application.

Publication
#

A publisher can have multiple publications, and each publication can have multiple tables.
When publishing, you can specify:

for table — publishes certain tables. New tables need to be explicitly added with ALTER PUBLICATION ADD TABLE. At minimum, the table owner is needed to create this publication.

for all tables — publishes all tables under the database. New tables are automatically published. Superuser is required to create this publication.

for all tables in schema — publishes all tables under the schema. New tables are automatically published. Superuser is required to create this publication. Supported starting from pg15.

Publications by default include INSERT, UPDATE, DELETE, and TRUNCATE. You can also specify to replicate only certain commands. DDL is not synchronized. (Official documentation verbatim. This means truncate is not considered DDL in PostgreSQL — leaving this as a topic for later research. Truncate is DDL in MySQL and Oracle.)
Only base tables can be published; temporary tables, foreign tables, views, sequences, etc. cannot be published. Partitioned table publishing is related to PostgreSQL version and partition attributes. pg15 defaults to publishing all partitions of a partitioned table.
publish_via_partition_root. Supported from pg13. This publication parameter indicates whether partitioned tables use partitions for filtering (false, default) or use the parent partition for row filtering. If set to true, heterogeneous partitioned table logical replication is supported, such as partitioned table to regular table replication. truncate replication is not possible when true.

Subscription
#

A subscription has only one publisher but can subscribe to multiple publications on the publisher.
A subscriber can have multiple subscriptions, each receiving data from one replication slot.
One subscription corresponds to one replication slot, which is on the publisher side.
When creating or deleting a subscription, the replication slot is automatically created or deleted on the publisher by default.
Creating a subscription requires superuser.
DDL is not synchronized; tables must already be created.
Existing data is synchronized by default, via COPY snapshot to the subscriber.
Synchronization can be paused and resumed with ALTER SUBSCRIPTION sub1 {ENABLE|DISABLE}.
When a publication adds new tables, refresh is needed on the subscriber side: alter subscription sub1 refresh publication.
Schema names, table names, and column names must be consistent between publication and subscription. Column types can differ (as long as implicit conversion succeeds). Column order can be different.
Subscriptions also have some attributes, such as binary transfer, streaming, synchronous commit, two-phase commit, etc.

logical replication launcher is used to start the subscriber-side worker processes and only exists at startup.

/*-------------------------------------------------------------------------
...
 * IDENTIFICATION

 * src/backend/replication/logical/launcher.c
...
 * NOTES
 * This module contains the logical replication worker launcher which
 * uses the background worker infrastructure to start the logical
 * replication workers for every enabled subscription.
 *-------------------------------------------------------------------------

 */

Publish-Subscribe Related Views
#

pg_publication; – View publications. Publications themselves are stateless; replication slots are stateful, so there’s no pg_stat_publication.

pg_publication_tables – View published tables, simple and clear.

pg_publication_rel – View published tables, all IDs.

pg_stat_subscription – View subscription status, pid is the worker process pid.

pg_subscription – View subscriptions.

pg_subscription_rel – View subscription tables. There’s no pg_subscription_tables. Additionally, this view can show the sync status of individual tables under a subscription, which the replication slot view cannot do.

\dRp list replication publications

\dRs list replication subscriptions

Creating a Publication and Subscription
#

Using a dedicated replication user replicate1, create a publication and subscription in the database lzldb to implement logical replication of table trep1.

`Role`	`Host IP`	`Port`	`Database`	`Schema`	`Table`	`Replication User`	`Version`
`Publisher`	`172.17.100.150`	`5410`	`lzldb`	`public`	`trep1`	`replicate1`	`pg13`
`Subscriber`	`172.17.100.150`	`5412`	`lzlbd`	`public`	`trep1`	`replicate1`	`pg13`

Creating the Publication
#

# Modify postgres.conf, wal_level parameter takes effect after restart

wal_level=logical 

# Modify pg_hba.conf file, takes effect after reload

host lzldb replicate1 172.17.100.150/32 md5

-- Create replication user and grant privileges

create user replicate1 with password 'replicate1';

alter user replicate1 with replication;

grant create on database lzldb to replicate1;

-- Create the table to be replicated and grant privileges to the replication user

\c lzldb replicate1 -- If the replication user is not the table owner, should grant select on trep1 to replicate1

create table trep1(a int primary key,b char(10));

insert into trep1 values(1,'abc') 

-- Create publication, superuser can also be used

\c lzldb replicate1

create publication pub_lzl1 for table trep1;

-- View publication. \dRp or pg_publication
lzldb=# select * from pg_publication;

 oid | pubname | pubowner | puballtables | pubinsert | pubupdate | pubdelete | pubtruncate | pubviaroot 

-------|----------|----------|--------------|-----------|-----------|-----------|-------------|-----------
 16400 | pub_lzl1 | 16392 | f | t | t | t | t | f

Creating the Subscription
#

-- Create table definition

create table trep1(a int primary key,b char(10));

-- Use superuser to create subscription

CREATE SUBSCRIPTION sub_test

CONNECTION 'host=172.17.100.150 port=5410 dbname=lzldb user=replicate1 password=replicate1'

PUBLICATION pub_lzl1;

lzlbd=# select * from pg_subscription; -- View subscription. \dRs or pg_subscription

 oid | subdbid | subname | subowner | subenabled | subconninfo | subslotname | subsynccommit | subpublications 

-------|---------|----------|----------|------------|--------------------------------------------------------------------------------|-------------|---------------+-----------------
 16394 | 16384 | sub_test | 10 | t | host=172.17.100.150 port=5410 dbname=lzldb user=replicate1 password=replicate1 | sub_test | off | {pub_lzl1}

lzlbd=# select * from trep1; -- Verify existing data has been synchronized

 a | b 

---+------------
 1 | abc

Publish-Subscribe Model Test 1: Truncate Synchronization
#

lzldb=# truncate table trep1;

TRUNCATE TABLE

lzldb=# select * from trep1;

 a | b 

---+---
(0 rows)

lzlbd=# select * from trep1; -- In publish-subscribe mode, truncate is synchronized

 a | b 

---+---
(0 rows)

Publish-Subscribe Model Test 2: Adding New Table Synchronization
#

-- Under an existing publish-subscribe, add a new table synchronization. lzldb is publisher, lzlbd is subscriber

lzldb=# create table tab_pk(a int,b varchar(10));

CREATE TABLE

lzldb=# alter table tab_pk add primary key(a);

ALTER TABLE

lzldb=# alter publication pub_lzl1 add table tab_pk;

ALTER PUBLICATION

-- After adding a table on the publisher, refresh must be executed on the subscriber. Refresh defaults to synchronizing existing data

lzlbd=# alter subscription sub_test refresh publication; 

ALTER SUBSCRIPTION

lzlbd=# select * from pg_subscription_rel ;

 srsubid | srrelid | srsubstate | srsublsn 

---------+---------+------------+-----------
 16394 | 16389 | r | 0/15F2898
 16394 | 16400 | d | 

-- Subscription state codes: i = initializing, d = copying data, s = synchronized, r = ready (normal replication)

-- At this point, table tab_pk data has not been synchronized because the subscriber's replication user lacks query permission on the table

lzldb=# grant select on tab_full to replicate1;

GRANT

lzlbd=# select * from pg_subscription_rel ;

 srsubid | srrelid | srsubstate | srsublsn 

---------+---------+------------+-----------
 16394 | 16389 | r | 0/15F2898
 16394 | 16400 | r | 0/172D830

-- Subscription is in ready state, new table synchronization complete

Replica Identity
#

Replica identity is written into WAL logs to identify a row of data. Whether it’s publish-subscribe or third-party logical sync tools, they all need to locate rows in the table to identify which row downstream the update or delete affects.

PostgreSQL supports 4 replica identity modes.

default(d): Default identity for non-system tables. Uses primary key if the table has one; if no primary key, it’s nothing.
index(i): Uses a non-null unique index as the identity. Must be non-null and unique to identify a row. If only unique, there can be multiple null values. You can also explicitly specify the primary key in index mode.
full(f): Uses all columns of the row as the identity. Full mode increases WAL log volume.
nothing(n): Default mode for system tables. No identity; update and delete cannot affect downstream.

-- View table's replica identity:
select relname,relreplident from pg_class where relname='tabname1';
-- When a table's replica identity is i, check if the index is the replica identity:
\d tabname
select rel.relname,idx.indisreplident from pg_index idx ,pg_class rel where idx.indexrelid=rel.oid and relname='idx_1';

Modify table replica identity:

ALTER TABLE tab1 REPLICA IDENTITY { DEFAULT | USING INDEX index_name | FULL | NOTHING };

Replica Identity Test 1: Setting a non-null unique index as replica identity for a table without a primary key
#

lzldb=# create table tab_idx(a int,b varchar(10));

CREATE TABLE

lzldb=# select relname,relreplident from pg_class where relname='tab_idx';

 relname | relreplident 

---------+--------------
 tab_idx | d

lzldb=# create unique index idx_1 on tab_idx(b);

CREATE INDEX

lzldb=# alter table tab_idx alter b set not null; -- The index used as replica identity must be a non-null unique index

ALTER TABLE

lzldb=# select rel.relname,idx.indisreplident from pg_index idx ,pg_class rel where idx.indexrelid=rel.oid and relname='idx_1';

 relname | indisreplident 

---------+----------------
 idx_1 | f

lzldb=# alter table tab_idx REPLICA IDENTITY using index idx_1; -- Modify table's replica identity

ALTER TABLE

lzldb=# select rel.relname,idx.indisreplident from pg_index idx ,pg_class rel where idx.indexrelid=rel.oid and relname='idx_1';

 relname | indisreplident 

---------+----------------
 idx_1 | t

lzldb=# \d tab_idx -- pg_index or \d to view index replica identity. \d can only display explicitly modified index replica identity

 Table "public.tab_idx"

 Column | Type | Collation | Nullable | Default 

--------+-----------------------+-----------+----------+---------
 a | integer | | | 
 b | character varying(10) | | not null | 

Indexes:
 "idx_1" UNIQUE, btree (b) REPLICA IDENTITY

Replica Identity Test 2: Full mode — can duplicate rows be synchronized normally?
#

-- Execute the following on the publisher

lzldb=# create table tab_full (a int,b varchar(10)); -- Add table sync without primary key and non-null index

CREATE TABLE

lzldb=# insert into tab_full values(1,'abc'); -- Insert 5 identical rows

INSERT 0 1

lzldb=# grant select on tab_full to replicate1;

GRANT

lzldb=# alter publication tab_full add table tab_pk;

ALTER PUBLICATION

--

lzlbd=# alter subscription sub_test refresh publication; 

ALTER SUBSCRIPTION

lzlbd=# select ctid,* from tab_full ;

 ctid | a | b 

-------+---+-----
 (0,1) | 1 | abc
 (0,2) | 1 | abc
 (0,3) | 1 | abc
 (0,4) | 1 | abc
 (0,5) | 1 | abc

lzldb=# delete from tab_full where ctid='(0,2)';

ERROR: cannot delete from table "tab_full" because it does not have a replica identity and publishes deletes

HINT: To enable deleting from the table, set REPLICA IDENTITY using ALTER TABLE.

lzldb=# update tab_full set a=2 where ctid='(0,5)';

ERROR: cannot update table "tab_full" because it does not have a replica identity and publishes updates

HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.

-- When the table's replica identity is d(default), without a primary key it's nothing. nothing cannot replicate delete and update.

lzldb=# alter table tab_full replica identity full;

ALTER TABLE

lzldb=# delete from tab_full where ctid='(0,2)'; -- After setting replica identity to full, delete succeeds

DELETE 1

lzlbd=# select ctid,* from tab_full ; --

 ctid | a | b 

-------+---+-----
 (0,2) | 1 | abc
 (0,3) | 1 | abc
 (0,4) | 1 | abc
 (0,5) | 1 | abc

lzldb=# update tab_full set a=2 where ctid='(0,5)'; 

UPDATE 1

lzldb=# select ctid,* from tab_full; 

 ctid | a | b 

-------+---+-----
 (0,1) | 1 | abc
 (0,3) | 1 | abc
 (0,4) | 1 | abc
 (0,6) | 2 | abc

lzlbd=# select ctid,* from tab_full ; 

 ctid | a | b 

-------+---+-----
 (0,3) | 1 | abc
 (0,4) | 1 | abc
 (0,5) | 1 | abc
 (0,6) | 2 | abc

– This example proves 3 points: – 1. When replica identity is d(default), it defaults to primary key; if no primary key, it’s nothing. – 2. nothing cannot replicate delete and update. – 3. Duplicate data in full mode can also be normally logically replicated. Although the ctid of data rows differs, the replication goal is still achieved.

Third-Party Synchronization Software
#

Third-party synchronization software already has relatively mature solutions and is widely used, such as OGG, DTS, KTL, etc.

These sync tools are very flexible. They can achieve true heterogeneous synchronization, from PostgreSQL databases to different databases or Kafka, big data consumption platforms, etc.

Of course, they can also sync from other architecture data platforms to PostgreSQL databases, such as the now common Oracle to PostgreSQL sync scenario.

Since we’re mainly discussing the PostgreSQL database itself, when PostgreSQL acts as the downstream target, it’s just some data write issues with very few problems. There won’t be logical decoding, replication slot issues, etc. So this small section won’t discuss PostgreSQL as a heterogeneous sync target. We’ll only observe and summarize scenarios where PostgreSQL acts as the upstream syncing to heterogeneous databases. These third-party tools generally utilize PostgreSQL’s own logical decoding, specify their own output plugin, and automatically create replication slots and replication links. Some tools automatically create subscriptions, while others only have replication slots without subscriptions.

Having already understood logical decoding, output plugins, replication slots, replica identity, and prerequisites for replication, let’s simulate a PostgreSQL to Oracle sync by directly configuring the prerequisites and starting synchronization.

Creating OGG Sync from PostgreSQL to Oracle
#

Software Installation:

ogg for oracle: Oracle GoldenGate 21.3.0.0.0 for Oracle on Linux x86-64

ogg for pg: Oracle GoldenGate 21.3.0.0.0 for PostgreSQL on Linux x86-64

oracle: 11.2.0.4

pg: 13.10

Installation steps:

OGG installation and deployment won’t be introduced here. I followed the article’s installation steps step by step. Installation article reference: https://liuzhilong.blog.csdn.net/article/details/129252320?spm=1001.2014.3001.5502

Sync architecture diagram:

lzldb=# select * from pg_replication_slots where slot_name='ext_pg_5d4b1d39f7494f79';

-[ RECORD 1 ]-------+------------------------
slot_name | ext_pg_5d4b1d39f7494f79
plugin | test_decoding -- OGG defaults to using test-decoding
slot_type | logical
datoid | 16385
database | lzldb
temporary | f
active | t -- As long as OGG extract is running, the replication slot is active
active_pid | 3509
xmin | 
catalog_xmin | 591
restart_lsn | 0/17F3E38
confirmed_flush_lsn | 0/17F4020
wal_status | reserved
safe_wal_size | 

select * from pg_stat_replication
-[ RECORD 2 ]----+------------------------------
pid | 3509
usesysid | 10
usename | pg
application_name |GoldenGateCapture
client_addr | 127.0.0.1
client_hostname | 
client_port | 43665
backend_start | 2023-02-28 15:12:17.350469+08
backend_xmin | 
state | streaming
sent_lsn | 0/17F4140
write_lsn | 0/17F4020
flush_lsn | 0/17F4020
replay_lsn | 
write_lag | 
flush_lag | 
replay_lag | 
sync_priority | 0
sync_state | async
reply_time | 2023-02-28 16:39:44.986625+08

-- replay_lsn has no value
-- Even lag has no value

Logical Replication Monitoring
#

An important method for logical replication lag monitoring is checking lag from the replication software. Without that, you can only check from the replication slot view. The replication slot view provides quite a lot of information, such as whether the replication slot is active directly indicating whether the replication link is syncing.

The replication slot view is very important for logical replication monitoring. Some additional monitoring for publish-subscribe was introduced earlier. Here we focus on broader logical replication monitoring.

pg_replication_slots
#

The replication slot view shows information about each replication slot and some slot statuses. Manually created slots or slots automatically created by tools and subscriptions are all displayed here.

slot_name	Replication slot name
plugin	Output plugin name for logical replication slots. If empty, it’s a physical replication slot
slot_type	physical or logical
datoid	Database ID for logical replication slot
database	Database for logical replication slot
temporary	Whether it’s a temporary replication slot. Temporary slots are not written to disk and are automatically deleted when the session ends. pg_basebackup uses temporary slots by default
active	Replication slot status: t or f. If f, you should quickly consider restarting the replication link or deleting it, as it may block WAL log deletion and fill up the primary database disk. This is related to the max_slot_wal_keep_size parameter
active_pid	walsender PID using this replication slot. Only present when the slot status is t
xmin	Minimum transaction ID the slot needs to hold
catalog_xmin	Minimum catalog transaction ID the slot needs to hold
restart_lsn	LSN position of WAL the slot needs to retain to ensure downstream consumer’s required WAL won’t be cleaned. max_slot_wal_keep_size parameter is the maximum WAL size the slot needs to retain. Beyond this value, WAL can also be deleted. Default -1 means never cleaned. This value represents the LSN position after the downstream’s latest checkpoint consumption and can help locate replication link lag
confirmed_flush_lsn	LSN confirmed received by the logical replication downstream. Empty for physical replication slots
wal_status	`Status of WAL claimed by this replication slot` reserved: the slot reserves WAL, WAL hasn’t exceeded max_wal_size (auto-checkpoint interval) extended: the slot reserves WAL, WAL has exceeded max_wal_size but the slot still retains it. WAL in this state is still within wal_keep_size or max_slot_wal_keep_size unreserved: the slot no longer retains needed WAL, WAL will be deleted at next checkpoint lost: WAL needed by the slot has been cleaned, slot is invalid. `The last two states are seen only when max_slot_wal_keep_size is non-negative. This is easy to understand, since max_slot_wal_keep_size is the criterion for whether WAL can be deleted. Without a mechanism to delete slot WAL, unreserved and lost states wouldn't appear.` `If restart_lsn is NULL, this field is null. Also easy to understand — if there's no WAL LSN, you can't know the WAL retention position or judge whether WAL has exceeded wal_keep_size or max_slot_wal_keep_size.`
safe_wal_size	Number of WAL bytes that can be written before WAL files would be deleted. If this value is negative or zero, it means max_slot_wal_keep_size has been exceeded, and WAL files will be deleted as soon as a checkpoint occurs, requiring the standby using this slot to be rebuilt

pg_stat_replication
#

Rather than replication status, it’s more accurate to call it walsender status. This view shows the status of each walsender, one record per walsender.

If present in pg_replication_slots but not in pg_stat_replication, the walsender is gone; logical replication is down; pg_replication_slots active should be f.
If absent in pg_replication_slots but present in pg_stat_replication, this is physical replication without a replication slot.

You can have replication stat info without a replication slot. Replication slots with walsenders also need this view because it reveals more replication status info than pg_replication_slots.

So when the replication slot hasn’t failed, pg_stat_replication is very important for monitoring logical replication lag.

pid	walsender PID, same as pg_replication_slots active_pid
usesysid	User OID connected to this walsender, i.e., the downstream’s replication user OID
usename	Username connected to this walsender
application_name	Downstream application name. If subscription, it’s the subscription name. If pg_recvlogical, it’s pg_recvlogical
client_addr	Downstream IP. If empty, it’s a local socket connection
client_hostname	Downstream hostname
client_port	Downstream port. If -1, it’s a local socket connection
backend_start	Backend start time, i.e., when downstream connected to walsender
backend_xmin	Standby’s xmin when hot_standby_feedback is enabled. This is clearly for physical replication
state	States are relatively easy to understand. startup: walsender starting. catchup: walsender catching up with primary logs. streaming: walsender has caught up with primary logs, normal replication state. backup: walsender sending backup, this state appears for walsender used for backup. stopping: walsender stopping
sent_lsn	LSN sent
write_lsn	LSN written to disk by downstream
flush_lsn	LSN flushed to disk by downstream
replay_lsn	LSN replayed by downstream
write_lag	Log lag between primary flush wal and downstream write
flush_lag	Log lag between primary flush wal and downstream flush
replay_lag	Log lag between primary flush wal and downstream relay
sync_priority	Synchronization priority
sync_state	Synchronization state
reply_time	Last reply time

Relationship between sent_lsn, write_lsn, flush_lsn, replay_lsn
#

The above nicely shows the hierarchical relationship of sent_lsn, write_lsn, flush_lsn.

These monitoring metrics look very much like streaming replication. For logical replication, sent_lsn, write_lsn, flush_lsn also generally have values.

However, when logical replication doesn’t know what the downstream is, the replay log replay action may not exist, so logical replication may not have replay_lsn.

But one thing is confirmed effective: sent_lsn.

After reviewing pg_replication_slots and pg_stat_replication view monitoring, we find that neither shows log parsing delay; at most, you can see log transmission delay.

pg_stat_replication_slots
#

This view has been available since pg14. It specifically monitors logical replication slot status and can additionally monitor spill status. For pg13, you can only check the pg_replslot directory. Spill will be introduced later.

Logical Replication Slot Transaction Snapshots and pg_logical Directory
#

The transaction snapshots needed by replication slots are persisted to disk. The source code is in snapbuild.c.

void
SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn)
{
if (builder->state < SNAPBUILD_CONSISTENT)
SnapBuildRestore(builder, lsn);
else
SnapBuildSerialize(builder, lsn);
}

Snap persistence has two behaviors: one is restore, loading from disk to memory; the other is serialize, persisting from memory to disk.

Transaction snapshot persistence:

SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
...
sprintf(path, "pg_logical/snapshots/%X-%X.snap",
(uint32) (lsn >> 32), (uint32) lsn);
...
else if (ret == 0)
{
/*
 * somebody else has already serialized to this point, don't overwrite
 * but remember location, so we don't need to read old data again.
 *
 * To be sure it has been synced to disk after the rename() from the
 * tempfile filename to the real filename, we just repeat the fsync.
 * That ought to be cheap because in most scenarios it should already
 * be safely on disk.
 */
fsync_fname(path, false);
fsync_fname("pg_logical/snapshots", true);

builder->last_serialized_snapshot = lsn;
goto out;
}

Transaction snapshot loading into memory:

SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn)
...
if (builder->state == SNAPBUILD_CONSISTENT)
return false;

sprintf(path, "pg_logical/snapshots/%X-%X.snap",
(uint32) (lsn >> 32), (uint32) lsn);

fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);

The transactions needed by logical replication slots, before being committed, store dirty transaction data and unconsumed data under pg_logical/snapshots/. After committing data or starting the replication slot, data is handed to reorderbuffer; or after cleaning the replication slot, the data is released.

My environment has a long-unused slot with restart_lsn at 0/1776858:

postgres=# select slot_name,plugin,slot_type,database,active,restart_lsn from pg_replication_slots where slot_name='logical_test';
 slot_name | plugin | slot_type | database | active | restart_lsn 
--------------+---------------+-----------+----------+--------+-------------
 logical_test | test_decoding | logical | lzldb | f | 0/1776858

The oldest snapshot under pg_logical/snapshots/ is it:

[pg@lzl snapshots]$ ll
total 300
-rw------- 1 pg pg 144 Feb 23 20:41 0-1776858.snap
-rw------- 1 pg pg 144 Feb 23 20:44 0-1776900.snap
-rw------- 1 pg pg 144 Feb 23 20:45 0-1776938.snap

Delete unwanted replication slot:

select pg_drop_replication_slot('logical_test');

After a few minutes, snap is deleted:

[pg@lzl snapshots]$ ll 0-1776858.snap
ls: cannot access 0-1776858.snap: No such file or directory

Logical Decoding Working Memory and Spill to pg_replslot
#

logical_decoding_work_mem
#

Before pg13, logical decoding would retain at most 4096 changes in memory (max_changes_in_memory hardcoded). Beyond 4096 changes, transaction data would be written to disk.

pg13 introduced the logical_decoding_work_mem parameter. Working memory used by logical decoding. All walsender decoding uses this shared memory area. If the data held by logical decoding exceeds this memory value, it’s written to disk. Logical decoding working memory size defaults to 64MB.

Related ReorderBuffer and Spill
#

Description in reorderbuffer.c:

 * This module gets handed individual pieces of transactions in the order
 * toplevel transaction sized pieces. When a transaction is completely
 * reassembled - signaled by reading the transaction commit record - it
 * will then call the output plugin (cf. ReorderBufferCommit()) with the
 * individual changes. The output plugins rely on snapshots built by
 * snapbuild.c which hands them to us.

When a transaction commits, reorderbuffer can receive transaction entries and sort them, then send data changes to the output plugin for output. The output plugin relies on snapshots built by snapbuild.c, which are handed to reorderbuffer.

/*
 * Maximum number of changes kept in memory, per transaction. After that,
 * changes are spooled to disk.
 *
 * The current value should be sufficient to decode the entire transaction
 * without hitting disk in OLTP workloads, while starting to spool to disk in
 * other workloads reasonably fast.
 *
 * At some point in the future it probably makes sense to have a more elaborate
 * resource management here, but it's not entirely clear what that would look
 * like.
 */
int logical_decoding_work_mem;
static const Size max_changes_in_memory = 4096; /* XXX for restore only */

When parsed data exceeds logical_decoding_work_mem, it’s written to disk. max_changes_in_memory is hardcoded at 4096, now only used to trigger disk restore. In pg12 source, there’s no int logical_decoding_work_mem, and subsequent serialization was also judged based on max_changes_in_memory.

In pg13, Disk serialization source code starts from line 2333. When parsed data in memory exceeds logical_decoding_work_mem, the largest transaction is spilled to disk. ReorderBufferLargestTXN(rb) finds the largest transaction. ReorderBufferSerializeTXN(rb, txn) persists this transaction. The immediately following code is ReorderBufferSerializeTXN():

/*
 * Spill data of a large transaction (and its subtransactions) to disk.
 */
static void
ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
dlist_iter subtxn_i;
dlist_mutable_iter change_i;
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;

elog(DEBUG2, "spill %u changes in XID %u to disk",
 (uint32) txn->nentries_mem, txn->xid);

/* do the same to all child TXs */
...

At debug2 level, spill logs are output:

/*
 * Given a replication slot, transaction ID and segment number, fill in the
 * corresponding spill file into 'path', which is a caller-owned buffer of size
 * at least MAXPGPATH.
 */
static void
ReorderBufferSerializedPath(char *path, ReplicationSlot *slot, TransactionId xid,
XLogSegNo segno)
{
XLogRecPtr recptr;

XLogSegNoOffsetToRecPtr(segno, 0, wal_segment_size, recptr);

snprintf(path, MAXPGPATH, "pg_replslot/%s/xid-%u-lsn-%X-%X.spill",
 NameStr(MyReplicationSlot->data.name),
 xid,
 (uint32) (recptr >> 32), (uint32) recptr);
}

Persisted to pg_replslot/replication_slot_name/xid-%u-lsn-%X-%X.spill.

Similarly, besides serialize, there’s also restore:

/*
 * Restore a number of changes spilled to disk back into memory.
 */
static Size
ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
TXNEntryFile *file, XLogSegNo *segno)
{
Size restored = 0;
XLogSegNo last_segno;
...
while (restored < max_changes_in_memory && *segno <= last_segno)
{
int readBytes;
ReorderBufferDiskChange *ondisk;

...
/*
 * Read the statically sized part of a change which has information
 * about the total size. If we couldn't read a record, we're at the
 * end of this file.
 */
ReorderBufferSerializeReserve(rb, sizeof(ReorderBufferDiskChange));
readBytes = FileRead(file->vfd, rb->outbuf,
 sizeof(ReorderBufferDiskChange),
 file->curOffset, WAIT_EVENT_REORDER_BUFFER_READ);
...
/*
 * ok, read a full change from disk, now restore it into proper
 * in-memory format
 */
ReorderBufferRestoreChange(rb, txn, rb->outbuf);
restored++;
}

return restored;
}

ReorderBufferRestoreChanges() just does judgment and looping (restored++), calling ReorderBufferRestoreChange():

static void
ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
 char *data)
{
...
/*
 * Update memory accounting for the restored change. We need to do this
 * although we don't check the memory limit when restoring the changes in
 * this branch (we only do that when initially queueing the changes after
 * decoding), because we will release the changes later, and that will
 * update the accounting too (subtracting the size from the counters). And
 * we don't want to underflow there.
 */
ReorderBufferChangeMemoryUpdate(rb, change, true,
ReorderBufferChangeSize(change));
}

Looking at ReorderBufferRestoreChanges(), its while loop judgment is restored < max_changes_in_memory, and restored starts at 0. It will loop 4096 times. There’s a comment in ReorderBufferRestoreChange explaining that although restore isn’t based on memory limit, it still needs to update memory usage to prevent underflow. Meaning: since I just restored it, don’t spill it again in a nested fashion. (It feels a bit odd — clearly judging by memory limit would be better rather than hardcoding the restore loop count.)

Interpreting the logical decoding process based on source code:

xtransaction snap preserves the metadata needed for parsing locks. When the replication slot is inactive or the transaction is uncommitted, snap persists to pg_logical/snapshots/%restart_lsn.snap. After the replication slot restarts or the transaction commits, the transaction snap metadata on disk is read into memory and sent to reorderbuffer for WAL parsing, sorted by transaction start order. If logical decoding data fills up the logical_decoding_work_mem memory area, change entries persist the largest transaction to pg_replslot/slot_name/xid-%u-lsn-%X-%X.spill, send other in-memory transactions to the output plugin for format conversion, and finally send the decoded information to the downstream.

In fact, we can see that long transactions and large transactions can make the entire logical replication link very slow. Large transactions are preferentially spilled to disk, then loaded back from disk to memory after the transaction completes.

Summary
#

Logical replication is managed through replication slots: one replication slot, one walsender process, one output plugin.
The output plugin determines the output form of logically decoded data, specified when creating the replication slot.
Replica identity priority recommendation: primary key -> non-null unique index -> full.
The publish-subscribe model is PostgreSQL’s built-in logical replication, using pgoutput by default. Publications can be used independently.
The publisher process is walsender, and the subscriber process is worker. Pay attention to their respective process parameters.
There are many third-party logical replication tools; they generally use PostgreSQL’s logical decoding system.
For monitoring replication links, pay attention to pg_replication_slots and pg_stat_replication.
The pg_logical directory stores transaction parsing metadata snaps, waiting for transaction commit before parsing.
The pg_replslot directory stores transaction information exceeding logical_decoding_work_mem, called spill.

References
#

Book: 《PostgreSQL实战》

Official Documentation:

PostgreSQL: Documentation: 15: Chapter 49. Logical Decoding

PostgreSQL: Documentation: 15: 49.1. Logical Decoding Examples

PostgreSQL: Documentation: 15: pg_recvlogical

PostgreSQL: Documentation: 14: 52.81. pg_replication_slots

PostgreSQL: Documentation: 13: 19.6. Replication

PostgreSQL: Documentation: 13: 48.6. Logical Decoding Output Plugins

PostgreSQL: Documentation: 15: 31.1. Publication

PostgreSQL: Documentation: 15: 31.2. Subscription

PostgreSQL: Documentation: 15: CREATE PUBLICATION

Highly Recommended:

https://www.pgconf.asia/JA/2017/wp-content/uploads/sites/2/2017/12/D2-A7-EN.pdf

Logical replication internals | Select * from Adrien

An Overview of Logical Replication in PostgreSQL - Highgo Software Inc.

Discussing Logical Decoding from Real Cases

Long-Troubling Logical Decoding Anomalies

Monitoring replication: pg_stat_replication - CYBERTEC

Other References:

https://zhuanlan.zhihu.com/p/311496301

A Guide to PostgreSQL Change Data Capture - DZone

Change data capture in Postgres: How to use logical decoding and wal2json - Microsoft Community Hub

PgSQL · The Secrets of PostgreSQL Logical Streaming Replication Technology · Database Kernel Monthly · KanCloud

Analyzing PostgreSQL Logical Replication Principles - CSDN Blog

http://pigsty.cc/zh/blog/2021/03/03/postgres逻辑复制详解/

Logical replication and logical decoding - Azure Database for PostgreSQL - Flexible Server | Microsoft Learn

PostgreSQL Streaming Replication

Tue, 13 Aug 2024 00:00:00 +0000

What is PostgreSQL Streaming Replication?
#

Streaming Replication is a method for transmitting WAL logs introduced in PostgreSQL 9.0. As soon as the primary database generates a log, it is immediately passed to the standby database. Before PostgreSQL 9.0, PostgreSQL could only transfer WAL logs one at a time (log shipping), and the standby database lagged behind the primary by at least one WAL log.

PostgreSQL Streaming Replication Processes
#

wal sender: The wal sender exists on the primary database. The wal sender process transmits the WAL between the primary’s latest LSN and the standby’s latest LSN to the standby. wal receiver: The wal receiver exists on the standby database. The wal receiver process transmits the standby’s latest LSN to the primary. The wal receiver receives WAL data passed by the wal sender and writes it to WAL logs. startup: The standby instance recovery process. It replays WAL logs on the standby database.

pg 16776 14632 0 13:33 ? 00:00:00 postgres: wal sender process lzl 172.17.100.150(13338) streaming 0/3002D30

pg 16775 15329 0 13:33 ? 00:00:00 postgres: wal receiver process streaming 0/3002D30
pg 15330 15329 0 10:26 ? 00:00:00 postgres: startup process recovering 000000010000000000000003

PostgreSQL Streaming Replication Principles
#

PostgreSQL streaming replication is primarily divided into two phases: the instance recovery phase and the primary-standby synchronization phase. Instance Recovery Phase: When a PostgreSQL database crashes abnormally, upon startup, PostgreSQL replays all WAL logs after the last checkpoint before the crash (this is the same principle as instance recovery in Oracle, MySQL, and other relational databases — the goal is to bring the database to a consistent state). When setting up a PostgreSQL standby database, the primary is generally not shut down. At this point, the backup taken from the primary is in an inconsistent state, and the startup process performs instance recovery when the standby starts. Primary-Standby Synchronization Phase: The wal receiver process transmits the standby’s latest LSN to the primary. The wal sender transmits the WAL between the primary’s latest LSN and the standby’s latest LSN to the wal receiver. The wal receiver receives the WAL and writes it to disk, and the startup process replays the WAL logs on the standby.

Synchronous and Asynchronous
#

PostgreSQL primary-standby has 5 modes, controlled by the synchronous_commit parameter. The essence of the synchronous_commit parameter is to control when the primary commits. remote_apply: The primary commits only after all standby databases have applied the WAL. This mode is synchronous — the primary and standby are consistent. Data that can be queried on the primary can definitely also be queried on the standby. In this mode there is no primary-standby lag, but it affects the primary commit time because the primary commit needs to wait for network transmission and standby application time.

The meaning of synchronous_commit has two scenarios: with and without standby databases (when synchronous_standby_names is empty or non-empty):

When synchronous_standby_names is non-empty: remote_apply: The standby has applied the WAL, only then can the primary commit. In this mode the primary and standby are synchronous. on: default. The primary commits when both primary and standby WAL have been written to disk. Similar to semi-synchronous, no data will be lost. remote_write: The primary commits when the standby has received the WAL and written the WAL log to the filesystem cache. At this point the standby has received the WAL but hasn’t flushed it to disk yet. If the OS crashes, data will be lost. local: The primary commits when its WAL is flushed to disk. This mode is asynchronous — the primary doesn’t need to confirm the standby’s status before committing. off: The primary can commit without its own WAL being flushed to disk. There is a risk of data loss. Not recommended.

When synchronous_standby_names is empty: (When synchronous_standby_names is empty, only on and off are effective for synchronous_commit. If set to remote_apply, remote_write, or local, they are still treated as on.) on: default. The database WAL must be written to disk before a transaction can commit. off: The primary can commit without its own WAL being flushed to disk. There is a risk of data loss. Not recommended.

Primary-Standby Synchronization Relationship

Primary-Standby Reliability

Failover
#

When the primary crashes, the standby needs to initiate failover, at which point the standby becomes the new primary. PostgreSQL does not provide a method to detect failures, but it does provide a method to activate the primary. (Typically, third-party tools call the PostgreSQL activation method, while primary-standby monitoring, primary crash detection, connection switching, etc. are not handled by PostgreSQL itself.) PostgreSQL provides 2 methods to activate a standby as the primary: the trigger_file file and the pg_ctl promote command. (In PostgreSQL 12 and later, trigger_file becomes promote_trigger_file.) Both trigger_file and pg_ctl promote can complete the task of activating the standby with a single command. The difference is that trigger_file requires the trigger_file configuration to be written in recovery.conf in advance. Using trigger_file for primary-standby switchover (pg_ctl promote has the same effect and is simpler):

Configure trigger_file in the standby’s recovery.conf
Shut down the primary
touch trigger_file to start the old standby as the new primary
Configure recovery.conf to start the old primary as the new standby
Observe the new and old primary/standby databases

Failover Example: Environment: Primary 172.17.100.150 5432 Standby 172.17.100.150 5433

1. Configure trigger_file in standby recovery.conf

$ cat recovery.conf|grep trigger
trigger_file = '/pg/pg96data_sla/trigger.kenyon'
$ ll /pg/pg96data_sla/trigger.kenyon
ls: cannot access /pg/pg96data_sla/trigger.kenyon: No such file or directory

Simply configure the trigger file path in recovery.conf. The trigger file won’t appear until it’s created.

Add configuration to standby postgres.conf

max_wal_senders = 6 #max_wal_senders is the maximum number of sender processes, default is 0, so the standby must configure this before switchover
hot_standby=on #Enable query functionality on standby

2. Shut down the primary

$ pg_ctl stop -D /pg/pg96data_pri -m fast
waiting for server to shut down.... done
server stopped

(Check if primary WAL has been fully applied by the standby: pg9.6- cd pg_xlog; pg 10+ cd pg_wal)

ls -ltr|tail -n 1 |awk '{print $NF}'|while read xlog;do pg_xlogdump $xlog;done

Look for the keyword “shutdown” in the standby’s WAL

3. touch to activate standby (or pg_ctl promote -D /pg/pg96data_sla)

$ touch /pg/pg96data_sla/trigger.kenyon

At this point recovery.conf becomes recovery.done

4. Set up primary as standby Configure the new standby’s recovery.conf file. You can directly copy from the old standby and modify the IP and directory.

vi $新备库/recover.conf
standby_mode = on
primary_conninfo = 'host=172.17.100.150 port=5433 user=lzl password=lzl'
recovery_target_timeline = 'latest'

Configure postgres.conf, write hot_standby = on to enable queries on the standby

vi $新备库/postgres.conf
hot_standby = on

Start the new standby

/pg/pg96/bin/pg_ctl -D /pg/pg96data_pri -l /pg/pg96data_pri/server.log start

5. Check primary and standby

postgres=# \x
Expanded display is on.
postgres=# select * from pg_stat_replication ;
-[ RECORD 1 ]----+------------------------------
pid | 24766
usesysid | 16384
usename | lzl
application_name | walreceiver
client_addr | 172.17.100.150
client_hostname | 
client_port | 47345
backend_start | 2021-07-30 07:44:05.582546+00
backend_xmin | 
state | streaming
sent_location | 0/4033790
write_location | 0/4033790
flush_location | 0/4033790
replay_location | 0/4033790
sync_priority | 0
sync_state | async

pg_basebackup
#

pg_basebackup is PostgreSQL’s built-in backup tool for performing base backups. pg_basebackup can be used for PITR and also for constructing log-shipping standby and streaming standby. It is PostgreSQL’s physical backup tool. https://liuzhilong.blog.csdn.net/article/details/119533506

pg_rewind
#

pg_rewind can be used as a maintenance tool for PostgreSQL primary-standby setups. When the timelines of two PostgreSQL instances diverge, pg_rewind can synchronize between the instances. (For example, if the standby is running after failover while the primary was still running, the timelines of primary and standby will have diverged.) https://liuzhilong.blog.csdn.net/article/details/119250794

Replication Slots
#

What are PostgreSQL Replication Slots? In a primary-standby architecture, if the standby hasn’t received WAL logs yet but the primary has already deleted them, such lag cannot be automatically recovered. Replication slots ensure that the primary won’t delete WAL logs that haven’t been transmitted to the standby yet. Without replication slots, you might need to use wal_keep_size/wal_keep_segments and archive_command to ensure WAL logs aren’t deleted, but this approach always retains too many WAL files and cannot guarantee that WAL won’t be deleted when lag is significant. This is exactly why replication slots were created. However, replication slots may cause the primary to never delete WAL (e.g., if the standby has crashed), causing disk space to fill up. In this case, max_slot_wal_keep_size is needed to set an upper limit on WAL file retention.

Replication Slot Parameters: max_slot_wal_keep_size: When replication slots are in use, this parameter defines the maximum size of WAL files in the pg_wal directory. The default value is -1, meaning there is no upper limit on the size of WAL files retained by the primary for the standby. wal_keep_segments/wal_keep_size: PostgreSQL 12 and below use wal_keep_segments, PostgreSQL 13 and above use wal_keep_size. Ensures that WAL files under pg_wal are not deleted. Without replication slots, WAL files exceeding this size may be deleted, potentially causing the standby to be unable to catch up. If set too large, it may cause the directory to grow excessively. The default is 0, meaning WAL files are not retained. If WAL is deleted, the following error may occur: ERROR: requested WAL segment xxxx has already been removed At this point the standby can only hope for archives; otherwise, it must be rebuilt. primary_slot_name: Sets the slot name, indicating that the PostgreSQL primary-standby setup uses replication slots. So enabling PostgreSQL replication slots requires at least the following configuration: primary_conninfo = ‘host=172.17.100.150 port=5433 user=lzl password=lzl’ primary_slot_name = ‘pg_slot_lzl’ max_replication_slots: The maximum number of replication slots. Takes effect upon restart. If there aren’t enough replication slots, the standby will fail to start. This value should be set relatively high. In PostgreSQL versions below 9.6, the default is 0; in PostgreSQL 10 and above, it’s 10.

Creating PostgreSQL Replication Slots

1. Set max_replication_slots on the primary Primary: (my PostgreSQL version is 9.6) max_replication_slots=10 Add to postgres.conf and restart the primary

2. Create replication slot Create replication slot:

postgres=# SELECT * FROM pg_create_physical_replication_slot('pg_slot_lzl');
 slot_name | xlog_position 
-------------+---------------
 pg_slot_lzl |

View replication slot

postgres=# SELECT slot_name, slot_type, active FROM pg_replication_slots;
 slot_name | slot_type | active 
-------------+-----------+--------
 pg_slot_lzl | physical | f

3. Set primary_slot_name on the standby primary_slot_name = 'pg_slot_lzl' Add to recovery.conf and restart the standby

4. Check replication slot

postgres=# select *,pg_xlogfile_name(restart_lsn)as current_xxlog from pg_replication_slots;
 slot_name | plugin | slot_type | datoid | database | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn | current_xxlog 
-------------+--------+-----------+--------+----------+--------+------------+------+--------------+-------------+---------------------+--------------------------
 pg_slot_lzl | | physical | | | t | 12802 | | | 0/A002340 | | 00000002000000000000000A
--pg_xlogfile_name(restart_lsn) to view current WAL log info

Query Conflicts
#

What are Query Conflicts? The standby may encounter the following error during queries: ERROR：canceling statement due to conflict with recovery

Why do conflicts occur? Let’s think carefully. For example, if the standby is executing a query based on a certain table (this query could be from an application or a manual connection), and the primary executes a drop table operation, this operation is written to WAL logs and transmitted to the standby for application. To ensure data consistency, PostgreSQL will inevitably replay the data quickly, at which point the drop table and select will conflict, as shown below:

Conflict scenarios: The above only introduces one type of query conflict. To summarize, there are several situations:

Primary exclusive locks (including explicit LOCK commands and various DDL operations)
Primary vacuum cleaning up dead tuples — if the standby is using those tuples, a conflict will occur
Primary drops the tablespace that the standby query is using
Primary drops the database that the standby is using

Consider a primary-only scenario: Scenario 1: A session issues a drop table and finds that a select statement is currently executing. The session can only wait for the select to complete its transaction. Scenario 2: A session issues a vacuum or automatic background vacuum — it won’t conflict with current database queries because vacuum won’t clean up tuples that are in use.

The standby’s handling is different. Because the primary doesn’t know the standby’s transaction status, and the standby needs to stay consistent with the primary, this is why “query conflicts” occur.

Query Conflict Parameters hot_standby_feedback: This is the most frequently mentioned parameter in the topic of query conflicts. Let’s explore it in detail below. Suppose, without a standby, Session 1 queries a row of data, Session 2 deletes that data and commits. Then Session 2 performs a vacuum. We know this vacuum won’t delete that row because Session 1’s transaction still needs to use that tuple, so it won’t be cleaned up. What about in a primary-standby setup? How does the primary know that the standby is still querying when it’s about to perform a vacuum? This is the purpose of this parameter. After setting hot_standby_feedback, the standby will periodically notify the primary of the minimum active transaction ID (xmin) value, so the primary vacuum process won’t clean up tuples with values greater than xmin. This parameter helps reduce conflicts but cannot completely avoid them. If you think about it carefully, this parameter only reduces conflicts caused by the primary vacuuming dead tuples — it cannot resolve conflicts caused by exclusive locks. Or conflicts caused by network interruptions: if the network between primary and standby is interrupted, the standby cannot send the xmin value to the primary normally. If the interruption is long enough, the primary will still clean up useless tuples during this period, and after the network recovers, the vacuum conflict described above may occur. It’s worth noting that the hot_standby_feedback parameter won’t override the value limited by the old_snapshot_threshold parameter on the primary. The old_snapshot_threshold parameter limits the infinite expansion of dead tuples. When transaction information exceeds the old_snapshot_threshold limit, cleanup will still occur.

max_standby_streaming_delay: The waiting time before the standby cancels a query due to a conflict caused by receiving WAL stream logs. Setting this parameter means that when a conflict occurs, the standby query won’t be immediately canceled but will wait for a period before throwing an error if it hasn’t finished. The value can be set based on the expected runtime of potential long transactions on the standby.

max_standby_archive_delay: The waiting time before the standby cancels a query due to a conflict caused by processing archived WAL logs. Similar to the parameter above.

vacuum_defer_cleanup_age: Specifies the number of transactions by which vacuum delays cleaning up dead tuples. Vacuum will delay clearing invalid records. The number of deferred transactions is set through vacuum_defer_cleanup_age. That is, vacuum and vacuum full operations won’t immediately clean up recently deleted tuples.

You can view conflict occurrences through the pg_stat_database and pg_stat_database_conflicts views.

Other Related Parameters
#

Transmission Parameters max_wal_senders: The maximum number of services that can fetch WAL using wal sender, i.e., the maximum number of standby databases + basebackup clients. PostgreSQL 9.6 defaults to 0; PostgreSQL 10 and later default to 10. wal_send_timeout: Interrupt replication after WAL transmission fails for xx seconds. When the standby crashes or the network is interrupted for a long time, WAL will no longer attempt transmission. Default is 60. 0 means never interrupt replication. track_commit_timestamp: Record transaction timestamps. Default is off.

Primary Parameters synchronous_standby_names: Configured on the primary. The standby replication list. There are several forms (s1, s2, s3 represent the standby’s application_name, configured in recovery.conf): synchronous_standby_names=‘s1’ means the primary can commit when s1 standby returns. synchronous_standby_names=‘FIRST 2 (s1,s2,s3)’ means the primary can commit when the first two of the three standbys (s1 and s2) return. synchronous_standby_names=‘ANY 2 (s1,s2,s3)’ means the primary can commit when any two of the three standbys return. synchronous_standby_names=’’ means matching any host — the primary can commit when any host returns. wal_level: WAL log level. This parameter determines how much information is written to WAL logs. The default is replica, which supports replication and WAL archiving while also supporting standby read-only queries. minimal: Other than records needed for instance crash recovery, nothing else is recorded. For example, CREATE TABLE AS, CREATE INDEX, CLUSTER, COPY can be skipped. The log information recorded in this mode is insufficient to support WAL archiving and streaming replication. logical: Adds additional information on top of replica to support logical decoding. This mode increases WAL log volume, especially for databases with many UPDATE and DELETE operations. Before PostgreSQL 9.6, there were also archive and hot_standby modes, which map to the current replica mode. synchronous_commit: As discussed earlier, 5 modes, each with pros and cons. archive_mode: archive_mode = on enables archiving. archive_command: Archiving command. PostgreSQL archiving directly calls operating system commands. Can be a simple cp command to the backup side. listen_addresses: Listening addresses. ‘’ means listen on all IPs. Default is local.

Standby Parameters hot_standby: on enables standby read-only queries. primary_conninfo: The connection string for the standby to connect to the primary. E.g., primary_conninfo = ‘host=172.17.100.150 port=5432 user=lzl password=lzl’. trigger_file/promote_trigger_file: The trigger file for activating the standby. Before PostgreSQL 12 it’s called trigger_file; PostgreSQL 12 and later use promote_trigger_file. Both trigger_file and pg_ctl promote can activate the standby with a single command, as demonstrated earlier. wal_receiver_create_temp_slot: When there is no slot, temporarily create one (named after primary_slot_name). Default is off.

References:
#

《The Way of PostgreSQL》(修炼之道) https://www.postgresql.org/docs/current/warm-standby.html

https://www.postgresql.org/docs/13/high-availability.html

https://www.postgresql.org/docs/current/runtime-config-replication.html

https://www.postgresql.org/docs/13/runtime-config-wal.html

https://www.postgresql.org/docs/current/app-pgbasebackup.html

https://www.postgresql.org/docs/current/hot-standby.html#HOT-STANDBY-CONFLICT

https://cloud.tencent.com/developer/article/1555354

https://www.modb.pro/db/29737

https://wiki.postgresql.org/wiki/Streaming_Replication

https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

https://www.cybertec-postgresql.com/en/the-synchronous_commit-parameter/

https://blog.csdn.net/m15217321304/article/details/88850146

https://blog.51cto.com/lishiyan/2460518?source=dra

A Brief Analysis of Linux Memory

Mon, 12 Aug 2024 00:00:00 +0000

Basic Memory Concepts
#

Operating system memory is very important and fairly complex. Many knowledge points need to be mastered to further analyze program issues. Since this is the first comprehensive and systematic exposure to OS memory, the goal is to understand Linux memory concepts thoroughly and at a low level without diving deep into principles, so this chapter will also try to avoid Linux source code knowledge.

Physical Memory and Virtual Memory
#

(https://en.wikipedia.org/wiki/Memory_address)

Physical Memory: Physical memory is the actual hardware memory present in a computer system, typically in the form of RAM (Random Access Memory).

Virtual Memory: Virtual memory is a linear region that has not been allocated actual physical memory. Programs think they have a larger address space than the actual physical memory. The implementation of virtual memory allows programs to access a larger address range than physical memory without requiring all data to be present in physical memory simultaneously. The kernel releases physical pages by releasing linear regions, finding the corresponding physical pages, and releasing them all.

Memory Management Unit (MMU): A hardware component responsible for converting virtual addresses used by programs into physical addresses where data is actually stored in physical memory. The MMU’s primary task is to perform address mapping.

Page Table: A page table is a data structure used to store the mapping between virtual address space and physical address space. When a program attempts to access virtual memory, the MMU determines the corresponding physical address by querying the page table.

System call flow: https://users.cs.utah.edu/~aburtsev/cs5460/lectures/lecture19-memory-management/lecture19-memory-management.pdf

(The image is a bit blurry, the topmost text is “User Space|Kernel Space”)

User programs can only access the kernel system through C libraries or system calls; user programs cannot directly access the kernel system
The kernel system accesses physical memory through the MMU; it accesses disks and other external devices through drivers
The virtual memory system (VM Subsystem in the figure above) includes buddy, slab algorithms, etc.

User Space and Kernel Space
#

The process virtual address space is divided into user space and kernel space.

User Space:

The space where user processes run in memory
This portion of space is protected, and the system prevents other processes from accessing it (except for shared memory)
However, kernel processes can directly access user processes

Kernel Space:

Kernel space is the space used by kernel processes
In kernel space, the operating system’s kernel code runs with higher privilege levels, allowing direct access to system hardware, process management, file system operations, etc.

Context Switching:

When a user program needs to access system services or perform operations requiring higher privileges, a context switch from user space to kernel space is triggered.
Context switching is an operating system mechanism for saving and restoring program state, ensuring no data loss occurs when switching between user programs and the kernel.

The division between user space and kernel space is to provide security isolation, preventing user programs from directly affecting critical parts of the operating system. Early operating systems and DOS did not distinguish between kernel and user space, so a single program’s error or malicious behavior could affect the entire system.

(https://www.zhihu.com/tardis/zm/art/66794639?source_id=1003)

32-bit systems: Total 4GB address space, 3G UserSpace | 1G KernelSpace

64-bit systems: Total 256TB address space, 128T UserSpace | 128T KernelSpace

2^32=4GB, 2^64=16777216TB, why does a 64-bit system only have 256TB address space?

The 64-bit computing wiki has an explanation. In short, 256TB (256 × 1024^4 bytes) of memory addresses is sufficient, and currently and in the imaginable future there won’t be 16EB (16 × 1024^6 bytes) of memory.

Process Virtual Address Space
#

Each process typically has its own independent virtual memory space. Virtual memory is an abstract concept that provides each running process with an address space that appears continuous and private, making each process feel like it has the entire computer system’s full memory.

Process virtual address space layout:

(https://www.sohu.com/a/392831824_467784)

The mmap mapping region expands from top to bottom, and the mmap mapping region and heap expand relative to each other until the remaining area in the virtual address space is exhausted. This structure facilitates the C runtime library’s use of the mmap mapping region and heap for memory allocation.
Stack: Stores local variables and function parameters during program execution, growing from high addresses to low addresses
Heap: Dynamic memory allocation area, managed through functions like malloc, new, free, and delete
BSS (Uninitialized Variables): Stores uninitialized global variables and static variables
Data: Stores global variables and static variables with predefined values in source code
Text: Stores read-only program execution code, i.e., machine instructions

Process virtual address space distribution and mapping:

(https://velog.io/@mysprtlty/%EA%B0%80%EC%83%81-%EB%A9%94%EB%AA%A8%EB%A6%AC%EC%99%80-%EA%B0%80%EC%83%81-%EC%A3%BC%EC%86%8C-%EA%B3%B5%EA%B0%84)

Shared Memory
#

As mentioned earlier, the user space in the virtual address space cannot be accessed by other user processes. If multi-process user access to the same memory data is implemented through the kernel area, context switching cannot be avoided. Multi-process applications clearly need inter-process access, so a method that directly allows user processes to access the same physical memory emerged — this is shared memory.

Shared memory is one of the mechanisms for implementing IPC (Inter Process Communication), with other methods including message queues and semaphores.

(https://www.geeksforgeeks.org/inter-process-communication-ipc/)

Since it is inherently multiple virtual memory address spaces corresponding to one physical memory address space, you just need to point a segment in the address spaces of two processes to the same physical memory.

(https://www.softprayog.in/programming/interprocess-communication-using-system-v-shared-memory-in-linux)

Shared memory (seems like) has many implementation methods. For example, PostgreSQL defaults to using mmap to implement shared memory, refer to the shared_memory_type parameter and Managing Kernel Resources. Other shared memory implementations can be found in this article: Song Baohua: The Best Shared Memory in the World (The Most Thorough Linux Shared Memory Article)

Page Table
#

The process virtual address space is per-process, while there is only one physical memory space. So how do you map and convert virtual memory and shared memory?

(https://courses.engr.illinois.edu/cs241/sp2014/lecture/09-VirtualMemory_II_sol.pdf)

The page table is where the correspondence between virtual memory addresses and physical memory addresses is stored. (There are concepts like MMU and TLB here — let’s simplify and just think of it as the virtual-to-physical memory conversion function (PAGING), and only look at the page table here). A page table consists of a set of Page Table Entries (PTEs), with each PTE storing the map between a virtual page and a physical page.

Although a single page table can implement memory-to-virtual-memory conversion, implementing it directly this way would consume too much memory for the page table itself.

(https://courses.engr.illinois.edu/cs241/sp2014/lecture/09-VirtualMemory_II_sol.pdf)

Therefore, the single page table needs to be subdivided: two-level page tables and four-level page tables.

Two-level page tables:

A two-level page table is a further subdivision of a single page table. 4G of space requires 4M of page tables to store the mapping table. If these 4M are divided into 1K pages (4K each), these 1K pages also need a table for management, which we call the page directory table. This page directory table has 1K entries, each 4 bytes, making the page directory table size 4K as well.

Four-level page tables:

For 64-bit systems, two-level page tables are insufficient; four-level page tables are needed.

(https://maodanp.github.io/2019/06/02/linux-virtual-space/)

Check page table size:

[pg@lzl 2345]$ cat /proc/meminfo |grep PageTables
PageTables: 46736 kB

NUMA
#

Uniform Memory Access (UMA): All CPUs have equivalent access time to memory. The problem with UMA is that multiple processors access memory through a single bus, increasing the load on the shared bus. Multiple processors contend for the memory controller causing conflicts. Additionally, the bus bandwidth is limited, leading to access delays.

Non-Uniform Memory Access (NUMA): A small group of CPUs access their own local memory together. When there are multiple groups of CPUs and their memory groups, each group of CPUs and memory constitutes a NUMA node.

UMA:

NUMA:

(https://users.cs.utah.edu/~aburtsev/cs5460/lectures/lecture19-memory-management/lecture19-memory-management.pdf)

Basic NUMA characteristics:

CPU access to local node memory is faster than remote
By default, Linux prioritizes allocating local memory on the CPU; the policy can be configured
Each node has its own memory structure
NUMA is not suitable for all scenarios; it requires adaptation by upper-layer applications

NUMA balancing: Achieves local access by automatically transferring tasks to remote CPUs or copying remote data to local memory. Enabled by default on Red Hat 7.

Transferring tasks or copying data itself consumes resources and can slow down tasks. This feature may not be suitable for some applications; for example, Oracle’s Exadata has targeted NUMA optimizations.

numactl: NUMA OS configuration tool.

numactl --show displays CPU and node information. Below is an example of 4 nodes with 64c 256g total, each node having 16c 64g:

available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
node 0 size: 65418 MB
node 0 free: 310 MB
node 1 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47
node 1 size: 65536 MB
node 1 free: 41 MB
node 2 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55
node 2 size: 65536 MB
node 2 free: 82 MB
node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63
node 3 size: 65536 MB
node 3 free: 43 MB

Zone
#

NUMA divides CPUs and memory into multiple nodes (node 0, node 1, node 2…). In UMA structures, the CPU memory as a whole can be viewed as node 0.

In Linux, each node is represented by the data structure struct pglist_data, with the data type typedef pg_data_t. Each node is further divided into multiple zones. A zone’s data structure is zone_t, with the data type zone_struct. There are generally 3 types: ZONE_DMA, ZONE_NORMAL, ZONE_HIGHMEM, each with different functions.

(https://www.kernel.org/doc/gorman/html/understand/understand005.html)

Zone distribution and functions in 32-bit:

ZONE_DMA: (<16MB), Direct Memory Access (DMA), the ancient 16 MiB limit, includes ISA devices.
ZONE_DMA32: Since many devices encounter problems accessing memory that cannot be addressed with 32 bits, this zone was added in x86-64. This zone only exists in x86-64 architecture. (See ZONE_DMA32)
ZONE_NORMAL: (16MB to 896MB), ordinary memory domain that can be directly mapped to the kernel segment; most kernel operations take place in the NORMAL zone, this is the most important zone
ZONE_HIGHMEM: (>896MB), marks physical memory beyond the kernel segment, cannot be directly called by the kernel.

Zone distribution diagram for 32-bit and 64-bit:

https://users.cs.utah.edu/~aburtsev/cs5460/lectures/lecture19-memory-management/lecture19-memory-management.pdf

Note that zones are for physical memory. Virtual memory must switch from user mode to kernel mode before it can call physical memory. The following diagram shows the relationship between kernel addresses in virtual memory space and zones in physical address space:

(https://wr.informatik.uni-hamburg.de/_media/teaching/wintersemester_2014_2015/kp-1415-memory-management.pdf)

Inspect zones:

cat /proc/zoneinfo

cat /proc/buddyinfo

cat /proc/pagetypeinfo

$ cat /proc/buddyinfo 
Node 0, zone DMA 1 1 1 1 1 0 1 1 1 1 2 
Node 0, zone DMA32 688 2080 1420 995 596 357 278 241 276 32 133 
Node 0, zone Normal 195748 204074 161167 119070 70791 33578 9556 2070 1034 2533 7328 
Node 1, zone Normal 11705 51467 36752 21326 11343 7309 5024 3403 2597 3056 10898

Pages
#

Virtual memory and physical memory are divided into fixed-size segments, typically 4KB in size. So after virtual memory is divided, we have virtual pages, and after physical memory is divided, we have physical pages (PP or PF, Physical Page or Page Frame), also called page frames, also 4KB. The page frame represents the minimum unit of system memory.

Each page in the virtual address space can be mapped to a page frame in the physical address space through its descriptor.

Huge Pages / Transparent Huge Pages
#

Pages are the minimum unit of memory allocation (default 4K). When mapping and allocating a large number of contiguous pages, performance is poor. Huge Pages solve this problem. Huge pages are not only cheaper to allocate, but the page table is also relatively smaller. hugepagesz is 2 MB or 1 GB, defaulting to 2MB. Huge Pages were implemented starting from Red Hat 6.

Since manually managing huge pages is cumbersome, Red Hat 6 also provided automatic huge page management, i.e., Transparent Huge Pages.

In Oracle database management, huge pages are generally enabled for SGA use, while transparent huge pages are disabled. There is plenty of related material available for searching.

Similarly, PostgreSQL can also enable huge pages. Since databases generally occupy more operating system memory, enabling huge pages for databases can generally reduce memory allocation pressure.

File Pages & Page Cache / Anonymous Pages & Swap Cache
#

File pages can be mapped to files on disk. File system reads and writes use Page Cache as buffered IO. Dirty data is synced (or fsynced, etc.) to the corresponding disk periodically or when called. Page Cache is the memory area used to “boost” disk performance.

Correspondingly, pages without associated files are called Anonymous Pages, generally corresponding to heap and stack. When memory resources are tight, the kernel writes infrequently used anonymous page data to swap partitions or swap files.

In short:

Page cache corresponds to file mappings
Swap cache corresponds to anonymous pages

(https://www.slideshare.net/raghusiddarth/memory-management-in-linux-11551521?from_search=2)

The above page cache diagram is from the operating system’s perspective. Application (such as database) writes can also be non-delayed, or even bypass Page Cache.

Memory Allocation
#

Memory allocation is also very complex, involving many concepts. Two common memory allocation methods are buddy and slab.

Buddy
#

The buddy system is used for allocating contiguous memory pages. Each zone has its own buddy system. The buddy system divides large blocks of memory to respond to memory allocation requests, and due to its coalescing characteristics, it can reduce system memory fragmentation.

The buddy allocator divides memory into pages of powers of 2, with the maximum order being 10:

When a memory request is larger than the existing block size, the system splits the larger block into two equally sized buddy blocks. When memory is freed, the system attempts to merge adjacent buddy blocks into a larger block:

When freeing a page, the page is directly placed back into the free list. If the other half of the previously split page is also unallocated, they are combined into a double-sized page and given to the next larger list, and so on, until it can no longer be merged or has reached the top.

When higher-order pages are depleted due to continuous allocation, fragmentation issues arise when requesting higher-order pages:

After waiting for memory reclamation to succeed, buddy itself merges lower orders into higher orders, then allocates higher-order pages:

(The implementations of anti pages fragmentation in Linux kernel https://teawater.github.io/presentation/antif.pdf)

However, memory reclamation may also not keep up with allocation speed, so the buddy system is not always ideal.

Analysis example:

$ cat /proc/buddyinfo 
Node 0, zone DMA 0 0 0 1 2 1 1 0 1 1 3 
Node 0, zone DMA32 7 6 5 6 5 6 7 7 6 2 272 
Node 0, zone Normal 317681 38869 31620 19250 8931 2579 815 182 19 5 0 

The above contains 3 ZONEs: DMA, DMA32, Normal
Orders: 0 ~ 10, i.e., the count of each order in buddy. The maximum order of buddy is 10, i.e., 1024 pages, which is 4MB
For example, the 3rd column in the Normal row indicates there are 31620 blocks of 2^2 contiguous memory available
By extension, the further back, the more contiguous the space. The larger the number, the more contiguous space of that size there is. When large contiguous spaces are scarce, it indicates significant memory fragmentation
Additionally, summing everything up gives the current free memory

Judging memory fragmentation issues through buddyinfo:

#host 1
Node 0, zone Normal 317681 38869 31620 19250 8931 2579 815 182 19 5 0 
#host 2
Node 0, zone Normal 7321 7833 10885 8514 2311 1644 1663 1302 1141 7384 80675 

The above shows the memory conditions of two hosts. Comparing them, the host below has more contiguous memory, while the host above has memory fragmentation issues.

Slab
#

The slab allocator manages memory based on objects. The slab system is a memory allocation algorithm specifically designed for kernel memory. It works by dividing memory into fixed-size caches, where each slab contains a set of objects of the same type. When there is a memory request, the algorithm first checks if available objects exist in the appropriate slab cache. If they exist, the object is returned. If not, the algorithm allocates a new slab and adds it to the appropriate cache.

Objects of different sizes correspond to different slab caches:

(https://bootlin.com/doc/training/linux-kernel/linux-kernel-slides.pdf)

Although slab has different caches and objects, slab still uses physically contiguous memory:

(https://i.stack.imgur.com/wo8Gg.png)

Slab also has 3 implementation methods:

Memory Reclamation
#

Memory Reclamation Overview
#

When system memory pressure is high, memory reclamation is performed on each zone under pressure. Memory reclamation mainly targets anonymous pages and file pages.
For anonymous pages, during memory reclamation, some infrequently used anonymous pages are selected, written to the swap partition, and then released as free page frames to the buddy system.
For file pages, during memory reclamation, some infrequently used file pages are also selected:
- If the content saved in this file page is consistent with the corresponding file content on disk, this file page is a clean file page and does not need to be written back; it is directly released as a free page frame to the buddy system.
- If the data saved in the file page is inconsistent with the corresponding data in the file on disk, this file page is considered a dirty page. It must first be written back to the corresponding data location on disk, and then released as a free page frame to the buddy system.
After memory reclamation completes, the number of free page frames in the system increases, alleviating memory pressure. However, the reclamation process puts significant IO pressure on the system. Therefore, a threshold is set for each zone in the system. When the number of free page frames falls below this threshold, memory reclamation operations are performed. When the number of free page frames meets this threshold, the system does not perform memory reclamation operations.

Zone Watermarks and kswapd
#

(https://vivani.net/2022/06/14/linux-kernel-tuning-page-allocation-failure/)

When available memory is low, the kswapd daemon is awakened to free pages.

pages_low: When the number of available free pages falls below pages_low, the buddy allocator wakes up the kswapd process, and the kernel begins swapping pages out to disk.
pages_min: When the number of available pages reaches pages_min, the pressure of page reclamation work is relatively high because the memory zone urgently needs free pages. The allocator will execute kswapd work in a synchronous manner, sometimes called direct reclaim.
pages_high: Once kswapd is awakened and begins freeing pages, the kernel considers the zone “balanced” only when the number of available pages reaches pages_high. If the watermark reaches pages_high, kswapd will re-enter the sleep state. If free pages exceed pages_high, the kernel considers the zone state ideal.

Memory reclamation is performed on a per-zone basis. /proc/zoneinfo can display the values of min, low, and high.

vm.min_free_kbytes is the min_pages watermark, a very important OS parameter. Very low values prevent the system from effectively reclaiming memory, potentially leading to system crashes and service interruptions. Too high values increase system reclamation activity, causing allocation delays, which may lead the system to immediately enter an out-of-memory state.

Types of Memory Allocation and Reclamation
#

Fast Memory Allocation: Performed by the get_page_from_freelist() function, which obtains a suitable zone from the zonelist using the low threshold for allocation. If the zone has not reached the low threshold, fast memory reclamation is performed, and allocation is retried after fast memory reclamation.

Slow Memory Allocation: When fast allocation fails, meaning no zone in the zonelist obtained memory in fast allocation, the min threshold is used for slow allocation. During slow allocation, three main things happen: asynchronous memory compaction, direct memory reclamation, and light synchronous memory compaction. Finally, OOM allocation may occur depending on the situation. And after each of these operations, fast memory allocation is called once to attempt to obtain page frames.

(https://blog.csdn.net/weixin_35094083/article/details/116688112)

Different memory allocation paths trigger different memory reclamation methods. Zone memory reclamation is divided into two types:

Background Memory Reclamation (kswapd): When physical memory is tight, the kswapd kernel thread is awakened to reclaim memory. This memory reclamation process is asynchronous and does not block process execution.
Direct Memory Reclamation (direct reclaim): If background asynchronous reclamation cannot keep up with process memory application speed, direct reclamation begins. This memory reclamation process is synchronous and blocks process execution.

Memory Compaction
#

Memory compaction: see Memory Monitoring - /proc/pagetypeinfo section

LRU
#

For zone memory reclamation, it targets three things for reclamation: slab, pages in LRU lists, and buffer_head. Here we only discuss memory reclamation targeting LRU lists.

The main purpose of LRU lists is to sort pages, placing pages most deserving of reclamation at the back and pages least deserving of reclamation at the front. Then, during memory reclamation, scanning proceeds from back to front, attempting to reclaim scanned pages.

LRU list descriptor, containing 5 LRU lists: active/inactive anonymous page LRU lists, active/inactive file page LRU lists, and unevictable page list:

(https://lpc.events/event/11/contributions/896/attachments/793/1493/slides-r2.pdf)

For memory reclamation, it only processes the first 4 LRU lists: active anonymous page LRU list, inactive anonymous page LRU list, active file page LRU list, and inactive file page LRU list. After reclaiming enough page frames, it returns directly: fast memory reclamation and kswapd memory reclamation do this.

Global lruvec can be viewed through meminfo (understood as LRU areas):

## cat /proc/meminfo |grep -i active
Active: 597380 kB
Inactive: 601920 kB
Active(anon): 10896 kB
Inactive(anon): 117376 kB
Active(file): 586484 kB
Inactive(file): 484544 kB

In reality, there is more than one lruvec. cgroup and NUMA nodes each have their own lruvec, and global also has its own lruvec.

Drop Cache
#

Drop cache records which pages are caching file system data pages and writes data back to disk when pages are forcibly reclaimed, so they can be cached again on the next access.

Default value: vm.drop_caches = 0. By default, the Linux kernel does not automatically clear caches.
Setting /proc/sys/vm/drop_caches to 1: The kernel clears unused page cache.
Setting /proc/sys/vm/drop_caches to 2: The kernel releases memory used by dentry and inode. Dentry and inode are file system metadata structures used to store file and directory information.
Setting /proc/sys/vm/drop_caches to 3: Equivalent to 1+2, releases all unused caches.

When the kernel decides to reclaim certain caches, it checks whether the data in the cache is consistent with the data on disk. If the data is inconsistent, the kernel needs to write the data back to disk before reclaiming that cache. This process can cause IO spikes. When performing Drop Cache operations, it is recommended to avoid any important I/O operations as this may affect system performance.

Operation commands:

echo 3 > /proc/sys/vm/drop_caches # Flush cache
echo 0 > /proc/sys/vm/drop_caches # Restore default

Memory Monitoring
#

Without understanding basic memory knowledge, it is actually very difficult to interpret memory monitoring information. With the above memory fundamentals in place, let’s go through memory-related monitoring commands and tools one by one.

What’s in the /proc Directory?
#

/proc mainly contains process information and system information.

In the system information part, some are interfaces provided by Linux for system status, allowing you to view monitoring information at the entire operating system level, such as slabinfo, swaps, zoneinfo, buddyinfo.

The other part, process, contains running data and status information for each process. cd into the corresponding process directory to see the FDs held by the corresponding process and process memory information.

Processes also have threads. Thread information directory: /proc/[pid]/task/[tid]/, with content similar to the process directory.

For more proc information, refer to proc(5) — Linux manual page

/proc/meminfo
#

/proc/meminfo is the primary interface for understanding the current Linux system memory usage. The most commonly used commands like free, vmstat, ps obtain data through it. /proc/meminfo information is more comprehensive. Below we only list some common information. For detailed meanings, refer to the Red Hat documentation

# General memory information
cat /proc/meminfo | grep "Mem"
MemTotal: 994328 kB # Total memory size (minus some reserved and kernel)
MemFree: 66428 kB # Completely unused physical memory
MemAvailable: 207192 kB # Maximum available memory for starting a new application without using swap space

# IO buffers
cat /proc/meminfo | grep -e "Buffers" -we "Cached" 
Buffers: 12820 kB # IO buffers used by raw disk blocks, not exceeding 20MB
Cached: 254592 kB # Page cache size used by disks (includes tmpfs and shmem, excludes SwapCached)

# swap
cat /proc/meminfo | grep "Swap" 
SwapCached: 13936 kB # Swap cache contains anonymous memory pages determined to be swapped but not yet written to physical swap area
SwapTotal: 945416 kB # Total swap space size
SwapFree: 851064 kB # Remaining swap size

# lru active and inactive page counts (self-explanatory)
cat /proc/meminfo | grep -e "Active" -e "Inactive" 
Active: 194308 kB
Inactive: 553172 kB
Active(anon): 59024 kB
Inactive(anon): 437264 kB
Active(file): 135284 kB
Inactive(file): 115908 kB

# Dirty pages
cat /proc/meminfo | grep -e "Dirty" -e "Writeback" 
Dirty: 0 kB # Dirty pages not yet written
Writeback: 0 kB # Dirty pages being written
WritebackTmp: 0 kB # Temporary buffer for writebacks used by the FUSE module

# Map information
cat /proc/meminfo | grep -e "AnonPages" -e "Map"
AnonPages: 95296 kB # Mapped anonymous pages
Mapped: 153192 kB # Mapped file pages
DirectMap4k: 113336 kB # Mapped 4k kernel pages
DirectMap2M: 1900544 kB # Mapped 2M kernel pages
DirectMap1G: 0 kB # Mapped 1G kernel pages

# Shared memory
cat /proc/meminfo | grep "Shmem"
Shmem: 28920 kB # Total memory size of shmem and tmpfs
ShmemHugePages: 0 kB # Total huge page memory size of shmem and tmpfs
ShmemPmdMapped: 0 kB # Shared memory mapped into userspace with huge pages

# Kernel memory (note: slab is kernel)
cat /proc/meminfo | grep -ie "reclaim" -e "slab" -e "kernel"
KReclaimable: 35008 kB # Reclaimable memory allocated to kernel
Slab: 88752 kB # Slab cache
SReclaimable: 35008 kB # Reclaimable memory in slab cache
SUnreclaim: 53744 kB # Non-reclaimable memory in slab cache
KernelStack: 5988 kB # Kernel stack memory used by all tasks

# Allocatable memory (different meaning from MemAvailable)
## CommitLimit=[("total RAM pages" - "total huge TLB pages") * overcommit_ratio]/100 + "total swap pages"
## In short, MemAvailable watermark plus swap equals allocatable memory
cat /proc/meminfo | grep -ie "commit"
CommitLimit: 1442580 kB # Allocatable memory
Committed_AS: 3035924 kB # Estimated memory needed in current worst-case scenario

# Virtual memory
cat /proc/meminfo | grep -e "Vmalloc"
VmallocTotal: 34359738367 kB # Total allocated virtual memory size
VmallocUsed: 34780 kB # Total used virtual memory size
VmallocChunk: 0 kB # Largest contiguous virtual memory block

# Page table memory (self-explanatory)
cat /proc/meminfo | grep PageTables
PageTables: 4120 kB

# Huge page memory
cat /proc/meminfo | grep -i hugepage
AnonHugePages: 32768 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

/proc/buddyinfo
#

Due to its concise and easy-to-understand information, buddyinfo is the most commonly used method for judging memory fragmentation issues. See “Memory Allocation - Buddy section” for details.

$ cat /proc/buddyinfo 
Node 0, zone DMA 0 0 0 1 2 1 1 0 1 1 3 
Node 0, zone DMA32 7 6 5 6 5 6 7 7 6 2 272 
Node 0, zone Normal 317681 38869 31620 19250 8931 2579 815 182 19 5 0 

/proc/pagetypeinfo
#

pagetypeinfo first provides information about page block sizes. It provides the same type of information as buddyinfo but broken down by type and detailing the number of pages of each type.

Before understanding pagetypeinfo, you need to first understand memory compaction.

Suppose the memory in a zone looks like this:

White represents free memory, red represents used memory. The memory fragmentation above is already quite severe. If a request for memory of order 2 or higher is made at this point, it cannot be allocated. This is where memory compaction comes into play. The compaction algorithm marks movable pages and free pages lists on the existing zone.

The movable scanner scans from bottom to top, and the free scanner scans from top to bottom. The movable and free scanners will eventually meet at some point in the middle. Then, through page migration, used pages are moved to the top of the zone.

Two trigger methods for page compaction:

When allocating pages, if allocation fails at the LOW watermark, slow memory allocation is attempted, during which page compaction occurs
Page compaction can be started with echo x > /proc/sys/vm/compact_memory. After starting, the kernel thread kcompactd begins page defragmentation.

Because page data is migrated to new locations, there are no performance issues as severe as those caused by memory reclamation. Moreover, since the goal is clearer, the cost of obtaining contiguous pages is lower. Additionally, ANON page reclamation requires SWAP, while this does not.

Now let’s look at /proc/pagetypeinfo:

$ cat /proc/pagetypeinfo 
Page block order: 9
Pages per block: 512

... (DMA omitted)
Node 0, zone Normal, type Unmovable 870 530 391 157 103 41 9 2 1 0 0 
Node 0, zone Normal, type Movable 5886 9235 5728 4072 1561 324 115 41 12 4 13018 
Node 0, zone Normal, type Reclaimable 3 4 8 11 2 3 1 1 1 0 0 
Node 0, zone Normal, type HighAtomic 0 0 0 0 0 0 0 0 0 0 0 
Node 0, zone Normal, type CMA 0 0 0 0 0 0 0 0 0 0 0 
Node 0, zone Normal, type Isolate 0 0 0 0 0 0 0 0 0 0 0 

Different pages are classified as pageblocks. Each pageblock is divided into several lists based on its type. When allocating memory, pages are requested from the corresponding list based on the requested page type, and when freed, they return to the corresponding list based on their pageblock. Different pageblocks:

Unmovable: Pages that cannot be compacted
Movable: Pages that can be compacted
Reclaimable: Pages that can be reclaimed
HighAtomic: Pageblock added to mitigate fragmentation issues. Only higher-order and same-level requests can request pages from this pageblock
CMA: CMA stands for Contiguous Memory Allocator
Isolate: Pages will not be allocated; used to help isolate pages. When isolating pages, pageblocks are first set to isolate to prevent them from being freed

CMA appears to be another large topic, which can be simply understood as a supplement to the buddy system:

(Memory Journey — How to Improve CMA Utilization? https://ost.51cto.com/posts/10815)

smaps & maps & pmap
#

VSS/RSS/PSS/USS

When viewing the memory occupied by a process, there are commonly four forms: VSS/RSS/PSS/USS, mainly differing in memory calculation methodology.

(https://cloud.tencent.com/developer/article/1683708)

VSS (Virtual Set Size) is just a virtual space size, with little significance for actual memory usage.
RSS (Resident Set Size) is used for calculating the total memory occupied by a process, including shared memory size occupied by shared libraries. For example, if private memory size is N and shared memory size is M, then RSS = N + M. This can be misleading, because for large shared libraries like libc, shared by many processes, counting it all against one process is not scientific.
PSS (Proportional Set Size) is the actual physical memory occupied by a single process when running, including proportionally allocated shared library memory. If a shared library is used by N processes, the size proportionally allocated to PSS is 1/N. PSS calculates process memory more accurately, including exclusive memory plus the shared portion.
USS (Unique Set Size) is the physical memory exclusively occupied by a process, not including shared memory.

/proc/[pid]/maps

/proc/[pid]/maps can view the user space memory mappings of the process’s virtual memory.

[pg@lzl 2345]$ cat maps 
StartAddr-EndAddr Perms Offset Dev Inode Filename
00400000-00bae000 r-xp 00000000 fd:00 1093852 /pg/pg15.3/bin/postgres ---text segment
00dad000-00dc3000 rw-p 007ad000 fd:00 1093852 /pg/pg15.3/bin/postgres
00dc3000-00df5000 rw-p 00000000 00:00 0 
00f1e000-00f60000 rw-p 00000000 00:00 0 [heap] ---heap area
33a6000000-33a6022000 r-xp 00000000 fd:00 1976006 /lib64/ld-2.17.so
...
7fbe2ae09000-7fbe2ae0a000 rw-p 0000c000 fd:00 1975966 /lib64/libnss_files-2.17.so
7fbe2ae1b000-7fbe33ca7000 rw-s 00000000 00:04 12556 /dev/zero (deleted)
7fbe33ca7000-7fbe39b38000 r--p 00000000 fd:00 1181300 /usr/lib/locale/locale-archive
7fbe39b38000-7fbe39b3d000 rw-p 00000000 00:00 0 
7fbe39b46000-7fbe39b4d000 rw-s 00000000 00:10 12559 /dev/shm/PostgreSQL.3661351388
7fbe39b4d000-7fbe39b4e000 rw-s 00000000 00:04 32769 /SYSV0010c0b6 (deleted)
7fbe39b4e000-7fbe39b4f000 rw-p 00000000 00:00 0 
7fffe3933000-7fffe3948000 rw-p 00000000 00:00 0 [stack] --stack area
7fffe397d000-7fffe397e000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

(1) Start-End Address: The address range of this segment in virtual memory (2) Permissions: Permissions of this segment; r-read, w-write, x-execute, p-private (3) Offset: The offset of this segment mapping in the file (4) Device: The device number of the device where the mapped file resides, corresponding to vm_file->f_dentry->d_inode->i_sb->s_dev. Anonymous mappings have 0. fd is the major device number, 00 is the minor device number. (5) Inode: Corresponds to vm_file->f_dentry->d_inode->i_ino, matches the content displayed by ls -i, anonymous mappings have 0. (6) Mapped File Name: For named mappings, it’s the mapped file name. For anonymous mappings, it’s the role of this memory segment in the process.

Below is an analysis by Wenxin (it actually analyzed it correctly, this is a PostgreSQL postmaster process):

/proc/[pid]/smaps

The /proc/[pid]/smaps file is an extension based on /proc/[pid]/maps, providing more detailed information than the maps file in the same directory. Each VMA has the following series of data:

[pg@lzl 2345]$ cat smaps 
00400000-00bae000 r-xp 00000000 fd:00 1093852 /pg/pg15.3/bin/postgres
Size: 7864 kB --VSS memory
Rss: 408 kB --RSS memory
Pss: 140 kB --PSS memory
Shared_Clean: 404 kB --Shared, clean memory size
Shared_Dirty: 0 kB --Shared, dirty (i.e., modified) memory size
Private_Clean: 4 kB --Private, clean memory size
Private_Dirty: 0 kB --Private, dirty memory size
Referenced: 408 kB --Current page marked as referenced or containing anonymous mappings
Anonymous: 0 kB --Anonymous pages
AnonHugePages: 0 kB --Anonymous huge pages
Swap: 0 kB --Swapped-out memory size
KernelPageSize: 4 kB --Kernel page size
MMUPageSize: 4 kB --Page table page size
...
7fffe3933000-7fffe3948000 rw-p 00000000 00:00 0 [stack]
Size: 88 kB
Rss: 16 kB
Pss: 16 kB
...

Now we know that maps are the process’s memory mapping information, and smaps also includes the memory size of each mapping segment (VSS, RSS, PSS).

You can calculate a process’s memory usage by looking at PSS, RSS, etc. data in process smaps. Note the unit is KB.

Total physical memory usage of all processes:

grep Pss /proc/[1-9]*/smaps | awk '{total+=$2}; END {printf "%d kB\n", total }'

PSS memory of a specific process:

cat /proc/90875/smaps |grep Pss |awk '{sum+=$2 };END {print sum/1024}'

RSS memory of a specific process:

cat /proc/68729/smaps |grep Rss |awk '{sum+=$2 };END {print sum/1024}'

Private memory of a specific process:

cat /proc/90875/smaps|sed '/zero/,/VmFlags/d' |grep Private |awk '{sum+=$2 };END {print sum/1024}'

pmap

The pmap command parses the /proc/[pid]/maps and /proc/[pid]/smaps files. It has few parameters; -x means show more information.

[root@lzl ~]# pmap -x 2345
2345: /pg/pg15.3/bin/postgres -D /pg/1503data
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 7864 212 0 r-x-- postgres
0000000000dad000 88 12 12 rw--- postgres
0000000000dc3000 200 36 32 rw--- [ anon ]
0000000000f1e000 264 12 8 rw--- [ anon ]
00000033a6000000 136 108 0 r-x-- ld-2.17.so
...
00007fbe2ae09000 4 0 0 rw--- libnss_files-2.17.so
00007fbe2ae1b000 145968 4396 4396 rw-s- zero (deleted)
00007fbe33ca7000 96836 8 0 r---- locale-archive
00007fbe39b38000 20 16 16 rw--- [ anon ]
00007fbe39b46000 28 4 4 rw-s- PostgreSQL.3661351388
00007fbe39b4d000 4 0 0 rw-s- [ shmid=0x8001 ]
00007fbe39b4e000 4 4 4 rw--- [ anon ]
00007fffe3933000 84 16 16 rw--- [ stack ]
00007fffe397d000 4 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------ ------ ------
total kB 268896 5532 4540

The pmap output format is similar to /proc/[pid]/maps, with one line per VMA address, but includes VSS and RSS in addition to maps, allowing you to directly see the size used by each region of the process’s virtual memory, helping to quickly determine where the regions with more memory are.

If the [heap] in the address space is too large, it might be a heap memory leak. For another example, if the process address space contains too many VMAs (each line in maps can be understood as a VMA), it’s likely that the application called many mmaps without munmap. Or, continuously observing changes in the address space — if certain entries are continuously growing, there’s likely an issue there.

Analysis Example

From the host’s TOP memory view, a certain PostgreSQL backend process memory appears relatively high. Further analysis of map information is needed:

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
68729 postgres 20 0 5579004 5.116g 5.114g R 97.4 1.4 128:27.94 postgres: lzl: lzldb lzl 30.78.14.174(58067) DELETE 

Check this process’s Rss, Pss, Uss:

cat /proc/68729/smaps |grep Rss |awk '{sum+=$2 };END {print sum/1024}'
5422.67 ---5.4G Rss
cat /proc/68729/smaps |grep Pss |awk '{sum+=$2 };END {print sum/1024}'
467.957 ---467mb Pss
cat /proc/68729/smaps|sed '/zero/,/VmFlags/d' |grep Private |awk '{sum+=$2 };END {print sum/1024}'
179.605 ---179mb Uss

Rss-Uss=5.3G of shared memory. From Pss-Uss=290mb of proportional shared memory, we can roughly see that this backend is only a small portion of this shared memory proportion.

$ pmap -x 68729
68729: postgres: pdmp: pdmpdata pdmp 30.78.14.174(46252) DELETE
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 6084 2444 0 r-x-- postgres
0000000000bf0000 4 4 4 r---- postgres
0000000000bf1000 52 52 52 rw--- postgres
...
00002b7f65bfa000 5441216 5365444 5365444 rw-s- zero (deleted) --this part takes the most
00002b80b1daa000 48 0 0 r-x-- libnss_files-2.17.so
00002b80b1db6000 2044 0 0 ----- libnss_files-2.17.so
00002b80b1fb5000 4 4 4 r---- libnss_files-2.17.so
00002b80b1fb6000 4 4 4 rw--- libnss_files-2.17.so
00002b80b1fb7000 24 0 0 rw--- [ anon ]
00002b80ba001000 516 516 516 rw--- [ anon ]
00007fffe16f7000 132 88 88 rw--- [ stack ]
00007fffe175b000 8 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]

Diving deeper into smap analysis, we can directly locate the zero (deleted) part:

$ cat smaps 
00400000-009f1000 r-xp 00000000 fd:06 58726481 /paic/postgres/base/9.6.6/bin/postgres
...
2b7f65bfa000-2b80b1daa000 rw-s 00000000 00:04 72254 /dev/zero (deleted)
Size: 5441216 kB
Rss: 5365444 kB
Pss: 264618 kB
Shared_Clean: 0 kB
Shared_Dirty: 5365444 kB --shared dirty data
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 5364764 kB
Anonymous: 0 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
VmFlags: rd wr sh mr mw me ms sd 

From the above analysis, we can conclude: this is a PostgreSQL private process that has modified a large amount of data without flushing dirty pages. Its own private memory is not much; most is occupied in shared memory. This is likely a transaction in PostgreSQL that has modified a lot of data but hasn’t committed yet.

Additionally, /dev/zero (deleted) is explained in proc(5) — Linux manual page:

Although these entries are present for memory regions that were mapped with the MAP_FILE flag, the way anonymous shared memory (regions created with the MAP_ANON | MAP_SHARED flags) is implemented in Linux means that such regions also appear on this directory. Here is an example where the target file is the deleted /dev/zero one:
 lrw-------. 1 root root 64 Apr 16 21:33
 7fc075d2f000-7fc075e6f000 -> /dev/zero (deleted)

“Unofficial translation”: Anonymous pages and shared pages are represented by /dev/zero (deleted).

/proc/[pid]/status
#

status can view process state information, including some memory information.

[root@lzl 2345]# cat status 
Name: postgres ---the command running this thread
State: S (sleeping) ---process state
Tgid: 2345 ---Thread group ID (i.e., Process ID)
Pid: 2345 ---Thread ID
PPid: 1 ---PID of parent process.
...
VmPeak: 268964 kB ---virtual memory peak
VmSize: 268896 kB ---virtual memory current
VmLck: 0 kB
VmHWM: 13400 kB ---RSS peak
VmRSS: 5532 kB ---RSS current
VmData: 528 kB ---data segment
VmStk: 88 kB ---stack segment
VmExe: 7864 kB ---text segment
VmLib: 3100 kB ---shared library code segment
VmPTE: 136 kB ---Page table entries
VmSwap: 308 kB ---swap size
Threads: 1 ---number of threads in this process
....

Compared to maps, status has no mapping information. The memory data is more summarized, allowing for a more intuitive view of the size occupied by each segment of virtual memory.

View processes with the most SWAP usage:

for file in /proc/*/status ; do awk '/VmSwap|Name|^Pid/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 3 -n -r | head

cgroup memory
#

cgroup memory control is now very common. Some host parameters need to be set in cgroup. Memory settings and monitoring information are under /sys/fs/cgroup/memory/.

cginfo to view CGROUP memory allocation and usage: /opt/cgtools/cginfo -t perf -s mem

cginfo -t perf -s mem
==================== Cgroup Performance: memory ====================
DB_TYPE INSTANCE_NAME MEM_OOM MEM_FILE_GB MEM_MAP_GB MEM_USED_GB MEM_ALLO_GB ALLO_RATE MEM_GLOB_GB GLOB_RATE 
------- ------------- ------- ----------- ---------- ----------- ----------- --------- ----------- --------- 
postgres LZLDB 0 154.3 0.0 4.2 160.0 2.6% 375 1.1% 

View relatively detailed CGROUP memory usage status: /sys/fs/cgroup/memory/[group]/memory.stat

$ cat memory.stat 
...
total_cache 167791534080
total_rss 4006932480
total_rss_huge 0
total_mapped_file 11747328
total_swap 0
total_pgpgin 792754417976
total_pgpgout 792712474991
total_pgfault 477971874868
total_pgmajfault 97318
total_inactive_anon 1610874880
total_active_anon 2408255488
total_inactive_file 73446166528
total_active_file 94332768256
total_unevictable 0

smem
#

smem is a powerful tool for displaying memory usage. It reads information from smaps, meminfo, etc. under /proc and outputs summaries. smem can output overall and specific map memory conditions, which is very intuitive and can be analyzed from different dimensions. Overall, it’s a very useful tool for analyzing memory usage.

The repo can be downloaded directly. Basically, just extract and use it. For more usage, refer to smem memory reporting tool. Below are just simple examples:

View system memory usage -w:

[root@lzl ~]# smem -w -k
Area Used Cache Noncache 
firmware/hardware 0 0 0 
kernel image 0 0 0 
kernel dynamic memory 183.9M 84.0M 99.9M 
userspace memory 112.3M 62.2M 50.1M 
free memory 700.3M 700.3M 0 

View memory consumption per user -u:

[root@lzl ~]# smem -s pss -urk
User Count Swap USS PSS RSS 
oracle 25 85.2M 30.8M 95.7M 383.0M 
root 93 112.4M 38.5M 42.3M 86.2M 
pg 12 5.9M 1.6M 2.5M 5.9M 
mysql 1 169.7M 1.7M 1.7M 2.0M 

View memory consumption for a specific user -U:

[root@lzl ~]# smem -U pg -k
 PID User Command Swap USS PSS RSS 
 2345 pg /pg/pg15.3/bin/postgres -D 364.0K 124.0K 134.0K 228.0K 
 2352 pg postgres: logical replicati 636.0K 144.0K 161.0K 196.0K 
 ...

Filter a specific process -P (PROCESSFILTER, not pid):

[root@lzl ~]# smem -P postgres -p
 PID User Command Swap USS PSS RSS 
 2346 pg /pg/pg16.0/bin/postgres -D 0.01% 0.01% 0.01% 0.01% 
 2350 pg postgres: walwriter 0.01% 0.01% 0.01% 0.01% 
 ...

View process mapping and memory usage -m:

[root@lzl ~]# smem -P postgres -mpr -s pss
Map PIDs AVGPSS PSS 
<anonymous> 13 0.02% 0.24% 
[heap] 3 0.07% 0.20% 
/usr/lib64/libpython2.6.so.1.0 1 0.11% 0.11% 
/pg/pg15.3/bin/postgres 6 0.01% 0.06% 
/pg/pg16.0/bin/postgres 6 0.01% 0.06% 
/dev/zero 12 0.00% 0.03% 
[stack] 13 0.00% 0.02% 
...

smem is very intuitive for viewing process USS\PSS\RSS. However, there is one issue: smem cannot filter by pid, only by username or PROCESSFILTER. When a host has multiple database instances deployed, filtering by parent PID or child PID is not very friendly.

top
#

top can display system running status in real time. top can be quite fancy in its usage. Running top directly can also display a lot of information.

Sorting in top:

command sorted-field supported
M %MEM Yes
N PID Yes
P %CPU Yes
T TIME+ Yes

You can use %MEM to sort processes with higher memory usage. %MEM represents the RES memory percentage.

top - 23:38:01 up 3 days, 22:32, 2 users, load average: 1.12, 1.42, 1.09
Tasks: 198 total, 13 running, 183 sleeping, 0 stopped, 2 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1020348k total, 325848k used, 694500k free, 1352k buffers
Swap: 4128760k total, 635872k used, 3492888k free, 150288k cached

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18537 oracle 20 0 636m 24m 21m S 0.0 2.4 0:05.41 oracle
18533 oracle 20 0 638m 24m 21m S 0.0 2.4 0:02.01 oracle
...
18509 oracle 20 0 634m 4384 4036 S 0.0 0.4 0:01.93 oracle
 2639 root 20 0 729m 4052 1444 S 0.0 0.4 8:45.32 nautilus 

Memory-related interpretation:

Line 4: Memory usage information: physical memory amount, used memory, free memory, buffer memory Line 5: Swap partition information: available swap total, used swap total, free swap total, kernel cached amount

Line 6 (memory-related):

VIRT: VSS
RES: RSS (likely), anything occupying physical memory
SHR: Shared Memory Size. It will include shared anonymous pages and shared file-backed pages
%MEM: RSS percentage, a task’s currently resident share of available physical memory.

Additionally, don’t forget to look at the process status when checking memory.

S (example column 8) Process Status:

D = uninterruptible sleep. Indicates the process is waiting for an external event to complete, such as disk I/O operations or network requests. Usually, D processes cannot be directly terminated.
I = idle
R = running
S = sleeping
T = stopped by job control signal
t = stopped by debugger during trace
Z = zombie

The top command can see the host’s memory summary information. Process memory usage information includes RSS and SHR. A rough calculation of RES-SHR=USS can also calculate the private memory usage size. Additionally, you can see process status, so top -p to view basic memory information for a specific process is very useful.

free
#

free displays the host’s swap, total and remaining memory, all parsed from /proc/meminfo.

user@ubuntu:~$ free

 total used free shared buff/cache available
Mem: 8029356 794336 6297928 183384 937092 6816804
Swap: 0 0 0

total: Total usable memory (MemTotal and SwapTotal in /proc/meminfo). This includes the physical and swap memory minus a few reserved bits and kernel binary code.
used: Used or unavailable memory (calculated as total - available)
free: Unused memory (MemFree and SwapFree in /proc/meminfo) shared Memory used (mostly) by tmpfs (Shmem in /proc/meminfo)
buffers: Memory used by kernel buffers (Buffers in /proc/meminfo)
cache: Memory used by the page cache and slabs (Cached and SReclaimable in /proc/meminfo). Not just pagecache, but also SReclaimable slab!
buff/cache: Sum of buffers and cache
available: cache includes pagecache and SReclaimable, free includes mem free and swap free; while available includes pagecache and memory about to be reclaimed. Indicates available memory, but their calculation methods differ. In practical applications, due to cache existence, available is usually larger than free.

Page Cache: Page cache is primarily used as a cache for file data on the file system, especially when processes have read/write operations on files.

Buffer Cache: Buffer cache is primarily designed for caching blocks when the system reads/writes block devices.

ps aux
#

The biggest advantage of ps is analyzing process status (including memory) from the process perspective. Processes with [ ] flags in the COMMAND are kernel processes.

[pg@lzl ~]$ ps aux|head -1;ps aux|grep postgres
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
pg 2345 0.0 0.0 268896 236 ? Ss Jan01 0:03 /pg/pg15.3/bin/postgres -D /pg/1503data
pg 2353 0.0 0.0 269040 196 ? Ss Jan01 0:00 postgres: checkpointer
pg 2354 0.0 0.0 269032 160 ? Ss Jan01 0:02 postgres: background writer
pg 2356 0.0 0.0 269032 116 ? Ss Jan01 0:01 postgres: walwriter
pg 2357 0.0 0.0 270508 824 ? Ss Jan01 0:02 postgres: autovacuum launcher
pg 2358 0.0 0.0 270492 620 ? Ss Jan01 0:00 postgres: logical replication launcher
pg 29818 0.0 0.0 103372 868 pts/0 S+ 09:16 0:00 grep postgres

VSZ and RSS units are KB. Memory information is limited; VSZ has little value, RSS can be referenced, but there’s no PSS or USS type information, so not much can be analyzed.

ipcs
#

ipcs -m is a command for querying IPC (Interprocess Communication) shared memory resources. It’s quite useful when analyzing shared memory.

[pg@lzl ~]$ ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status 
0x0010c0b6 32769 pg 600 56 6 

Shared memory key value
Shared memory ID (shmid)
User who created this shared memory
Permissions (perms)
Created size (bytes)
Number of processes attached to this shared memory (nattach)
Shared memory status

When connecting a session to PostgreSQL, one more backend process appears:

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status 
0x0010c0b6 32769 pg 600 56 7 

nattch+1, indicating that the private backend process also shares a portion of the PG shared memory. At this point, the following diagram is understood more deeply:

(http://gauss.ececs.uc.edu/Courses/c4029/code/memory/virtual.pdf)

vmstat
#

vmstat is an abbreviation for Virtual Memory Statistics, and can monitor the operating system’s virtual memory, processes, and CPU activity. It provides statistics on the overall system situation; the shortcoming is that it cannot perform in-depth analysis of a specific process.

Useful parameter explanations:

vmstat [options] [delay [count]]
OPTIONS:
-a Display active and inactive memory
-m Display slabinfo
-s Display memory-related statistics and various system activity counts
-t Append timestamp to each line
-w Wide output mode. Without w, the output is narrow, reducing alignment issues

-bash-4.1$ vmstat -w 1 3
procs -------------------memory------------------ ---swap-- -----io---- --system-- -----cpu-------
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 0 1 661652 763348 324 76100 15 12 54 21 18 45 0 0 79 21 0
 2 1 661652 763340 304 75764 0 0 0 32 12 84 0 1 0 99 0
41 1 661652 760744 244 78300 228 0 3216 0 265 442 0 0 0 100 0

pidstat
#

pidstat is a command from the sysstat tool, used to monitor all or specified processes’ CPU, memory, threads, device IO, and other system resource usage.

Useful parameter explanations:

pidstat OPTIONS interval [ count ] 
-d :Report I/O statistics 
-u :Report CPU utilization
-r :Report page faults and memory utilization
-w :Report task switching activity
-p :pid[,...] 
-l :Display the process command name and all its arguments.

View memory status of a specific process:

-bash-4.1$ pidstat -r -l -p 2345
Linux 2.6.32-431.el6.x86_64 (lzl) 01/06/2024 _x86_64_ (1 CPU)

02:48:32 PM PID minflt/s majflt/s VSZ RSS %MEM Command
02:48:32 PM 2345 0.23 0.00 268896 240 0.02 /pg/pg15.3/bin/postgres -D /pg/1503data 

Various indicators are relatively easy to understand. VSZ, RSS — tired of talking about them.

minflt/s: Abbreviation for “minor page faults”, indicating the number of “minor page faults” that occur per second. A page fault occurs when a program tries to access a page that is not in physical memory. If the page is indeed in the swap area on disk, this is a minor page fault.
majflt/s: Abbreviation for “major page faults”, indicating the number of “major page faults” that occur per second. Unlike minor page faults, major page faults occur when a program tries to access a page that is not in physical memory and is also not in the swap area on disk.

sar
#

sar (System Activity Reporter) is currently one of the most comprehensive system performance analysis tools on Linux. It can report on various aspects of system activity, including: file read/write status, system call usage, disk I/O, CPU efficiency, memory usage, process activity, and IPC-related activity. The SAR tool is part of the sysstat software package.

(https://www.brendangregg.com/Perf/linux_observability_sar.png)

sar is very powerful. The man parameter introduction alone has over 1k lines. This article cannot possibly explain everything (being lazy).

Memory-related parameters:

sar OPTIONS interval [ count ] 
-B :Report paging statistics
-r :Report memory utilization statistics
-W :Report swapping statistics.
-H :Report hugepages utilization statistics
-s [ start_time ] ] [ -e [ end_time ] 

Example: sar view memory utilization sar -r 1 3

kbmemfree: This value is basically consistent with the free value in the free command, so it does not include buffer and cache space
kbmemused: This value is basically consistent with the used value in the free command, so it includes buffer and cache space
%memused: This value is kbmemused as a percentage of total memory (excluding swap)
kbbuffers: buffer in the free command
kbcached: cache in the free command
kbcommit: Memory needed to guarantee the current system, i.e., memory needed to ensure no overflow (RAM + swap)
%commit: This value is kbcommit as a percentage of total memory (including swap)

Example: sar view memory page status sar -B 1 3

pgpgin/s: Kilobytes paged in from disk or SWAP to memory per second
pgpgout/s: Kilobytes paged out from memory to disk or SWAP per second
fault/s: Number of page faults per second, i.e., sum of major and minor faults
majflt/s: Number of major faults per second
pgfree/s: Number of pages placed on the free queue per second
pgscank/s: Number of pages scanned by kswapd per second
pgscand/s: Number of pages directly scanned per second
pgsteal/s: Number of pages reclaimed from cache to meet memory needs per second
%vmeff: Pages stolen (pgsteal) as a percentage of total scanned pages (pgscank + pgscand) per second

Example: sar view swap information sar -W 1 3

Report explanation:

pswpin/s: Number of swap pages swapped in per second
pswpout/s: Number of swap pages swapped out per second

Example: sar view historical memory information sar -B -s "08:00:00" -e "10:00:00"

#Without -e, it shows information from the start time to now
$ sar -B -s "08:00:00"
09:45:01 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
09:46:01 PM 414429.37 395024.08 179478.63 0.07 352922.62 12003.78 4266.52 16269.42 99.99
09:47:01 PM 879907.08 337948.43 157970.97 0.02 402290.21 0.00 0.00 0.00 0.00
09:48:01 PM 772977.43 507343.30 150255.50 0.05 466742.08 0.00 5821.28 5821.27 100.00

Above, pgscank represents the speed at which the kswapd process intervenes in memory reclamation, and pgscand represents the speed of direct memory reclamation.

gcore
#

gcore is part of gdb and can generate a core dump file for a process.

Example: dump a PostgreSQL backend process:

[root@lzl ~] ps -ef|grep 8296
pg 8296 2345 0 09:41 ? 00:00:00 postgres: pg lzldb [local] idle 
[root@lzl ~] cat /proc/8296/smaps |grep Pss |awk '{sum+=$2 };END {print sum/1024}' 
0.351562
[root@lzl ~] cat /proc/8296/smaps |grep Rss |awk '{sum+=$2 };END {print sum/1024}'
0.445312
[root@lzl ~] cat /proc/8296/smaps|sed '/zero/,/VmFlags/d' |grep Private |awk '{sum+=$2 };END {print sum/1024}'
0.0078125

Process 8296’s USS is only 7.8 KB, RSS 445 KB. Dump memory:

gcore -o /tmp/dump 8296

Dumping takes some time, and the dumped file is relatively large, and it will hang the process.

[root@lzl 8296]# ls -lh /tmp/dump.8296 
-rw-r--r-- 1 root root 252M Jan 7 10:59 /tmp/dump.8296

gdb
#

gdb can view specific locations and content in memory.

Example: view PostgreSQL backend cached data:

Open a new session to query a partitioned table, keeping the session open:

[pg@lzl ~]$ psql
psql (15.3)
Type "help" for help.

postgres=> \c lzldb
You are now connected to database "lzldb" as user "pg".
lzldb=> select * from lzlpartition limit 1;
 appl_no | is_deleted | date_created | date_updated 
---------+------------+--------------+--------------
(0 rows)

Use pmap, smaps to view process memory usage and find the memory segment to dump:

[root@lzl 13393]# pmap -x 13393
13393: postgres: pg lzldb [local] idle 
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 7864 1204 0 r-x-- postgres
..
00007fbe2ae1b000 145968 2164 176 rw-s- zero (deleted) ---RSS takes the most here
00007fbe33ca7000 96836 0 0 r---- locale-archive
00007fbe39b38000 20 0 0 rw--- [ anon ]
00007fbe39b46000 28 0 0 rw-s- PostgreSQL.3661351388
00007fbe39b4d000 4 0 0 rw-s- [ shmid=0x8001 ]
00007fbe39b4e000 4 0 0 rw--- [ anon ]
00007fffe3933000 84 36 0 rw--- [ stack ]
00007fffe397d000 4 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]

[root@lzl 13393]# cat /proc/13393/smaps |grep -A 13 zero
7fbe2ae1b000-7fbe33ca7000 rw-s 00000000 00:04 12556 /dev/zero (deleted)
Size: 145968 kB
Rss: 2164 kB
Pss: 2164 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 1988 kB
Private_Dirty: 176 kB
Referenced: 2164 kB
Anonymous: 0 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB

gdb dump memory:

The starting position for dumping memory is the vm address in smaps + 0x:

[pg@lzl tmp]$ gdb
(gdb) attach 13393
(gdb) dump memory /tmp/delete.dump 0x7fbe2ae1b000 0x7fbe33ca7000

View the dump file:

You can simply view it through strings:

[root@lzl 13393]# strings /tmp/delete.dump|grep lzl|sort|uniq
...
 @lzlpartition_202301
lzlpartition_202301
lzlpartition_202301_appl_no_idx
lzlpartition_202301_date_created_idx
...
lzlpartition_202306
lzlpartition_202306_appl_no_idx
lzlpartition_202306_date_created_idx
 @lzlpartition_attach
lzlpartition_attach
 @nk_lzlpartition
nk_lzlpartition
select * from lzlpartition limit 1;

As long as the session queries a partitioned table, all partition and index metadata is cached in the backend process.

Note:

gdb attach [pid] will hang the process; do not execute casually
The dump file size equals VSS, generally much larger than RSS/PSS/USS

Memory Summary
#

References
#

Easily Break Through File I/O Bottlenecks: Memory-Mapped mmap Technology https://blog.51cto.com/u_15481245/6582927

Step by Step with Diagrams: Deep Understanding of Linux Physical Memory Management https://cloud.tencent.com/developer/article/2352771?areaId=106001

Systematically Learning Memory Management from a DBA’s Perspective https://mp.weixin.qq.com/s/CybzGP44dVWQN5hfFrVx7A

https://linux2me.wordpress.com/2017/09/15/linux-introduction-to-memory-management/

Memory management in Linux https://www.slideshare.net/raghusiddarth/memory-management-in-linux-11551521?from_search=2

Linux Performance Tunning Memory https://www.slideshare.net/shayc1/linux-performance-tunning-memory?from_search=4

How to Learn the Linux Kernel (Memory Chapter) https://mp.weixin.qq.com/s/lKKHH1MMiZbnIbDQt3-IAQ

https://courses.engr.illinois.edu/cs241/sp2014/lecture/09-VirtualMemory_II_sol.pdf

Linux Process Virtual Address Space https://maodanp.github.io/2019/06/02/linux-virtual-space/

Red Hat Official Documentation https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/chap-virtualization_tuning_optimization_guide-numa

Data Processing on Modern Hardware https://db.in.tum.de/teaching/ss21/dataprocessingonmodernhardware/MH_8.pdf?lang=de

Chapter 2 Describing Physical Memory https://www.kernel.org/doc/gorman/html/understand/understand005.html

Various command man pages

Linux Forced Memory Reclamation, Linux Memory Source Code Analysis - Memory Reclamation (Overall Process) https://blog.csdn.net/weixin_35094083/article/details/116688112

Memory Journey — How to Improve CMA Utilization? https://ost.51cto.com/posts/10815

The implementations of anti pages fragmentation in Linux kernel https://teawater.github.io/presentation/antif.pdf

T H E /proc F I L E S Y S T E M https://www.kernel.org/doc/Documentation/filesystems/proc.txt

The /proc/meminfo File in Linux https://www.baeldung.com/linux/proc-meminfo

the proc filesystem https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-proc-meminfo

Introduction and Usage of Linux /proc/{pid}/maps (Locating Memory Leaks) https://blog.csdn.net/mijichui2153/article/details/123934531

CPU and Memory Usage in Linux top Command https://blog.csdn.net/weixin_45030965/article/details/127693042

smem memory reporting tool https://www.selenic.com/smem/

Linux performance optimization https://feiyang233.club/post/linux/

gdb onlinedocs https://sourceware.org/gdb/current/onlinedocs/gdb

Linux_Core_Dumps https://averageradical.github.io/Linux_Core_Dumps.pdf

A Brief Analysis of PostgreSQL FDW

Mon, 12 Aug 2024 00:00:00 +0000

FDW Basic Concepts
#

What is SQL/MED?
#

SQL/MED aims to unify access methods for heterogeneous data sources. In 2003, SQL/MED was added to the ISO/IEC 9075-9 standard, defined as a SQL standard extension for managing external data via foreign-data wrappers (FDW) or datalink (such as Oracle or PG’s dblink). In short, SQL/MED is an international SQL extension standard. Many databases already support SQL/MED, such as DB2, MariaDB, PG, and more.

Without SQL/MED, applications must access required data sources themselves and process data at the application layer:

With SQL/MED, the data access architecture becomes clearer:

However, while this architecture diagram appears simpler, it increases the database’s IO and computation pressure. This goes against the modern trend of decoupling computation from the database to the application layer.

Of course, both approaches have their pros and cons, and SQL/MED is still used in certain scenarios.

SQL/MED exists as a standard, and PostgreSQL supports the SQL/MED standard excellently through FDW.

What is FDW?
#

PostgreSQL has supported FDW since version 9.1. Users can access external data (foreign data) through regular SQL statements. Foreign data is accessed via a foreign data wrapper (FDW). The FDW in PostgreSQL is itself a library — because different external data sources correspond to different FDW extensions, we often call it an FDW plugin.

PG’s FDW functionality is extremely powerful: it not only supports multiple data sources but also optimizes data access, and can even be used for “beyond expectations” purposes, such as implementing cluster functionality.

Installation and Download
#

Basically every type of database and data format has its own FDW plugin: oracle_fdw for Oracle databases, mysql_fdw for MySQL databases, and so on. FDW plugins can be installed directly or downloaded:

FDWs already included as extensions: file_fdw, postgres_fdw, cstore_fdw
Other FDW plugins can be downloaded from PGXN or the wiki, such as: oracle_fdw, mysql_fdw, json_fdw. Be sure to read the README carefully to understand each FDW’s limitations and usage rules.

FDW plugin download: https://pgxn.org/tag/fdw/
More FDWs (mostly beta): https://wiki.postgresql.org/wiki/Foreign_data_wrappers

Write your own FDW: https://www.postgresql.org/docs/current/fdwhandler.html

Advantages of FDW over dblink in PG
#

PG also has dblink. FDW and dblink are functionally similar — both access external tables. But FDW has more advantages:

FDW supports many more data sources (a LOT more). dblink only supports PostgreSQL databases, equivalent to just one FDW plugin — postgres_fdw (which is actually much more powerful).
Transparent to developers. External tables can be accessed just like regular tables.
More compliant with standard SQL syntax.
Better performance in many scenarios.

The functionality provided by this module overlaps substantially with the functionality of the older dblink module. But postgres_fdw provides more transparent and standards-compliant syntax for accessing remote tables, and can give better performance in many cases.

In summary, FDW is stronger than the dblink plugin — you can basically forget about dblink.

FDW’s Four Objects
#

Different FDWs have different usage patterns, but generally all require creating 4 objects: foreign data wrapper, server, user mapping, foreign table. Some objects are not mandatory — for example, file_fdw doesn’t need a user mapping, while relational database FDWs generally require one.

foreign data wrapper
#

After creating the corresponding FDW extension with CREATE EXTENSION, the foreign data wrapper is automatically created.

For example, creating a file_fdw extension:

=# create extension file_fdw;
CREATE EXTENSION

=# \dx
 Name | Version | Schema | Description
--------------------+---------+------------+------------------------------------------------------------------------
 file_fdw | 1.0 | public | foreign-data wrapper for flat file access

## select * from information_schema.foreign_data_wrappers;
 foreign_data_wrapper_catalog | foreign_data_wrapper_name | authorization_identifier | library_name | foreign_data_wrapper_language
------------------------------+---------------------------+--------------------------+--------------+-------------------------------
 postgres | file_fdw | postgres | [null] | c

You can also create a foreign data wrapper manually without using an extension. See CREATE FOREIGN DATA WRAPPER.

server
#

CREATE SERVER creates an external service, essentially specifying the data source. The OPTIONS syntax varies by foreign-data wrapper — for example, the OPTION syntax for file_fdw and postgres_fdw is definitely different. At this point, you need to read the FDW plugin’s README or official documentation. For example:

Create a file_fdw external service named fileserver:

CREATE SERVER fileserver FOREIGN DATA WRAPPER file_fdw;

Create a postgres_fdw external service named pgserver, pointing to the lzldb database on a PG instance at 172.0.0.1:5432:

CREATE SERVER pgserver FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '172.0.0.1', dbname 'lzldb', port '5432');

View servers:

=# select * from information_schema.foreign_servers;
 foreign_server_catalog | foreign_server_name | foreign_data_wrapper_catalog | foreign_data_wrapper_name | foreign_server_type | foreign_server_version | authorization_identifier
------------------------+---------------------+------------------------------+---------------------------+---------------------+------------------------+--------------------------
 postgres | pgserver | postgres | postgres_fdw | [null] | [null] | postgres
 postgres | fileserver | postgres | file_fdw | [null] | [null] | postgres

user mapping
#

User mapping defines the correspondence between external service users and local users. Therefore, relational database FDWs generally have user mappings, while file-type FDWs without user definitions don’t need them.

For example, create a user mapping using the pgserver from above:

CREATE USER MAPPING FOR localuser SERVER pgserver OPTIONS (user 'remoteuser', password 'mypasswd');

View user mappings:

=# select * from information_schema.user_mappings;
 authorization_identifier | foreign_server_catalog | foreign_server_name
--------------------------+------------------------+---------------------
 localuser | lzldb | pgserver

foreign table
#

Foreign tables map remote tables locally, allowing them to be accessed like regular tables. Since local objects are involved and there are many OPTIONS, the full syntax is somewhat complex. See CREATE FOREIGN TABLE. Simply put, you create a locally corresponding remote table.

Two common ways to create foreign tables: creation and import.

Create a foreign table:

CREATE FOREIGN TABLE localtable (
 id char(5) NOT NULL,
 name varchar(40) NOT NULL
)
SERVER pgserver OPTIONS (table_name 'remotetable');

Creating foreign tables one by one is tedious — you can import all tables from a remote schema at once:

IMPORT FOREIGN SCHEMA remoteschema FROM SERVER pgserver INTO localschema;

View foreign tables:

 information_schema.foreign_tables; -- Intuitive view of foreign tables
 pg_foreign_server; -- Less intuitive, but shows OPTION settings

Using FDW
#

Viewing Foreign Table Information
#

psql’s built-in shortcuts are quite clear for viewing the 4 objects of foreign tables, but pay attention to search_path settings:

psql command	Meaning
\des	list foreign servers
\deu	list user mappings
\det	list foreign tables
\dtE	list both local and foreign tables

Foreign table object views/tables can be messy — here’s a quick organization:

foreign data wrapper tables/views	Meaning
information_schema._pg_foreign_data_wrappers	More complete information
information_schema.foreign_data_wrappers	Less information
information_schema.foreign_data_wrapper_options	Targeted query of foreign data wrapper options
pg_foreign_data_wrapper	Slightly less info, but has permission info that other views lack

foreign server tables/views	Meaning
information_schema._pg_foreign_servers	More complete information
information_schema.foreign_servers	Less information
information_schema.foreign_server_options	Targeted option query — one record per option, not per server
pg_foreign_server	Less information, base table

user mapping tables/views	Meaning
information_schema._pg_user_mappings	Fairly complete user mapping information
information_schema.user_mappings	Less information
information_schema.user_mapping_options	Targeted query of UM options
pg_user_mappings	Slightly less than _pg_user_mappings. Viewable by unprivileged users — passwords show as null
pg_user_mapping	Less information, base table, mainly options. Inaccessible to unprivileged users

foreign table tables/views	Meaning
information_schema._pg_foreign_tables	More complete, shows all foreign tables
information_schema._pg_foreign_table_columns	Shows column-to-column mappings
information_schema.foreign_table_options	Targeted display of foreign table options
foreign_tables	Less information, base table

These views/tables look messy but actually have a clear structure. The 4 object types all follow the same data dictionary pattern:

pg_xxx are base tables, the foundational information source for the 4 objects
information_schema._pg_xxx joins pg_xxx base tables with other info — it’s a summary view with comprehensive information
information_schema.xxx is a view on information_schema._pg_xxx, with less information
information_schema.xxx_options provides targeted option information, sourced only from the full view information_schema._pg_xxx
A special view: pg_user_mappings, usable even by unprivileged users

Permission Considerations
#

If you use the postgres superuser throughout to create foreign tables, you’ll rarely encounter issues. But in production, application users are typically not superusers. Therefore, permissions are extremely important — not only important but also quite troublesome. Using a regular user for testing is crucial (as with any testing). PG’s permission system is like a boss battle — missing any link won’t work.

Key permission points:

Foreign data wrapper, server, and user mapping owners are their creators. Users must be granted USAGE privilege or be the owner themselves to use them.
Accessing remote data sources requires users with appropriate permissions — specified in the user mapping step with suitable remote login credentials.
After creating/importing foreign tables locally, these objects are treated as local objects (only the data dictionary is local). So PG’s local object access permission system must also be properly configured.

FDW Usage Examples
#

There are hundreds of FDW implementations for various data sources worldwide — relational databases, NoSQL databases, various file types, Web Services, columnar storage, big data, and more. Here are a few common FDWs.

Using postgres_fdw
#

This is probably the most commonly used and most powerful FDW. It allows accessing external PostgreSQL databases from a local database. It can also be used for self-access — this is important because: PostgreSQL cannot access across databases internally! To solve this problem, a good approach is using FDW for cross-database access within the same instance — accessing yourself through an external connection.

Here’s an example of cross-database access using postgres_fdw:

An instance has two databases: aka and bkb. You can’t query both databases in a single SQL statement — databases in PG are logically isolated, somewhat like Oracle 12c PDBs.

[lzl@postgres]=# \l
 aka | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
 | | | | | postgres=CTc/postgres
 bkb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
 | | | | | postgres=CTc/postgres

Although both databases are local, when using FDW we still need the local/remote database concept. Here we treat aka as the local database and bkb as the remote database, enabling access to bkb’s tables from aka while handling permission issues.

1. Install FDW plugin

\c aka
create extension postgres_fdw;

Note: Extensions are database-level — switch to the local database first.

2. Grant user permissions

grant usage on foreign data wrapper postgres_fdw to akadata;

3. Create server

\c aka akadata
CREATE SERVER bkb_server FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '127.0.0.1', port '5432', dbname 'bkb');

4. Create user mapping

CREATE USER MAPPING FOR akadata SERVER bkb_server OPTIONS (user 'bkbdata', password 'bkbpasswd');

5. Create schema in aka database, grant to akadata user

\c aka postgres
create schema bkb;
grant usage on schema bkb TO akadata;
--GRANT select ON ALL TABLES IN SCHEMA bkb TO akadata;
grant all privileges on schema bkb TO akadata;

6. Import bkb tables

\c aka akadata

Import entire schema:

IMPORT FOREIGN SCHEMA public FROM SERVER bkb_server INTO bkb;

Import a single table:

 IMPORT FOREIGN SCHEMA public LIMIT TO (tab1) FROM SERVER bkb_server INTO bkb

7. View foreign tables

=# select * from information_schema.foreign_tables;
 foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_name
-----------------------+----------------------+-------------------------------------+------------------------+---------------------
 aka | bkb | tab1 | aka | bkb_server

Using file_fdw
#

The file_fdw extension provides PG with read-only access to external files. file_fdw is already in contrib and can be installed with CREATE EXTENSION. External files must conform to COPY rules.

Here’s a classic example of mapping PG output logs to a foreign table, script from the official documentation:

1. Create file_fdw extension

CREATE EXTENSION file_fdw;

2. Create external server

CREATE SERVER fileserver FOREIGN DATA WRAPPER file_fdw;

3. Create foreign table

CREATE FOREIGN TABLE pglog (
 log_time timestamp(3) with time zone,
 user_name text,
 database_name text,
 process_id integer,
 connection_from text,
 session_id text,
 session_line_num bigint,
 command_tag text,
 session_start_time timestamp with time zone,
 virtual_transaction_id text,
 transaction_id bigint,
 error_severity text,
 sql_state_code text,
 message text,
 detail text,
 hint text,
 internal_query text,
 internal_query_pos integer,
 context text,
 query text,
 query_pos integer,
 location text,
 application_name text
) SERVER fileserver
OPTIONS ( filename 'pg_log/postgresql-07-06.csv', format 'csv' );

4. Query the log table

=# select user_name,database_name,process_id,error_severity,message from pglog where error_severity<>'LOG';
 user_name | database_name | process_id | error_severity | message
-----------+---------------+------------+----------------+-----------------------------------------------
 appuser1 | db1 | 102349 | ERROR | value too long for type character varying(20)
 appuser1 | db1 | 55378 | ERROR | value too long for type character varying(20)
 appuser2 | db2 | 219377 | ERROR | relation "dual" does not exist

Deep Dive into postgres_fdw
#

postgres_fdw Performance Optimization
#

Unlike most FDW plugins, postgres_fdw is an official plugin maintained by the PostgreSQL Global Development Group, with its source code in contrib. Because external services differ in functionality and structure, some features — such as obtaining remote database access costs or aggregate pushdown in certain scenarios — are difficult to implement in other FDWs. But in postgres_fdw they’re achievable. The official team has done extensive optimization for postgres_fdw, making it extremely powerful.

SQL Execution Process
#

The parser generates a query tree from the foreign table definition.
The planner connects to the foreign server.
Obtain cost information. If use_remote_estimate is true (default), the planner executes EXPLAIN on the remote database to get access costs (step 3); if false, it calculates locally instead.
Deparse generates remote SQL text. FDW accesses remote database objects by sending SQL text — the planner generates SQL text for remote execution. The Remote SQL part of the execution plan directly shows the deparsed SQL:

=> explain (verbose) select a from bkb.tab1 where a=1;
 QUERY PLAN
-----------------------------------------------------------------
 Foreign Scan on bkb.tab1 (cost=100.00..146.86 rows=15 width=4)
 Output: a
 Remote SQL: SELECT a FROM public.tab1 WHERE ((a = 1))

Send SQL statement and receive data. The remote database executes the SQL independently and returns results to the local database based on fetch_size (default 100 rows).

Cost Estimation
#

postgres_fdw can pass remote database object access costs to the local database for calculating the overall SQL execution plan cost. However, simply returning the remote estimated cost isn’t enough — the cost of remote access itself must also be considered. postgres_fdw provides 3 OPTIONS to adjust foreign table cost estimation:

use_remote_estimate: When set to true, the planner runs EXPLAIN on the remote database to get estimated costs, adding fdw_startup_cost and fdw_tuple_cost. When false (default), the planner calculates locally and adds fdw_startup_cost and fdw_tuple_cost. Local foreign table statistics may differ from actual values.

fdw_startup_cost: Startup cost for foreign tables, default 100. Represents the cost of establishing a connection, parsing, and generating a plan on the external service.

fdw_tuple_cost: Additional cost per tuple scanned from a foreign table, default 0.01. Represents data transfer cost — higher latency should mean higher settings.

Aggregate Pushdown
#

Aggregate pushdown executes computations on the remote database, with the local database directly receiving the remote execution results. Without aggregate pushdown, all data must be returned to the local database for computation, increasing data transfer’s impact on SQL execution efficiency and the local database’s computational burden.

(In this environment, bkb. are all foreign tables, local tables are public.)

Predicate Pushdown: postgres_fdw supports WHERE pushdown — no need to return all data to the local database.

=> explain (verbose,costs off) select f1.a from bkb.tab1 f1 where f1.a=1;
 QUERY PLAN
---------------------------------------------------------
 Foreign Scan on bkb.tab1 f1
 Output: a
 Remote SQL: SELECT a FROM public.tab1 WHERE ((a = 1))

Sort Pushdown: postgres_fdw supports sort pushdown, sending sorts to the remote database.

=> explain (verbose,costs off) select f1.a from bkb.tab1 f1 order by 1 desc nulls first;
 QUERY PLAN
---------------------------------------------------------------------
 Foreign Scan on bkb.tab1 f1
 Output: a
 Remote SQL: SELECT a FROM public.tab1 ORDER BY a DESC NULLS FIRST

Join Pushdown: Some joins cannot be pushed down, like local table JOIN foreign table — only the foreign table results can be brought locally for joining.

=> explain (verbose,costs off) select f1.a,l2.a from bkb.tab1 f1,tab1 l2 where f1.a=l2.a;
 QUERY PLAN
-----------------------------------------------------
 Hash Join
 Output: f1.a, l2.a
 Hash Cond: (l2.a = f1.a)
 -> Seq Scan on public.tab1 l2
 Output: l2.a, l2.b
 -> Hash
 Output: f1.a
 -> Foreign Scan on bkb.tab1 f1
 Output: f1.a
 Remote SQL: SELECT a FROM public.tab1

When both tables are foreign tables, joins can be pushed down to the remote database:

=> explain (verbose,costs off) select f1.a,f1.b from bkb.tab1 f1 left join bkb.tab2 f2 on f1.a=f2.a;
 QUERY PLAN
-----------------------------------------------------------------------------------------------------
 Foreign Scan
 Output: f1.a, f1.b
 Relations: (bkb.tab1 f1) LEFT JOIN (bkb.tab2 f2)
 Remote SQL: SELECT r1.a, r1.b FROM (public.tab1 r1 LEFT JOIN public.tab2 r2 ON (((r1.a = r2.a))))

Aggregate Function Pushdown: Supports pushing down aggregate functions — functions must be IMMUTABLE.

=> explain (verbose,costs off) select b,count(*),avg(a) from bkb.tab1 group by b;
 QUERY PLAN
----------------------------------------------------------------------------
 GroupAggregate
 Output: b, count(*), avg(a)
 Group Key: tab1.b
 -> Foreign Scan on bkb.tab1
 Output: a, b
 Remote SQL: SELECT a, b FROM public.tab1 ORDER BY b ASC NULLS LAST

Some scenarios aren’t supported, such as HAVING clauses that can only filter locally:

=> explain (verbose,costs off) select b,count(*) from bkb.tab1 group by b having count(*)>=2;
 QUERY PLAN
-------------------------------------------------------------------------
 GroupAggregate
 Output: b, count(*)
 Group Key: tab1.b
 Filter: (count(*) >= 2)
 -> Foreign Scan on bkb.tab1
 Output: a, b
 Remote SQL: SELECT b FROM public.tab1 ORDER BY b ASC NULLS LAST

Other Features
#

Remote Execution OPTION Settings
#

extensions: User-specified FDW extensions that can use “remote computation”. Can only be set at the server level.

fetch_size: Number of rows fetched per batch from the remote database, default 100. Can be set at server or table level.

updatable: By default, postgres_fdw foreign tables are updatable. The updatable option can control this. If a foreign table is inherently non-updatable, setting updatable to false at the table level causes errors directly locally.

truncatable: Starting from PG14, postgres_fdw supports truncating foreign tables, controlled by the truncatable option, defaulting to true.

Connection Management
#

On the first foreign table access in a session, a connection to the remote database is established. As long as the local session hasn’t disconnected, this connection is reused. If multiple user mappings are used, a connection is established for each user mapping.

Starting from PG14, the keep_connections option controls this behavior. Defaults to on, meaning the session can reuse this connection later; when off, the connection is closed at transaction end.

PG14+: postgres_fdw_get_connections() can view connection status.

Transaction Management
#

Important FDW transaction characteristics:

The remote database executes SQL based on the text sent by the local database.
When the local database has SERIALIZABLE isolation level, the remote also uses SERIALIZABLE; otherwise, the remote uses REPEATABLE READ.
When the local transaction commits or rolls back, the remote transaction also commits or rolls back.
FDW does not support 2PC transactions.

Without distributed 2PC transaction support, partial commits may occur. For example, even if a remote update fails, the local update can still complete:

=> select * from tab1;
 a | b
---+-----
 1 | abc
=> begin;
BEGIN
=> update tab1 set b='123' ;
UPDATE 6
=> update bkb.tab1 set b='a' where c=1;
ERROR: 42703: column "c" does not exist
LINE 1: update bkb.tab1 set b='a' where c=1;
=> commit;
COMMIT
=> select * from tab1;
 a | b
---+-----
 1 | 123

No Distributed Lock Management
#

FDW has no distributed lock management, hence no distributed deadlock detection mechanism.

Deadlock detection works for local tables but not for foreign tables.

Asynchronous Execution
#

Starting from PG14, postgres_fdw supports asynchronous execution. When there are multiple Append nodes in the execution plan, they can execute in parallel, improving performance when accessing multiple foreign tables.

Asynchronous execution only occurs with multiple sessions — i.e., multiple user mappings. The async_capable option controls this, defaulting to false. The enable_async_append parameter must also be enabled (default on).

Parallel Commit
#

Starting from PG15, postgres_fdw supports parallel commit. Remote transactions commit alongside local transactions. Without parallel commit/rollback, PG can only commit/rollback remote transactions serially.

postgres_fdw Version History
#

Version	Release Support Notes
9.3	postgres_fdw released
9.6	Support pushdown of join, sort, update, delete; fetch_size support
10	Push down aggregate functions to remote server; more join pushdown scenarios
11	Push down operators to partitioned tables; UPDATE/DELETE joins can push down
12	More order by/limit pushdown scenarios
13	Enhanced password authentication; pg_dump can export foreign tables
14	Parallel scanning for queries with multiple foreign tables (async_capable); bulk insert; postgres_fdw_get_connections(); TRUNCATE foreign tables
15	Push down CASE expressions; parallel commit (parallel_commit)
16	Interruptible parallel transactions; foreign table analyze_sampling; COPY batch_size; foreign table truncate triggers

Sharding Implementation
#

FDW-based Sharding
#

Many PostgreSQL forks (XC/XL, Citus, etc.) have implemented sharding, but PostgreSQL itself is a single-instance database without native sharding support. Since SQL/MED was defined for accessing external data, postgres_fdw can implement sharding by accessing external instances.

Core Sharding Features
#

Key features needed for usable sharding:

Partition management — SQL/MED transparency allows sharding on partitioned tables.
Partition optimization — partition pruning, PARTITION WISE JOIN, etc.
Aggregate pushdown — push computation to shard nodes.
Parallel scanning — PG14 implemented.
2PC transactions — FDW doesn’t yet support this.
Shard management — foreign table partitions must be manually created and added.
Global transactions — global clocks, global snapshot management needed.
Distributed locks — stronger distributed lock mechanisms needed.
Batch writes — DML/COPY distribution to shards needs batch write support.

Summary
#

PostgreSQL’s FDW functionality derives from the SQL/MED standard for accessing external data, supporting many data source types.
FDW has 4 basic objects: foreign data wrapper, server, user mapping, foreign table.
postgres_fdw has many feature enhancements and performance optimizations, capable of pushing operators down to remote databases.
Sharding can be implemented based on postgres_fdw, though some features still need improvement.

References
#

https://www.interdb.jp/pg/pgsql04.html https://www.postgresql.org/docs/13/postgres-fdw.html https://www.postgresql.org/docs/current/file-fdw.html https://wiki.postgresql.org/wiki/WIP_PostgreSQL_Sharding https://www.percona.com/blog/postgres_fdw-enhancement-in-postgresql-14/ https://www.percona.com/blog/foreign-data-wrappers-postgresql-postgres_fdw/ https://www.percona.com/blog/parallel-commits-for-transactions-using-postgres_fdw-on-postgresql-15/ https://www.enterprisedb.com/blog/postgresql-aggregate-push-down-postgresfdw https://www.postgresql.fastware.com/postgresql-insider-fdw-ove https://momjian.us/main/writings/pgsql/sharding.pdf https://www.slideserve.com/johnna/sql-med-and-more-powerpoint-ppt-presentation https://dbaplus.cn/news-19-2090-1.html https://www.highgo.ca/2019/08/08/horizontal-scalability-with-sharding-in-postgresql-where-it-is-going-part-3-of-3/ https://www.highgo.ca/2021/06/28/parallel-execution-of-postgres_fdw-scans-in-pg-14-important-step-forward-for-horizontal-scaling/

A Brief Analysis of PostgreSQL Memory

Mon, 12 Aug 2024 00:00:00 +0000

Architecture
#

(https://www.postgresql.fastware.com/blog/lets-get-back-to-basics-postgresql-memory-components)

(http://geekdaxue.co/read/fcant@sql/qts5is)

Shared Memory
#

Linux Shared Memory Implementation
#

(https://momjian.us/main/writings/pgsql/inside_shmem.pdf)

Shared Memory on Linux Shared memory is an IPC (Inter-Process Communication) mechanism supported by Unix-based operating systems (including Linux). It is a type of memory that multiple processes can simultaneously use to communicate with each other. Shared memory is one of the fastest IPC mechanisms because it does not require processes to copy data between each other. Processes can access shared memory through their own address space.

Two Forms of Shared Memory One form of shared memory is memory-mapped files. Once multiple processes map the same file into their address space, they can access the file’s contents and simultaneously update the file directly using the mapped memory. Another form of shared memory is anonymous memory. This refers to shared memory regions allocated by programs without associating them with a file or persistent storage mechanism.

mmap() Mapping a file into a process’s address space uses mmap(). Anonymous memory can also be created with mmap(). mmap is part of the standard C library. For anonymous memory, the flags should be MAP_ANONYMOUS or MAP_ANON, in which case fd should be NULL or -1, and offset should be 0.

http://www.tutorialsdaddy.com/courses/linux-device-driver/lessons/mmap/

Shared Memory in PostgreSQL
#

https://www.interdb.jp/pg/pgsql02.html

PostgreSQL has many types of shared memory: shared buffers, WAL buffer, CLOG buffer, lock space, etc.

Shared Buffer The shared memory area where PostgreSQL caches data, similar to Oracle’s SGA. When data hits the shared buffer, it is read directly from memory without requiring disk I/O. PostgreSQL loads table pages and indexes from persistent storage into this area and operates on them directly.

WAL Buffer To ensure no data is lost in the event of a server failure, PostgreSQL supports the WAL mechanism. WAL data (also called XLOG records) is PostgreSQL’s transaction log. The WAL BUFFER is the buffer for WAL data before it is written to persistent storage.

CLOG BUFFER The Commit Log (CLOG) maintains the status of all transactions (e.g., in_progress, committed, aborted) for the concurrency control mechanism. The corresponding CLOG BUFFER is the buffer for CLOG data before it is written to disk.

PostgreSQL Shared Memory Parameters
#

shared_buffers Default 128MB. Recommended to configure at 25% of total memory. Because PostgreSQL’s private memory generally takes up a significant portion and relies on cache, sufficient memory must be left for the OS. It is therefore not recommended to set this to as high a value (relative to total memory) as you would for Oracle’s SGA.

shared_memory_type Specifies the shared memory implementation method, not only for shared_buffers but also for other shared data areas. The shared memory implementation varies by platform. (It appears) on Linux the default is mmap. Other values are:

posix (for POSIX shared memory allocated using shm_open)
sysv (for System V shared memory allocated via shmget)
windows (for Windows shared memory)
mmap (to simulate shared memory using memory-mapped files stored in the data directory)

By default, PostgreSQL uses a very small amount of System V shared memory, with the vast majority being mmap shared memory. Due to differences between POSIX and System V IPC, signal implementations differ. The shared_memory_type parameter can be explicitly adjusted for the IPC implementation mechanism:

Setting System V IPC (default is mmap): On Linux and FreeBSD systems, the default shared memory system settings are generally sufficient. Setting shared_memory_type to sysv does not take effect on these two platforms (System V semaphores are not used on this platform). On OpenBSD systems, if shared_memory_type is set to sysv, the default shared memory system parameters are insufficient and need to be adjusted via sysctl.
Setting POSIX IPC: POSIX semaphores are effective on Linux and FreeBSD.

dynamic_shared_memory_type The mechanism for dynamic shared memory, defaults to posix. This parameter is important for parallel queries. A community email about /dev/shm describes:

PostgreSQL creates segments in /dev/shm for parallel queries (via
shm_open()), not for shared buffers. The amount used is controlled by
work_mem. Queries can use up to work_mem for each node you see in the
EXPLAIN plan, and for each process, so it can be quite a lot if you
have lots of parallel worker processes and/or lots of
tables/partitions being sorted or hashed in your query.

Translation:

Parallel queries use POSIX and create segments in /dev/shm
Parallel queries do NOT use shared_buffers
Each plan node in a query is limited by work_mem!

min_dynamic_shared_memory The initial size of memory used by parallel queries, allocated at server startup. Related to huge_pages and dynamic_shared_memory_type.

huge_pages This parameter controls whether the main shared memory area uses huge pages. This means private memory and OS-level memory are not affected by this setting. PostgreSQL’s use of huge pages is currently only supported on Linux and Windows systems; on Linux systems, it is only supported when shared_memory_type is set to mmap!

Setting	Description
try	default, attempts to allocate huge pages
on	uses huge pages; server will not start if allocation fails
off	does not use huge pages

huge_page_size Controls the size of huge pages. Default is 0, meaning PostgreSQL uses the huge page size provided by the operating system. Setting a non-default value is only supported on Linux.

The pg_shmem_allocations View
#

pg_shmem_allocations is a view introduced in PG13 that allows viewing the allocation of major shared memory segments, including those from PostgreSQL itself and extensions.

> select sum(allocated_size)/1024/1024/1024 gb from pg_shmem_allocations;
 gb 
--------------------
 2.7658920288085938
 
 >select * from pg_shmem_allocations order by 4 desc;
 name | off | size | allocated_size 
-------------------------------------+------------+------------+----------------
 Buffer Blocks | 38575360 | 2415919104 | 2415919104
 [null] | 2729553280 | 240300672 | 240300672
 <anonymous> | [null] | 240198528 | 240198528
 Buffer Descriptors | 19700992 | 18874368 | 18874368
 XLOG Ctl | 171008 | 16803472 | 16803584
 Backend Activity Buffer | 2707733248 | 10680320 | 10680320
...

NULL indicates unused memory, anonymous indicates anonymous page allocations. Most of the memory modules in the pg_shmem_allocations view are difficult to understand. You can find them by searching the source code, but there is no intuitive explanation — it simply displays the data from the source code’s init memory module.

Example: Buffer Blocks: Searching the source code directly for “buffer blocks”:

// Initialize shared buffer pool
// Called only once, during shared memory initialization
void
InitBufferPool(void)
{
	bool		foundBufs,
				foundDescs,
				foundIOCV,
				foundBufCkpt;

	/* Align descriptors to a cacheline boundary. */
	BufferDescriptors = (BufferDescPadded *)
		ShmemInitStruct("Buffer Descriptors",
						NBuffers * sizeof(BufferDescPadded),
						&foundDescs);

	BufferBlocks = (char *)
		ShmemInitStruct("Buffer Blocks",
						NBuffers * (Size) BLCKSZ, &foundBufs);

	/* Align condition variables to cacheline boundary. */
	BufferIOCVArray = (ConditionVariableMinimallyPadded *)
		ShmemInitStruct("Buffer IO Condition Variables",
						NBuffers * sizeof(ConditionVariableMinimallyPadded),
						&foundIOCV);

// Checkpoint BufferIds are used to sort checkpoints in shared memory
	CkptBufferIds = (CkptSortItem *)
		ShmemInitStruct("Checkpoint BufferIds",
						NBuffers * sizeof(CkptSortItem), &foundBufCkpt);
}

The InitBufferPool() function initializes the shared buffer.
The shared buffer has 4 sub-pools: Buffer Descriptors, Buffer Blocks, Buffer IO Condition Variables, Checkpoint BufferIds.

Private Memory
#

Private memory is memory areas allocated by PostgreSQL for each session or process. Unlike shared buffers, there is not just one. Private memory of each process cannot be accessed by other processes.

temp_buffers Temp buffers are used to cache temporary table data, default 8MB. temp_buffers is private memory, so temporary tables are only visible to the current session.

work_mem The maximum memory used by query operations, such as sorts and hash tables. Default 4MB. Each query or each plan node? Official documentation:

Note that a complex query might perform several sort and hash operations at the same time, with each operation generally being allowed to use as much memory as this value specifies before it starts to write data into temporary files.

Community email about /dev/shm:

Queries can use up to work_mem for each node you see in the
EXPLAIN plan,

This parameter applies to each operation (plan node) in a query, not to each query. A query can have many parallel operations, so a single query can also consume a lot of memory. Therefore, the work_mem setting must be made very carefully to avoid exhausting OS memory. The worst case: multiple sessions, each session having multiple plan nodes, and those plan nodes using operations that heavily consume work_mem. Which operations use work_mem? For sort operations: ORDER BY, DISTINCT, merge joins. For hash table usage: hash joins, hash-based aggregation, memoize nodes, hash-based IN subqueries.

hash_mem_multiplier Used to limit the memory size of hash-based operations. The limit is hash_mem_multiplier * work_mem. hash_mem_multiplier defaults to 2. Although work_mem can be limited, you cannot limit how many hash operations a query uses, so PG13 added this parameter. This means that before version 12 (inclusive), it was very difficult to limit hash table memory. In our 9.6 production environment, we found a single session consuming 300GB of memory. The culprit was the lack of hash table limits in older versions combined with an execution plan that incorrectly used hash tables.

maintenance_work_mem Memory area used by operations such as VACUUM, CREATE INDEX, and ALTER TABLE ADD FOREIGN KEY. These are session-initiated operations with independent processes that use private memory. These maintenance operations cannot run in parallel within a single session, and concurrency is generally low, so this parameter can be set relatively high. Autovacuum may also use this memory area and limit. See autovacuum_work_mem explanation.

autovacuum_work_mem Maximum memory used by each autovacuum worker process. Default -1, meaning the maintenance_work_mem parameter is used to limit autovacuum workers. Vacuum uses at most 1GB of memory, and autovacuum has the same limit, so setting the vacuum/autovacuum memory limit above 1GB is meaningless.

vacuum_buffer_usage_limit Limits the number of pages that VACUUM and ANALYZE can access from shared memory, to prevent too many pages from being evicted. Default is 256KB, 0 means no limit. When using VACUUM or ANALYZE commands, BUFFER_USAGE_LIMIT can be specified, which takes precedence over the GUC parameter vacuum_buffer_usage_limit.

max_stack_depth The maximum safe depth of the execution stack, generally meaning the stack depth of a recursive function executed on a single backend process. Default is 2MB. The OS kernel stack limit should be set slightly larger than max_stack_depth. If a recursive function exceeds the stack depth, the following error is reported:

ERROR: stack depth limit exceeded HINT: 
Increase the configuration parameter max_stack_depth (currently 2048kB), 
after ensuring the platform's stack depth limit is adequate.

logical_decoding_work_mem Before PG13, logical decoding would retain at most 4096 changes in memory (max_changes_in_memory hardcoded in the source). PG13 introduced the parameter logical_decoding_work_mem. If the data held by logical decoding exceeds this memory value, it is written to disk. Default 64MB.

each replication connection only uses a single buffer of this size,

Generally, the number of logical replication connections is not large, so logical_decoding_work_mem can be set relatively high without issues.

xxCache
#

xxCache is also private memory. For example, PostgreSQL caches relation metadata in relcache. The official documentation has relatively little description about this, but PostgreSQL memory problems are often related to it. For instance, the issue of catalog cache causing each backend process to consume a lot of memory without releasing it has appeared in many environments. Here is a community email from 2016 by Digoal about catalog cache consuming excessive memory

Every PostgreSQL session holds system data in own cache. Usually this cache is pretty small (for significant numbers of users). But can be pretty big if your catalog is untypically big and you touch almost all objects from
catalog in session. A implementation of this cache is simple - there is not
delete or limits. There is not garabage collector (and issue related to
GC), what is great, but the long sessions on big catalog can be problem.
The solution is simple - close session over some time or over some number of operations. Then all memory in caches will be released.

The community’s explanation of catalog cache:

Each session has its own cache for storing system data (metadata, etc.)
Generally, this cache is small. When the catalog is large and a session has accessed all catalog objects, the cache can become very large.
Cache management is simple: there is no deletion mechanism or limit (though invalidation messages do exist).
Closing the session releases the cache.

Tom Lane’s solution was also simple and blunt — add more hardware resources:

I do not think you should complain if that takes a great deal of memory. Either rethink why you need so many tables, or buy hardware commensurate with the size of your problem.

In fact, there are many knowledge points about caches worth paying attention to. After understanding their principles, the solutions to cache-caused memory issues may not be limited to just one approach. There are many types of xxCache, such as relcache, syscache, plancache, etc. Since documentation is scarce, understanding xxCache requires reading the source code. The main xxCache source code is under src/backend/utils/cache. Source structure:

inval.c				-- Invalidation message dispatcher for private caches. The corresponding shared cache invalidation message handler is sinval.c
relfilenodemap.c	-- relfilenode to oid mapping cache
ts_cache.c			-- Cache for Tsearch (text search) related objects
relmapper.c			-- catalog to relfilenode mapping cache
typcache.c			-- type cache
spccache.c			-- tablespace cache
evtcache.c			-- event trigger cache
attoptcache.c		-- attribute cache
plancache.c			-- plan cache 
relcache.c			-- relation cache 							 								*Focus of this article*
catcache.c			-- system catalog cache 					 						*Focus of this article*
syscache.c			-- one layer above catcache, also system catalog cache	 *Focus of this article*
lsyscache.c			-- routines for conveniently querying catalog cache, 'l' likely stands for lookup
partcache.c			-- routines for operating on partition information in relcache

In addition to handling various caches, there is also source code for operations and messages. Below we focus on relcache, catcache/syscache, and invalidation messages.

relcache
#

What data does a relcache entry store? Defined in src/include/utils/rel.h:

* POSTGRES relation descriptor (a/k/a relcache entry) definitions.

RelationData is the primary data structure for relcache entries:

typedef struct RelationData
{
	RelFileNode rd_node;		/* physical identifier of relation */
	SMgrRelation rd_smgr;		/* cached file handle, or NULL */
	int			rd_refcnt;		/* reference count */
	BackendId	rd_backend;		/* if temp relation, the owning backend id */
	bool		rd_islocaltemp; /* is it a temp rel of the current session */
	bool		rd_isnailed;	/* is it nailed in cache */
	bool		rd_isvalid;		/* is the relcache entry valid */
	bool		rd_indexvalid;	/* are the indexes on the relation valid */
	bool		rd_statvalid;	/* are the statistics on the relation valid */
...
	/* some subtransaction info */
	SubTransactionId rd_createSubid;	/* rel was created in current xact */
	SubTransactionId rd_newRelfilenodeSubid;	/* highest subxact changing rd_node to current value */
	SubTransactionId rd_firstRelfilenodeSubid;	/* highest subxact changing rd_node to any value */
	SubTransactionId rd_droppedSubid;	/* dropped with another Subid set */

	Form_pg_class rd_rel;		/* pointer to the relation's pg_class tuple */
	TupleDesc	rd_att;			/* tuple descriptor */
	Oid			rd_id;			/* relation's oid */
	LockInfoData rd_lockInfo;	/* lock info on the relation */
	RuleLock *rd_rules;		/* rewrite rules */
	MemoryContext rd_rulescxt;	/* private memory cxt for rd_rules */
	TriggerDesc *trigdesc;		/* trigger info, NULL if none */
...
	/* foreign key info */
	List	 *rd_fkeylist;	/* list of ForeignKeyCacheInfo (see below) */
	bool		rd_fkeyvalid;	/* true if list has been computed */
	/* partition info */
	PartitionKey rd_partkey;	/* partition key, or NULL */
	MemoryContext rd_partkeycxt;	/* private context for rd_partkey, if any */
...
	List	 *rd_indexlist;	/* list of all index OIDs */
	Oid			rd_pkindex;		/* primary key oid */
	Oid			rd_replidindex; /* replica identity index oid */

	List	 *rd_statlist;	/* list of extended stats OIDs */
...

	PublicationDesc *rd_pubdesc;	/* publication descriptor, or NULL */
...
	bytea	 *rd_options;		/* parsed pg_class.reloptions */
...
	Form_pg_index rd_index;		/* index descriptor in pg_index tuple */
	struct HeapTupleData *rd_indextuple;	/* all pg_index tuples */
	MemoryContext rd_indexcxt;	/* index cxt */
...
	void	 *rd_amcache;		/* available for use by index/table AM */
...
	struct FdwRoutine *rd_fdwroutine;	/* cached function pointers, or NULL */

...
} RelationData;

RelationData contains a large amount of relation-related metadata: oid, pg_class, partition tables, subtransactions, row security policies, statistics, index metadata, AM, etc.

relcache ROUTINES The ROUTINES source code is located at src/backend/utils/cache/relcache.c. There are mainly 5 stages:

RelationCacheInitialize - Initialize relcache, initially empty
RelationCacheInitializePhase2 - Initialize shared catalogs
RelationCacheInitializePhase3 - Complete relcache initialization
RelationIdGetRelation - Get relation descriptor by relation id
RelationClose - Close a relation

These 5 stages are the 5 main logical steps for a rel entry, equivalent to the lifecycle of a rel entry, not the lifecycle of relcache. The first three stages are all relcache initialization — they initialize relcache and load some system tables and their indexes. The last two stages are the logic for obtaining a reldesc and closing a relation; the relcache itself still exists.

Stage 1: RelationCacheInitialize RelationCacheInitialize initializes relcache:

// Define initial size 400
#define INITRELCACHESIZE		400

void
RelationCacheInitialize(void)
{
	HASHCTL		ctl;
	int			allocsize;

	/*
	 * make sure cache memory context exists
	 */
	// Check if cache mctx exists, create one if not
	if (!CacheMemoryContext)
		CreateCacheMemoryContext();

	// Create hash table indexed by OID for relcache
	ctl.keysize = sizeof(Oid);
	ctl.entrysize = sizeof(RelIdCacheEnt);
	RelationIdCache = hash_create("Relcache by OID", INITRELCACHESIZE,
								 &ctl, HASH_ELEM | HASH_BLOBS);
...
	// Initialize relation mapper
	RelationMapInitialize();
}

RelationCacheInitialize does not allocate any relation operations; it only initializes relcache memory, hash tables, mappers, etc.

Stage 2: RelationCacheInitializePhase2

void
RelationCacheInitializePhase2(void)
{
	MemoryContext oldcxt;
	
	// Initialize relation mapper
	RelationMapInitializePhase2();

	// If in bootstrap mode, shared catalogs don't exist yet, so do nothing
	if (IsBootstrapProcessingMode())
		return;

	// Switch to current cache mctx
	oldcxt = MemoryContextSwitchTo(CacheMemoryContext);

	// Try to load shared relcache file
	if (!load_relcache_init_file(true)) // If init file not loaded
	{
		formrdesc("pg_database", DatabaseRelation_Rowtype_Id, true,
				 Natts_pg_database, Desc_pg_database);
		formrdesc("pg_authid", AuthIdRelation_Rowtype_Id, true,
				 Natts_pg_authid, Desc_pg_authid);
		formrdesc("pg_auth_members", AuthMemRelation_Rowtype_Id, true,
				 Natts_pg_auth_members, Desc_pg_auth_members);
		formrdesc("pg_shseclabel", SharedSecLabelRelation_Rowtype_Id, true,
				 Natts_pg_shseclabel, Desc_pg_shseclabel);
		formrdesc("pg_subscription", SubscriptionRelation_Rowtype_Id, true,
				 Natts_pg_subscription, Desc_pg_subscription);

#define NUM_CRITICAL_SHARED_RELS	5	/* fix if you change list above */
	}

	MemoryContextSwitchTo(oldcxt);
}

The init file is divided into shared and local cache init files. load_relcache_init_file() attempts to load data from these two types of files into relcache (here it should only load the shared ones). If loading fails, it creates descriptors for the 5 basic system tables: pg_database, pg_authid, etc.

Stage 3: RelationCacheInitializePhase3 is the third stage of initialization and contains the most content:

void
RelationCacheInitializePhase3(void)
{
	HASH_SEQ_STATUS status;
	RelIdCacheEnt *idhentry;
	MemoryContext oldcxt;
	bool		needNewCacheFile = !criticalSharedRelcachesBuilt;

	RelationMapInitializePhase3();

	// Switch to CacheMemoryContext
	oldcxt = MemoryContextSwitchTo(CacheMemoryContext);

	// Like stage 2, load more system table descriptors
	if (IsBootstrapProcessingMode() ||
		!load_relcache_init_file(false))
	{
		needNewCacheFile = true;

		formrdesc("pg_class", RelationRelation_Rowtype_Id, false,
				 Natts_pg_class, Desc_pg_class);
		formrdesc("pg_attribute", AttributeRelation_Rowtype_Id, false,
				 Natts_pg_attribute, Desc_pg_attribute);
		formrdesc("pg_proc", ProcedureRelation_Rowtype_Id, false,
				 Natts_pg_proc, Desc_pg_proc);
		formrdesc("pg_type", TypeRelation_Rowtype_Id, false,
				 Natts_pg_type, Desc_pg_type);

#define NUM_CRITICAL_LOCAL_RELS 4	/* fix if you change list above */
	}

	MemoryContextSwitchTo(oldcxt);
...
	// If we haven't obtained critical system indexes yet, do it now
	// Because catcache and/or opclass cache depend on critical system indexes in relcache
	if (!criticalRelcachesBuilt) // If critical indexes not loaded
	{
		load_critical_index(ClassOidIndexId,
							RelationRelationId);
...
		load_critical_index(TriggerRelidNameIndexId,
							TriggerRelationId);

#define NUM_CRITICAL_LOCAL_INDEXES	7	/* fix if you change list above */

		criticalRelcachesBuilt = true; // Mark: critical system table indexes obtained
	}

	// Continue processing shared critical system table indexes.
	// These shared critical system tables are needed in certain situations (autovacuum, client authentication, etc.)
	if (!criticalSharedRelcachesBuilt)
	{
		load_critical_index(DatabaseNameIndexId,
							DatabaseRelationId);
...
		load_critical_index(SharedSecLabelObjectIndexId,
							SharedSecLabelRelationId);

#define NUM_CRITICAL_SHARED_INDEXES 6	/* fix if you change list above */

		criticalSharedRelcachesBuilt = true; // Mark: shared critical system table indexes obtained
	}

	// Scan all entries in relcache and update those that are erroneous
	// from formrdesc or init file
	// If erroneous, read pg_class data and replace the erroneous entry
	// Because the cache file does not contain rules, triggers, security policies,
	// also fetch from pg_class
...
	while ((idhentry = (RelIdCacheEnt *) hash_seq_search(&status)) != NULL)
	{
		Relation	relation = idhentry->reldesc;
		bool		restart = false;


		// Ensure relations in use are not flushed
		RelationIncrementReferenceCount(relation);

		// If it's an erroneous entry, read the tuple from pg_class
		if (relation->rd_rel->relowner == InvalidOid)
		{
		...
			memcpy((char *) relation->rd_rel, (char *) relp, CLASS_TUPLE_SIZE);

			// Update rd_option
			if (relation->rd_options)
				pfree(relation->rd_options);
			RelationParseRelOptions(relation, htup);
...
			ReleaseSysCache(htup);
...
			restart = true;
		}

		// Fix data not in the init file
		// For example, relhasrules, relhastriggers may be outdated or incorrect
		if (relation->rd_rel->relhasrules && relation->rd_rules == NULL)
		{
			RelationBuildRuleLock(relation);
			if (relation->rd_rules == NULL)
				relation->rd_rel->relhasrules = false;
			restart = true;
		}
		if (relation->rd_rel->relhastriggers && relation->trigdesc == NULL)
		{
			RelationBuildTriggers(relation);
			if (relation->trigdesc == NULL)
				relation->rd_rel->relhastriggers = false;
			restart = true;
		}

		// Reload row security policies, since init file doesn't contain them
		if (relation->rd_rel->relrowsecurity && relation->rd_rsdesc == NULL)
		{
			RelationBuildRowSecurity(relation);

			Assert(relation->rd_rsdesc != NULL);
			restart = true;
		}

		// If tableam needs reloading
		if (relation->rd_tableam == NULL &&
			(RELKIND_HAS_TABLE_AM(relation->rd_rel->relkind) || relation->rd_rel->relkind == RELKIND_SEQUENCE))
		{
			RelationInitTableAccessMethod(relation);
			Assert(relation->rd_tableam != NULL);

			restart = true;
		}

		// Decrement reference count
		RelationDecrementReferenceCount(relation);

...
	// Finally, if needed, update the init file (since there may have been reloads, don't waste them)
	if (needNewCacheFile)
	{
		InitCatalogCachePhase2();

		/* now write the files */
		write_relcache_init_file(true); // Write global init file
		write_relcache_init_file(false); // Write private init file
	}
}

Compared to Stage 2 which loads 5 system tables, RelationCacheInitializePhase3() loads more system tables, such as pg_class, pg_proc, and the indexes on these tables. Of course, the precondition for loading these rels is that they are not in cache or have expired. After reloading is complete, the “new” catalog is written to the init file. Looking at the write_relcache_init_file function source code when writing the init file, we can understand the meaning of the true and false parameters:

static void
write_relcache_init_file(bool shared)
{
...
	if (shared)
	{
		snprintf(tempfilename, sizeof(tempfilename), "global/%s.%d",
				 RELCACHE_INIT_FILENAME, MyProcPid);
		snprintf(finalfilename, sizeof(finalfilename), "global/%s",
				 RELCACHE_INIT_FILENAME);
	}
	else
	{
		snprintf(tempfilename, sizeof(tempfilename), "%s/%s.%d",
				 DatabasePath, RELCACHE_INIT_FILENAME, MyProcPid);
		snprintf(finalfilename, sizeof(finalfilename), "%s/%s",
				 DatabasePath, RELCACHE_INIT_FILENAME);
	}
...
}

true means write to the global init file. false means write to the local init file.

The RELCACHE_INIT_FILENAME parameter macro definition:

#define RELCACHE_INIT_FILENAME "pg_internal.init"

So the written init files are:

shared: global/pg_internal.init
local: DatabasePath/pg_internal.init and DatabasePath/pg_internal.init.myPID

Let’s look at real init file paths:

[postgres]$ find ./ -name *init*
./global/pg_internal.init #shared
./base/1/pg_internal.init #local
./base/13577/pg_internal.init #local
./base/13578/pg_internal.init	#local
./base/16398/pg_internal.init	#local
./base/16811/pg_internal.init	#local
./base/17674/pg_internal.init	#local

Diagram of the three initialization stages call flow: (https://blog.japinli.top/2022/07/postgres-relcache-and-syscache/)

Stage 4: RelationIdGetRelation Find a reldesc by OID. The caller only needs an AccessShareLock on the OID and is responsible for incrementing/decrementing the rel’s reference count.

Relation
RelationIdGetRelation(Oid relationId)
{
	Relation	rd;

	// Ensure we're in a transaction
	Assert(IsTransactionState());

	// First try to find in cache via reldesc
	RelationIdCacheLookup(relationId, rd);

	if (RelationIsValid(rd))
	{
		// Return NULL for dropped relations
		if (rd->rd_droppedSubid != InvalidSubTransactionId)
		{
			Assert(!rd->rd_isvalid);
			return NULL;
		}

		RelationIncrementReferenceCount(rd);
		if (!rd->rd_isvalid) // If cached rel is invalid, revalidate it
		{
			if (rd->rd_rel->relkind == RELKIND_INDEX ||
				rd->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) // Load index info directly
				RelationReloadIndexInfo(rd);
			else // For non-index, clear the reldesc
				RelationClearRelation(rd, true);
...
		}
		return rd;
	}

	// No reldesc found, create a new one
	rd = RelationBuildDesc(relationId, true);
	if (RelationIsValid(rd))
		RelationIncrementReferenceCount(rd);
	return rd;
}

RelationIdGetRelation is relatively simple: it obtains a reldesc and index info via OID.

Stage 5: RelationClose The code for RelationClose is also quite simple:

void
RelationClose(Relation relation)
{
	// No lock operations needed, simply decrement refcount
	RelationDecrementReferenceCount(relation);

	// If no sessions have the relation open, partition descriptors can be deleted
	if (RelationHasReferenceCountZero(relation))
	{
		if (relation->rd_pdcxt != NULL &&
			relation->rd_pdcxt->firstchild != NULL)
			MemoryContextDeleteChildren(relation->rd_pdcxt);

		if (relation->rd_pddcxt != NULL &&
			relation->rd_pddcxt->firstchild != NULL)
			MemoryContextDeleteChildren(relation->rd_pddcxt);
	}

#ifdef RELCACHE_FORCE_RELEASE
	if (RelationHasReferenceCountZero(relation) &&
		relation->rd_createSubid == InvalidSubTransactionId &&
		relation->rd_firstRelfilenodeSubid == InvalidSubTransactionId)
		RelationClearRelation(relation, false);
#endif
}

RelationClose is the operation for closing access to a relation. Generally, this function only decrements the refcount of sessions accessing the relation. However, there are exceptions:

When refcount is 0, MemoryContextDeleteChildren() is executed. This function deletes the mctx related to child partition descriptors, which does release memory.
When refcount is 0 and the macro RELCACHE_FORCE_RELEASE is defined, the RelationClearRelation() function deletes the hash table entry. This step does not release memory. The RELCACHE_FORCE_RELEASE macro was not found (only available with explicit compilation?).

relcache is not completely without memory release logic, but the trigger conditions are relatively strict, and the freed memory is not all of the relcache memory.

syscache/catcache
#

CatCache caches tuples from system tables. Built on top of CatCache is another layer called SysCache (KV interface). Essentially, CatCache and SysCache together reorganize data from system tables in memory using a KV approach. syscache/catcache is more complex. Here I’ll briefly extract some easily interpretable content, mainly to understand the cached content and loading mechanism of syscache. For deeper source code analysis, refer to PostgreSQL Source Analysis — Storage Management — Memory Management (3) and PostgreSQL RelCache and SysCache Caches.

catcache struct

typedef struct catcache
{
	int			id;				// cache id, defined in syscache.h
	int			cc_nbuckets;	// number of hash buckets for this cache
	TupleDesc	cc_tupdesc;		// tuple descriptor, copied from reldesc
...
	const char *cc_relname;		// system table name corresponding to the tuple
	Oid			cc_reloid;		// system table OID
	Oid			cc_indexoid;	// index OID for cache key
	bool		cc_relisshared; // is the table shared across databases?
...
	// Statistics used by catcache
#ifdef CATCACHE_STATS
	long		cc_searches;	// number of queries against this catcache
	long		cc_hits;		// hit count
	long		cc_neg_hits;	// negative entry hit count
...
#endif
} CatCache;

catcache entry

typedef struct catctup
{
	int			ct_magic;		// identifies this catctup entry
#define CT_MAGIC 0x57261502

	uint32		hash_value;		// hash key value for this tuple
...

	// Dead tuples won't be returned, but will be removed from catcache when refcount reaches zero
	int			refcount;		// tuple refcount, indicates whether it's being accessed
	bool		dead;			// dead tuple, but not yet cleaned up
	bool		negative;		// is this a negative cache entry?
	HeapTupleData tuple;		// tuple header structure

...
	CatCache *my_cache;		// link to the catcache this tuple belongs to
} CatCTup;

SearchCatCacheMiss() Function SearchCatCacheMiss() is the main function for catcache hit/miss, and after a miss it accesses tuples from the dictionary.

static pg_noinline HeapTuple
SearchCatCacheMiss(CatCache *cache,
				 int nkeys,
				 uint32 hashValue,
				 Index hashIndex,
				 Datum v1,
				 Datum v2,
				 Datum v3,
				 Datum v4)
{
	ScanKeyData cur_skey[CATCACHE_MAXKEYS];
	Relation	relation;
	SysScanDesc scandesc;
	HeapTuple	ntp;
	CatCTup *ct;
	Datum		arguments[CATCACHE_MAXKEYS];

...
	
	// Tuple not found in cache, so try to find it directly from the table
	// If found, add it to cache
	// If not found, add a negative cache entry
	relation = table_open(cache->cc_reloid, AccessShareLock);

	scandesc = systable_beginscan(relation,
								 cache->cc_indexoid,
								 IndexScanOK(cache, cur_skey),
								 NULL,
								 nkeys,
								 cur_skey);

	ct = NULL;
	// When tuple is valid, create an entry
	while (HeapTupleIsValid(ntp = systable_getnext(scandesc)))
	{
		ct = CatalogCacheCreateEntry(cache, ntp, arguments,
									 hashValue, hashIndex,
									 false); // Create an entry
		// Immediately increment refcount
		ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
		ct->refcount++;
		ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
		break;					/* assume only one match */
	}

	systable_endscan(scandesc);

	table_close(relation, AccessShareLock);

	/*
	// If no tuple found, create a negative cache entry (a dummy tuple)
	// The dummy tuple has key columns, all others are null
	// During startup, the invalidation mechanism is not active and entries
	// cannot be cleaned up if a tuple is actually created later
	// So during this phase, negative entries are not created
	 */
	if (ct == NULL) // If no tuple found, enter the following logic
	{
		if (IsBootstrapProcessingMode()) // Return NULL directly if in startup phase
			return NULL;

		ct = CatalogCacheCreateEntry(cache, NULL, arguments,
									 hashValue, hashIndex,
									 true); // Create entry

		CACHE_elog(DEBUG2, "SearchCatCache(%s): Contains %d/%d tuples",
				 cache->cc_relname, cache->cc_ntup, CacheHdr->ch_ntup);
		CACHE_elog(DEBUG2, "SearchCatCache(%s): put neg entry in bucket %d",
				 cache->cc_relname, hashIndex);

		// Negative entries are not returned to caller, refcount remains 0
		return NULL;
	}
...
	return &ct->tuple;
}

The dummy tuple (negative cache entry) here is brilliant — caching a non-existent tuple in catcache prevents needing to query the data dictionary again on the next access, avoiding repeated pointless data dictionary lookups.

Cache Validation Messages
#

When a tuple is updated or deleted, due to transaction visibility rules, these tuples that become invisible after the transaction ends need to be communicated to caches, invalidating the cached tuples so they can be reloaded on the next read. Similarly, when new tuples are inserted, negative cache entries in caches may also need to be flushed to match the new tuples. One common scenario is DDL — DDL may cause certain tuples in the metadata to become invalid, at which point cache validation messages need to be sent to various private caches to clean up cache entries. This cache validation mechanism applies to managing private cache pools like syscache and relcache. Since idle backends won’t read sinval events, messages must be actively sent to allow lagging backends to “catch up.” When completing a transaction, invalidation events must be broadcast to other backends via the SI message queue.

The source code is split into two parts: sinval and inval.

Invalidation interface: src/include/utils/inval.h
Invalidation dispatch: src/backend/utils/cache/inval.c
Invalidation message sharing interface: src/include/storage/sinval.h
Invalidation message sharing dispatch: src/backend/storage/ipc/sinval.c
Invalidation message sharing data structures interface: src/include/storage/sinvaladt.h
Invalidation message sharing data structures: src/backend/storage/ipc/sinvaladt.c

In src/backend/utils/cache/inval.c, the shared-invalidation message structure is defined:

typedef union
{
	int8		id;				/* type field --- must be first */
	SharedInvalCatcacheMsg cc;
	SharedInvalCatalogMsg cat;
	SharedInvalRelcacheMsg rc;
	SharedInvalSmgrMsg sm;
	SharedInvalRelmapMsg rm;
	SharedInvalSnapshotMsg sn;
} SharedInvalidationMessage;

Shared-invalidation messages include the following types:

Invalidate a specific catcache entry
Invalidate the entire catcache entry for a particular system catalog
Invalidate a specific relcache entry
Invalidate ALL relcache entries
Invalidate the smgr cache entry for a particular physical relation
Invalidate a mapped-relation
Invalidate saved snapshots that scanned a relation

Messages are located in the shared memory queue until all other processes read them. Normally, receiving processes only read messages at specific times, so if a receiving process is idle (not processing any user requests) or busy doing other things such that they don’t have time to read these messages, the messages may remain in shared memory indefinitely. In unfortunate situations, if this shared memory space is no longer available for processes to store new messages, that process will have to take on the cleanup task. (In practice, this cleanup is done proactively, so space rarely runs out.) To discard old messages, it must be ensured that all other processes have read them. If some processes cannot do so for the above reasons, it must explicitly signal the lagging processes to catch up. Once the lagging processes have caught up, these messages can be freely discarded. When processing a message, it first checks whether the catalog tuple specified in the message is currently in the cache (the message also specifies the syscache identifier). If so, it is removed from the cache’s hash table. The next time that tuple is requested, it will be re-read from the underlying catalog table and added to the hash table, so subsequent accesses will read the new value. If a process has already locked a particular database object preventing concurrent processes from modifying it, it can continue using the cached tuple until the lock is released.

xxCache Issues Summary
#

There are many types of xxCache, among which the more notable ones are plancache, relcache, and syscache. These caches belong to private memory and exist in each backend process. These caches have no LRU mechanism to evict stale data; they use invalidation messages to clean up globally-unneeded snapshots and metadata information, such as when an object is deleted.

relcache is the place most likely to occupy significant memory. relcache loads metadata information, and during initialization it reads *.init files to speed up loading metadata into relcache. Later, when other metadata needs to be read, loading also occurs.
catcache caches tuple information from the data dictionary. syscache is one layer above catcache — they can be understood as jointly implementing this data dictionary cache. If a tuple does not exist, a negative entry is created to avoid accessing the data dictionary again on the next visit. Similarly, a catcache miss will also read tuples from the data dictionary.
Cache validation messages exist to inform caches that cached tuples and snapshot information have become stale. They can invalidate corresponding relcache and catcache entries. Entries are removed from the cache’s hash table, which releases memory.

Since the cache memory release mechanisms are very limited, when there is a lot of metadata (many tables, partition tables), relcache and catcache can consume a lot of memory — and this can happen for every backend. Possible solutions:

Global cache. Like Oracle’s dictionary cache, cache in one place with shared access. For example, PolarDB’s Global RelCache has already implemented this functionality.
LRU. An LRU mechanism suitable for caches is needed to separate hot and cold ends, cleaning excessively old cache entries from the hash table. This might require cache limit parameters to restrict cache size — ideally one per cache…
Threading mode. Memory is shared and accessed by all threads — a natural advantage.
Periodically disconnect long connections. All of the above are just wishful thinking.
Don’t create too many tables or partitions (note that in PostgreSQL, partitions are also tables).

Memory Contexts
#

PostgreSQL manages memory through the memory context mechanism. I previously did a translation about memory contexts, roughly summarized as follows:

C language requires explicit memory deallocation. To reduce the risk of memory leaks, PostgreSQL implemented memory contexts to manage private memory.
Memory contexts do not require freeing memory after each use; instead, memory is released by deleting a particular context.
Memory contexts form a hierarchical structure — releasing a parent context recursively deletes all child contexts.
Aside from debugging, observing memory context usage is quite difficult. Starting from PG14, the pg_backend_memory_contexts view can observe the current memory context usage of the current session.

Timing of memory context creation during SQL operations:

(https://www.pgcon.org/2019/schedule/attachments/514_introduction-memory-contexts.pdf)

Source Code Analysis
#

In PostgreSQL, all memory allocation, deallocation, and resetting is done within memory contexts, so the malloc(), realloc(), and free() system call functions are not used directly. Instead, palloc(), repalloc(), and pfree() are used for memory allocation, reallocation, and deallocation.

C Library Memory Functions C library dynamic memory allocation functions include:

malloc(): The C library’s malloc() function (memory allocation) is used to allocate large blocks of memory.
calloc(): The C library’s calloc() function (contiguous allocation) is used to allocate contiguous memory.
free(): Used to release memory. malloc() and calloc() do not release memory; after dynamic memory allocation, free() must be used to release it.
realloc(): Used for memory re-allocation.

There is also a C library function memset(), used to fill a memory block with a specific value.

PostgreSQL-Defined Memory Functions The functions actually heavily used in PostgreSQL source code for memory allocation, deallocation, etc., are palloc(), palloc0(), repalloc(), and pfree(). They mostly do not directly interact with OS memory (C library functions); only in certain cases do they call C library memory functions. This essentially adds a layer of protection over OS memory operations, with PostgreSQL handling small memory operations on its own.

palloc(): palloc() primarily calls the alloc method of MemoryContext. alloc corresponds to calling the MemoryContextAlloc function, which in turn calls the AllocSetAlloc function specified in the methods field of the current memory context.

void *
palloc(Size size)
{
	/* duplicates MemoryContextAlloc to avoid increased overhead */
	void	 *ret;
	MemoryContext context = CurrentMemoryContext;
...
	ret = context->methods->alloc(context, size);
....
	return ret;
}

palloc0():

void *
palloc0(Size size)
{
...
	ret = context->methods->alloc(context, size);
...
	MemSetAligned(ret, 0, size);

	return ret;
}

MemSetAligned is macro-defined and actually calls C library memset for memory filling, but MemSetAligned passes 0 as the value.

#define MemSetAligned(start, val, len)\
...\
	memset(_start, _val, _len); \
...	

Compared to palloc, palloc0 not only calls alloc(context, size) but also zeroes out the memory content.

repalloc(): repalloc() primarily calls the realloc method of MemoryContext. The realloc function pointer corresponds to the AllocSetRealloc function.

/*
 * repalloc
 *		Adjust the size of a previously allocated chunk.
 */
void *
repalloc(void *pointer, Size size)
{
	MemoryContext context = GetMemoryChunkContext(pointer);
...
	ret = context->methods->realloc(context, pointer, size);
...
	return ret;
}

pfree(): pfree calls the free_p function pointer in the methods field of the memory context to which the memory chunk belongs, to release the memory chunk’s space. Currently, in PostgreSQL, the free_p pointer actually points to the AllocSetFree function.

/*
 * pfree
 *		Release an allocated chunk.
 */
void
pfree(void *pointer)
{
	MemoryContext context = GetMemoryChunkContext(pointer);

	context->methods->free_p(context, pointer);
	VALGRIND_MEMPOOL_FREE(context, pointer);
}

AllocSetAlloc Memory Allocation Looking at the alloc method within, alloc ultimately points to the AllocSetAlloc function. AllocSetAlloc looks rather complex, but it becomes easier to understand when read in segments:

static void *
AllocSetAlloc(MemoryContext context, Size size)
{
	AllocSet	set = (AllocSet) context;
	AllocBlock	block;
	AllocChunk	chunk;
	int			fidx;
	Size		chunk_size;
	Size		blksize;
...

	// If requested memory exceeds the max chunk size, allocate an entire memory block
	if (size > set->allocChunkLimit)
	{
...
		block = (AllocBlock) malloc(blksize);
...
	}

	// If requested memory is less than chunk size, check free list for available free chunks
	fidx = AllocSetFreeIndex(size);
	chunk = set->freelist[fidx];
	if (chunk != NULL) // There are chunks available in the free list
	{
		Assert(chunk->size >= size);

		set->freelist[fidx] = (AllocChunk) chunk->aset;

		chunk->aset = (void *) set;
...
		return AllocChunkGetPointer(chunk);
	}

...
	// If there's space, try to place the chunk in the allocation block; if not, create a new block
	if ((block = set->blocks) != NULL)
	{
		Size		availspace = block->endptr - block->freeptr;

		if (availspace < (chunk_size + ALLOC_CHUNKHDRSZ))
		{
...
			block = NULL;
		}
	}
	// No space, create a new block
	if (block == NULL)
	{
		Size		required_size;

...
		// Requested block size is a power of 2, not exceeding maxBlockSize
		required_size = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
		while (blksize < required_size)
			blksize <<= 1;

		// Use malloc to allocate the block, size is a power of 2
		block = (AllocBlock) malloc(blksize);

...
}

(https://smartkeyerror.com/PostgreSQL-MemoryContext)

palloc() => AllocSetAlloc() only calls malloc() to request memory from the OS when the requested memory exceeds the chunk size limit or when there are no free blocks in the freelist. In all other cases, it takes existing free chunks from the freelist.

pfree() is similar (not demonstrated here): pfree() => AllocSetFree() releases a specified memory chunk in a memory context. If the chunk to be freed is the only chunk in the memory block, free() is called directly to release that memory block. Otherwise, the specified chunk is added to the freelist for the next allocation.

Viewing Memory Context Size
#

PG14+: pg_backend_memory_contexts view to directly inspect memory context memory within the database.

lzldb=> SELECT * FROM pg_backend_memory_contexts ORDER BY used_bytes DESC LIMIT 5;
 name | ident | parent | level | total_bytes | total_nblocks | free_bytes | free_chunks | used_bytes 
-------------------------+-------+------------------+-------+-------------+---------------+------------+-------------+------------
 CacheMemoryContext | | TopMemoryContext | 1 | 1048576 | 8 | 508216 | 1 | 540360
 Timezones | | TopMemoryContext | 1 | 104120 | 2 | 2616 | 0 | 101504
 TopMemoryContext | | | 0 | 97680 | 5 | 12904 | 7 | 84776
 ExecutorState | | PortalContext | 3 | 49208 | 4 | 4424 | 3 | 44784
 WAL record construction | | TopMemoryContext | 1 | 49768 | 2 | 6360 | 0 | 43408

PG14+: pg_log_backend_memory_contexts function outputs memory information to the log file, producing output similar to MemoryContextStats(TopMemoryContext) log output.

SELECT pg_log_backend_memory_contexts(9293);

Universal — gdb MemoryContextStats(TopMemoryContext)

Use gdb to call MemoryContextStats(TopMemoryContext):

gdb 
(gdb) attach 9293
(gdb) p MemoryContextStats(TopMemoryContext)
$2 = void

Log output:

TopMemoryContext: 97680 total in 5 blocks; 16856 free (16 chunks); 80824 used
 TableSpace cache: 8192 total in 1 blocks; 2088 free (0 chunks); 6104 used
 RowDescriptionContext: 8192 total in 1 blocks; 6888 free (0 chunks); 1304 used
 MessageContext: 8192 total in 1 blocks; 6888 free (1 chunks); 1304 used
 Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
...
 Relcache by OID: 16384 total in 2 blocks; 3504 free (2 chunks); 12880 used
 CacheMemoryContext: 524288 total in 7 blocks; 90840 free (0 chunks); 433448 used
 index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: pg_statistic_ext_relid_index
...
 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_database_oid_index
 index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_authid_rolname_index
 WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks); 43408 used
 PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
 MdSmgr: 8192 total in 1 blocks; 7592 free (0 chunks); 600 used
 LOCALLOCK hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
 Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
 ErrorContext: 8192 total in 1 blocks; 7928 free (3 chunks); 264 used

Summary
#

references
#

src/backend/utils/mmgr/mcxt.c

src/backend/utils/mmgr/README

https://momjian.us/main/writings/pgsql/inside_shmem.pdf

https://www.interdb.jp/pg/pgsql02.html

https://www.postgresql.org/docs/current/runtime-config-resource.htm

https://www.postgresql.org/docs/16/kernel-resources.html

https://blog.csdn.net/weixin_45644897/article/details/121340327

https://help.aliyun.com/zh/polardb/polardb-for-postgresql/global-cache

https://www.cnblogs.com/feishujun/p/PostgreSQLSourceAnalysis_cache02.html

https://blog.japinli.top/2022/07/postgres-relcache-and-syscache/

https://amitlan.com/2019/06/14/caches-inval.html

https://www.cybertec-postgresql.com/en/memory-context-for-postgresql-memory-management/

https://www.geeksforgeeks.org/dynamic-memory-allocation-in-c-using-malloc-calloc-free-and-realloc/

https://www.cnblogs.com/feishujun/p/PostgreSQLSourceAnalysis_mmgr01.html

https://www.cnblogs.com/feishujun/p/PostgreSQLSourceAnalysis_mmgr02.html

https://smartkeyerror.com/PostgreSQL-MemoryContext

https://jnidzwetzki.github.io/2022/05/28/postgres-memory-context.html

https://www.pgcon.org/2019/schedule/attachments/514_introduction-memory-contexts.pdf

A Deep Dive into PostgreSQL Transactions

Mon, 12 Aug 2024 00:00:00 +0000

PostgreSQL Transactions

To guarantee ACID properties, an RDBMS must implement concurrency control. PostgreSQL, like Oracle and MySQL (InnoDB), uses MVCC (Multi-Version Concurrency Control) for concurrency control. MVCC works by continuously generating new versions of objects as data changes while allowing queries to access a bounded range of older versions. It captures a snapshot of data at a given point in time and selects one version to read.

Oracle and MySQL both use undo segments to record old versions of objects. PostgreSQL has no undo. Instead, during DML operations it writes historical data directly into the original table (UPDATE creates a new row, DELETE marks the row) and records additional columns — xmin and xmax — in the table to store transaction IDs. By comparing transaction IDs and other metadata, PostgreSQL implements its MVCC mechanism.

Among relational databases, PostgreSQL’s transaction mechanism is truly distinctive. Understanding it is key to grasping how PostgreSQL operates under the hood.

Transaction Isolation Levels
#

Most relational databases support multiple transaction isolation levels. Under different isolation levels, concurrent transaction behavior varies.

Setting the Transaction Isolation Level
#

PostgreSQL supports four isolation levels (though only three are actually effective):

{ SERIALIZABLE | REPEATABLE READ | READ COMMITTED | READ UNCOMMITTED }

Isolation level parameters

default_transaction_isolation: sets the default isolation level for all transactions globally.

transaction_isolation: displays the isolation level of the current session.

The default isolation level is read committed.

Changing the global default isolation level

Modify the default_transaction_isolation parameter and reload:

postgres=# alter system set default_transaction_isolation to 'serializable';
ALTER SYSTEM
postgres=# select pg_reload_conf();
 pg_reload_conf 
----------------
 t
 (1 row)
 postgres=# show transaction_isolation;
 transaction_isolation 
-----------------------
 serializable

After the change, every new transaction will use the default_transaction_isolation isolation level.

Setting the session isolation level

Note: transaction_isolation only displays the current session’s isolation level. This parameter cannot be modified directly.

lzldb=# alter system set transaction_isolation to 'REPEATABLE READ';
ERROR: parameter "transaction_isolation" cannot be changed

Use SET SESSION to change the session’s isolation level:

lzldb=# SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SET
lzldb=# show transaction_isolation ;
-[ RECORD 1 ]---------+----------------
transaction_isolation | repeatable read

Setting the transaction-level isolation level

PostgreSQL allows specifying the isolation level for an individual transaction. You can set it when starting the transaction:

lzldb=# BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
BEGIN
lzldb=# start TRANSACTION ISOLATION LEVEL REPEATABLE READ;
START TRANSACTION

Or use set transaction after starting a transaction:

lzldb=# begin;
BEGIN
lzldb=*# set transaction ISOLATION LEVEL REPEATABLE READ;
SET

ANSI-92 Transaction Isolation Levels
#

The ANSI SQL-92 standard defines four isolation levels:

Serializable

All transactions in the system execute serially, without interfering with each other. Executing transactions one after another avoids all data inconsistency scenarios.

Early implementations used exclusive locks to control concurrent transactions. Serial execution caused queuing and dramatically reduced system concurrency. After ANSI-92, more serializable implementation methods emerged, greatly improving both concurrency and performance.

Repeatable Read

Once a transaction begins, all data read during the transaction cannot be modified by other transactions. Repeatable Read is MySQL’s default isolation level.

Note: in ANSI SQL, Repeatable Read can experience phantom reads, but PostgreSQL’s Repeatable Read does not.

Read Committed

A transaction can read data committed by other transactions. If a transaction reads a piece of data multiple times and that data happens to be modified and committed by another transaction in between, the current transaction will see different values for the same data. This is the default isolation level for both Oracle and PostgreSQL.

At this isolation level, both “non-repeatable read” and “phantom read” scenarios can occur.

Read Uncommitted

A transaction can read data that has been modified but not yet committed by other transactions. Since uncommitted data can still be rolled back, reading such data leads to “dirty reads.”

At this isolation level, “dirty read” scenarios can occur.

PostgreSQL does not have a Read Uncommitted isolation level. Setting Read Uncommitted is treated as Read Committed.

Standard concurrency phenomena and isolation level matrix

Isolation Level	Dirty Read	Non-repeatable Read	Phantom Read
Read Uncommitted	Possible	Possible	Possible
Read Committed	Impossible	Possible	Possible
Repeatable Read	Impossible	Impossible	Possible
Serializable	Impossible	Impossible	Impossible

PostgreSQL concurrency phenomena and isolation level matrix

Isolation Level	Dirty Read	Non-repeatable Read	Phantom Read
Read Uncommitted	Impossible	Possible	Possible
Read Committed	Impossible	Possible	Possible
Repeatable Read	Impossible	Impossible	Impossible
Serializable	Impossible	Impossible	Impossible

A Brief History of Transaction Isolation Levels
#

The isolation levels and anomaly phenomena defined by ANSI SQL-92 have had a profound impact on the database industry. Even today, over 30 years later, most engineers’ understanding of transaction isolation levels still revolves around them, and many real-world database isolation level implementations still follow them. However, the post-ANSI-92 era has seen much discussion and even criticism regarding isolation levels. Here is a summary of the key historical developments:

1992: The database industry was in a chaotic state regarding transactions, so ANSI defined the SQL-92 standard — the widely known 4 isolation levels and 4 anomaly phenomena.
1995: Snapshot Isolation and other isolation levels were proposed, along with more anomaly phenomena. Microsoft engineers proposed the Snapshot Isolation level and criticized ANSI SQL-92, noting that the standard was vaguely defined and many isolation levels and anomalies were left undefined. See A Critique of ANSI SQL Isolation Levels. By this point, there were more than 4 isolation levels and more anomaly phenomena, including write skew.
1999: Due to the proliferation of lock-based isolation levels, Atul Adya’s paper organized these phenomena and mapped the various isolation levels back to ANSI SQL-92 based on anomaly phenomena and functionality.
2005: Because most databases claimed to be serializable but were actually Snapshot Isolation, Alan Fekete et al proposed Making Snapshot Isolation Serializable — achieving serializability on top of Snapshot Isolation by eliminating its anomalies.
2008: Fekete extended serializability and proposed a database-level implementation called Serializable Snapshot Isolation (SSI).
2012: PostgreSQL became the first database to implement SSI. See the PostgreSQL SSI implementation paper.

Isolation levels and anomaly phenomena from the 1995 Critique of ANSI SQL Isolation Levels:

Isolation Levels Supported by Various Databases
#

Many databases claim “full ACID” compliance, but without serializability, ACID cannot be fully realized (especially consistency). Yet many databases claim ACID support even without serializability. The truth is, most do not fully implement it — including the veteran Oracle.

Serializable
#

There are many misconceptions about serializability.

The meaning of serializable: if each transaction is itself correct (satisfying certain integrity conditions), then any schedule that executes those transactions serially is also correct (the transactions still satisfy their conditions). “Serial” means transactions do not overlap in time and cannot interfere with each other — they are fully isolated.

In the 1970s, serializability was achieved through Strict Two-Phase Locking (SS2PL), where reads and writes block each other until the transaction ends. SS2PL sacrifices high availability but eliminates anomaly phenomena.

Beyond SS2PL, there are other ways to achieve serializability, such as Serializable Snapshot Isolation (SSI).

To guarantee no anomalies, serializability sacrifices some concurrency (how much depends on the implementation), but it can truly guarantee data consistency (the “C” in ACID). In other words, databases that do not implement serializability do not fully support ACID.

Serializability has been mathematically proven achievable, but the real database world is somewhat “abnormal.” In practice, serializability is the highest transaction isolation level and the one strongly recommended by academics and experts. However, the vast majority of databases run at Read Committed or Snapshot Isolation.

Why Do Weaker Isolation Levels Cause Academic Problems but Few Real-World Disasters?
#

Anomalies in non-serializable isolation levels generally require high concurrency. Low-concurrency databases rarely encounter problems.
When anomalies do occur, some applications may not detect them or may not consider them important.
It is possible that data becomes anomalous but the application simply returns an error and enters exception-handling logic.
Cost is too high. Not only is the development cost of serializable isolation high for the database, but applications also need to adapt. Simply understanding this complex theory is no easy task.
Higher isolation levels lose some performance. Extensive rework may not be worth it; applications must choose between “high concurrency” and “freedom from anomalies.”
Business logic is built around mechanisms, not rules. Applications have somewhat adapted to the anomalies of weaker isolation levels, especially Read Committed or Snapshot Isolation.

Snapshot Isolation
#

ANSI SQL-92 did not define Snapshot Isolation (SI). This isolation level emerged as the database industry evolved.

Quoting the Wikipedia definition: a transaction executing under Snapshot Isolation operates on a snapshot of the database taken at the start of the transaction. When the transaction ends, it will only commit successfully if the values it updated have not been externally changed since the snapshot was taken. Write conflicts thus cause transaction aborts.

As the name implies, Snapshot Isolation uses snapshots. It exists in databases that use MVCC, where the multi-version concurrency mechanism supports concurrent transaction execution.

The 1992 ANSI SQL-92 standard was defined based on database locks, so it did not define Snapshot Isolation. The concept only emerged with the 1995 Critique.

Serializable Snapshot Isolation
#

Due to the widespread adoption of Snapshot Isolation and the academic goal that databases should achieve serializability, Serializable Snapshot Isolation (SSI) was born. As the name suggests, it achieves serializability on top of Snapshot Isolation.

Because of the ambiguity of the ANSI-92 standard, although Snapshot Isolation was not defined, many databases actually use it. Snapshot Isolation also has certain anomaly phenomena (including write skew), and SSI was created to resolve them.

Mainstream databases implement concurrency control via S2PL or MVCC. Under S2PL, write operations block reads and writes from other transactions, so there is no write skew. MVCC, however, allows reads and writes not to block each other — only write-write conflicts. In concurrent read-write patterns, this leads to write skew. Starting from PostgreSQL 9.1, SSI has been embedded into Snapshot Isolation (PostgreSQL only has Snapshot Isolation, even at the serializable level), resolving write skew and other anomalies.

Write Skew
#

When certain conflicts form a cycle, serialization anomalies occur. One of the easier ones to understand is write skew.

Write skew only happens in read-write patterns (not write-write or write-read), and only under concurrent conditions. A dependency cycle forms when a preceding transaction’s write depends on a later transaction’s write.

There are many real-world cases of write skew. Let’s understand it through the classic black-and-white ball problem:

A bag contains 10 balls: 5 white and 5 black. Two transactions, P and Q, are running. P changes all black balls to white; Q changes all white balls to black. There are two possible serial executions: P then Q, or Q then P. In both cases, the final result is either 10 white balls or 10 black balls. However, Snapshot Isolation allows another outcome:

Transaction P picks up 5 black balls
Transaction Q picks up 5 white balls
Transaction P changes all the balls in hand to white and puts them back
Transaction Q changes all the balls in hand to black and puts them back

Now the bag still has 5 black and 5 white balls — an outcome impossible in any serial execution. Yet this is valid under Snapshot Isolation: each transaction maintains a consistent view of the database, and its write set does not overlap with any concurrent transaction’s write set. Hence, the black and white balls are swapped.

The black-and-white ball problem illustrates: the result under Snapshot Isolation is inconsistent with the result under serial execution. Write skew occurs under Snapshot Isolation, and the data outcome does not match expectations.

SSI in PostgreSQL
#

PostgreSQL was the first database to implement SSI. Here is the black-and-white ball example using the Wikipedia code:

create table dots
 (
 id int not null primary key,
 color text not null
 );
 insert into dots
 with x(id) as (select generate_series(1,10))
 select id, case when id % 2 = 1 then 'black'
 else 'white' end from x;

set default_transaction_isolation = ‘serializable’;	set default_transaction_isolation = ‘serializable’;
begin; update dots set color = ‘black’ where color = ‘white’;
	begin; update dots set color = ‘white’ where color = ‘black’;
commit
	commit
(PostgreSQL SSI: first committer succeeds, second throws an error)	ERROR: could not serialize access due to read/write dependencies among transactions DETAIL: Reason code: Canceled on identification as a pivot, during commit attempt. HINT: The transaction might succeed if retried.

(At Read Committed and Repeatable Read, no error is thrown; the black and white balls simply swap colors. Test results omitted.)

Strict Two-Phase Locking (S2PL) can also achieve serializability, but S2PL requires heavy read-write locks held until transaction commit. S2PL severely impacts concurrency performance, and users generally won’t accept reads and writes blocking each other, so PostgreSQL does not use S2PL.

SSI is an alternative approach to serializability. It still uses Snapshot Isolation but additionally checks for anomaly phenomena. The two approaches also handle anomalies differently: when one occurs, S2PL blocks transactions, while SSI aborts a transaction to break the cycle.

One reason people avoid serializability is that it supposedly reduces database performance. This is understandable — SSI, which performs “anomaly checks,” must be slower than weaker isolation levels that do no such checking. However, with advances in SSI implementation theory and PostgreSQL’s optimizations for read-only transactions, SSI’s performance is now on par with SI.

Serializability greatly simplifies applications’ consistency concerns. PostgreSQL 9.1 has implemented SSI with optimizations. Let’s hope applications will one day truly adopt the serializable isolation level.

Transaction Isolation Level References
#

https://wiki.postgresql.org/wiki/SSI

https://en.wikipedia.org/wiki/Serializability

https://en.wikipedia.org/wiki/Snapshot_isolation

https://justinjaffray.com/what-does-write-skew-look-like/

http://www.bailis.org/blog/when-is-acid-acid-rarely/

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-95-51.pdf — 1995 paper on SI isolation levels and critique of SQL-92

https://www.cse.iitb.ac.in/infolab/Data/Courses/CS632/2009/Papers/p492-fekete.pdf — SSI paper

https://drkp.net/papers/ssi-vldb12.pdf — PostgreSQL SSI implementation

https://ristret.com/s/f643zk/history_transaction_histories — History of transaction isolation levels

Transaction Processing
#

Transaction Blocks
#

Transactions can be implicit or explicit. An implicit transaction is a standalone SQL statement that auto-commits upon completion. An explicit transaction requires an explicit declaration; multiple SQL statements grouped together form a transaction block.

Transaction blocks begin with begin, begin transaction, or start transaction.

They end with COMMIT, END, or ABORT, ROLLBACK, where COMMIT=END and ABORT=ROLLBACK.

BEGIN;
select * from lzl1 limit 1;
update lzl1 set a=2;
END;

If an error occurs during a transaction block, the transaction can only be rolled back due to atomicity:

lzldb=# begin;
BEGIN
lzldb=*# select * from lzl2;
ERROR: relation "lzl2" does not exist
LINE 1: select * from lzl2;
^
lzldb=!# commit;
ROLLBACK

Transaction Processing Functions
#

Transaction processing functions are organized into three layers: top-level transaction functions, middle-level transaction functions, and bottom-level transaction functions.

Top-level transaction functions handle transaction block commands like BEGIN, COMMIT, ROLLBACK, SAVEPOINT, etc.:

BeginTransactionBlock	Start a transaction block
EndTransactionBlock	End a transaction block
UserAbortTransactionBlock	User-initiated transaction abort
DefineSavepoint	Create a savepoint
RollbackToSavepoint	Roll back to a savepoint
ReleaseSavepoint	Release a savepoint

Middle-level transaction functions: every SQL statement calls middle-level functions before and after execution, including after detecting an exception:

StartTransactionCommand	Start a transaction command
CommitTransactionCommand	Complete a transaction command (not commit)
AbortCurrentTransaction	Abort the current transaction

Bottom-level transaction functions: the actual transaction processing functions, responsible for maintaining transaction state, allocating and reclaiming transaction resources, etc.:

StartTransaction	Start a transaction
CommitTransaction	Commit a transaction
AbortTransaction	Rollback/abort a transaction
CleanupTransaction	Clean up a transaction
StartSubTransaction	Start a subtransaction
CommitSubTransaction	Commit a subtransaction
AbortSubTransaction	Rollback/abort a subtransaction
CleanupSubTransaction	Clean up a subtransaction

These functions are fairly easy to distinguish. Aside from a few special functions (top-level savepoint-related, middle-level abort function), the three layers are organized as: *Block (transaction block functions), *Command (command functions), and *Transaction (actual transaction processing functions). Savepoints/subtransactions are treated as transaction-block-level functions (subtransactions can be rolled back within a transaction block, so placing them at the block level makes sense), and abort is treated as a command-level function.

Transaction Block States
#

Top-level and middle-level functions jointly control the transaction block state; bottom-level functions control the transaction state.

Both transaction block states and transaction states are in src/backend/access/transam/xact.c:

typedef enum TBlockState
{
/* states not in a transaction block */
TBLOCK_DEFAULT, /* idle state; entering or exiting a transaction returns to this state */
TBLOCK_STARTED, /* just entered a transaction block; transitions from TBLOCK_DEFAULT; short-lived */

/* transaction block states */
TBLOCK_BEGIN, /* start a transaction block; at this point data block is started, entering block-level state */
TBLOCK_INPROGRESS, /* active transaction; after BEGIN, the block stays in this state until transaction ends */
TBLOCK_IMPLICIT_INPROGRESS, /* active transaction with an implicit BEGIN */
TBLOCK_PARALLEL_INPROGRESS, /* active transaction in parallel execution */
TBLOCK_END, /* received COMMIT command */
TBLOCK_ABORT, /* transaction failed, waiting for ROLLBACK */
TBLOCK_ABORT_END, /* transaction failed, received ROLLBACK */
TBLOCK_ABORT_PENDING, /* active transaction, received ROLLBACK */
TBLOCK_PREPARE, /* active transaction, received PREPARE (explicit 2PC) */

/* subtransaction states (still transaction-block level) */
TBLOCK_SUBBEGIN, /* start a subtransaction */
TBLOCK_SUBINPROGRESS, /* active subtransaction */
TBLOCK_SUBRELEASE, /* received RELEASE (release savepoint) */
TBLOCK_SUBCOMMIT, /* parent transaction COMMIT while subtransaction is still running (SUBINPROGRESS) */
TBLOCK_SUBABORT, /* failed subtransaction, waiting for rollback command */
TBLOCK_SUBABORT_END, /* failed subtransaction, received rollback command */
TBLOCK_SUBABORT_PENDING, /* active subtransaction, received rollback command */
TBLOCK_SUBRESTART, /* active subtransaction, received rollback to command */
TBLOCK_SUBABORT_RESTART /* failed subtransaction, received ROLLBACK TO command */
} TBlockState;

Most states are self-explanatory. A note on rollback vs. abort: their subsequent behavior is similar — both need to clean up transaction resources and exit the current transaction. Yet PostgreSQL separates them into two behaviors with two states: TBLOCK_ABORT and TBLOCK_ABORT_END (and similarly for subtransactions). Why?

src/backend/access/transam/README offers a detailed explanation:

Scenario 1 Scenario 2

1) User types BEGIN 1) User types BEGIN

2) User executes some commands 2) User executes some commands

3) User doesn’t like what she sees, types ABORT 3) The transaction system aborts for some reason (syntax error, etc.)

In Scenario 1, we want to abort the transaction and return to the default state.

In Scenario 2, more commands may follow that are still part of the current transaction block. We must ignore these commands until we see COMMIT or ROLLBACK.

AbortCurrentTransaction handles internal transaction aborts; UserAbortTransactionBlock handles user-initiated aborts. Both rely on AbortTransaction to do all the real work. The only difference is what state we enter after AbortTransaction finishes:

* AbortCurrentTransaction leaves us in TBLOCK_ABORT

* UserAbortTransactionBlock leaves us in TBLOCK_ABORT_END

Bottom-level transaction abort processing has two phases:

* As soon as we realize the transaction has failed, AbortTransaction is executed. This should release all shared resources (locks, etc.) to avoid unnecessarily increasing latency for other backends.

* When we finally see the user’s COMMIT or ROLLBACK, CleanupTransaction is executed; this function cleans up resources and gets us completely out of the transaction. In particular, we cannot destroy TopTransactionContext before this point.

Scenario 1	Scenario 2
1) User types `BEGIN`	1) User types `BEGIN`
2) User executes some commands	2) User executes some commands
3) User doesn’t like what she sees, types `ABORT`	3) The transaction system aborts for some reason (syntax error, etc.)

Transaction States
#

Transaction states are straightforward (note: these are different from transaction block states):

typedef enum TransState
{
TRANS_DEFAULT, /* idle */
TRANS_START, /* transaction started */
TRANS_INPROGRESS, /* active transaction */
TRANS_COMMIT, /* transaction commit */
TRANS_ABORT, /* abort transaction */
TRANS_PREPARE /* prepare transaction (2PC) */
} TransState;

Transaction State Flow
#

Each command in a transaction block calls transaction functions, which in turn transition the transaction and transaction block states.

Let’s use the simplest transaction block as an example (from the README):

1)BEGIN
2)SELECT * FROM foo
3)INSERT INTO foo VALUES (...)
4)COMMIT

Command call relationships:

 / StartTransactionCommand; -- middle-level: start transaction command
 / StartTransaction; -- bottom-level: actually start the transaction
 1)< ProcessUtility; -- ProcessUtility handles the BEGIN command
 \ BeginTransactionBlock; -- top-level: start transaction block
 \ CommitTransactionCommand; -- middle-level: complete command

 / StartTransactionCommand; -- middle-level: start transaction command
2) / PortalRunSelect; -- execute SELECT statement
 \ CommitTransactionCommand; -- middle-level: complete command
 \ CommandCounterIncrement; -- middle-level: command counter increment

 / StartTransactionCommand; -- middle-level: start transaction command
3) / ProcessQuery; -- execute INSERT statement
 \ CommitTransactionCommand; -- middle-level: complete command
 \ CommandCounterIncrement; -- command counter +1

 / StartTransactionCommand; -- middle-level: start transaction command
 / ProcessUtility; -- ProcessUtility handles COMMIT command
4) < EndTransactionBlock; -- top-level: end transaction block
 \ CommitTransactionCommand; -- middle-level: complete command
 \ CommitTransaction; -- bottom-level: actually commit the transaction

Every command in a transaction block begins with the middle-level StartTransactionCommand and ends with CommitTransactionCommand.
Between these two middle-level functions is where the actual command processing occurs.

The transaction block state for 2) SELECT and 3) INSERT is TBLOCK_INPROGRESS. The state transitions for BEGIN and COMMIT:

Transaction Function References
#

PostgreSQL Internals (book)

src/backend/access/transam/README

Transaction ID
#

Every transaction in PostgreSQL is assigned a transaction ID. Transaction IDs come in two forms: virtual transaction IDs and persistent transaction IDs. Understanding transaction IDs is crucial for grasping transactions, data visibility, transaction ID wraparound, and more.

Virtual Transaction ID
#

Read-only transactions are not assigned a transaction ID — transaction IDs are a precious resource. A simple SELECT, for instance, won’t consume one. However, to identify transactions for purposes such as shared locks, a non-persistent transaction ID is needed. This is the virtual transaction ID (VXID).

VXID consists of two parts: a backend ID and a backend-local counter.

Source: src/include/storage/lock.h

 typedef struct
{
BackendId backendId; /* backendId from PGPROC */
LocalTransactionId localTransactionId; /* lxid from PGPROC */
} VirtualTransactionId;

(PGPROC is a structure storing process information; we’ll cover it later.)

You can see VXID in pg_locks. Querying pg_locks itself is a SQL statement, so it generates a VXID:

lzldb=# begin;
BEGIN
lzldb=*# select locktype,virtualxid,virtualtransaction,mode from pg_locks;
 locktype | virtualxid | virtualtransaction | mode 
------------+------------+--------------------+-----------------
 relation		| 		 | 4/16 | AccessShareLock
 virtualxid | 4/16 	 | 4/16 | ExclusiveLock
(2 rows)
 
lzldb=*# savepoint p1;
SAVEPOINT
lzldb=*# select locktype,virtualxid,virtualtransaction,mode from pg_locks;
 locktype 	| virtualxid | virtualtransaction | mode 
------------+------------+--------------------+-----------------
 relation 	| 		 | 4/16 | AccessShareLock
 virtualxid | 4/16 | 4/16 | ExclusiveLock
lzldb=*# rollback;
ROLLBACK
lzldb=# select locktype,virtualxid,virtualtransaction,mode from pg_locks;
 locktype 	| virtualxid | virtualtransaction | mode 
------------+------------+--------------------+-----------------
 relation 	| 	 | 4/17 | AccessShareLock
 virtualxid | 4/17 | 4/17 | ExclusiveLock

After \q (disconnect) and immediately logging back in, the counter continues: 4/19.

Opening another window gives backendID+1:

lzldb=# select locktype,virtualxid,virtualtransaction,mode from pg_locks;
 locktype | virtualxid | virtualtransaction | mode 
------------+------------+--------------------+-----------------
 relation | | 5/3 | AccessShareLock
 virtualxid | 5/3 | 5/3 | ExclusiveLock

From these tests we can observe:

The VXID’s backend ID is not the actual process PID; it’s simply an incrementing number.
Both the VXID’s backend ID and command counter are incrementing.
Subtransactions do not have their own VXID; they use the parent transaction’s VXID.
VXID also has wraparound, but it’s not a serious issue since it isn’t persisted — after an instance restart, VXID starts counting from scratch.

Persistent Transaction ID
#

32-bit TransactionId
#

When a data-modifying transaction begins, the transaction manager assigns it a unique identifier: TransactionId. TransactionId is a 32-bit unsigned integer, capable of storing 2^32 = 4,294,967,296 — about 4.2 billion — transactions. The range of a 32-bit unsigned integer is 0 ~ 2^32 - 1.

Three special transaction IDs

src/include/access/transam.h defines several special transaction IDs:

#define InvalidTransactionId ((TransactionId) 0)
#define BootstrapTransactionId ((TransactionId) 1)
#define FrozenTransactionId ((TransactionId) 2)
#define FirstNormalTransactionId ((TransactionId) 3)
#define MaxTransactionId ((TransactionId) 0xFFFFFFFF)

0: Invalid TransactionId
1: Bootstrap Transaction ID, used only during database initialization. Older than all normal transactions.
2: Frozen Transaction ID. Older than all normal transactions.

#define TransactionIdIsNormal(xid) ((xid) >= FirstNormalTransactionId)

A transaction ID >= 3 is a normal transaction ID.

The maximum transaction ID, MaxTransactionId, is 0xFFFFFFFF = 4,294,967,295 = 2^32 - 1.

So the allocatable range for normal transaction IDs is: 3 ~ 2^32 - 1.

64-bit FullTransactionId
#

Transaction IDs increment sequentially. PostgreSQL has used 32-bit transaction IDs for a long time. Before PostgreSQL 7.2, when the 32-bit transaction ID was exhausted, you had to dump and restore the database. A 64-bit transaction ID, on the other hand, is practically inexhaustible. The source defines a 64-bit FullTransactionId as a struct:

/*
 *A 64-bit value containing an epoch and a TransactionId.
 *It is wrapped in a struct to prevent implicit conversion to TransactionId.
 *Not all values represent valid normal XIDs.
 */
typedef struct FullTransactionId
{
uint64 value;
} FullTransactionId;

The 64-bit value consists of an epoch and a 32-bit TransactionId, converted via these functions:

#define EpochFromFullTransactionId(x)	((uint32) ((x).value >> 32))
#define XidFromFullTransactionId(x)		((uint32) (x).value)

The epoch is FullTransactionId shifted right 32 bits; the XID (TransactionId) is FullTransactionId modulo 2^32. This is like treating the 32-bit TransactionId as a “circle” that loops, while the 64-bit FullTransactionId is a “line” that keeps growing, nearly inexhaustible.

A full transaction ID can exceed 2^32:

Transaction ID Assignment
#

Let’s run a few experiments to see how transaction IDs are assigned. We’ll use two functions that return transaction IDs:

pg_current_xact_id(): returns the current transaction ID; if the current transaction has not yet been assigned one, it allocates one. (In pg12 and earlier, use txid_current().)

pg_current_xact_id_if_assigned(): returns the current transaction ID; if the current transaction has not yet been assigned one, returns NULL. (In pg12 and earlier, use txid_current_if_assigned().)

Transaction IDs are assigned sequentially:

lzldb=# select pg_current_xact_id();
 pg_current_xact_id 
--------------------
 612
lzldb=# select pg_current_xact_id();
 pg_current_xact_id 
--------------------
 613
lzldb=# select pg_current_xact_id();
 pg_current_xact_id 
--------------------
 614

BEGIN does not immediately allocate a transaction ID:

lzldb=# begin; -- explicitly start a transaction
BEGIN
lzldb=*# select pg_current_xact_id_if_assigned () ; -- BEGIN does not immediately allocate a transaction ID
 pg_current_xact_id_if_assigned 
-------------------------------- 
(1 row)
lzldb=*# select * from lzl1; -- query immediately after BEGIN
 a 
---
(0 rows)
lzldb=*# select pg_current_xact_id_if_assigned () ; -- queries do not allocate transaction IDs
 pg_current_xact_id_if_assigned 
-------------------------------- 
(1 row)
lzldb=*# insert into lzl1 values(1); -- insert data, a data change
INSERT 0 1
lzldb=*# select pg_current_xact_id_if_assigned () ; -- the first non-query statement after BEGIN allocates a transaction ID
 pg_current_xact_id_if_assigned 
--------------------------------
 611
lzldb=*# commit;
COMMIT
lzldb=# select xmin, pg_current_xact_id_if_assigned () from lzl1; -- the INSERT transaction writes to xmin
 xmin | pg_current_xact_id_if_assigned 
------+--------------------------------
 611

Some records in system catalogs were assigned BootstrapTransactionId=1 during database initialization:

postgres=# select xmin,count(*) from pg_class where xmin=1 group by xmin;
 xmin | count 
------+-------
 1 | 184

Conclusions from the experiments:

During database initialization, the special transaction ID 1 is assigned, visible in system catalogs.
Transaction IDs are assigned incrementally.
BEGIN does not immediately allocate a transaction ID; the first non-query statement after BEGIN allocates one.
When a transaction inserts a tuple, the transaction’s txid is written into the tuple’s xmin.

Transaction ID Comparison
#

PostgreSQL compares the age of transactions by their transaction IDs. src/backend/access/transam/transam.c defines four comparison functions: <, <=, >, >=:

bool TransactionIdPrecedes()
bool TransactionIdPrecedesOrEquals()
bool TransactionIdFollows()
bool TransactionIdFollowsOrEquals()

They are similar. Let’s examine TransactionIdPrecedes() as the representative:

bool
TransactionIdPrecedes(TransactionId id1, TransactionId id2)
{
/*
 * If either ID is a permanent XID then we can just do unsigned
 * comparison. If both are normal, do a modulo-2^32 comparison.
 */
int32 diff;
 
if (!TransactionIdIsNormal(id1) || !TransactionIdIsNormal(id2))
return (id1 < id2);
 
diff = (int32) (id1 - id2);
return (diff < 0);
}

Key points from this source code:

TransactionIdIsNormal() is a macro defined in the header to check for normal transactions. FirstNormalTransactionId is the constant 3. So a normal transaction ID is >= 3.

#define TransactionIdIsNormal(xid) ((xid) >= FirstNormalTransactionId)

int32 is a signed integer: the first bit being 0 means positive, 1 means negative. Range: -2^31 ~ 2^31 - 1.
Integer overflow: when a value exceeds the storage range (e.g., 2^31 barely overflows for int32), the value wraps around.

The transaction ID comparison code can be understood in two parts:

Non-normal transaction ID comparison:

if (!TransactionIdIsNormal(id1) || !TransactionIdIsNormal(id2))
return (id1 < id2);

When id1=2, id2=100: return(2<100), precedes is true — the normal transaction is newer.

When id1=100, id2=2: return(100<2), precedes is false — the normal transaction is newer.

So, txid 1 and 2 are older than normal transactions.

Normal transaction ID comparison:

diff = (int32) (id1 - id2);
return (diff < 0);

id1 - id2 can be negative, so diff cannot be unsigned int. It must be cast to signed int. Now the crucial part:

Since int32 ranges from -2^31 to 2^31 - 1:

When id1 = 2^31 + 99, id2 = 100: id1 - id2 = 2^31 - 1. Fine — int32 can hold this. → Larger txid is newer.

When id1 = 2^31 + 100, id2 = 100: id1 - id2 = 2^31. Problem — exactly exceeds int32 storage. The value becomes 2^31 - 2^32 = -2^31 < 0. → Smaller txid is considered newer.

When id1 = 100, id2 = 2^31 + 100: id1 - id2 = -2^31. Fine — int32 can hold this. → Larger txid is newer.

When id1 = 100, id2 = 2^31 + 101: id1 - id2 = -2^31 - 1. Problem — exactly exceeds int32 storage. The value becomes -2^31 - 1 + 2^32 = 2^31 - 1 > 0. → Smaller txid is considered newer.

From this analysis, when integer overflow occurs, a transaction with a larger txid cannot see a transaction with a smaller txid. The overflow itself is an exceptional event, so this is acceptable. To address this, PostgreSQL divides the 4-billion transaction ID space into two halves: one half is visible, the other invisible.

For example, for transaction txid 100, the 2 billion transactions in its past are visible, and the 2 billion transactions in its future are invisible. Therefore, the maximum difference between the oldest and newest transaction IDs (the database age) in PostgreSQL is |-2^31| = 2^31, roughly 2 billion.

Transaction ID Wraparound
#

What is transaction ID wraparound?

Understanding transaction ID wraparound itself is not difficult, but when I first studied it, I found two different definitions:

PostgreSQL official definition:

Because transaction IDs are limited in size (32 bits), a cluster that runs for a long time (more than 4 billion transactions) will suffer transaction ID wraparound: the XID counter wraps around to zero, and suddenly past transactions appear to be in the future — meaning they become invisible. In short, catastrophic data loss. (The data is still there, but you can’t access it.)

interdb explanation:

A tuple’s t_xmin records the minimum transaction of that tuple. If the tuple never changes, this t_xmin stays the same. Suppose tuple_1 was created by transaction txid=100, so its t_xmin=100. If the database advances by 2^31 transactions, reaching 2^31+100, tuple_1 is still visible. Then another transaction starts, advancing txid to 2^31+101. Now txid=100 is in the “future,” so tuple_1 becomes invisible. This is severe data loss — this is transaction ID wraparound.

Yes, the official documentation and some classic articles define transaction ID wraparound differently. They are indeed describing two different things. I attribute this to a translation issue: both behaviors are wraparound in English semantics. If you reconsider the meaning of “wraparound,” they are both forms of it.

However, they differ: one is when transaction IDs (2^32) are fully exhausted and wrap back to 0; the other is when the “oldest transaction ID” and “newest transaction ID” differ by more than 2^31.

The official definition of transaction ID wraparound introduces the concept that “transaction IDs form a circle.”
The generally understood transaction ID wraparound problem is the “circle divided into two halves, one visible, one invisible” concept — when the “more than half” threshold is crossed, that’s wraparound.

In practice, the wraparound problem you actually need to worry about is the latter: the difference between the newest and oldest transaction IDs must not exceed 2.1 billion (2^31).

How long does 2.1 billion transactions take?

2.1 billion transactions sounds like a lot, but it can still be exhausted.

For example, a PostgreSQL database with 100 TPS (not counting SELECT statements, since simple SELECTs don’t allocate transaction IDs) uses 8,640,000 transactions per day. It takes only about 2,147,483,648 / 8,640,000 ≈ 248 days to exhaust 2.1 billion transaction IDs and trigger wraparound. At 1,000 transactions per second, it takes less than one month. So transaction ID wraparound is something you must pay attention to in PostgreSQL.

Transaction ID Freezing
#

To solve the serious data loss problem caused by transaction ID wraparound, PostgreSQL introduced the concept of transaction freezing.

XIDs are reused cyclically and divided into two halves: one visible, one invisible. For a tuple with xid=100, if no operations are performed and transaction IDs keep advancing, the once-visible tuple will eventually become invisible.

As mentioned earlier, there is a frozen transaction ID. If the tuple with xid=100 is marked with the frozen transaction ID, it will remain visible. This is the purpose of transaction freezing.

The frozen transaction ID FrozenTransactionId = 2, and it is older than all normal transactions. That means txid=2 is visible to all normal transactions (txid >= 3). When t_xmin is older than current_txid - vacuum_freeze_min_age (default 50 million), the tuple is rewritten with the frozen transaction ID 2. In version 9.4 and later, the xmin_frozen flag in t_infomask is used to indicate a frozen tuple, rather than rewriting t_xmin to 2.

There are many optimization approaches to the transaction ID wraparound problem, but none can avoid transaction freezing. Freezing involves reading every row of every table and resetting flags — a massive I/O and CPU operation. There’s no escaping it; the database may even reject all operations until freezing completes. This is known as the “freeze bomb.” The busier the system and the higher the transaction rate, the more likely it is to trigger. (We’ll expand on freeze optimization in a future chapter.)

64-bit Transaction IDs
#

The ultimate solution to transaction ID exhaustion and wraparound is using 64-bit transaction IDs. A 32-bit txid provides 2^32 IDs; a 64-bit txid provides 2^64. Even at 10,000 transactions per second — 864 million per day — it would take 58.49 million years to exhaust them. With 64-bit transaction IDs, they are practically inexhaustible. No wraparound, no freezing, no “freeze bomb”…

Why hasn’t 64-bit transaction ID been implemented yet?

Note: 64-bit transaction IDs already exist in PostgreSQL (as FullTransactionId described earlier). However, because tuple storage is limited, the xmin, xmax, etc. in tuples still use 32-bit XIDs, and transaction ID comparison still relies on 32-bit XIDs. xmin and xmax — the transaction IDs for insert and delete — are stored in each tuple’s header (we’ll cover tuple structure later), and header space is limited. A 32-bit txid is 4 bytes; a 64-bit txid is 8 bytes. Storing both xmin and xmax as 64-bit would require an extra 8 bytes, which the current header cannot accommodate. The community has discussed two approaches:

Extend the header to store 64-bit transaction IDs directly.
Keep the header size unchanged. Retain 64-bit transaction IDs in memory, adding an epoch concept to convert between the two.

The first approach has been essentially abandoned — compared to other systems, PostgreSQL’s tuple header is already large enough.

The second approach already has epochs and FullTransactionId-to-TransactionId conversion. The key is how to convert the TransactionId in tuples to FullTransactionId (though some extra storage for the epoch would still be needed — otherwise, how to implement it?).

See community mailing list discussions:

https://www.postgresql.org/message-id/CAEYLb_UfC+HZ4RAP7XuoFZr+2_ktQmS9xqcQgE-rNf5UCqEt5A@mail.gmail.com

https://www.postgresql.org/message-id/flat/DA1E65A4-7C5A-461D-B211-2AD5F9A6F2FD%40gmail.com

The community proposed 64-bit transaction IDs as a permanent solution to the freeze problem back in 2014, and began discussing practical implementation in 2017. But after several PostgreSQL versions, it’s still vaporware. Given the sensitivity and importance of data in databases, and how many things transaction ID changes touch — one slip could mean data loss or unknown bugs — PostgreSQL is moving cautiously. However, the community is still considering it. Hopefully one day, in some PostgreSQL version, the transaction ID wraparound problem will be completely solved.

Transaction ID References
#

The Internals of PostgreSQL

https://www.interdb.jp/pg/pgsql05.html

https://www.interdb.jp/pg/pgsql06.html

https://www.slideshare.net/masahikosawada98/introduction-vauum-freezing-xid-wraparound?from_action=save

https://www.modb.pro/db/427012

https://www.modb.pro/db/377530

https://www.postgresql.org/docs/13/routine-vacuuming.html

https://blog.csdn.net/weixin_30916255/article/details/112365965

https://wiki.postgresql.org/wiki/FullTransactionId

https://www.bookstack.cn/read/aliyun-rds-core/bd7e1c1955b35f7d.md

https://github.com/digoal/blog/blob/master/201605/20160520_01.md

Transaction-Related Tuple Structure
#

The tuple structure contains much of the information essential to PostgreSQL’s MVCC. The following sections cover xmin, xmax, t_ctid, cmin, cmax, combo CID, and tuple ID — their meanings and relationships.

Physical Structure
#

HeapTupleHeaderData is the tuple header. Its structure is defined in src/include/access/htup_details.h:

typedef struct HeapTupleFields
{
	TransactionId t_xmin;		/* transaction ID of inserter */
	TransactionId t_xmax;		/* transaction ID of deleter or locker */

	union
	{
		CommandId	t_cid;		/* command ID of insert or delete */
		TransactionId t_xvac;	/* VACUUM FULL transaction ID */
	}			t_field3;
} HeapTupleFields;

typedef struct DatumTupleFields
{
...
} DatumTupleFields;

struct HeapTupleHeaderData
{
	union
	{
		HeapTupleFields t_heap;
		DatumTupleFields t_datum;
	}			t_choice;
	ItemPointerData t_ctid;		/* TID of current tuple or updated tuple */
...
};

Five definitions in HeapTupleHeaderData are critically important to MVCC. (Here, “x” = transaction, “c” = command, “t” = tuple — helpful for categorization.)

t_xmin: the transaction ID that inserted this tuple.
t_xmax: the transaction ID that deleted this tuple, or the transaction ID that rolled back. If the tuple has not been deleted or updated, xmax is 0. If the delete or update was rolled back, xmax is the rolling-back transaction’s ID.
t_xvac: the transaction ID set when the tuple is vacuumed. At that point, the tuple is detached from its original transaction.
t_cid: the command ID (cid). A transaction can contain multiple SQL statements. Commands within a transaction are numbered starting from 0, incrementing sequentially. CommandId is a uint32 type, supporting up to 2^32 - 1 commands. To conserve resources, and because queries don’t affect row transaction ordering, queries do not increment cid (similar to how transaction IDs are allocated).
t_ctid: stores a pointer to itself or to a newer tuple. TID identifies a tuple within a table — it is the tuple’s physical address. If a record is modified multiple times, multiple versions exist. These versions are linked via t_ctid, forming a version chain that can be followed to find the latest version.

System Columns
#

Every tuple has 6 system columns (directly queryable): tableoid, xmin, xmax, cmin, cmax, ctid. tableoid is the table’s OID and doesn’t change during queries or DML. Here we focus on the remaining 5:

lzldb=# select xmin,xmax,cmin,cmax,ctid from lzl1;

 xmin | xmax | cmin | cmax | ctid 

------+------+------+------+-------

 616 | 619 | 0 | 0 | (0,3)

cmin: the command ID that inserted the tuple.
cmax: the command ID that deleted the tuple.

xmin, xmax, and xvac are physically stored in struct HeapTupleFields. But cmin and cmax are not separate fields — they are derived from t_cid in the struct.

The source for cmin and cmax is in src/include/access/htup_details.h:

/* SetCmin is reasonably simple since we never need a combo CID */
#define HeapTupleHeaderSetCmin(tup, cid) \
do { \
	Assert(!((tup)->t_infomask & HEAP_MOVED)); \
	(tup)->t_choice.t_heap.t_field3.t_cid = (cid); \
	(tup)->t_infomask &= ~HEAP_COMBOCID; \
} while (0)

/* SetCmax must be used after HeapTupleHeaderAdjustCmax; see combocid.c */
#define HeapTupleHeaderSetCmax(tup, cid, iscombo) \
do { \
	Assert(!((tup)->t_infomask & HEAP_MOVED)); \
	(tup)->t_choice.t_heap.t_field3.t_cid = (cid); \
	if (iscombo) \
		(tup)->t_infomask |= HEAP_COMBOCID; \
	else \
		(tup)->t_infomask &= ~HEAP_COMBOCID; \
} while (0)


/*
 * HeapTupleHeaderGetRawCommandId will give you what's in the header whether
 * it is useful or not. Most code should use HeapTupleHeaderGetCmin or
 * HeapTupleHeaderGetCmax instead, but note that those Assert that you can
 * get a legitimate result, ie you are in the originating transaction!
 */
#define HeapTupleHeaderGetRawCommandId(tup) \
( \
	(tup)->t_choice.t_heap.t_field3.t_cid \
)

Combo CID
#

Before 8.3, cmin and cmax were separate. Later, considering that it’s rare for a single transaction to both insert and delete the same row, and that cmin/cmax are not needed after the transaction ends, the two were merged into a “combo command ID,” or combocid, to save header space.

combocid source: src/backend/utils/time/combocid.c

/* Key and entry structures for the hash table */
typedef struct
{
	CommandId	cmin;
	CommandId	cmax;
} ComboCidKeyData;
/* comboid structure is cmin and cmax */
static CommandId
GetComboCommandId(CommandId cmin, CommandId cmax)
{
...
/*
 * The hash table is only created the first time a combo cid is used
 */
if (comboHash == NULL)
{
	HASHCTL		hash_ctl;

	/* generate array and hash table */
	comboCids = (ComboCidKeyData *)
		MemoryContextAlloc(TopTransactionContext,
						 sizeof(ComboCidKeyData) * CCID_ARRAY_SIZE);
	sizeComboCids = CCID_ARRAY_SIZE;
	usedComboCids = 0;

	memset(&hash_ctl, 0, sizeof(hash_ctl));
...
	comboHash = hash_create("Combo CIDs",
							CCID_HASH_SIZE,
							&hash_ctl,
							HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
}
...
}

combocid is stored in a hash table. The first time a transaction uses combocid, a small block of memory is allocated to store it.

So the relationship among these command IDs is: combocid → (cmin, cmax) → (t_cid, t_cid).

Simple Relationships Among Transaction IDs and System Columns
#

With all these IDs and source code, things might seem confusing. Here’s a diagram to help understand and remember the relationships among transaction IDs, command IDs, and tuple IDs:

A First Taste of Transactions
#

Without any tools or extensions, let’s get a feel for how these system columns change during a transaction:

lzldb=# select xmin,xmax,cmin,cmax,ctid from lzl1;
 xmin | xmax | cmin | cmax | ctid 
------+------+------+------+-------
 622 | 0 | 0 | 0 | (0,1)
lzldb=# begin ;
BEGIN
lzldb=*# update lzl1 set a=2;
UPDATE 1
-- after update, xmin+1, ctid+1; a new tuple appears
lzldb=*
select xmin,xmax,cmin,cmax,ctid from lzl1; 

xmin | xmax | cmin | cmax | ctid 
------+------+------+------+-------
 623 | 0 | 0 | 0 | (0,2)

lzldb=*# rollback;
ROLLBACK
-- xmax records the rollback transaction ID
-- xmin and ctid return to old values; the old tuple barely changes
lzldb=# select xmin,xmax,cmin,cmax,ctid from lzl1;
 xmin | xmax | cmin | cmax | ctid 
------+------+------+------+-------
 622 | 623 | 0 | 0 | (0,1)
lzldb=# update lzl1 set a=2;
UPDATE 1
-- update again; tuple number jumps over 2 directly to 3
lzldb=# select xmin,xmax,cmin,cmax,ctid from lzl1;
 xmin | xmax | cmin | cmax | ctid 
------+------+------+------+-------
 624 | 0 | 0 | 0 | (0,3)

Tuple Header and Transactions
#

The pageinspect Extension
#

Simply looking at row changes won’t show old tuples. You need the pageinspect extension. pageinspect is a contrib module bundled with PostgreSQL that can display the detailed contents of data pages. To observe how tuples support transactions, we’ll use get_raw_page() and heap_page_items().

get_raw_page(): returns the binary content of a specified block. The fork parameter accepts main, fsm, vm, or init. main is the main data file; fsm is the free space map; vm is the visibility map; init is the initialization fork. Defaults to main if not specified.

heap_page_items(): displays all line pointers on a heap page, including rows invisible under MVCC.

Generally, get_raw_page() is passed as a parameter to heap_page_items() to display tuple headers, pointers, and the data itself.

heap_tuple_infomask_flags: converts decimal infomask/infomask2 values into their meanings (flags), outputting two columns: all individual flags and combined flags. (Infomask is covered later.)

lzldb=# create extension pageinspect;
CREATE EXTENSION
lzldb=# select t_xmin,t_xmax,t_field3 as t_cid,t_ctid from heap_page_items(get_raw_page('lzl1',0));
 t_xmin | t_xmax | t_cid | t_ctid 
--------+--------+-------+--------
 633 | 0 | 0 | (0,1)

lp (Line Pointer)
#

A line pointer is essentially a row pointer number within a page, marking a tuple’s location. t_ctid looks more like a tuple ID, but ctid is simply the combination of (table page number, line pointer number). ctid can point to the next lp.

For example, after one UPDATE, a new tuple is added. The new tuple’s lp number increments by 1, the old tuple’s ctid points to the new tuple’s lp, and the new tuple’s ctid points to itself:

lzldb=# select lp,t_ctid from heap_page_items(get_raw_page('lzl1',0));
 lp | t_ctid 
----+--------
 1 | (0,1)
(1 row)

lzldb=# update lzl1 set a=2;
UPDATE 1
lzldb=# select lp,t_ctid from heap_page_items(get_raw_page('lzl1',0));
 lp | t_ctid 
----+--------
 1 | (0,2)
 2 | (0,2)

lp source: src/include/storage/itemid.h. The ItemIdData struct stores the tuple’s offset, state, and length:

typedef struct ItemIdData
{
	unsigned	lp_off:15,		/* tuple offset within the page */
				lp_flags:2,		/* lp state */
				lp_len:15;		/* tuple length */
} ItemIdData;
typedef ItemIdData *ItemId;*

*
/* lp_off:15 is a bit-field; lp_off occupies 15 bits of the unsigned. The 3 fields together total 32 bits. So ItemIdData is an int, 4 bytes, 32 bits. */

lp_flags defines 4 states:

/*
 *lp_flags has these possible states. An UNUSED line pointer is available
 *for immediate re-use, the other states are not.
 */
#define LP_UNUSED		0		/* lp not in use, tuple length lp_len always 0 */
#define LP_NORMAL		1		/* lp in use, tuple length lp_len always > 0 */
#define LP_REDIRECT		2		/* HOT redirect to another lp (should have lp_len=0) */
#define LP_DEAD			3		/* dead lp, vacuumable */

lzldb=# select lp,lp_flags,lp_off,lp_len from heap_page_items(get_raw_page('lzl1',0));
 lp | lp_flags | lp_off | lp_len 
----+----------+--------+--------
 1 | 1 | 8160 | 28

Infomask
#

Infomask provides information about transactions, locks, tuple state, etc. — such as committed, aborted, lock, HOT info, and more. There are two infomask fields in the header: infomask and infomask2. They store different information.

infomask and infomask2
#

infomask source is in src/include/access/htup_details.h:

#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK2 2
	uint16		t_infomask2;	/* number of attributes + various flags */

#define FIELDNO_HEAPTUPLEHEADERDATA_INFOMASK 3
	uint16		t_infomask;		/* various flag bits, see below */

infomask Flag Meanings
#

/*
 * information stored in t_infomask:
 */
#define HEAP_HASNULL			0x0001	/* tuple has null values */
#define HEAP_HASVARWIDTH		0x0002	/* tuple has variable-width attributes, e.g. varchar */
#define HEAP_HASEXTERNAL		0x0004	/* tuple has TOAST storage */
#define HEAP_HASOID_OLD			0x0008	/* tuple has OID */
#define HEAP_XMAX_KEYSHR_LOCK	0x0010	/* tuple has FOR KEY SHARE lock */
#define HEAP_COMBOCID			0x0020	/* t_cid is a combo CID */
#define HEAP_XMAX_EXCL_LOCK		0x0040	/* tuple has FOR UPDATE lock */
#define HEAP_XMAX_LOCK_ONLY		0x0080	/* xmax is only a locker */

 /* xmax is a shared locker */
#define HEAP_XMAX_SHR_LOCK	(HEAP_XMAX_EXCL_LOCK | HEAP_XMAX_KEYSHR_LOCK)

#define HEAP_LOCK_MASK	(HEAP_XMAX_SHR_LOCK | HEAP_XMAX_EXCL_LOCK | \
						 HEAP_XMAX_KEYSHR_LOCK)
#define HEAP_XMIN_COMMITTED		0x0100	/* inserting transaction committed */
#define HEAP_XMIN_INVALID		0x0200	/* inserting transaction invalid or aborted */
#define HEAP_XMIN_FROZEN		(HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID)
#define HEAP_XMAX_COMMITTED		0x0400	/* deleting transaction committed */
#define HEAP_XMAX_INVALID		0x0800	/* deleting transaction invalid or aborted */
#define HEAP_XMAX_IS_MULTI		0x1000	/* t_xmax is a MultiXactId */
#define HEAP_UPDATED			0x2000	/* this is an updated version of a row */
#define HEAP_MOVED_OFF			0x4000	/* moved elsewhere by pre-9.0 VACUUM FULL; kept for binary upgrade compatibility */
#define HEAP_MOVED_IN			0x8000	/* moved from elsewhere, opposite of HEAP_MOVED_OFF; kept for compatibility */
#define HEAP_MOVED (HEAP_MOVED_OFF | HEAP_MOVED_IN)
#define HEAP_XACT_MASK			0xFFF0	/* visibility-related bits */

infomask2 Flag Meanings
#

#define HEAP_NATTS_MASK			0x07FF	/* 11 bits for the number of columns (MaxHeapAttributeNumber is 1600) */
/* bits 0x1800 are available */
#define HEAP_KEYS_UPDATED		0x2000	/* tuple updated or deleted */
#define HEAP_HOT_UPDATED		0x4000	/* tuple updated, new tuple is HOT */
#define HEAP_ONLY_TUPLE			0x8000	/* HOT tuple */
#define HEAP2_XACT_MASK			0xE000	/* visibility-related bits */
#define HEAP_TUPLE_HAS_MATCH	HEAP_ONLY_TUPLE 
/* flag temporarily used in Hash Join, only for Hash table tuples that don't need visibility info; we can reuse a visibility flag instead of a separate bit */

infomask Bit Calculation
#

Converting hex to binary makes it easier to understand the bit meanings:

-- convert hex 1600 to binary
lzldb=# select x'1600'::bit(16);
 bit 
------------------
 0001011000000000

infomask:

0000000000000001 0x0001 HEAP_HASNULL			
0000000000000010 0x0002 HEAP_HASVARWIDTH		
0000000000000100 0x0004 HEAP_HASEXTERNAL		
0000000000001000 0x0008 HEAP_HASOID_OLD			
0000000000010000 0x0010 HEAP_XMAX_KEYSHR_LOCK	
0000000000100000 0x0020 HEAP_COMBOCID
0000000001000000 0x0040 HEAP_XMAX_EXCL_LOCK
0000000010000000 0x0080 HEAP_XMAX_LOCK_ONLY		
0000000001010000 0x0050 HEAP_XMAX_SHR_LOCK bitwise OR: (HEAP_XMAX_EXCL_LOCK | HEAP_XMAX_KEYSHR_LOCK)=10|40=50
0000000001010000 0x0050 HEAP_LOCK_MASK bitwise OR: (HEAP_XMAX_SHR_LOCK | HEAP_XMAX_EXCL_LOCK | HEAP_XMAX_KEYSHR_LOCK)=50|40|10=50
0000000100000000 0x0100 HEAP_XMIN_COMMITTED		
0000001000000000 0x0200 HEAP_XMIN_INVALID		
0000001100000000 0x0300 HEAP_XMIN_FROZEN bitwise OR: (HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID)=100|200=300
0000010000000000 0x0400 HEAP_XMAX_COMMITTED		
0000100000000000 0x0800 HEAP_XMAX_INVALID		
0001000000000000 0x1000 HEAP_XMAX_IS_MULTI		
0010000000000000 0x2000 HEAP_UPDATED			
0100000000000000 0x4000 HEAP_MOVED_OFF			
1000000000000000 0x8000 HEAP_MOVED_IN			
1100000000000000 0xC000 HEAP_MOVED bitwise OR: (HEAP_MOVED_OFF | HEAP_MOVED_IN)=4000|8000=C000
1111111111110000 0xFFF0 HEAP_XACT_MASK

infomask2:

0000011111111111 0x07FF HEAP_NATTS_MASK PostgreSQL max columns is 1600 = 0000011001000000, so 11 bits suffice for column count
0001100000000000 0x1800 available bits, apparently unused
0010000000000000 0x2000 HEAP_KEYS_UPDATED 
0100000000000000 0x4000 HEAP_HOT_UPDATED 
1000000000000000 0x8000 HEAP_ONLY_TUPLE 
1110000000000000 0xE000 HEAP2_XACT_MASK

How to Compute Infomask?
#

Infomask flags are hexadecimal. pageinspect returns them as decimal. Use to_hex() to convert from decimal to hexadecimal:

lzldb=# select lp,t_ctid,to_hex(t_infomask) infomask,to_hex(t_infomask2) infomask2 from heap_page_items(get_raw_page('lzl1',0));
 lp | t_ctid | infomask | infomask2 
----+--------+----------+-----------
 1 | (0,1) | 2b00 | 1

infomask=2b00 — still a bit opaque. Convert to binary and match against the flag meanings: 0010101100000000 = HEAP_UPDATED + HEAP_XMAX_INVALID + HEAP_XMIN_FROZEN.

Meaning: the tuple was updated, xmax is invalid (0), xmin is frozen (visible to all transactions).

infomask2=1 — the first 11 bits of binary (first 2047 in decimal, for up to 1600 columns) represent the number of user columns. So 1 means the tuple has only 1 column.

Manually computing infomask is tedious. Starting from pg13, pageinspect provides the heap_tuple_infomask_flags function to decode infomask and infomask2. Individual bits are shown as raw_flags; combined multi-bit flags are shown as combined_flags:

lzldb=# SELECT t_ctid, raw_flags, combined_flags
 FROM heap_page_items(get_raw_page('lzl1', 0)),
 LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2)
 WHERE t_infomask IS NOT NULL OR t_infomask2 IS NOT NULL;
 t_ctid | raw_flags | combined_flags 
--------+------------------------------------------------------------------------+--------------------
 (0,1) | {HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID,HEAP_UPDATED} | {HEAP_XMIN_FROZEN}

Commit Log (CLOG)
#

PostgreSQL uses the commit log (CLOG) to store transaction status. PostgreSQL writes the transaction to WAL before completion — that’s what WAL means. If a transaction aborts, its status is written to both WAL and CLOG so that during instance recovery, PostgreSQL knows the transaction was not committed.

When transaction status is needed — for example, when determining visibility — PostgreSQL reads the CLOG.

Transaction status

Source: src/include/access/clog.h

#define TRANSACTION_STATUS_IN_PROGRESS		0x00
#define TRANSACTION_STATUS_COMMITTED		0x01
#define TRANSACTION_STATUS_ABORTED			0x02
#define TRANSACTION_STATUS_SUB_COMMITTED	 0x03

The CLOG defines four transaction states: IN_PROGRESS, COMMITTED, ABORTED, SUB_COMMITTED.

Transaction status size

Source: src/backend/access/transam/clog.c

/* We need two bits per xact, so four xacts fit in a byte */
#define CLOG_BITS_PER_XACT	2
#define CLOG_XACTS_PER_BYTE 4
#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE)
#define CLOG_XACT_BITMASK	((1 << CLOG_BITS_PER_XACT) - 1)

Transaction status is very small — only 2 bits per transaction. One byte can store 4 transaction states. A standard page can hold 8K * 4 = 32,768 transaction states.

CLOG persistence

When PostgreSQL shuts down or checkpoints, CLOG data is written to the pg_clog directory. In version 10.0 and later, pg_clog was renamed to pg_xact.

[pg@lzl pg_xact]$ ll
total 8
-rw------- 1 pg pg 8192 Mar 28 23:33 0000

On disk, CLOG files are named 0000, 0001, etc. CLOG files are 256KB in size, while in-memory pages storing transaction states are 8KB. So the 0000 file’s size will always be a multiple of 8192. After 32 CLOG pages are written, the next page goes into the 0001 file. PostgreSQL reads transaction states from pg_xact into memory at startup.

During system operation, not all transaction states need to be retained in CLOG files forever, so VACUUM periodically deletes no-longer-needed CLOG files.

Hint Bits
#

What Are Hint Bits?
#

Hint bits mark whether the transaction that created or deleted a row has committed or aborted. Without hint bits, determining transaction visibility requires accessing on-disk pg_clog or pg_subtrans — a relatively expensive operation. If a tuple has hint bits set, you can determine the tuple’s state just by reading the page — no extra access needed.

The source code uses SetHintBits() to set hint bits:

SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED,
			InvalidTransactionId);

SetHintBits only sets 2 bits in infomask, for 4 hint bit flags (these 2 bits also combine into HEAP_XMIN_FROZEN — it’s clear that hint bits exist purely to mark transaction state):

#define HEAP_XMIN_COMMITTED	0x0100	/* inserting or updating transaction committed */
#define HEAP_XMIN_INVALID		0x0200	/* inserting or updating transaction invalid or aborted */
#define HEAP_XMAX_COMMITTED		0x0400	/* deleting or updating transaction committed */
#define HEAP_XMAX_INVALID		0x0800	/* deleting or updating transaction invalid or aborted */

Queries Can Cause Writes
#

When a transaction starts, PostgreSQL DML transactions record the transaction ID and status (like t_xmin) in the tuple header. But when the transaction ends, nothing is done to the header. Instead, a subsequent DML, DQL, or VACUUM that scans the relevant tuple triggers SetHintBits (this happens in HeapTupleSatisfiesMVCC() when a new snapshot accesses data — we’ll cover visibility rules later).

Before SetHintBits is triggered, PostgreSQL looks up transaction status in the CLOG. After SetHintBits is triggered, it reads the hint bits in the data page’s tuple header.

For example, an INSERT statement:

lzldb=# insert into lzl1 values(1);
INSERT 0 1
lzldb=# SELECT t_ctid, raw_flags, combined_flags
lzldb-# FROM heap_page_items(get_raw_page('lzl1', 0)),
lzldb-# LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2)
lzldb-# WHERE t_infomask IS NOT NULL OR t_infomask2 IS NOT NULL;
 t_ctid | raw_flags | combined_flags 
--------+---------------------+----------------
 (0,1) | {HEAP_XMAX_INVALID} | {}
(1 row)

lzldb=# select * from lzl1; -- just a single query
a 
---
 1
(1 row)
lzldb=# SELECT t_ctid, raw_flags, combined_flags
 FROM heap_page_items(get_raw_page('lzl1', 0)),
 LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2)
 WHERE t_infomask IS NOT NULL OR t_infomask2 IS NOT NULL;
 t_ctid | raw_flags | combined_flags 
--------+-----------------------------------------+----------------
 (0,1) | {HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID} | {} 

After one query, t_infomask changed — the tuple header changed.

After INSERT, SetHintBits only had HEAP_XMAX_INVALID, because INSERT only updates xmin. Whether the transaction commits or aborts (exits or rolls back), xmax is unused and can be set to HEAP_XMAX_INVALID along with the transaction.

But the transaction may commit or abort (exit/rollback). Since transaction completion does not update the tuple, HEAP_XMIN_COMMITTED cannot be set upon completion. During visibility checking (heapam_visibility.c), the visibility check updates the transaction state by calling SetHintBits on t_infomask. Thus, the query updated HEAP_XMIN_COMMITTED.

Hint bits advantage: completing (or failing) data modifications in a transaction produces no writes to the tuple. Commit and rollback are very fast.

Hint bits disadvantage: if a transaction updates many rows, the next query performing visibility checks may need to read transaction states from pg_clog and update many pages.

Do Hint Bits Generate WAL?
#

When checksums are enabled or wal_log_hints is true, if the first operation to make a page dirty after a checkpoint is updating hint bits, a WAL record is generated — specifically, a Full Page Image — to prevent partial writes that would cause checksum mismatches.

Therefore, with checksums enabled or wal_log_hints set to true, even a SELECT can modify page hint bits, which may generate WAL — increasing WAL storage to some extent. If you observe SELECT triggering disk writes, check whether CHECKSUM or wal_log_hints is enabled.

Why Are Hint Bits Deferred?
#

In src/backend/access/heap/heapam_visibility.c, within the HeapTupleSatisfiesMVCC() visibility function, a comment explains why hint bits are deferred:

/*
*While insert/delete operations are still running, hint bits on tuples are not updated,
*even if the transaction has committed or aborted.
*In high-concurrency scenarios, sharing data structures can cause contention,
*and this doesn't affect visibility decisions anyway.
*Hint bits are only set the first time a fresh snapshot accesses data after transaction completion.
*So HeapTupleSatisfiesMVCC always runs TransactionIdIsCurrentTransactionId and XidInMVCCSnapshot
*to determine whether the tuple belongs to the current transaction.
*In older versions, PostgreSQL tried to update hint bits immediately (even while transactions were running),
*but this caused more contention on the PGXACT array.
*/

Simply put: immediate hint bit updates perform very poorly. So transaction status is first stored in CLOG to reduce PGXACT contention and improve performance. Deferred hint bits are why later queries may update tuple headers.

Tuple DML Operations
#

Now that we’ve built up knowledge of tuple headers, system columns, CLOG, and hint bits, let’s see how PostgreSQL performs INSERT, UPDATE, and DELETE.

Observing DML Transactions
#

We’ll observe PostgreSQL’s DML transaction behavior by examining tuple header fields: lp, lp_flags, ctid, xmin, xmax, cid (cmin, cmax), infomask, and infomask2.

We’ll use the following query:

select t_ctid,lp,case lp_flags when 0 then '0:LP_UNUSED' when 1 then 'LP_NORMAL' when 2 then 'LP_REDIRECT' when 3 then 'LP_DEAD' end as lp_flags,t_xmin,t_xmax,t_field3 as t_cid, raw_flags, info.combined_flags from heap_page_items(get_raw_page('lzl1',0)) item,LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2) info order by lp;

(A side note: some sources like to write SELECT '(0,'||lp||')' AS ctid. This is misleading — lp and ctid are different things. lp is like a row number; ctid points to a line pointer number. lp can be different from ctid.)

For readability, create a view:

create view vlzl1 as select t_ctid,lp,case lp_flags when 0 then '0:LP_UNUSED' when 1 then 'LP_NORMAL' when 2 then 'LP_REDIRECT' when 3 then 'LP_DEAD' end as lp_flags,t_xmin,t_xmax,t_field3 as t_cid, raw_flags, info.combined_flags from heap_page_items(get_raw_page('lzl1',0)) item,LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2) info order by lp;

Now the query looks like:

lzldb=# \x
Expanded display is on.
lzldb=# select * from vlzl1;
-[ RECORD 6 ]--+-------
t_ctid | (0,6)
lp | 6
lp_flags | LP_NORMAL
t_xmin | 653
t_xmax | 0
t_cid | 0
raw_flags | {HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE}
combined_flags | {}

INSERT
#

Truncate the table, then insert a row:

lzldb=# begin ;
BEGIN
lzldb=*# insert into lzl1 values(1);
INSERT 0 1
lzldb=*# insert into lzl1 values(2);
INSERT 0 1
lzldb=*# commit;
lzldb=# select * from vlzl1;
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+---------------------+----------------
 (0,1) | 1 | LP_NORMAL | 664 | 0 | 0 | {HEAP_XMAX_INVALID} | {}
 (0,2) | 2 | LP_NORMAL | 664 | 0 | 1 | {HEAP_XMAX_INVALID} | {}

ctid points to (page 0, lp 1), i.e., to itself.
lp (line pointer number) increments.
Both tuples share the same xmin — they were inserted by the same transaction.
xmax is 0 (invalid transaction ID). Infomask only indicates xmax is invalid: this tuple has not yet “experienced” a delete transaction.
cid increments from 0: 0 for the first command, 1 for the second.

DELETE
#

lzldb=# begin;
BEGIN
lzldb=*# delete from lzl1 where a=1;
DELETE 1
lzldb=*# commit;
COMMIT
lzldb=# select * from vlzl1;
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+-----------------------------------------+----------------
 (0,1) | 1 | LP_NORMAL | 664 | 665 | 0 | {HEAP_XMIN_COMMITTED,HEAP_KEYS_UPDATED} | {}
 (0,2) | 2 | LP_NORMAL | 664 | 0 | 1 | {HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID} | {}

The first tuple was deleted. The tuple wasn’t physically removed — only a few attributes were marked:

ctid unchanged, still points to itself.
xmax updated to the delete transaction ID.
Infomask shows HEAP_KEYS_UPDATED, indicating the tuple was deleted (actually, HEAP_KEYS_UPDATED means either deleted or updated).
Although only the first tuple was modified, the second tuple’s infomask was also updated with HEAP_XMIN_COMMITTED.

UPDATE
#

lzldb=# begin;
BEGIN
lzldb=# update lzl1 set a=3;
UPDATE 1
lzldb=*# commit;
COMMIT
lzldb=# select * from vlzl1;
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+-------------------------------------------------------------+----
 (0,1) | 1 | LP_NORMAL | 664 | 665 | 0 | {HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_KEYS_UPDATED} | {}
 (0,3) | 2 | LP_NORMAL | 664 | 666 | 0 | {HEAP_XMIN_COMMITTED,HEAP_HOT_UPDATED} | {}
 (0,3) | 3 | LP_NORMAL | 666 | 0 | 0 | {HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {}

An UPDATE doesn’t modify the tuple in place. Instead, it marks the old tuple as unavailable and inserts a new one:

lp=2 is the old tuple from the update transaction. t_xmax is the update transaction ID. Infomask adds HEAP_HOT_UPDATED, indicating the tuple is HOT. ctid points to the new tuple.
lp=3 is the new tuple from the update. It’s equivalent to an inserted tuple, but xmin matches the old tuple’s xmax. Infomask has the extra flag HEAP_UPDATED, indicating this is the updated version.
Additionally, the invisible deleted tuple at lp=1 had its infomask updated with HEAP_XMAX_COMMITTED by an unrelated subsequent update transaction.

Rollback
#

lzldb=# truncate table lzl1;
TRUNCATE TABLE
lzldb=# begin;
BEGIN
lzldb=*# insert into lzl1 values(1); -- INSERT
INSERT 0 1
lzldb=*# select * from vlzl1;
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+---------------------+----------------
 (0,1) | 1 | LP_NORMAL | 679 | 0 | 0 | {HEAP_XMAX_INVALID} | {}
(1 row)
lzldb=*# rollback; -- INSERT rolled back
ROLLBACK
lzldb=# select * from vlzl1;
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+---------------------+----------------
 (0,1) | 1 | LP_NORMAL | 679 | 0 | 0 | {HEAP_XMAX_INVALID} | {}
lzldb=# select * from lzl1;
 a 
---
(0 rows)
-- After INSERT and rollback, the tuple header shows no changes.

lzldb=# insert into lzl1 values(2);
INSERT 0 1
lzldb=# begin ;
BEGIN
lzldb=*# delete from lzl1 ; -- DELETE
DELETE 1
lzldb=*# select * from vlzl1;
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+-----------------------------------------+----------------
 (0,1) | 1 | LP_NORMAL | 684 | 0 | 0 | {HEAP_XMIN_INVALID,HEAP_XMAX_INVALID} | {}
 (0,2) | 2 | LP_NORMAL | 685 | 686 | 0 | {HEAP_XMIN_COMMITTED,HEAP_KEYS_UPDATED} | {}
(2 rows)

lzldb=*# rollback; -- DELETE rolled back
ROLLBACK
lzldb=# select * from vlzl1;
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+-----------------------------------------+----------------
 (0,1) | 1 | LP_NORMAL | 684 | 0 | 0 | {HEAP_XMIN_INVALID,HEAP_XMAX_INVALID} | {}
 (0,2) | 2 | LP_NORMAL | 685 | 686 | 0 | {HEAP_XMIN_COMMITTED,HEAP_KEYS_UPDATED} | {}
-- After DELETE and rollback, the tuple header shows no changes.

lzldb=*# update lzl1 set a=100 ; -- UPDATE
UPDATE 1
lzldb=*# select * from vlzl1; 
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+--------------------------------------------------+---------------
 (0,1) | 1 | LP_NORMAL | 684 | 0 | 0 | {HEAP_XMIN_INVALID,HEAP_XMAX_INVALID} | {}
 (0,3) | 2 | LP_NORMAL | 685 | 688 | 0 | {HEAP_XMIN_COMMITTED,HEAP_HOT_UPDATED} | {}
 (0,3) | 3 | LP_NORMAL | 688 | 0 | 0 | {HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {}
(3 rows)
lzldb=*# rollback; -- UPDATE rolled back
ROLLBACK
lzldb=*# select * from vlzl1; 
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+--------------------------------------------------+---------------
 (0,1) | 1 | LP_NORMAL | 684 | 0 | 0 | {HEAP_XMIN_INVALID,HEAP_XMAX_INVALID} | {}
 (0,3) | 2 | LP_NORMAL | 685 | 688 | 0 | {HEAP_XMIN_COMMITTED,HEAP_HOT_UPDATED} | {}
 (0,3) | 3 | LP_NORMAL | 688 | 0 | 0 | {HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {}
-- After UPDATE and rollback, the tuple header shows no changes.

When a transaction rolls back, tuple information does not change at all. This is why PostgreSQL’s MVCC doesn’t worry about running out of rollback segments — rollback is purely a visibility operation, not a data update.
xmax doesn’t change after rollback either, which means a non-zero xmax doesn’t necessarily indicate the tuple was deleted — the delete or update transaction may have rolled back.
However, once visibility checking occurs, even without data changes, all tuples’ infomask will be updated with HEAP_XMIN_INVALID. Non-HOT tuples get HEAP_XMIN_INVALID, and HOT-referenced tuples naturally get it too.

References for Tuple and Transaction
#

Books:

The Internals of PostgreSQL
PostgreSQL in Action
PostgreSQL Internals: Deep Dive into Transaction Processing
PostgreSQL Database Kernel Analysis

https://edu.postgrespro.com/postgresql_internals-14_parts1-2_en.pdf

Official resources:

https://en.wikipedia.org/wiki/Concurrency_control

https://wiki.postgresql.org/wiki/Hint_Bits

https://www.postgresql.org/docs/current/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND

https://www.postgresql.org/docs/10/storage-page-layout.html

https://www.postgresql.org/docs/13/pageinspect.html3

Essential PostgreSQL transaction reads (interdb):

https://www.interdb.jp/pg/pgsql05.html

https://www.interdb.jp/pg/pgsql06.html

Source code experts:

https://blog.csdn.net/Hehuyi_In/article/details/102920988

https://blog.csdn.net/Hehuyi_In/article/details/127955762

https://blog.csdn.net/Hehuyi_In/article/details/125023923

PostgreSQL snapshot optimization performance comparison:

https://techcommunity.microsoft.com/t5/azure-database-for-postgresql/improving-postgres-connection-scalability-snapshots/ba-p/1806462

Other resources:

https://brandur.org/postgres-atomicity

https://mp.weixin.qq.com/s/j-8uRuZDRf4mHIQR_ZKIEg

Snapshots in PostgreSQL
#

A snapshot is a data structure that records the instantaneous state of the database. PostgreSQL’s snapshot stores: the minimum and maximum transaction IDs among all active transactions, the list of currently active transactions, the current transaction’s command ID, and more.

Snapshot data is stored in the SnapshotData struct type. Source: src/include/utils/snapshot.h

typedef struct SnapshotData
{
	SnapshotType snapshot_type; /* snapshot type */
TransactionId xmin;			/* txid < xmin are visible to the snapshot */
TransactionId xmax;			/* txid >= xmax are invisible to the snapshot */

/* list of active transactions at snapshot time. Only includes txids between xmin and xmax */
TransactionId *xip;
uint32		xcnt;			/* xip_list stored in xip[] */

/* list of active subtransactions at snapshot time */
TransactionId *subxip;
int32		subxcnt;		/* subtransactions stored in subxip[] */
bool		suboverflowed;	/* whether subtransactions overflowed; overflows occur with many subtransactions */

bool		takenDuringRecovery;	/* is this a recovery snapshot? */
bool		copied;			/* whether the snapshot is a copy (RR and serializable copy their snapshots); false if static */

CommandId	curcid;			/* command ID in the transaction; CID < curcid is visible */
...
TimestampTz whenTaken;		/* timestamp when snapshot was taken */
XLogRecPtr	lsn;			/* LSN when snapshot was taken */
} SnapshotData;
typedef struct SnapshotData *Snapshot;

The most important snapshot information is xmin, xmax, and xip_list. Use pg_current_snapshot() (in pg12 and earlier, txid_current_snapshot()) to display the current transaction’s snapshot.

Note: snapshot xmin/xmax are different from tuple xmin/xmax — they have different meanings.

lzldb=*# select pg_current_snapshot();
 pg_current_snapshot 
---------------------
 100:104:100,102

xmin	Earliest active txid. All txids older than xmin have either committed (visible) or aborted (dead tuples).
xmax	First unassigned txid. xmax = latestCompletedXid + 1. All txid >= xmax have not yet started and are invisible to the current snapshot.
xip_list	Stored in array xip[]. Since transactions can start and finish out of order (a later-started transaction may finish earlier), xmin and xmax alone cannot fully express all active transactions at snapshot time. xip_list stores the active transactions at snapshot time.

Snapshot Types
#

Beyond MVCC snapshots, PostgreSQL defines several other snapshot types in src/include/utils/snapshot.h:

typedef enum SnapshotType
{
	/* Tuple is visible if and only if it satisfies MVCC snapshot visibility rules.
 * The most important snapshot type — used to implement MVCC.
 * Tuple visibility is judged based on snapshot xmin, xmax, xip_list, curcid, etc.
 * If a command changed data, the current MVCC snapshot won't see it; a new MVCC snapshot is needed.
 */
SNAPSHOT_MVCC = 0,
/* Tuple is visible if its transaction committed.
 * In-progress transactions are invisible.
 * Data changes from the current command are visible to the SELF snapshot.
 */
SNAPSHOT_SELF,

/*
 * Any tuple is visible.
 */
SNAPSHOT_ANY,

/*
 * Visible if the TOAST tuple is valid. TOAST visibility depends on the main table tuple's visibility.
 */
SNAPSHOT_TOAST,

/*
 * Data changes from the current command are visible to the DIRTY snapshot.
 * The DIRTY snapshot preserves version info for in-progress tuples.
 * Snapshot xmin is set to the xmin of other in-progress transactions' tuples; xmax is similar.
 */
SNAPSHOT_DIRTY,

/* HISTORIC_MVCC snapshot follows MVCC rules, used for logical decoding.
 */
SNAPSHOT_HISTORIC_MVCC,

/*
 Determines whether dead tuples are visible to certain transactions.
 */
SNAPSHOT_NON_VACUUMABLE
} SnapshotType;

Snapshots and Isolation Levels
#

Different isolation levels acquire snapshots differently:

Read Committed requires a new snapshot for each SQL statement in the transaction, while Repeatable Read uses only one snapshot for the entire transaction. The function that acquires snapshots is GetTransactionSnapshot().

Process-Level Transaction Structures
#

When PostgreSQL acquires snapshot data, it needs to scan the transaction state of all backend processes.

Before understanding the GetSnapshotData() function, we need to understand several backend process structures: PGPROC, PGXACT, PROC_HDR (PROCGLOBAL), and ProcArray.

These process-related structures contain process and lock information. Here we only study the transaction-related parts. Source code examples are based on pg13.

PGPROC Struct
#

Source: src/include/storage/proc.h

// Every backend process stores a PGPROC struct in memory.
// Think of this as the backend process's main structure.
struct PGPROC
{
...
LocalTransactionId lxid;	/* local id of top-level transaction currently
								 * being executed by this proc, if running;
								 * else InvalidLocalTransactionId */
...
struct XidCache subxids;	/* cached subtransaction XIDs */
...
/* clog group transaction status update */
bool		clogGroupMember;	/* whether this proc uses clog group commit */
pg_atomic_uint32 clogGroupNext; /* atomic int, pointing to the next group member proc */
TransactionId clogGroupMemberXid;	/* xid to be committed */
XidStatus	clogGroupMemberXidStatus;	/* status of the xid to be committed */
int			clogGroupMemberPage;	/* which page the xid to be committed belongs to */
									
XLogRecPtr	clogGroupMemberLsn; /* LSN of the commit log for the xid to be committed */
};
/* NOTE: "typedef struct PGPROC PGPROC" appears in storage/lock.h. Not written with the struct itself. */

PGXACT Struct
#

// Before 9.2, PGXACT information was inside PGPROC. Stress testing showed that on multi-CPU systems,
// separating them makes GetSnapshotData faster by reducing the number of cache lines fetched.
typedef struct PGXACT
{
	TransactionId xid;			/* id of top-level transaction currently being
								 * executed by this proc, if running and XID
								 * is assigned; else InvalidTransactionId */
								// appears to be the current process's xmax

	TransactionId xmin;			/* excluding lazy vacuum; minimum xid at transaction start;
								 vacuum cannot remove tuples with xid >= xmin */
	uint8		vacuumFlags;	/* vacuum-related flags, see above */
	bool		overflowed; // whether PGXACT overflowed

	uint8		nxids;
} PGXACT;

PGXACT stores relatively simple information — the backend’s xmin, xmax, and other transaction-related fields. PGPROC leans toward storing basic backend info; some less frequently accessed transaction info remains in PGPROC, but the core process transaction info is in PGXACT.

PROC_HDR (PROCGLOBAL) Struct
#

Every backend process has a proc struct. In high-concurrency scenarios, scanning all proc structs to find transaction info is time-consuming. An instance-level structure is needed to store all proc info — this is PROCGLOBAL.

The source typically uses the struct type PROC_HDR to define a struct pointer to PROCGLOBAL. PROC_HDR stores global proc info: the full array of proc structs, free procs, etc.

Source: src/include/storage/proc.h

typedef struct PROC_HDR
{
	/* pgproc array (not including dummies for prepared txns) */
	PGPROC	 *allProcs;
	/* pgxact array (not including dummies for prepared txns) */
	PGXACT	 *allPgXact;
	...
	/* Current shared estimate of appropriate spins_per_delay value */
	int			spins_per_delay;
	/* The proc of the Startup process, since not in ProcArray */
	PGPROC	 *startupProc;
	int			startupProcPid;
	/* Buffer id of the buffer that Startup process waits for pin on, or -1 */
	int			startupBufferPinWaitBufId;
} PROC_HDR;

ProcArray Struct
#

ProcArray is in procarray.c, which maintains the PGPROC and PGXACT structures for all backends.

Source: src/backend/storage/ipc/procarray.c

typedef struct ProcArrayStruct
{
	int			numProcs;		/* number of procs */
	int			maxProcs;		/* size of proc array */

	// handling assigned xids
	int			maxKnownAssignedXids;	/* allocated size of array */
	int			numKnownAssignedXids;	/* current # of valid entries */
	int			tailKnownAssignedXids;	/* index of oldest valid element */
	int			headKnownAssignedXids;	/* index of newest element, + 1 */
	slock_t		known_assigned_xids_lck;	/* protects head/tail pointers */

	/*
	 * Highest subxid that has been removed from KnownAssignedXids array to
	 * prevent overflow; or InvalidTransactionId if none. We track this for
	 * similar reasons to tracking overflowing cached subxids in PGXACT
	 * entries. Must hold exclusive ProcArrayLock to change this, and shared
	 * lock to read it.
	 */
	TransactionId lastOverflowedXid;

	/* oldest xmin of any replication slot */
	TransactionId replication_slot_xmin;
	/* oldest catalog xmin of any replication slot */
	TransactionId replication_slot_catalog_xmin;

	/* pgprocnos, equivalent to allPgXact[] array indices, used to look up allPgXact[]; this array has PROCARRAY_MAXPROCS entries */
	int			pgprocnos[FLEXIBLE_ARRAY_MEMBER];
} ProcArrayStruct;
static ProcArrayStruct *procArray;

Acquiring a Snapshot
#

GetTransactionSnapshot()
#

Snapshots are acquired via GetTransactionSnapshot().

Source: src/backend/utils/time/snapmgr.c

// GetTransactionSnapshot() allocates the appropriate snapshot for SQL in a transaction
Snapshot
GetTransactionSnapshot(void)
{

	 // Return historic snapshot if doing logical decoding. We'll never need a
	 // non-historic snapshot after this, so return directly.
	if (HistoricSnapshotActive())
	{
		Assert(!FirstSnapshotSet);
		return HistoricSnapshot;
	}

	/* If it's not the first call in this transaction, enter this if */
	if (!FirstSnapshotSet)
	{
		/*
		 * Ensure the catalog snapshot is fresh.
		 */
		InvalidateCatalogSnapshot();

		Assert(pairingheap_is_empty(&RegisteredSnapshots));
		Assert(FirstXactSnapshot == NULL);
	// Return error if in parallel mode
		if (IsInParallelMode())
			elog(ERROR,
				 "cannot take query snapshot during a parallel operation");

		 // For Repeatable Read or Serializable, use the same snapshot for the entire transaction; only copy once
		 // IsolationUsesXactSnapshot() means the isolation level is RR or Serializable — they use one snapshot per transaction
		if (IsolationUsesXactSnapshot())
		{
			// First, create the snapshot in CurrentSnapshotData
			// If SI isolation level, initialize SSI-required data structures
			if (IsolationIsSerializable()) 
				CurrentSnapshot = GetSerializableTransactionSnapshot(&CurrentSnapshotData);
			else
				CurrentSnapshot = GetSnapshotData(&CurrentSnapshotData);
			/* Make a saved copy */
			/* For Repeatable Read or Serializable, this snapshot lasts the entire transaction; copy once */
			CurrentSnapshot = CopySnapshot(CurrentSnapshot);
			FirstXactSnapshot = CurrentSnapshot;
			/* Mark it as "registered" in FirstXactSnapshot */
			FirstXactSnapshot->regd_count++;
			pairingheap_add(&RegisteredSnapshots, &FirstXactSnapshot->ph_node);
		}
		else
			// For Read Committed, acquire a snapshot
			CurrentSnapshot = GetSnapshotData(&CurrentSnapshotData);

// Modify flag to indicate this is the first snapshot; subsequent calls in this transaction won't enter this if
		FirstSnapshotSet = true;
		return CurrentSnapshot;
	}

// If not the first call in this transaction (already have a first snapshot)
// For Repeatable Read or Serializable, return a copy of the first snapshot
	if (IsolationUsesXactSnapshot())
		return CurrentSnapshot;

	/* Don't allow catalog snapshot to be older than xact snapshot. */
	InvalidateCatalogSnapshot();
	// Read Committed: re-acquire snapshot
	CurrentSnapshot = GetSnapshotData(&CurrentSnapshotData);

	return CurrentSnapshot;
}

About IsolationUsesXactSnapshot() and IsolationIsSerializable():

Defined as macros in src/include/access/xact.h:

#define XACT_READ_UNCOMMITTED	0
#define XACT_READ_COMMITTED	1
#define XACT_REPEATABLE_READ	2
#define XACT_SERIALIZABLE	3
// Internally only 3 isolation levels: 1, 2, 3
// 2 isolation levels use one snapshot per transaction; others use one snapshot per SQL statement
#define IsolationUsesXactSnapshot() (XactIsoLevel >= XACT_REPEATABLE_READ)
#define IsolationIsSerializable() (XactIsoLevel == XACT_SERIALIZABLE)

IsolationUsesXactSnapshot() is true for Repeatable Read or Serializable.

IsolationIsSerializable() is true for Serializable only.

GetTransactionSnapshot() flow chart:

(image from CSDN: https://blog.csdn.net/Hehuyi_In)

The main logic of GetTransactionSnapshot():

For historic snapshots during logical decoding, return the snapshot result directly.
For Repeatable Read or Serializable: on the first call, return the snapshot and copy it so subsequent calls (non-first) can directly reference it.
For Read Committed: generate a new snapshot on every call.
For the first call in Serializable, additionally acquire SSI data information.
GetTransactionSnapshot() acquires the snapshot; the actual data comes from GetSnapshotData().

GetSnapshotData()
#

Source: src/backend/storage/ipc/procarray.c

Snapshot
GetSnapshotData(Snapshot snapshot)
{
	// Initialize some variables: arrayP pointer, procarray, xmin, xmax, replication slot txid, etc.
	ProcArrayStruct *arrayP = procArray;
	TransactionId xmin;
	TransactionId xmax;
	TransactionId globalxmin;
	int			index;
	int			count = 0;
	int			subcount = 0;
	bool		suboverflowed = false;
	TransactionId replication_slot_xmin = InvalidTransactionId;
	TransactionId replication_slot_catalog_xmin = InvalidTransactionId;

	Assert(snapshot != NULL);

	if (snapshot->xip == NULL)
	{
		/*
		 * First call for this snapshot. Snapshot is same size whether or not
		 * we are in recovery, see later comments.
		 */
		snapshot->xip = (TransactionId *) // get current transaction's xip
			malloc(GetMaxSnapshotXidCount() * sizeof(TransactionId));
		...
		Assert(snapshot->subxip == NULL);
		snapshot->subxip = (TransactionId *) // get current subtransaction's subxip
			malloc(GetMaxSnapshotSubxidCount() * sizeof(TransactionId));
		...
	}

	// Acquire procarray; need shared LWLock
	LWLockAcquire(ProcArrayLock, LW_SHARED);

	/* xmax = max completed xid + 1 */
	xmax = ShmemVariableCache->latestCompletedXid;
	Assert(TransactionIdIsNormal(xmax));
	TransactionIdAdvance(xmax); // xmax + 1

	/* xmax value retrieved; xmin needs scanning pgproc, pgxact, procarray */
	/* Set globalxmin and xmin to xmax first; if backends have no transaction info, this is simpler */
	globalxmin = xmin = xmax; 
	
	// Recovery snapshots handled separately
	snapshot->takenDuringRecovery = RecoveryInProgress();
	// Non-recovery snapshots need transaction info from backends
	if (!snapshot->takenDuringRecovery)
	{
		int		 *pgprocnos = arrayP->pgprocnos;
		int			numProcs;

		/*
		 * Spin over procArray checking xid, xmin, and subxids. The goal is
		 * to gather all active xids, find the lowest xmin, and try to record
		 * subxids. It appears that while scanning procarray, it will spin
		 * to collect all active xids, the smallest xmin, and subtransaction subxids.
		 */
		numProcs = arrayP->numProcs;
		for (index = 0; index < numProcs; index++)
		{
			int			pgprocno = pgprocnos[index]; // iterate numProcs, get all pgprocno indices
			PGXACT	 *pgxact = &allPgXact[pgprocno]; // iterate all pgxact structs via pgprocno
			TransactionId xid;
			...
			/* Update globalxmin to be the smallest valid xmin */
			xid = UINT32_ACCESS_ONCE(pgxact->xmin);
			if (TransactionIdIsNormal(xid) &&
				NormalTransactionIdPrecedes(xid, globalxmin))
				globalxmin = xid;

			/* Fetch xid just once - see GetNewTransactionId */
			xid = UINT32_ACCESS_ONCE(pgxact->xid);
			...
			/* Save backend's xmin into snapshot xip */
			/* i.e., iterate all pgxact to find all active xids */
			snapshot->xip[count++] = xid;
			...
			/* Subtransaction info handling */
			if (!suboverflowed) // if subtransaction hasn't overflowed
			{
				if (pgxact->overflowed)
					suboverflowed = true; // if transaction overflowed, mark subtransaction as overflowed too
				else
				{
					int			nxids = pgxact->nxids;

					if (nxids > 0)
					{
						PGPROC	 *proc = &allProcs[pgprocno];

						pg_read_barrier();	/* pairs with GetNewTransactionId */

						memcpy(snapshot->subxip + subcount,
							 (void *) proc->subxids.xids,
							 nxids * sizeof(TransactionId));
						subcount += nxids;
					}
				}
			}
		}
	}
	else // the else corresponds to if (!snapshot->takenDuringRecovery)
	{
		// These checks are for standby; when the instance is in hot standby mode and queries run on the replica
		subcount = KnownAssignedXidsGetAndSetXmin(snapshot->subxip, &xmin,
												 xmax);

		if (TransactionIdPrecedesOrEquals(xmin, procArray->lastOverflowedXid))
			suboverflowed = true;
	}


	// Replication slot xmin and catalog cluster-wide xmin, first save to local variables
	// Replication slot xmin prevents tuple reclamation
	// The comment says this is to avoid holding ProcArrayLock for too long, so save to local variables
	replication_slot_xmin = procArray->replication_slot_xmin;
	replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
	
	// Backend transaction info gathering is done; below is a series of ifs for cleanup and code robustness
	if (!TransactionIdIsValid(MyPgXact->xmin))
		MyPgXact->xmin = TransactionXmin = xmin;

	LWLockRelease(ProcArrayLock); // release ProcArrayLock

	if (TransactionIdPrecedes(xmin, globalxmin))
		globalxmin = xmin; // globalxmin and process xmin: assign globalxmin to the smaller one

	RecentGlobalXmin = globalxmin - vacuum_defer_cleanup_age;
	if (!TransactionIdIsNormal(RecentGlobalXmin))
		RecentGlobalXmin = FirstNormalTransactionId; // edge case: if RecentGlobalXmin <= 2, assign 3

	/* Check whether there's a replication slot requiring an older xmin. */
	if (TransactionIdIsValid(replication_slot_xmin) &&
		NormalTransactionIdPrecedes(replication_slot_xmin, RecentGlobalXmin))
		RecentGlobalXmin = replication_slot_xmin;

	/* Non-catalog tables can be vacuumed if older than this xid */
	RecentGlobalDataXmin = RecentGlobalXmin;

	// Re-check and compare catalog, globalxmin
	if (TransactionIdIsNormal(replication_slot_catalog_xmin) &&
		NormalTransactionIdPrecedes(replication_slot_catalog_xmin, RecentGlobalXmin))
		RecentGlobalXmin = replication_slot_catalog_xmin;

	RecentXmin = xmin;
	
	// Start assigning values to the snapshot struct, returning snapshot data
	snapshot->xmin = xmin;
	snapshot->xmax = xmax;
	snapshot->xcnt = count;
	snapshot->subxcnt = subcount;
	snapshot->suboverflowed = suboverflowed;

	snapshot->curcid = GetCurrentCommandId(false);

	// If it's a new snapshot, initialize some snapshot info
	snapshot->active_count = 0;
	snapshot->regd_count = 0;
	snapshot->copied = false;
	
	// Snapshot-too-old logic below; oddly written here
	if (old_snapshot_threshold < 0)
	{
		/*
		 * If not using "snapshot too old" feature, fill related fields with
		 * dummy values that don't require any locking.
		 */
		// When old_snapshot_threshold < 0 (no "snapshot too old" issue)
		// assign simple constant values that won't require any locks
		snapshot->lsn = InvalidXLogRecPtr;
		snapshot->whenTaken = 0;
	}
	else
	{
		// When old_snapshot_threshold >= 0, need to handle old snapshot logic
		snapshot->lsn = GetXLogInsertRecPtr(); // get LSN
		snapshot->whenTaken = GetSnapshotCurrentTimestamp(); // get snapshot timestamp
		MaintainOldSnapshotTimeMapping(snapshot->whenTaken, xmin); //
		// GetXLogInsertRecPtr(), GetSnapshotCurrentTimestamp(), MaintainOldSnapshotTimeMapping() 
		// all contain SpinLockAcquire and SpinLockRelease
		// MaintainOldSnapshotTimeMapping() also has LWLockAcquire and LWLockRelease
		// Since this is called for every snapshot, GetSnapshotData should be very frequent
		// So in pg13 source, setting old_snapshot_threshold to negative avoids many spinlocks and lwlocks
	}

	return snapshot;
}

pg14 Snapshot Optimizations
#

pg14 Optimization Source Analysis
#

From the pg13 source, we can see that GetSnapshotData() hardcodes old_snapshot_threshold >= 0, causing each snapshot acquisition to incur many SpinLock and LWLock operations. Since snapshot acquisition is extremely frequent, this inevitably causes performance issues. So pg14 simply removed the old_snapshot_threshold logic from GetSnapshotData().

Beyond that removal, pg14 made many other optimizations:

Removed RecentGlobalXmin and RecentGlobalDataXmin, added the GlobalVisTest* family of functions.

Introduced the boundaries concept with two boundaries: definitely_needed and maybe_needed:

struct GlobalVisState
{
	/* XIDs >= are considered running by some backend */
	// rows with XID >= definitely_needed are definitely visible
	FullTransactionId definitely_needed;

	/* XIDs < are not considered to be running by any backend */
	// rows with XID < maybe_needed can definitely be cleaned up
	FullTransactionId maybe_needed;
};

Added ComputeXidHorizons() for more precise horizon calculation (storing xmin and removable xid information). This function still needs to iterate PGPROC. The calculation range is XID >= maybe_needed && XID < definitely_needed.

Added GlobalVisTestShouldUpdate() to determine whether boundaries need recalculation.

First, understand the variable ComputeXidHorizonsResultLastXmin:

static TransactionId ComputeXidHorizonsResultLastXmin; // last precisely computed xmin

GlobalVisTestShouldUpdate(GlobalVisState *state)
{
	// If xmin=0, need to recalculate boundaries. This is an edge case for tuples created during database initialization.
	if (!TransactionIdIsValid(ComputeXidHorizonsResultLastXmin))
		return true;

	/*
	 * If the maybe_needed/definitely_needed boundaries are the same, it's
	 * unlikely to be beneficial to refresh boundaries.
	 */
	// When maybe_needed equals definitely_needed, no need to recalculate
	// Uses FullTransactionIdFollowsOrEquals (not strict equality)
	// "Greater than" scenario: no rows definitely visible. "Equal" scenario: only one row definitely visible.
	if (FullTransactionIdFollowsOrEquals(state->maybe_needed,
										 state->definitely_needed))
		return false;

	/* does the last snapshot built have a different xmin? */
	// When the last snapshot's xmin equals the last precisely computed xmin, no need to recalculate boundaries
	return RecentXmin != ComputeXidHorizonsResultLastXmin;
}

We can see that maybe_needed and definitely_needed are similar to snapshot xmin/xmax, but with an additional layer of computation. First calculate boundaries, then further refine with ComputeXidHorizons(). GlobalVisTestShouldUpdate reduces the scenarios where boundaries need recalculation, and ComputeXidHorizons() is also more efficient for precise calculation.

Optimization Results
#

Recommended article on PostgreSQL snapshot optimization:

https://techcommunity.microsoft.com/t5/azure-database-for-postgresql/improving-postgres-connection-scalability-snapshots/ba-p/1806462

The before-and-after comparison is striking:

In pg13 production environments, GetSnapshotData consistently shows high performance overhead. (No screenshot, so I’ll borrow another expert’s chart:)

Snapshot References
#

Books:

The Internals of PostgreSQL
PostgreSQL in Action
PostgreSQL Internals: Deep Dive into Transaction Processing
PostgreSQL Database Kernel Analysis

https://edu.postgrespro.com/postgresql_internals-14_parts1-2_en.pdf

Official resources:

https://en.wikipedia.org/wiki/Concurrency_control

https://wiki.postgresql.org/wiki/Hint_Bits

https://www.postgresql.org/docs/current/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND

https://www.postgresql.org/docs/10/storage-page-layout.html

https://www.postgresql.org/docs/13/pageinspect.html3

Essential PostgreSQL transaction reads (interdb):

https://www.interdb.jp/pg/pgsql05.html

https://www.interdb.jp/pg/pgsql06.html

Source code experts:

https://blog.csdn.net/Hehuyi_In/article/details/102920988

https://blog.csdn.net/Hehuyi_In/article/details/127955762

https://blog.csdn.net/Hehuyi_In/article/details/125023923

PostgreSQL snapshot optimization performance comparison:

https://techcommunity.microsoft.com/t5/azure-database-for-postgresql/improving-postgres-connection-scalability-snapshots/ba-p/1806462

Other resources:

https://brandur.org/postgres-atomicity

https://mp.weixin.qq.com/s/j-8uRuZDRf4mHIQR_ZKIEg

Visibility Checking
#

With a snapshot, we can determine tuple visibility. Let’s review the key information (ignoring subtransactions for now): tuple header transaction info, snapshot info, and CLOG transaction status (before SetHintBits).

Tuple header has: xmin, xmax, cmin, cmax, infomask, etc.
Snapshot data has: snapshot xmin, xmax, xip_list, curcid, etc.
CLOG has additional transaction status info, which may also be written to infomask as hint bits.

Different snapshot types have slightly different visibility rules:

bool
HeapTupleSatisfiesVisibility(HeapTuple tup, Snapshot snapshot, Buffer buffer)
{
	switch (snapshot->snapshot_type)
	{
		case SNAPSHOT_MVCC:
			return HeapTupleSatisfiesMVCC(tup, snapshot, buffer);
			break;
		...
		case SNAPSHOT_NON_VACUUMABLE:
			return HeapTupleSatisfiesNonVacuumable(tup, snapshot, buffer);
			break;
	}
	...
}

Each snapshot type has its own visibility rules. Here we’ll use the most common SNAPSHOT_MVCC visibility rules to understand tuple visibility.

static bool
HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot,
				 Buffer buffer)
{
	HeapTupleHeader tuple = htup->t_data; 

	Assert(ItemPointerIsValid(&htup->t_self)); // lp valid, i.e., tuple valid
	Assert(htup->t_tableOid != InvalidOid); // oid valid, i.e., table valid

	// t_xmin not committed: the transaction that INSERTed or UPDATEd this new tuple has not committed
	// In htup_details.h, macro: HeapTupleHeaderXminCommitted() is ((tup)->t_infomask & HEAP_XMIN_COMMITTED) != 0
	// So if (!HeapTupleHeaderXminCommitted(tuple)) means the tuple infomask does not have HEAP_XMIN_COMMITTED
	// Literally: t_xmin has not committed
	if (!HeapTupleHeaderXminCommitted(tuple)) 
	{
		// If a transaction updated the tuple but then aborted or failed, this tuple's xmin is the failed transaction ID
		// If the inserting transaction failed, directly return invisible
		if (HeapTupleHeaderXminInvalid(tuple))
			return false;
		
		// When infomask has HEAP_MOVED_OFF, visibility is judged separately for VACUUM tuples, with hint bits set
		/* Used by pre-9.0 binary upgrades */
		if (tuple->t_infomask & HEAP_MOVED_OFF)
		{
			TransactionId xvac = HeapTupleHeaderGetXvac(tuple);

			if (TransactionIdIsCurrentTransactionId(xvac))
				return false;
			if (!XidInMVCCSnapshot(xvac, snapshot))
			{
				if (TransactionIdDidCommit(xvac))
				{
					SetHintBits(tuple, buffer, HEAP_XMIN_INVALID,
								InvalidTransactionId);
					return false;
				}
				SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED,
							InvalidTransactionId);
			}
		}
		// When infomask has HEAP_MOVED_IN, visibility is judged separately for VACUUM tuples, with hint bits set
		/* Used by pre-9.0 binary upgrades */
		else if (tuple->t_infomask & HEAP_MOVED_IN)
		{
			TransactionId xvac = HeapTupleHeaderGetXvac(tuple);

			if (!TransactionIdIsCurrentTransactionId(xvac))
			{
				if (XidInMVCCSnapshot(xvac, snapshot))
					return false;
				if (TransactionIdDidCommit(xvac))
					SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED,
								InvalidTransactionId);
				else
				{
					SetHintBits(tuple, buffer, HEAP_XMIN_INVALID,
								InvalidTransactionId);
					return false;
				}
			}
		}
		// When the tuple was written by the current transaction
		else if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmin(tuple)))
		{
			if (HeapTupleHeaderGetCmin(tuple) >= snapshot->curcid) // tuple cid >= snapshot current command id
				return false;	// tuple was inserted after visibility check started; invisible

			if (tuple->t_infomask & HEAP_XMAX_INVALID) // infomask has HEAP_XMAX_INVALID
				return true; // tuple not deleted; visible
				// A pure insert, whether committed, not yet committed, or rolled back, has HEAP_XMAX_INVALID
				// But this check is under the "written by current transaction" condition, so:
				// Tuple inserted by current transaction, not committed (logically equivalent to not deleted within the same tx),
				// and t_cid < curcid → visible

			// xmax is set in two scenarios: 1) tuple locked, 2) tuple deleted
			// Even without HEAP_XMAX_INVALID, the tuple may not be deleted — it may just be locked
			// Locked tuples have xmax set but are visible
			if (HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask))	/* not deleter */
				return true;
			
			// HEAP_XMAX_IS_MULTI is set when multiple transactions acquire locks on the same row, producing MultiXactId
			// Still judging visibility under xmax lock scenarios
			if (tuple->t_infomask & HEAP_XMAX_IS_MULTI)
			{
				TransactionId xmax;

				xmax = HeapTupleGetUpdateXid(tuple);

				/* not LOCKED_ONLY, so it has to have an xmax */
				Assert(TransactionIdIsValid(xmax));

				/* updating subtransaction must have aborted */
				// If xmax is not the current transaction, visible
				if (!TransactionIdIsCurrentTransactionId(xmax))
					return true;
				// If xmax is the current transaction, judge by command id:
				// snapshot acquired before update/delete → tuple was visible at snapshot time
				else if (HeapTupleHeaderGetCmax(tuple) >= snapshot->curcid)
					return true;	/* updated after scan started */
				else
					return false;	/* updated before scan started */
			}
			
			// The following scenario: a subtransaction's delete command was rolled back, need SetHintBits HEAP_XMAX_INVALID
			// Delete rolled back, so tuple is visible
			if (!TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmax(tuple)))
			{
				/* deleting subtransaction must have aborted */
				SetHintBits(tuple, buffer, HEAP_XMAX_INVALID,
							InvalidTransactionId);
				return true;
			}

			// cmax is the command ID that deleted the tuple
			// If tuple cmax >= snapshot curcid: delete happened after snapshot scan → visible
			// If tuple cmax < snapshot curcid: delete happened before snapshot scan → invisible
			if (HeapTupleHeaderGetCmax(tuple) >= snapshot->curcid)
				return true;	/* deleted after scan started */
			else
				return false;	/* deleted before scan started */
		}
		// XidInMVCCSnapshot() checks if xid was in-progress at snapshot time
		// "in-progress" means: 1. snapshot xmin <= xid < snapshot xmax AND xid in xip_list 2. xid >= snapshot xmax
		// The xid below is t_xmin
		// So this means: if t_xmin was in-progress at snapshot time → invisible
		// Equivalent to: t_xmin not committed → invisible. This seems redundant.
		// Because this whole block is under !HeapTupleHeaderXminCommitted(tuple) — also meaning t_xmin not committed.
		// But with the preceding checks, this else if is reasonable. Meaning:
		// t_xmin not committed, tuple not deleted, not current transaction → invisible
		else if (XidInMVCCSnapshot(HeapTupleHeaderGetRawXmin(tuple), snapshot))
			return false;
			
		// If t_xmin transaction committed, SetHintBits HEAP_XMIN_COMMITTED
		// This seems odd: the entire block is for t_xmin NOT committed, how could it be committed here?
		// And if this case really happens, why no visibility judgment?
		else if (TransactionIdDidCommit(HeapTupleHeaderGetRawXmin(tuple)))
			SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED,
						HeapTupleHeaderGetRawXmin(tuple));
		// If t_xmin transaction did not commit, SetHintBits HEAP_XMIN_INVALID
		else
		{
			/* it must have aborted or crashed */
			SetHintBits(tuple, buffer, HEAP_XMIN_INVALID,
						InvalidTransactionId);
		// t_xmin transaction not committed, return invisible again. Similar to XidInMVCCSnapshot() above?
		// Currently: not committed, and doesn't satisfy XidInMVCCSnapshot() (xid was not in-progress at snapshot time)
		// The only case: transaction hadn't started at snapshot time, later started, still not committed → invisible
		return false; 
		}
	}
	// xmin-not-committed visibility judgments finally done
	// Everything after the else is for when xmin IS committed (hint bit HEAP_XMIN_COMMITTED is set)
	else
	{
		// xmin is committed, but maybe not according to our snapshot
		/* xmin is committed, but maybe not according to our snapshot */
		// If infomask has no HEAP_XMIN_FROZEN AND xmin was in-progress at snapshot time → invisible
		// Translating the if: at snapshot time, xmin was not committed; at visibility check time,
		// tuple xmin is committed but not marked FROZEN → invisible
		// Even though tuple xmin is now committed, from the current snapshot's perspective it was still in-progress
		if (!HeapTupleHeaderXminFrozen(tuple) &&
			XidInMVCCSnapshot(HeapTupleHeaderGetRawXmin(tuple), snapshot))
			return false;		/* treat as still in progress */
	}

	// HEAP_XMAX_INVALID means tuple not deleted
	// This if means: tuple committed, and was committed at snapshot time, and not deleted (no delete marker at all) → visible
	if (tuple->t_infomask & HEAP_XMAX_INVALID)	/* xid invalid or aborted */
		return true;
	
	// Tuple has xmax, but it's not a delete — it's a lock marker
	// This if means: tuple committed, was committed at snapshot time, has xmax but xmax is a lock → visible
	if (HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask))
		return true;
	
	// HEAP_XMAX_IS_MULTI means the tuple is in shared-row-lock state, typically when multiple transactions process one row
	if (tuple->t_infomask & HEAP_XMAX_IS_MULTI)
	{
		TransactionId xmax;

		/* already checked above */
		Assert(!HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));

		// Get the transaction ID that updated the tuple
		xmax = HeapTupleGetUpdateXid(tuple);

		/* not LOCKED_ONLY, so it has to have an xmax */
		Assert(TransactionIdIsValid(xmax));

		// If the shared-row-lock tuple's transaction ID is the current transaction
		if (TransactionIdIsCurrentTransactionId(xmax))
		{
			// tuple cmax >= snapshot curcid: tuple not yet deleted at snapshot time → visible
			if (HeapTupleHeaderGetCmax(tuple) >= snapshot->curcid)
				return true;	/* deleted after scan started */
			// tuple cmax < snapshot curcid: tuple already deleted at snapshot time → invisible
			else
				return false;	/* deleted before scan started */
		}
		// If the shared-row-lock tuple's transaction ID is not the current transaction, and xmax was in-progress at snapshot time
		// This if means: xmin committed, tuple not deleted, MULTI XMAX marker present, xmax not yet committed at snapshot time → visible
		if (XidInMVCCSnapshot(xmax, snapshot))
			return true;
		// If the shared-row-lock tuple transaction committed → invisible
		if (TransactionIdDidCommit(xmax))
			return false;		/* updating transaction committed */
		/* it must have aborted or crashed */
		// Updating transaction aborted or crashed → still visible
		return true;
	}
	
	// Tuple xmin committed, xmax not yet marked committed, not yet deleted
	// Seems !HEAP_XMAX_COMMITTED differs from HEAP_XMAX_INVALID
	// This looks like: tuple experienced a delete, but the delete transaction hasn't committed
	// While HEAP_XMAX_INVALID above is: definitely no delete or delete aborted/rolled back, so can directly return true
	if (!(tuple->t_infomask & HEAP_XMAX_COMMITTED))
	{
		// If xmax is the same as the checking transaction
		if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmax(tuple)))
		{
			// Same old pattern: visibility via command id
			// cmax >= snapshot curcid: delete happened after snapshot → visible
			if (HeapTupleHeaderGetCmax(tuple) >= snapshot->curcid)
				return true;	/* deleted after scan started */
				
			// cmax < snapshot curcid: delete happened before snapshot → invisible
			else
				return false;	/* deleted before scan started */
		}
		
		// Delete transaction not committed, and xmax not the checking transaction
		// If xmax was in-progress at snapshot time → visible
		if (XidInMVCCSnapshot(HeapTupleHeaderGetRawXmax(tuple), snapshot))
			return true;
		
		// Confirm xmax delete transaction aborted or failed; SetHintBits HEAP_XMAX_INVALID
		// Similar to HEAP_XMAX_INVALID above → visible
		if (!TransactionIdDidCommit(HeapTupleHeaderGetRawXmax(tuple)))
		{
			/* it must have aborted or crashed */
			SetHintBits(tuple, buffer, HEAP_XMAX_INVALID,
						InvalidTransactionId);
			return true;
		}

		/* xmax transaction committed */
		// Remaining case: xmax delete transaction committed. SetHintBits HEAP_XMAX_COMMITTED
		// Visibility should be judged here, but it's deferred to the last few lines, because this is a sub-case of a larger condition
		SetHintBits(tuple, buffer, HEAP_XMAX_COMMITTED,
					HeapTupleHeaderGetRawXmax(tuple));
	}
	else
	{
		/* xmax is committed, but maybe not according to our snapshot */
		// xmax delete transaction now committed, but was in-progress at snapshot time → visible
		if (XidInMVCCSnapshot(HeapTupleHeaderGetRawXmax(tuple), snapshot))
			return true;		/* treat as still in progress */
	}

	/* xmax transaction committed */
	// Only remaining case: xmax committed and was not in-progress at snapshot time → invisible
	return false;
}

The entire visibility judgment source code looks complex. Stripping out the SetHintBits parts and the convoluted if-else chains, focusing only on the core visibility rules, the key points are:

Core visibility rule logic:
- Delete committed → tuple invisible
- Insert committed, delete rolled back → tuple visible
- Insert committed, delete not committed → current transaction compares cid; other transactions see the tuple as visible
- Insert rolled back → tuple invisible
- Insert not committed → same transaction compares cmin; other transactions see the tuple as invisible
Visibility checking involves two time points: the check time and the snapshot time. The logic distinguishes between the same transaction (checking transaction = snapshot transaction) and different transactions.
- Same transaction: compare tuple cmin/cmax against snapshot->curcid.
  - cmin >= snapshot->curcid: tuple inserted after snapshot → invisible. Otherwise visible.
  - cmax >= snapshot->curcid: tuple deleted after snapshot → visible. Otherwise invisible.
- Different transactions: use XidInMVCCSnapshot() to check whether xid (t_xmin or t_xmax) was in-progress at snapshot time.
  - xmin was in-progress at snapshot time → invisible.
  - xmax was in-progress at snapshot time → visible.
Beyond basic DML operations, there are 4 additional cases:
- VACUUM tuple insert/delete visibility
- Lock-only marker (HEAP_XMAX_IS_LOCKED_ONLY): tuple visible
- MultiXact state (HEAP_XMAX_IS_MULTI): visibility for tuples under multi-transaction locks
- Frozen tuples: visibility when frozen marker is set

MultiXact
#

What Is MultiXact?
#

When multiple transactions lock the same row, there may be multiple associated transaction IDs on the tuple. PostgreSQL groups multiple transaction IDs together and manages them with a single MultiXactId. The relationship between TransactionId and MultiXactId is many-to-one.

Like TransactionId, MultiXactId is also 32-bit and also subject to wraparound.

MultiXactId values 0 and 1 are reserved for system use. Allocatable MultiXactIds start from 2.

Source: src/include/access/multixact.h
#define InvalidMultiXactId	((MultiXactId) 0)
#define FirstMultiXactId	((MultiXactId) 1)
#define MaxMultiXactId		((MultiXactId) 0xFFFFFFFF)

Row Lock Types
#

MultiXact only exists when rows are locked. MultiXact defines 6 states:

typedef enum
{
MultiXactStatusForKeyShare = 0x00,
MultiXactStatusForShare = 0x01,
MultiXactStatusForNoKeyUpdate = 0x02,
MultiXactStatusForUpdate = 0x03,
/* an update that doesn't touch "key" columns */
MultiXactStatusNoKeyUpdate = 0x04,
/* other updates, and delete */
MultiXactStatusUpdate = 0x05
} MultiXactStatus;

There are 4 explicitly declarable row lock states: ForKeyShare, ForShare, ForNoKeyUpdate, ForUpdate.

MultiXact Infomask Flags
#

PostgreSQL marks row locks on xmax and records them in infomask.

Source: src/include/access/htup_details.h

#define HEAP_XMAX_KEYSHR_LOCK	0x0010	/* xmax is a key-shared locker */
#define HEAP_XMAX_EXCL_LOCK		0x0040	/* xmax is exclusive locker */
#define HEAP_XMAX_LOCK_ONLY		0x0080	/* xmax, if valid, is only a locker */
#define HEAP_XMAX_SHR_LOCK	(HEAP_XMAX_EXCL_LOCK | HEAP_XMAX_KEYSHR_LOCK)
#define HEAP_LOCK_MASK	(HEAP_XMAX_SHR_LOCK | HEAP_XMAX_EXCL_LOCK | \
						 HEAP_XMAX_KEYSHR_LOCK)
#define HEAP_XMAX_IS_MULTI		0x1000	/* t_xmax is a MultiXactId */

Here we focus on the HEAP_XMAX_IS_MULTI flag. Only when multiple transactions hold shared locks on the same row is a true MultiXact ID generated and this flag set.

lzldb=# insert into lzl1 values(1); -- initially one row
INSERT 0 1
lzldb=# select * from vlzl1;
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+----------------------------------+----------------
 (0,1) | 1 | LP_NORMAL | 742 | 0 | 0 | {HEAP_HASNULL,HEAP_XMAX_INVALID} | {}
(1 row)

Session 1	Session 2
lzldb=# begin; BEGIN lzldb=# select from lzl1 for share; a — 1
	lzldb=# begin; BEGIN lzldb=# select from lzl1 for share; a — 1
lzldb=*# update lzl1 set a=2; –hang
	commit；
UPDATE 1 –update completed

-- Check tuple xmax and infomask
lzldb=*# select t_ctid,lp,t_xmin,t_xmax,(t_infomask&4096)!=0 is_multixact from heap_page_items(get_raw_page('lzl1',0));
 t_ctid | lp | t_xmin | t_xmax | is_multixact 
--------+----+--------+--------+--------------
 (0,2) | 1 | 742 | 4 | t
 (0,2) | 2 | 744 | 3 | t

HEAP_XMAX_IS_MULTI is 0x1000 in hex, which is 4096 in decimal. Using (t_infomask&4096)!=0 is_multixact shows whether the tuple uses a MultiXact ID. From the example:

MultiXact IDs have their own value space, separate from transaction IDs.
MultiXact IDs are generally smaller than transaction IDs — here t_xmax < t_xmin.
For an UPDATE, old and new tuples typically share the same xmax. In MultiXact scenarios, they may differ.

MultiXact SLRU
#

Although src/backend/access/transam/multixact.c defines many variables and functions at the top — page, member, membergroup, offset — they are all about defining variable values and conversion functions between them.

Before reading multixact.c, understand a few macros:

src/include/c.h defines MultiXactOffset as a 32-bit type:

typedef uint32 MultiXactOffset;

src/include/access/slru.h defines how many SLRU pages per segment:

#define SLRU_PAGES_PER_SEGMENT	32

Back to the top of src/backend/access/transam/multixact.c:

define MULTIXACT_OFFSETS_PER_PAGE (BLCKSZ / sizeof(MultiXactOffset)) 
// MULTIXACT_OFFSETS_PER_PAGE = 8k / 32B = 2048. One page stores 2048 offset markers, i.e., 2048 MultiXactIds.

#define MultiXactIdToOffsetPage(xid) \
	((xid) / (MultiXactOffset) MULTIXACT_OFFSETS_PER_PAGE)
// Convert xid to the page where the corresponding record resides: xid / 2048

#define MultiXactIdToOffsetEntry(xid) \
	((xid) % (MultiXactOffset) MULTIXACT_OFFSETS_PER_PAGE)
// Convert xid to the offset within the page: xid % 2048

#define MultiXactIdToOffsetSegment(xid) (MultiXactIdToOffsetPage(xid) / SLRU_PAGES_PER_SEGMENT)
// Convert xid to the segment: xid / 2048 / 32

Now read the comments at the top of the source:

/* 
 * Defines for MultiXactOffset page sizes. A page is the same BLCKSZ as is
 * used everywhere else in Postgres.
 *
 * Note: because MultiXactOffsets are 32 bits and wrap around at 0xFFFFFFFF,
 * MultiXact page numbering also wraps around at
 * 0xFFFFFFFF/MULTIXACT_OFFSETS_PER_PAGE, and segment numbering at
 * 0xFFFFFFFF/MULTIXACT_OFFSETS_PER_PAGE/SLRU_PAGES_PER_SEGMENT. We need
 * take no explicit notice of that fact in this module, except when comparing
 * segment and page numbers in TruncateMultiXact (see
 * MultiXactOffsetPagePrecedes).
 */

Since MultiXactOffsets are 32-bit and subject to wraparound:

MultiXact page numbering wraps at 0xFFFFFFFF / MULTIXACT_OFFSETS_PER_PAGE = 2^32 / 2048 = 2^21
Segment numbering wraps at 0xFFFFFFFF / MULTIXACT_OFFSETS_PER_PAGE / SLRU_PAGES_PER_SEGMENT = 2^32 / 2^11 / 2^5 = 2^16

TruncateMultiXact() cleans up these segments and page numbers. It is called by VACUUM.

The pg_multixact Directory
#

Like CLOG and SUBTRANS, MultiXact logs use an SLRU buffer pool implementation. The pg_multixact directory has only two subdirectories: members and offsets.

[pg@lzl pg_multixact]$ ll
total 8
drwx------ 2 pg pg 4096 Feb 14 21:29 members
drwx------ 2 pg pg 4096 Feb 14 21:29 offsets

One MultiXactId corresponds to multiple TransactionIds — the members. The offset is the starting position of each MultiXact.

typedef struct mXactCacheEnt
{
	MultiXactId multi; // one MultiXactId
	int		nmembers;
	dlist_node	node;
	MultiXactMember members[FLEXIBLE_ARRAY_MEMBER]; // multiple TransactionIds; expanded via MultiXactIdExpand() if needed
} mXactCacheEnt;

multixact.h defines MultiXactMember as just a single transaction ID and its status:

typedef struct MultiXactMember
{
	TransactionId xid;
	MultiXactStatus status;
} MultiXactMember;

MultiXact References
#

https://www.postgresql.org/docs/current/routine-vacuuming.html

https://pgpedia.info/m/multixact-id.html

https://www.postgresql.org/docs/15/explicit-locking.html

https://www.modb.pro/db/14939

https://www.highgo.ca/2020/06/12/transactions-in-postgresql-and-their-mechanism/

Two-Phase Commit (2PC) Transactions
#

What Is a 2PC Transaction?
#

Transaction atomicity requires that a transaction either completes entirely or rolls back entirely. In distributed transactions spanning multiple connected databases, a consistent state must be provided to satisfy distributed transaction atomicity. Like other databases, PostgreSQL provides the Two-Phase Commit Protocol (2PC).

There are many distributed transaction implementations; 2PC is the most fundamental and common. Distributed transactions encompass atomic commit, atomic visibility, and global consistency. 2PC is only an implementation for atomic commit.

PREPARE TRANSACTION
#

Foreign Data Wrappers (FDWs) can handle 2PC internally. PostgreSQL also provides an explicit way to use 2PC: PREPARE TRANSACTION. Once issued, the prepared transaction is detached from the session; its state is persisted. PREPARE TRANSACTION is not designed for use in applications or interactive sessions — unless you’re writing a transaction manager — so it is recommended (and default) to keep it disabled.

Syntax:

PREPARE TRANSACTION transaction_id
COMMIT PREPARED transaction_id 
ROLLBACK PREPARED transaction_id

Notes:

The transaction_id here is not the internal transaction ID — it’s just a user-declared string.
PREPARE TRANSACTION must be inside a transaction block, started with BEGIN or START TRANSACTION.
max_prepared_transactions controls the number of prepared transactions. Default is 0 (disabled). Must be increased to use prepared transactions.

Starting a Prepared Transaction
#

lzldb=# begin;
BEGIN
lzldb=*# PREPARE TRANSACTION 'lzl';
PREPARE TRANSACTION

lzldb=# select * from pg_prepared_xacts ;
 transaction | gid | prepared | owner | database 
-------------+-----+-------------------------------+-------+----------
 719 | lzl | 2023-04-29 16:08:45.866022+08 | pg | lzldb
(1 row)

lzldb=# rollback prepared 'lzl';
ROLLBACK PREPARED 

lzldb=# select * from pg_prepared_xacts ;
 transaction | gid | prepared | owner | database 
-------------+-----+----------+-------+----------
(0 rows)

The pg_twophase Directory
#

As mentioned, prepared transactions are session-independent. When a prepared transaction is started, its state information is stored in a cache. To ensure the transaction is not lost, prepared transactions are also persisted to the pg_twophase directory. This doesn’t only happen on shutdown — it’s tied to checkpoint.

Source: src/backend/access/transam/twophase.c

void
CheckPointTwoPhase(XLogRecPtr redo_horizon)
{
	...
	TRACE_POSTGRESQL_TWOPHASE_CHECKPOINT_START(); // checkpoint start
	...
	fsync_fname(TWOPHASE_DIR, true); // call fsync to flush to disk
	
	TRACE_POSTGRESQL_TWOPHASE_CHECKPOINT_DONE(); // checkpoint done
	...
}

Let’s test: start a prepared transaction and run a checkpoint:

[pg@lzl pg_twophase]$ ll
total 0

lzldb=*# PREPARE TRANSACTION 'lzl';
PREPARE TRANSACTION
lzldb=# checkpoint;
CHECKPOINT

[pg@lzl pg_twophase]$ ll
total 4
-rw------- 1 pg pg 116 Apr 29 16:33 000002D0

Orphaned Prepared Transactions
#

If a prepared transaction is never completed (neither committed nor rolled back), and since it is session-independent, it will persist unless explicitly terminated. (Normally, a regular transaction rolls back when the session disconnects.) This is an orphaned prepared transaction.

Orphaned prepared transactions hold locks and tuple resources indefinitely, preventing VACUUM from reclaiming dead tuples and even blocking transaction ID wraparound. For example, if a prepared transaction is forgotten and not committed or rolled back, and there is no external transaction management monitoring it, it may go unnoticed and exist forever — ultimately causing severe problems. Therefore, it’s recommended to keep max_prepared_transactions=0 (default) or monitor prepared transactions via the pg_prepared_xacts view.

Here’s a simulation of an orphaned prepared transaction causing indefinite blocking:

-- Start a prepared transaction and disconnect
lzldb=# begin;
BEGIN
lzldb=*# insert into lzl1 values(1);
INSERT 0 1
lzldb=*# PREPARE TRANSACTION 'lzl';
PREPARE TRANSACTION
lzldb=# \q

-- After disconnecting, the prepared transaction still exists
postgres=# select * from pg_prepared_xacts ;
 transaction | gid | prepared | owner | database 
-------------+-----+-------------------------------+-------+----------
 721 | lzl | 2023-04-29 17:08:59.597678+08 | pg | lzldb

-- DDL blocked
lzldb=# alter table lzl1 add column b int;
-- Check locks
lzldb=# select locktype,relation,pid,mode from pg_locks where relation=32808;
 locktype | relation | pid | mode 
----------+----------+-------+---------------------
 relation | 32808 | 26136 | AccessExclusiveLock
 relation | 32808 | | RowExclusiveLock

-- End the prepared transaction; DDL completes
lzldb=# rollback prepared 'lzl';
ROLLBACK PREPARED
lzldb=# alter table lzl1 add column b int;
ALTER TABLE

2PC Transaction References
#

http://postgres.cn/docs/13/sql-prepare-transaction.html

https://www.highgo.ca/2020/01/28/understanding-prepared-transactions-and-handling-the-orphans/

https://wiki.postgresql.org/wiki/Atomic_Commit_of_Distributed_Transactions

Subtransactions
#

What Is a Subtransaction?
#

A regular transaction can only commit or roll back as a whole. Subtransactions allow partial rollback.

SAVEPOINT p1 places a savepoint marker inside a transaction. You cannot directly commit a subtransaction — subtransactions are committed when the parent transaction commits. However, you can use ROLLBACK TO SAVEPOINT p1 to roll back to that savepoint.

Subtransactions are useful for bulk data loading. If a transaction contains multiple subtransactions and one small segment fails, only that segment needs to be retried — not the entire transaction.

Using Subtransactions in SQL
#

SAVEPOINT savepoint_name
ROLLBACK [ WORK | TRANSACTION ] TO [ SAVEPOINT ] savepoint_name
RELEASE [ SAVEPOINT ] savepoint_name

Notes:

Savepoint statements must be inside a transaction block.
SAVEPOINT creates a savepoint; ROLLBACK TO rolls back to the named savepoint; RELEASE erases the savepoint without rolling back subtransaction data.
Cursors are not affected by savepoint operations.

Example:

lzldb=# begin;
BEGIN
lzldb=*# insert into lzl1 values(0);
INSERT 0 1
lzldb=*# savepoint p1;
SAVEPOINT
lzldb=*# insert into lzl1 values(1);
INSERT 0 1
lzldb=*# savepoint p2;
SAVEPOINT
lzldb=*# insert into lzl1 values(2);
INSERT 0 1
lzldb=*# savepoint p3;
SAVEPOINT
lzldb=*# insert into lzl1 values(3);
INSERT 0 1
lzldb=*# rollback to savepoint p2;
ROLLBACK
lzldb=*# commit;
COMMIT
lzldb=# select xmin,xmax,cmin,a from lzl1;
 xmin | xmax | cmin | a 
------+------+------+---
 731 | 0 | 0 | 0
 732 | 0 | 1 | 1
(2 rows)
-- Rolling back to p2 also rolled back p3

lzldb=# select * from vlzl1;
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags 
--------+----+-----------+--------+--------+-------+------------------------------------------------------+----------------
 (0,1) | 1 | LP_NORMAL | 731 | 0 | 0 | {HEAP_HASNULL,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID} | {}
 (0,2) | 2 | LP_NORMAL | 732 | 0 | 1 | {HEAP_HASNULL,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID} | {}
 (0,3) | 3 | LP_NORMAL | 733 | 0 | 2 | {HEAP_HASNULL,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID} | {}
 (0,4) | 4 | LP_NORMAL | 734 | 0 | 3 | {HEAP_HASNULL,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID} | {}
(4 rows)
-- Subtransaction infomask is not very different from regular transactions.
-- Multiple commands within the same transaction are differentiated by cid and HEAP_XMIN_INVALID, etc.
-- Subtransaction writes also consume transaction IDs, and cid increments within the parent transaction framework.

Other Sources of Subtransactions
#

Even without explicit SAVEPOINT, subtransactions can be created by other means:

EXCEPTION blocks trigger subtransactions. This is common in tools and frameworks and easily overlooked. Every EXCEPTION creates a subtransaction.

Syntax: BEGIN / EXCEPTION WHEN .. / END

Reference: https://fluca1978.github.io/2020/02/05/PLPGSQLExceptions.html
PL/Python code using plpy.subtransaction().

Subtransaction SLRU Cache
#

Subtransaction commit logs are in pg_xact. Parent-child relationships are stored in pg_subtrans, which caches the mapping of subXID to parent XID. When PostgreSQL needs to look up a subXID, it calculates which memory page the ID resides on and searches within that page. If the page is not in cache, it evicts a page and loads the required page from pg_subtrans into memory. Large numbers of subtransaction cache misses consume system I/O and CPU.

The subtransaction buffer is only 32 pages, hardcoded in the source.

Source: src/include/access/subtrans.h

/* Number of SLRU buffers to use for subtrans */
\#define NUM_SUBTRANS_BUFFERS 32

Buffer default is 8KB; xid is 32 bits (4 bytes). Therefore:

SUBTRANS_BUFFER size: 32 * 8K = 256KB
SUBTRANS_BUFFER can store at most: 32 * 8K / 4 = 65,536 xids

Finding a subtransaction’s position in a page by transaction ID:

Source: src/backend/access/transam/subtrans.c

/* We need four bytes per xact */
#define SUBTRANS_XACTS_PER_PAGE (BLCKSZ / sizeof(TransactionId))
// Each page can store up to 8K / 4 bytes = 2048 subtransaction IDs

#define TransactionIdToPage(xid) ((xid) / (TransactionId) SUBTRANS_XACTS_PER_PAGE)
// Calculate page number from subtransaction xid: xid / 2048
#define TransactionIdToEntry(xid) ((xid) % (TransactionId) SUBTRANS_XACTS_PER_PAGE)
// Calculate offset within page from subtransaction xid: xid % 2048

Subtransaction xids may not be densely packed within a page — a page may hold fewer than 2048 subtransaction IDs.

The Dangers of Subtransactions
#

1. PGPROC_MAX_CACHED_SUBXIDS Overflow

PGPROC_MAX_CACHED_SUBXIDS is not a GUC parameter — it’s hardcoded. You can only change it by modifying the source.

Source: src/include/storage/proc.h

	/* 
	*Each backend has a subtransaction cache limit of PGPROC_MAX_CACHED_SUBXIDS.
	*We must track whether the cache has overflowed (i.e., the transaction has at least one subtransaction that couldn't be cached).
	*If no cache has overflowed, we can be sure that an xid not in the PGPROC array is definitely not a running transaction.
	*If there is an overflow, we must consult pg_subtrans.
	*/
	#define PGPROC_MAX_CACHED_SUBXIDS 64	/* XXX guessed-at value */
	
	struct XidCache
	{
		TransactionId xids[PGPROC_MAX_CACHED_SUBXIDS];
	};

Two key takeaways from this source:

Every backend’s subtransaction cache is capped at PGPROC_MAX_CACHED_SUBXIDS: 64 subtransactions.
Beyond 64 subtransactions, they overflow to the pg_subtrans directory.

An expert’s benchmark: performance drops when subtransactions just exceed 64. So it’s best to keep per-session subtransactions below 64.

Reference: https://postgres.ai/blog/20210831-postgresql-subtransactions-considered-harmful

2. Subtransactions Causing MultiXact Contention

Reference: https://buttondown.email/nelhage/archive/notes-on-some-postgresql-implementation-details/

FOR UPDATE itself is a row-level exclusive lock and should not generate a MultiXact ID. But in this scenario, multiple MultiXact waits occurred, causing a cliff-like performance drop:

LWLock:MultiXactMemberControlLock
LWLock:MultiXactOffsetControlLock
LWLock:multixact_member
LwLock:multixact_offset

It was later discovered that the Django framework was issuing subtransaction statements:

SELECT [some row] FOR UPDATE;
 SAVEPOINT save;
 UPDATE [the same row];

3. Replica Performance Cliff

Reference: https://about.gitlab.com/blog/2021/09/29/why-we-spent-the-last-month-eliminating-postgresql-subtransactions/

A single long transaction with a savepoint subtransaction can also cause a performance cliff on replicas.

If a read occurs on a snapshot taken on the primary, the snapshot includes xmin, xmax, the txip transaction list, and subxip (the list of in-progress subtransactions). However, neither the original arrays nor the snapshot are directly shared with replicas — replicas read all needed data from WAL.

When subtransactions exist, a single long-running transaction can cause replica performance to drop off a cliff:

4. Production Performance Cliff

When the database is busy and many subtransactions exist, performance can drop sharply, accompanied by subtransaction wait events. This scenario can occur even when per-session subtransactions don’t exceed 64, and even on the primary (not just replicas).

We found that a tool (OGG) defaulted to 50 subtransactions. Reducing the subtransaction count in that tool to 10–20 alleviated the database performance issue.

Subtransaction usage recommendations:

Besides explicit SAVEPOINT, EXCEPTION blocks, frameworks, and tools can also generate subtransactions.
If you have replica query workloads, disable subtransactions.
Use row locks cautiously. FOR UPDATE + subtransactions can also trigger MultiXactId issues.
If you must use subtransactions, keep them well below 64 per session — preferably much lower.

Subtransactions have caused countless production issues worldwide, with many case studies and analyses. To quote: “Subtransactions are basically cursed. Rip ’em out.”

Subtransaction References
#

https://postgres.ai/blog/20210831-postgresql-subtransactions-considered-harmful

https://www.cybertec-postgresql.com/en/subtransactions-and-performance-in-postgresql/

https://fluca1978.github.io/2020/02/05/PLPGSQLExceptions.html

https://about.gitlab.com/blog/2021/09/29/why-we-spent-the-last-month-eliminating-postgresql-subtransactions/

https://buttondown.email/nelhage/archive/notes-on-some-postgresql-implementation-details/

References
#

Books:

The Internals of PostgreSQL
PostgreSQL in Action
PostgreSQL Internals: Deep Dive into Transaction Processing
PostgreSQL Database Kernel Analysis

https://edu.postgrespro.com/postgresql_internals-14_parts1-2_en.pdf

Official resources:

https://en.wikipedia.org/wiki/Concurrency_control

https://wiki.postgresql.org/wiki/Hint_Bits

https://www.postgresql.org/docs/current/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND

https://www.postgresql.org/docs/10/storage-page-layout.html

https://www.postgresql.org/docs/13/pageinspect.html3

Essential PostgreSQL transaction reads (interdb):

https://www.interdb.jp/pg/pgsql05.html

https://www.interdb.jp/pg/pgsql06.html

Source code experts:

https://blog.csdn.net/Hehuyi_In/article/details/102920988

https://blog.csdn.net/Hehuyi_In/article/details/127955762

https://blog.csdn.net/Hehuyi_In/article/details/125023923

PostgreSQL snapshot optimization performance comparison:

https://techcommunity.microsoft.com/t5/azure-database-for-postgresql/improving-postgres-connection-scalability-snapshots/ba-p/1806462

Other resources:

https://brandur.org/postgres-atomicity

https://mp.weixin.qq.com/s/j-8uRuZDRf4mHIQR_ZKIEg

https://blog.csdn.net/postgrechina/article/details/49130743?spm=a2c6h.12873639.article-detail.7.41b32cda2KR1QM

http://mysql.taobao.org/monthly/2018/12/02/

Originally published in Chinese on lastdba.com.

PostgreSQL Localization

Mon, 12 Aug 2024 00:00:00 +0000

Localization Concepts
#

The purpose of localization is to support the language features and rules of different countries and regions. With localization support, you can use character sets that handle Chinese, French, Japanese, and more. Beyond character sets, there are also character sorting rules and other language-related rule support. For example, we know how to sort (‘a’, ‘b’), but how should (‘a’, ‘A’) and (‘啊’, ‘阿’) be sorted?

If you search Google for information about localization, character sets, and collation, you might end up with knowledge that feels both complex and distant. The best teacher is still

Localization knowledge is divided into three parts: locale support, collation, and character sets.

locale
#

PostgreSQL’s localization is provided by the operating system. You need to check whether the OS supports it via locale -a. The locale can be specified when initializing the database:

initdb --locale=en_US

You can also set localization subcategories individually: string sort order, character classification, numeric formatting, date formatting, time formatting, currency formatting, etc.

initdb --locale=zh_CN --lc-monetary=en_US

All localization subcategories:

Subcategory	Rule
LC_COLLATE	String sort order
LC_CTYPE	Character classification (What is a letter? Its upper-case equivalent?)
LC_MESSAGES	Language of messages
LC_MONETARY	Formatting of currency amounts
LC_NUMERIC	Formatting of numbers
LC_TIME	Formatting of dates and times

These subcategories can be split into two groups. lc_messages, lc_monetary, lc_numeric, and lc_time can be adjusted via parameters after initialization. LC_COLLATE and LC_CTYPE belong to collation — see the collation section for adjustment details.

Locale settings affect the following behaviors:

Sort order in queries using ORDER BY or the standard comparison operators on textual data
The upper, lower, and initcap functions
Pattern matching operators (LIKE, SIMILAR TO, and POSIX-style regular expressions); locales affect both case insensitive matching and the classification of characters by character-class regular expressions
The to_char family of functions
The ability to use indexes with LIKE clauses

COLLATION
#

Collation defines the sort order of characters and character classification behavior. Some database operators depend on collation, such as ORDER BY, lower, upper, initcap, to_char, and others.

Use the following SQL to query the system table pg_collation to get LC_COLLATE and LC_CTYPE information for supported character sets:

 select pg_encoding_to_char(collencoding) as encoding,collname,collcollate,collctype from pg_collation where collname in ('default','C','POSIX','en_US.utf8','zh_CN.utf8','zh_CN.gb2312','zh_SG.gb2312') ;
 encoding | collname | collcollate | collctype 
----------+--------------+--------------+--------------
 | default | | 
 | C | C | C
 | POSIX | POSIX | POSIX
 UTF8 | en_US.utf8 | en_US.utf8 | en_US.utf8
 EUC_CN | zh_CN.gb2312 | zh_CN.gb2312 | zh_CN.gb2312
 UTF8 | zh_CN.utf8 | zh_CN.utf8 | zh_CN.utf8
 EUC_CN | zh_SG.gb2312 | zh_SG.gb2312 | zh_SG.gb2312

encoding is the character set, and collname is the collation name.

When encoding is empty, it means this collation supports all character sets.
default, C, POSIX are collations supported on all platforms, provided by libc. Other collations depend on whether the operating system supports them (locale -a).
default means using the collation set at database creation time, which can be viewed via \l.
C is semantically equivalent to POSIX, but PostgreSQL still considers them different collations. They both compare characters by ASCII code, strictly by byte order.

=> SELECT 'a' COLLATE "C" < 'b' COLLATE "POSIX" ;
ERROR: 42P21: collation mismatch between explicit collations "C" and "POSIX"
LINE 1: SELECT 'a' COLLATE "C" < 'b' COLLATE "POSIX" ;
LOCATION: merge_collation_state, parse_collate.c:834

UTF8 is the most common character set, and the most common language environments are en_US and zh_CN.
You can create custom collations via CREATE COLLATION .... However, cases where LC_COLLATE and LC_CTYPE differ are very rare.

LC_COLLATE
#

LC_COLLATE affects character comparison (sorting, character operations, etc.).

The COLLATE clause can transform the collation of an expression:

expr COLLATE collation

Note that this specifies a collation, not lc_collate. If no collation is explicitly specified, the database uses the column’s collation by default. If the column has no collation specified, it uses the database’s default collation.

Sorting test with different collations:

 select col1 from (values ('a'), ('A'), ('啊'), ('阿')) 
-> AS l(col1)
-> order by col1 collate "C";
 col1 
------
 A
 a
 啊
 阿
 select col1 from (values ('a'), ('A'), ('啊'), ('阿')) 
-> AS l(col1)
-> order by col1 collate "en_US.utf8";
 col1 
------
 a
 A
 啊
 阿
 select col1 from (values ('a'), ('A'), ('啊'), ('阿')) 
-> AS l(col1)
-> order by col1 collate "zh_CN.utf8";
 col1 
------
 a
 A
 阿
 啊

These three different collations have different lc_collate values, and the sort methods are indeed different — we can see three distinct sort results from the output.

Why does collation C put ‘A’ before ‘a’? Collation C uses ASCII encoding order. In ASCII, uppercase letters come before lowercase. Meanwhile, en_US.utf8 and zh_CN.utf8 clearly do not follow this order for English letters.

Order of Chinese characters Even with the same UTF8 character set, the order of Chinese characters differs between Chinese and English locales. Different lc_collate values correspond to different alphabets for different localized languages. The sort order with lc_collate=C is always by byte order. Although ASCII does not include Chinese, C can still sort Chinese — (essentially) every Chinese character maps to a UTF8 encoding, and C sorts by byte order.

LC_CTYPE
#

LC_CTYPE affects character operations (such as upper, initcap, etc.).

If the string is all English, e.g., 'abcD', initcap converts it to 'Abcd' under all three collations — nothing special to show here.

But when Chinese is introduced, the results differ:

select initcap('啊aAAa阿bBBb' collate "C");
 initcap 
--------------
 啊Aaaa阿Bbbb

select initcap('啊aAAa阿aAAa' collate "en_US.utf8");
 initcap 
--------------
 啊aaaa阿aaaa

select initcap('啊aAAa阿aAAa' collate "zh_CN.utf8");
 initcap 
--------------
 啊aaaa阿aaaa

When LC_CTYPE=C, initcap capitalizes the first letter of every non-contiguous English character sequence, whereas en_US.utf8 and zh_CN.utf8 only capitalize the very first character (Chinese characters remain unchanged) and lowercase other English characters.

The behavior of initcap with Chinese may be an undefined requirement, but we can conclude: different LC_CTYPE settings lead to different results from character-sensitive functions like initcap.

Furthermore, Chinese is case-insensitive, but some other localized languages do have case distinctions — different LC_CTYPE settings lead to even more complex outcomes.

Character Sets
#

Character Set Basics
#

PostgreSQL supports different character sets (also called encodings). Character sets and collation are two separate concepts, but the character set must be compatible with LC_CTYPE and LC_COLLATE. As seen in pg_collation, C/POSIX support all character sets, while other collations only support one character set (on Linux systems).

Chinese-related character sets available in PostgreSQL: *(The C collation is provided by the libc library; some collations can be provided by the ICU library, requiring compilation in advance.)

Name	Description	Language	Server-side support?	ICU support?	Bytes/Char	Aliases
BIG5	Big Five	Traditional Chinese	No	No	1–2	WIN950, Windows950
EUC_CN	Extended UNIX Code-CN	Simplified Chinese	Yes	Yes	1–3	GB2312
GB18030	National Standard	Chinese	No	No	1–4
GBK	Extended National Standard	Simplified Chinese	No	No	1–2	WIN936, Windows936
UTF8	Unicode, 8-bit	all	Yes	Yes	1–4	Unicode

Traditional Chinese: BIG5 is the most common character set standard for Traditional Chinese. It was once the industry standard and was later incorporated as a national standard.

Simplified Chinese: GB stands for “Guobiao” (national standard). GB2312, GB18030, and GBK are all Chinese national character set standards. Due to issues such as rare characters and years of development producing several historical versions, there appear to be multiple standards. EUC_CN stands for Extended UNIX Code-CN, which is essentially GB2312, but it cannot handle all rare characters either. Similarly named encodings include EUC_KR, EUC_JP, EUC_TW, and so on.

International Standards: The character sets above are all national standards — they support English and Chinese but not other languages. The international standard that supports all languages of the world is Unicode (which even includes emoji 👍). (There is also the well-known international standards organization ISO, which maintains character sets as well — there is some overlap, but we’ll set ISO aside for now.)

Due to different Unicode encoding schemes, there are three encoding formats: UTF-8, UTF-16, and UTF-32.

UTF-8 encoding format:

Bytes	Format	Actual encoding bits	Code point range
1 byte	0xxxxxxx	7	0 ~ 127
2 byte	110xxxxx 10xxxxxx	11	128 ~ 2047
3 byte	1110xxxx 10xxxxxx 10xxxxxx	16	2048 ~ 65535
4 byte	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx	21	65536 ~ 2097151

UTF8 encoding is variable-length. For characters in the range 0x00-0x7F (1 byte), UTF-8 encoding is exactly identical to ASCII (American Standard Code for Information Interchange). Therefore, UTF-8 is fully backward-compatible with ASCII.

Due to shared origins, meanings, and similarities, Chinese, Japanese, Korean, and Vietnamese characters use a unified encoding in Unicode called CJK Unified Ideographs (CJKV Unified Ideographs). CJK Unified Ideographs encoding ranges: 3400-4DBF/4E00-9FFF/20000-3FFFF.

Character Set Conversion
#

When server_encoding and client_encoding differ, automatic conversion of the character set returned by the server can occur. For setting server-side and client-side character sets, see the “Configuring Character Sets” section.

Chinese-related character sets — Server/Client convertible table:

Server Character Set	Available Client Character Sets
BIG5	not supported as a server encoding
EUC_CN (GB2312)	EUC_CN (GB2312), `MULE_INTERNAL`, `UTF8`
GB18030	not supported as a server encoding
GBK	not supported as a server encoding
UTF8	all supported encodings
GB18030 and GBK are not supported on the server side, so in practice only EUC_CN (GB2312) and UTF8 can perform Server/Client conversion.

The above lists the character sets that can be converted, but conversion still requires CONVERSION support. PostgreSQL has built-in conversion functions visible via pg_conversion:

Conversion Name	Source Encoding	Destination Encoding
big5_to_utf8	BIG5	UTF8
euc_cn_to_utf8	EUC_CN	UTF8
gb18030_to_utf8	GB18030	UTF8
gbk_to_utf8	GBK	UTF8
utf8_to_big5	UTF8	BIG5
utf8_to_euc_cn	UTF8	EUC_CN
utf8_to_gb18030	UTF8	GB18030
utf8_to_gbk	UTF8	GBK

You can create custom conversions via the CREATE CONVERSION statement, specifying the conversion function.

Some character sets appear to be interconvertible, but the server side doesn’t support storing them at all (such as BIG5, GB18030, GBK), so it’s not practically useful. All we need to know here is that euc_cn and utf8 can be converted to/from each other.

Without CONVERSION support, conversion cannot happen:

-- EUC_CN database
=> \encoding EUC_KR
EUC_KR: invalid encoding name or conversion procedure not found

Character set conversion test: Pay attention to the client-side character set settings (e.g., CRT’s “session” - “Appearance” - “Character encoding”)

There are at least three endpoints with character set concepts: database server, database client, and UI client. CONVERSION only controls: database server → database client.

Server with UTF8 conversion test:

create table zh(col1 varchar(20));
insert into zh values('>'),('阿'),('〇'); -- 〇 (líng) is a Chinese character
-- If CRT is not set to UTF8, Chinese characters are all garbled; only set CRT to UTF8 for insertion

=> show server_encoding;
 server_encoding 
-----------------
 UTF8
=> show client_encoding;
 client_encoding 
-----------------
 UTF8
-- With no conversion at all, UTF8 displays correctly. Currently three endpoints: UTF8 - UTF8 - UTF8
=> select * from zh;
 col1 
------
 >
 阿
 〇

-- Switch database client character set. Now three endpoints: UTF8 - EUC_CN - UTF8
=> \encoding EUC_CN; -- Set client character set

=> select * from zh where col1 in ('阿');
ERROR: 22021: invalid byte sequence for encoding "EUC_CN": 0xe9 0x98
LOCATION: report_invalid_encoding, mbutils.c:1597
Time: 0.112 ms
=> select * from zh where col1 in ('〇');
ERROR: 22021: invalid byte sequence for encoding "EUC_CN": 0xe3 0x80
ERROR: 22021: invalid byte sequence for encoding "EUC_CN": 0xe3 0x80
-- It looks like "阿" and "〇" cannot be converted to EUC_CN, but that's not the whole story
 
=> select * from zh limit 2;
 col1 
------
 >
 <B0><A2>
(2 rows)
-- The second row is "阿". The database server/client appears to have converted the character set from UTF8 to EUC_CN.
-- However, it may not display correctly due to UI client issues (currently CRT is set to UTF8)

-- Even changing CRT to GB2312 still won't display correctly
select * from zh limit 2;
 col1 
------
 >
 <B0><A2>
(2 rows)

-- When querying 〇, the database throws an error directly, indicating 〇 cannot be converted from UTF8 to EUC_CN
select * from zh ;
ERROR: 22P05: character with byte sequence 0xe3 0x80 0x87 in encoding "UTF8" has no equivalent in encoding "EUC_CN"
LOCATION: report_untranslatable_char, mbutils.c:1631

Server with EUC_CN conversion test:

=> show server_encoding; -- Database has EUC_CN character set
 server_encoding 
-----------------
 EUC_CN
 
-- Create the same zh table under the EUC_CN database, but inserting already has issues
=> insert into zh values('〇');
ERROR: 22P05: character with byte sequence 0xe3 0x80 0x87 in encoding "UTF8" has no equivalent in encoding "EUC_CN"
LOCATION: report_untranslatable_char, mbutils.c:1631

Again, the error says 〇 cannot be converted from UTF8 to EUC_CN. EUC_CN (GB2312) Chinese encoding is not fully identical to UTF8 — EUC_CN (GB2312) does not include all Chinese characters, especially rare ones.

Configuring locale, collation, and character set
#

Now that we’ve covered localization and character sets, here’s a summary.

Database cluster locale, collation, character set
#

At initialization time, you can set the database cluster’s locale and character set:

initdb -D $DATADIR -E UTF8 --locale=en_US.UTF8 
initdb -D $DATADIR -E UTF8 --locale=en_US.UTF8 --lc_collate=C --lc_ctype=C
initdb -D $DATADIR -E UTF8 --locale=en_US.UTF8 --lc_collate=C --lc_ctype=C --lc-messages=en_US.UTF8 --lc-monetary=en_US.UTF8 --lc-numeric=en_US.UTF8 --lc-time=en_US.UTF8

initdb creates three databases: postgres, template1, and template0. The CREATE DATABASE statement defaults to using template1 to create databases.

encoding sets the character set; locale sets LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC, and LC_TIME, unless specifically overridden (e.g., via --lc_collate).
LC_COLLATE and LC_CTYPE are called collation and can also be set at the database, column, and index levels. LC_MESSAGES, LC_MONETARY, LC_NUMERIC, and LC_TIME are instance parameters that can be changed at any time.
encoding can only be set at initialization or at database creation — once set, it cannot be changed.

Database collation and character set
#

When creating a database, you can set the database’s character set, lc_collate, and lc_ctype.

Both CREATE DATABASE and createdb can specify the character set at database creation time. Once created, the database character set cannot be changed. Both commands use a template database to create the new database.

There are two templates: template0 and template1. The official documentation states:

Another common reason for copying template0 instead of template1 is that new encoding and locale settings can be specified when copying template0, whereas a copy of template1 must use the same settings it does. This is because template1 might contain encoding-specific or locale-specific data, while template0 is known not to.

template1 is a writable template database that may contain localized data, while template0 cannot be written to. Therefore, to create a database with different localization settings, you should use template0.

And you must explicitly use template0, because the default is template1. Attempting to create a database without specifying template1 and with a different character set will result in an error:

=> create database db_GB2312 ENCODING 'EUC_CN' LC_COLLATE 'zh_CN.gb2312' LC_CTYPE 'zh_CN.gb2312';
ERROR: 22023: new encoding (EUC_CN) is incompatible with the encoding of the template database (UTF8)
HINT: Use the same encoding as in the template database, or use template0 as template.

Additionally, you cannot set the character set by specifying locale when creating a database:

=> create database db_GB2312 locale 'zh_CN.gb2312' template 'template0';
ERROR: 22023: encoding "UTF8" does not match locale "zh_CN.gb2312"
DETAIL: The chosen LC_CTYPE setting requires encoding "EUC_CN".
LOCATION: check_encoding_locale_matches, dbcommands.c:773

The error indicates you need to specify the LC_CTYPE sub-option. Adding all collation-related sub-options still produces an error:

=> create database db_GB2312 LOCALE 'EUC_CN' LC_COLLATE 'zh_CN.gb2312' LC_CTYPE 'zh_CN.gb2312';
ERROR: 42601: conflicting or redundant options
DETAIL: LOCALE cannot be specified together with LC_COLLATE or LC_CTYPE.

LOCALE cannot be used together with LC_CTYPE and other sub-options.

Removing locale and setting via character set, LC_COLLATE, and LC_CTYPE works successfully.

The correct way to create a database with a specific character set:

CREATE DATABASE:

create database db_GB2312 ENCODING 'EUC_CN' LC_COLLATE 'zh_CN.gb2312' LC_CTYPE 'zh_CN.gb2312' template 'template0';

createdb: Use the CLI command createdb, which wraps CREATE DATABASE — they are equivalent:

 createdb -E EUC_CN -T template0 --lc-collate=zh_CN.gb2312 --lc-ctype=zh_CN.gb2312 db_GB2312

Viewing database character set:

\l
pg_database

select datname,pg_encoding_to_char(encoding),datcollate,datctype,datlocprovider,daticulocale from pg_database;

SHOW parameters

SERVER_ENCODING, LC_COLLATE, and LC_CTYPE are all immutable parameters that display the current database’s server-side character set, LC_COLLATE, and LC_CTYPE, respectively.

Column collation
#

Collation is only related to character sorting and character functions — it is not related to encoding. Without indexes, changing a column’s collation is essentially just adjusting the default sort output for that column. With indexes, it will rebuild the index. If no collation is specified for a column, it defaults to the database’s collation.

Specifying collation when creating a table (note: some data types are un-collatable, such as int):

create table t1(col1 varchar(10) collate "en_US.utf8");
alter table t1 alter column col1 type varchar(10) collate "C";

Note: ALTER TABLE without changing the length will not rewrite the table, but it will definitely rebuild the index.

Viewing a column’s default collation:

1. \d+ t1

2. information_schema.columns
select table_catalog,table_schema,table_name,column_name,collation_name from information_schema.columns where table_name='t1';

3. pg_attribute
select a.attrelid::regclass,a.attname,a.attcollation,c.collname,c.collcollate,c.collctype from pg_attribute a left join pg_collation c on a.attcollation=c.oid where a.attrelid::regclass='tlzl'::regclass and a.attcollation<>0;

Method 3 is recommended. While \d+ and information_schema.columns can show collname, collname is not unique. Only method 3 reveals collate and ctype.

Test: specifying collate and viewing pg_attribute:

create table tlzl(
 col1 varchar(10) ,
 col2 varchar(10) collate "C",
 col3 varchar(10) collate "zh_CN",
 col4 varchar(10) collate "en_US.utf8"
);

 -- Column collation is like tagging the column with a default sort order; you can't see the specific collate and ctype
db_utf8_c=> Table "public.tlzl"
 Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description 
--------+-----------------------+------------+----------+---------+----------+-------------+--------------+-------------
 col1 | character varying(10) | | | | extended | | | 
 col2 | character varying(10) | C | | | extended | | | 
 col3 | character varying(10) | zh_CN | | | extended | | | 
 col4 | character varying(10) | en_US.utf8 | | | extended | | | 
-- collname and collate/ctype are not one-to-one; col3's zh_CN alone doesn't reveal which collate is used
db_utf8_c=> select pg_encoding_to_char(collencoding) as encoding,collname,collcollate,collctype from pg_collation where collname like 'zh_CN%';
 encoding | collname | collcollate | collctype 
----------+--------------+--------------+--------------
 EUC_CN | zh_CN | zh_CN | zh_CN
 EUC_CN | zh_CN.gb2312 | zh_CN.gb2312 | zh_CN.gb2312
 UTF8 | zh_CN.utf8 | zh_CN.utf8 | zh_CN.utf8
 UTF8 | zh_CN | zh_CN.utf8 | zh_CN.utf8
 
-- pg_attribute shows more precisely than \d+
db_utf8_c=> select a.attrelid::regclass,a.attname,a.attcollation,c.collname,c.collcollate,c.collctype from pg_attribute a left join pg_collation c on a.attcollation=c.oid where a.attrelid::regclass='tlzl'::regclass and a.attcollation<>0;
 attrelid | attname | attcollation | collname | collcollate | collctype 
----------+---------+--------------+------------+-------------+------------
 tlzl | col1 | 100 | default | | 
 tlzl | col2 | 950 | C | C | C
 tlzl | col4 | 12562 | en_US.utf8 | en_US.utf8 | en_US.utf8
 tlzl | col3 | 13200 | zh_CN | zh_CN.utf8 | zh_CN.utf8
-- Now we know that col3 zh_CN's collate is zh_CN.utf8 

Test: table rewrite when modifying column collate:

-- Add an index to the column and check rewrite behavior
db_utf8_c=> create index idxcol4 on tlzl(col4);
CREATE INDEX

db_utf8_c=> select pg_relation_filepath('tlzl') TableRelid, pg_relation_filepath('idxcol4') IndexRelid; 
 tablerelid | indexrelid 
------------------+------------------
 base/40996/41006 | base/40996/41015

db_utf8_c=> alter table tlzl alter column col4 type varchar(10) collate "C";
ALTER TABLE
db_utf8_c=> select pg_relation_filepath('tlzl') TableRelid, pg_relation_filepath('idxcol4') IndexRelid; 
 tablerelid | indexrelid 
------------------+------------------
 base/40996/41006 | base/40996/41016
-- Table was not rewritten; index was rewritten

A column’s collation is merely a marker. Modifying the column’s collation does not rewrite the table, but if there is an index on it, the index will be rewritten (sometimes not — see the next section).

Index collation
#

When creating an index, if the index’s collation is not explicitly specified, the index uses the collation declared on the column.

Explicitly specifying collation when creating an index:

create index idx_C on tlzl(col3 collate "C");

Additionally, indexes can be created with text_pattern_ops, varchar_pattern_ops, bpchar_pattern_ops — in this case, the index does not depend on collation rules but compares character by character:

The difference from the default operator classes is that the values are compared strictly character by character rather than according to the locale-specific collation rules.

CREATE INDEX test_index ON test_table (col varchar_pattern_ops);

In fact, this type of index is not entirely unrelated to collation — an index always has a sort order. This type of index’s sort order appears to be consistent with C. See the “LIKE not using index” section.

Viewing an index’s collation:

\d+ -- \d+ shows indexes with explicitly specified collate; if not specified, the column's default collation is used
db_utf8_c=> \d+ tlzl
 Table "public.tlzl"
 Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description 
--------+-----------------------+------------+----------+---------+----------+-------------+--------------+-------------
 col1 | character varying(10) | | | | extended | | | 
 col2 | character varying(10) | C | | | extended | | | 
 col3 | character varying(10) | zh_CN | | | extended | | | 
 col4 | character varying(10) | en_US.utf8 | | | extended | | | 
Indexes:
 "idx_c" btree (col3 COLLATE "C")
 "idxcol4" btree (col4)
Access method: heap

Viewing via pg_index is clearer (the indcollation type in pg_index is oidvector and cannot be directly cast to oid, making queries a bit cumbersome):

db_utf8_c=> select indcollation,indexrelid::regclass from pg_index where indexrelid::regclass ='idx_C'::regclass;
 indcollation | indexrelid 
--------------+------------
 950 | idx_c

db_utf8_c=> select oid,pg_encoding_to_char(collencoding) as encoding,collname,collcollate,collctype from pg_collation where oid=950;
 oid | encoding | collname | collcollate | collctype 
-----+----------+----------+-------------+-----------
 950 | | C | C | C

Also, you cannot change an index’s collation via ALTER INDEX — you must drop and recreate it.

Test: After specifying an index collate, does modifying the column’s collate rewrite the index?

db_utf8_c=> select pg_relation_filepath('tlzl') TableRelid, pg_relation_filepath('idxcol4') IndexRelid4,pg_relation_filepath('idx_c') IndexRelidC; 
 tablerelid | indexrelid4 | indexrelidc 
------------------+------------------+------------------
 base/40996/41020 | base/40996/41023 | base/40996/41024
(1 row)

db_utf8_c=> alter table tlzl alter column col3 type varchar(10) collate "en_US.utf8";
ALTER TABLE
db_utf8_c=> select pg_relation_filepath('tlzl') TableRelid, pg_relation_filepath('idxcol4') IndexRelid4,pg_relation_filepath('idx_c') IndexRelidC; 
 tablerelid | indexrelid4 | indexrelidc 
------------------+------------------+------------------
 base/40996/41020 | base/40996/41023 | base/40996/41024 -- idx_c's relfileid did not change

If an index’s collate has been explicitly specified, modifying the column’s default collate will not rewrite that index.

Client character set
#

When the client sets a character set different from the database, character set conversion occurs — though conversion may not always succeed. See the “Character Set Conversion” section for details.

The server-side character set cannot be changed after database creation, but the client character set can be adjusted at any time.

There are many ways to set the client character set:

Set directly on the client:

\encoding UTF8 -- psql only
SET CLIENT_ENCODING TO UTF8; -- session-level parameter change
SET NAMES UTF8; -- SQL standard

Set the PGCLIENTENCODING environment variable
Set the client_encoding server configuration parameter

Priority: client-side setting > PGCLIENTENCODING environment variable > client_encoding server configuration parameter

Viewing the client character set:

\encoding -- psql only
SHOW client_encoding;

Expression collate
#

Adding COLLATE to an expression overrides the expression’s original collation, effectively specifying a sort collation.

Add the COLLATE keyword at the end of the expression:

expr COLLATE collation

-- For example
select * from tab1 order by name COLLATE "C";

For details on sorting and collate index selection, see the “Sort Result Issues” section.

MORE
#

Concept Summary
#

PostgreSQL localization has three important concepts: character set, locale, and collation — it’s essential to understand their relationships.

The server-side character set setting is very important: it can only be specified at initialization and database creation time, and cannot be modified after the database is created. The character set choice directly affects the encoding method. Collation does not, but there is a dependency between the two. Locale can likewise be specified at initialization, and among them, collation can be set at database creation time or individually on columns — note that these are merely defaults. Only when specifying collation at index creation does it affect the actual storage order. Different collations cannot use the same index, even if they share the same origin.

Client character set and the four parameters (LC_MESSAGES, etc.) are relatively simple — they can be modified directly via parameters and are unrelated to data storage.

Sort Result Issues
#

Since UTF8 is the most common character set, we’ll test sorting with UTF-related collations:

create database db_UTF8 ENCODING 'UTF8' template 'template0'; -- Create a UTF8 database; collation doesn't matter
use db_UTF8;
create table tzlz(name varchar(10));
insert into tzlz values('a'),('aa'),('A'),('AA'),('啊'),('阿'),('〇');

ORDER BY results with different collations:

select name from tzlz where name in ('a','aa','A','AA','啊','阿','〇') order by name;
select name from tzlz where name in ('a','aa','A','AA','啊','阿','〇') order by name collate "C";
select name from tzlz where name in ('a','aa','A','AA','啊','阿','〇') order by name collate "en_US";
select name from tzlz where name in ('a','aa','A','AA','啊','阿','〇') order by name collate "en_US.utf8";
select name from tzlz where name in ('a','aa','A','AA','啊','阿','〇') order by name collate "zh_CN";
select name from tzlz where name in ('a','aa','A','AA','啊','阿','〇') order by name collate "zh_CN.utf8";

Order	default	C	en_US	en_US.utf8	zh_CN	zh_CN.utf8
1	〇	A	〇	〇	a	a
2	a	AA	a	a	A	A
3	A	a	A	A	aa	aa
4	aa	aa	aa	aa	AA	AA
5	AA	〇	AA	AA	阿	阿
6	啊	啊	啊	啊	啊	啊
7	阿	阿	阿	阿	〇	〇

Here, default is en_US.utf8 (column collation(default) → database collation(en_US.utf8))

🌟 C, en_US.utf8, and zh_CN.utf8 all produce different sort results!

Collate and index scan test:

insert into tzlz values(generate_series(1,10000));
create index idxzlz_default on tzlz(name);
create index idxzlz_C on tzlz(name collate "C");
create index idxzlz_enUS_utf8 on tzlz(name collate "en_US.utf8");

Using collate for index optimization:

-- Without any collate keyword, a simple index scan; no extra sorting
db_utf8_c=> explain select name from tzlz where name in ('a','aa','A','AA','啊','阿','〇') order by name;
 QUERY PLAN 
---------------------------------------------------------------------------------
 Index Only Scan using idxzlz_default on tzlz (cost=0.29..30.13 rows=8 width=4)
 Index Cond: (name = ANY ('{a,aa,A,AA,啊,阿,〇}'::text[]))

-- Adding collate conversion to the predicate hits the correct index
db_utf8=> explain select name from tzlz where name collate "C" in ('a','aa','A','AA','啊','阿','〇');
 QUERY PLAN 
---------------------------------------------------------------------------
 Index Only Scan using idxzlz_c on tzlz (cost=0.29..30.12 rows=7 width=4)
 Index Cond: (name = ANY ('{a,aa,A,AA,啊,阿,〇}'::text[]))

db_utf8=> explain select name from tzlz where name collate "en_US.utf8" in ('a','aa','A','AA','啊','阿','〇');
 QUERY PLAN 
-----------------------------------------------------------------------------------
 Index Only Scan using idxzlz_enus_utf8 on tzlz (cost=0.29..30.12 rows=7 width=4)
 Index Cond: (name = ANY ('{a,aa,A,AA,啊,阿,〇}'::text[]))
 
-- However, the collation name must match exactly
db_utf8=> explain select name from tzlz where name collate "en_US" in ('a','aa','A','AA','啊','阿','〇');
 QUERY PLAN 
-----------------------------------------------------------------
 Seq Scan on tzlz (cost=0.00..232.63 rows=7 width=4)
 Filter: ((name)::text = ANY ('{a,aa,A,AA,啊,阿,〇}'::text[]))
 
-- ORDER BY also needs the collate conversion expression
-- Here, the correct index is used, but ORDER BY treats them as different collations (even though they are the same)
db_utf8=> explain select name from tzlz where name collate "en_US.utf8" in ('a','aa','A','AA','啊','阿','〇') order by name;
 QUERY PLAN 
-----------------------------------------------------------------------------------------
 Sort (cost=30.22..30.23 rows=7 width=4)
 Sort Key: name
 -> Index Only Scan using idxzlz_enus_utf8 on tzlz (cost=0.29..30.12 rows=7 width=4)
 Index Cond: (name = ANY ('{a,aa,A,AA,啊,阿,〇}'::text[]))

-- Adding collate conversion to both WHERE and ORDER BY selects the right index and avoids extra sorting
db_utf8=> explain select name from tzlz where name collate "en_US.utf8" in ('a','aa','A','AA','啊','阿','〇') order by name collate "en_US.utf8";
 QUERY PLAN 
------------------------------------------------------------------------------------
 Index Only Scan using idxzlz_enus_utf8 on tzlz (cost=0.29..30.12 rows=7 width=42)
 Index Cond: (name = ANY ('{a,aa,A,AA,啊,阿,〇}'::text[]))

After specifying a collation on an index, the SQL must explicitly use the COLLATE keyword to convert the expression. Even if the default is the same as the current collation, PostgreSQL will not use the index.

LIKE not using index
#

The drawback of using locales other than C or POSIX in PostgreSQL is its performance impact. It slows character handling and prevents ordinary indexes from being used by LIKE

PostgreSQL’s own words: using non-C or non-POSIX prevents ordinary indexes from being used!

db_utf8=> explain select name from tzlz where name like 'a%';

 QUERY PLAN 
--------------------------------------------------------------------------

 Index Only Scan using idxzlz_c on tzlz (cost=0.29..4.31 rows=1 width=4)
 Index Cond: ((name >= 'a'::text) AND (name < 'b'::text))
 Filter: ((name)::text ~~ 'a%'::text)
(3 rows)

db_utf8=> explain select name from tzlz where name collate "en_US.utf8" like 'a%';

 QUERY PLAN 
--------------------------------------------------------------------------

 Index Only Scan using idxzlz_c on tzlz (cost=0.29..4.31 rows=1 width=4)
 Index Cond: ((name >= 'a'::text) AND (name < 'b'::text))
 Filter: ((name)::text ~~ 'a%'::text)

PostgreSQL converts LIKE to >= and < during index scans, where < adds a “one step greater” value. This is where the problem lies: collation is strongly tied to sorting order. In ASCII, a+1 is b, but what about Chinese characters?

db_utf8=> explain select name from tzlz where name collate "en_US.utf8" like '阿%';
 QUERY PLAN 
--------------------------------------------------------------------------
 Index Only Scan using idxzlz_c on tzlz (cost=0.29..6.49 rows=1 width=4)
 Index Cond: ((name >= '阿'::text) AND (name < '陿'::text))
 Filter: ((name)::text ~~ '阿%'::text)

Sure enough, another Chinese character appears!

If it’s a sequential scan, the >= and < won’t appear:

db_utf8=> drop index idxzlz_c;
DROP INDEX
db_utf8=> explain select name from tzlz where name collate "en_US.utf8" like '阿%';
 QUERY PLAN 
------------------------------------------------------
 Seq Scan on tzlz (cost=0.00..170.09 rows=1 width=4)
 Filter: ((name)::text ~~ '阿%'::text)

You can create an index that is (claimed by the PostgreSQL docs to be) unrelated to collation rules:

CREATE INDEX idx_pattern ON tzlz (name varchar_pattern_ops);

Let’s look at its execution plan:

db_utf8=> explain select name from tzlz where name like '阿%';

 QUERY PLAN 
-----------------------------------------------------------------------------

 Index Only Scan using idx_pattern on tzlz (cost=0.29..6.49 rows=1 width=4)
 Index Cond: ((name ~>=~ '阿'::text) AND (name ~<~ '陿'::text))
 Filter: ((name)::text ~~ '阿%'::text)

It still auto-generates the “one greater” string — this is definitely related to collation. It appears to be using C.

So we can conclude:

When PostgreSQL uses a regular index for LIKE, it needs to convert it to >= and <, which requires a “one greater” value relative to the current string. Since collation is strongly tied to ordering, only an index using the same collation can guarantee data correctness. PostgreSQL chooses the non-localized C collation for this.

The quickest workaround is to create a C collation index or a pattern index:

create index idxzlz_C on tzlz(name collate "C");
CREATE INDEX idx_pattern ON tzlz (name varchar_pattern_ops);

For other adjustments to default collation at various levels, refer to the sections above.

Developers typically don’t specify collation when creating indexes. If it’s not C or pattern, LIKE won’t use the index. Combined with the common choice of the international character set UTF8, this leaves very few localization options in database operations. The recommended setup: character set UTF8, collation C.

References
#

https://dbafix.com/what-is-the-impact-of-lc_ctype-on-a-postgresql-database/#:~:text=Having%20LC_CTYPE%20set%20to%20%E2%80%98C%E2%80%99%20implies%20that%20C,Postgres%20on%20top%20of%20these%20libc%20functions%2C%20they%E2%80%99re https://www.postgresql.org/docs/current/charset.html https://www.bookstack.cn/read/rds-best-pratice/bfc0037fe00d87dc.md https://help.aliyun.com/zh/rds/apsaradb-rds-for-postgresql/configure-the-collation-of-a-database-on-an-apsaradb-rds-for-postgresql-instance https://baike.baidu.com/item/%E7%BB%9F%E4%B8%80%E7%A0%81/2985798?fromModule=lemma_inlink&fromtitle=Unicode&fromid=750500 https://baike.baidu.com/item/%E4%B8%AD%E6%97%A5%E9%9F%A9%E8%B6%8A%E7%BB%9F%E4%B8%80%E8%A1%A8%E6%84%8F%E6%96%87%E5%AD%97/1301611?fromModule=lemma_inlink

https://blog.csdn.net/songyundong1993/article/details/128739919

Original article (Chinese): PostgreSQL本地化

PostgreSQL Table Partitioning Deep Dive

Mon, 12 Aug 2024 00:00:00 +0000

What is a Partitioned Table
#

Database partitioning splits table data into smaller physical shards to improve performance, availability, and manageability. Partitioned tables are a common optimization technique for large tables in relational databases. DBMS generally provide partition management, and applications can access partitioned tables directly without changing their architecture—though good performance requires proper partition access patterns.

Partitioned tables are common database technology, but PostgreSQL partitioned tables have many unique characteristics: multiple implementation approaches, partitions being regular tables, partition maintenance strategies, SQL optimization considerations, and some known issues.

Partition Table Implementations
#

PostgreSQL provides various partition implementation approaches. The officially supported methods are declarative partitioning and inheritance partitioning, while third-party plugins include pg_pathman, pg_partman, etc. Since the introduction of official declarative partitioning, only one approach is generally recommended: declarative partitioning. Covering every implementation’s features, details, and history would make this article excessively long and is less relevant going forward. This article focuses mainly on declarative partitioning, with brief introductions to other approaches. However, due to existing deployments and feature differences, understanding declarative partitioning, inheritance partitioning, and pg_pathman remains valuable.

Declarative Partitioning
#

Declarative partitioning, also called native partitioning, has been supported since PG10. It is the “officially supported” partitioning approach and the most recommended method. Although different from inheritance partitioning, declarative partitioning is also implemented internally using table inheritance. It supports only three partition methods: RANGE, LIST, and HASH.

RANGE Partitioning
#

RANGE partitioned tables split data by range, with partition boundaries defined as [t1, t2) (inclusive lower bound, exclusive upper bound).

CREATE TABLE PUBLIC.LZLPARTITION1
(
 id int,
 name varchar(50) NULL, 
 DATE_CREATED timestamp NOT NULL DEFAULT now()
) PARTITION BY RANGE(DATE_CREATED);

alter table public.lzlpartition1 add primary key(id,DATE_CREATED)

create table LZLPARTITION1_202301 partition of LZLPARTITION1 for values from ('2023-01-01 00:00:00') to ('2023-02-01 00:00:00');

create table LZLPARTITION1_202302 partition of LZLPARTITION1 for values from ('2023-02-01 00:00:00') to ('2023-03-01 00:00:00');

-- Insert some data into the partitioned table
=> INSERT INTO lzlpartition1 SELECT random() * 10000, md5(g::text),g 
FROM generate_series('2023-01-01'::date, '2023-02-28'::date, '1 minute') as g;
INSERT 0 83521

For RANGE partitioning, the FROM t1 TO t2 boundary uses the [t1, t2) convention: the lower bound is inclusive and the upper bound is exclusive.

Inspecting the partitioned table shows that each partition is also an independent table:

lzldb=> \d+ lzlpartition1
 Partitioned table "public.lzlpartition1"
 Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description 
--------------+-----------------------------+-----------+----------+---------+----------+-------------+--------------+-------------
 id | integer | | not null | | plain | | | 
 name | character varying(50) | | | | extended | | | 
 date_created | timestamp without time zone | | not null | now() | plain | | | 
Partition key: RANGE (date_created)
Indexes:
 "lzlpartition1_pkey" PRIMARY KEY, btree (id, date_created)
Partitions: lzlpartition1_202301 FOR VALUES FROM ('2023-01-01 00:00:00') TO ('2023-02-01 00:00:00'),
 lzlpartition1_202302 FOR VALUES FROM ('2023-02-01 00:00:00') TO ('2023-03-01 00:00:00')

lzldb=> \d+ lzlpartition1_202301
 Table "public.lzlpartition1_202301"
 Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description 
--------------+-----------------------------+-----------+----------+---------+----------+-------------+--------------+-------------
 id | integer | | not null | | plain | | | 
 name | character varying(50) | | | | extended | | | 
 date_created | timestamp without time zone | | not null | now() | plain | | | 
Partition of: lzlpartition1 FOR VALUES FROM ('2023-01-01 00:00:00') TO ('2023-02-01 00:00:00')
Partition constraint: ((date_created IS NOT NULL) AND (date_created >= '2023-01-01 00:00:00'::timestamp without time zone) AND (date_created < '2023-02-01 00:00:00'::timestamp without time zone))
Indexes:
 "lzlpartition1_202301_pkey" PRIMARY KEY, btree (id, date_created)
Access method: heap

Primary keys, indexes, and NOT NULL/CHECK constraints are automatically created on partitions. Since partitions are independent tables, constraints and indexes can also be created on individual partitions. (ATTACH does not automatically create these — see the ATTACH section for details.)

LIST Partitioning
#

LIST partitioning stores data in the corresponding partition based on specified partition key values.

CREATE TABLE cities (
 city_id bigserial not null,
 name text,
 population bigint
) PARTITION BY LIST (left(lower(name), 1));

CREATE TABLE cities_ab
 PARTITION OF cities FOR VALUES IN ('a', 'b');
CREATE TABLE cities_null
 PARTITION OF cities (
 CONSTRAINT city_id_nonzero CHECK (city_id != 0)
) FOR VALUES IN (null);

insert into cities(name,population) values('Acity',10);
insert into cities(name,population) values(null,20);

=> SELECT tableoid::regclass,* FROM cities;
 tableoid | city_id | name | population 
-------------+---------+--------+------------
 cities_ab | 1 | Acity | 10
 cities_null | 2 | [null] | 20

LIST partitioned tables support creating a NULL partition.

HASH Partitioning
#

HASH partitioning distributes data across partitions to spread out hot data evenly.

CREATE TABLE orders (order_id int,name varchar(10)) PARTITION BY HASH (order_id);
CREATE TABLE orders_p1 PARTITION OF orders
 FOR VALUES WITH (MODULUS 3, REMAINDER 0);
CREATE TABLE orders_p2 PARTITION OF orders
 FOR VALUES WITH (MODULUS 3, REMAINDER 1);
CREATE TABLE orders_p3 PARTITION OF orders
 FOR VALUES WITH (MODULUS 3, REMAINDER 2);

You cannot create a default partition, nor can you create more partitions than the specified MODULUS.

=> CREATE TABLE orders_p2 PARTITION OF orders
-> FOR VALUES WITH (MODULUS 3, REMAINDER 4);
ERROR: 42P16: remainder for hash partition must be less than modulus
LOCATION: transformPartitionBound, parse_utilcmd.c:3939

=> CREATE TABLE orders_p4 PARTITION OF orders default;
ERROR: 42P16: a hash-partitioned table may not have a default partition
LOCATION: transformPartitionBound, parse_utilcmd.c:3909

Insert data:

=>insert into orders values(generate_series(1,10000),'a');
INSERT 0 10000
=>SELECT tableoid::regclass,count(*) FROM orders group by tableoid::regclass;
 tableoid | count 
-----------+-------
 orders_p1 | 3277
 orders_p3 | 3354
 orders_p2 | 3369
 =>select tableoid::regclass,* from orders limit 30;
 tableoid | order_id | name 
-----------+----------+------
 orders_p1 | 2 | a
 orders_p1 | 4 | a
 orders_p1 | 6 | a
 orders_p1 | 8 | a
 orders_p1 | 15 | a
 orders_p1 | 16 | a
 orders_p1 | 18 | a
 orders_p1 | 19 | a
 orders_p1 | 20 | a

HASH partition data is distributed evenly across partitions:

-- Insert 100 NULL rows
=> insert into orders values(null,generate_series(1,100)::text);
INSERT 0 100
=> SELECT tableoid::regclass,count(*) FROM orders where order_id is null group by tableoid::regclass;
 tableoid | count 
-----------+-------
 orders_p1 | 100
-- All NULL data ends up on the remainder 0 partition
=>\d+ orders_p1
 Table "public.orders_p1"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
----------+-----------------------+-----------+----------+---------+----------+--------------+-------------
 order_id | integer | | | | plain | | 
 name | character varying(10) | | | | extended | | 
Partition of: orders FOR VALUES WITH (modulus 3, remainder 0)
Partition constraint: satisfies_hash_partition('412053'::oid, 3, 0, order_id)

Although HASH partitioned tables have no concept of a NULL partition, they can store NULL data. NULL values are placed on the remainder 0 partition.

Multi-level (Mixed) Partitioning
#

Partitions can themselves be further partitioned, forming a cascading structure. Sub-partitions can use different partition methods — this is called mixed partitioning. Creating a mixed partition:

create table part_1000(id bigserial not null,name varchar(10),createddate timestamp) partition by range(createddate);
create table part_2001 partition of part_1000 for values from ('2023-01-01 00:00:00') to ('2023-02-01 00:00:00') partition by list(name) ;
create table part_2002 partition of part_1000 for values from ('2023-02-01 00:00:00') to ('2023-03-01 00:00:00') partition by list(name) ;
create table part_2003 partition of part_1000 for values from ('2023-03-01 00:00:00') to ('2023-04-01 00:00:00') partition by list(name) ;
create table part_3001 partition of part_2001 FOR VALUES IN ('abc');
create table part_3002 partition of part_2001 FOR VALUES IN ('def');
create table part_3003 partition of part_2001 FOR VALUES IN ('jkl');

\d+ only shows the immediate next-level partitions:

 \d+ part_1000
 Partitioned table "dbmgr.part_1000"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
-------------+-----------------------------+-----------+----------+---------------------------------------+----------+--------------+-------------
 id | bigint | | not null | nextval('part_1000_id_seq'::regclass) | plain | | 
 name | character varying(10) | | | | extended | | 
 createddate | timestamp without time zone | | | | plain | | 
Partition key: RANGE (createddate)
Partitions: part_2001 FOR VALUES FROM ('2023-01-01 00:00:00') TO ('2023-02-01 00:00:00'), PARTITIONED,
 part_2002 FOR VALUES FROM ('2023-02-01 00:00:00') TO ('2023-03-01 00:00:00'), PARTITIONED,
 part_2003 FOR VALUES FROM ('2023-03-01 00:00:00') TO ('2023-04-01 00:00:00'), PARTITIONED
=> \d+ part_2001
 Partitioned table "dbmgr.part_2001"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
-------------+-----------------------------+-----------+----------+---------------------------------------+----------+--------------+-------------
 id | bigint | | not null | nextval('part_1000_id_seq'::regclass) | plain | | 
 name | character varying(10) | | | | extended | | 
 createddate | timestamp without time zone | | | | plain | | 
Partition of: part_1000 FOR VALUES FROM ('2023-01-01 00:00:00') TO ('2023-02-01 00:00:00')
Partition constraint: ((createddate IS NOT NULL) AND (createddate >= '2023-01-01 00:00:00'::timestamp without time zone) AND (createddate < '2023-02-01 00:00:00'::timestamp without time zone))
Partition key: LIST (name)
Partitions: part_3001 FOR VALUES IN ('abc'),
 part_3002 FOR VALUES IN ('def'),
 part_3003 FOR VALUES IN ('jkl') 

Now insert a row:

=> insert into part_1000 values(random() * 10000,'abc','2023-01-01 08:00:00');
INSERT 0 1
=> SELECT tableoid::regclass,* FROM part_1000;
 tableoid | id | name | createddate 
-----------+------+------+---------------------
 part_3001 | 6385 | abc | 2023-01-01 08:00:00

Data is stored in the lowest-level sub-partition.

Declarative Partitioning Feature Summary
#

No INTERVAL partitioning. There is no built-in automatic partition creation feature, which makes maintenance more cumbersome.
Partitions themselves are tables. This is a distinctive characteristic. This not only allows PostgreSQL to flexibly operate on sub-partitions but, more importantly, affects functionality and behavior.
TRUNCATE, VACUUM, and ANALYZE on a partitioned table operate on all partitions. TRUNCATE ONLY cannot be executed on the parent table but can be executed on a child table containing data, clearing only that sub-partition.
RANGE and HASH partition keys can have multiple columns; LIST partition keys can only be a single column or expression.
The partitioned parent table itself is empty; only the lowest-level sub-partitions contain data.
A DEFAULT partition receives data that falls outside declared ranges. Without a DEFAULT partition, inserting out-of-range data will raise an error.
When adding a new partition, check whether the DEFAULT partition contains data belonging to the new partition.
Partitions created via PARTITION OF automatically create indexes, constraints, and row-level triggers from the parent table.
ATTACH does not handle any indexes, constraints, or other objects.

Inheritance Partitioning
#

Inheritance partitioning is also officially supported. It leverages PostgreSQL’s table inheritance feature to implement partitioning functionality. Inheritance partitioning is more flexible than declarative partitioning. Implementing inheritance partitioning requires two PostgreSQL features: table inheritance and write redirection. Write redirection can be implemented via rules or triggers.

Creating Inheritance Partition Tables
#

Example of creating inheritance partitioned tables: 1. Create the parent table

CREATE TABLE measurement (
 city_id int not null,
 logdate date not null,
 peaktemp int,
 unitsales int
);

2. Create child tables with CHECK constraints for partitioning ranges

CREATE TABLE measurement_202308 (
 CHECK ( logdate >= DATE '2023-08-01' AND logdate < DATE '2023-09-01' )
) INHERITS (measurement);
CREATE TABLE measurement_202309 (
 CHECK ( logdate >= DATE '2023-09-01' AND logdate < DATE '2023-10-01' )
) INHERITS (measurement);

3. Create rules or triggers to redirect inserted data to the corresponding child tables

CREATE OR REPLACE FUNCTION measurement_insert_trigger()
RETURNS TRIGGER AS $$
BEGIN
 IF ( NEW.logdate >= DATE '2023-08-01' AND
 NEW.logdate < DATE '2023-09-01' ) THEN
 INSERT INTO measurement_202308 VALUES (NEW.*);
 ELSIF ( NEW.logdate >= DATE '2023-09-01' AND
 NEW.logdate < DATE '2023-10-01' ) THEN
 INSERT INTO measurement_202309 VALUES (NEW.*);
 ELSE
 RAISE EXCEPTION 'Date out of range. Fix the measurement_insert_trigger() function!';
 END IF;
 RETURN NULL;
END;
$$
LANGUAGE plpgsql;

CREATE TRIGGER insert_measurement_trigger
 BEFORE INSERT ON measurement
 FOR EACH ROW EXECUTE FUNCTION measurement_insert_trigger();

A basic inheritance partitioned table is now set up.

=>\d+ measurement
 Table "public.measurement"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
-----------+---------+-----------+----------+---------+---------+--------------+-------------
 city_id | integer | | not null | | plain | | 
 logdate | date | | not null | | plain | | 
 peaktemp | integer | | | | plain | | 
 unitsales | integer | | | | plain | | 
Triggers:
 insert_measurement_trigger BEFORE INSERT ON measurement FOR EACH ROW EXECUTE FUNCTION measurement_insert_trigger()
Child tables: measurement_202308,
 measurement_202309
Access method: heap

Test insertion and querying:

-- Inserting data outside the defined range raises an error
=> insert into measurement values(1001, now() - interval '31' day ,1,1);
ERROR: P0001: Date out of range. Fix the measurement_insert_trigger() function!
CONTEXT: PL/pgSQL function measurement_insert_trigger() line 10 at RAISE
LOCATION: exec_stmt_raise, pl_exec.c:3889
-- Inserting data is redirected to the child table
=> insert into measurement values(1001,now(),1,1);
INSERT 0 0
-- Querying the parent table returns data from child tables
=> select tableoid::regclass,* from measurement;
 tableoid | city_id | logdate | peaktemp | unitsales 
--------------------+---------+------------+----------+-----------
 measurement_202308 | 1001 | 2023-08-03 | 1 | 1

RULE vs. Trigger Besides triggers, PostgreSQL can also use rules to redirect inserts. Example rule statements:

CREATE RULE measurement_insert_202308 AS
ON INSERT TO measurement WHERE
 ( logdate >= DATE '2023-08-01' AND logdate < DATE '2023-08-01' )
DO INSTEAD
 INSERT INTO measurement_202308 VALUES (NEW.*);
CREATE RULE measurement_insert_202309 AS
ON INSERT TO measurement WHERE
 ( logdate >= DATE '2023-09-01' AND logdate < DATE '2023-09-01' )
DO INSTEAD
 INSERT INTO measurement_202309 VALUES (NEW.*);

Differences between rules and triggers:

Rules have worse performance than triggers in general, but for bulk inserts rules perform better since they only check once. In all other cases, triggers are preferable.
COPY does not fire rules but does fire triggers. When using rules, data can be COPY’d directly into child tables.
When inserting data outside defined ranges, rules will insert into the parent table, while triggers will raise an error.

Indexes To improve performance, you also need to create indexes and enable constraint_exclusion. Indexes on partitions are generally essential. For inheritance tables, indexes must be manually created on child tables. Example of creating indexes:

CREATE INDEX idx_measurement_202308_logdate ON measurement_202308 (logdate);
CREATE INDEX idx_measurement_202309_logdate ON measurement_202309 (logdate);

Insert some data and check the execution plan:

-- '2023-08-04' has only 1 row, allowing it to use the index
=> insert into measurement values(1001,now()+interval '1' day,1,1);
INSERT 0 0
 insert into orders values(generate_series(1,10000),'a');
=> insert into measurement values(generate_series(1,1000),now(),1,1);
INSERT 0 0

=> explain select * from measurement where logdate='2023-08-04';
 QUERY PLAN 
------------------------------------------------------------------------------------------------------------------------------
 Append (cost=0.00..5.17 rows=2 width=16)
 -> Seq Scan on measurement measurement_1 (cost=0.00..0.00 rows=1 width=16)
 Filter: (logdate = '2023-08-04'::date)
 -> Index Scan using idx_measurement_202308_logdate on measurement_202308 measurement_2 (cost=0.14..5.16 rows=1 width=16)
 Index Cond: (logdate = '2023-08-04'::date)

In the above execution plan, the August partition uses the index on the partition. Since constraint_exclusion is enabled by default for inheritance tables, the September partition was excluded and only August was scanned. However, because the parent table has no constraints (and cannot have them), it always appears in the execution plan—but since the parent table is generally empty, this has minimal impact.

constraint_exclusion
#

constraint_exclusion controls whether the optimizer uses constraints to reduce unnecessary table access. This parameter is commonly used in inheritance partitioning optimization — by reducing child table access, it improves SQL performance. (This functionality is similar to the enable_partition_pruning parameter, which controls partition pruning for declarative partitioned tables.) constraint_exclusion has three values: on: All tables are checked for constraints. partition: Inheritance tables and UNION ALL subqueries are checked for constraints (default). off: Constraints are not checked. Constraint exclusion only occurs during execution plan generation, not during actual execution (partition pruning can occur during execution). This means constraint exclusion does not happen when using bound parameters or variable values. For example, when using functions like now() whose specific value the optimizer cannot determine, the optimizer cannot exclude partitions that don’t need to be accessed at all:

=> select now();
 now 
-------------------------------
 2023-08-03 17:12:04.772658+08
-- The optimizer did not exclude the September partition
=> explain select * from measurement where logdate<=now();
 QUERY PLAN 
-----------------------------------------------------------------------------------------------------
 Append (cost=0.00..55.98 rows=1628 width=16)
 -> Seq Scan on measurement measurement_1 (cost=0.00..0.00 rows=1 width=16)
 Filter: (logdate <= now())
 -> Seq Scan on measurement_202308 measurement_2 (cost=0.00..21.15 rows=1010 width=16)
 Filter: (logdate <= now())
 -> Bitmap Heap Scan on measurement_202309 measurement_3 (cost=7.44..26.69 rows=617 width=16)
 Recheck Cond: (logdate <= now())
 -> Bitmap Index Scan on idx_measurement_202309_logdate (cost=0.00..7.28 rows=617 width=0)
 Index Cond: (logdate <= now())

Additionally, constraint exclusion itself needs to check all child table constraints. If there are too many child table constraints, the efficiency of generating execution plans will be affected. Therefore, inheritance partitioning is not recommended for creating too many child partitions.

Adding/Removing Partitions in Inheritance Partitioning
#

To turn an inherited partition into a regular table:

ALTER TABLE measurement_202308 NO INHERIT measurement;

To add an existing regular table (with data) as a child table in the inheritance partition:

CREATE TABLE measurement_202310 
(LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS);
ALTER TABLE measurement_202310 ADD CONSTRAINT measurement_202310_logdate_check 
CHECK ( logdate >= DATE '2023-10-01' AND logdate < DATE '2023-11-01' );
--insert into measurement_202310 values(2001,'20231010',3,3);
ALTER TABLE measurement_202310 INHERIT measurement;

Inheritance Partitioning Feature Summary
#

Inheritance partitioning is more flexible than declarative partitioning, but some declarative partitioning features are unavailable.
Child tables inherit parent table constraints, so global constraints should not be set on the parent table.
Indexes are not inherited; they must be created individually on each child table.
Declarative partitioning only supports RANGE, LIST, and HASH partitions. Inheritance partitioning can support more, including custom partitioning methods.
Dropping a child table does not invalidate the trigger. PostgreSQL does not have Oracle’s concept of invalidated objects (indexes do have an invalidation concept).
Generally, using triggers for insert redirection is more efficient than rules.
When adding a new partition, if the trigger function lacks a rule for that partition, the trigger function needs to be updated.
Inheritance partitioning supports multiple inheritance.
Constraint exclusion cannot occur during execution; using fixed values for queries is recommended.
With inheritance partitioning, avoid creating too many child partitions.

pg_pathman
#

pg_pathman is a third-party plugin implementing partitioning functionality. The pg_pathman README on GitHub and articles on using pg_pathman already describe pathman in great detail. Here we only highlight key points and do some simple testing.

pg_pathman Basics
#

No Longer Maintained

NOTE: this project is not under development anymore

pg_pathman supports PostgreSQL 9.5 through 15. Later PostgreSQL versions are no longer supported, and existing versions only receive bug fixes — no new features will be added. pg_pathman emerged because older PostgreSQL versions had incomplete partitioning features. Now that native partitioned tables (declarative partitioning) are very mature, pg_pathman also recommends using native partitioned tables. Existing pg_pathman partitioned tables are also recommended to be migrated to native partitioned tables. pg_pathman, once recognized by many users, is now history. Even though it’s no longer updated, its feature set is still richer than the current native partitioned tables. Feature Highlights pg_pathman is quite powerful, supporting some features that native partitioned tables do not. However, pathman is not perfect either and has many issues in practice. Key points to note about pg_pathman include:

pg_pathman can manage partitions through partition management functions. It supports replace, merge, split partition operations; attach and detach operations; and INTERVAL partitioning.
pg_pathman has many optimizations for partitioned table execution plans.
pg_pathman only supports RANGE and HASH partition types.
The pathman_config table stores partition configuration information; it provides partition task views.
Partition information is cached in memory for execution plan generation.

Basic pg_pathman Usage
#

Creating pathman RANGE partitions

-- The regular table serves as the parent table
CREATE TABLE journal (
 id SERIAL,
 dt TIMESTAMP NOT NULL,
 level INTEGER,
 msg TEXT);

-- Indexes on the parent table are automatically created on child partitions
CREATE INDEX ON journal(dt);
-- Create partitions
select 
create_range_partitions('journal'::regclass, 
 'dt',
 '2023-01-01 00:00:00'::timestamp,
 interval '1 month', 
 6, 
 false) ; 

-- View table definition
=> \d+ journal
 Table "public.journal"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
--------+-----------------------------+-----------+----------+-------------------------------------+----------+--------------+-------------
 id | integer | | not null | nextval('journal_id_seq'::regclass) | plain | | 
 dt | timestamp without time zone | | not null | | plain | | 
 level | integer | | | | plain | | 
 msg | text | | | | extended | | 
Indexes:
 "journal_dt_idx" btree (dt)
Child tables: journal_1,
 journal_2,
 journal_3,
 journal_4,
 journal_5,
 journal_6
Access method: heap

=> \d+ journal_6
 Table "public.journal_6"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
--------+-----------------------------+-----------+----------+-------------------------------------+----------+--------------+-------------
 id | integer | | not null | nextval('journal_id_seq'::regclass) | plain | | 
 dt | timestamp without time zone | | not null | | plain | | 
 level | integer | | | | plain | | 
 msg | text | | | | extended | | 
Indexes:
 "journal_6_dt_idx" btree (dt)
Check constraints:
 "pathman_journal_6_check" CHECK (dt >= '2023-06-01 00:00:00'::timestamp without time zone AND dt < '2023-07-01 00:00:00'::timestamp without time zone)
Inherits: journal
Access method: heap

-- Insert data
INSERT INTO journal (dt, level, msg)
SELECT g, random() * 10000, md5(g::text)
FROM generate_series('2023-01-01'::date, '2023-02-28'::date, '1 hour') as g;

-- Insert data for which no corresponding partition has been created yet
=> INSERT INTO journal (dt, level, msg) values('2023-07-01'::date,'11','1');
INSERT 0 1
-- Check partition data distribution; the INTERVAL partition has been automatically created
=> SELECT tableoid::regclass AS partition, count(*) FROM journal group by partition;
 partition | count 
-----------+-------
 journal_7 | 1
 journal_2 | 649
 journal_1 | 744

-- View execution plan
-- Partition pruning has occurred
=> explain select * from journal where dt='2023-01-01 22:00:00';
 QUERY PLAN 
-----------------------------------------------------------------------------------------------------
 Append (cost=0.00..5.30 rows=2 width=48)
 -> Seq Scan on journal journal_1 (cost=0.00..0.00 rows=1 width=48)
 Filter: (dt = '2023-01-01 22:00:00'::timestamp without time zone)
 -> Index Scan using journal_1_dt_idx on journal_1 journal_1_1 (cost=0.28..5.29 rows=1 width=49)
 Index Cond: (dt = '2023-01-01 22:00:00'::timestamp without time zone)

Creating pathman HASH partitions

-- Create parent table
CREATE TABLE items (
 id SERIAL PRIMARY KEY,
 name TEXT,
 code BIGINT);
-- Create HASH partitions
select create_hash_partitions('items'::regclass, 
 'id',
 3, 
 false) ; 
-- Insert data 
INSERT INTO items (id, name, code)
SELECT g, md5(g::text), random() * 100000
FROM generate_series(1, 1000) as g;

=> SELECT tableoid::regclass AS partition, count(*) FROM items group by partition;
 partition | count 
-----------+-------
 items_2 | 344
 items_0 | 318
 items_1 | 338
=> \d+ items
 Table "public.items"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
--------+---------+-----------+----------+-----------------------------------+----------+--------------+-------------
 id | integer | | not null | nextval('items_id_seq'::regclass) | plain | | 
 name | text | | | | extended | | 
 code | bigint | | | | plain | | 
Indexes:
 "items_pkey" PRIMARY KEY, btree (id)
Child tables: items_0,
 items_1,
 items_2
Access method: heap

=> \d+ items_1
 Table "public.items_1"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
--------+---------+-----------+----------+-----------------------------------+----------+--------------+-------------
 id | integer | | not null | nextval('items_id_seq'::regclass) | plain | | 
 name | text | | | | extended | | 
 code | bigint | | | | plain | | 
Indexes:
 "items_1_pkey" PRIMARY KEY, btree (id)
Check constraints:
 "pathman_items_1_check" CHECK (get_hash_part_idx(hashint4(id), 3) = 1)
Inherits: items
Access method: heap
=> SELECT tableoid::regclass AS partition, count(*) FROM items group by partition;
 partition | count 
-----------+-------
 items_2 | 344
 items_0 | 318
 items_1 | 338

Pros and Cons of PostgreSQL Partitioned Tables
#

Advantages of Partitioned Tables
#

SQL performance improvement. In certain scenarios, such as splitting a large amount of data into multiple partitions where SQL only needs to query one partition, SQL performance can be dramatically improved.
Partitions can work together with indexes. For example, accessing an index on a single partition is more efficient than accessing a large unpartitioned index.
Dropping a single partition is much more efficient than deleting many rows. This is common in time-range partitioning — dropping an unused historical partition is very fast, but without partitioning, DELETE operations are not only slow but also require additional maintenance.
VACUUM is faster. Reclaiming old version information or collecting statistics on a large table is very slow. If VACUUM hasn’t finished executing, SQL may already be experiencing problems. With partitioning, VACUUM becomes much faster.
I/O distribution capability. Different partitions can be placed on different paths or different disks. Rarely-used data can be placed on cheaper disks.
More maintenance techniques. Directly maintaining a very large table is difficult — for example, VACUUM on an extremely large table has many issues. With partitioned tables, each partition can run VACUUM independently. Moreover, ATTACH/DETACH, local indexes/constraints, and more can be flexibly used in many scenarios.

Disadvantages of Partitioned Tables
#

In PostgreSQL, every partition of a partitioned table can be treated as a regular table. Too many partitions can lead to longer SQL parsing times and higher memory load, even causing errors. See the previous article: Too many range table entries even with a modest number of partitions
Even if having too many partitions doesn’t cause errors, and partition pruning doesn’t happen during execution plan generation (it might happen during execution), the EXPLAIN output will be extremely long. At that point, the logs will also contain lengthy execution plans, affecting log readability.
Some strange issues: Different users see different execution plans

Limitations of Partitioned Tables
#

No native automatic partition creation feature
Only local partition indexes are supported; global indexes are not supported
Primary keys must include the partition key. PostgreSQL currently can only enforce uniqueness within each partition, hence this limitation. Oracle and MySQL do not have this restriction.
Unique indexes must include the partition key. PostgreSQL currently can only enforce uniqueness within each partition. Same applies to primary keys.
Cannot create globally-defined constraints
BEFORE ROW INSERT triggers cannot update the partition into which the row is being inserted.
Temporary table partitions and regular table partitions cannot coexist under the same partitioned table.
In declarative partitioning, parent and child table columns must be identical; in inheritance partitioning, child tables can have more columns than the parent table.
In declarative partitioning, CHECK and NOT NULL constraints are always inherited; these two constraints cannot be set independently on individual partitions.
RANGE partitions cannot store NULL values. HASH partitions have no concept of NULL partitions but can store NULL values — they are placed on the remainder 0 partition. LIST partitions can explicitly create a NULL partition to store NULL data.

When Should You Use Partitioned Tables?
#

First, to use partitioned tables you must understand the advantages, disadvantages, and limitations they bring. For example, when data volume is large, partitioning can improve performance; hot/cold data separation also makes partition data management easier. You should decide whether to partition and how to partition based on your specific business situation and hardware resources. However, developers will always ask questions like “how much data warrants partitioning.” Advice on using partitioned tables can only be given in general terms. If you don’t know how to partition, you can refer to the following recommendations (if you already have sufficient understanding of table partitioning, please ignore):

The table data is large enough, and SQL queries on the table always or can include the partition key column.
Clear hot/cold data separation. For example, new data is always inserted into the current month’s partition, while the other 11 months of old partitions are read-only.
VACUUM can no longer keep up.

Partition Table Permissions
#

Permission issues are less discussed in the context of partitioned table knowledge, but they are still worth paying attention to. Because PostgreSQL has the concept that “partition child tables are also regular tables,” this differs from other common databases (Oracle, MySQL). For example, in Oracle you don’t need to worry about partition child table permissions, but in PostgreSQL you do.

PARTITION OF / ATTACH do not inherit the parent table’s permissions to child tables:

-- Grant SELECT on the partitioned table to a regular user
=> grant select on lzlpartition1 to userlzl;
GRANT

-- Check permissions — only the parent table has been granted; existing partition child tables are not automatically granted
=> select grantee,table_schema,table_name,privilege_type from information_schema.table_privileges where grantee='userlzl';
 grantee | table_schema | table_name | privilege_type 
---------+--------------+---------------+----------------
 userlzl | public | lzlpartition1 | SELECT
 
 -- Create a partition using PARTITION OF
 => create table LZLPARTITION1_202303 partition of LZLPARTITION1 for values from ('2023-03-01 00:00:00') to ('2023-04-01 00:00:00');
CREATE TABLE

-- Create a partition using ATTACH
=> CREATE TABLE lzlpartition1_202304
-> (LIKE lzlpartition1 INCLUDING DEFAULTS INCLUDING CONSTRAINTS);
CREATE TABLE
=> alter table lzlpartition1 attach partition lzlpartition1_202304 for values from ('2023-04-01 00:00:00') to ('2023-05-01 00:00:00');
ALTER TABLE

-- Check permissions again — newly created child partitions are not automatically granted to other users (but permissions are automatically granted to the owner)
=> select grantee,table_schema,table_name,privilege_type from information_schema.table_privileges where grantee='userlzl';
 grantee | table_schema | table_name | privilege_type 
---------+--------------+---------------+----------------
 userlzl | public | lzlpartition1 | SELECT

At this point, user userlzl has no access permissions to any child tables, but has permissions on the parent table. userlzl can access partition data through the parent table, but cannot access data by directly querying child tables:

=> \c - userlzl;
You are now connected to database "dbmgr" as user "userlzl".
=> select * from LZLPARTITION1 where date_created='2023-01-02 10:00:00';
 id | name | date_created 
------+----------------------------------+---------------------
 2159 | d05d716da126ff4b44d934344cc4dd7a | 2023-01-02 10:00:00

=> select * from LZLPARTITION1_202301 where date_created='2023-01-02 10:00:00';
ERROR: 42501: permission denied for table lzlpartition1_202301
LOCATION: aclcheck_error, aclchk.c:3466

Since ATTACH/DETACH does not handle permissions, if we DETACH a partition at this point, that partition will also be inaccessible to userlzl:

=> alter table lzlpartition1 detach partition lzlpartition1_202303;
ALTER TABLE
=> \dp+ lzlpartition1_202303;
 Access privileges
 Schema | Name | Type | Access privileges | Column privileges | Policies 
--------+----------------------+-------+-------------------+-------------------+----------
 dbmgr | lzlpartition1_202303 | table | | | 
=> select * from LZLPARTITION1_202301 where date_created='2023-01-02 10:00:00';
ERROR: 42501: permission denied for table lzlpartition1_202301 

From this we can conclude:

Partition child tables and the parent table exist as regular tables in PostgreSQL, each with their own permission system.
If you lack child table permissions but have parent table permissions, you can still access child table data.
PARTITION OF, ATTACH, and DETACH do not handle permission issues.

However, partition table permissions do not merely control whether access is possible. Lacking partition child table permissions can lead to abnormal execution plans. Reference article: Different users see different execution plans This issue is an intermittent phenomenon that causes superusers and regular users to see different SQL execution plans. The actual business SQL execution plan is abnormal but goes unnoticed, making it difficult to diagnose. Partition child tables have their own statistics, and child table permissions are inconsistent with the parent table (even for partitions created via PARTITION OF), resulting in users being able to access child table data through the parent table but unable to view the child table’s statistics. This permission issue leads to differences in execution plans. This contradicts the general concept that “permissions only control whether you can access a table, not how you access it,” so attention must be paid to this permission issue. To provide permission for child table statistics, it is recommended to explicitly grant SELECT on all child tables to the user, which avoids the issues above:

grant select on table_partition_allname to username;

Partition Table Maintenance
#

ATTACH/DETACH Basic Operations
#

ATTACH/DETACH can add/detach an existing table as a partition of/detach from a partitioned table. ATTACH/DETACH is very useful in maintenance work. First, let’s look at the locking behavior of adding partitions via “CREATE TABLE … PARTITION OF” and deleting partitions via “DROP TABLE”:

Lock Matrix: https://www.postgresql.org/docs/current/explicit-locking.html

Lock Requests: https://postgres-locks.husseinnasser.com

Adding a partition via PARTITION OF

-- Session 1: Start a transaction, read-only data
=> select * from lzlpartition1 where date_created='2023-01-01 00:00:00';
 id | name | date_created 
------+----------------------------------+---------------------
 8249 | 256ac66bb53d31bc6124294238d6410c | 2023-01-01 00:00:00

-- Session 3: Check lock status. When reading data from one partition, locks are acquired on both the child partition and the parent table.
=> select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+--------+-----------------+---------
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 311449 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 311449 | AccessShareLock | t

-- Session 2: Add a partition via PARTITION OF
=> create table LZLPARTITION1_202305 partition of LZLPARTITION1 for values from ('2023-05-01 00:00:00') to ('2023-06-01 00:00:00');
-- Waiting

-- Session 3: Check locks again
=> select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+--------+---------------------+---------
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 311449 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 311449 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 308525 | AccessExclusiveLock | f -- This is the PARTITION OF session

-- Session 4: Run an arbitrary query
=> select * from lzlpartition1 where date_created='2023-01-01 00:00:00';
-- Waiting

-- Session 4: Check locks again
=> select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+--------+---------------------+---------
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 311449 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 311449 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 308525 | AccessExclusiveLock | f
 relation | dbmgr | lzlpartition1 | [null] | [null] | 84774 | AccessShareLock | f -- Query is blocked

When adding a partition via PARTITION OF, an AccessExclusiveLock is requested on the parent table. This waits for all transactions on the parent table and also blocks all transactions on the parent table.

Although the PARTITION OF statement itself executes quickly, if there are long-running transactions on the parent table, all operations on the partitioned table will stall for an extended period. Without a maintenance window, using PARTITION OF to add partitions directly is not recommended.

Dropping a partition via DROP TABLE

-- Session 1: Start another read-only transaction
-- Session 2: Drop a child partition of the partitioned table
=> drop table lzlpartition1_202305;
-- Waiting

-- Session 3: Check lock status
=> 
select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+--------+---------------------+---------
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 311449 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 311449 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 308525 | AccessExclusiveLock | f

Dropping a child partition with DROP TABLE requests an AccessExclusiveLock on the parent table, waiting for all and blocking all. Similarly, this must be used with caution in production environments.

ATTACH — adding a partition ATTACH attaches an existing regular table to a partitioned table. Although both ATTACH and PARTITION OF can add partitions, note that ATTACH does not automatically create indexes, constraints, default values, or row-level triggers — this differs from PARTITION OF. First, create a table:

-- To reduce tedious DDL, use LIKE to create the table
CREATE TABLE lzlpartition1_202305
 (LIKE lzlpartition1 INCLUDING DEFAULTS INCLUDING CONSTRAINTS);

Now observe whether ATTACH is blocked:

-- Session 1: Start a read-write transaction
=>begin;
BEGIN
=> insert into lzlpartition1 values('1234','abcd','2023-01-01 01:00:00');
INSERT 0 1
-- Session 3: Check lock status
=> 
select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+--------+------------------+---------
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 311449 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 311449 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 311449 | RowExclusiveLock | t
-- DML statements acquire RowExclusiveLock on the partition parent table and the corresponding partition child table

-- Session 2: ATTACH the newly created table to the partition parent table
=> alter table lzlpartition1 attach partition lzlpartition1_202305 for values from ('2023-05-01 00:00:00') to ('2023-06-01 00:00:00');
ALTER TABLE

ATTACH only requests a SHARE UPDATE EXCLUSIVE lock, which is much lighter than ACCESS EXCLUSIVE.

ATTACH does not block reads or writes, so ATTACH is recommended for adding partitions — it does not affect business operations and can be executed online.

DETACH — removing a partition

DETACH removes a partition from the partitioned table, turning it into a regular table:

-- Session 1: Keep the DML transaction uncommitted
-- Session 2: DETACH a partition
 alter table lzlpartition1 detach partition lzlpartition1_202305;
 -- Waiting

-- Session 3: Check lock status
 => select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+--------+---------------------+---------
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 311449 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 311449 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 308525 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 308525 | AccessExclusiveLock | f

Unlike ATTACH, DETACH requests an AccessExclusiveLock on the parent table, waiting for all and blocking all.

DETACH CONCURRENTLY

Starting from PostgreSQL 14, DETACH gained two new syntax variants: CONCURRENTLY and FINALIZE.

ALTER TABLE [ IF EXISTS ] name DETACH PARTITION partition_name [ CONCURRENTLY | FINALIZE ]

DETACH CONCURRENTLY internally starts two transactions. The first transaction requests a SHARE UPDATE EXCLUSIVE lock on both the parent and child tables, marking the partition as being in a detaching state, at which point it waits for all transactions on the partitioned table to commit. Once all those transactions have committed, the second transaction requests a SHARE UPDATE EXCLUSIVE lock on the parent table and an ACCESS EXCLUSIVE lock on that child table, after which DETACH CONCURRENTLY completes.

Additionally, after DETACH CONCURRENTLY, the detached child table retains its constraint — the partition constraint is converted into a CHECK constraint on the detached table.

DETACH CONCURRENTLY limitations:

DETACH CONCURRENTLY cannot be placed inside a transaction block.
The partitioned table cannot have a DEFAULT partition.

Locking behavior of CONCURRENTLY:

-- Session 1
lzldb=> begin;
BEGIN
lzldb=*> insert into lzlpartition1 values('1234','abcd','2023-01-01 01:00:00');
INSERT 0 1

-- Session 2: DETACH CONCURRENTLY
lzldb=> alter table lzlpartition1 detach partition lzlpartition1_202301 concurrently;
-- Waiting

-- Session 3: Check locks
 3691 | insert into lzlpartition1 values('1234','abcd','2023-01-01 01:00:00'); | Client | ClientRead
 3940 | alter table lzlpartition1 detach partition lzlpartition1_202301 concurrently; | Lock | virtualxid
 3947 | select pid,query,wait_event_type,wait_event from pg_stat_activity; | | 
-- The DETACH session is 3940. Interestingly, the DETACH wait event is virtualxid, and the wait event type is Lock.

-- Check lock details
lzldb=> select locktype,database,relation,virtualtransaction,pid,mode,granted from pg_locks where pid in (3691,3940);
 locktype | database | relation | virtualtransaction | pid | mode | granted 
---------------+----------+----------+--------------------+------+------------------+---------
 virtualxid | | | 6/9 | 3940 | ExclusiveLock | t
 relation | 16387 | 40969 | 5/179 | 3691 | RowExclusiveLock | t
 relation | 16387 | 40963 | 5/179 | 3691 | RowExclusiveLock | t
 virtualxid | | | 5/179 | 3691 | ExclusiveLock | t
 virtualxid | | | 6/9 | 3940 | ShareLock | f
 transactionid | | | 5/179 | 3691 | ExclusiveLock | t
-- At this point, DETACH is not yet waiting for a table-level lock; it is waiting for a ShareLock on virtualxid

-- Session 4: Try an insert
lzldb=> insert into lzlpartition1 values('12345','abcd','2023-01-01 01:00:00');
ERROR: no partition of relation "lzlpartition1" found for row
DETAIL: Partition key of the failing row contains (date_created) = (2023-01-01 01:00:00).
lzldb=> insert into lzlpartition1 values('12345','abcd','2023-02-01 01:00:00');
INSERT 0 1
-- The detaching partition can no longer accept inserts, but other partitions can.
-- What if we insert directly into the partition? It works fine.
lzldb=> insert into lzlpartition1_202301 values('12345','abcd','2023-01-01 01:00:00');
INSERT 0 1
-- Note: at this point it is still a partition of the partitioned table, not yet a regular table, but it has been marked as unavailable.

-- \d+ shows the partition in DETACH PENDING state
Partitions: lzlpartition1_202301 FOR VALUES FROM ('2023-01-01 00:00:00') TO ('2023-02-01 00:00:00') (DETACH PENDING),
 lzlpartition1_202302 FOR VALUES FROM ('2023-02-01 00:00:00') TO ('2023-03-01 00:00:00')
 

-- Commit/rollback the insert session (Session 1)
lzldb=> rollback;
ROLLBACK

-- Session 2 completes immediately
lzldb=> alter table lzlpartition1 detach partition lzlpartition1_202301 concurrently;
ALTER TABLE

FINALIZE:

-- Session 1
lzldb=> begin;
BEGIN
lzldb=*> insert into lzlpartition1 values('1234','abcd','2023-01-01 01:00:00');
INSERT 0 1

-- Session 2: DETACH CONCURRENTLY, manually canceled
lzldb=> alter table lzlpartition1 detach partition lzlpartition1_202301 concurrently;
^CCancel request sent
ERROR: canceling statement due to user request

-- \d+ shows the partition in DETACH PENDING state
Partitions: lzlpartition1_202301 FOR VALUES FROM ('2023-01-01 00:00:00') TO ('2023-02-01 00:00:00') (DETACH PENDING),
 lzlpartition1_202302 FOR VALUES FROM ('2023-02-01 00:00:00') TO ('2023-03-01 00:00:00')

-- In DETACH PENDING state, SQL no longer accesses this partition
lzldb=> explain select * from lzlpartition1;
 QUERY PLAN 
-----------------------------------------------------------------------------------------
 Seq Scan on lzlpartition1_202302 lzlpartition1 (cost=0.00..752.81 rows=38881 width=45)

-- Use FINALIZE to complete the detach
lzldb=> alter table lzlpartition1 detach partition lzlpartition1_202301 finalize; 
-- Waiting

-- Check lock status
lzldb=> select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';lzldb-# 
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+------+--------------------------+---------
 relation | lzldb | lzlpartition1 | | | 3691 | RowExclusiveLock | t
 relation | lzldb | lzlpartition1_202301 | | | 3940 | AccessExclusiveLock | f
 relation | lzldb | lzlpartition1 | | | 3940 | ShareUpdateExclusiveLock | t
 relation | lzldb | lzlpartition1_202301 | | | 3691 | RowExclusiveLock | t
-- 3940, FINALIZE requests ShareUpdateExclusiveLock on the parent table and AccessExclusiveLock on the child table
-- Since the inserted data happened to be in the detaching partition, it is waiting

-- Session 1 ends
lzldb=!> rollback;
ROLLBACK
-- Session 2 completes immediately
lzldb=> alter table lzlpartition1 detach partition lzlpartition1_202301 finalize; 
ALTER TABLE
 

Although DETACH requests an 8-level lock on the partition, generally business operations don’t write directly through child partitions, so you only need to ensure that long-running transactions on the partitioned table complete quickly. Usually, there’s no need to worry about subsequent blocking on that partition’s child table.

Online DETACH summary:

The blocking behavior of DETACH CONCURRENTLY is somewhat similar to CIC (CREATE INDEX CONCURRENTLY) — it does not block other transactions, but it itself waits for existing transactions to complete. This is not easily visible from lock information alone.
During DETACH CONCURRENTLY, the partition enters a DETACH PENDING intermediate state. This state is somewhat like INVISIBLE — SQL will not find this partition.
If DETACH PENDING is caused by long-running transactions, promptly end those transactions; if it’s caused by interruption, use FINALIZE to complete the detach.

Using Constraints to Reduce ATTACH Time
#

Partition data overview — prepare to ATTACH a relatively large partition:

=> SELECT tableoid::regclass AS partition, count(*) FROM lzlpartition1 group by partition;
 partition | count 
----------------------+---------
 lzlpartition1_202301 | 2592001
 lzlpartition1_202302 | 38881

Note: this 202301 partition has a PARTITION CONSTRAINT:

=> \d+ lzlpartition1_202301
 Table "public.lzlpartition1_202301"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
--------------+-----------------------------+-----------+----------+---------+----------+--------------+-------------
id | integer | | not null | | plain | | 
name | character varying(50) | | | | extended | | 
date_created | timestamp without time zone | | not null | now() | plain | | 
Partition of: lzlpartition1 FOR VALUES FROM ('2023-01-01 00:00:00') TO ('2023-02-01 00:00:00')
Partition constraint: ((date_created IS NOT NULL) AND (date_created >= '2023-01-01 00:00:00'::timestamp without time zone) AND (date_created < '2023-02-01 00:00:00'::timestamp without t
Indexes:
 "lzlpartition1_202301_pkey" PRIMARY KEY, btree (id, date_created)
Access method: heap

DETACH the partition:

 alter table lzlpartition1 detach partition lzlpartition1_202301;

-- After DETACH, the PARTITION CONSTRAINT is gone
=> \d+ lzlpartition1_202301
 Table "public.lzlpartition1_202301"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
--------------+-----------------------------+-----------+----------+---------+----------+--------------+-------------
id | integer | | not null | | plain | | 
name | character varying(50) | | | | extended | | 
date_created | timestamp without time zone | | not null | now() | plain | | 
Indexes:
 "lzlpartition1_202301_pkey" PRIMARY KEY, btree (id, date_created)
Access method: heap

ATTACH without adding a CHECK constraint:

=> alter table lzlpartition1 attach partition lzlpartition1_202301 for values from ('2023-01-01 00:00:00') to ('2023-02-01 00:00:00');
ALTER TABLE
Time: 343.498 ms

Because it must scan the partition data to verify it satisfies the partition range, ATTACH took 300+ ms.

Add a CHECK constraint first, then ATTACH:

=> alter table lzlpartition1 detach partition lzlpartition1_202301;
=> alter table lzlpartition1_202301 add constraint chk_202301 CHECK
-> ((date_created IS NOT NULL) AND (date_created >= '2023-01-01 00:00:00'::timestamp without time zone) AND (date_created < '2023-02-01 00:00:00'::timestamp without time zone));
ALTER TABLE
Time: 355.458 ms

The time taken to add the CHECK constraint is roughly the same as the ATTACH operation without a CHECK — because adding a CHECK constraint also needs to scan and validate all data. Once the CHECK constraint is added, the subsequent ATTACH completes very quickly:

=> alter table lzlpartition1 attach partition lzlpartition1_202301 for values from ('2023-01-01 00:00:00') to ('2023-02-01 00:00:00');
ALTER TABLE
Time: 1.480 ms

Drop the CHECK constraint:

=> \d+ lzlpartition1_202301;
 Table "public.lzlpartition1_202301"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
--------------+-----------------------------+-----------+----------+---------+----------+--------------+-------------
 id | integer | | not null | | plain | | 
 name | character varying(50) | | | | extended | | 
 date_created | timestamp without time zone | | not null | now() | plain | | 
Partition of: lzlpartition1 FOR VALUES FROM ('2023-01-01 00:00:00') TO ('2023-02-01 00:00:00')
Partition constraint: ((date_created IS NOT NULL) AND (date_created >= '2023-01-01 00:00:00'::timestamp without time zone) AND (date_created < '2023-02-01 00:00:00'::timestamp without t
Indexes:
 "lzlpartition1_202301_pkey" PRIMARY KEY, btree (id, date_created)
Check constraints:
 "chk_202301" CHECK (date_created IS NOT NULL AND date_created >= '2023-01-01 00:00:00'::timestamp without time zone AND date_created < '2023-02-01 00:00:00'::timestamp without time 
Access method: heap

Note: CHECK CONSTRAINT and PARTITION CONSTRAINT are different concepts, even though their constraint content can be identical. ATTACH uses the CHECK constraint but does not merge it. You can explicitly drop this redundant CHECK:

=> alter table lzlpartition1_202301 drop constraint chk_202301;
ALTER TABLE

Additionally, note that DROP CONSTRAINT requests an AccessExclusiveLock on the current child partition — this is the highest-level lock and blocks all operations. So, if there are transactions on that child partition, be cautious with DROP CONSTRAINT.

=> select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+--------+---------------------+---------
 relation | dbmgr | lzlpartition1 | [null] | [null] | 448243 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 448243 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 444399 | AccessShareLock | t
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 444399 | AccessExclusiveLock | f -- This is the DROP CONSTRAINT session
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 448243 | RowExclusiveLock | t

So, When ATTACH-ing a partition, adding a CHECK constraint beforehand is useful — it reduces ATTACH execution time. The data validation just needs to be completed before ATTACH.

The Correct Way to Add Partitions to a Partitioned Table
#

We now know that ATTACH can be executed online, while PARTITION OF / DROP TABLE / DETACH all request an AccessExclusiveLock that waits for and blocks everything. So, It is recommended to use ATTACH to create new partitions. PARTITION OF / DETACH both wait for and block all transactions, while ATTACH is not blocked by read-only/DML transactions. Therefore, adding partitions should use ATTACH, and a CHECK constraint should be created beforehand. When dropping constraints, be mindful of long-running transactions. The correct way to add a partition to a partitioned table:

-- To reduce tedious DDL, use LIKE to create the table
CREATE TABLE lzlpartition1_202303
 (LIKE lzlpartition1 INCLUDING DEFAULTS INCLUDING CONSTRAINTS);
-- Refer to the PARTITION CONSTRAINT of other partitions, add a CHECK constraint on the table to reduce ATTACH constraint validation time
alter table lzlpartition1_202303 add constraint chk_202303 CHECK ((date_created IS NOT NULL) AND (date_created >= '2023-03-01 00:00:00'::timestamp without time zone) AND (date_created < '2023-04-01 00:00:00'::timestamp without time zone));
-- Add partition using ATTACH
alter table LZLPARTITION1 attach partition LZLPARTITION1_202303 for values from ('2023-03-01 00:00:00') to ('2023-04-01 00:00:00');
-- Optional. Drop the redundant CHECK constraint before transactions start on the new partition
alter table lzlpartition1_202303 drop constraint chk_202303;

Locks on Partition Indexes
#

Creating/dropping partition indexes during read-only transactions

When a partition has a shared lock (AccessShareLock), meaning there is a query transaction on the partitioned table: CREATE INDEX ON lzlpartition1 succeeds (note: without CONCURRENTLY); DROP INDEX lzlpartition1 fails:

-- Session 1: Start a transaction, read data from the partitioned table
=> begin;
BEGIN
=> select count(*) from lzlpartition1 where date_created>='2023-01-01 00:00:00' and date_created<='2023-01-02 00:00:00';
 count 
-------
 86401
(1 row)
-- Session 2: Create index, succeeds
=> create index idx_datecreated on lzlpartition1(date_created);;
CREATE INDEX

-- Session 2: Drop index, waits
=> drop index idx_datecreated;

-- Session 3: Check locks
=> select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+---------------------------+------------+---------------+--------+---------------------+---------
 relation | dbmgr | lzlpartition1_202301_pkey | [null] | [null] | 300371 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 99598 | AccessExclusiveLock | f
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 300371 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 300371 | AccessShareLock | t

CREATE INDEX does not request an AccessExclusiveLock on the table, but DROP INDEX does. From this example we can conclude: Read-only transactions do not block CREATE INDEX, but they do block DROP INDEX.

Creating/dropping partition indexes during update transactions

-- Session 1: Start an update transaction
=> begin;
BEGIN
=> update lzlpartition1 set name='abc' where date_created='2023-01-01 10:00:00';
UPDATE 1
-- Session 2: Create partition index, waits
=> create index idx_datecreated on lzlpartition1(date_created);

-- Session 3: Check lock status
=>select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
-> where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+---------------------------+------------+---------------+--------+------------------+---------
 relation | dbmgr | lzlpartition1_202301_pkey | [null] | [null] | 300371 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 99598 | ShareLock | f
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 300371 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 300371 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 300371 | RowExclusiveLock | t

The CREATE INDEX session (99598) requests a ShareLock on the partition parent table; the DML transaction session (300371) holds RowExclusiveLock on the child partition and parent table.

CREATE INDEX (without CONCURRENTLY) requests ShareLock on the parent table; Read-only transactions request AccessShareLock on the parent and child tables; Update transactions request RowExclusiveLock on the parent and child tables; ==> AccessShareLock does not block ShareLock, so queries do not block CREATE INDEX (without CONCURRENTLY); RowExclusiveLock blocks ShareLock, so DML blocks CREATE INDEX (without CONCURRENTLY);

Creating partitioned indexes with CONCURRENTLY

Note: You cannot create indexes with CONCURRENTLY on a partitioned table.

=> create index concurrently idx_datecreated on lzlpartition1(date_created);
ERROR: 0A000: cannot create index on partitioned table "lzlpartition1" concurrently
LOCATION: DefineIndex, indexcmds.c:665

There is a patch at https://commitfest.postgresql.org/35/2815/ working on solving this issue.

Currently, you can create indexes with CONCURRENTLY on individual partition child tables:

-- Session 1: Still using the previous DML transaction
-- Session 2: Create index with CONCURRENTLY on a child table, waits
=> create index concurrently idx_datecreated_202301 on lzlpartition1_202301(date_created);

-- Session 3: Check lock status
=> select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+---------------------------+------------+---------------+--------+--------------------------+---------
 relation | dbmgr | lzlpartition1_202301_pkey | [null] | [null] | 300371 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 99598 | ShareUpdateExclusiveLock | t
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 300371 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 300371 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 300371 | RowExclusiveLock | t

With CONCURRENTLY, the requested lock is one level lower and no longer conflicts with ROW EXCL. The locks don’t conflict, so why is CONCURRENTLY itself still blocked?

it must wait for all existing transactions that could potentially modify or use the index to terminate.

The official documentation explains that CONCURRENTLY needs to wait for transactions that could potentially modify or use the index to terminate. In our case, the UPDATE statement modified the indexed column, so CONCURRENTLY needs to wait for it to complete. Although CONCURRENTLY itself hasn’t completed due to the prior DML statement, there’s a benefit: CONCURRENTLY does not block subsequent DML statements.

-- While CONCURRENTLY has not yet completed
-- Session 4: Update a record
=> update lzlpartition1 set name='abc' where date_created='2023-01-01 12:00:00';
UPDATE 1

Summary of partition index locking issues:

Locking for read-only/read-write/index creation on partitioned tables is similar to regular tables. Just note that transactions acquire locks on both the partition parent table and child tables, so when subsequent blocking chains involve heavier locks, all partitions are affected.
Read-only transactions do not block CREATE INDEX, but they do block DROP INDEX.
DML blocks CREATE INDEX and also blocks CREATE INDEX CONCURRENTLY, but CONCURRENTLY does not block DML.
Although CREATE INDEX on a partitioned table automatically creates indexes on all existing and future partitions, it is not recommended for direct use in production due to blocking issues.
You cannot use CONCURRENTLY directly on the partition parent table, so you need to create indexes with CONCURRENTLY on each partition child table.
CONCURRENTLY does not block subsequent transactions but itself gets blocked by prior long-running transactions and may cause the created index to be invalid. Attention must be paid to long-running transactions.

The Correct Way to Create Partition Indexes
#

Although you cannot create indexes with CONCURRENTLY on a partitioned table, you can create indexes with CONCURRENTLY on partition child tables using the following syntax: CREATE INDEX ON ONLY : Creates an invalid index on the parent table; does not automatically create indexes on child partitions. CREATE INDEX CONCURRENTLY : Creates an index with CONCURRENTLY on a child partition. ALTER INDEX .. ATTACH PARTITION : Attaches the partition index to the parent index. After all child partition indexes have been attached, the partition parent table index is automatically marked as valid. However, when executing these commands, you still need to pay attention to locking behavior.

Below, observe the lock requests and blocking for the above two statements: (DML explicit transaction in Session 1 is kept open throughout)

Blocking behavior of CREATE INDEX ON ONLY:

=> CREATE INDEX IDX_DATECREATED ON ONLY lzlpartition1(date_created);
-- Waiting

-- Check lock status
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+--------+------------------+---------
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 448243 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 448243 | RowExclusiveLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 444399 | ShareLock | f

CREATE INDEX ON ONLY requests a ShareLock. ShareLock and RowExclusiveLock block each other. So, although ONLY itself executes very quickly, CREATE INDEX ON ONLY should not be used casually either.

-- After the DML transaction ends, CREATE INDEX ON ONLY completes
 "idx_datecreated" btree (date_created) INVALID

CREATE INDEX ON ONLY creates an invalid index on the partition parent table and does not create indexes on child partitions.

Blocking behavior of ATTACH index:

-- After ONLY index creation completes, start another DML explicit transaction in Session 1
=> begin;
BEGIN
=> insert into lzlpartition1 values('1111','abc','2023-01-01 00:00:00');
INSERT 0 1

-- Session 2: Create index with CONCURRENTLY on child partition
=> create index concurrently idx_datecreated_202302 on lzlpartition1_202302(date_created);
CREATE INDEX -- 202302 partition index created
=> create index concurrently idx_datecreated_202304 on lzlpartition1_202304(date_created);
CREATE INDEX -- 202304 partition index created
=> create index concurrently idx_datecreated_202301 on lzlpartition1_202301(date_created);
---- Creating 202301 partition index, waiting

CONCURRENTLY waits for transactions that might use the index to complete. Our explicit transaction only inserted into the 202301 partition, so only this partition’s CONCURRENTLY index creation hasn’t completed.

-- Complete the DML explicit transaction in Session 1, wait for the index to finish, then start another transaction
=> commit;
COMMIT
=> begin;
BEGIN
=> insert into lzlpartition1 values('1111','abc','2023-01-01 00:00:01');
INSERT 0 1
-- Session 2: ATTACH index
 => ALTER INDEX idx_datecreated ATTACH PARTITION idx_datecreated_202302;
ALTER INDEX -- ATTACH successful
=> \d+ idx_datecreated
 Partitioned index "public.idx_datecreated"
 Column | Type | Key? | Definition | Storage | Stats target 
--------------+-----------------------------+------+--------------+---------+--------------
 date_created | timestamp without time zone | yes | date_created | plain | 
btree, for table "public.lzlpartition1", invalid
Partitions: idx_datecreated_202302 -- 202302 child partition index has been attached, index still invalid
Access method: btree
-- Attach the remaining child partition indexes
=> ALTER INDEX idx_datecreated ATTACH PARTITION idx_datecreated_202301;
ALTER INDEX -- ATTACH successful
=> ALTER INDEX idx_datecreated ATTACH PARTITION idx_datecreated_202304;
ALTER INDEX -- ATTACH successful
-- After all child partition indexes are attached, the parent table index automatically becomes valid
=> \d+ idx_datecreated
 Partitioned index "public.idx_datecreated"
 Column | Type | Key? | Definition | Storage | Stats target 
--------------+-----------------------------+------+--------------+---------+--------------
 date_created | timestamp without time zone | yes | date_created | plain | 
btree, for table "public.lzlpartition1"
Partitions: idx_datecreated_202301,
 idx_datecreated_202302,
 idx_datecreated_202304
Access method: btree

ATTACH is not blocked by DML and completes immediately. At this point, new partitions created via PARTITION OF will also automatically get the child partition index.

In summary,

CREATE INDEX ON ONLY requests a ShareLock, which mutually blocks with the RowExclusiveLock requested by DML.
CREATE INDEX CONCURRENTLY requests a ShareUpdateExclusiveLock, which does not block the RowExclusiveLock requested by DML. However, CREATE INDEX CONCURRENTLY needs to wait for DML transactions to complete before it can finish (CONCURRENTLY can acquire the lock but cannot complete).
ALTER INDEX .. ATTACH PARTITION requests an AccessShareLock, which is the lightest lock and does not block the RowExclusiveLock requested by DML.
Queries request AccessShareLock, the lightest lock. Unless DDL requests AccessExclusiveLock (the heaviest lock), blocking does not occur.

Therefore, directly running CREATE INDEX on a partition blocks DML and is not acceptable. The correct way to create partition indexes:

-- Use ONLY to create an invalid index on the partition parent table. Fast, but blocks subsequent DML, affects business — watch for long-running transactions.
CREATE INDEX IDX_DATECREATED ON ONLY lzlpartition1(date_created);
-- Use CONCURRENTLY to create indexes on each partition child table. Slow, does not block subsequent DML, does not affect business, but watch for long-running DML transactions to prevent failure.
create index concurrently idx_datecreated_202302 on lzlpartition1_202302(date_created);
-- ATTACH all indexes. Fast, does not cause business blocking.
 ALTER INDEX idx_datecreated ATTACH PARTITION idx_datecreated_202302;

Adding Primary Keys and Unique Indexes to Partitioned Tables
#

A “primary key index” is functionally equivalent to “unique index + NOT NULL constraint” (but there can only be one primary key). Creating unique indexes on partitioned tables can follow the index creation best practices above: ONLY on parent, CONCURRENTLY on children, ATTACH. However, while primary keys on regular tables support the USING INDEX syntax, partitioned tables currently do not support this:

=> ALTER TABLE lzlpartition1 ADD CONSTRAINT pk_id_date_created PRIMARY KEY USING INDEX idx_uniq;
ERROR: 0A000: ALTER TABLE / ADD CONSTRAINT USING INDEX is not supported on partitioned tables
LOCATION: ATExecAddIndexConstraint, tablecmds.c:8032

In other words, you can create a NOT NULL unique index by pre-creating a NOT NULL constraint + ATTACH-ing indexes, but the final step of USING INDEX to add the primary key does not work.

Now let’s look at the blocking behavior of directly adding/dropping primary keys:

Directly dropping a primary key:

-- Session 1
=> begin;
BEGIN
Time: 0.318 ms
=> select * from lzlpartition1 where date_created='2023-01-01 22:00:00';
 id | name | date_created 
------+----------------------------------+---------------------
 7715 | beee680a86e1d12790489e9ab4a4351b | 2023-01-01 22:00:00
 -- Session 2: Drop primary key, waits
 => alter table lzlpartition1 drop constraint lzlpartition1_pkey;
 
 -- Session 3: Observe
 => select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+---------------------------+------------+---------------+-------+---------------------+---------
 relation | dbmgr | lzlpartition1_202301_pkey | [null] | [null] | 21659 | AccessShareLock | t
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 21659 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 95016 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 95016 | AccessExclusiveLock | f
 relation | dbmgr | lzlpartition1 | [null] | [null] | 21659 | AccessShareLock | t

Dropping a primary key requests an AccessExclusiveLock, blocking everything.

Directly adding a primary key:

-- Session 1 transaction ends; Session 2's drop primary key completes
-- Session 1 starts another read-only transaction
-- Session 2: Add a primary key on the partitioned table, waits
=> ALTER TABLE lzlpartition1 ADD PRIMARY KEY(id, date_created);

-- Session 3: Observe locks
=> select l.locktype,d.datname,r.relname,l.virtualxid,l.transactionid,l.pid,l.mode,l.granted from pg_locks l left join pg_database d on l.database=d.oid left join pg_class r on l.relation=r.oid 
where relname like '%lzlpartition1%';
 locktype | datname | relname | virtualxid | transactionid | pid | mode | granted 
----------+---------+----------------------+------------+---------------+-------+---------------------+---------
 relation | dbmgr | lzlpartition1_202301 | [null] | [null] | 21659 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 95016 | AccessShareLock | t
 relation | dbmgr | lzlpartition1 | [null] | [null] | 95016 | AccessExclusiveLock | f -- Session adding primary key
 relation | dbmgr | lzlpartition1 | [null] | [null] | 21659 | AccessShareLock | t

Adding a primary key requests an AccessExclusiveLock on the parent table, blocking everything. Adding an index on a partitioned table is very slow, and a primary key causes subsequent blocking. Currently, there is no low-impact way to add a primary key on a partitioned table. As a workaround, you can consider using the “ATTACH unique index + NOT NULL constraint” approach; or you may have to schedule a long maintenance window for the partitioned table business and wait for index creation to complete; or use a third-party sync tool to insert data into a partitioned table that already has the primary key.

Adding Partitions to HASH Partitioned Tables
#

If the new number of partitions is an integer multiple of the old number, we can know which old partition the data in the new partition came from. For example, expanding a 3-partition HASH partitioned table to 6 partitions, we can determine the data source:

Although understanding this simple data characteristic is helpful, in practice it may not be very useful, because new HASH partitions are always populated by brute-force INSERT. In terms of operations, going from “3→4” partitions is no different from “3→6”. Mature data sync tools are now widely available. For example, using DTS to insert the table into a new table and then performing a table switch — this results in very short downtime and should be the preferred approach in production. Below is primarily testing and observing the manual addition of integer-multiple partitions to a HASH partitioned table:

Partition info:

SELECT tableoid::regclass,count(*) FROM orders group by tableoid::regclass;
 tableoid | count 
-----------+-------
 orders_p1 | 3377
 orders_p3 | 3354
 orders_p2 | 3369

2. DETACH partitions:
 Adding 3 more partitions to a 3-partition HASH native partitioned table:

ALTER TABLE orders DETACH PARTITION orders_p1;
ALTER TABLE orders DETACH PARTITION orders_p2;
ALTER TABLE orders DETACH PARTITION orders_p3;

RENAME partitions:

ALTER TABLE orders_p1 RENAME TO bak_orders_p1;
ALTER TABLE orders_p2 RENAME TO bak_orders_p2;
ALTER TABLE orders_p3 RENAME TO bak_orders_p3;

Create 6 HASH partitions on the old table:

CREATE TABLE orders_p1 PARTITION OF orders FOR VALUES WITH (MODULUS 6, REMAINDER 0);
CREATE TABLE orders_p2 PARTITION OF orders FOR VALUES WITH (MODULUS 6, REMAINDER 1);
CREATE TABLE orders_p3 PARTITION OF orders FOR VALUES WITH (MODULUS 6, REMAINDER 2);
CREATE TABLE orders_p4 PARTITION OF orders FOR VALUES WITH (MODULUS 6, REMAINDER 3);
CREATE TABLE orders_p5 PARTITION OF orders FOR VALUES WITH (MODULUS 6, REMAINDER 4);
CREATE TABLE orders_p6 PARTITION OF orders FOR VALUES WITH (MODULUS 6, REMAINDER 5);

View partition info: Note the function used in the partition constraint:

\d+ orders_p1
 Table "public.orders_p1"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
----------+-----------------------+-----------+----------+---------+----------+--------------+-------------
 order_id | integer | | | | plain | | 
 name | character varying(10) | | | | extended | | 
Partition of: orders FOR VALUES WITH (modulus 6, remainder 0)
Partition constraint: satisfies_hash_partition('412053'::oid, 6, 0, order_id)
Access method: heap

Calculate which new partition old partition data should be inserted into. For example, the old modulus 3, remainder 0 partition’s data needs to be split into the modulus 6, remainder 0 and remainder 3 partitions:

 select count(*) from bak_orders_p1 where satisfies_hash_partition('412053'::oid, 6, 0, order_id)=true;
 count 
-------
 1776
 select count(*) from bak_orders_p1 where satisfies_hash_partition('412053'::oid, 6, 3, order_id)=true;
 count 
-------
 1601
 select count(*) from bak_orders_p1;
 count 
-------
 3377

Insert data directly into partition child tables: You can insert data directly into the corresponding partition child tables rather than through the partition parent table:

INSERT INTO orders_p1 SELECT * FROM bak_orders_p1 where satisfies_hash_partition('412053'::oid, 6, 0, order_id)=true;
INSERT INTO orders_p2 SELECT * FROM bak_orders_p2 where satisfies_hash_partition('412053'::oid, 6, 1, order_id)=true;
INSERT INTO orders_p3 SELECT * FROM bak_orders_p3 where satisfies_hash_partition('412053'::oid, 6, 2, order_id)=true;
INSERT INTO orders_p4 SELECT * FROM bak_orders_p1 where satisfies_hash_partition('412053'::oid, 6, 3, order_id)=true;
INSERT INTO orders_p5 SELECT * FROM bak_orders_p2 where satisfies_hash_partition('412053'::oid, 6, 4, order_id)=true;
INSERT INTO orders_p6 SELECT * FROM bak_orders_p3 where satisfies_hash_partition('412053'::oid, 6, 5, order_id)=true;

Verify data from 3 old partitions has been inserted into 6 new partitions:

SELECT tableoid::regclass,count(*) FROM orders group by tableoid::regclass;
tableoid | count 
-----------+-------
orders_p3 | 1665
orders_p5 | 1678
orders_p1 | 1776
orders_p6 | 1689
orders_p4 | 1601
orders_p2 | 1691

Changing Column Length on Partitioned Tables Rebuilds Indexes
#

Modifying a column involves three considerations: table rewrite, index rebuild, and statistics loss.

Changing column type or reducing column length rewrites the table.
Increasing column length only causes statistics loss; an exception is reducing the length (or changing int4 to int8), which rewrites the table.
Increasing column length does not rebuild indexes, with one exception: increasing column length on a partitioned table rebuilds indexes (if the column has an index).

For column modifications, refer to the PostgreSQL apprentice.

Here we mainly test the scenario of increasing column length on a partitioned table. If an index exists, it may cause transaction blocking on the partitioned table. Regular table, increasing the length of an indexed column:

-- Create regular table and index
=> create table t111(id int,name varchar(50));
CREATE TABLE
=> insert into t111 values(1001,'abc');
INSERT 0 1
=> create index idx111 on t111(name);
CREATE INDEX

-- Index file relfilenode is 417728
 select pg_relation_filepath('idx111');
 pg_relation_filepath 
----------------------
 base/16398/417728
(1 row)

-- Increase column length
=> alter table t111 alter column name type varchar(60);
ALTER TABLE
-- Index file relfilenode is still 417728, unchanged. Regular table index was NOT rebuilt.
=> select pg_relation_filepath('idx111');
 pg_relation_filepath 
----------------------
 base/16398/417728

Partitioned table, increasing the length of an indexed column:

-- Create an index on the partitioned table
=> create index idx_name on lzlpartition1(name);
CREATE INDEX
-- Check the index on one partition
=> \d+ lzlpartition1_202301
 Table "dbmgr.lzlpartition1_202301"
 Column | Type | Collation | Nullable | Default | Storage | Stats target | Description 
--------------+-----------------------------+-----------+----------+---------+----------+--------------+-------------
 id | integer | | | | plain | | 
 name | character varying(50) | | | | extended | | 
 date_created | timestamp without time zone | | not null | now() | plain | | 
Partition of: lzlpartition1 FOR VALUES FROM ('2023-01-01 00:00:00') TO ('2023-02-01 00:00:00')
Partition constraint: ((date_created IS NOT NULL) AND (date_created >= '2023-01-01 00:00:00'::timestamp without time zone) AND (date_created < '2023-02-01 00:00:00'::timestamp without time zone))
Indexes:
 "lzlpartition1_202301_name_idx" btree (name)
Access method: heap

=> select pg_relation_filepath('lzlpartition1_202301_name_idx') idx,pg_relation_filepath('lzlpartition1_202301') tbl;
 idx | tbl 
-------------------+-------------------
 base/16398/417810 | base/16398/417800
(1 row)

-- Increase the indexed column length — partitioned table index is rebuilt
=> alter table lzlpartition1 alter column name type varchar(60);
ALTER TABLE
=> select pg_relation_filepath('lzlpartition1_202301_name_idx') idx,pg_relation_filepath('lzlpartition1_202301') tbl;
 idx | tbl 
-------------------+-------------------
 base/16398/417814 | base/16398/417800

-- Reduce the indexed column length — partitioned table is rewritten
=> alter table lzlpartition1 alter column name type varchar(40);
ALTER TABLE
Time: 609.585 ms
=> select pg_relation_filepath('lzlpartition1_202301_name_idx') idx,pg_relation_filepath('lzlpartition1_202301') tbl;
 idx | tbl 
-------------------+-------------------
 base/16398/417828 | base/16398/417825

-- Keep the indexed column length the same — partitioned table index is still rebuilt
=> alter table lzlpartition1 alter column name type varchar(40);
ALTER TABLE
=> select pg_relation_filepath('lzlpartition1_202301_name_idx') idx,pg_relation_filepath('lzlpartition1_202301') tbl;
 idx | tbl 
-------------------+-------------------
 base/16398/417834 | base/16398/417825

For regular tables, increasing column length only requires attention to statistics loss (except int to bigint). However, for partitioned tables, when increasing column length, if the column has an index, not only are statistics lost but the index is also rebuilt. Since ALTER COLUMN is an 8-level lock, the index rebuild period causes extended blocking. Recommendation: first drop the index, modify the column, then rebuild the index using the “parent table ONLY + child tables CIC + ATTACH” approach.

Partition Table Maintenance Summary
#

PARTITION OF / DROP TABLE / DETACH require ACCESS EXCLUSIVE locks. ATTACH / DETACH CONCURRENTLY are recommended — they do not cause blocking. For DETACH CONCURRENTLY, watch for existing long-running transactions.
Before ATTACH-ing a partition, you can pre-create a constraint on the partition. This eliminates the time spent scanning partition data during ATTACH.
Currently, CIC (CREATE INDEX CONCURRENTLY) is not supported on partitioned tables. You can create partition indexes using the “ONLY on parent + CONCURRENTLY on children + ATTACH index” approach to reduce business blocking time.
Partitioned tables do not support the USING INDEX method for creating primary keys.
Pay attention to the exceptional case of modifying column length on partitioned tables.

Partition Table Optimization
#

Partition Pruning
#

Partition Pruning can improve performance for declarative partitioning and is a very important feature for partitioned table optimization. Without partition pruning, queries would scan all partitions. With partition pruning, the optimizer can filter out partitions that don’t need to be accessed through the WHERE condition. Partition pruning relies on the PARTITION CONSTRAINT (visible with \d+), which means queries must include partition key conditions for pruning to occur. This constraint differs from regular CHECK constraints — it is automatically created when the partition is created. Partition pruning is controlled by the enable_partition_pruning parameter, which defaults to on.

-- Without partition pruning, all partitions are accessed
=> set enable_partition_pruning=off;
SET

=> explain select count(*) from lzlpartition1 where date_created='2023-01-01';
 QUERY PLAN 
--------------------------------------------------------------------------------------------------
 Aggregate (cost=1872.08..1872.09 rows=1 width=8)
 -> Append (cost=0.00..1872.07 rows=4 width=0)
 -> Seq Scan on lzlpartition1_202301 lzlpartition1_1 (cost=0.00..992.30 rows=1 width=0)
 Filter: (date_created = '2023-01-01 00:00:00'::timestamp without time zone)
 -> Seq Scan on lzlpartition1_202302 lzlpartition1_2 (cost=0.00..864.12 rows=1 width=0)
 Filter: (date_created = '2023-01-01 00:00:00'::timestamp without time zone)
 -> Seq Scan on lzlpartition1_202304 lzlpartition1_3 (cost=0.00..15.62 rows=2 width=0)
 Filter: (date_created = '2023-01-01 00:00:00'::timestamp without time zone)

-- With partition pruning enabled, partitions that don't need to be accessed are excluded
=> set enable_partition_pruning=on;
SET

=> explain select count(*) from lzlpartition1 where date_created='2023-01-01';
 QUERY PLAN 
------------------------------------------------------------------------------------------
 Aggregate (cost=992.30..992.31 rows=1 width=8)
 -> Seq Scan on lzlpartition1_202301 lzlpartition1 (cost=0.00..992.30 rows=1 width=0)
 Filter: (date_created = '2023-01-01 00:00:00'::timestamp without time zone)
(3 rows)

(The official documentation says pruning happens during execution plan generation, and EXPLAIN would show “Subplans Removed.” In testing, this isn’t always the case, as in the EXPLAIN example above.) Partition pruning can occur at two stages: during execution plan generation, and during actual execution. Why does this happen? Because sometimes only at execution time can we know which partitions can be pruned. There are two scenarios:

Parameterized Nested Loop Joins: The parameter from the outer side of the join can be used to determine the minimum set of inner side partitions to scan.
Initplans: Once an initplan has been executed we can then determine which partitions match the value from the initplan.

Simulating runtime pruning: When fetching data from another table, the optimizer certainly doesn’t know what the data is, so it cannot use that as a basis for partition pruning during plan generation:

-- Create another table
=> create table x(date_created timestamp);
CREATE TABLE

=> insert into x values('2023-01-01 09:00:00');
INSERT 0 1

-- Generate execution plan only, don't execute — no pruning occurred
=> explain select count(*) from lzlpartition1 where date_created=(select date_created from x);
 QUERY PLAN 
--------------------------------------------------------------------------------------------------
 Aggregate (cost=1904.68..1904.69 rows=1 width=8)
 InitPlan 1 (returns $0)
 -> Seq Scan on x (cost=0.00..32.60 rows=2260 width=8)
 -> Append (cost=0.00..1872.07 rows=4 width=0)
 -> Seq Scan on lzlpartition1_202301 lzlpartition1_1 (cost=0.00..992.30 rows=1 width=0)
 Filter: (date_created = $0)
 -> Seq Scan on lzlpartition1_202302 lzlpartition1_2 (cost=0.00..864.12 rows=1 width=0)
 Filter: (date_created = $0)
 -> Seq Scan on lzlpartition1_202304 lzlpartition1_3 (cost=0.00..15.62 rows=2 width=0)
 Filter: (date_created = $0)
(10 rows)

-- Execute the SQL — pruning occurred. Notice the "never executed" keyword.
=> explain analyze select count(*) from lzlpartition1 where date_created=(select date_created from x);
 QUERY PLAN 
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate (cost=1904.68..1904.69 rows=1 width=8) (actual time=5.680..5.682 rows=1 loops=1)
 InitPlan 1 (returns $0)
 -> Seq Scan on x (cost=0.00..32.60 rows=2260 width=8) (actual time=0.013..0.014 rows=1 loops=1)
 -> Append (cost=0.00..1872.07 rows=4 width=0) (actual time=0.029..5.676 rows=2 loops=1)
 -> Seq Scan on lzlpartition1_202301 lzlpartition1_1 (cost=0.00..992.30 rows=1 width=0) (actual time=0.008..5.652 rows=2 loops=1)
 Filter: (date_created = $0)
 Rows Removed by Filter: 45382
 -> Seq Scan on lzlpartition1_202302 lzlpartition1_2 (cost=0.00..864.12 rows=1 width=0) (never executed)
 Filter: (date_created = $0)
 -> Seq Scan on lzlpartition1_202304 lzlpartition1_3 (cost=0.00..15.62 rows=2 width=0) (never executed)
 Filter: (date_created = $0)
 Planning Time: 0.157 ms
 Execution Time: 5.732 ms
(13 rows)

Partition Wise Join
#

Partition wise join can reduce the cost of partition joins. Suppose there are two partitioned tables t1 and t2, both with 3 partitions (p1, p2, p3) with identical partition definitions. t1 has 10 rows per partition, t2 has 20 rows per partition:

	t1	t2
p1	10 rows	20 rows
p2	10 rows	20 rows
p3	10 rows	20 rows
When t1 and t2 join,

Normally, all data from both partitioned tables needs to be extracted for joining. The number of row comparison operations would be: (10+10+10)*(20+20+20)=180
With partition wise join, since the structures are similar, only corresponding partitions need to be joined, e.g.: t1.p1<=>t2.p1, t1.p2<=>t2.p2, t1.p3<=>t2.p3, The number of row comparison operations becomes: (10*20)*3=90

When there are many partitions, the cost savings of partition wise join are significant. Parameter enable_partitionwise_join: whether to enable partition wise join, default is off.

The prerequisites for partition wise join are very strict:

The join condition must include the partition key.
The partition keys must be of the same data type.
Partitions must correspond one-to-one.

While these conditions seem strict, it’s relatively rare for tables with different purposes to produce partition wise join scenarios. A common case would be both tables using RANGE time partitioning. Another scenario: a partitioned table self-joining also meets partition wise join prerequisites:

-- Without partition wise join enabled
=> explain select p1.*,p2.name from lzlpartition1 p1,lzlpartition1 p2 where p1.date_created=p2.date_created and p2.name='256ac66bb53d31bc6124294238d6410c';
 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------
 Hash Join (cost=546.64..9256.34 rows=182252 width=288)
 Hash Cond: (p1.date_created = p2.date_created)
 -> Append (cost=0.00..2085.46 rows=85364 width=150)
 -> Seq Scan on lzlpartition1_202301 p1_1 (cost=0.00..878.84 rows=45384 width=150)
 -> Seq Scan on lzlpartition1_202302 p1_2 (cost=0.00..765.30 rows=39530 width=150)
 -> Seq Scan on lzlpartition1_202304 p1_3 (cost=0.00..14.50 rows=450 width=150)
 -> Hash (cost=541.30..541.30 rows=427 width=146)
 -> Append (cost=7.17..541.30 rows=427 width=146)
 -> Bitmap Heap Scan on lzlpartition1_202301 p2_1 (cost=7.17..284.30 rows=227 width=146)
 Recheck Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Bitmap Index Scan on lzlpartition1_202301_name_idx (cost=0.00..7.12 rows=227 width=0)
 Index Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Bitmap Heap Scan on lzlpartition1_202302 p2_2 (cost=6.95..248.52 rows=198 width=146)
 Recheck Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Bitmap Index Scan on lzlpartition1_202302_name_idx (cost=0.00..6.90 rows=198 width=0)
 Index Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Bitmap Heap Scan on lzlpartition1_202304 p2_3 (cost=2.66..6.35 rows=2 width=146)
 Recheck Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Bitmap Index Scan on lzlpartition1_202304_name_idx (cost=0.00..2.66 rows=2 width=0)
 Index Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
(20 rows)

-- With partition wise join enabled
=> set enable_partitionwise_join =on;
SET

M=> explain select p1.*,p2.name from lzlpartition1 p1,lzlpartition1 p2 where p1.date_created=p2.date_created and p2.name='256ac66bb53d31bc6124294238d6410c';
 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------
 Append (cost=287.14..2529.83 rows=438 width=288)
 -> Hash Join (cost=287.14..1338.49 rows=232 width=288)
 Hash Cond: (p1_1.date_created = p2_1.date_created)
 -> Seq Scan on lzlpartition1_202301 p1_1 (cost=0.00..878.84 rows=45384 width=150)
 -> Hash (cost=284.30..284.30 rows=227 width=146)
 -> Bitmap Heap Scan on lzlpartition1_202301 p2_1 (cost=7.17..284.30 rows=227 width=146)
 Recheck Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Bitmap Index Scan on lzlpartition1_202301_name_idx (cost=0.00..7.12 rows=227 width=0)
 Index Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Hash Join (cost=250.99..1166.55 rows=202 width=288)
 Hash Cond: (p1_2.date_created = p2_2.date_created)
 -> Seq Scan on lzlpartition1_202302 p1_2 (cost=0.00..765.30 rows=39530 width=150)
 -> Hash (cost=248.52..248.52 rows=198 width=146)
 -> Bitmap Heap Scan on lzlpartition1_202302 p2_2 (cost=6.95..248.52 rows=198 width=146)
 Recheck Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Bitmap Index Scan on lzlpartition1_202302_name_idx (cost=0.00..6.90 rows=198 width=0)
 Index Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Hash Join (cost=6.37..22.60 rows=4 width=288)
 Hash Cond: (p1_3.date_created = p2_3.date_created)
 -> Seq Scan on lzlpartition1_202304 p1_3 (cost=0.00..14.50 rows=450 width=150)
 -> Hash (cost=6.35..6.35 rows=2 width=146)
 -> Bitmap Heap Scan on lzlpartition1_202304 p2_3 (cost=2.66..6.35 rows=2 width=146)
 Recheck Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
 -> Bitmap Index Scan on lzlpartition1_202304_name_idx (cost=0.00..2.66 rows=2 width=0)
 Index Cond: ((name)::text = '256ac66bb53d31bc6124294238d6410c'::text)
(25 rows)

Without partition wise join enabled, the optimizer first accesses all partition data from p2 (matching the filter) and combines them (Append), then Hash Joins with all partition data from p1 through the partition key. With partition wise join enabled, the optimizer joins corresponding partitions from p1 and p2 (actually the same table accessed twice): p1_1<=>p2_1 Hash Join p1_2<=>p2_2 Hash Join p1_3<=>p2_3 Hash Join Then combines the data together (Append). If there are enough data partitions, combined with partition pruning, partition wise join can have very good optimization effects.

Partition Wise Grouping/Aggregation
#

When performing aggregation on partitioned data, partitions can each compute independently — there is no need to scan all partition data for aggregation. Each partition computes its own aggregation, then the results are collected and returned. Without partition wise grouping, it’s essentially “scan all partitions first, then aggregate.” With partition wise grouping, it’s “aggregate per partition first, then combine results.”

Advantages of partition wise grouping:

When partitions are on foreign servers, the aggregation operator can be pushed down to the foreign server.
When aggregating into hash tables, each partition rather than the entire table uses the memory hash table space, reducing memory usage.
Aggregation algorithms pushed down to individual partitions can better utilize features like indexes and parallelism.
Fewer data comparisons. Although data scanning is the same, there are fewer data comparisons — for example, data from the last partition does not need to be compared with data from the first partition.

Parameter enable_partitionwise_aggregate: whether to enable partition wise grouping/aggregation, default is off.

Partition wise aggregate example:

=> vacuum (analyze) lzlpartition1;

-- Without wise agg
=> set enable_partitionwise_aggregate =off;
SET
=> explain select date_created,min(id),count(*) from lzlpartition1 group by date_created order by 1,2,3;
 QUERY PLAN 
-------------------------------------------------------------------------------------------------------------
 Sort (cost=10354.94..10562.89 rows=83180 width=20)
 Sort Key: lzlpartition1.date_created, (min(lzlpartition1.id)), (count(*))
 -> HashAggregate (cost=2725.69..3557.49 rows=83180 width=20)
 Group Key: lzlpartition1.date_created
 -> Append (cost=0.00..2085.46 rows=85364 width=12)
 -> Seq Scan on lzlpartition1_202301 lzlpartition1_1 (cost=0.00..878.84 rows=45384 width=12)
 -> Seq Scan on lzlpartition1_202302 lzlpartition1_2 (cost=0.00..765.30 rows=39530 width=12)
 -> Seq Scan on lzlpartition1_202304 lzlpartition1_3 (cost=0.00..14.50 rows=450 width=12)


-- With wise agg enabled
=> set enable_partitionwise_aggregate =on;
SET
=> explain select date_created,min(id),count(*) from lzlpartition1 group by date_created order by 1,2,3;
 QUERY PLAN 
-------------------------------------------------------------------------------------------------------------
 Sort (cost=10356.08..10564.32 rows=83296 width=20)
 Sort Key: lzlpartition1.date_created, (min(lzlpartition1.id)), (count(*))
 -> Append (cost=1219.22..3548.31 rows=83296 width=20)
 -> HashAggregate (cost=1219.22..1663.09 rows=44387 width=20)
 Group Key: lzlpartition1.date_created
 -> Seq Scan on lzlpartition1_202301 lzlpartition1 (cost=0.00..878.84 rows=45384 width=12)
 -> HashAggregate (cost=1061.77..1448.86 rows=38709 width=20)
 Group Key: lzlpartition1_1.date_created
 -> Seq Scan on lzlpartition1_202302 lzlpartition1_1 (cost=0.00..765.30 rows=39530 width=12)
 -> HashAggregate (cost=17.88..19.88 rows=200 width=20)
 Group Key: lzlpartition1_2.date_created
 -> Seq Scan on lzlpartition1_202304 lzlpartition1_2 (cost=0.00..14.50 rows=450 width=12)
(12 rows)

Without partition wise aggregate: first scan all data then combine (Append), then aggregate (HashAggregate). With partition wise aggregate: first aggregate on each partition (HashAggregate), then combine results (Append).

Partial Aggregation The aggregation algorithm can be pushed down to partitions for computation. At this point, the aggregated results fall into two categories: non-duplicate aggregation data (GROUP BY includes the partition key), and duplicate aggregation data (GROUP BY does not include the partition key). When aggregation data is non-duplicate, simply appending the per-partition computed aggregation data is sufficient (as in the example above). When per-partition aggregation data has duplicates, an additional aggregation step (Finalize Aggregate) is needed. Aggregation that does not include the partition key is partial aggregation.

Partial aggregation example:

-- When GROUP BY is not the partition key
=> show enable_partitionwise_aggregate;
 enable_partitionwise_aggregate 
--------------------------------
 on
=> explain select id,count(*) from lzlpartition1 group by id ;
 QUERY PLAN 
------------------------------------------------------------------------------------------------------------
 Finalize HashAggregate (cost=2474.80..2573.80 rows=9900 width=12)
 Group Key: lzlpartition1.id
 -> Append (cost=1105.76..2377.47 rows=19467 width=12)
 -> Partial HashAggregate (cost=1105.76..1202.28 rows=9652 width=12)
 Group Key: lzlpartition1.id
 -> Seq Scan on lzlpartition1_202301 lzlpartition1 (cost=0.00..878.84 rows=45384 width=4)
 -> Partial HashAggregate (cost=962.95..1059.10 rows=9615 width=12)
 Group Key: lzlpartition1_1.id
 -> Seq Scan on lzlpartition1_202302 lzlpartition1_1 (cost=0.00..765.30 rows=39530 width=4)
 -> Partial HashAggregate (cost=16.75..18.75 rows=200 width=12)
 Group Key: lzlpartition1_2.id
 -> Seq Scan on lzlpartition1_202304 lzlpartition1_2 (cost=0.00..14.50 rows=450 width=4)

When GROUP BY does not include the partition key, aggregation can still be performed, but a subsequent Finalize HashAggregate is required.

Even without GROUP BY, Partial Aggregate can still occur:

=> show enable_partitionwise_aggregate;
 enable_partitionwise_aggregate 
--------------------------------
 on
=> explain select count(*) from lzlpartition1;
 QUERY PLAN 
------------------------------------------------------------------------------------------------------------
 Finalize Aggregate (cost=1872.10..1872.11 rows=1 width=8)
 -> Append (cost=992.30..1872.10 rows=3 width=8)
 -> Partial Aggregate (cost=992.30..992.31 rows=1 width=8)
 -> Seq Scan on lzlpartition1_202301 lzlpartition1 (cost=0.00..878.84 rows=45384 width=0)
 -> Partial Aggregate (cost=864.12..864.13 rows=1 width=8)
 -> Seq Scan on lzlpartition1_202302 lzlpartition1_1 (cost=0.00..765.30 rows=39530 width=0)
 -> Partial Aggregate (cost=15.62..15.63 rows=1 width=8)
 -> Seq Scan on lzlpartition1_202304 lzlpartition1_2 (cost=0.00..14.50 rows=450 width=0)

=> explain select max(date_created) from lzlpartition1;
 QUERY PLAN 
------------------------------------------------------------------------------------------------------------
 Finalize Aggregate (cost=1872.10..1872.11 rows=1 width=8)
 -> Append (cost=992.30..1872.10 rows=3 width=8)
 -> Partial Aggregate (cost=992.30..992.31 rows=1 width=8)
 -> Seq Scan on lzlpartition1_202301 lzlpartition1 (cost=0.00..878.84 rows=45384 width=8)
 -> Partial Aggregate (cost=864.12..864.13 rows=1 width=8)
 -> Seq Scan on lzlpartition1_202302 lzlpartition1_1 (cost=0.00..765.30 rows=39530 width=8)
 -> Partial Aggregate (cost=15.62..15.63 rows=1 width=8)
 -> Seq Scan on lzlpartition1_202304 lzlpartition1_2 (cost=0.00..14.50 rows=450 width=8)
 

The precondition for triggering Partial Aggregate is not GROUP BY. We should think from the purpose of Partial Aggregate — it aims to push aggregation down to partitions. Aggregation without GROUP BY can also be done this way, as shown in the two examples above: they both compute aggregation on each partition first (Partial Aggregate), then combine and aggregate once more (Finalize Aggregate). Without the parameter enabled, these aggregations would occur after scanning all partitions.

History of Partitioned Tables
#

Declarative partitioning has gone through many version enhancements and is now very mature. Here’s a summary of declarative partitioning feature enhancements across PostgreSQL versions:

Pre-PG9.6

Only inheritance tables could implement partitioning functionality.

PG10

Declarative partitioning supported.
RANGE and LIST partitioning supported.
ATTACH/DETACH table partitions supported.
Partition pruning supported.

PG11

Added HASH partition support.
Support for creating primary keys, foreign keys, indexes, and triggers.
Support for updating partition key; automatic creation of indexes on partitions.
Support for DEFAULT partition.
Support for ATTACH index.
Support for FOR EACH ROW triggers, automatically created on existing and future child partitions.
New enable_partition_pruning parameter; pruning enhancements.
Support for partition wise join.
Support for partition wise aggregation.

PG12

Enhanced query, insert, pruning, and COPY performance.
Support for foreign key constraints referencing partitioned tables.
Support for non-blocking partition ATTACH: ALTER TABLE ATTACH PARTITION.

PG13

Enhanced pruning.
Enhanced partition wise join.
Support for BEFORE triggers.
Support for publishing partitioned tables; support for subscribing and writing to partitioned tables.

PG14

Enhanced UPDATE and DELETE performance.
Support for non-blocking partition DETACH: ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY.
Support for REINDEX on partitioned table indexes.

PG15

Enhanced execution plan generation, reducing generation time with many partitions.
Enhanced sorting.
Support for CLUSTER on partitioned tables.

PG16

Enhanced GENERATED column restrictions: if the parent table has a generated column, child partitions must also include it.
Enhanced lookup for RANGE and LIST partitions.

References
#

《PostgreSQL修炼之道》

https://mp.weixin.qq.com/s/NW8XOZNq0YlDZvx24H737Q https://www.postgresql.org/docs/current/ddl-partitioning.html https://www.postgresql.org/docs/current/ddl-inherit.html https://www.postgresql.org/docs/13/sql-altertable.html https://github.com/postgrespro/pg_pathman https://developer.aliyun.com/article/62314 https://hevodata.com/learn/postgresql-partitions https://www.postgresql.fastware.com/postgresql-insider-prt-ove https://www.buckenhofer.com/2021/01/postgresql-partitioning-guide/ https://www.depesz.com/2018/05/01/waiting-for-postgresql-11-support-partition-pruning-at-execution-time/ https://blog.csdn.net/horses/article/details/86164273

http://www.pgsql.tech/article_0_10000102

https://brandur.org/fragments/postgres-partitioning-2022

Vector Database Core Concepts

Mon, 12 Aug 2024 00:00:00 +0000

Vector Database Core Concepts
#

A Bit of History
#

The development history of LLM models, from Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond¹:

Many people only gradually learned about large models after the ChatGPT explosion, but in the years before that tipping point, the development of large models had already begun a war of the gods. Several institutions published many revolutionary papers — on the corporate side: Google, DeepMind, OpenAI, Meta, Microsoft; on the academic side: Stanford, Berkeley, CMU, Princeton, MIT².

There are three main camps:

Google & DeepMind camp — Gemini, Bard
Microsoft & OpenAI camp — ChatGPT, Bing
Meta open-source community camp — Llama

Timeline of recent large model product releases, from A Survey of Large Language Models³:

Generative AI Basics
#

AIGC (Artificial Intelligence Generated Content): The precise concept of AIGC is a mode of production that uses AI to automatically generate content. In a broader sense, AIGC can be approximated as AI technology trained to possess human-like generative and creative capabilities — i.e., Generative AI. It can autonomously generate and create new text, images, music, videos, 3D interactive content, and various other forms of content and data based on data and generative algorithm models, and even includes enabling new scientific discoveries and creating new meanings.

LLM (Large Language Model): LLMs are large language models capable of capturing and processing complex language patterns and semantics — that is, they can understand and generate human language. GPT-3, ChatGPT, BERT, T5, ERNIE Bot, and others are typical large language models.

NLP (Natural Language Processing): Natural Language Processing (NLP) studies how to enable computers to read and understand human language — i.e., converting natural human language into instructions that computers can process. LLM is an important component of NLP.

AIGC has achieved remarkable growth, largely due to Natural Language Processing (NLP), and the biggest driver behind NLP’s progress is the Large Language Model (LLM). This year (2024), AIGC is also developing rapidly in areas such as video and audio.⁴

prompt: Instructions or directives — natural language provided to AI describing a task, used to guide a language model (such as GPT-3 or GPT-4) to generate the corresponding output⁵. (Everyone basically knows what this is already, no need to elaborate.)

embedding:

Embedding is a method of representing objects (such as text, images, and audio) as points in a continuous vector space, where the positions of these points in space carry semantic meaning for machine learning algorithms.

Based on GloVe word-vector relevance for English words, there is an interactive 2D embedding explorer. This shows natural language embedded as 2D vectors:

RAG
#

RAG (Retrieval-Augmented Generation) is a two-stage process consisting of document retrieval and large language model (LLM) answer generation. The initial stage leverages dense embeddings to retrieve documents. Depending on the specific use case, this retrieval can be based on various database formats, such as vector databases, summary indexes, tree indexes, and key indexes⁵.

The original RAG paper⁶ was published on May 22, 2020, by researchers from Facebook (Meta), University College London, and New York University, proposing a general fine-tuning approach for RAG. RAG includes the following characteristics²:

RAG models combine pre-trained memory to assist language generation
RAG models generate language that is more specific, diverse, and factual

On March 23, 2023, OpenAI released the chatgpt-retrieval-plugin repository, recommending the use of vector databases in RAG. From that point on, vector databases gained widespread attention in the application domain, riding the wave of large model popularity.

What Can Vector Databases Bring to AI?
#

Vector databases can provide large models with data retrieval and long-term data storage capabilities within RAG⁷.

Why use RAG? No words carry more weight than those of the master, OpenAI. The following passage is from the retrieval plugin usage guide released by OpenAI in March 2023⁸, translated by ChatGPT:

The open-source retrieval plugin enables ChatGPT to access personal or organizational information sources (with permission). Users can ask questions or express needs in natural language and obtain the most relevant document snippets from their data sources (such as files, notes, emails, or public documents).

As an open-source and self-hosted solution, developers can deploy their own version of the plugin and register it with ChatGPT. The plugin leverages OpenAI’s embeddings and allows developers to choose a vector database (such as Milvus, Pinecone, Qdrant, Redis, Weaviate, or Zilliz) to index and search documents. Information sources can be synchronized with the database using webhooks.

In short, OpenAI recommends everyone use vector databases.

Has the vector database cooled off? Not only has it not cooled off — RAG has developed to the point of being everywhere today — Has RAG Technology Really Become “Commonplace”?. And vector databases, with their high retrieval efficiency, data storage reliability, and other characteristics, are an important part of RAG.

Common Vector Databases
#

Since OpenAI released the RAG repo, many vector databases have emerged (though some existed before). Several companies have also secured considerable funding⁹:

Company	Headquartered in	Funding
Weaviate	🇳🇱 Amsterdam	$68M Series B
Qdrant	🇩🇪 Berlin	$11M Seed
Pinecone	🇺🇸 San Francisco	$138M Series B
Milvus/Zilliz	🇨🇳 / 🇺🇸 Redwood City	$113M Series B
Chroma	🇺🇸 San Francisco	$20M Seed
LanceDB	🇺🇸 San Francisco	Venture
Vespa	🇳🇴 / 🇺🇸 Indianapolis	Yahoo!
Vald	🇯🇵 Tokyo	Yahoo! Japan

Vector database release timeline:

Vector database performance comparison¹⁰:

Dedicated vector databases generally perform better than traditional databases with vector plugins, for roughly two reasons:

Dedicated vector databases are built with vector-specific underlying storage, and their performance is generally better than untargeted traditional databases.
Dedicated vector databases are generally newer (mostly implemented in Go or Rust), making code-level optimization easier.

However, this does not mean plugin-based vector databases have no place:

Traditional databases natively support more features, not just similarity computation.
ACID — traditional database storage is safer.
It’s easier to manipulate data within a single database.

Vector database feature comparison:

The description of pgvector above is no longer entirely accurate — pgvector now supports HNSW, and the pgvector ecosystem project pgvectorscale also supports DiskANN.

Mathematical Concepts
#

Mathematics says: “I stand on the mountaintop watching you all play.”

Scalar
#

A scalar is a specific number. Scalars have no direction and are generally defined in contrast to vectors.

Vector
#

In Euclidean space, a vector has both magnitude and direction. For example, vector a from point A to point B (contains information about both points and direction)¹¹:

Unit Vector
#

A vector with magnitude one is a unit vector. The unit vector equals the vector divided by its Euclidean length¹²: $$ \vec a = \frac{\mathbf a}{||\mathbf a||} $$

In mathematics, the Unit Vector is called a “normalized vector” in pgvector and OpenAI embeddings. (Note: do not confuse this with the mathematical concept of the normal vector — a normal vector is a different concept entirely.)

Why use unit vectors?

OpenAI embeddings’ explanation for using unit vectors¹³:

OpenAI embeddings are normalized to length 1, which means that:

Cosine similarity can be computed slightly faster using just a dot product

Cosine similarity and Euclidean distance will result in the identical rankings

Sparse Vector
#

Sparse vectors are called “sparse” because the information in the vector is sparsely distributed. Typically, we need to find a few ones (relevant information) among thousands of zeros. Therefore, these vectors can contain many dimensions, usually in the tens of thousands.

Comparison of sparse and dense vectors: Sparse vectors contain sparsely distributed bits of information, while dense vectors carry more information in every dimension — information-dense.¹⁴

Euclidean Space
#

Simply called Euclidean space, it is the most fundamental space in mathematics. In modern mathematics, a space of positive integer n dimensions is called Euclidean space.

There are other space definitions, such as inner product space and Hilbert space. They differ in mathematical definitions, but in database/real-world contexts, the distinctions are not so fine-grained. The key takeaway is that inner product space, Euclidean space, and Hilbert space can all contain elements such as points, vectors, and inner products — we can simply call them “multi-dimensional spaces”. For their differences, see A Casual Discussion of Various Spaces in Mathematics¹⁵.

Euclidean Distance
#

Simply called Euclidean distance, this is what we generally think of as the distance between points — i.e., the length of a line segment¹⁶.

In 2D space, the Euclidean distance between points q and p is: $$ d(\mathbf p,\mathbf q)=\sqrt{(p_1-q_1)^2+(p_2-q_2)^2} $$

In n-dimensional space, the Euclidean distance between points q and p is: $$ d(\mathbf p,\mathbf q)=\sqrt{(p_1-q_1)^2+(p_2-q_2)^2+\cdots+(p_n-q_n)^2} $$

Manhattan Distance (or Taxicab Distance)
#

$$ d(\mathbf p,\mathbf q)= \sum_{i=1}^n | p_i-q_i| $$

Manhattan distance is the sum of the absolute differences of two points across each dimension¹⁷.

In the figure above, the green line is Euclidean distance; the red, yellow, and blue lines are Manhattan distances.

Minkowski Distance
#

$$ d(\mathbf a,\mathbf b)= \left( \sum_{i=1}^n | a_i-b_i|^p \right)^{1/p} $$

The figure below shows the distance from the origin to a point of unit length at different values of p in Minkowski distance¹⁸:

When p=1, it is Manhattan distance, also written as “L1 distance”
When p=2, it is Euclidean distance, also written as “L2 distance”
When p=n, it is Minkowski distance, also written as “Ln distance”

Cosine Similarity
#

The cosine value of the angle between two vectors — also called cosine similarity. Cosine similarity depends only on the angle between the two vectors, not on the vectors’ lengths¹⁹.

The smaller the angle between two vectors, the larger the cosine similarity. Value range: [-1, 1]. cos(0)=1, cos(90)=0, cos(180)=-1.

Cosine similarity between two vectors is written as: $$ cos (\theta) $$ Expressed in vector form: $$ cos (\theta)=\frac{\mathbf a\cdot \mathbf b }{||\mathbf a|| , ||\mathbf b||}= \frac{ \sum_{i=1}^n \mathbf a_i \mathbf b_i}{ \sqrt {\sum_{i=1}^n \mathbf a_i ^2} \cdot \sqrt {\sum_{i=1}^n \mathbf b_i ^2}} $$

Inner Product
#

Also called the dot product, it can be used to represent the length and angle of vectors. The inner product equals the Euclidean distance of the vectors multiplied by the cosine of the angle between them.

Inner product in 2D space: $$ \mathbf a\cdot \mathbf b=||\mathbf a|| , ||\mathbf b||, cos \theta $$ or $$ \mathbf a\cdot \mathbf b= a_1 b_1 + a_2 b_2 $$ Inner product in n-dimensional space (a=[a1,a2,···,an], b=[b1,b2,···,bn]): $$ \mathbf a\cdot \mathbf b=\sum_{i=1}^n a_ib_i= a_1b_1 + a_2b_2 + \cdots + a_nb_n $$

Now the following diagram should make sense. Using the formulas above, you can also reverse-engineer what the distance operators mean for n-dimensional vectors.

They are: Euclidean distance, cosine distance, and inner product²⁰.

All three can describe the similarity between two vectors.

Euclidean distance: contains only distance information between the two vectors
Cosine distance: contains only angle information between the two vectors
Inner product: contains both distance information and angle information

Of course, there are more mathematical models for vector similarity computation, but it depends on whether the vector database supports them.

Jaccard Distance
#

In short: intersection divided by union²¹.

Formula: $$ J(A,B)= \frac{|A\cap B| }{|A \cup B|} $$

Expressed in vectors, it computes the ratio of the count of equal elements to the count of unequal elements²².

Hamming Distance
#

The number of differing positions between two strings or vectors of equal length²³.

Examples:

“karolin” and “kathrin” is 3.
“karolin” and “kerstin” is 3.
“kathrin” and “kerstin” is 4.
0000 and 1111 is 4.
2173896 and 2233796 is 3.

Illustration²⁴:

Delaunay Triangulation
#

Delaunay triangulation is an operation on a set of points in a plane. It subdivides the convex hull of these points (which contains multiple points) into multiple triangles, where the circumcircle of each triangle contains no point from the set. This maximizes the minimum angle among all triangles and tends to avoid producing skinny triangles²⁵.

Does NOT satisfy “the circumcircle of each triangle contains no point from the set”:

DOES satisfy “the circumcircle of each triangle contains no point from the set”:

For example, triangulating a point set:

A valid triangulation:

Delaunay triangulation is not actually an algorithm — it merely defines what a “good” triangular mesh looks like. Its excellent properties are the empty-circle property and the maximized-minimum-angle property. These two properties avoid the creation of skinny triangles and make Delaunay triangulation widely applicable.

Voronoi Diagram
#

Delaunay triangulation is a triangulation of a discrete point set P in general position, and it corresponds to the dual graph of P’s Voronoi diagram. The circumcenters of Delaunay triangles are the vertices of the Voronoi diagram. In 2D, Voronoi vertices are connected by edges, which can be derived from the adjacency relationships of Delaunay triangles: if two triangles share an edge in the Delaunay triangulation, their circumcenters should be connected by an edge in the Voronoi tessellation²⁶:

The key property of a Voronoi diagram is: the distance from a centroid to any point within its region is smaller than the distance from that point to any other centroid. $$ R_k={x \in X ,|,d(x,P_k) \le d(x,P_j) ; \mathrm{for ,all },j \neq k} $$ Rk is the centroid, d(x,Pk) is the distance from the centroid to any point within its region, and d(x,Pj) is the distance from other centroids to any point in that region.

Due to different ways of computing the distance d, Voronoi diagrams can take on different appearances²⁷:

Vector Database Indexes
#

Nearest Neighbor Search
#

ENN (Exact Nearest Neighbor): Finding the point or vector closest to a query point in a given dataset. This method guarantees the highest accuracy, but as the dataset size increases, the computational cost rises sharply because it requires evaluating the distance between the query point and every point in the dataset.

ANN (Approximate Nearest Neighbor): To improve efficiency, approximately finding the nearest point to the query point at the cost of some accuracy. This method is implemented through various algorithms and can significantly reduce computational cost, especially effective when dealing with large-scale datasets.

KNN (K-Nearest Neighbors): A commonly used machine learning algorithm that works by finding the K nearest neighbors to a given query point in the dataset.

Index Evaluation Criteria
#

Evaluating the quality of an index always depends on the specific data model, but in general, it includes the following points:

Query time: Query speed is critical, especially important in large model contexts.
Query quality: ANN queries won’t always return perfectly accurate results, but the query quality must not deviate too much. Query quality has many metrics, the most common being recall.
Memory consumption: The memory consumed by the query index — searching in memory is clearly faster than searching on disk.
Training time: Some search methods require training to reach a good state.
Write time: The impact on the index when writing vectors, including all maintenance.

Most of these metrics are straightforward. Here we’ll focus on query quality:

In ANN search, results are not always exact. When searching a set of elements, the concepts include: the query scope (retrieved elements), all correct elements (relevant elements), the returned correct elements (true positives), and the returned incorrect elements (false positives)²⁸:

TP = True positive; FP = False positive; TN = True negative; FN = False negative

Accuracy: $$ Accuracy=\frac{TP+TN}{TP+FP+TN+FN} $$ or: $$ Accuracy=\frac{\text{all correct elements}}{\text{all elements}} $$

Precision: $$ Precision=\frac{TP}{TP+FP} $$ or: $$ Precision=\frac{\text{retrieved correct elements}}{\text{all retrieved elements}} $$

Recall: $$ Recall=\frac{TP}{TP+FN} $$ or: $$ Recall=\frac{\text{retrieved correct elements}}{\text{all correct elements}} $$

F-measure: Equivalent to weighted precision and recall $$ Recall=2 \cdot \frac{precision \cdot recall}{precision+recall} $$

Example: Consider a computer program designed to identify dogs (and related elements) in digital photos. When processing a photo containing ten cats and twelve dogs, the program identifies eight dogs. Among the eight identified as dogs, only five are actually dogs (true positives), while the other three are cats (false positives). Seven dogs were missed (false negatives), and seven cats were correctly excluded (true negatives). For this program:

Accuracy = 12/(10+12) (largely independent of the identification program itself)
Precision = 5/8 (true positives / all retrieved elements)
Recall = 5/12 (true positives / all correct elements)
F-measure = 2*[(5/18)*(5/12)]/[(5/18)+(5/12)]

Locality-Sensitive Hashing (LSH)
#

LSH is a method for narrowing the search scope by converting data vectors into hash values while preserving information about their similarity.

LSH Construction
#

LSH has many implementations. Here we introduce the more traditional one. This traditional LSH implementation consists of three parts²²:

Shingling: Encode the original text into vectors.
MinHashing: Convert the vectors into a special representation called a signature, used for comparing similarity between them.
LSH function: Hash the signatures into different buckets. If a pair of vectors’ signatures fall into the same bucket at least once, they are considered candidates.

Shingling
#

Shingling is a method of embedding (in my personal opinion). Shingling identifies natural language as k consecutive tokens, with duplicate tokens removed²²:

At this point, we have a set of tokens based on k-grams. The next step is to convert them into vectors.

Start with an all-zero vector, whose length equals the length of the token set. Set the position corresponding to each token to 1:

The final result is a very long vector containing only 0s and 1s, where the vector’s information captures the semantics of a sentence.

MinHashing
#

Since the vector dimensionality is extremely high, directly computing approximate distances using one-hot encoded vectors yields very poor results. We need to convert sparse vectors into dense vectors — this process is called MinHashing in LSH, and the converted vector is called a MinHashing signature.

MinHashing can be a bit tricky for beginners at first, but once you grasp it, you’ll find it very simple.

MinHashing is a hash function that permutes the components of an input vector and then returns the first index where the permuted vector component equals 1.

First, apply a permutation: rearrange the components of a vector.
Return the index of the first element that equals 1 after permutation.

For example:

u1 vector (0,0,1,1,0): after the first random permutation, the corresponding index is 0; after the second random permutation, the corresponding index is 0²⁹. u1’s MinHashing signature is (0,0).

In practice, multiple minhash values can be used to approximately compute the Jaccard similarity between vectors. The more minhash values used, the more accurate the approximation.

LSH Function
#

Even after converting sparse vectors into dense vectors, the dense vectors can still have high dimensionality, making direct retrieval inefficient.

We can improve query efficiency using hash tables. However, note that using a completely random hash algorithm easily places nearby vectors into different hash buckets. We need a hash algorithm that places nearby vectors into the same hash bucket — this is LSH: Locality-Sensitive Hashing.

The LSH mechanism builds a hash table consisting of several parts which puts a pair of signatures into the same bucket if they have at least one corresponding part.

The concept of locality-sensitive hashing is also simple: split the signature into bands, compute hash values for each sub-signature band, and designate those with colliding sub-hash values as candidates.

The following example is easy to understand — read through it:

Thinking in terms of extremes: b=1 means no banding at all — direct hashing, completely defeating the purpose of LSH. b=number of signature elements means one band per element, i.e., one hash value per element — this can achieve relatively accurate approximate comparison, but it imposes a massive burden on computation and memory.

LSH Parameters and Error Rate
#

The probability that a vector becomes a candidate vector directly affects recall. The probability of a candidate vector is as follows, where:

s represents similarity
b represents the number of bands
r represents the number of rows per band

If we plot P against s using the formula, the relationship between vector similarity and candidate probability is as follows:

The larger the number of bands b, the smaller the candidate similarity probability.

At the same time, adjusting b and s affects P, and P is related to FP and TN.

For example, returning more candidates naturally leads to more false positives — i.e., returning non-similar “candidate pairs.” This is an inevitable consequence of modifying the parameter b.

TP = True positive; FP = False positive; TN = True negative; FN = False negative

LSH is susceptible to high-dimensional data: more dimensions require longer signatures and more computation to maintain good search quality. In such cases, other indexes are recommended.

More
#

There are two more articles I haven’t finished digesting — they seem to be related to binary vectors and Euclidean distance:

https://towardsdatascience.com/similarity-search-part-6-random-projections-with-lsh-forest-f2e9b31dcc47

https://towardsdatascience.com/similarity-search-part-7-lsh-compositions-1b2ae8239aca

HNSW Index
#

The HNSW algorithm (Hierarchical Navigable Small World) is a multi-layer graph-based proximity algorithm. HNSW is currently one of the most popular vector index algorithms.

At a high level, HNSW is based on the Small World Theory. The Small World Theory originally stems from the Six Degrees of Separation theory in social psychology — any two people can be connected through at most five layers of social relationships. In other words, any two people on Earth can be connected through at most six steps of social connections. The Small World Theory was later widely accepted through experimental and empirical evidence and extended to non-social relationship networks. Note that the Small World Theory is a phenomenon.

In short, the Small World Theory explains that “the connection between two entities is actually very short.” What HNSW does is establish connections between elements and reduce the number of connections.

HNSW Index Construction
#

Let’s look at the HNSW paper’s algorithm for constructing HNSW graph layers³⁰:

Several elements in the construction algorithm are important:

M is the number of new edges (connections) added, representing the number of new edges for a newly inserted node.
Mmax is the maximum number of edges per node. If neighboring nodes are inserted continuously, the edge count of existing neighboring nodes could keep increasing, wasting computational resources during search. When inserting a new node causes an existing neighboring node’s edge count to exceed Mmax, shrink connection is needed.
efConstruction is the set of neighboring nodes.

Construction illustration³¹:

Steps for HNSW node insertion (without shrink connection):

When a new node is inserted, first find neighboring nodes at the top layer using efConstruction. Use the found nearest neighbor as the entry point to descend to the next layer, then continue searching for neighbors using that layer’s efConstruction.
Perform node insertion at a certain layer (e.g., L=2). Select M nodes from efConstruction and connect them to the new node — at this point, 1 new node is added with M edges connected to it.
Repeat step 2 until reaching the bottom layer (layer0).

HNSW Heuristic Neighbor Selection
#

The basic HNSW index structure construction has another problem: if two clusters are relatively far apart, according to the basic HNSW construction algorithm, the two clusters are almost impossible to connect, because the basic HNSW construction algorithm is built on the nearest neighbor nodes in efConstruction.

The HNSW original paper not only proposed the basic HNSW construction algorithm but also introduced a heuristic algorithm for solving the isolated cluster problem:

Fig.2 Heuristic for selecting graph neighbors for two isolated clusters. A new element is inserted on the boundary of cluster 1. All the element’s nearest neighbors belong to cluster 1, thus missing the Delaunay triangulation edges between the clusters. However, the heuristic selects element e2 from cluster 2, so if the inserted element is closer to e2 than to any other element from cluster 1, global connectivity is maintained.

“The heuristic algorithm not only considers the nearest distance between nodes in the graph but also considers connectivity between different regions of the graph.”

As shown below, when adding node X, the heuristic algorithm should be applied here — establishing connectivity with cluster A, rather than simply adding to the nearest neighbor nodes:

HNSW Index Search
#

The main logic of HNSW’s KNN search method as described in the HNSW original paper consists of the following two algorithms:

Algorithm 2 appears slightly more complex, but the logic is actually simple — Algorithm 2 finds the set of nearest neighbor nodes ef for q at that layer. In simple terms, Algorithm 2 adds candidate nodes to the ef set, compares distances, and removes the farthest nodes, so the returned W is the ef for q at that layer.
Algorithm 5 returns the K nearest neighbor nodes of q. It calls Algorithm 2 twice (or more). The first line in the for loop has input parameter ef=1, meaning non-bottom layers only find the single nearest ep (entry point). The bottom layer (lc=0) returns the K nearest neighbor node set W.

HNSW Complexity
#

The number of HNSW layers is a function of log(N).

Search complexity: Complexity can be rigorously evaluated in a Delaunay graph, with the average complexity being O(log(N)) (for non-Delaunay graphs, such as graphs with heuristic neighbor selection, the paper does not provide a specific complexity formula).

Construction complexity: HNSW is constructed by iteratively inserting all elements, with average complexity O(N∙log(N)).

HNSW Index Parameters
#

Generally, HNSW indexes for vector data have several adjustable parameters that affect index construction speed, recall, etc. Different databases may have slightly different parameters. Here we use pgvector’s HNSW parameters as an example:

Index construction parameters:

m: Maximum number of edges per vector, default 16. Equivalent to Mmax in the paper.
ef_construction: Number of vectors in the neighbor list during index construction, default 64. Equivalent to ef_construction in the paper.

Index search parameters:

hnsw.ef_search: Adjusts the number of vectors in the neighbor list during search (also equivalent to ef_construction in the paper). Must be greater than or equal to limit.

Impact of adjusting ef_construction on creation time and recall during index construction²⁰:

Increasing ef_construction improves recall but extends index creation time. After ef_construction=256, index construction time increases noticeably but recall improvement is not obvious.

Increasing m also improves recall and extends index creation time. After m=36, index construction time increases noticeably but recall improvement is not obvious.

Similarly, increasing hnsw.ef_search improves recall at the cost of performance.

IVFFlat Index
#

IVFFlat stands for Inverted File with Flat Compression. (What’s the relationship with “invert”? Do all indexes that can’t be categorized get called inverted?) The core concept of the IVFFlat index is based on the Voronoi diagram:

The key property of a Voronoi diagram is: the distance from a centroid to any point within its region is smaller than the distance from that point to any other centroid.

This property is expressed in formula form: $$ R_k={x \in X ,|,d(x,P_k) \le d(x,P_j) ; \mathrm{for ,all },j \neq k} $$ Rk is the centroid, d(x,Pk) is the distance from the centroid to any point within its region, and d(x,Pj) is the distance from other centroids to any point in that region.

Using this concept, we can partition many vectors into regions by setting centroids, and then use the Voronoi diagram property to roughly find nearby points.

IVFFlat Index Construction
#

Let’s reduce high-dimensional space to 2D for understanding IVFFlat index construction³².

For example, the following set of X marks represents points (or vectors). Suppose we have three centroids:

The three centroids partition 3 Voronoi cells, and all points are assigned to their respective Voronoi cells:

IVFFlat Index Search
#

Now there is a query node. Compute its distance to all centroids, find the nearest centroid, and the cell containing that centroid is the region to search next. Finally, within that region, find the neighboring nodes³³:

Boundary Problem:

The above search path has a boundary problem. When the query is near a region boundary, if the true nearest node is in another region, the algorithm of “only searching for neighboring nodes within the region” will not find the true nearest neighbor.

The boundary problem is fundamentally because:

The Voronoi diagram only guarantees that the distance from a node to its own region’s centroid is smaller than the distance to other centroids, but it does NOT guarantee that the distance from a node to other nodes in its own region is smaller than the distance to nodes in other regions.

This problem can be mitigated by increasing the number of regions searched. For example, increasing the number of regions searched from 1 to 3:

Increasing the number of search regions is generally set as a parameter in databases, such as ivfflat.probes in pgvector.

IVFFlat Search Summary:

Compute the distance from the query node to all other centroids, find the nearest one.
Based on the input parameter for the number of cells to query (e.g., probes), search for neighboring points in the top probes cells.

IVFFlat Index Parameters
#

Similarly, vector databases that support IVFFlat indexes generally have at least two parameters: list and probe. These parameters affect index search performance and recall. Here we use Faiss parameters as an example³².

nlist: Number of regions to construct. Increasing nlist increases the time to search for the nearest centroid but reduces the time to search for nodes within a region.
nprobe: Number of regions to search. Increasing nprobe increases the number of regions searched, which obviously reduces search performance but improves recall.

Theoretically, for nlist, it’s best to test specifically against the structure of the vector data and the database type — increasing nlist does not always reduce response time. For nprobe, increasing nprobe definitely reduces search performance and improves recall, but making nprobe too large is meaningless and goes against the original intent of ANN.

The following is from Pinecone’s performance testing of the Faiss IVFFlat index:

PQ Product Quantization
#

One million dense vectors may require gigabytes of memory, and real-world vectors far exceed this number. Without management, similarity vector search can require enormous amounts of memory — yet RAM is limited. Vector size increases with vector dimensionality and the number of vectors.

Product Quantization (PQ) aims to reduce memory usage and can also improve query speed (because the amount of computation is reduced). PQ is a lossy compression method, which leads to reduced vector retrieval accuracy, but this is acceptable within ANN requirements.

PQ’s algorithm logic is slightly more complex than other algorithms. I strongly recommend this article: Similarity Search, Part 2: Product Quantization³⁴.

PQ Construction
#

Step description:

Subvectors — Split the original high-dimensional vector into n low-dimensional sub-vectors.
Codebook — Use the k-means algorithm (or other algorithms) to compute the Voronoi diagram for each set of all sub-vectors, producing n different Voronoi diagrams. These Voronoi diagrams are the codebooks (assuming each Voronoi diagram has k centroids).
Clustering — Place the n sub-vectors into their respective already-clustered Voronoi diagrams and compute the nearest centroid.
Quantized vectors — Take these n nearest centroids as the new vector — the quantized vector.
Reproduction values — Take the nearest centroid index for each of the n subspaces as new values; the combined new values are called the PQ code.

Step 5, reproduction values, in detail:

Based on the n sub-vectors and the k centroids in each subspace, we obtain an n×k centroid matrix. Taking the index of the nearest centroid for each sub-vector gives the PQ code.

(btw: to be rigorous, all element indices in the diagram below should start from 1, not 0.)

The new PQ code is equivalent to a lossy-compressed new vector (reproduction value) of the original vector. New distance calculations can directly compute the L2 distance of the PQ codes.

PQ Retrieval
#

Based on the PQ original paper³⁵, there are two PQ retrieval modes:

Symmetric mode: The distance between vector x and vector y is approximated by the distance between their centroids q(x) and q(y). In other words, the distance between two vectors can be approximated by the distance between their PQ codes.
Asymmetric mode: The distance between vector x and vector y is approximated by the distance from x to the centroid q(y). In other words, the distance between two vectors can be computed using the original query vector value and the other vector’s PQ code.

Clearly, the distance accuracy differs between the two modes:

The figure above shows the distance accuracy between two vectors under different modes, with 8 subspaces and 256 centroids. It can be seen that the asymmetric mode has higher accuracy than the symmetric mode.

When comparing distances between two vectors, the symmetric and asymmetric distance computation models are quite useful. However, in the scenario of finding PQ approximate vectors, there are some differences — especially the symmetric mode, where distortion can be quite severe:

The symmetric mode’s query speed is very fast because the code table has already been computed and preserved during the PQ construction process. You only need to first compute the query vector x’s PQ code via the code table (minimal computation), then reverse-lookup the code table to get the corresponding sub-code-table — all vectors in this sub-code-table are approximate vectors at equal distance. This method requires extremely little computation — just a direct table lookup.
The symmetric mode’s distortion is relatively severe (the two figures above don’t fully capture it — imagine it as a Voronoi diagram where one cell contains multiple vectors, and you’ll realize how severe the symmetric distortion can be). The asymmetric mode can slightly alleviate this problem. In asymmetric mode, first compute the PQ code of vector x, then similarly reverse-lookup the code table to get the corresponding sub-code-table, then compute distances between vector x and the vectors in this sub-code-table to obtain KNN. Its computational cost is n×km (n = number of subspaces, km ≈ total vector count / centroid count).

Asymmetric mode requires finding the centroid via the PQ code, then searching for KNN within the subspace where the centroid resides. The distance between the query vector x and an existing vector y is approximated by the distance between x and y’s centroid.

PQ asymmetric retrieval³⁴:

Steps of PQ asymmetric retrieval:

Split the query vector into multiple sub-vectors.
Compute the distance between sub-vectors and the centroid matrix.
Take the nearest centroid in each subspace as the query vector’s PQ code.
Compute the approximate distance using the query vector and the centroid corresponding to the PQ code. Distances can be computed independently in each subspace and then summed.

As mentioned earlier, asymmetric mode’s approximate distance computation is slightly better than symmetric mode, but in some scenarios, the asymmetric distance can still deviate significantly from the actual distance:

This is easier to understand from the figure above. Within the same cell, the distance between the farthest vector and the centroid can differ significantly from the distance between the closest vector and the centroid. Computing only the partial distance to the centroid cannot capture this difference.

PQ Parameters and Their Impact
#

PQ has at least two parameters that significantly affect performance and memory: the number of subspaces m and the number of centroids per subspace k.

Recall:

The product quantizer is parametrized by the number of subvectors m and the number of quantizers per subvector k*, producing a code of length m × log2 k

With m subspaces, each having k* centroids, the length (in bits) of a PQ code is³⁵: $$ code ; length , (bits)=m \cdot \log_2 k^* $$

The more subspaces m, the higher the recall; the longer the PQ code, the higher the recall. Longer PQ code essentially means more centroids. Note that the specific values here are based on the paper’s dataset.

Memory and complexity:

k represents the number of cluster centroids, D represents dimension, m represents the number of subspaces. k* represents centroids within a subspace, D* represents dimensions within a subspace.

For example, with k=2048, D=128, m=8, the complexity is as follows³⁶:

Operation	Memory and complexity
k-means	kD = 2048×128 = 262144
PQ	mkD = (k^(1/m))×D = (2048^(1/8))×128 = 332

It can be seen that PQ significantly reduces complexity during search.

DiskANN & Vamana
#

The DiskANN original paper Abstract³⁷:

Current state-of-the-art approximate nearest neighbor search (ANNS) algorithms generate indices that must be stored in main memory for fast high-recall search. This makes them expensive and limits the size of the dataset. We present a new graph-based indexing and search system called DiskANN that can index, store, and search a billion point database on a single workstation with just 64GB RAM and an inexpensive solid-state drive (SSD).

At the time (the paper was published in 2019), state-of-the-art ANN algorithms all relied on RAM for high recall and performance. This approach was not only expensive but also limited dataset size. DiskANN requires only 64GB RAM and an affordable SSD.

Vamana Construction
#

Vamana iteratively builds a directed graph, starting from a random graph where each node represents a data point in the vector space. Initially, the graph is highly connected — all nodes are connected to each other. The graph is then optimized using an objective function that aims to maximize connectivity between the closest nodes. This is achieved by pruning most random short-range edges while adding certain long-range edges that connect distant nodes (to accelerate graph traversal)³⁷.

The figure shows 200 2D points after two iterations. The first iteration aggressively prunes edges but also removes long edges that reduce path length; when alpha is increased to relax the pruning condition, long edges are added back³⁸. For the specific algorithm, refer to the paper — this is roughly the idea.

The DiskANN Algorithm
#

From the paper’s “The DiskANN Index Design”:

The high-level idea is simple: run Vamana on a dataset P and store the resulting graph on an SSD. At search time, whenever Algorithm 1 requires the out-neighbors of a point p, we simply fetch this information from the SSD. However, note that just storing the vector data for a billion points in 100 dimensions would far exceed the RAM on a workstation! This raises two questions: how do we build a graph over a billion points, and how do we do distance comparisons between the query point and points in our candidate list at search time in Algorithm 1, if we cannot even store the vector data?

Run Vamana on the vector set and store it on SSD. When the dataset is very large, two problems must be addressed:

How to index such a large-scale dataset with limited memory resources?

k-means + Vamana stacking algorithm: First, use k-means to partition the data into k clusters, then assign each point to the nearest i clusters. Usually, i=2 is sufficient. Build an in-memory Vamana index for each cluster, and finally merge the k Vamana indexes into one.

If the original data cannot be loaded into memory, how to compute distances during search?

Use compressed vectors (e.g., PQ) and store the compressed vectors in main memory.

If index data is stored on SSD, disk access count and disk read/write requests must be minimized to ensure low search latency; at the same time, lossy compression reduces recall. Therefore, the DiskANN paper proposes three optimization strategies:

Beam Search: Simply put, preload neighbor information. When searching for point p, if p’s neighbors are not in memory, they must be loaded from disk. Since the time required for a small number of random SSD accesses is roughly the same as the time for a single SSD sector access, the neighbor information of W unvisited points can be loaded in one batch. W should not be set too large or too small. Setting W too large wastes computational resources and SSD bandwidth, while setting it too small increases search latency.
Caching Frequently Visited Vertices: Aims to reduce disk access count. Cache all points within C hops from the starting point in memory. The value of C is best set between 3 and 4.
Implicit Re-Ranking Using Full-Precision Vectors: Since PQ is lossy compression, PQ-based distance algorithms only approximate the actual distance. To eliminate this discrepancy, we store the distance from each point to all its neighbors — this is full-precision. As for the implementation principle, in simple terms, it also leverages disk loading efficiency.

Based on the paper, DiskANN’s execution efficiency and recall outperform IVF and HNSW:

References
#

Original article (Chinese): 向量数据库相关概念

Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond ↩︎
Chih-Hao Liu 66 Classic LLM Papers ↩︎ ↩︎
A Survey of Large Language Models ↩︎
一文讲清楚，AI、AGI、AIGC与AIGC、NLP、LLM，ChatGPT等概念 ↩︎
https://en.wikipedia.org/wiki/Prompt_engineering ↩︎ ↩︎
RAG original paper ↩︎
Jonathan Katz pgconfdev2024 Vectors: How to better support a nasty data type ↩︎
OpenAI recommends using vector databases https://openai.com/index/chatgpt-plugins/ ↩︎
Vector databases (1): What makes each one different? ↩︎
Vector database performance comparison ↩︎
https://en.wikipedia.org/wiki/Vector_(mathematics_and_physics) ↩︎
https://en.wikipedia.org/wiki/Unit_vector ↩︎
OpenAI on unit vector usage https://platform.openai.com/docs/guides/embeddings/frequently-asked-questions ↩︎
Pinecone Natural Language Processing for Semantic Search https://www.pinecone.io/learn/series/nlp/dense-vector-embeddings-nlp/ ↩︎
Yao Yuan A Casual Discussion of Various Spaces in Mathematics ↩︎
https://en.wikipedia.org/wiki/Euclidean_distance ↩︎
https://en.wikipedia.org/wiki/Taxicab_geometry ↩︎
https://en.wikipedia.org/wiki/Minkowski_distance ↩︎
https://en.wikipedia.org/wiki/Sine_and_cosine ↩︎
Jonathan Katz pgconfeu2023 Vectors are the new JSON ↩︎ ↩︎
https://en.wikipedia.org/wiki/Jaccard_index ↩︎
Vyacheslav Efimov Similarity Search, Part 5: Locality Sensitive Hashing (LSH) ↩︎ ↩︎ ↩︎
https://en.wikipedia.org/wiki/Hamming_distance ↩︎
Vyacheslav Efimov Similarity Search, Part 6: Random Projections with LSH Forest ↩ ↩︎
earthwjl Delaunay Triangulation Study Notes ↩︎
https://en.wikipedia.org/wiki/Delaunay_triangulation ↩︎
https://en.wikipedia.org/wiki/Voronoi_diagram ↩︎
https://en.wikipedia.org/wiki/Precision_and_recall ↩︎
Jianshu LSH (Locality Sensitive Hashing) Algorithm ↩︎
HNSW Original Paper ↩︎
Vyacheslav Efimov Similarity Search, Part 4: Hierarchical Navigable Small World (HNSW) ↩︎
https://www.pinecone.io/learn/series/faiss/vector-indexes/ ↩︎ ↩︎
Vyacheslav Efimov Similarity Search, Part 1: kNN & Inverted File Index ↩︎
Vyacheslav Efimov Similarity Search, Part 2: Product Quantization ↩︎ ↩︎
PQ Original Paper ↩︎ ↩︎
Pinecone Faiss Manual https://www.pinecone.io/learn/series/faiss/product-quantization/ ↩︎
DiskANN Original Paper ↩︎ ↩︎
DiskANN, A Disk-based ANNS Solution with High Recall and High QPS on Billion-scale Dataset https://milvus.io/blog/2021-09-24-diskann.md ↩︎

Order	default	C	en_US	en_US.utf8	zh_CN	zh_CN.utf8
1	〇	A	〇	〇	a	a
2	a	AA	a	a	A	A
3	A	a	A	A	aa	aa
4	aa	aa	aa	aa	AA	AA
5	AA	〇	AA	AA	阿	阿
6	啊	啊	啊	啊	啊	啊
7	阿	阿	阿	阿	〇	〇

Order	default	C	en_US	en_US.utf8	zh_CN	zh_CN.utf8
1	〇	A	〇	〇	a	a
2	a	AA	a	a	A	A
3	A	a	A	A	aa	aa
4	aa	aa	aa	aa	AA	AA
5	AA	〇	AA	AA	阿	阿
6	啊	啊	啊	啊	啊	啊
7	阿	阿	阿	阿	〇	〇

PostgreSQL内功修炼 on Last DBA

UUID v4 and v7: Collision Incidents and Performance Benchmarks

TL;DR #

The UUID v4 Collision Incident #

Cause 1: Unreliable entropy sources #

Cause 2: Known npm uuid package bugs #

Cause 3: Linux kernel /dev/random race condition #

Cause 4: Go UUID library not checking return values #

Cause 5: Historical AMD CPU RNG defects #

UUID v7 Performance Comparison in PG 16 #

Test Conditions #

Performance Results #

Why Such a Big Difference #

Summary #

From collation mismatch Exception to Its Principles

Problem Phenomenon #

Problem Analysis #

Why is the error reported? #

Source Code Understanding #

pg_newlocale_from_collation Caching and Checking pg_collation #

CheckMyDatabase Caching and Checking pg_database #

Function Differences #

Why Are There Fewer Errors After Only Refreshing the Database? #

Testing #

Test: Refresh db without refreshing pg_collation, only db coll version changes #

Test: Refresh db without refreshing pg_collation, specifying expression sort reports warning #

Test: Refresh db without refreshing pg_collation, creating a new index with specified collation reports warning #

Test: Refresh db without refreshing pg_collation, existing index with specified collation does not report warning #

Summary of This Problem #

Solutions for This Problem #

More #

Key Summary of glibc Upgrade Related Issues #

What to Do After Physical Migration #

ref #

A Brief Review of Logical Replication in Oracle, MySQL, and PostgreSQL

PostgreSQL Logical Replication #

MySQL’s binlog #

Oracle Logical Replication #

Query Conflicts: From a Static Table Conflict to Its Root Cause

Problem Symptoms #

The Symptom #

Why a Query Conflict on a Static Table Matters #

Root Cause Analysis #

XLOG_HEAP2_CLEAN #

ResolveRecoveryConflictWithSnapshot #

GetConflictingVirtualXIDs #

In-Page Pruning #

Testing #

Test: Cross-Table Query Conflict #

Test: Vacuum Produces In-Page Pruning #

Test: UPDATE Produces In-Page Pruning #

Test: Hint-Bit Writeback Producing In-Page Pruning? #

Test: SELECT Produces In-Page Pruning #

Test: Shared Table Cross-Database Query Conflict #

Conclusions #

Developer Perspective #

Operations Perspective #

PostgreSQL DDL Pitfalls and Clever Solutions

Linux Memory Advanced

Memory Basic Concepts #

buddy #

page table & PTE #

TLB #

Reverse Mapping #

Huge Pages #

C Library and System Calls #

Page Fault Exception #

Copy-On-Write (COW) #

mmap, brk & Shared Memory Mapping Area, Heap Area #

VM #

compact #

concept & param #

dirty #

concept & param #

Observing Dirty Pages #

os dirty != pg dirty #

swappiness #

swappiness=0 #

inconsistent swap behavior #

memory overcommitment #

TL;DR
#

The UUID v4 Collision Incident
#

Cause 1: Unreliable entropy sources
#

Cause 2: Known npm uuid package bugs
#

Cause 3: Linux kernel /dev/random race condition
#

Cause 4: Go UUID library not checking return values
#

Cause 5: Historical AMD CPU RNG defects
#

UUID v7 Performance Comparison in PG 16
#

Test Conditions
#

Performance Results
#

Why Such a Big Difference
#

Summary
#

Problem Phenomenon
#

Problem Analysis
#

Why is the error reported?
#

Source Code Understanding
#

`pg_newlocale_from_collation` Caching and Checking `pg_collation`
#

`CheckMyDatabase` Caching and Checking `pg_database`
#

Function Differences
#

Why Are There Fewer Errors After Only Refreshing the Database?
#

Testing
#

Test: Refresh db without refreshing pg_collation, only db coll version changes
#

Test: Refresh db without refreshing pg_collation, specifying expression sort reports warning
#

Test: Refresh db without refreshing pg_collation, creating a new index with specified collation reports warning
#

Test: Refresh db without refreshing pg_collation, existing index with specified collation does not report warning
#

Summary of This Problem
#

Solutions for This Problem
#

More
#

Key Summary of glibc Upgrade Related Issues
#

What to Do After Physical Migration
#

ref
#

PostgreSQL Logical Replication
#

MySQL’s binlog
#

Oracle Logical Replication
#

Problem Symptoms
#

The Symptom
#

Why a Query Conflict on a Static Table Matters
#

Root Cause Analysis
#

XLOG_HEAP2_CLEAN
#

ResolveRecoveryConflictWithSnapshot
#

GetConflictingVirtualXIDs
#

In-Page Pruning
#

Testing
#

Test: Cross-Table Query Conflict
#

Test: Vacuum Produces In-Page Pruning
#

Test: UPDATE Produces In-Page Pruning
#

Test: Hint-Bit Writeback Producing In-Page Pruning?
#

Test: SELECT Produces In-Page Pruning
#

Test: Shared Table Cross-Database Query Conflict
#

Conclusions
#

Developer Perspective
#

Operations Perspective
#

Memory Basic Concepts
#

buddy
#

page table & PTE
#

TLB
#

Reverse Mapping
#

Huge Pages
#

C Library and System Calls
#

Page Fault Exception
#

Copy-On-Write (COW)
#

mmap, brk & Shared Memory Mapping Area, Heap Area
#

VM
#

compact
#

concept & param
#

dirty
#

concept & param
#

Observing Dirty Pages
#

os dirty != pg dirty
#

swappiness
#

swappiness=0
#

inconsistent swap behavior
#

memory overcommitment
#

concept & param
#

Reserve Memory and Overcommit
#

Observing Overcommit
#

watermark
#

min_free_kbytes
#

watermark_scale_factor
#

watermark_boost_factor
#

Order	default	C	en_US	en_US.utf8	zh_CN	zh_CN.utf8
1	〇	A	〇	〇	a	a
2	a	AA	a	a	A	A
3	A	a	A	A	aa	aa
4	aa	aa	aa	aa	AA	AA
5	AA	〇	AA	AA	阿	阿
6	啊	啊	啊	啊	啊	啊
7	阿	阿	阿	阿	〇	〇