The Last DBA on Last DBA

A DBA's Perspective on the 0526 Approved Database List

Fri, 29 May 2026 00:00:00 +0000

AI rate 5%

TL;DR
#

On May 26, the Xinchuang Database List 2026 No. 2 was released, with 23 products passing (8 centralized + 15 distributed) — the most ever. Most notably: Ping An, UnionPay, China Mobile, and China Telecom — four major buyers — had their self-incubated databases debut on the list. The Xinchuang logic has changed — buyers are no longer just buyers.

The Latest List
#

Historical batch statistics for the Xinchuang database list. Data source: China Information Security Evaluation Center (itsec.gov.cn), 8 batches total, 4 containing databases.

By Batch

Batch	Date	Database Products	Achieved Level II
2023#1	2023-12-26	11 (centralized)	None
2024#2	2024-09-30	17 (6 centralized + 11 distributed)	GaussDB
2025#2	2025-08-22	3 (centralized)	None
2026#2	2026-05-26	23 (8 centralized + 15 distributed)	Dameng/Yashan/GaussDB/GoldenDB

By Appearances (≥2 times)

Vendor	Count
Dameng	3
GBASE	3
Alibaba Cloud	3
HighGo	2
Tencent Cloud	2
East Golden	2
Vastdata	2
Huawei Cloud	2
ZTE (GoldenDB)	2
OceanBase	2
Kingbase	2
Shentong	2
Xugu	2
Yashan	2
Only 1 time	PingCAP/Wanli/Uxin/Ping An/China Mobile/UnionPay/Telecom Cloud/Timecho/Transwarp/DolphinDB/Z-Range/CM Suzhou

By Category: Big Tech / Unicorn / Major Buyer

Category	Vendors
Big Tech	Huawei Cloud (GaussDB/TaurusDB/DWS), Alibaba Cloud (PolarDB/AnalyticDB), Tencent Cloud (TDSQL), ZTE (GoldenDB), OceanBase (Ant Group)
Unicorns	PingCAP (TiDB), Yashan (SICS), Transwarp (ArgoDB), Timecho (TimechoDB), DolphinDB
Major Buyers	Ping An Tech (RASESQL), China UnionPay (UPDRDB), China Mobile (Panwei + He3DB), China Telecom Cloud (TeleDB)
Traditional Xinchuang	Dameng, Kingbase, GBASE, Shentong, HighGo, Xugu, Vastdata, East Golden, Wanli, Uxin

The Floodgates Open
#

When this list came out, my reaction was four words: the floodgates opened. 23 products — the most ever. A few highlights:

Ping An RASESQL. The most unexpected. Ping An Group’s fintech capabilities have always been strong, but there was almost no public information about them building a database. Seeing “RASESQL” on the list stunned me for several seconds. A financial buyer of Ping An’s scale — once their self-developed database passes national testing, their internal Xinchuang replacement roadmap gains one more path.

UnionPay UPDRDB. Equally mysterious. I had no idea UnionPay was building a distributed database before this. UnionPay’s transaction volume speaks for itself — a distributed database that can handle their own business won’t be technically weak.

Alibaba Cloud PolarDB for MySQL. The MySQL-compatible edition of PolarDB not passing had been something many people remembered. Now, all three of PolarDB’s main lines — PG edition, distributed edition, MySQL edition — have passed. Add AnalyticDB, and Alibaba Cloud’s database family is basically complete.

China Mobile Panwei + China Telecom TeleDB. China Mobile already had He3DB (CM Suzhou) pass national testing last year; this year Panwei is their second product. China Telecom TeleDB debuts. Both telecom operators now have their own incubated Xinchuang databases, which should significantly reduce their respective Xinchuang replacement pressure. Interestingly, China Unicom has been silent — their Xinchuang strategy is clearly different from Mobile and Telecom.

Transwarp ArgoDB. Transwarp started in the big data/Hadoop ecosystem and now their distributed database has passed national testing. Once crowned “China’s First Domestic Big Data Infrastructure Software Stock” with a market cap exceeding 30 billion, their path from data lake to Xinchuang database has been validated.

Impact
#

The most important signal from this floodgate opening: buyers can self-develop databases.

What are the implications?

Major buyers who succeed at self-development don’t have to be lambs to the slaughter.
Those major buyers who haven’t built one yet may restart their self-development efforts.
The market share that big tech and unicorns could compete for in the domestic database market just shrank.

Financial industry players UnionPay and Ping An, telecom players China Mobile and China Telecom — all passed national testing, effectively earning a “R&D Success” gold badge. Internally, each organization must be celebrating. For external vendors, what they’ve lost isn’t just major clients — more precisely, they’ve lost absolute bargaining power.

“I know you’re in a tough spot, and I know you can’t afford not to buy, so I’ll swap the butcher’s knife for a dragon-slaying blade and slaughter you to death” — for buyers who successfully incubated their own databases, this kind of predicament has been substantially eased. That’s significant.

As for where Xinchuang policy goes next, nobody can say. Based on previous lists, things should be getting stricter (last time only 3 databases passed), but this time they unexpectedly opened the floodgates. A sharp contraction next round isn’t impossible. Not just China Unicom — insurance industry players like CPIC and PICC, and even capable financial institutions, could consider jumping in to hand-roll their own database.

Bittersweet Reflections
#

Since our kernel team sits right behind me, I have some understanding of the Xinchuang R&D process. After consecutive failed submissions, the entire team’s morale was extremely low. I believe we weren’t the only ones — many teams whose submissions failed felt the same. For industries like finance and telecom, there’s a Xinchuang mandate, but if your self-developed product doesn’t pass approval, there’s no choice at the corporate strategy level, and at the team level, there’s no reason for existence. That’s why “passing national testing” carries such weight and influence. Thankfully they passed — heartfelt congratulations to them! RaseSQL No.1!

At the same time, it’s clear that Xinchuang results and direction are unstable, volatile, and impactful. It determines some companies’ strategies and many people’s fates. I myself am even a piece on this wheel of fortune.

Beyond those on the list, many organizations poured enormous effort but remain off the list. Their products might be terrible, or they might be excellent. But national testing is that stark watershed — a mysterious ticket of admission. Pass or fail — in the domestic market, those are two entirely different concepts.

OK, just some thoughts — might delete later.

Reference
#

https://www.itsec.gov.cn/aqkkcp/cpgg/

Original link: https://lastdba.com/2026/05/29/xinchuang-db-2026-review/

UUID v4 and v7: Collision Incidents and Performance Benchmarks

Fri, 29 May 2026 00:00:00 +0000

Source material: HN UUID v4 Collision Thread, dev.to UUID Benchmark

AI-generated ratio: 99%

TL;DR
#

UUID v4 collided — someone on HackerNews actually hit a real collision. The root cause was a software stack bug, not math. v4 and v7 have no fundamental difference in collision safety. The real difference is index performance: v7 is time-ordered, B-tree is more compact, writes are 35% faster, indexes are 22% smaller. Your UUID v4 is probably fine, but if you care about index performance, switching to v7 is a cheap win.

The UUID v4 Collision Incident
#

A HackerNews thread blew up — Ask HN: We just had an actual UUID v4 collision…, 479 upvotes, 347 comments.

The OP’s own words:

I know what you’re thinking… and I still can’t believe it, but… This morning, our database flagged a duplicate UUID (v4).

It wasn’t a double-insert bug. The code didn’t write it twice. Only ~15,000 rows in the table, using npm’s uuid package uuidv4(), and two rows created at different times collided on the same UUID:

b6133fd6-70fe-4fe3-bed6-8ca8fc9386cd

What’s the probability of a UUID v4 collision? 122 random bits, 2^122 ≈ 5.3×10^36 possibilities. With 15,000 records, collision probability is roughly 2×10^-29. Theoretically “impossible.”

But it happened.

Cause 1: Unreliable entropy sources
#

HN’s top-voted comment (jandrewrogers):

UUIDv4 security depends on high-quality entropy sources. Hardware defects, software bugs, and misunderstandings of “high-quality entropy” all break this assumption. Detecting entropy source failures is expensive, so nobody checks — until a collision happens.

UUID v4 is explicitly banned in high-reliability systems because entropy source quality cannot be verified.

Cause 2: Known npm uuid package bugs
#

The npm uuid package README itself warns:

This module may generate duplicate UUIDs when run in clients with deterministic random number generators, such as Googlebot crawlers.

More seriously, its internal rng() function has global mutable state. One commenter pointed out: calling rng() and sending the result effectively overwrites someone else’s random number, and you can predict it.

Related commit: 91805f665c

Community advice: use Node.js built-in crypto.randomUUID(), not the npm uuid package.

Cause 3: Linux kernel /dev/random race condition
#

Another comment:

I encountered duplicate UUIDs during soak testing of a distributed system. After extensive debugging, I found it was a Linux kernel race condition bug — on multi-processor systems, two processes simultaneously reading /dev/random could, with extremely low probability (~one in a million), get the same bytes.

Cause 4: Go UUID library not checking return values
#

Early Go UUID libraries called random functions without checking the return value length. “Request N bytes, got 3 bytes back” never happened on most hardware, so nobody checked — until production, where it generated thousands of duplicate UUIDs.

Cause 5: Historical AMD CPU RNG defects
#

Certain AMD CPUs had built-in random number generator issues. VM environments can also “virtualize away” entropy — both time sources and entropy sources can degrade inside VMs.

v4 and v7 have no fundamental difference in collision safety. The difference is in the first 48 bits — v4 is random, v7 is a timestamp. You’re unlikely to encounter timestamp source issues, and random source issues are equally rare. The HN thread is an interesting edge case. Knowing that a tiny number of people hit it is enough — you don’t need to distrust the UUID v4 in your own systems.

When choosing v4 vs v7, what you should really look at isn’t collisions — it’s index performance.

UUID v7 Performance Comparison in PG 16
#

UUID v7 has one concrete advantage over v4 in PostgreSQL: temporal clustering, more B-tree-friendly. v4 can bloat and v7 can bloat too — the difference is simply that v7’s first 48 bits are time-ordered, so inserts concentrate on the right side of the B-tree, reducing page splits.

Umang Sinha’s benchmark ran a rigorous comparison on a PG 16 Docker container (8 cores, 16GB, NVMe).

Test Conditions
#

CREATE TABLE uuid_v4_test (id UUID PRIMARY KEY, payload TEXT);
CREATE TABLE uuid_v7_test (id UUID PRIMARY KEY, payload TEXT);

Parameter	Value
Data volume	10 million rows per table
Batch size	10,000 rows per batch
Client	Go + pq driver
UUID generation	Pre-generated in memory, not timed

Performance Results
#

Metric	UUID v4	UUID v7	Improvement
Write 10M rows	5 min 35 sec	3 min 38 sec	35% faster
Table + index total size	3618 MB	3443 MB	5% smaller
B-tree index size	776 MB	602 MB	22% smaller
Point lookup	0.167 ms	0.038 ms	4.4x faster
Range scan	8.283 ms	3.791 ms	2.2x faster

Why Such a Big Difference
#

UUID v4 is fully random. Newly inserted UUIDs scatter randomly across the B-tree index, causing massive page splits and severe index fragmentation. UUID v7 has a millisecond-precision timestamp in the first 48 bits, so newly generated UUIDs are naturally ordered — writes cluster on the right side of the B-tree, page splits drop dramatically, and the index is much more compact.

The 22% smaller index isn’t magic — it’s reduced fragmentation. Point lookups being 4x faster isn’t surprising either — fewer B-tree levels, higher cache hit rates.

Summary
#

UUID v4 and v7 are identical in collision safety — both depend on entropy source quality, one fills the first 48 bits with random numbers, the other with a timestamp. Collisions are edge cases that a tiny number of people hit in specific environments. Your environment is probably fine — that basic judgment doesn’t change.

What you really should think about is index performance. v7’s temporal property makes B-trees more compact, with measured results of 35% faster writes, 22% smaller indexes, and 2-4x faster queries. If your system writes UUIDs at high volume, switching to v7 saves meaningful storage and CPU.

PG 18 will natively support gen_uuid_v7(). For now, generate UUIDs at the application layer. Whichever version you use, always add a UNIQUE constraint.

This article was originally published in Chinese on lastdba.com.

When PostgreSQL Becomes AI's Hands — Bruce Momjian's MCP Server in Practice

Wed, 27 May 2026 00:00:00 +0000

Original: Building an MCP Server Using Postgres, Bruce Momjian, PGDay Armenia 2026, CC BY 4.0.

AI-generated ratio: 80%

Bruce Momjian (PG core team, the one who has written release notes for 20+ years) recently gave a talk at PGDay Armenia 2026: Building an MCP Server Using Postgres. 70 slides, extremely dense. Theory and practice — a solid reference.

Reading it directly is hard work. Even having AI interpret it probably won’t make sense at first glance. I had to read for a while and ask several questions before it clicked.

These 70 slides can be cleanly split into two layers — the first half is theory, the second half is a hands-on demo. The two layers don’t have much to do with each other.

Theory Layer: Explaining the RAG → MCP Evolution Through Transformers (Slides 1-33)
#

The theory layer takes up nearly half the content, from LLM fundamentals to how MCP works. The outline is clear:

RAG vs MCP: In One Sentence
#

Everyone knows the RAG workflow: the programmer decides what data to query → retrieval results are appended to the system prompt → the LLM reads and generates a response. Pre-orchestrated — what the LLM can see is decided before the user even asks.

MCP is different. Tool descriptions are registered with the LLM, and the LLM decides for itself during generation whether to call a tool and which one. Dynamic decision-making — the programmer only exposes tools, the LLM handles orchestration.

Bruce sums it up in one sentence:

RAG can only do what the programmer pre-planned. MCP can dynamically adjust based on output quality, can iteratively call multiple tools, and can trigger external tasks.

“Word or MCP” — That Set of Vector Embedding Diagrams
#

Slides 18-33 are the core of the theory layer. Bruce draws a detailed internal Transformer flow diagram:

His logic: take each MCP tool’s description text (e.g., “Return the radiation level (CPM) at 13 Roberts Road…”), embed it into a vector using a text embedding model, and inject it into the attention layer’s vector space. Then at each inference step, the output vector matches against the nearest vector —

“The closest vector might be a word or an MCP.”

Is This Model Correct?
#

This is what puzzled me the most. Here are my thoughts.

Bruce’s 15 slides are beautifully drawn, but if you try to understand them as engineering implementation, there are problems:

① MCP tools don’t need “embedding.” In actual engineering, tool definitions are written directly into the system prompt as text. The LLM reads “You have these tools: geiger(), get_pretzel_inventory()…” and uses semantic understanding to decide when to call them. There’s no need to compute tool descriptions as vectors, no need to do cosine distance comparisons against word vectors. The essence of Bruce’s teaching model is explaining “LLM decision-making” as “nearest vector matching” — this is closer to the retrieval paradigm than the generation paradigm.

② Attention doesn’t produce a “find nearest” operation. output = Σ(softmax(Q·K) × V) yields a weighted-mixed context vector. There’s no step of “binary choice between the word embedding table and the tool embedding table.” The actual mechanism for LLM tool selection is: attention produces hidden states → LM head → softmax over vocabulary → output tool call JSON. There’s never a “word vs tool” choice, only a softmax over the entire vocabulary.

③ System prompt and user prompt have no boundary in attention. A token sequence is just a token sequence — attention blocks do Q·K dot products on all tokens equally. There is no “system zone” or “user zone.”

So these 33 theory slides can be seen as a simplified teaching model Bruce built for DBAs without an AI background — visually appealing and easy to understand, but don’t use it as an architecture diagram. MCP’s truly revolutionary aspect is protocol standardization (unified tool registration/discovery/calling spec), not any vectorization trick.

Practice Layer: Two Working Demos (Slides 34-69)
#

Starting from Slide 34, the style abruptly shifts — all code, terminal output, hardware photos. That entire Transformer vector model from the theory layer completely disappears, replaced by curl, psql, and Perl scripts.

The only thread connecting the two layers is that “they’re both talking about MCP.” But the vector matching mechanism painted in the theory layer and the actual implementation in the practice layer are nearly two different logic systems. This may be exactly the tension Bruce intended — the theory layer helps you understand why MCP is stronger than RAG, and the practice layer tells you how to actually implement it today.

Demo 1: Letting ChatGPT Read a Real-World Geiger Counter
#

Bruce set up a GQ GMC-800 Geiger counter (radiation detector) in his backyard, connected via USB to a Raspberry Pi, taking environmental radiation readings every 15 minutes. First, see ChatGPT using MCP to call real data:

MCP can call external tools to get real-time data — something RAG cannot do.

Connected to hardware:

Wrote a Python wrapper using fastmcp:

from fastmcp import FastMCP

mcp = FastMCP("Geiger counter MCP server")

@mcp.tool
def geiger() -> int:
 """Return the radiation level (CPM) at 13 Roberts Road, Newtown Square, PA, USA"""
 return subprocess.check_output(
 "/var/lib/postgresql/tmp/geiger", shell=True, text=True
 )

The underlying layer is a Perl script that sends <GETCPM>> over serial, reads back a 4-byte CPM value. Apache reverse-proxies port 443 (OpenAI only talks to 443). After registering with ChatGPT:

User: What's the radiation level at 13 Roberts Road?
GPT: I don't have public data for that location...

User: Use my custom app
GPT: [calls geiger tool] → 14 CPM. Normal background radiation (5-25 CPM).

User: Take five readings and give me the average
GPT: [calls ×5] 15 16 13 15 15 → average 14.8 CPM

Two key behaviors:

The LLM can iteratively call tools and compute — RAG is a one-shot data dump, MCP is “call → get result → decide → call again → compute”
The user must explicitly authorize — the first time, ChatGPT didn’t say “I have your Geiger counter data.” Only when the user said “use my custom app” did the tool call trigger. The security model is conservative

Demo 2: Using PG as a Pretzel Shop Inventory System
#

From hardware back to software. Building a pretzel inventory database:

CREATE TABLE pretzel (
 quantity INTEGER CHECK (quantity >= 0)
);
INSERT INTO pretzel VALUES (0); -- initial inventory 0

MCP tools use psql to operate on PG directly:

@mcp.tool
def get_pretzel_inventory() -> int:
 """Return the number of unsold pretzels"""
 return subprocess.check_output(
 "psql --tuples-only -c 'SELECT quantity FROM pretzel;' -d mcp",
 shell=True, text=True
 )

@mcp.tool
def sold_one_pretzel() -> str:
 """Call this when a pretzel is sold; reduces inventory by one"""
 return subprocess.check_output(
 "psql --tuples-only -c 'UPDATE pretzel SET quantity = quantity - 1;' -d mcp",
 shell=True, text=True
 )

@mcp.tool
def baked_6_pretzels() -> str:
 """Call this when a tray of 6 pretzels is baked; increases inventory"""
 return subprocess.check_output(
 "psql --tuples-only -c 'UPDATE pretzel SET quantity = quantity + 6;' -d mcp",
 shell=True, text=True
 )

Interaction flow:

User: How many pretzels available?
GPT: 0 pretzels.

User: I just baked a tray → 6 pretzels
User: I sold two → 4 remaining
User: I sold four → 0 remaining

User: I sold one pretzel → ERROR! CHECK constraint prevented negative quantity

The LLM doesn’t write SQL directly — it calls your predefined, controlled interfaces. PG’s CHECK constraints naturally form a safety net — even if the LLM is tricked into calling the wrong function, the database-level constraint provides a second line of defense.

But this also exposes a problem: the LLM faithfully executed sold_one_pretzel, but didn’t anticipate that “inventory is 0, calling it will error.” MCP is the execution layer, not the reasoning layer.

How Far from Production
#

On the final slide, Bruce frankly admits the current implementation’s limitations:

No authentication — anyone can call your MCP Server
No parameterization — all three tools are parameterless functions; real-world tools need to accept parameters
No security restrictions on dynamic SQL — tool descriptions declare semantics, but the LLM could be injected with malicious content
Connection pooling, transaction management, rate limiting — none addressed

Two recommended practical reads:

Between the Two Layers
#

Looking back at these 70 slides, the most interesting part isn’t any single demo — it’s how the theoretical thinking and hands-on work together explain what MCP can do:

The theory layer uses Transformer vector spaces to explain “how the LLM chooses between words and tools” — this is a teaching model
The practice layer uses psql, curl, and Perl scripts to actually implement things — this is engineering

The real MCP mechanism — tool definitions inserted as text into the system prompt, the LLM using semantic understanding to decide which tool to call, outputting tool call JSON — needs none of the vector embedding model from the theory layer. Between the two layers, Bruce didn’t draw the connecting line. This might not be a bug — it might be a feature.

This article was originally published in Chinese on lastdba.com.

My Blog is Live

Sat, 16 May 2026 00:00:00 +0000

It’s Live!
#

The blog is finally live.

URL: https://lastdba.com

Accessible from China, mobile-friendly too.

76 articles — all PostgreSQL writing from the past few years: case studies, internals, source code analysis, paper deep reads.

This is a proper launch: new framework, new domain, new theme — rebuilt from the ground up.

Highlights
#

Clean Interface

Minimalist, reader-friendly design with a useful search feature.

Framework: Jekyll → Hugo

Version 1: Jekyll + minima theme + 2000 lines of CSS

Version 2: Hugo + Blowfish theme + 0 lines of CSS

V1 was decent, but building the UI myself was exhausting. I remembered vonng had written an article about website architecture choices, so I just went and borrowed from it. I explained the architecture to AI and had it learn from vonng.com — the page quality jumped up a level instantly. A few more tweaks and it was done.

Domain: github.io → lastdba.com

Bought lastdba.com, configured Cloudflare. GitHub Pages with custom domain, free HTTPS certificate, auto-renewal. Now accessible without VPN!

Image Localization

Previously, article images were scattered everywhere — CSDN CDN, GitHub PicBed, Modb OSS. CSDN has hotlink protection. GitHub PicBed on foreign networks often failed to load domestically. This time I had AI consolidate everything to local paths. No more worrying about image hosts going down. Cross-network image loading problems solved — very good.

Reflections on Going Live
#

I’d actually set up a blog URL before — just fork a blog project and deploy via GitHub Pages. The domain was liuzhilong62.github.io/blogs. But being somewhat of a quality freak (not really), the results were mediocre so I took it down. Later I just used the GitHub repo as my blog, without even enabling Pages. Recently, with more free time for various reasons, I revisited this and used Hermes to build the blog from scratch.

As a DBA and backend engineer, I know nothing about frontend stuff like Jekyll, Hugo, Blowfish, CSS. I just give Hermes a target and it does the work. When it explains things to me I don’t understand (and I’m too embarrassed to admit it), I basically just say “keep going.” I check the result in the browser — if I’m satisfied, great; occasionally I say “revert this.”

Honestly, my biggest takeaway from switching to Hugo wasn’t technical — it was “don’t reinvent the wheel.” I’d spent so much time hand-coding dark mode, TOC, search, only to discover a theme swap includes it all, and theirs looks better than mine.

Also, after hooking up lastdba.com, the blog suddenly felt “official.” liuzhilong62.github.io/blogs felt like a personal experiment; now it feels like a real website. Same content, different feeling.

What It Cost
#

All expenses:

Item	Cost
`lastdba.com` domain (Cloudflare, 1 year)	¥70
GitHub Pages hosting	¥0
Hugo framework	¥0
Blowfish theme	¥0
Cloudflare DNS + CDN	¥0
Tokens	¥60
Total	¥130

Possibly the most cost-effective personal website solution out there.

Finally
#

Some details may not be polished — feedback, bug reports, and optimization suggestions welcome.

I’ll likely keep updating.

Reference
#

https://vonng.com/

Original link: https://lastdba.com/2026/05/16/个人博客上线/

Case Study: Startup Failure and SysV Shared Memory

Mon, 09 Mar 2026 00:00:00 +0000

Problem Symptoms
#

The database instance’s RSS memory was maxed out, OOM messages appeared in the logs, and the instance died. We won’t analyze the OOM cause here.

But startup kept failing — 4 or 5 attempts according to the logs:

2026-02-12 09:15:21 CST::@:[578272]: FATAL: pre-existing shared memory block (key 2048, ID 1328250881) is still in use
2026-02-12 09:15:21 CST::@:[578272]: HINT: Terminate any old server processes associated with data directory "/data".
2026-02-12 09:15:21 CST::@:[578272]: LOG: database system is shut down
2026-02-12 09:21:03 CST::@:[658824]: FATAL: pre-existing shared memory block (key 2048, ID 1328250881) is still in use
2026-02-12 09:21:03 CST::@:[658824]: HINT: Terminate any old server processes associated with data directory "/data".
2026-02-12 09:21:03 CST::@:[658824]: LOG: database system is shut down
2026-02-12 09:31:12 CST::@:[794791]: LOG: redirecting log output to logging collector process
2026-02-12 09:31:12 CST::@:[794791]: HINT: Future log output will appear in directory "/data/pg_log".
2026-02-12 09:31:37 CST::@:[801049]: FATAL: lock file "postmaster.pid" already exists
2026-02-12 09:31:37 CST::@:[801049]: HINT: Is another postmaster (PID 794791) running in data directory "/data"?
2026-02-12 09:32:34 CST::@:[814396]: FATAL: lock file "postmaster.pid" already exists
2026-02-12 09:32:34 CST::@:[814396]: HINT: Is another postmaster (PID 794791) running in data directory "/data"?

Startup succeeded after the DBA ran ipcrm -m xxx before starting.

Although the issue was quickly resolved, many questions remained:

Why isn’t this scenario more common in practice?
The start.log shows two different error types — what operations and logic do they correspond to?
Can shared memory still exist even if the postmaster is gone?
How do you locate and clean up this shared memory segment?
PG has multiple shared memory segments — which one is this?
Besides ipcrm -m, are there other ways to get the instance started?

Error Analysis: `pre-existing shared memory block`
#

Three Types of Shared Memory
#

Normally, after PG starts, there are three shared memory segments.

Using the default shared_memory_type='mmap' without huge pages as an example:

## View PG's actual shared memory usage from its virtual memory map
cat /proc/`head -1 $PGDATA/postmaster.pid`/smaps | grep -E "\-s"

2b61b0563000-2b61b0564000 rw-s 00000000 00:04 116293664 /SYSV00001000 (deleted)
2b61b057f000-2b61b05b3000 rw-s 00000000 00:12 1501001168 /dev/shm/PostgreSQL.1193490778
2b61bbac2000-2b61fa67a000 rw-s 00000000 00:04 1500999610 /dev/zero (deleted)

From top to bottom, these are: the SysV shared memory used at startup, shared memory for parallel queries, and shared memory for shared_buffers.

If shared_buffers uses huge pages, or if the shared_memory_type is SysV instead of mmap, the output differs slightly.

Huge pages:

2aaaaac00000-2aba9ca00000 rw-s 00000000 00:0e 48453452 /anon_hugepage (deleted)
2b08f2eea000-2b08f2eeb000 rw-s 00000000 00:04 50692152 /SYSV00001000 (deleted)
2b08f2f05000-2b08f302d000 rw-s 00000000 00:12 48436142 /dev/shm/PostgreSQL.1345689218

shared_memory_type = ‘sysv’:

2b03b3ceb000-2b03b3d1f000 rw-s 00000000 00:12 1572332304 /dev/shm/PostgreSQL.2883611352
2b03bf0c2000-2b03fdc7a000 rw-s 00000000 00:04 143917075 /SYSV00001000 (deleted)

Summary:

PG Shared Memory Config	smaps Segments	shared_buffers smaps	sysv smaps
shared_memory_type=mmap, no huge pages	3 segments	/dev/zero	/SYSV00001000
shared_memory_type=sysv, no huge pages	2 segments	/SYSV00001000	/SYSV00001000
shared_memory_type=mmap, with huge pages	3 segments	/anon_hugepage	/SYSV00001000
shared_memory_type=sysv, with huge pages	not supported	not supported

Now the key question: when the error says pre-existing shared memory block, which shared memory segment is it talking about?

Source Code Analysis
#

Searching for the error message in the source quickly leads to the key location: src/backend/port/sysv_shmem.c

First, understand what the SysV shmem is for. From scattered README content:

We still require a SysV shmem block to
 * exist, though, because mmap'd shmem provides no way to find out how
 * many processes are attached, which we need for interlocking purposes.

 * As of PostgreSQL 9.3, we normally allocate only a very small amount of
 * System V shared memory, and only for the purposes of providing an
 * interlock to protect the data directory. The real shared memory block
 * is allocated using mmap(). This works around the problem that many
 * systems have very low limits on the amount of System V shared memory
 * that can be allocated. Even a limit of a few megabytes will be enough
 * to run many copies of PostgreSQL without needing to adjust system settings.

SysV shmem can determine whether shared memory is still attached; mmap cannot
This SysV shmem is used to protect the data directory; shared_buffers uses mmap (by default), not SysV
This SysV shmem segment is tiny (from the virtual addresses we can see it’s just 4K = 2b61b0563000-2b61b0564000)

Now look at the shm state enum:

typedef enum
{
	SHMSTATE_ANALYSIS_FAILURE,	/* unexpected failure to analyze the ID */
	SHMSTATE_ATTACHED,			/* pertinent to DataDir, has attached PIDs */
	SHMSTATE_ENOENT,			/* no segment of that ID */
	SHMSTATE_FOREIGN,			/* exists, but not pertinent to DataDir */
	SHMSTATE_UNATTACHED			/* pertinent to DataDir, no attached PIDs */
} IpcMemoryState;

The key states are ATTACHED, FOREIGN, and UNATTACHED.

The SysV shmem protects the data directory — the common scenario is ensuring the directory isn’t running two instances. Since it’s shared memory, weird scenarios could mean the segment doesn’t belong to this directory or this process (FOREIGN state). If the shared memory corresponds to the data directory but no processes are running, it should be UNATTACHED. With processes running, it’s ATTACHED.

Now look at the error thrown by PGSharedMemoryCreate:

PGShmemHeader *
PGSharedMemoryCreate(Size size,
					 PGShmemHeader **shim)
{...
 for (;;) // infinite loop
	{..
 shmid = shmget(NextShmemSegID, sizeof(PGShmemHeader), 0);// shmget to fetch the SysV shmem and return its shmid
		if (shmid < 0)
		{
			oldhdr = NULL;
			state = SHMSTATE_FOREIGN;
		}
		else
			state = PGSharedMemoryAttach(shmid, NULL, &oldhdr);// determine this shmem segment's state

 switch (state)// take different actions based on the shared memory state
		{
 ...// only showing 2 states here: attached and unattached
			case SHMSTATE_ATTACHED: // shm is attached — throw the error (this is the fault symptom we saw)
				ereport(FATAL,
						(errcode(ERRCODE_LOCK_FILE_EXISTS),
						 errmsg("pre-existing shared memory block (key %lu, ID %lu) is still in use",
								(unsigned long) NextShmemSegID,
								(unsigned long) shmid),
						 errhint("Terminate any old server processes associated with data directory \"%s\".",
								 DataDir)));
				break;
 ...
			case SHMSTATE_UNATTACHED:// shm is unattached

				/*
				 * The segment pertains to DataDir, and every process that had
				 * used it has died or detached. Zap it, if possible, and any
				 * associated dynamic shared memory segments, as well. This
				 * shouldn't fail, but if it does, assume the segment belongs
				 * to someone else after all, and try the next candidate.
				 * Otherwise, try again to create the segment. That may fail
				 * if some other process creates the same shmem key before we
				 * do, in which case we'll try the next key.
				 */
 // The segment belongs to the data directory, and no process still holds it
				if (oldhdr->dsm_control != 0)
					dsm_cleanup_using_control_segment(oldhdr->dsm_control);
				if (shmctl(shmid, IPC_RMID, NULL) < 0)
					NextShmemSegID++; // Note: ShmemSegID increments and retries
				break;
		}
 ...
 }
 ...
 }

When shmem is ATTACHED, it throws the error. When unattached, it loops infinitely, trying to clean up the segment and incrementing ShmemSegID to request a new one.

The first case corresponds to this fault
The second case corresponds to normal crash recovery (instance can still start after a crash)

SysV shmem
#

From PG10 onwards, the postmaster.pid and SysV shmem logic was significantly reworked and has been largely stable since. This article only covers the PG10+ logic.

pidfile.h:

#define LOCK_FILE_LINE_SHMEM_KEY	7

sysv_shmem.c, InternalIpcMemoryCreate():

	{
		char		line[64];

		sprintf(line, "%9lu %9lu",
				(unsigned long) memKey, (unsigned long) shmid);
		AddToDataDirLockFile(LOCK_FILE_LINE_SHMEM_KEY, line);
	}

From the source code, shmem info is saved on line 7 of postmaster.pid, containing the shmkey and shmid.

> cat postmaster.pid
242712
/data
1772698474
8531
/tmp
0.0.0.0
 4096 143917078 # <----here
ready

What Are shmkey and shmid?
#

In PG’s source, the call path is: InternalIpcMemoryCreate():

			shmid = shmget(memKey, 0, IPC_CREAT | IPC_EXCL | IPCProtection);

PG uses shmkey/memkey as a seed key to request shared memory from the kernel, which returns a unique identifier, shmid.

shmid is highly dependent on the server or rather the server’s memory state. For PG, when quickly restarting an instance, the shmid may be the same or +1 — this depends on Linux kernel internals. After a full server reboot, it’ll be completely different.

To aid understanding: whether the server reboots or not, shmkey/memkey can remain constant (since it’s user/PG input). But across a server reboot, even with the same shmkey, the returned shmid is very unlikely to be the same value.

How PG Obtains the shmkey
#

PGSharedMemoryCreate():

	/*
	 * We use the data directory's ID info (inode and device numbers) to
	 * positively identify shmem segments associated with this data dir, and
	 * also as seeds for searching for a free shmem key.
	 */
	if (stat(DataDir, &statbuf) < 0)
		ereport(FATAL,
				(errcode_for_file_access(),
				 errmsg("could not stat data directory \"%s\": %m",
						DataDir)));
...
	/*
	 * Loop till we find a free IPC key. Trust CreateDataDirLockFile() to
	 * ensure no more than one postmaster per data directory can enter this
	 * loop simultaneously. (CreateDataDirLockFile() does not entirely ensure
	 * that, but prefer fixing it over coping here.)
	 */
	NextShmemSegID = statbuf.st_ino;

	for (;;)
	{
		IpcMemoryId shmid;
		PGShmemHeader *oldhdr;
		IpcMemoryState state;

		/* Try to create new segment */
		memAddress = InternalIpcMemoryCreate(NextShmemSegID, sysvsize);
		if (memAddress)
			break;				/* successful create and attach */

		/* Check shared memory and possibly remove and recreate */

		/*
		 * shmget() failure is typically EACCES, hence SHMSTATE_FOREIGN.
		 * ENOENT, a narrow possibility, implies SHMSTATE_ENOENT, but one can
		 * safely treat SHMSTATE_ENOENT like SHMSTATE_FOREIGN.
		 */
		shmid = shmget(NextShmemSegID, sizeof(PGShmemHeader), 0);

PG calls stat() on the data directory, which returns the directory’s inode. PG directly uses datadir.inode as the shmkey.

In PG, the shmem key is tightly coupled to the data directory’s inode. Under normal circumstances, shmem key = datadir inode.

Verification example:

> ls -id $PGDATA
4096 /lzlcloud/pg8574/data
> cat postmaster.pid |head -7|tail -1
 4096 143917090

We can see datadir.inode = shmkey = 4096.

PG shmkey in Cloud Environments
#

Above I said generally shmkey = datadir.inode, but in cloud environments this is typically not the case.

Our cloud environment:

> ls -id /lzlcloud/pg8298/data
4096 /lzlcloud/pg8298/data
> ls -id /lzlcloud/pg8388/data
4096 /lzlcloud/pg8388/data
> ls -id /lzlcloud/pg8095/data
4096 /lzlcloud/pg8095/data

> cat /lzlcloud/pg8298/data/postmaster.pid|head -7|tail -1
 4096 971833391
> cat /lzlcloud/pg8388/data/postmaster.pid|head -7|tail -1
 4097 62128161
> cat /lzlcloud/pg8095/data/postmaster.pid|head -7|tail -1
 4098 143163441

The data disk directories all have inode 4096, but the shmkeys are 4096, 4097, 4098.

Why?

The inode issue relates to the filesystem:

Each filesystem has independent inodes
The filesystem reserves some inodes — the first few are unusable. Depending on mount options, our data disk’s real inodes start at 4096

So datadir.inode = 4096 is the default behavior of our cloud environment’s disk mounts. Other environments may differ — I haven’t analyzed those deeply. But with the same filesystem and mount approach for PG data directories, inode collisions are still possible.

The shmkey issue relates to PG’s source code, PGSharedMemoryCreate():

	for (;;)
	{
 ...
 NextShmemSegID = statbuf.st_ino;
 ...
		shmid = shmget(NextShmemSegID, sizeof(PGShmemHeader), 0);
		...
		switch (state)
		{
			case SHMSTATE_FOREIGN:
				NextShmemSegID++;
				break;

The initial shmkey = datadir.inode, but since the requested shmem might be FOREIGN (used by another process), PG increments shmkey by 1 and tries again.

For example, the instance with shmkey=4097 in postmaster.pid: at startup it tried shmkey=4096, but found that shmid’s memory segment was already in use by another instance (the one with shmkey=4096). So it used shmkey+1 to request a different shmid segment.

Similarly, the instance with shmkey=4098 had to increment twice to find a free shmkey-shmid pair.

shmid Relationships
#

The SysV shmid can be found in the startup error log, line 7 of postmaster.pid, and virtual memory smaps. It can be inspected via the ipcs command and cleaned up with ipcrm.

Example — note shmid=143917078 throughout:

Startup error log:

pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 16:02:19 CST::@:[262388]: FATAL: pre-existing shared memory block (key 4096, ID 143917078) is still in use

postmaster.pid line 7:

> cat postmaster.pid |head -7|tail -1
 4096 143917078

Virtual memory smaps:

cat /proc/`head -1 $PGDATA/postmaster.pid`/smaps | grep -E "\-s"
2ad2b5189000-2ad2b518a000 rw-s 00000000 00:04 143917078 /SYSV00001000 (deleted)

Inspecting and cleaning via SysV shmid:

ipcs -m -i 143917078 # cleanup: ipcrm -m shmid

Shared memory Segment shmid=143917078
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=242712 cpid=242712 nattch=10
att_time=Thu Mar 5 16:14:51 2026
det_time=Thu Mar 5 16:14:49 2026
change_time=Thu Mar 5 16:14:34 2026

Testing
#

Reproducing the Production Issue
#

Hold a backend process alive indefinitely, then kill -9 the postmaster:

> cat postmaster.pid
 4096 143917076

> ipcs -m -i 143917076 # shmem id
Shared memory Segment shmid=143917076
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=241567 cpid=64757 nattch=23

> kill -stop 107648 # any backend

> kill -9 64757 # postmaster or another process

> ipcs -m -i 143917076
Shared memory Segment shmid=143917076
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=252283 cpid=64757 nattch=1 # nattch != 0

> pg_ctl start -D $PGDATA
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 16:02:19 CST::@:[262388]: FATAL: pre-existing shared memory block (key 4096, ID 143917076) is still in use
2026-03-05 16:02:19 CST::@:[262388]: HINT: Terminate any old server processes associated with data directory "/data".
 stopped waiting
pg_ctl: could not start server

nattch=1 — the instance cannot start.

Normal Crash Recovery (Successful Startup)
#

Essentially, kill the instance and then start it:

> cat postmaster.pid
 4096 143917077

> ipcs -m -i 143917077 # shmem id
Shared memory Segment shmid=143917077
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=154800 cpid=134329 nattch=18

> kill -9 134329 # postmaster or another process

> cat postmaster.pid
 4096 143917077

> ipcs -m -i 143917077 # shmem id unchanged, segment still exists
Shared memory Segment shmid=143917077
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=169360 cpid=134329 nattch=0 # nattch=0

> ipcs -m -i 143917077 # shmem id unchanged, segment still exists

> pg_ctl start -D $PGDATA # startup succeeds
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 16:14:34 CST::@:[242712]: LOG: redirecting log output to logging collector process
2026-03-05 16:14:34 CST::@:[242712]: HINT: Future log output will appear in directory "/data/pg_log".
 done
server started

> ipcs -m -i 143917077 # residual shmem cleaned up during startup
ipcs: id 143917077 not found
> ipcs -m -i 143917078 # shmid incremented by 1 at startup
Shared memory Segment shmid=143917078
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=273571 cpid=242712 nattch=26

> cat postmaster.pid # shmkey unchanged, shmid +1
 4096 143917078

A normal kill -9 followed by startup works fine — the residual shmem is cleaned up during startup. shmkey stays the same because inode=4096 and shmkey=4096 wasn’t occupied. shmid+1 is Linux kernel behavior, at least indicating a different shared memory segment was used.

Holding a File Descriptor But Not shmem
#

Since startup is tied to the data directory inode, and inode is tied to shmem id, startup essentially checks whether the shmem is held by another process, not whether a file descriptor is still open. So let’s test with the logger process, which holds file descriptors but not shared memory:

$ cat /proc/77300/smaps | grep -E "\-s" # logger process — verify it has no shared memory
$ kill -stop 77300 # stop logger
$ kill -9 77076 # kill -9 pm
$ cat postmaster.pid # file still exists
77076
/lzlcloud/pg8531/data
1772700343
8531
/tmp
0.0.0.0
 4096 143917080
ready
$ ipcs -m -i 143917080 # shared memory still exists

Shared memory Segment shmid=143917080
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=77319 cpid=77076 nattch=0
att_time=Thu Mar 5 17:27:11 2026
det_time=Thu Mar 5 17:27:15 2026
change_time=Thu Mar 5 16:45:43 2026

$ ps -ef|grep 77300 # process still alive
postgres 77300 1 0 16:45 ? 00:00:00 postgresql: lzldb: logger
postgres 135246 46622 0 17:27 pts/1 00:00:00 grep --color=auto 77300
$ pg_ctl start -D $PGDATA # startup succeeds
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 17:27:55 CST::@:[140497]: LOG: redirecting log output to logging collector process
2026-03-05 17:27:55 CST::@:[140497]: HINT: Future log output will appear in directory "/data/pg_log".
 done
server started

The logger holds files in the data directory but is not associated with shared memory — it does not block startup.

Deleting postmaster.pid Then Failing to Start
#

Same procedure: hold a backend process, kill -9 the PM, delete postmaster.pid, attempt startup.

I’ll skip the full output — result: startup fails with:

waiting for server to start....2026-03-06 15:29:48 CST::@:[22475]: FATAL: pre-existing shared memory block (key 4098, ID 171868173) is still in use
2026-03-06 15:29:48 CST::@:[22475]: HINT: Terminate any old server processes associated with data directory "/data".
2026-03-06 15:29:48 CST::@:[22475]: LOG: database system is shut down

This shows: even with a zombie process holding shmem, deleting the postmaster.pid (which contains the shmid) doesn’t stop PG from finding the corresponding shmid.

Stop a Different Instance, Start the Current One
#

PG analyzes shmid from two sources to determine if it belongs to the current instance:

The shmid corresponding to datadir.inode as shmkey, or after shmkey++
The shmid stored in postmaster.pid

Even if postmaster.pid is deleted, PG can still tell whether shmem is held by another process. But we can exploit datadir.inode and shmkey++ behavior to get it started.

Since in our cloud environment all data directory inodes are 4096, and shmkeys differ due to the shmkey++ source logic, we can: start or stop a PG instance whose datadir.inode = 4096 to shift the current instance’s shmkey++ by one, obtaining a different shmid.

$ kill -stop 165245
$ kill -9 164411 # stop current instance, keep one of its backend processes alive

$ pg_ctl stop -D /pg8531/data # stop a different instance
waiting for server to shut down.... done
server stopped
$ pg_ctl start -D /pg8574/data # try starting the current instance — fails because postmaster.pid still exists
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-05 18:22:35 CST::@:[196209]: FATAL: pre-existing shared memory block (key 4097, ID 143917087) is still in use
2026-03-05 18:22:35 CST::@:[196209]: HINT: Terminate any old server processes associated with data directory "/pg8574/data".
 stopped waiting
pg_ctl: could not start server
Examine the log output.

$ mv /lzlcloud/pg8574/data/postmaster.pid{,.bak} # delete current instance's postmaster.pid
$ pg_ctl start -D /lzlcloud/pg8574/data # try again — succeeds
2026-03-05 18:23:09 CST::@:[207725]: LOG: redirecting log output to logging collector process
2026-03-05 18:23:09 CST::@:[207725]: HINT: Future log output will appear in directory "/lzlcloud/pg8574/data/pg_log".
 done
server started

$ ipcs -m -i 143917087 # the shmid's SysV segment is still held by our zombie process

Shared memory Segment shmid=143917087
uid=6001 gid=6001 cuid=6001 cgid=6001
mode=0600 access_perms=0600
bytes=56 lpid=196209 cpid=164411 nattch=1
att_time=Thu Mar 5 18:22:35 2026
det_time=Thu Mar 5 18:22:35 2026
change_time=Thu Mar 5 18:21:04 2026

Startup succeeds — the current instance requested a different shared memory segment. The old segment wasn’t cleaned up. This is the “hack” of stopping another instance to start the current one in a cloud environment.

A small prerequisite: the other instance must have not only inode = current instance inode, but also shmkey < current instance shmkey.

Error Analysis: `lock file "postmaster.pid" already exists`
#

This problem is much simpler than the shared memory one.

During startup, PG checks the lock file and its contained PID, in CreateLockFile():

		if (other_pid != my_pid && other_pid != my_p_pid &&
			other_pid != my_gp_pid)
		{
			if (kill(other_pid, 0) == 0 ||
				(errno != ESRCH && errno != EPERM))
			{
				/* lockfile belongs to a live process */
				ereport(FATAL,
						(errcode(ERRCODE_LOCK_FILE_EXISTS),
						 errmsg("lock file \"%s\" already exists",
								filename),
						 isDDLock ?
						 (encoded_pid < 0 ?
						 errhint("Is another postgres (PID %d) running in data directory \"%s\"?",
								 (int) other_pid, refName) :
						 errhint("Is another postmaster (PID %d) running in data directory \"%s\"?",
								 (int) other_pid, refName)) :
						 (encoded_pid < 0 ?
						 errhint("Is another postgres (PID %d) using socket file \"%s\"?",
								 (int) other_pid, refName) :
						 errhint("Is another postmaster (PID %d) using socket file \"%s\"?",
								 (int) other_pid, refName))));
			}
		}

Testing is even simpler — just start it a second time while already running:

$ pg_ctl start -D /pg8531/data
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2026-03-06 15:59:05 CST::@:[89145]: FATAL: lock file "postmaster.pid" already exists
2026-03-06 15:59:05 CST::@:[89145]: HINT: Is another postmaster (PID 255500) running in data directory "/pg8531/data"?
 stopped waiting
pg_ctl: could not start server
Examine the log output.

So the later errors in the fault’s start.log were because the instance was already running and someone tried starting it multiple more times.

Summary
#

When starting, PG first allocates a SysV shmem segment (not the mmap-based shared_buffers) to lock the data directory. The lock is obtained by using the data directory’s inode as the shmkey passed to shmget(), which returns a unique shmid. Since the requested shmem may already be in use by another process, PG increments shmkey++ in an infinite loop until it finds an unclaimed segment. postmaster.pid line 7 stores both the shmkey and shmid. In cloud environments, you’ll often see adjacent PG instances with incrementing shmkeys — this happens because the data disks are mounted identically and share the same starting inode, causing shmkey++ to kick in.

If a PG instance is killed unexpectedly, the shmem is not automatically cleaned up. Under normal conditions, no zombie process holds the shared memory, so startup cleans it up and proceeds normally. Under abnormal conditions, a zombie process still holds the shared memory — startup fails and manual intervention is required.

Recommended handling:

ipcrm -m (most recommended)
Use lsof to find the zombie process and kill it
Reboot the host

Not recommended but possible workarounds:

mv postmaster.pid + stop a different PG instance (where the other instance’s shmkey < current instance’s shmkey)
mv postmaster.pid + remount the data disk to change its inode

Finally, answering the opening questions:

Why isn’t this scenario more common in practice?

Abnormal instance crash + zombie processes still alive. Many crash scenarios leave no zombie processes, so startup just works.

The start.log shows two different error types — what do they correspond to?

The “shared memory in use” error means abnormal crash + zombie processes still exist. The “postmaster.pid already exists” error means the instance was started multiple times.

Can shared memory still exist if the postmaster is gone?

Yes, shared memory can persist when the postmaster is gone — PG processes don’t always cleanly exit or get cleaned up by the OS. However, if all processes are gone, the shared memory should not exist.

How do you locate and clean up this shared memory segment?

The shmid can be found in the startup error log (start.log). Clean it with ipcrm -m $shmid.

PG has multiple shared memory segments — which one is this?

The SysV shmem used to protect the data directory. It always exists. See the “Three Types of Shared Memory” section. It’s distinct from the mmap-based shared_buffers.

Can you find the corresponding shmem via inode or file?

Linux does not provide a userspace interface to find SysV shmem by inode or file (this statement is 100% AI-generated, cross-validated across multiple models). PG uses the data directory’s inode as a seed shmkey to request shared memory — it does not directly find shmem by inode. PG has its own mechanism for locating SysV shmem, but it’s not an absolute mapping; shmkey++ is a compromise startup logic for this reason.

DBA, Writing, Learning and the Future in the AI Era

Wed, 21 Jan 2026 00:00:00 +0000

AI rate: This article has approximately 60% AI involvement, with about 20 rounds of battling with AI

Recommendation reason: Contains some reflections and insights on AI Ops, hence recommended

Writing in the AI Era
#

For authors who write blogs or WeChat public accounts, AI may be a fatal blow, because AI writing is simply too easy. As someone who writes articles myself, I have many internal struggles about how AI affects writing habits, and it pains me too. Let me revisit some earlier thoughts on writing:

Why write?

For myself: To consolidate knowledge. Output is what strengthens input. Glancing at something once versus writing it out are completely different experiences — writing can take several times longer than just reading. For example, when you see a profound and seemingly familiar sentence, rewriting it yourself reveals countless details within it.
For myself: To leverage others’ biases constructively. Mainly to use readers’ expectations as motivation to persist in writing and to enhance the credibility of content. Knowledge you consume yourself may be “good enough,” but writing for a public audience forces you to weigh every word and take responsibility for others. (Relatively speaking — not actual word-by-word scrutiny.)
For myself: To build reputation. This depends heavily on writing quality.
For others/the community: To spread knowledge. Good things should be shared and used by everyone — this is at the core of the PostgreSQL open-source community. Encouraging sharing, not hoarding, is a principle I’ve always upheld.
Building connections: This wasn’t my goal, but I have indeed met some friends through it.

Human writing was already difficult; in the AI era, human writing is essentially Hell Mode — like walking against the current without a destination, unable to see any light, while everyone else is heading the opposite direction. I’ve certainly experienced AI-powered interpretation, translation, and article generation, but it never feels like mine, or it loses the original purpose of training myself. Or, at a deeper level, I want to feel the vitality of the work.

The DBA community’s articles can be described as a mixed bag — people write about everything. I’ve always preferred substantive, content-rich articles focused on PostgreSQL internals and operations, like those by Cancan and Xiangbo — I eagerly anticipate every piece and read them carefully. Generally, content-oriented articles don’t get much traffic (both Cancan and Xiangbo have complained about this on their public accounts…), and I’m quite easygoing about it myself.

However, my previous article “PG Operations Database Operations Experience 2025” gained a surprising number of followers, which truly astonished me. So I’ve been pondering this question for days: Why would a non-AI-written, non-comprehensive, DBA-focused, knowledge-oriented article attract so much interest? What does AI mean for DBAs?

Reflections on Operations
#

The Essence of Operations and AI Ops
#

Operations involve many things. To narrow the scope of discussion, I’ll focus on just one small part of operations work — incident response — to interpret the essence of DB Ops. First, my position: “Operations is not merely a technical problem.”

Many people argue that since both humans and AI make mistakes, AI can be given authority to act boldly — specifically, if AI’s error rate ≤ human error rate, replacement is justified. I thought the same two years ago, but I no longer do. Because the real-world environment is far more complex, with at least the following factors to consider:

The consensus problem. There is consensus that a DBA might accidentally delete data, but another consensus is easily overlooked: in normal circumstances, the team assumes the DBA won’t delete data. How to understand this? For example, when hiring a DBA, a responsible team will assess whether the person is mentally stable, then default to assuming they won’t delete data, and maintain this assumption throughout long-term work. At the very least, I don’t constantly worry that my colleague will drop the database. But when “hiring” an AI DBA, it has no mental state, and no one assumes it won’t delete data. “It will delete data” is everyone’s consensus, creating deployment resistance.
The importance of data. C-end (consumer) data and B-end (business) data have different importance levels. Retail, internet, government, and financial industry data also differ in criticality. The more an industry values data, the more sensitive it is to data reliability and business continuity. A personal computer has no business continuity and only one person cares about data reliability, but in the financial industry, business continuity can directly trigger widespread social concern — financial data reliability simply cannot be questionable. AI Ops deployment must consider system criticality; it cannot be rolled out across all domains simultaneously.
The management system. For example, in financial systems, DBAs hold high privileges and are governed by a set of management procedures. So shouldn’t an AI DBA also have corresponding management procedures before it can be deployed? What about abnormal login detection, or abnormal backend access? How does it request permissions, and for how long? What level of permission in what scenario? These are all unresolved issues.
AI’s own security. For instance, the paper STRATUS mentions prompt injection attacks, for which there is currently no effective solution. If someone injects a “drop database” prompt, it might just execute it. But humans basically don’t have this problem — if you tell a DBA “drop database,” they’ll just ask you what you’re trying to do.
The responsibility problem. Operations engineering is not a “knowledge problem” but a “responsibility problem.” One of the core tasks of operations is to make irreversible decisions about the system within limited time during an incident, and take responsibility for those actions. AI can replace “formalizable operations” but cannot replace “judgments that must bear consequences” — at least not yet.
Full of noise. Operations is an “open system,” not a closed reasoning system. Databases run in extremely complex environments, while AI’s reasoning premise is that the world can be described in text. But the real operations world is filled with noise, contingency, and undocumented behaviors.
Situational pressure. Real business environments include recovery time pressure, organizational and customer emotional management, etc. The book Google SRE describes a common recovery scenario: customers asking when it will be restored, leadership asking why failover hasn’t happened, engineers gathering various information under pressure while calling people to confirm recovery procedures. AI cannot feel this pressure. The first two questions are fundamentally not technical problems, but they must be answered. In real scenarios, the answers at that moment are likely to be rough at best.

Let’s imagine what conditions would be needed for fully automated AI operations to truly happen:

AI won’t destroy critical data — at least, the vast majority of people need to reach this consensus about AI.
Complete management procedures are needed, including how to grant AI permissions, just like how we grant DBA permissions.
Solve the problem of AI itself being attacked. Not just LLMs, but the entire IT system encompassing AI.
A no-blame operations culture (or eliminating operations altogether is another approach).
Accept erroneous judgments. Form consensus around the existence of noise and environment, and tolerate AI Ops iteration cycles.
If recovery takes too long or the blast radius expands, don’t allow human intervention — because if human intervention is required, that person is still the operator (semi-automated AI Ops?).
Pressure-free recovery context. This means leaders, customers, and public opinion don’t need responses, or they trust some AI’s response. This is a human transformation, not an IT system transformation.

AIOps and Agent Research Results
#

The Tsinghua AIDB repository’s directory contains many AI4DB papers — too many for a person to read. I used NotebookLM to summarize the paper categories:

Again, to narrow the scope (mainly to reduce my own effort), let’s focus on database diagnostics content.

AIOps has made decent academic progress. AIOps research integrates machine learning, reinforcement learning, and large language models into database management, covering key tasks such as parameter tuning, index recommendations, query optimization, and fault diagnosis. The goal is to build “self-driving” database systems with self-awareness and self-healing capabilities. While significantly improving complex workload performance and operational efficiency, this also drives the DBA’s transformation from low-efficiency manual intervention to high-level architectural supervision.

Regarding whether “DBAs will be eliminated,” current research trends and industry practices (especially self-driving databases and LLM applications) show that the DBA role is undergoing a profound transformation from “manual operator” to “senior manager/supervisor,” rather than simple replacement. The DBA’s core value will shift toward managing AI operations strategies, ensuring data security and compliance, and handling extreme anomaly scenarios that AI cannot resolve.

Another AI Ops Frontier Survey article describes Agents this way:

“This shows that AI Agents are not a silver bullet. To apply Agents, we need not only progress at the model and agent level, but also sufficient support capabilities from the entire operational system — such as Kubernetes-like declarative interfaces, good observability, and reversible operation design. Stratus’s preliminary experiments demonstrate the potential of Agents in automated operations, but there remain enormous gaps in performance, reliability, and security before production deployment.”

The development domain, fueled by the booming vibe coding movement, is clearly advancing much faster than AI in operations. I’d also love to have a confirm/redo operations remote control — the problem is, it doesn’t exist yet. Even if we fantasize about “vibe maintaining” one day, I doubt many ops people would turn on yolo mode.

The Value of a DBA
#

Is a DBA’s Value Just Being the Decision-Maker and Scapegoat?
#

Endorsement indeed seems to be something AI cannot solve. So is the DBA’s value just being the decision-maker and scapegoat? After all, a DBA’s knowledge is far less than AI’s — it’s just that AI can’t make the final call.

1. Instantaneous Context

“The DBA’s knowledge is far less than AI’s” — this is true for general knowledge (like how to optimize a SQL query, or the meaning of a configuration parameter). But AI lacks instantaneous runtime context. AI knows database principles, but it doesn’t know the accumulated historical debt hiding behind the load balancer during the sudden traffic spike of your company’s Double Eleven (Singles’ Day). The DBA possesses unstructured experience about “this specific machine, this specific business, these specific people.” In the face of extreme failures, AI offers the “highest-probability suggestion,” while the DBA offers “the operation that best preserves the system’s life under this specific pressure.”

2. The Last Gate of a Chaotic System

The database is the most fragile and least fault-tolerant part of all IT architectures (code can be rolled back, but data loss can bankrupt a company). AI’s logic is extrapolation based on historical data. When encountering unprecedented underlying hardware bad sectors, extremely rare distributed deadlocks, or novel hacker attack methods, AI’s “suggestions” often fail or even cause secondary damage. The core of “making the call” is not “which solution to choose,” but “hedging against risk.” This kind of control over extreme situations is something current AI cannot provide.

3. Chain of Trust

The DBA is the maintainer of the chain of trust: for example, if you let AI audit AI, then who audits the AI’s audit logic? At the levels of data security, compliance, and ethics, there must be a human with the highest privileges who can be held accountable as the endpoint of the trust chain.

Let’s flip the perspective: if DBAs really were just “less knowledgeable decision-makers and scapegoats,” then enterprises would have long ago transferred DBA decision-making authority to SREs, architecture committees, or even AI and other responsible entities. But the reality is, at truly critical moments, enterprises still call “that person.” This shows the question was never “who is smarter,” but who can bear the consequences for the organization amid uncertainty. The DBA is the last human in this chaotic database system who holds the authority to stop losses, the responsibility, and the terminal point of trust.

So is every decision made by the DBA? Obviously not. The DBA does not hold “objective decision-making authority” but rather “risk veto power” — they cannot decide whether the business should take risks, but they can determine which risks the system cannot bear. In simple, low-risk, rollback-able scenarios, decisions are often made automatically by processes or systems; only when decisions enter high-risk, irreversible territory where responsibility must converge is the DBA pushed to the forefront.

The Uniqueness of the Postgres DBA
#

For the specific group of Postgres (PG) DBAs, this uniqueness is even more pronounced.

In modern technical organizations, DBAs do not naturally hold architectural decision-making authority, nor do they monopolize index or parameter formulation. Architects can design solutions, developers can write SQL, and AI can even provide seemingly comprehensive best-practice recommendations. But these decisions mostly occur at the abstraction layer, design layer, and probability layer — they assume the system is rollback-able, replay-able, and correctable.

Postgres’s uniqueness lies in the fact that it hands a great deal of freedom to its users, and these freedoms ultimately translate into long-term side effects in real systems: write amplification, I/O pattern changes, Vacuum imbalance, WAL bloat, and unpredictable performance degradation. These side effects cannot be fully rehearsed at the design stage, cannot be subcontracted to a single role, and cannot simply be “withdrawn” after an incident occurs. When the system enters an unstoppable, unreplayable state, the only person still responsible for the overall outcome is often the DBA.

Therefore, the value of a Postgres DBA lies not in “making decisions for others” (though you certainly can), but in continuously managing the real-world consequences of all decisions after they have already been made. “Architects define the ideal, developers implement functionality, AI predicts the future; and the DBA guards reality.”

This ability to guard reality is based on the PG DBA having sufficient understanding of Postgres, sufficient understanding of the system’s real environment, sufficient understanding of the system’s history, and sufficient immediate context. In the AI era, one more thing needs to be added: sufficient understanding of AI.

Why Keep Learning
#

In the past two years, I’ve heard “learning is useless” rhetoric more than ever before. I generally scoff at such talk. Let me take this opportunity to properly address it.

Does foundational database knowledge still have value? The answer is: its value is higher than ever. Let’s interpret this from three angles: the right to explain, active learning, and why I keep revisiting the classics.

1. The Right to Explain

Foundational knowledge enables three things:

Identifying “systemic inevitable failure points” in advance
Clearly articulating the judgment logic
Transforming “I’m going with my gut” into “this is the system-determined outcome”

The true meaning of learning database fundamentals is not to “do more work,” but to:

Delineate responsibility boundaries
Enhance discourse power
Let the system endorse your judgments

2. Active Learning Becomes an Even Rarer Ability

In the AI era, the “technical barrier” to knowledge acquisition approaches zero. Active learning ability is scarce. Why is “active learning” even rarer in the AI era? This is counter-intuitive but very real. AI makes “passive learning” extremely comfortable — ask and answer anytime, no long-term investment required, no need to endure cognitive discomfort. But the result is that more and more people stay in the “instant gratification layer,” unwilling to learn foundational knowledge anymore. When everyone else is regressing, you find yourself advancing faster.

3. Why Do I Keep Re-reading Classics like The Internals of PostgreSQL?

Technical people need to read books because books don’t just give answers — they help build a cognitive model that can be run repeatedly and continuously refined. AI is currently better at answering questions rather than shaping such models. AI struggles to become this kind of “long-term dialogue partner” — its answers are unstable.

From another perspective, looking at the value of books through cognitive economics + information theory + token cost: First, you don’t need to battle with AI back and forth. The real cost of battling is not money, but your attention and context-maintenance ability. Second, the hundreds of thousands of words in a book require neither massive prompt input nor excessive token expenditure from you. Third, the knowledge in books has been repeatedly verified by authors and readers — it is already compressed knowledge, the easiest to learn. So: Classic books = extremely low token cost to obtain high-density, human-repeatedly-verified, focused knowledge in compressed form.

4. Learning AI Itself

This needs no elaboration from me.

My battles with AI led me to an interesting conclusion about “why read books”:

In the AI era, knowledge is cheap, but judgment is expensive => And judgment comes from a stable, calibratable cognitive model => A stable cognitive model is itself a byproduct of “long-term high-quality knowledge intake.”

Another real-world piece of evidence supporting the “reading is useful” argument: this very article depends on books, papers, other articles, and information I’ve read. Without that foundation, this article would not exist.

Why People Still Love Reading “Human-Written” Articles
#

The Psychology of Preferring Imperfection
#

Human-written technical articles are inevitably riddled with flaws. Looking back at my own “Operations Experience 2024” from last year, I can find many holes. Even “Operations Experience 2025,” completed just days ago, I consider incomplete — vastly different from something AI would write. So why do readers still enjoy such flawed technical articles?

The reason may be that humans are not attracted by “information correctness,” but by “empathetically imperfect traces of a human mind.” From the book A Brief History of Intelligence, we know that the human intelligence model inherently includes self-trial-and-error exploration and observing others’ behaviors to map onto oneself — this is a learning process, innate to humans. Our brains automatically scan text for hesitation, uncertainty, logical gaps, awkward expressions, emotional leakage, etc. — all things systematically absent from AI text. In the imperfections and emotions of writing, readers can feel the author’s thinking and emotions, whereas AI merely presents results. Readers almost never have emotional resonance with AI. Generally speaking, only those who have truly experienced something leave these “unattractive” traces.

So I believe many people, like me, can identify purely AI-written technical articles at a glance (not guaranteed 100% accurate) and generally won’t have the emotional drive to read through them. But if it’s something a human has seriously written, they’ll read carefully, feeling the author’s feelings, catching their shortcomings or contextual gaps.

Of course, authors could feed prompts to mimic their previous writing style or deliberately leave flaws. But I haven’t seriously tried this — I briefly generated a few pieces and felt the emotional immersion was still quite poor. I don’t plan to explore this further; there’s not much point.

Borrowing an Expert’s ATTENTION for Free
#

The core difference between AI articles and expert articles is not “how well they’re written,” but a matter of economic questioning and industry-leading Attention. Expert writing is about allocating attention on behalf of the reader; AI writing is about avoiding missing any potentially relevant information. This is not a capability issue — it’s a difference in objective functions. Truly high-value technical articles don’t tell you all the correct answers — they block 80% of what you shouldn’t be paying attention to right now.

Why do experts dare to “delete,” while AI doesn’t? Because they bear cognitive responsibility for your understanding outcomes. AI does not bear the consequences of you learning or applying things wrong. So experts deliberately filter out details that don’t need attention right now. This filtering is itself the value of expertise. For humans, the bottleneck in learning is not insufficient information, but limited attention and not knowing where to look first. An expert’s article directly hands you the result and says: just focus on this. But when facing an LLM, do you know what to look for?

This is not a denial of the value of AI articles. AI excels at “rapidly expanding the information space when you already know the problem boundaries,” while expert articles excel at “contracting the problem space for you before you’ve established a judgment framework.” The former is good for filling gaps and lateral expansion; the latter is good for building core understanding and key intuition. The truly efficient learning approach is not choosing one over the other, but first using experts to achieve Attention alignment, then using AI to do amplified search within the bounded space.

This isn’t saying AI content is useless or human-written content is useless — it’s that each has its own use.

Can AGI Solve All Problems?
#

Refuting Musk
#

Recently Musk has been painting big pictures again. After reading, I don’t agree.

1. Shared Prosperity or the Useless Class?

The “useless class” is a concept from Yuval Noah Harari’s Homo Deus. He argues that when AI’s productivity surpasses that of ordinary people, using AI to do work will replace having ordinary people do it. These people become the useless class. Resources will increasingly concentrate in the hands of a few elites and large corporations, and most people will lose their jobs — yet there is currently no effective policy to provide a safety net. This view happens to contradict the Musk-style shared prosperity vision. Musk believes that when AGI is realized, no one will need to worry about survival, education, or healthcare — productivity will be so high that governments will provide a safety net for most people. I currently support Harari’s view. In fact, from anecdotal perceptual statistics around us, the population of the useless class is indeed rising.

2. Can High Productivity Create a Utopia?

One theory supporting my disagreement with the AGI utopia comes from another book, Evolutionary Psychology — Mate Selection Criteria. One particularly striking insight: Due to social division of labor and the biological drive to raise well-adapted offspring, men tend to prefer young, healthy women, while women tend to prefer healthy, resourceful men. This default filtering engraved in our genes means that humans cannot live equally — you don’t want to be the one eliminated. So if a non-comparing, non-competitive, resource-equal utopia could be sustained, productivity is merely one necessary condition among many — there are many other social problems that must be solved, which the public tends to overlook. This isn’t narrowly referring only to evolutionary psychology; some things haven’t been carefully discussed, such as the power struggles in Chimpanzee Politics, which should also be considered.

3. Calhoun’s Mouse Utopia Experiment

In 1972, animal behaviorist John B. Calhoun designed and described in detail a famous experimental environment — “Universe 25.” This was a laboratory “utopia” specially crafted for mice, striving for perfection in almost every aspect: abundant food, water, and nesting materials; regularly cleaned living environment; no predator threats; temperature maintained between 20°C and 31°C via fans and heating, stable and comfortable.

The mouse population’s march toward extinction seems somewhat insane. I’ll focus only on the process: 1) Increased violence 2) No longer pursuing the opposite sex 3) Increased homosexual behavior 4) Increased solitary behavior 5) Males grooming themselves excessively 6) Apathy, etc. Of course, this experiment has flaws. From the intelligence model described in A Brief History of Intelligence, the intelligence gap between mice and humans still spans a primate — it cannot represent human society. But at minimum, it shows that utopia triggers new social problems; people won’t just quietly live their lives.

4. An Economics-Based Society

From the perspective of modern economics, whether AGI can achieve a “shared-prosperity utopia” can be divided into two types: retaining the modern economy or not retaining it.

If we retain the modern economy, AGI can be viewed as an extremely efficient “universal factor of production.” It significantly reduces the costs of knowledge production, decision support, organizational coordination, and marginal labor, raising the ceiling of society-wide productivity. Under this premise, wealth distribution, public service provision, and social security mechanisms still rely on markets, price signals, incentive structures, and institutional constraints. AGI’s role is more about expanding the size of the “distributable pie” rather than automatically solving distribution problems. In other words, shared prosperity remains a political economy problem, not a technical one. AGI can only lower the cost of achieving goals; it cannot replace institutional design.

So, if we don’t retain the modern economy and instead try to bypass markets, prices, and incentive systems to directly rely on AGI to achieve some kind of “techno-utopia” — is it feasible?

The answer can almost certainly be determined as: no.

A utopia without a modern economy was repeatedly verified as a failure in the 1960s–70s. The fundamental reason was not that “technology wasn’t advanced enough” at the time, but that the problems of information and incentives were structurally unsolvable: Even with powerful centralized computing capability, you cannot replace the preference information transmitted by dispersed individuals through price mechanisms, nor can you sustain innovation drive, responsibility constraints, and resource allocation efficiency over the long term. AGI can improve the computational capacity of centralized decision-making, but it cannot eliminate the fundamental economic question of “who is responsible for decisions, who bears consequences, who holds the right to choose.”

Therefore, AGI is not a replacement for the modern economy, but an amplifier within the modern economy’s framework. Any “techno-utopia” that detaches from market mechanisms, incentive structures, and institutional constraints, whether AGI is introduced or not, will essentially replay the historical path of failure — just in more subtle forms and at higher cost.

Productivity (including the intellectual enhancement brought by AGI) is only one of the conditions required for utopia, and far from the most critical one. Utopia is not a computing power problem, nor an intelligence problem — it is a problem of the stability of human behavior under institutional constraints.

The Mathematical Foundation for Why AI Cannot Solve Everything
#

The following is excerpted from Wu Jun’s The Beauty of Mathematics:

“In 1900, Hilbert posed many problems, one of which was: ‘Can any (polynomial) Diophantine equation be determined, through a finite number of operations, to have integer solutions or not?’ If the universal answer to Hilbert’s question is negative, then it means that for many mathematical problems, even God doesn’t know whether an answer exists — because the Diophantine equation solving problem is only a very small part of all mathematical problems. For problems whose very answer-existence cannot be determined, the answer naturally cannot be found. It was precisely Hilbert’s contemplation of the boundaries of mathematical problems that made Turing understand the limits of computation… Matiyasevich rigorously proved that, except for a very small number of special cases, in general, it is impossible to determine through finite operations whether a Diophantine equation has integer solutions. The resolution of this problem had a far greater impact on human cognition than its mathematical influence… If even the solution’s existence is unknown, it’s even more impossible to solve them through computation.”

“A rational-state Turing machine can only solve a subset of problems that have answers… Many engineering problems are not artificial intelligence problems… Today, what we should worry about is not how powerful artificial intelligence or computers are, much less should we think they are omnipotent, because their boundaries have already been clearly delineated by the boundaries of mathematics… There are still many problems in the world that need to be solved by humans. How to make good use of AI tools to more effectively solve human problems is what deserves more attention.”

(See? Reading is useful, right? It explains it clearly to you directly — you probably couldn’t ask the right question or get such an accessible answer. See? Following hardcore tech content creators is useful — I filtered it for you. Hit that follow button ⭐)

Conclusion
#

As a technical blogger, I rarely write about such social issues. I originally just wanted to briefly write about why my previous article got traffic, but explaining this phenomenon somewhat expanded the scope of the problem 😓.

Limitations of this article:

Only discussed a very small part of DBA work — incident recovery — without discussing the intelligentization of other tasks.
GPT knows me too well and seems to be flattering me. It indeed makes very valid points, but I cannot endorse what it says. This is somewhat circular: AI helps me confirm that AI cannot endorse things — an output that inherently cannot be endorsed. From my own perspective, its reasoning is indeed good, with quotable lines throughout.

Some Ops scenarios are certainly easy to AI-ify. But through the discussion in this article, AI-ifying the incident recovery domain still faces considerable difficulty. I have never given up on using AI, nor have I ever given up on using the human brain. I simply enjoy identifying in which scenarios AI works well, in which it works poorly, and in which it cannot be used at all. This may give the article a tone that seems pessimistic about AI’s future, but my thinking is not pessimistic.

At the beginning of this article, you can see the AI rate is 50%. In reality, I also discussed similar issues with several friends and included my own thinking, so the true intellectual composition of this article is:

AI rate 50%, other human brain rate 10%, my brain rate 40%

So this article is also a typical case of “not giving up on using AI, nor giving up on using the human brain.”

Let me conclude with a few questions to briefly state my views:

Why do people still love reading human-written articles? Psychological preference and attention alignment.

Is reading useful (not just books)? Useful, and more useful than ever (bad books are more useless than ever; knowledge taste is more important than ever).

Will AIOps be realized? Yes, but it will take time, and it won’t be easy. This requires academic breakthroughs and the thinking and practice of operations (including DBAs).

Will DBAs be replaced? No. Like software developers, they will experience changes in work patterns but will not disappear.

Which DBAs will remain? “Those who understand both DB and AI, who don’t depend on AI, yet don’t give up on judgment.”

Will AGI be realized? Yes.

Will AGI achieve universal prosperity? No.

If you’d like to discuss AI Ops or the issues in this article with me, you can find me in various PG groups — I should be easy to find. You can also leave me a message.

ref
#

https://github.com/TsinghuaDatabaseGroup/AIDB

https://mp.weixin.qq.com/s/urqh4NZDmkXvDllBCCdZDA

Zhao, Y., et al. (2025). “STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds”. Advances in Neural Information Processing Systems (NeurIPS)

https://zhuanlan.zhihu.com/p/631632685

The Beauty of Mathematics (《数学之美》)

PostgreSQL Operations Experience 2025

Sun, 11 Jan 2026 00:00:00 +0000

This is a technical operations summary, focused on being accessible and practical. It also serves as a periodic reflection on PostgreSQL database operations. Hope it helps fellow PGers.

Previous ops experience: PostgreSQL Operations Experience 2024. Note: this article does not repeat content from that one.

CPU
#

SQL performance problems are the most common root cause in PostgreSQL incident handling. This includes poor SQL performance, suboptimal indexing, sudden high concurrency, and execution plan regressions. For a database like PostgreSQL that lacks a robust plan-binding mechanism, having a DBA team to help design data models, access patterns, indexes, and tune execution plans is crucial — it can significantly reduce sudden CPU saturation incidents.

Execution Plans
#

Execution plan instability is an age-old problem with cost-based optimizers, and PostgreSQL is no exception.

Inaccurate DISTINCT Estimates
#

Case Study: From Inaccurate DISTINCT to DISTINCT Calculation Principles

The default maximum sample size is 30,000 rows. For tables exceeding this size, the estimated distinct count is likely to be low. Note: this assumes the data doesn’t have too many unique values.

Testing on a table with different sample sizes:

Table: reltuples=800 million, relpages=20 million, size=175GB, actual distinct on the target column: 100 million.

target statistics	pages sampling rate	tuples sampling rate	n_distinct	execution time
50	0.00075	0.00001875	60K	2s
100	0.0015	0.0000375	110K	5s
1000	0.015	0.000375	1.03M	58s
3000	0.045	0.001125	2.68M	3m01s
10000	0.15	0.00375	6.75M	7m21s

(target statistics max value: 10000)

Rough summary: n_distinct and analyze execution time grow proportionally with sample size.

n_distinct increases with sample size, while pages and tuples estimates remain consistently accurate.

Generic Plan Interference
#

PostgreSQL execution plans must account for generic plans. A generic plan is parameter-independent — it uses default values to compute cost, then compares against the first five custom plan costs; whichever is cheaper wins.

Case Study: Adding an Index Causes Performance Degradation and Generic Plans

I. Classification of generic plan estimation problems

Because of the 5-execution comparison mechanism, generic plan problems fall into two categories:

The first 5 SQL executions are not representative. Heavily dependent on data skew and whether the first 5 parameter values are representative.
The generic plan itself is flawed. Due to data skew or inability to accurately compute selectivity even with balanced data, the generic plan is inherently inefficient.

II. Solution reference

Generic plan problems often surface on partitioned tables. When the partition key is continuous, scanning all partitions should yield a selectivity of 1, but the generic plan estimates 0.05 — likely resulting in a “full index scan” scenario.

Consider these when optimizing:

Don’t create too many indexes that confuse the optimizer
Eliminate generic plan interference. Execute the prepared statement 6 times for real
Compare plans with session-level set plan_cache_mode='force_generic_plan'; or set plan_cache_mode='force_custom_plan';; or on PG 16+, use explain (GENERIC_PLAN) to compare

Syntax reference:

--prepare/execute
PREPARE sql1(text) AS
SELECT COUNT(*) FROM LZL where a=$1;

EXECUTE sql1('zzz'); --run 6 times first
EXPLAIN EXECUTE sql1('zzz');

select * from pg_prepared_statements --view prepared statement info, current session only

--Compare execution plans, set session parameter then EXPLAIN EXECUTE
set plan_cache_mode='force_generic_plan'
set plan_cache_mode='force_custom_plan'
--Directly view generic plan, 16+
explain (GENERIC_PLAN) xx 

LWLock:Lockmanager Caused by Row Locks
#

LWLock Lockmanager issues typically occur on partitioned tables under high concurrency with queries lacking partition keys. This year, a new scenario was discovered: Row Locks Causing LWLock:Lockmanager

This isn’t a major issue — blocking on concurrent updates to the same row is well known. I just hadn’t expected that updating the same row could also produce LWLock:Lockmanager. Not a particularly valuable case study, but when you see LWLock:Lockmanager as a wait event, consider row locks.

Idle Connections
#

PostgreSQL performance generally improves with each major release. PG 14 made significant optimizations to snapshot acquisition and backend transaction tracking, yielding noticeable improvements for high idle connection counts:

(https://techcommunity.microsoft.com/blog/adforpostgresql/improving-postgres-connection-scalability-snapshots/1806462)

However, this doesn’t mean you can ignore idle connections after PG 14. They still consume backend transaction maintenance overhead, cause context switches, fragment memory, etc. — the more idle connections, the worse the performance.

Typically, application connections have keepalive and pooling. Maintaining some idle connections avoids creating new connections for every request, which would be far more expensive. Small databases generally don’t need to worry much about connection counts (as long as they’re not absurd) — CPUs are cheap, the system isn’t critical, and scaling is easy. But large databases are different. CPU count is the hard limit; you can’t just add more. Large databases already have many idle connections; adding more doesn’t necessarily increase throughput — when CPU is already tight, it can backfire.

PG 15 benchmark experience: with 5K idle as baseline, increasing to 10K idle adds ~2-5 vCPU overhead for idle maintenance; 20K idle adds ~5-10 vCPU. Approximate.

Idle in Transaction
#

Last year I thoroughly criticized long transactions, because they impact PostgreSQL more severely than other databases (Oracle, MySQL, etc.). But this is manageable — with proper alerting and operations, long transactions are solvable.

When monitoring session states, you need to check them. active means running SQL, idle in transaction means in a transaction but not currently executing SQL. All pg_stat_activity states, PG 15:

Current overall state of this backend. Possible values are:

active: The backend is executing a query.
idle: The backend is waiting for a new client command.
idle in transaction: The backend is in a transaction, but is not currently executing a query.
idle in transaction (aborted): This state is similar to idle in transaction, except one of the statements in the transaction caused an error.
fastpath function call: The backend is executing a fast-path function.
disabled: This state is reported if track_activities is disabled in this backend.

Common states are: active, idle, idle in transaction, idle in transaction (aborted). A common misconception about idle in transaction: it only means no SQL is running right now and the transaction hasn’t committed — it does NOT mean the transaction has been idle for a long time. Don’t use xact_start + idle in transaction to judge how long a transaction has been idle. Use state_change + idle in transaction instead.

Memory
#

Memory issues are extremely tricky, and I handled many this year, finding some good solutions. But memory knowledge is broad — I’ll try to simplify as much as possible, going straight to symptoms, results, and solutions.

Memory Issues and Huge Pages
#

Classification of PostgreSQL memory problems:

Relevant wchan states for PG memory issues:

Huge pages are very effective against memory fragmentation and direct memory reclaim within cgroups.

Benchmark results for huge pages: https://docs.paic.com.cn/#/post/84479375

Theoretical benefits of huge pages:

Reduced TLB pressure
Reduced page table size in main memory
Huge pages are physically contiguous. Contiguous physical memory access is better than non-contiguous
With huge pages, pages are directly mapped without multi-level PTE entries

However, huge pages bring management challenges:

Must pre-allocate huge pages
Must calculate huge page size in advance to avoid memory waste

Memory knowledge is extensive. For more, refer to Advanced Linux Memory. Key takeaways:

Rule out OS-level issues before tackling PG instance-level issues
Huge pages have remarkable effects, but in rare cases they don’t help
Many people don’t monitor pgpgin/pgpgout/pgfree, or even pgscank/pgscand — they only look at CPU and memory usage. That’s insufficient for operating PostgreSQL.
Without good operational practices, PG memory can be very unstable

Notable Cgroup Knowledge
#

Cgroup knowledge is also extensive. Refer to earlier articles; here’s a quick summary.

Cgroup v1 has inherent flaws:

Does not account for cgroup page tables
Does not account for cgroup slab
Does not account for cgroup huge pages (huge pages are not charged, not just uncounted)
Does not account for cgroup async/sync page reclaim
Cgroup RSS and process RSS have inconsistent accounting methods
shmem accounting is messy

Unsolved Mysteries
#

Huge pages have solved many problems, but not all. The unsolved portion remains to be researched — hopefully clarified in 2026.

Pay Attention to the OS
#

Pay Attention to Everything OS
#

To operate open-source databases well, you need to understand the operating system.

(Source forgotten)

To operate PostgreSQL well, understanding OS principles is essential. PostgreSQL is built on top of the OS (especially Linux) — it uses whatever Linux provides. PostgreSQL is part of the Linux ecosystem. To truly understand how it works, understand the OS first.

Rule out OS-level issues before tackling PG instance-level issues.

(My own words)

I. CPU

Since PostgreSQL doesn’t use NUMA, whether on bare metal or cgroup/pod-managed CPU, you rarely need to dive into OS-level CPU internals. CPU issues can mostly be diagnosed from SQL or PG stack traces.

II. Memory

See the Memory section. Memory issues require OS-level investigation.

III. Processes

Inspecting PG process states from the OS is critical. You need to check D state, wchan, RSS, syscalls, at minimum.

IV. Host Status and Logs

Monitor host status — CPU, memory, IO, network, logs at the host level. Very important.

It’s hard to imagine that a vague network IO alert like “an I/O error occurred while sending to the backend” is related to underlying storage. Beyond /var/log/messages, PG itself shows nothing. (Of course, this error may have other causes — don’t misinterpret.)

V. Others

Uncategorized.

Physical Reads
#

PostgreSQL itself does not directly expose a “true physical disk read” metric. The various reads in pg_stat_* (e.g., pg_stat_database.blks_read) are reads from the OS cache.

So how do you monitor physical reads?

Reads or buffer allocation metrics are supplementary. The best approach is monitoring the OS itself.

The OS is PostgreSQL’s ecosystem. Never look at the database in isolation. Not being able to monitor physical reads at the database level is nothing to be ashamed of — as long as you have a solution.

Monitor iostat and other disk metrics. For cloud environments, OS-level observability is already mature — don’t waste cloud-native observability.

Autovacuum
#

SQL for monitoring autovacuum processes: sql autovacuum_queue_and_progress

Autovacuum Freeze on Large Databases
#

With properly configured parameters, monitoring, and alerting, autovacuum freeze requires little attention in most databases.

However, in databases with extremely high transaction throughput and very large data volumes, you still can’t ignore it. Autovacuum prevent wraparound may be running constantly. At minimum, watch these two points:

Age alerting: handle promptly and try to prevent the next alert. Don’t wait until the last moment to panic (acceleration options depend on version, e.g., INDEX_CLEANUP OFF, BUFFER_USAGE_LIMIT adjustments)
Impact on memory (especially cache). If autovacuum runs nonstop on a very large database, it impacts cache and memory

For principles and parameters, see this howtos diagram:

Large Tables That Won’t Finish Vacuuming
#

“Large tables” means hundreds of GB, typically with many indexes and dead tuples that prevent vacuum from completing.

The main bottleneck: (auto)vacuum cleans dead index tuples one by one per dead row. Large table (auto)vacuum is slow here — you’ll typically see many dead tuples on the table. Worse, (auto)vacuum may run slower than the rate of dead tuple generation — vacuum never finishes, infinite bloat.

Experience with large tables that can’t finish:

For the same table, dead tuple count is roughly proportional to execution time
From autovacuum log’s user time and elapsed time, you can observe CPU time and execution time, and roughly estimate delay sleep time
Disabling autovacuum cost-based delay can reduce execution time by ~3× (index-size dependent; based on a 200GB table with 280GB indexes)
Adjusting a table’s autovacuum cost-based delay means letting autovacuum rest less when processing that table — consuming more CPU and scan IO in a shorter time

How to accelerate?

Repack. Repack is a nuclear option — fast table rebuild for emergencies. But repack is a CLI tool; running it manually each time is cumbersome.
Tune autovacuum cost-based delay parameters. Either 1. Increase cost limit: alter table t1 SET (autovacuum_vacuum_cost_limit=1000);, or 2. Disable delay entirely: alter table t1 SET (autovacuum_vacuum_cost_delay=0);. Recommended only for tables that can’t keep up.
Drop unnecessary indexes. Scanning indexes and updating index entries takes the most time — dropping unnecessary indexes is effective.
Partitioned tables. Recommended partition size ≤10GB. Converting to partitioned tables is the best solution.
Drop updated_time column indexes to leverage HOT, reducing bloat rate.

Checkpoint and Bgwriter
#

The checkpointer not only creates checkpoints (affecting recovery time) but also flushes dirty buffers. The bgwriter only flushes dirty buffers. Starting from PG 17, some metrics moved to pg_stat_checkpointer. For PG ≤16, mainly look at pg_stat_bgwriter.

I. Checkpoint intervals

Metric checkpoints_timed: corresponds to checkpoint_timeout parameter
Metric checkpoints_req: corresponds to max_wal_size parameter

Recommend using checkpoint_timeout as the primary checkpoint interval. If checkpoints_req appears, increase max_wal_size and tune flush parameters accordingly. When FPIs are present, also check these two metrics.

II. Flush metrics

Metric buffers_checkpoint: dirty buffers flushed by checkpointer
Metric buffers_clean: dirty buffers flushed by bgwriter
Metric buffers_backend: dirty buffers flushed by backends — should be as close to zero as possible; occurrence means bgwriter isn’t aggressive enough
Metric buffers_backend_fsync: meaning unclear

The tuning goal is flush priority: bgwriter flush > checkpointer flush > backend flush

The checkpointer can flush as a side effect, but checkpointer flush speed is hard to control — it can cause IO spikes. So bgwriter flush priority should be higher than checkpointer. Backend flush is obviously worst — minimize it.

III. Bgwriter flush parameters

Bgwriter controls flush speed through a “write some, pause, write again” cycle:

Parameter bgwriter_delay: how long to pause
Parameter bgwriter_lru_maxpages: max pages to write per cycle
Parameter bgwriter_lru_multiplier: pages per cycle = (recent buffer allocation × lru_multiplier), capped at lru_maxpages
Parameter bgwriter_flush_after: fsync after writing this many buffers
Metric pg_buffers_alloc: represents shared memory buffer allocation (allocation means actual eviction occurred, somewhat indicative of pgpgin)
Metric maxwritten_clean: number of times bgwriter_lru_maxpages was reached

Default bgwriter flush logic: each cycle: flush (new buffer count × 2, max 100 dirty buffers), delay 200ms, fsync every 64 buffers flushed.

Per-cycle flush volume depends on recent buffer allocation and bgwriter_lru_multiplier. During peak times, buffer allocation is typically high, so it usually hits bgwriter_lru_maxpages. Thus: bgwriter_lru_maxpages caps peak flush volume; bgwriter_lru_multiplier prevents excessive flushing during off-peak times.

IV. Flush parameter reference

Default max bgwriter flush = 100 × 5 × 8KB = 3.9MB/s. The defaults are definitely too low. If tuning upward, adjust based on shared_buffers size and workload.

After all that theory, here’s a practical reference:

#Read/write ratio 2:8, high load
shared_buffers=40GB
checkpoint_timeout=20min;
max_wal_size=80GB
bgwriter_delay=20ms
bgwriter_lru_maxpages=1000
bgwriter_lru_multiplier=4

Adjust further as needed.

As for effects: from practical experience, don’t expect standalone bgwriter tuning to yield great results. Overly aggressive bgwriter tuning can even backfire.

So: If your database hasn’t been clearly diagnosed with checkpoint flush spikes or other flush issues, don’t touch this. Only recommended for core large databases with high concurrency, as a supplementary tuning strategy alongside other changes (migrations, shared_buffer adjustments, etc.).

V. Flush parameter summary

Bgwriter flushing can be summarized as “three hard’s”:

“Hard to understand, hard to tune, hard to see results.”

DB4AI
#

AI Task Scheduling Writes to Database
#

AI applications are widely deployed at the development level. One scenario: AI task invocations write to the database. Task invocations can spike instantly, and the database writes may lack concurrency control, causing CPU or other resource spikes.

This is a new database incident pattern in the AI era. Be careful.

Vector HNSW
#

Reference: https://postgresql.us/events/pgconfnyc2024/sessions/session/1862/slides/172/pgvector_best_practices_pgconfnyc2024.pdf

HNSW Index Build Acceleration
#

HNSW index builds can be extremely slow — millions of rows can take hours.

Factors affecting HNSW build speed include instance memory (and CPU) as well as index build parameters:

maintenance_work_mem=3g
max_parallel_maintenance_workers=2
m=12
ef_construction=100

Building HNSW indexes can be painful. Ways to accelerate:

Building the index before data load is an option. Though the total initial time is slower, developers may accept “a bit slower” but cannot accept “index building for 1 hour.”
Optimizing post-load index builds:
- SET maintenance_work_mem = '8GB'
- SET max_parallel_maintenance_workers = 8

Post-load index builds need attention to memory — strongly related to instance memory and free memory.

Note: maintenance_work_mem can protect OS memory. If maintenance_work_mem exceeds available OS memory and the table is large, the connection is terminated immediately (fast failure):
```
ERROR: 53200: could not resize shared memory segment "/PostgreSQL.1390017142" to 6439348672 bytes: Cannot allocate memory
LOCATION: dsm_impl_posix, dsm_impl.c:314
```

Note: if memory used during build exceeds maintenance_work_mem, an info notice appears (after some time):

NOTICE: 00000: hnsw graph no longer fits into maintenance_work_mem after 886990 tuples
DETAIL: Building will take significantly more time.
HINT: Increase maintenance_work_mem to speed up builds.
LOCATION: InsertTuple, hnswbuild.c:525

HNSW Index Query Performance
#

Query recall and performance need to be balanced via the ef_search parameter.

Besides ef_search, one more factor significantly impacts query speed: whether the HNSW index is cached in memory.

Index NOT in memory:

explain (analyze,buffers) SELECT image_id, applyNo, feature_vector <-> (select vectorsit
 FROM image_features_test2
 ORDER BY distance
 LIMIT 10;
 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=11852.80..11865.74 rows=10 width=35) (actual time=82193.073..82193.185 rows=10 loops=1)
 Buffers: shared hit=1796 read=9309
 I/O Timings: shared/local read=82108.559
 InitPlan 1 (returns $0)
 -> Limit (cost=0.00..0.02 rows=1 width=32) (actual time=0.008..0.009 rows=1 loops=1)
 Buffers: shared hit=1
 -> Seq Scan on test_0 (cost=0.00..23.60 rows=1360 width=32) (actual time=0.007..0.008 rows=1 loops=1)
 Buffers: shared hit=1
 -> Index Scan using idx_feature_hnsw on image_features_test2 (cost=11852.78..1292546.60 rows=989705 width=35) (actual time=82193.071..82193.179 rows=10 loops=1)
 Order By: (feature_vector <-> $0)
 Buffers: shared hit=1796 read=9309
 I/O Timings: shared/local read=82108.559
 Planning:
 Buffers: shared hit=1
 Planning Time: 0.130 ms
 Execution Time: 82193.279 ms

Index IN memory:

 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=11852.80..11865.74 rows=10 width=35) (actual time=20.240..20.350 rows=10 loops=1)
 Buffers: shared hit=11105
 InitPlan 1 (returns $0)
 -> Limit (cost=0.00..0.02 rows=1 width=32) (actual time=0.007..0.008 rows=1 loops=1)
 Buffers: shared hit=1
 -> Seq Scan on test_0 (cost=0.00..23.60 rows=1360 width=32) (actual time=0.007..0.007 rows=1 loops=1)
 Buffers: shared hit=1
 -> Index Scan using idx_feature_hnsw on image_features_test2 (cost=11852.78..1292546.60 rows=989705 width=35) (actual time=20.239..20.344 rows=10 loops=1)
 Order By: (feature_vector <-> $0)
 Buffers: shared hit=11105
 Planning:
 Buffers: shared hit=1
 Planning Time: 0.093 ms
 Execution Time: 20.392 ms

Same index, same execution plan — the performance difference between index-in-memory and index-not-in-memory is 82193.279 / 20.392 = 4000×!

This gap cannot be ignored. When monitoring HNSW index performance, always check whether the index is in memory. Reference SQL:

--Check if HNSW index is cached in shared buffers via pg_buffercache
SELECT c.relname, pg_size_pretty(count(*) * 8192) as buffered, round(100.0 * count(*) / (SELECT setting FROM pg_settings WHERE name='shared_buffers')::integer, 1) AS buffer_percent, round(100.0 * count(*) * 8192 / pg_table_size(c.oid), 1) AS percent_of_relation FROM pg_class c INNER JOIN pg_buffercache b ON b.relfilenode = c.relfilenode INNER JOIN pg_database d ON (b.reldatabase = d.oid AND d.datname = current_database()) GROUP BY c.oid, c.relname ORDER BY 3 DESC LIMIT 10; 

 relname | buffered | buffer_percent | percent_of_relation 
---------------------------------+------------+----------------+---------------------
 idx_feature_hnsw_1 | 2117 MB | 91.9 | 44.5
 idx_feature_hnsw | 78 MB | 3.4 | 2.0
 pg_inherits_parent_index | 8192 bytes | 0.0 | 100.0

Application Releases
#

DDL Tips
#

Online DDL tools like pg-osc and pg_migrate don’t support partitioned tables, and they have other issues — real-world use is difficult. So DDL tips are still useful: lowering lock levels, proactively identifying blocking, etc., to reduce DDL blocking and rewrite risks.

Key points for understanding this diagram:

Before changes:

Ensure no long transactions on the table — long transactions hold locks on tables persistently. Long transactions are a well-known hazard in PG; handle them first.
Ensure no autovacuum (to prevent wraparound) on the table — autovacuum generally doesn’t block SQL, except when running to prevent wraparound:

Autovacuum workers generally don’t block other commands. If a process attempts to acquire a lock that conflicts with the SHARE UPDATE EXCLUSIVE lock held by autovacuum, lock acquisition will interrupt the autovacuum. However, if the autovacuum is running to prevent transaction ID wraparound (i.e., the autovacuum query name in the pg_stat_activity view ends with (to prevent wraparound)), the autovacuum is not automatically interrupted.
lock_timeout=2000 — if a lock cannot be acquired within 2 seconds, bail out to avoid mass blocking.

Special cases for “small-to-large” type changes:

Small-to-large type changes generally don’t rewrite the table, but there are exceptions. Pay special attention to int → bigint (common for PK columns) and char(n) → char(m).
Partitioned table indexes. Small-to-large type changes on partitioned tables don’t rewrite the table, but they do rebuild indexes — and rebuilding indexes on partitioned tables is typically very slow, potentially causing prolonged level-8 lock blocking. This behavior is unique to partitioned tables.

Changing column types:

Almost always rewrites the table, except for equivalent types or small-to-large cases.

DDL lock-level reduction tips:

Use CIC (CREATE INDEX CONCURRENTLY) for indexes. If partitions don’t support it, do CIC on child tables (remember to attach the index).
CIC has multiple phases. Phases 2 and 3 acquire a SHARE lock, blocking DML. (Official docs only mention SHARE UPDATE EXCLUSIVE — CIC isn’t a simple explicit lock.)
Add primary keys with USING INDEX. For partitions, leverage “add PK on child table + add PK on parent can merge existing child PKs.”
Use VALIDATE CONSTRAINT for constraints.
PG <17 doesn’t support NOT NULL VALIDATE. Use CHECK(col1 IS NOT NULL) instead. This CHECK-to-NOT-NULL conversion won’t produce extra scans.
Adding a column with a volatile DEFAULT rewrites the table. Use the non-volatile-no-rewrite property: add the column first (no rewrite), then UPDATE legacy data as needed.
When attaching partitions, use CHECK constraints to reduce downtime, and use VALIDATE CONSTRAINT for the CHECK.
CREATE TABLE LIKE + ATTACH has much lower lock levels than PARTITION OF (though I still prefer PARTITION OF).

After changes:

Remember to collect statistics (needed in many scenarios).

Parallel Index Creation
#

In production, you may need to create indexes on very large tables that take a long time. Parallel index creation can shorten build time.

Parallel index creation on regular tables:

Parallel parameter: max_parallel_maintenance_workers

Prerequisites:

Enough workers: check max_parallel_workers, max_worker_processes
Increase maintenance_work_mem to GB scale

Notes:

Effective for B-tree and BRIN
maintenance_work_mem limits the entire utility command. Unlike parallel query, where resource limits are per worker process.

From test results, parallel index creation shows diminishing returns beyond 8 workers (this conclusion may not hold in all environments).

Parallel index creation on partitioned tables:

Recommend manual parallel creation across child partitions — run index creation on multiple partitions simultaneously rather than using native parallelism. This reduces multi-process coordination overhead.

Cached Plan Must Not Change Resource
#

After adding a new column the previous night, application connections started throwing errors the next morning: “cached plan must not change result type in PostgreSQL”

Reproduction:

create table a(b varchar(10));
PREPARE p1 (varchar) AS SELECT * FROM a WHERE b=$1;
ALTER TABLE a ALTER COLUMN b TYPE varchar(20);
EXECUTE p1 ('abcd');

ERROR: 0A000: cached plan must not change result type
LOCATION: RevalidateCachedQuery, plancache.c:718

Test environment solutions: DEALLOCATE ALL — actively discard prepared statements Or, DISCARD ALL — actively discard all session state

DEALLOCATE ALL; --DISCARD ALL
PREPARE p1 (varchar) AS SELECT * FROM a WHERE b=$1;
EXECUTE p1 ('abcd');

Production environment solutions:

Since the error occurs at the application layer, JDBC can handle DEALLOCATE ALL / DISCARD ALL, but the application may not have implemented this. Immediate production solutions:

Solutions (choose one):

Since connection pools like HikariCP have connection cycling and timeout mechanisms, killing idle sessions will gradually reduce errors.
Similarly, due to connection pool cycling, you can do nothing — as the pool gradually establishes new connections, the errors fade.
If business pressure is high enough, consider killing all application connections.
Rolling restart of the application.

Not recommended:

“Restart the application after every DDL.” It works but don’t recommend this as a standard practice.
autosave=conservative. It works but enables subtransactions. A savepoint is set for each query; rollback happens only for rare cases like ‘cached statement cannot change return type’ or ‘statement XXX is not valid,’ where the JDBC driver rolls back and retries.

JDBC configuration suggestions:

Configure automatic retry after transaction rollback: https://developer.aliyun.com/article/741750
Other JDBC config references: https://jdbc.postgresql.org/documentation/server-prepare/#corner-cases. Note: some suggestions are not suitable for production.

Physical Replication
#

Query Conflicts
#

Query conflicts are a notoriously frustrating feature that directly impacts the usability of PG standby queries. Query conflicts increase standby lag, yet long-running queries on the standby are logically reasonable. This forces PG administrators to balance between lag management and long-query management — a problem that doesn’t exist in other relational databases.

Hidden characteristics of query conflicts:

Even static tables can trigger query conflicts (see: From Static Table Query Conflicts to Their Principles). The conflict is a snapshot conflict, largely unrelated to table-level locks — snapshot conflicts are cross-table.
Long queries affect short queries. Once a long query pushes standby lag to max_standby_streaming_delay, even short queries get canceled.
Continuous short queries also cause query conflicts. For example, one short query hasn’t finished when the next starts — the two queries may be logically similar, and the startup process hasn’t had time to apply WAL. Both short queries hold the XID that needs to be applied. Check whether pg_stat_activity.backend_xmin is less than the XID the startup process is applying.

Recommended standby query practices:

Using RTO SLO to tune max_standby_streaming_delay is a good approach. When arguments lead nowhere, SLO-based IT management saves the day.
Separate short/fast business queries from long queries (data extraction, reporting) onto different standbys to reduce mutual interference.
Standby queries still need SQL optimization.
Standby WAL apply lag must be monitored.

Logical Replication
#

Logical replication has countless pitfalls. 2024 had many nasty cases; 2025 had some too, but less severe, mostly on older PG versions. Overall, logical replication on newer PG versions is trending toward stability.

Slow DDL/DCL Parsing on Older PG Versions
#

Case Study: GRANT and Walsender Stuck

On PG 13 and earlier, certain DDL/DCL statements parse slowly and may affect walsender lag. These include:

Batch GRANT (including grant all tables) + pathman extension installed (whether used or not)
Batch DDL/TRUNCATE/DCL/DROP PUBLICATION

Older PG + Multiple Replication Links + Flink
#

Flink requires one link per table. Since PostgreSQL walsenders re-decode independently, dozens of Flink links on one PG database are common — and hard to refactor.

On PG 11 and earlier, the walsender main loop calls PostmasterIsAlive(), causing poor loop performance. Starting from PG 12, WalSndLoop no longer polls PostmasterIsAlive() in the main loop; instead, status checks are placed inside WalSndWait, using event-based passive notification. This greatly reduces CPU contention.

If you have multiple Flink links on an older PG version, upgrading can alleviate certain walsender resource contention issues, including:

May resolve the problem where walsender startup resource contention prevents the database from coming up for a long time
May resolve upstream heavy data changes (including DDL rewrites) causing runtime walsender log decoding CPU saturation

Older PG Cannot Auto-Sync New Partitions
#

On older PG versions with declarative partitioning, note that you can only publish child tables individually. PG ≥13 supports publishing by parent table. Below that, you must configure sync per partition child table name:

Allow partitioned tables to be logically replicated via publications (Amit Langote) § §

Previously, partitions had to be replicated individually. Now a partitioned table can be published explicitly, causing all its partitions to be published automatically. Addition/removal of a partition causes it to be likewise added to or removed from the publication. The CREATE PUBLICATION option publish_via_partition_root controls whether changes to partitions are published as their own changes or their parent’s.

In other words, if this partitioned table is an upstream for sync, every time a new partition is added, you must adapt the sync tool to publish it — otherwise, new partition data won’t sync.

Migration and Upgrades
#

Xinchuang Migration and glibc Upgrades
#

Whether it’s Xinchuang (domestic tech migration) or Linux OS version upgrades, glibc upgrades may be involved — and glibc upgrades can be extremely painful. PG sorting was entirely OS-dependent before PG 17.

PostgreSQL cannot detect compatibility issues from glibc upgrades. Every minor version of GNU C library makes locale changes. The most problematic version in practice is glibc 2.28, because 2.28 upgraded to a major Unicode 9.0.0 release (has been updated to a new upstream version from ISO which is in sync with Unicode 9.0.0).

Collations come in many types, and many environments use linguistic sorting (e.g., en_US.utf8), which is the most version-sensitive. Collation changes most commonly cause database crashes during index scans, but also uncommon issues like duplicate primary keys, data landing in wrong partitions, inconsistent merge join results, etc.

Fortunately, PG 17 provides a very safe locale provider: builtin, no longer dependent on OS-provided glibc, ICU, etc. Example:

initdb --locale-provider=builtin --bultin-locale=C.UTF-8 dbname1

However,

builtin is great but arrived too late. Converting existing production instances to builtin collation is no small task. Moreover, Xinchuang migrations or OS upgrades may not mandate database upgrades.

During Xinchuang migration, the target host’s glibc version is typically higher than the old Intel server’s — likely crossing version 2.28. Combined with tight deadlines, KPI pressure, staffing shortages, and large databases, physical migration is unavoidable. So physical Xinchuang migration must account for glibc version and collation-induced anomalies.

What can you do after physical migration?

I. Official required steps

Check indexes, rebuild those clearly problematic
REFRESH DATABASE COLLATION VERSION
Check dependent objects
REFRESH COLLATION VERSION

II. Unofficial “dark arts” approaches

I don’t have a complete solution, just ideas:

Handle partitioned table data landing in wrong partitions
- Partition key is int/bigint/float: unrelated to collation, don’t worry
- Partition key is timestamp: don’t worry; if varchar or other character types: evaluate
- Partition key is character type: refer to “a” vs “-” sort order (pgconf Collation Challenges Sorting It Out). But note:
  - If querying data, don’t query from the parent table — may crash or return nothing
  - No simple detection method
Handle primary key / unique key conflicts
Handle FDW sort range anomalies
Unknown issues

Reference: collation

Smooth Major Version Upgrades
#

https://gitlab.com/postgres-ai/postgresql-consulting/postgres-howtos/-/blob/main/0077_zero_downtime_major_upgrade.md?ref_type=heads

https://www.postgresql.eu/events/pgconfeu2023/sessions/session/4791/slides/439/2023.pgconf.eu%20Zero%20Downtime%20PostgreSQL%20Upgrades.pdf

Common major version upgrade approaches:

pg_upgrade in-place upgrade. Not recommended — may blow up in place.
pg_dump: suitable for small databases, longer maintenance windows.
Logical sync + switchover (pub/sub, pg_logical, DTS, etc.): suitable for small databases, shorter windows.
Physical forward sync + logical reverse sync: suitable for large databases, not-too-short windows.
Physical replication full sync + logical incremental sync + switchover: suitable for large databases, extremely short windows.

Syncing full data via logical replication can be extremely slow. In-place upgrade of a new standby carries uncertainty and upgrade time, plus the need for reverse logical sync. “Smooth major version upgrade” is essentially “physical replication full sync + logical incremental sync + switchover.”

Key technique: the primary creates a slot and returns an LSN. The new standby uses recovery_target_lsn to recover to that LSN, then logical sync begins.

Approximate workflow:

Pre-checks. Multi-database (consider applying one slot LSN for all), extensions, pathman, triggers, foreign keys, unlogged tables, crontab, etc.
Physical sync. Old and new version software, compare and backup conf files, pg_basebackup to build new standby on old version.
Logical sync prep 1. Primary keys and replica identity, create publication; prohibit application DDL/DCL.
Restore new standby to target LSN. Stop new standby; create slot on old primary and record LSN; start new standby with target LSN.
New standby major version upgrade. Upgrade, handle issues, switch environment variables.
Logical sync prep 2. Disable triggers, foreign keys, jobs, extensions, etc.
Logical sync. Create subscription with specified slot, copy_data=false.
Post logical sync. Check for index corruption, check logs for errors and fix, rebuild remote standbys.
Switchover. Stop application; advance sequences, enable foreign keys, triggers, jobs, etc.
Switchover. Build reverse link (old primary subscribes).
Switchover. Application cutover.

The smooth major upgrade approach is smooth for the business but complex for the DBA. It combines all the drawbacks of logical and physical migration — quite painful to execute. The steps above are already simplified. This approach consumes DBA manpower; consider it only for the most critical databases.

Partitioned Table Management
#

PostgreSQL partitioned tables are very flexible, lack built-in interval partitioning, and have varied behavior across versions — making partition management problems an annual occurrence. I believe many PG DBAs still worry about new partition issues.

My observations on partition management and usage issues:

Not using declarative partitioning. Older versions still use pathman partitioning or inheritance-based partitioning, or continue using them even after upgrading. Declarative partitioning was introduced in PG 10. Due to early version limitations, recommend only using declarative partitioning from at least PG 12 onward to reduce environmental complexity.
Developers building child table indexes/primary keys directly. Creating indexes/PKs directly on child tables via SQL rather than through parent table inheritance means the next developer writing SQL may forget. This leads not only to parent-child inconsistency but also child-child inconsistency, eventually making the partition structure unrecognizable.
No new partition management strategy. Forgetting to create new partitions or using a DEFAULT partition. Typically, developers create partitions for a few years ahead; next time, the developers may have moved on, and no one manages new partition creation. This is a ticking time bomb, or data lands in the DEFAULT partition, defeating the purpose of partitioning.
Lack of DBA management. Yes, DBA! PG partitioned table knowledge is extensive (see PostgreSQL Partitioned Tables). How to build management strategies and implement them in your environment requires proactive DBA involvement. This may be the most important factor.

My partition management goals (from Case Study: 2026-01-01 Partition Data Update Failure):

Use the parent table structure as the canonical structure — the parent table faces developers; it should have primary keys, indexes, and replica identity (unless the PG version doesn’t support it).
Keep parent and child tables consistent. Use PARTITION OF when creating new partitions (yes, I don’t recommend ATTACH).
Keep child tables consistent with each other.
Create new partitions in advance. Partition data volume should not be too large.
DEFAULT partitions are not recommended. If created, must monitor writes to them.
Queries on frequently accessed tables must include the partition key for partition pruning. Otherwise, convert to a regular table.

Observability
#

The official documentation clearly explains database, table, index, SQL, flush, and other metrics.

A few metrics deserve special attention — not only are they unclearly explained, but they’re frequently used and have a learning curve.

buffers_alloc, blks_read
#

pg_stat_bgwriter.buffers_alloc: Number of buffers allocated — shared memory eviction volume.
pg_stat_database.blks_read: OS cache reads.

(buffers_alloc may appear in different views across PG versions, but the meaning is the same.)

pg_stat_bgwriter.buffers_alloc is the shared memory buffer allocation count (called buffer allocation in the source). It represents shared memory eviction volume — newly started databases typically have higher values. When observing shared memory busyness, buffer allocation may be better than hit ratio — high hit ratios can be inflated by frequent small-table access, while allocation represents actual eviction.

buffers_alloc counts buffers allocated after reading from cache and loading into a new shared buffer — somewhat representative of OS cache reads too? But in practice, buffers_alloc and blks_read have similar meanings yet can differ significantly in value. Why? Unclear, pending research.

Source: numBufferAllocs

tup_fetched, tup_returned
#

These are metrics in pg_stat_database:

tup_fetched: Number of rows ultimately returned from index scans, after removing filtered rows, dead tuples, and invisible rows. Result-oriented.
tup_returned: Number of rows fetched from the table during index scans, regardless of filter conditions, dead tuples, or visibility. Process-oriented.

Thus, tup_returned is typically much higher than tup_fetched. An abnormally high tup_returned suggests optimization opportunity — after all, many rows were accessed but few returned to the client.

idx_tup_fetch, idx_tup_read
#

These are metrics in pg_stat_all_indexes:

idx_tup_read: Number of index entries accessed (counted from the index side), includes bitmap scans.
idx_tup_fetch: Number of rows ultimately returned from index scans (counted from the table side), excludes bitmap scans.

Madness.

One thing to remember: xx_tup_fetch refers to the final rows returned after index access + table fetch — result-oriented.

References
#

postgres-ai howtos

Best practices for using pgvector

Case Study: 2026-01-01 Partition Data Update Failure

PostgreSQL Partitioned Tables

Case Study: From Inaccurate DISTINCT to DISTINCT Calculation Principles

Case Study: Adding an Index Causes Performance Degradation and Generic Plans

From Static Table Query Conflicts to Their Principles

Control File Parameters and Primary-Standby Parameter Mismatch

https://liuzhilong.blog.csdn.net/article/details/130783036

https://techcommunity.microsoft.com/blog/adforpostgresql/improving-postgres-connection-scalability-snapshots/1806462

https://www.postgresql.org/docs/17/sql-prepare.html

https://www.postgresql.org/docs/17/sql-deallocate.html

https://www.postgresql.org/docs/release/13.0/

https://jdbc.postgresql.org/documentation/use/

https://jdbc.postgresql.org/documentation/server-prepare/#server-prepared-statements

https://www.postgresql.eu/events/pgconfeu2023/sessions/session/4791/slides/439/2023.pgconf.eu%20Zero%20Downtime%20PostgreSQL%20Upgrades.pdf

Thanks to Master Gao for the 2025 battles.

Case: Partition Data UPDATE Failure on 2026-01-01

Sun, 04 Jan 2026 00:00:00 +0000

Symptoms
#

On December 30, business errors were reported — data could not be updated:

ERROR: 55000: cannot update table "tablzl_202601" because it does not have a replica identity and publishes updates
HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.
LOCATION: CheckCmdReplicaIdentity, execReplication.c:575

Temporary Recovery
#

The error message was clear: no replica identity. The table was a partitioned table and a 2026 partition, so I immediately suspected the new partition lacked a primary key. (A new table’s replica identity defaults to default, which only uses a primary key as the replica identity. Without a primary key, updates are impossible.)

Further investigation revealed: the parent table had no primary key or indexes, child partitions from 2025 and earlier had both primary keys and indexes, but 2026 and later child partitions had neither — and all child partitions were published. Roughly:

p_parent -- no PK, no indexes
p_child_202511 -- has PK, has indexes, published
p_child_202512 -- has PK, has indexes, published
p_child_202601 -- no PK, no indexes, published
p_child_202602 -- no PK, no indexes, published

Since the parent table had nothing, a partition of child would also have nothing — you must manually create the primary key and indexes for each child partition. So the new partition creation was problematic; the old partitions presumably had them added after creation.

Additionally, publishing partitioned tables via the parent was only supported starting from PG13. Previously, you couldn’t publish via the parent — only via child tables. This database was on PG11.

Allow partitioned tables to be logically replicated via publications (Amit Langote) § §

Previously, partitions had to be replicated individually. Now a partitioned table can be published explicitly, causing all its partitions to be published automatically. Addition/removal of a partition causes it to be likewise added to or removed from the publication. The CREATE PUBLICATION option publish_via_partition_root controls whether changes to partitions are published as their own changes or their parent’s.

After the initial diagnosis and given the urgency, there were three ways to temporarily resolve:

Add primary keys to the 2026 partitions
Set replica identity full on the 2026 partitions
Remove the 2026 partitions from the publication

Since recovery time was about the same for all options, we chose adding primary keys — the lowest operational cost — to at least stop the business errors.

Root Cause Analysis
#

The issue seems clear: “no replica identity + published + no primary key” prevents updates. But several questions still needed answers.

Question 1: Why does the UPDATE fail even though there’s no 202601 data at all (the new partition has zero rows)?
#

The SQL text was:

UPDATE tablzl_202601
 SET idid = $1,...
 date_updated = now()
 WHERE mykey = $4

The partition key for tablzl_202601 is created_date. The SQL WHERE clause didn’t include the partition key, so when attempting to update the 202601 partition, it found no primary key and errored out.

As for whether row existence or replica identity is checked first, we can see from ExecSimpleRelationUpdate. This function has changed very little across PG versions:

/*
 * Find the searchslot tuple and update it with data in the slot,
 * update the indexes, and execute any constraints and per-row triggers.
 *
 * Caller is responsible for opening the indexes.
 */
void
ExecSimpleRelationUpdate(EState *estate, EPQState *epqstate,
					 TupleTableSlot *searchslot, TupleTableSlot *slot)
{
...
	CheckCmdReplicaIdentity(rel, CMD_UPDATE); // check replica identity

	/* BEFORE ROW UPDATE Triggers */
	if (resultRelInfo->ri_TrigDesc &&
		resultRelInfo->ri_TrigDesc->trig_update_before_row)
	{
		slot = ExecBRUpdateTriggers(estate, epqstate, resultRelInfo,
									&searchslot->tts_tuple->t_self,
									NULL, slot);

		if (slot == NULL)		/* "do nothing" */
			skip_tuple = true;
	}

	if (!skip_tuple)
	{
		List	 *recheckIndexes = NIL;

		/* Check the constraints of the tuple */
		if (rel->rd_att->constr)
			ExecConstraints(resultRelInfo, slot, estate);
		if (resultRelInfo->ri_PartitionCheck)
			ExecPartitionCheck(resultRelInfo, slot, estate, true);

		/* Materialize slot into a tuple that we can scribble upon. */
		tuple = ExecMaterializeSlot(slot);

		/* OK, update the tuple and index entries for it */
		simple_heap_update(rel, &searchslot->tts_tuple->t_self,
						 slot->tts_tuple);

		if (resultRelInfo->ri_NumIndices > 0 &&
			!HeapTupleIsHeapOnly(slot->tts_tuple))
			recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
												 estate, false, NULL,
												 NIL);

		/* AFTER ROW UPDATE Triggers */
		ExecARUpdateTriggers(estate, resultRelInfo,
							 &searchslot->tts_tuple->t_self,
							 NULL, tuple, recheckIndexes, NULL);

		list_free(recheckIndexes);
	}
}

ExecSimpleRelationUpdate flow:

Check replica identity
BEFORE ROW UPDATE triggers
Check constraints (both non-partition and partition constraints)
Update the row
Insert index entries
AFTER ROW UPDATE triggers

So PG’s logic checks replica identity first, before row updates and everything else.

Even though the SQL didn’t include the partition key, would adding it trigger partition pruning? The answer is: maybe not.

Partition pruning improvements across versions:

PG10 introduced declarative partitioning. There was no enable_partition_pruning parameter; pruning was done at planning time via constraint_exclusion. So PG10 had no query-execution-time pruning.
PG11 added runtime partition pruning: Allow partition elimination during query execution (David Rowley, Beena Emerson). But it only supports pruning with bound variables, not non-immutable functions (including now()).
PG14 added final pruning: This wins in UPDATEs on partitioned tables when only some of the partitions will actually receive updates. i.e., supports pruning with non-immutable functions.

Since PG11 doesn’t support now() pruning, adding a now() condition to the business SQL wouldn’t trigger pruning — the error would still occur. However, if the business passed a bound variable, pruning would trigger and the error wouldn’t appear. Note: “the error wouldn’t appear” means updating 202512 data wouldn’t error out on the 202601 partition; updating 202601 data would still fail regardless.

Question 2: The partition was created on 2025-12-26, so why was the problem only discovered on December 30?
#

This is even simpler: “no replica identity + published + no primary key” is an AND condition.

Although the new partitions were created early, they were published on the evening of December 29 at 20:47:

cat postgresql-12-29.csv.bak |grep "alter publication"
2025-12-29 20:48:07.730 CST,"userlzlreplication","lzldb",xxx"statement: alter publication publzl add table ""public"".""tablzl_202601"", ""public"".""tablzl_202602"",...

The first error appeared on December 29 at 22:26, about 1.5 hours later:

 cat postgresql-12-29.csv.bak |grep "REPLICA IDENTITY"
2025-12-29 22:26:01.404 CST,"userlzlreplication","lzldb",375121,xxx,"cannot update table ""tablzl_202601"" because it does not have a replica identity and publishes updates",,"To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.",,,,"UPDATE tablzl

Summary
#

Root cause overview: The parent table had no primary key, so partition of child partitions naturally also had none. Old child partitions had their primary keys added manually; new child partitions did not, resulting in the 202601 partition lacking a primary key. Logical replication relies on the primary key (default replica identity) for synchronization. Without replica identity, changes can’t be sent downstream, and UPDATE/DELETE statements on published tables cannot execute. In PG11, an UPDATE SQL that does include the partition key condition may still visit the new partition.

A stroke of luck: Due to various factors, this problem was discovered early in this particular database. We had a one-day buffer on December 31 to fix all database instances, ensuring at least that January 1 new partition data updates wouldn’t error out. Otherwise, on January 1, 2026, multiple systems would have likely gone up in flames.

Temporary measures (pick one):

Add primary keys to 2026 partitions
Set replica identity full on 2026 partitions
Remove 2026 partitions from the publication

For replication pipeline optimization:

Tables without primary keys should be detected proactively, otherwise publishing them could cause business-side UPDATE failures

For partition management strategy:

PG’s partitioned tables are highly flexible, and developers generally don’t know how to create partitions correctly. Combined with significant new partitioning features across roughly PG10-15, and the lack of INTERVAL partitioning in PG, partitioned tables can end up a mess. Standardized management of partitioned tables is thus critical. For partition table features and operational tips, see: PostgreSQL Partitioned Tables

As for management tools, I’ll skip those.

Management goals:

Use the parent table structure as the standard: the parent table, being developer-facing, should have primary keys, indexes, and replica identity (unless the PG version doesn’t support it)
Keep parent and child tables consistent; use partition of to create new partitions (yes, I don’t recommend attach)
Keep child tables consistent with each other
Create new partitions in advance; partition data volumes should not be excessive
Default partitions are not recommended; if created, their writes must be monitored
Frequently accessed tables must have partition keys in their SQL queries and use partition pruning; otherwise, convert them to regular tables

References
#

https://www.postgresql.org/docs/release/10.0/

https://www.postgresql.org/docs/release/11.0/

https://www.postgresql.org/docs/release/12.0/

https://www.postgresql.org/docs/release/13.0/

https://www.postgresql.org/docs/release/14.0/

src/backend/executor/execReplication.c

PostgreSQL Partitioned Tables

Paper Deep Read: Anarchy in the Database

Sat, 03 Jan 2026 00:00:00 +0000

Paper: Anarchy in the Database: A Survey and Evaluation of Database Management System Extensibility

GitHub: https://github.com/cmu-db/ext-analyzer

PGConf: The trouble with extensions (PGConf.dev 2025)

Why This Paper
#

This is a survey of database extensions (mainly Postgres), covering the implementation approaches of extensions across different databases, existing problems, and most importantly, compatibility. The most significant finding: an evaluation of over 400 PostgreSQL extensions shows that 16.8% of extensions have compatibility issues with at least one other extension, potentially leading to system failures.

Analysis tools and results are on GitHub; Marco Slot’s presentation is at PGConf.

Extension Categories
#

Extension Classification
#

The extension classification chapter is particularly lengthy — a single diagram actually clarifies everything.

Extensions across 6 databases:

PostgreSQL (1986): Written in C, designed from the beginning as an extensible architecture. Consequently, PostgreSQL has the richest and most diverse extensible ecosystem.
MySQL (1994): Written in C++, best known for its storage engine plugin architecture.
MariaDB (2009): A fork of MySQL, also C++ based, supporting more extensions than the original MySQL.
SQLite (2000): Embedded database written in C, adaptable to various hardware devices and operating systems.
Redis (2009): In-memory key-value store written in C++, uniquely extensible — only supports running above the DBMS key-value storage layer.
DuckDB (2018): Embedded analytical database written in C++, with a rapidly emerging extensible ecosystem.

Flexibility and Security
#

Extension security and flexibility are a trade-off — PG extensions are the most flexible but least secure; Redis is the most secure but least flexible:

How PostgreSQL Extensions Are Typically Implemented
#

PG generally has two ways to implement extensions:

Through handler functions, such as UDFs, UDTs, external tables, storage engines, and index access methods.
Through hooks. Hooks are declared as function pointers in global variables; if a hook is set, it will call these pointers instead of its own code.

Implementations may use both approaches — they’re not mutually exclusive. The other 5 databases have generally similar implementations, but none of them have hook-based implementations.

Extensions may use different implementation approaches, e.g., function + types + index AM — this is the number of extensibility types. From Figure 1, we can see that extensions with 1-3 types are the most common, and the most-used implementation approach is function.

From Table 3, 92.5% of extensions use UDFs — after all, it’s a user-facing feature, easiest to develop with the lowest barrier to entry. The least used is client authentication, as this scenario itself is uncommon.

Extension Code Copy Rate
#

The paper also conducted an interesting survey: the extent to which extension code is copied from built-in code:

Out of 441 extensions, 16.6% — 73 extensions — contain at least one line copied from PG source code. The detailed distribution is shown in the left chart above.

Why are so many extensions copying code? Because:

Some functions in PG source are declared static, only callable within their own file, so they can only be copied.
Due to the extension’s own requirements, functions may need slight adjustments, so they can only be copied and adjusted.

And how much were these copied functions adjusted? See the right chart above.

As can be seen, unmodified copies are actually rare.

In summary, extension code is copied from PG source out of necessity, and the overall copy rate isn’t high.

The Heavyweight! — PG Extension Compatibility
#

This is the most interesting part of the paper: pairwise compatibility testing was conducted on 96 extensions, and testing found that 16.8% of extension pairs are incompatible!

Testing methodology:

Installation. Yes, installation alone can cause problems. The authors tested both A→B and B→A installation orders, hence the asymmetric diagram.
Running the extension’s provided unit tests.
pgbench. Smoke testing. pgbench is of course simple, but good results here can still indicate something.

Among the top 20 least compatible extensions, many commonly-used ones appear:

Common extensions: pg_hint_plan, vector, pg_show_plans, pgsentinel, pg_cron, pg_stat_kcache
Heavy extensions: citus, timescaledb

The fact that such extremely common and star extensions can have such poor compatibility is jaw-dropping.

What’s even more chilling: this is just simple pairwise testing. Running 3-10 extensions should be the production norm, and production environments are far more complex and variable than the paper’s three testing methods.

Finally, the paper identifies the reason for poor extension compatibility: extensions that use more components, extension types, and hooks are more likely to be incompatible with other extensions.

Nitpicking
#

It’s really still about Postgres

The paper’s title says DBMS, but it’s mainly about PG compatibility. MySQL, Redis, etc. compatibility is only covered in the survey, with no experimental data at all. (Though the survey is interesting — you can learn how MySQL and Redis extensions are implemented.)

On the other hand, this paper has a kind of alternative “general-specific-general” feel: “DBMS-Postgres-DBMS” 😅

Insufficient compatibility testing

PG has 400+ extensions, but only 96 were tested for compatibility, and only 1-on-1 compatibility testing, without tests involving 3 or more extensions. The compatibility testing isn’t particularly comprehensive.

Conclusion
#

PG extensions are indeed numerous and flexible — you’d struggle to find functionality that PG extensions don’t support. But the extensions themselves are almost in a state of “anarchy” — both extension development and usage have problems.

From the compatibility results, extension compatibility is quite poor — even the installation order affects compatibility. Multiple extensions also depend on hook execution order; for example, two extensions both requiring themselves to execute last becomes awkward. “Having everything” doesn’t mean “install everything.”

Extension Security Issues
#

PG extensions have virtually no security management, whether from inherently unsafe extensions or user privilege escalation through extensions.

If an extension contains unsafe languages, only the OS can restrict its behavior, not the DBMS.
If an extension can access user space, the OS layer cannot manage it.
Extensions implemented through queries (e.g., UDFs) generally won’t bypass ACL policies. While UDFs are more secure, they’re not absolutely secure, as UDFs with admin privileges can exist.
A single hook may not be restricted by ACL, because in PostgreSQL, ACL is only enforced at the planning and execution layers. PG provides SECURITY LABEL to restrict access control for objects (including extensions).

Philosophical Thoughts on Software Management
#

“If an extension contains unsafe languages, only the OS can restrict its behavior, not the DBMS.”

This statement itself isn’t wrong, but it carries an implication of “your directory could be deleted.” To counter this, consider the following:

If you use this software, you trust it, just like PG itself (but even when using PG, you create a postgres OS user rather than using root directly). As for extensions, treat them as part of the PG software. PG is trusted and can be installed directly in production because of its industry reputation. The same goes for extensions — choose reputable extensions rather than using them indiscriminately. This is essentially the difference between PostgreSQL community gatekeeping and extension provider gatekeeping. For cloud service providers, many extensions aren’t supported — the cloud provider assumes the gatekeeping function and the responsibility of taking the blame.

Version Convergence
#

PG extension versions have these characteristics:

The same extension may have different extension packages for different database versions.
Extensions have different versions.

This means that without version management, you’ll end up with unmanageable numbers of software versions. To address this, limiting specific PG versions to installing specific extension versions is a good approach. As for extension upgrades needed for certain requirements, implement them through PG version upgrades. This strategy sacrifices some flexibility to ensure stability. I personally think it’s worthwhile — the need to upgrade extensions itself isn’t common, but it can reduce many software management issues and unknown compatibility problems.

Consider Compatibility When Using Extensions
#

Since extension compatibility isn’t great, managing extensions becomes especially important — we don’t want the database returning strange results or even crashing while running.

Extension management strategy: 1. Install necessary extensions. 2. Create needed extensions on demand. 3. Don’t install obscure extensions.
Search the compatibility matrix. While PG compatibility testing isn’t perfect, it’s still valuable. Since the paper isn’t directly searchable for the compatibility matrix, you can “ctrl+f” search the ext-analyzer compatibility table to preliminarily assess whether extensions you need have good compatibility.

Trivia
#

In the 1976 INGRES paper, UDFs were already implemented through extensions. Even POSTGRES carried forward this functionality in its 1986 initial release. Oracle’s UDF implementation came in Oracle 7, released in 1992 — much later than PG.

The SQL standard didn’t include UDFs until 1996 — a full 20 years after INGRES’s UDF. Stonebraker indeed wasn’t very focused on driving standards.

Original link: https://lastdba.com/2026/01/03/论文精读插件无政府状态/

Case Study: Row Locks and LWLock LockManager

Sun, 21 Dec 2025 00:00:00 +0000

Symptoms
#

The database showed a large number of row locks and a smaller number of LWLock LockManager waits. CPU was maxed out and active sessions spiked. The blocking PID associated with the locks kept changing, with no obvious long-transaction blocker. (Imagine high CPU and active sessions.)

The SQL corresponding to the large number of locks was as follows:

UPDATE lzl_record SET rc_lzl1= rc_lzl1 + $1, pc_lzl2 = pc_lzl2 + $2, rc_lzl3 = rc_lzl3 + $3 where lzl_id = $4

Analysis
#

No Increase in SQL Concurrency Observed
#

From the correlation between hits and CPU, we can analyze from the SQL hit perspective. That UPDATE SQL accounted for about 80% of activity. The SQL’s execution count had not changed, but blks hit was clearly abnormal.

We also analyzed metadata access — within snapshots, no metadata tables showed unusually high access.

From the symptom analysis, neither SQL concurrency increase nor metadata anomalies were apparent. The reason for the SQL hit increase wasn’t obvious at this point.

LWLock LockManager Analysis
#

Since the SQL itself is simple — the lzl_id field in the lzl_record table is a unique field, meaning the update is done by unique key.

In addition to the large number of explicit locks, the wait events at the scene also included LWLock LockManager.

However, the table is a regular table (not partitioned), with only 4 or 5 indexes on it.

LWLock LockManager is related to not using the fast path. Simple queries and DML can use the fast path:

Weak relation locks. SELECT, INSERT, UPDATE, and DELETE must acquire a lock on every relation they operate on, as well as various system catalogs that can be used internally. Many DML operations can proceed in parallel against the same table at the same time; only DDL operations such as CLUSTER, ALTER TABLE, or DROP – or explicit user action such as LOCK TABLE – will create lock conflicts with the “weak” locks (AccessShareLock, RowShareLock, RowExclusiveLock) acquired by DML operations.

So a SELECT/DML accessing no more than 16 relations (including indexes) should be able to use the fast path, and there shouldn’t be much LWLock LockManager.

However, DML certainly can’t simply use the fast path — fast path handles lock operations entirely locally, but DML must check whether other sessions hold locks on the row and needs to access shared memory. Combined with the fact that this SQL updates by unique field yet still encounters row locks, it must be updating the same row.

From the logs, we could see instances of updating the same row — one row had tens of thousands of lock-waiting updates.

Benchmark Testing
#

Benchmarking Same-Row Updates to Reproduce LWLock LockManager
#

Given that row locks definitely can’t rely solely on the fast path, and knowing that LWLock LockManager degrades database performance, we benchmarked different scenarios.

#prompt
Give me a pgbench benchmark script
Table structure: primary key, unique field + unique index, other fields
Update: update by unique field

Benchmark repeated updates on the same row (repeated row-lock updates)
Benchmark random updates on different rows (no row-lock updates)

Script omitted. Environment: 20 cores, 96GB RAM.

pgbench commands:

pgbench -h localhost -p $PGPORT -d lzldb -U dbmgr -f update_same_unique_key.sql -c 200 -j 32 -T 600 -r -S
pgbench -h localhost -p $PGPORT -d lzldb -U dbmgr -f update_random_unique_key.sql -c 200 -j 32 -T 600 -r -S

Wait events during the benchmark:

 -- Update same row, 2 typical samples
 usename | state | wait_event | wait_event_type | cnt 
----------+--------+---------------------+-----------------+-----
 dbmgr | active | LockManager | LWLock | 105
 dbmgr | active | transactionid | Lock | 61
 dbmgr | active | tuple | Lock | 25
 dbmgr | active | [null] | [null] | 8
 dbmgr | active | WALSync | IO | 1
 
 usename | state | wait_event | wait_event_type | cnt 
----------+--------+---------------------+-----------------+-----
 dbmgr | active | transactionid | Lock | 180
 dbmgr | active | LockManager | LWLock | 18
 dbmgr | active | tuple | Lock | 1
 dbmgr | active | WALSync | IO | 1

-- Update different rows, 2 typical samples
usename | state | wait_event | wait_event_type | cnt 
----------+---------------------+---------------------+-----------------+-----
 dbmgr | active | [null] | [null] | 106
 dbmgr | idle | ClientRead | Client | 34
 dbmgr | idle in transaction | ClientRead | Client | 25
 dbmgr | active | WALWrite | LWLock | 21
 dbmgr | active | BufferMapping | LWLock | 7
 dbmgr | idle in transaction | [null] | [null] | 4
 dbmgr | idle in transaction | WALWrite | LWLock | 2
 
 usename | state | wait_event | wait_event_type | cnt 
----------+---------------------+---------------------+-----------------+-----
 dbmgr | active | [null] | [null] | 117
 dbmgr | idle | ClientRead | Client | 42
 dbmgr | idle in transaction | ClientRead | Client | 24
 dbmgr | active | WALWrite | LWLock | 12
 dbmgr | active | XactGroupUpdate | IPC | 1
 dbmgr | active | WALSync | IO | 1
 dbmgr | active | XactSLRU | LWLock | 1
 dbmgr | active | BufferContent | LWLock | 1
 dbmgr | active | ClientRead | Client | 1

From the wait events, the difference is clear: updating the same row produces LWLock LockManager, sometimes at a high proportion. Updating different rows mostly just waits on CPU. Scenario 1 matches the production situation.

A Brief Analysis of Row Locks and Fast Path
#

The lmgr README’s explanation of the fast path:

Fast Path Locking
-----------------

Fast path locking is a special purpose mechanism designed to reduce the
overhead of taking and releasing certain types of locks which are taken
and released very frequently but rarely conflict. Currently, this includes
two categories of locks:

(1) Weak relation locks. SELECT, INSERT, UPDATE, and DELETE must acquire a
lock on every relation they operate on, as well as various system catalogs
that can be used internally. Many DML operations can proceed in parallel
against the same table at the same time; only DDL operations such as
CLUSTER, ALTER TABLE, or DROP -- or explicit user action such as LOCK TABLE
-- will create lock conflicts with the "weak" locks (AccessShareLock,
RowShareLock, RowExclusiveLock) acquired by DML operations.

Conditions for locks that can use the fast path, from lmgr/lock.c:

/*
 * The fast-path lock mechanism is concerned only with relation locks on
 * unshared relations by backends bound to a database. The fast-path
 * mechanism exists mostly to accelerate acquisition and release of locks
 * that rarely conflict. Because ShareUpdateExclusiveLock is
 * self-conflicting, it can't use the fast-path mechanism; but it also does
 * not conflict with any of the locks that do, so we can ignore it completely.
 */
#define EligibleForRelationFastPath(locktag, mode) \
	((locktag)->locktag_lockmethodid == DEFAULT_LOCKMETHOD && \
	(locktag)->locktag_type == LOCKTAG_RELATION && \
	(locktag)->locktag_field1 == MyDatabaseId && \
	MyDatabaseId != InvalidOid && \
	(mode) < ShareUpdateExclusiveLock)

SELECT/DML can use the fast path, but only for locktype=relation.

Let’s look at the actual lock situation when there’s a row lock:

-- Session 1
begin;
update lzl1 set b='zzz' where a=1;

-- Session 2
begin;
update lzl1 set b='zzz' where a=1;
-- waiting


-- Session 3
 select * from pg_locks where pid<>(select pg_backend_pid()) order by pid,locktype;
 locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath 
---------------+----------+----------+--------+--------+------------+---------------+---------+--------+----------+--------------------+--------+------------------+---------+----------
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 5/4791 | 220559 | RowExclusiveLock | t | t
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706189 | [null] | [null] | [null] | 5/4791 | 220559 | ExclusiveLock | t | f
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706190 | [null] | [null] | [null] | 5/4791 | 220559 | ExclusiveLock | t | f
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706187 | [null] | [null] | [null] | 5/4791 | 220559 | ShareLock | f | f
 tuple | 4267681 | 5290151 | 0 | 1 | [null] | [null] | [null] | [null] | [null] | 5/4791 | 220559 | ExclusiveLock | t | f
 virtualxid | [null] | [null] | [null] | [null] | 5/4791 | [null] | [null] | [null] | [null] | 5/4791 | 220559 | ExclusiveLock | t | t
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 7/562 | 253641 | RowExclusiveLock | t | t
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706187 | [null] | [null] | [null] | 7/562 | 253641 | ExclusiveLock | t | f
 virtualxid | [null] | [null] | [null] | [null] | 7/562 | [null] | [null] | [null] | [null] | 7/562 | 253641 | ExclusiveLock | t | t

PG’s row lock implementation is quite complex — it involves not only tuple locks, but also transactionid and relation locks. Among these, only locktype=relation and virtualxid can use the fast path; all others cannot.

Compare with the no-row-lock case:

-- Session 1
begin;
update lzl1 set b='zzz' where a=1;

-- Session 2
begin;
update lzl1 set b='zzz' where a=2;
-- waiting

select * from pg_locks where pid<>(select pg_backend_pid()) order by pid,locktype;
 locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath 
---------------+----------+----------+--------+--------+------------+---------------+---------+--------+----------+--------------------+--------+------------------+---------+----------
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 5/4792 | 220559 | RowExclusiveLock | t | t
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 5/4792 | 220559 | AccessShareLock | t | t
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706214 | [null] | [null] | [null] | 5/4792 | 220559 | ExclusiveLock | t | f
 virtualxid | [null] | [null] | [null] | [null] | 5/4792 | [null] | [null] | [null] | [null] | 5/4792 | 220559 | ExclusiveLock | t | t
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 7/563 | 253641 | AccessShareLock | t | t
 relation | 4267681 | 5290151 | [null] | [null] | [null] | [null] | [null] | [null] | [null] | 7/563 | 253641 | RowExclusiveLock | t | t
 transactionid | [null] | [null] | [null] | [null] | [null] | 170706212 | [null] | [null] | [null] | 7/563 | 253641 | ExclusiveLock | t | f
 virtualxid | [null] | [null] | [null] | [null] | 7/563 | [null] | [null] | [null] | [null] | 7/563 | 253641 | ExclusiveLock | t | t

There are only 2-3 fewer fastpath=f entries. The transactionid locks held by both sessions definitely can’t use the fast path.

Summary of conditions for using the fast-path lock mechanism (all must be met):

Lock level <= 3, i.e., SELECT/DML statements
locktype=relation. PG’s row locks also require at least transactionid and tuple locks, so these two can’t use the fast path
Fewer than 16 relations accessed (typically exceeded only with full partition access on partitioned tables)

Conclusion
#

Is the row lock the cause or the effect? Is it a row lock problem, or did database performance degrade causing SQL to run slower and produce row locks?

Row lock is the cause. The SQL execution count didn’t change, but the SQL parameters shifted from scattered to concentrated — i.e., updates to the same row noticeably increased. From the benchmark data, updating the same row produces row lock and LWLock LockManager waits.

SQL execution count didn’t increase — did SQL performance degrade?

SQL performance did degrade, but the index was definitely not chosen incorrectly — it was simply because the same row was being updated repeatedly.

Solution:

From the business side, the SQL was tied to a certain API endpoint: after being called, it updates the call count into the table. If the same endpoint is called repeatedly, it’s possible to repeatedly update the same row. Therefore, reducing repeated calls to the same endpoint, or batching the database updates into fewer, larger batches, is expected to mitigate this problem.

My 2025 Year-End Summary

Sun, 21 Dec 2025 00:00:00 +0000

As a DBA
#

As a DBA, I strongly believe in first principles and information theory when it comes to problem analysis. A DBA needs to deeply understand the system, understand PostgreSQL, to explain anomalies from first principles. For example, in the first half of the year I spent considerable effort understanding Linux memory, exploring the essence of memory issues and their solutions. At the same time, this year I took a step forward in system operations — no longer focusing solely on technical problems and handling, but more on providing solutions. These should encompass thinking across the PostgreSQL database technology dimension, the system dimension, and the management dimension.

Here’s a simple classification of cloud DBA work:

Many Ops papers only talk about incident handling, but in reality, incident handling probably accounts for less than 5% of actual operational workload. And whether in academia or practice, anomaly ops itself isn’t very effective anyway. So I’m not very bullish on AIOps being able to significantly help DBAs. Note that DBAs using AIOps and DBAs using AI are two different things.

Actually, this diagram is just so-so, because it doesn’t include leadership tasks, which are definitely the bulk.

Looking back at the 2023 and 2024 year-end summaries, I can simply summarize my DBA work year by year:

2023: Comprehensive PostgreSQL learning
2024: Comprehensive PostgreSQL operations
2025: Responsible for 1510 emotional value

What’s deeply ironic is that last year’s conclusion — “DBAs are providing 1510 emotional value to their leaders” — became my lived reality this year. I don’t want to say more about it. In short, it’s been exhausting, mentally draining. I hope next year brings improvement.

READING
#

This year I read even more books than last year (from 20+ to 30+), but wrote even fewer reading notes. Writing is indeed troublesome and energy-consuming, and I’ve grown to prefer the feeling of reading itself. Compared to last year, this year’s reading shows a clear decrease in PostgreSQL technical books, an increase in comprehensive technical books, and I even started reading psychology, economics, and philosophy. In short, broader hunting grounds, not limited to databases alone. Also fewer novels — novels are like snacks, and I’m increasingly losing interest in such non-nutritious content.

This year’s book list generally falls into: IT Systems, Economics, Popular Science, Spiritual, and Fiction categories. As with last year, ranked by personal preference.

IT Systems Book List:

“SRE: Google’s Approach to Service Reliability” — DBAs are not SREs, but their work involves system stability objectives, which has similarities with DBA work. Some content in this book about cloud environments or management aspects was truly enlightening — for example, SLA, systems engineering, operational pressure, busy work, role rotation, “trust the team rather than a single technical expert,” and more. Absolutely brilliant. Recently I also heard the term DBRE — Database Reliability Engineer — which fits my current role even better than DBA. In short, an excellent book, a must-read for modern ops.
“Running Linux Kernel: Introduction” — operating open-source databases requires understanding the operating system. One of my books for studying Linux memory.
“Deep Understanding of Linux Processes and Memory” — one of my books for studying Linux memory.
“Understanding the Linux Kernel” — one of my books for studying Linux memory.
“Observability Engineering” — the patterns and flaws of traditional monitoring and traditional ops, and what observability essentially means. Quite helpful.

Economics Book List:

“Microeconomics” — a masterpiece, by Daron Acemoglu. I consider it essential reading for life. This book has my best notes of any book. Not only understanding economics, but further understanding society. Some viewpoints left a deep impression on me:
- Proves why the market is an invisible hand that maximizes social surplus value — any intervention reduces social surplus value.
- Under what circumstances markets are ineffective: externalities, public resources, and common-pool resources.
- Women earn less than men in the workplace partly because women bear children and cannot participate in production during that time.
- The function of academic credentials is signaling — to a certain degree, they certify the productive value of the person.
- Business entry and exit are normal market signals, not signs of disorder.
- The trade-off between equity and efficiency is a subject of study.
“Why Nations Fail” — a masterpiece, by Daron Acemoglu. This book can be summarized in one sentence: Why do nations succeed? Because of creative destruction. Daron Acemoglu won the 2024 Nobel Prize in Economics for “research on how institutions are formed and how they affect prosperity.” What’s even more remarkable is that this book is easier to understand than other economics works. The top-recommended economics masterpiece.
“The Rational Optimist” — said to rival “Sapiens,” but it’s definitely a notch below. However, the content quality isn’t bad, and it’s more economics-oriented. Some viewpoints are very fresh, for example:
- Modern economics makes the rich richer, but the poor are not getting poorer.
- Self-sufficiency is poverty.
- What distinguishes humans from animals is barter exchange (in “Sapiens” it’s the cognitive revolution).
- Higher income leads to greater happiness — this is a fact.
- The elevation of trade in social status came from the rise of maritime trade, because land trade was unstable and easily plundered.
“Reminiscences of a Stock Operator” — feels like I learned something and nothing at the same time. Decent read though.
“Game Theory” — honestly, I found it average. Not much content, quite superficial. I mainly read it because economics books keep mentioning game theory, so I flipped through it to evaluate.
“The Wealth of Nations” — extremely dense, not for normal people to read. Incredibly content-rich. Adam Smith must have been a genius — hard to imagine what kind of mind produced this. Too difficult for me, didn’t finish, gave up.

Popular Science Book List:

“A Brief History of Intelligence” — a masterpiece, essential reading for the AI era. This book is worn from my constant reading, covered in notes everywhere. Deconstructing the human brain, understanding what intelligence is, understanding how AI came to be. I give it full marks! Now whenever I see any animal, I first think about what intelligence level it’s at…
“On Top of Tides” — by Wu Jun. Every IT professional should read this book. It tells the rise and fall of major IT companies. You can learn about Oracle, Google, Fairchild, Bell Labs, and even basics about venture capital. Every company has its own DNA, which is nearly unchangeable and determines the company’s culture and characteristics. A programmer’s must-read.
“The Almanack of Naval Ravikant” — has many useful perspectives, like views on marginal utility. And more importantly, it recommended one of my favorite books this year — “Microeconomics.” It also recommended meditation, which changed my habits.
“How to Manage a Software Company” — by Frank Slootman, a legendary Silicon Valley CEO who led three software companies (ServiceNow, Data Domain, Snowflake) to successful IPOs. A very good book, looking at company development, employee management, execution, decision-making, and decision failures from an IT company manager’s perspective. Highly recommended.
“The Economics of Aging” — by Kenichi Ohmae. Using Japan’s aging problem to glimpse China’s aging problems and opportunities. The demographic structural risks in our country are severe and about to come to a head. In this era, highly recommended reading.
“The Fourth Wave” — by Kenichi Ohmae. Mainly about how Japan missed the IT technology wave, still relying on old industries to support the national economy, appearing somewhat envious of South Korea and China. I personally love the author’s attitude of directly criticizing the prime minister, haha.
“The Checklist Manifesto” — explains the necessity of checklist inspections before Western surgical procedures. Seemingly simple steps can dramatically increase surgical success rates. This book had a big impact on my work — I genuinely brought the checklist concept into my work. I treat database operations like a surgical procedure — checklists are a simple yet necessary means to improve success rates.
“The Mythical Man-Month” — “adding people” cannot linearly reduce systems engineering project timelines, but you also can’t simply reject “adding people” because large systems engineering projects genuinely require many people collaborating. It’s a good book, but calling it a programmer’s must-read feels like a stretch.
“The Beauty of Mathematics” — by Wu Jun. Also quite good. Technology always has its mathematical foundations. This book accessibly tells the beauty of mathematics.
“McKinsey Structured Thinking” — any problem should be structurally decomposed. When I encounter new problems, I think this way. A useful book.
“The Chrysanthemum and the Sword” — stock from years ago that I dug out to read. An American’s post-WWII perspective on Japan. You can glimpse aspects of Japanese culture like modified Confucianism without “benevolence (ren),” the psychology of indebtedness, etc. One drawback is it’s quite dated — modern Japan is largely different from that era.
“The Black Swan” — a black swan refers to unforeseen extreme events. Black swan events will always happen — there’s no such thing as 100% accurate prediction. It also discusses classification, which reminded me of content from “Structured Thinking” and “The Worlds I See”: “The essence of human understanding is classifying things,” but classification always awkwardly leaves some things unclassifiable or unable to be classified. Black swan events exist from the moment of classification. An interesting and noteworthy reflection.
“The Professional” — by Kenichi Ohmae. Very mediocre, not recommended.

Spiritual / Self-Help Book List:

“The Evolution of Desire” — evolutionary psychology, a masterpiece.
“Die with Zero” — experience the right things at different life stages. Even if you revisit something after missing it, it won’t feel the same as experiencing it at the right time. A life manual, highly recommended.
“Ten Minutes Meditation” — mainly about the importance of meditation and how to do it. I learned meditation through this book. When I first completed meditation, I fell in love with it. It gave me a feeling of being taken to outer space and then returning to Earth. More importantly, it truly relieves stress. Meditation has become part of my life.
“The Manipulation Bible” — okay.
“Siddhartha” — incomprehensible, rubbish.
“The Book of Life” — pure chicken soup, rubbish.

Fiction Book List:

“The Stranger” — a masterpiece. An indescribable sense of authenticity, feeling like an outsider oneself.
“Yellowface” — a very interesting book about a white American woman who plagiarizes an unpublished work by a deceased Asian writer, even using a very Chinese pen name. When fans discover she’s white, you can feel the embarrassment. Playfully explores racial prejudice. As thrilling as watching a TV drama — twists and turns, gripping. Highly recommended.
“The World of Yesterday” — by Stefan Zweig. Austria, Europe, WWI and WWII through a writer’s eyes. Returning to that turbulent Europe from a different angle. A very good book.
“Project Hail Mary” — sci-fi. I increasingly dislike reading sci-fi. This one is okay: imagine you’re on an alien exploration mission, all your crewmates have died, and you happen to encounter a friendly alien. How do you communicate with them…
“Letter from an Unknown Woman” — by Stefan Zweig. Not good. Only the first story is somewhat novel. No interest in seriously reading the other two.
“Satantango” — incomprehensible. Even Nobel Prize in Literature winners vary in quality.

Blog and WeChat Official Account
#

The name of my WeChat Official Account has always been a struggle. I didn’t put much thought into maintaining it anyway, so I casually used a few names. This year I watched a documentary — “The Last Porter” (最后的棒棒), which moved me deeply. The DBA profession, like the porters of Chongqing, is undergoing tremendous change. So I simply changed it to “最后的DBA” (The Last DBA). This name rolls off the tongue nicely and carries some historical context and philosophical reflection. Seems like a good name.

Since a lot of time goes into work, I didn’t have much time for writing to begin with. Plus, this year my operational approach kept changing, and no matter how I adjusted my daily schedule, I couldn’t carve out a good time slot. I even invested some money, and my time still didn’t increase, which frustrated me for quite a while. Looking back now, I only published 12 articles this year — not even one in the first half of the year.

Very dissatisfied. 😠

I don’t know if my skills have improved or if the system is genuinely stable, but cases worth deep research seem to have become fewer. But this isn’t really a big problem. This year I also started treating paper interpretation as an article type. I personally feel the results are decent — I can learn quite a bit, without being too insular or reinventing the wheel. Using AI to interpret papers would certainly be fast, but I personally feel there are two problems:

Do I truly understand? I feel like I don’t — it’s not the same concept as reading through it myself. Reading it yourself not only allows deeper understanding but also lets you discover all sorts of quirky details.
Can’t pad articles. If I can interpret a paper with one prompt, then I feel the dissemination value is minimal — surely there’s no one not using AI now, right?

Of course, I don’t read every paper word by word — that would be too inefficient. I only select papers that I feel are good and worth frame-by-frame interpretation, and savor them carefully.

A quick summary of this year’s articles:

Too few in quantity
Slightly improved quality, and useful content (several articles I’m personally very satisfied with)
Explored new formats

Final Thoughts
#

This year was a busy one, with both bad and good memories. Many important things were left unfinished. Next year should bring significant changes. Writing this year-end summary is quite interesting — looking back to see what my past selves were up to is a fun experience.

Last year’s 2025 OKRs:

Continue some things — FAILED
Think about how to produce output — FAILED
Master another track — HALF SUCCESSFUL
PostgreSQL… haven’t figured out what more to do — FAILED
Find a way to resume fitness — FAILED

2026 Plan:

Continue some things
Pay attention to my psychological and physical health — next year’s annual health inspection alerts should be lower than this year’s
Pay attention to article readership, maintain the WeChat Official Account
Explore DB AI Ops, report to myself next year
Manage upward — don’t invest too much time in work
Travel during holidays instead of grinding
Read no fewer than 30 books, but don’t focus solely on quantity

Paper Deep Read: DBAIOps

Sun, 21 Dec 2025 00:00:00 +0000

Paper: DBAIOps: A Reasoning LLM-Enhanced Database Operation and Maintenance System using Knowledge Graphs

Repo: https://github.com/weAIDB/DBAIOps/

What is DBAIOps
#

Why DBAIOps:

Manual operations are extremely time-consuming.
Manual operations are difficult to scale.
Manual operations are often trapped in recurring failures.
Documentation + RAG models are inaccurate (limited DBA experience integration).

In short, both manual operations and existing solutions are mediocre, hence DBAIOps — an operations system combining LLM reasoning and knowledge graphs to achieve DBA-like diagnostic capabilities.

Comparison of database failure analysis approaches:

Rule-based approach: Traditional, rigid.
Machine learning approach: Essentially rule-based with similar limitations; depends on training data leading to lower generation capability; generally suitable for diagnosing common specific problems.
LLM-based approach: Uses general documentation and LLMs (e.g., decision-tree-based), prone to giving generic results.
LLM+RAG approach: Searches based on chunked top-k approximate knowledge; results are inaccurate.

After comparing the above approaches, the advantages of DBAIOps combining graph knowledge, DBA experience, and LLMs are clear:

Incorporates DBA experience.
Preserves original relationships.
Supports new root cause identification and solutions.
Extensible.

Overview
#

Left side is architecture, right side is an example.

Offline: DBA experience is embedded into Neo4j, with the resulting graph model called ExperienceGraph, where edges represent anomaly phenomena or metric relationships. The embedded anomaly model is called AnomalyModel.

Online: Anomaly analysis, retrieval, and report generation. The AnomalyProcessor extracts standard failure information and AnomalyModel information, then retrieves the graph via ExperienceRetriever; finally, RootCauseAnalyzer calls the LLM to generate analysis reports.

From the right-side example, we can see graph relevance finding LOG FILE SYNC associated with LOG WRITE performance and IO performance; through REDO ALLOCATION, we can find table structure changes and DDL.

The Operations Experience Graph Model
#

Unlike rule-based or document-chunk-based RAG, ExperienceGraph is a graph model encoding heterogeneous operations experience information. The graph contains three elements: (vertices, directed edges, relationships on edges).

Based on the characteristics of operations experience, DBAIOps classifies vertices:

trigger vertex: Used to detect database anomalies; the entry point for anomaly analysis. For example, LOG FILE SYNC is an entry vertex.
metric vertex: Database runtime metrics. For offline knowledge, this refers to metrics from operations case studies (if present).
experience vertex: Encodes domain-specific operations experience, covering anomaly meanings and handling methods. For example, LOG FILE SYNC exceeding 60ms indicates overly frequent commits or parameter adjustments needed.
tool vertex: Executable scripts for collecting and analyzing anomaly metrics.
tag vertex: Semantic categories of graph vertices. For example, “Concurrent Transactions” involves multiple vertex types; tag vertices strengthen cross-case associations.
auxiliary vertex: Explains the meaning of metrics.

Edge classification:

containment edge: Trigger Vertex - Experience Vertex
relevance edge: Trigger Vertex - Metric Vertex
diagnosis edge: Experience Vertex - Metric Vertex
synonym edge: Only appears between Tag Vertices, indicating semantic synonymy, e.g., physical_read and disk_read; shared_pool and shared_buffer.

Analyzing the operations experience graph model through an example:

LOG FILE SYNC has multiple TAGs, and TAGs are associated with Experience, metrics, and tools. The strong relevance is evident — it represents a human DBA’s understanding and operations experience of LOG FILE SYNC.

Graph Construction
#

Manual graph construction is unreliable, and existing ML-generated graphs may generate irrelevant relationships, so a semi-automatic graph generation approach is proposed.

Graph initialization: This part is manually generated, defining trigger vertices according to rules. Once trigger vertices are generated, their associated metric vertices, experience vertices, etc., are automatically generated. This is somewhat like a human DBA guiding the creation of a knowledge sketch — the overall framework cannot be changed; nothing bizarre should be generated.
Graph storage: Stored in Neo4J. Additionally, different database types are marked with tags, making much knowledge reusable and avoiding duplicate graph construction.
Graph augmentation: Generating more edges.
Graph updates: DBAIOps supports incremental updates. Updates here include both adding new vertices and removing old vertices.

Anomaly Model
#

Metrics
#

Metrics come from many sources, including runtime information (CPU %, throughput, etc., routine monitoring), logs, traces, etc. Combined with relevance differences, strongly correlated metrics need to be extracted. So metrics are divided into 2 categories:

Immediately collected metrics: Runtime information, logs, traces.
Subsequently collected metrics: Periodic, delta, etc., metrics generated when needed, such as AWR/ASH data.

Regarding metric-anomaly correlation, unlike baseline-based approaches, DBAIOps uses specific metric combinations for each anomaly type.

Finally, a formula determines whether an anomaly has actually occurred:

Two-Stage Graph Evolution
#

Database anomalies rarely occur in isolation — one performance issue may simultaneously trigger or exacerbate others. However, connections between different anomaly models (e.g., LOG_FILE_SYNC and REDO_ALLOCATION) in pre-built knowledge graphs tend to be loose, with shared experience fragments sparse and fragmented. This makes it difficult for traditional methods to discover cross-model composite root causes, such as combined I/O bottleneck and memory pressure issues.

To address this challenge, DBAIOps proposes an automatic “graph evolution” mechanism that dynamically discovers and connects relevant experience fragments between different anomaly models, evolving the knowledge graph from an initially sparse structure into a densely interconnected network, thus supporting more comprehensive root cause analysis.

Stage 1 - Graph Inference and Proximity Discovery: Uses graph query language (Cypher) to collect and aggregate relevant metrics, traversing related nodes and edges based on configurable thresholds to build association networks. For example, starting from LOG_FILE_SYNC latency, traverse up to 3 hops of associated nodes. Establish connections between LOG_FILE_SYNC and REDO_ALLOCATION models because they are both related to I/O-related concurrency issues. Through multiple iterations, the knowledge graph gradually evolves into a denser structure, enabling diagnosis to consider more potential factors and composite causes.
Stage 2 - Adaptive Abnormal Metric Detection: Identifies truly anomalous metrics along graph expansion paths. Using an Adaptive Detection Function (ADF), it calculates composite anomaly scores considering dimensions such as metric volatility and dynamic baseline deviation. Based on anomaly scoring results, it decides whether further knowledge graph structure expansion is needed, filtering a precise subset of anomaly metrics for subsequent LLM root cause reasoning.

Generating Analysis Reports
#

Once the graph is ready, prompts need to be fed to the LLM to generate desired reports. A well-structured prompt can also improve report accuracy.

Anomalies have 5 components, which serve as the prompt for the LLM:

Anomaly: Anomaly description (“CPU usage spiked to 95% at 16:00 on 2023-10-05”)
Condition: Anomaly trigger condition (“exceeds 90% for >5 min”)
Metrics
Experience: Provides normal load values or recent maintenance tasks.
Output: Describes the report’s composition — anomaly verification (requiring further analysis), root cause analysis, recovery plan, summary, SQL text.

Some personal thoughts:

Recent maintenance tasks are very useful — maintenance tasks generally have strong correlation, and failure analysis can’t just be simple technical analysis. However, who updates these maintenance tasks and which ones to update or not update is a problem.

The first few items in output are easy to understand, but the last one — SQL text — is a stroke of genius. In production environments, aside from hardware failures, database runtime status is strongly correlated with SQL. I personally believe you can unthinkingly capture SQL and discuss causality later. From an operations perspective, failures always require joint investigation with developers, so SQL text is basically mandatory to capture.

Evaluation
#

Comparison of analysis report quality across different tools and approaches:

Impressive results. Notably, DBAIOps specifically emphasizes that mid-sized LLMs already produce good analysis results. This is important — DeepSeek-R1 671B running bare isn’t bad, but the cost is on a completely different level.

Nitpicking
#

Can’t really be called “Ops” — it only has failure analysis functionality. Ops content is vast; failure analysis is just the tip of the iceberg.
Graph classification doesn’t match the graph example. The defined tag vertices and edges differ significantly from the example.

The vertices in the example play important roles, but these edge types aren’t defined: tag vertex-tool vertex, tag vertex-experience vertex, tag vertex-metric vertex. And the edges that should exist seem mostly absent, with only synonym edges present.

Undescribed parts of the example should be listed, otherwise it’s confusing.

The two-stage graph evolution results are a bit odd:

w/o ADF means without Stage 2 graph evolution (adaptive abnormal metric detection). w/o ADF should mean without Stage 1 graph evolution (graph inference and proximity discovery). w/o ADF means without either stage of graph evolution.

Here, the case with both stages of graph evolution is missing — having it would better demonstrate the effectiveness of two-stage graph evolution.

Root causes are somewhat limited:

The circled ones should be relatively common (I only looked at Oracle and Postgres), but these root causes are currently absent.

PG’s root causes are a bit sparse. Dirty page flushing generally isn’t a major issue — as a root cause, it probably ranks behind many other root causes.

Summary
#

Points I personally really like:

GraphRAG should be better than vector RAG for failure diagnosis.

(GraphRAG original paper: From Local to Global: A GraphRAG Approach to Query-Focused Summarization)

SS represents vector RAG, TS represents source text summaries, and C0/C1/C2/C3 represent GraphRAG at different knowledge granularities. From this chart, we can simply conclude: GraphRAG is better suited for multi-document complex scenarios and multi-angle analysis, but may not necessarily outperform vector RAG in precision.

Semi-automatic graph generation approach.

Graph generation is semi-automatic — trigger vertices are manually created, others can be auto-generated. For example, LOG FILE SYNC is a trigger vertex. Failure entry points can indeed be made into obvious anomaly points — these are the entry points. Same for PG, same for any failure — it aligns with human logic for understanding failures.

Automatic graph evolution.

Strengthening associations between certain vertices is meaningful, as evident from the “Performance of DBAIOps Variants” table.

Automatic baseline adjustment.

In Observability Engineering, there’s this passage about AIOps:

AI can only help when there are clearly discernible patterns and it can identify shifting baselines for prediction — such AIOps doesn’t exist yet.

DBAIOps in my eyes:

Clearly discernible patterns = DBAIOps’s graph, which includes failure models, anomaly relationships, monitoring data, and logs.

Shifting baselines = DBAIOps’s adaptive abnormal metric detection.

In summary, it’s a significant advancement over random chunking of failure knowledge, setting a single baseline, and vector approximate search in RAG models.

Original link: https://lastdba.com/2025/12/21/论文精读dbaio-ps/

From collation mismatch Exception to Its Principles

Sat, 13 Dec 2025 00:00:00 +0000

Problem Phenomenon
#

After physical migration to Xinchuang, occasional errors appear in the pg log, version pg15:

WARNING: 01000: collation "zh_CN.utf8" has version mismatch
DETAIL: The collation in the database was created using version 2.17, but the operating system provides version 2.28.
HINT: Rebuild all objects affected by this collation and run ALTER COLLATION pg_catalog."zh_CN.utf8" REFRESH VERSION, or build RaseSQL with the right library version.
LOCATION: pg_newlocale_from_collation, pg_locale.c:1660

Context: During the physical switch, invalid index rebuilding and refresh database collation version were performed.

Although the libc version was upgraded after physical migration, indexes were rebuilt and are now valid, and the collation version in the database is already consistent with the OS libc.

So,

Why is the error reported?

Where is the error triggered?

What is the impact of the error?

How to resolve it?

Problem Analysis
#

Why is the error reported?
#

The collation inside the database mainly involves 3 aspects: database, columns, and indexes. The first two use default collation, and the index collation is the real collation.

First, check the database collation. All databases use en_US.UTF8, and refresh database collation has already been done, so the “collation "zh_CN.utf8" has version mismatch” error should not be thrown at the database layer.

Then check columns without specially specified default collation:

select attrelid,attname,attcollation from pg_attribute where attcollation not in (0,100,950,951);
 attrelid | attname | attcollation 
----------+---------+--------------
(0 rows)

0 means no collation, default oid=100, C oid=950, POSIX oid=951; “zh_CN.utf8” definitely won’t be any of these four.

Finally, check indexes without specially specified collation:

select * from (select indexrelid ,unnest(indcollation) coll from pg_index) i where coll not in (0,100,950,951);
 indexrelid | coll 
------------+------
(0 rows)

Having ruled out database, columns, and indexes, only one situation remains: the application layer specifies a sort rule:

select col1 from (values ('a'), ('A'), ('啊'), ('阿')) AS l(col1) order by col1 collate "zh_CN.utf8";
WARNING: 01000: collation "zh_CN.utf8" has version mismatch
DETAIL: The collation in the database was created using version 2.17, but the operating system provides version 2.28.
HINT: Rebuild all objects affected by this collation and run ALTER COLLATION pg_catalog."zh_CN.utf8" REFRESH VERSION, or build RaseSQL with the right library version.
LOCATION: pg_newlocale_from_collation, pg_locale.c:1660
 col1 
------
 阿
 啊
 a
 A

This zh_CN.utf8 version is inconsistent with the actual one:

 select collname,collversion,pg_collation_actual_version(oid) from pg_collation where collname ='zh_CN.utf8';
 collname | collversion | pg_collation_actual_version 
------------+-------------+-----------------------------
 zh_CN.utf8 | 2.17 | 2.28

Not only zh_CN.utf8 is different, all are different (except a few collations without version concept).

So it’s very likely that the application itself specified a sort rule “zh_CN.utf8”, but the coll version in the database is inconsistent with the OS, which triggered the error.

Source Code Understanding
#

The error message makes it easy to locate the source code position. Two main functions are of interest: pg_newlocale_from_collation and CheckMyDatabase.

`pg_newlocale_from_collation` Caching and Checking `pg_collation`
#

pg_newlocale_from_collation was introduced in pg10.

/*
 * Create a locale_t from a collation OID. Results are cached for the
 * lifetime of the backend. Thus, do not free the result with freelocale().
 *
 * As a special optimization, the default/database collation returns 0.
 * Callers should then revert to the non-locale_t-enabled code path.
 * In fact, they shouldn't call this function at all when they are dealing
 * with the default locale. That can save quite a bit in hotspots.
 * Also, callers should avoid calling this before going down a C/POSIX
 * fastpath, because such a fastpath should work even on platforms without
 * locale_t support in the C library.
 *
 * For simplicity, we always generate COLLATE + CTYPE even though we
 * might only need one of them. Since this is called only once per session,
 * it shouldn't cost much.
 */
/* locale_t means non-ICU. This function caches a locale_t type collation OID for the backend
* the default/database collation returns 0. "default" means using the database's collation
*/
pg_locale_t
pg_newlocale_from_collation(Oid collid) // Note: passes in collation oid, not fetching all pg_collation
{
...
	/* Return 0 for "default" collation, just in case caller forgets */
	if (collid == DEFAULT_COLLATION_OID) // Three special collations:
		return (pg_locale_t) 0; // default oid=100, C oid=950, POSIX oid=951

...
	if (cache_entry->locale == 0)
	{
...
		collversion = SysCacheGetAttr(COLLOID, tp, Anum_pg_collation_collversion,
									 &isnull); // Get version from pg_collation data dictionary
		if (!isnull)
		{
...
			actual_versionstr = get_collation_actual_version(collform->collprovider, collcollate); // Get actual version via get_collation_actual_version
...
			collversionstr = TextDatumGetCString(collversion);

			if (strcmp(actual_versionstr, collversionstr) != 0) // Compare data dictionary version and actual version, throw error if different
				ereport(WARNING,
						(errmsg("collation \"%s\" has version mismatch",
								NameStr(collform->collname)),
						 errdetail("The collation in the database was created using version %s, "
								 "but the operating system provides version %s.",
								 collversionstr, actual_versionstr),
						 errhint("Rebuild all objects affected by this collation and run "
								 "ALTER COLLATION %s REFRESH VERSION, "
								 "or build PostgreSQL with the right library version.",
								 quote_qualified_identifier(get_namespace_name(collform->collnamespace),
															NameStr(collform->collname)))));
		}
...
	return cache_entry->locale;
}

The main check is: through the coll oid, check whether the version in the pg_collation data dictionary is consistent with the actual version; if inconsistent, throw an error.

`CheckMyDatabase` Caching and Checking `pg_database`
#

CheckMyDatabase has existed for a long time, performing many database-side checks. However, pg15 added logic for checking the database version.

/*
 * CheckMyDatabase -- fetch information from the pg_database entry for our DB
 */
static void
CheckMyDatabase(const char *name, bool am_superuser, bool override_allow_connections)
{
...
	/* Fetch our pg_database row normally, via syscache */
	tup = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId));
...
	default_locale.provider = dbform->datlocprovider; // default is the db's

	/*
	 * Default locale is currently always deterministic. Nondeterministic
	 * locales currently don't support pattern matching, which would break a
	 * lot of things if applied globally.
	 */
	default_locale.deterministic = true; // byte-order sensitive

	/*
	 * Check collation version. See similar code in
	 * pg_newlocale_from_collation(). Note that here we warn instead of error
	 * in any case, so that we don't prevent connecting.
	 */
	datum = SysCacheGetAttr(DATABASEOID, tup, Anum_pg_database_datcollversion,
							&isnull); // Get datcollversion from pg_database
	if (!isnull)
	{
		char	 *actual_versionstr;
		char	 *collversionstr;

		collversionstr = TextDatumGetCString(datum);

		actual_versionstr = get_collation_actual_version(dbform->datlocprovider, dbform->datlocprovider == COLLPROVIDER_ICU ? iculocale : collate); // Get actual version via get_collation_actual_version
...
		else if (strcmp(actual_versionstr, collversionstr) != 0) // Compare db datcollversion and actual version, throw warning if not equal
			ereport(WARNING,
					(errmsg("database \"%s\" has a collation version mismatch",
							name),
					 errdetail("The database was created using collation version %s, "
							 "but the operating system provides version %s.",
							 collversionstr, actual_versionstr),
					 errhint("Rebuild all objects in this database that use the default collation and run "
							 "ALTER DATABASE %s REFRESH COLLATION VERSION, "
							 "or build PostgreSQL with the right library version.",
							 quote_identifier(name))));
	}
...
}

The CheckMyDatabase function compares the datcollversion in the pg_database data dictionary with the actual version.

Function Differences
#

In pg14 and before, there was only 1 collation comparison logic: when a session first caches the corresponding collation, it calls pg_newlocale_from_collation to access the version of the corresponding collation in the pg_collation data dictionary and compare it with the real version.
In PG15 and later, because the datcollversion field was added to the pg_database table, a new logic for checking db collation version was added: when a session first accesses the db in pg_database, it calls CheckMyDatabase to check the datcollversion of the corresponding database in pg_database and compare it with the real version.

Why Are There Fewer Errors After Only Refreshing the Database?
#

After refreshing the database collation version, the warning about inconsistent pg_database coll version won’t be triggered, but it still cannot rule out the situation where pg_collation’s coll version is inconsistent. Why are there so many fewer errors after only refreshing the database? Could it be that pg_collation’s coll version simply won’t be loaded?

select c.coll,count(*) from (select unnest(indcollation) coll from pg_index ) c group by c.coll;
 coll | count 
------+-------
 950 | 37 --C
 0 | 2841 --No collation
 100 | 723 --default

In real environments, default is the most used. Generally, no one specifies a collation; if not specified it’s default, and default is the database’s default collation.

Here we need to revisit the pg_newlocale_from_collation function. The function starts like this:

pg_locale_t
pg_newlocale_from_collation(Oid collid)
{
	collation_cache_entry *cache_entry;

	/* Callers must pass a valid OID */
	Assert(OidIsValid(collid));

	/* Return 0 for "default" collation, just in case caller forgets */
	if (collid == DEFAULT_COLLATION_OID)
		return (pg_locale_t) 0;
...

When collid==DEFAULT_COLLATION_OID==100, it directly returns without executing the real version check below, so it won’t throw a warning. This logic is reasonable because the db coll version has already been verified when logging into the database; if there’s a problem, a warning must have already been thrown at the session layer.

Furthermore, even if a possible value like collid=37 is passed in, the corresponding C also has no version concept.

Therefore, after refreshing the database, in the vast majority of scenarios, as long as the database’s internal sorting is used (not expression sorting or specified index sorting), no error will be thrown.

Testing
#

Here we only test whether there is a refresh warning, not testing index corruption or database crashes.

# Check libc version
getconf GNU_LIBC_VERSION

Source host version glibc 2.17
Target host glibc 2.28
pg version pg15+

Test: Refresh db without refreshing pg_collation, only db coll version changes
#

select datname,datlocprovider,datcollate,datctype,datcollversion from pg_database 
 datname | datlocprovider | datcollate | datctype | datcollversion 
------------+----------------+-------------+-------------+----------------
 lzldb | c | en_US.UTF-8 | en_US.UTF-8 | 2.17
 
select collname,collprovider,collversion,pg_collation_actual_version(oid) from pg_collation where collname ~ 'en_US.utf8';
 collname | collprovider | collversion | pg_collation_actual_version 
------------+--------------+-------------+-----------------------------
 en_US.utf8 | c | 2.17 | 2.28

alter database lzldb refresh collation version;
NOTICE: 00000: changing version from 2.17 to 2.28
LOCATION: AlterDatabaseRefreshColl, dbcommands.c:2399
ALTER DATABASE

Check pg_collation and pg_database again:

 collname | collprovider | collversion | pg_collation_actual_version 
------------+--------------+-------------+-----------------------------
 en_US.utf8 | c | 2.17 | 2.28

 datname | datlocprovider | datcollate | datctype | datcollversion 
------------+----------------+-------------+-------------+----------------
 lzldb | c | en_US.UTF-8 | en_US.UTF-8 | 2.28

Consistent with the official documentation description: refresh database collation version only refreshes the db’s default collation; pg_collation itself won’t change.

Test: Refresh db without refreshing pg_collation, specifying expression sort reports warning
#

As analyzed at the beginning, expression sorting will report a warning, omitted.

Test: Refresh db without refreshing pg_collation, creating a new index with specified collation reports warning
#

Test 1: Specify collation when creating index

 collname | collversion | pg_collation_actual_version 
------------+-------------+-----------------------------
 zh_CN.utf8 | 2.17 | 2.28
 
 > create index idx11 on tt(a collate "zh_CN.utf8");
WARNING: 01000: collation "zh_CN.utf8" has version mismatch
DETAIL: The collation in the database was created using version 2.17, but the operating system provides version 2.28.
HINT: Rebuild all objects affected by this collation and run ALTER COLLATION pg_catalog."zh_CN.utf8" REFRESH VERSION, or build PostgreSQL with the right library version.
LOCATION: pg_newlocale_from_collation, pg_locale.c:1664
CREATE INDEX

Test 2: Specify column default collation when creating table, don’t specify when creating index

\c lzldb -- Reconnect a session
You are now connected to database "lzldb" as user "postgres".
create table ttt(a varchar(10) collate "zh_CN.utf8");
CREATE TABLE

> create index idxttt on ttt(a);
WARNING: 01000: collation "zh_CN.utf8" has version mismatch
DETAIL: The collation in the database was created using version 2.17, but the operating system provides version 2.28.
HINT: Rebuild all objects affected by this collation and run ALTER COLLATION pg_catalog."zh_CN.utf8" REFRESH VERSION, or build PostgreSQL with the right library version.
LOCATION: pg_newlocale_from_collation, pg_locale.c:1664
CREATE INDEX
Time: 7.904 ms

Column default collation and index specification of collation are essentially the same thing, both for specifying the index’s collation. Both can report warnings.

Test: Refresh db without refreshing pg_collation, existing index with specified collation does not report warning
#

Scenario: The original database already has an index specifying collation zh_CN.utf8, different from the db. Refreshing the db won’t catch it. But after migrating to a new database, the vendor’s coll version definitely changed.

select collname,collprovider,collversion,pg_collation_actual_version(oid) from pg_collation where collname ~ 'zh_CN.utf8';
 collname | collprovider | collversion | pg_collation_actual_version 
------------+--------------+-------------+-----------------------------
 zh_CN.utf8 | c | 2.17 | 2.28

Without using expression sorting, the index can be used, but index sorting cannot be used:

> set enable_seqscan =off;
SET
> EXPLAIN ANALYZE SELECT a FROM tt ORDER BY a LIMIT 1000;
 QUERY PLAN 
----------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=6667.80..6670.30 rows=1000 width=33) (actual time=44.928..45.145 rows=1000 loops=1)
 -> Sort (cost=6667.80..6892.81 rows=90004 width=33) (actual time=44.926..45.021 rows=1000 loops=1)
 Sort Key: a
 Sort Method: top-N heapsort Memory: 127kB
 -> Index Only Scan using idxtt on tt (cost=0.42..1732.98 rows=90004 width=33) (actual time=0.029..15.434 rows=90004 loops=1)
 Heap Fetches: 4

Existing indexes with specified collation do not report warnings when used.

Summary of This Problem
#

The refresh database and refresh collation warnings are session-level. In each session, for each database or each collation, it only reports once.

Only refreshing the database very likely won’t report warnings again, but there are situations where creating an index with a specified collation or running SQL with specified expression collation may still report warnings.

The coll version in the data dictionary is only for tracking whether the collation provider version has changed at the database layer. Imagine if there were no coll version in the data dictionary - the database might not even be able to return a warning saying “your sort rule provider has upgraded its version, your data sorting might have problems, you need to check it” (and of course it’s not just about sorting).

Solutions for This Problem
#

Corrupt indexes have already been rebuilt, the database has been refreshed, only collation hasn’t been refreshed. The inconsistency of coll version in the data dictionary is not a big problem, it’s just a warning. As for other hidden and strange pitfalls, refer to the more section.

Solution for this problem:

Step 1: Check if there are still dependencies

SELECT pg_describe_object(refclassid, refobjid, refobjsubid) AS "Collation",
 pg_describe_object(classid, objid, objsubid) AS "Object"
 FROM pg_depend d JOIN pg_collation c
 ON refclassid = 'pg_collation'::regclass AND refobjid = c.oid
 WHERE c.collversion <> pg_collation_actual_version(c.oid)
 ORDER BY 1, 2;

If there are returns, it’s best to rebuild the dependent objects; if not, follow step 2:

Solution 1: Do nothing. If there aren’t many warnings, leaving them alone is fine.
Solution 2: Only refresh collation zh_CN.UTF8. Fix one as it comes.
Solution 3: Refresh all collations. Even if the application incrementally uses expressions or index-specified collation, no warnings will be reported.

More
#

Key Summary of glibc Upgrade Related Issues
#

Locale is a very tricky area, and glibc upgrades cause many collation-related problems. Referencing reference materials, here’s a summary of some important points:

pg_collation is obtained from the OS command locale -a; the provider is basically glibc, so you need to look at the glibc version.

In pg_collation, “C” and “posix” have collprovider c, which looks the same as “C.UTF8” etc., but they’re not. “C.UTF8”’s provider is glibc, has a version, generally Unicode codepoint sorting or Unicode semantic sorting; “C” and “POSIX” are equivalent, the most basic locale defined by the POSIX standard, implemented by libc, not in locale -a, has no version, sorts directly by byte order.

Root cause of collation problems: The database requires that locale definitions never change during the database lifecycle, but OS vendors, especially the GNU C library, make changes to locale in every minor version, and this is legitimate.

GNU C library makes changes to locale in every minor version. The version most prone to problems in reality is glibc 2.28, because 2.28 upgraded the major version unicode 9.0.0 (has been updated to a new upstream version from ISO which is in sync with Unicode 9.0.0).

pg has no way to detect compatibility issues caused by glibc upgrades. Index corruption checking is not an all-check, and indexes are only one aspect. After physical replication or upgrade, even if indexes are rebuilt, you cannot rule out the possibility that the database crashes one day due to collation version issues.

Data anomalies include: duplicate primary keys, sort-dependent constraints, range partition table data written to wrong partitions, mergejoin and other sort operations, etc.

Character types depend on collation. Data types that don’t depend on collation:

bytea
tsvector gin indexes
pg_trgm indexes
numeric data types: int, bigint, numeric, float, …
custom data types like geometry (PostGIS)
timestamp

ASCII sorting is relatively common but doesn’t conform to human understanding, i.e., not semantic. Semantically conforming international sorting standards are generally Unicode standards.

Unicode-based sorting rules are divided into 2 types: codepoint sorting, UCA (Unicode Collation Algorithm).

UCA is based on DUCET (Default Unicode Collation Element Table). The DUCET table itself may have sorting changes between different versions. For example, en_US.UTF8 is UCA sorting, equivalent to semantic sorting; version upgrades will change sorting rules. C.UTF8 is codepoint sorting; once codepoints are confirmed they won’t change, and sorting rules won’t change.

PG 17+ provides a very safe locale provider method: builtin, no longer depending on OS-provided glibc, ICU and other providers. Example enable command:

initdb --locale-provider=builtin --bultin-locale=C.UTF-8 dbname1

17 only supports C, C.UTF-8. C is byte-order sorting (approximately ASCII sorting), C.UTF-8 is Unicode codepoint sorting; 18 adds one more PG_UNICODE_FAST, also Unicode codepoint sorting, with slight differences from C.UTF-8.

Because the database must maintain stable sorting, custom application sorting can only be pushed to the application layer. For example, expression sorting is semantically clear and doesn’t affect the database’s own choice of collation. If one day pg also supports built-in en_US.utf8, then we can consider built-in semantic sorting.

During Xinchuang migration, the glibc version of Xinchuang hosts is generally higher than old Intel server glibc versions, likely crossing the 2.28 version. Combined with tight deadlines, KPI pressure, insufficient manpower, and large databases, physical migration is unavoidable. So Xinchuang physical migration needs to pay attention to glibc versions and many anomalies caused by collation.

What to Do After Physical Migration
#

Assuming the database is en_US.utf8, provider c, and physical migration across libc versions has already been done, the following operations should be performed:

I. Official Required Solution

At minimum, rebuild problematic indexes. Install the amcheck extension and use the bt_index_check function:

SELECT bt_index_check('idx1'::regclass, true);

Refresh database version (pg15+):

ALTER DATABASE name REFRESH COLLATION VERSION

Check if there are other dependent objects. If there are, handle them accordingly:

SELECT pg_describe_object(refclassid, refobjid, refobjsubid) AS "Collation",
 pg_describe_object(classid, objid, objsubid) AS "Object"
 FROM pg_depend d JOIN pg_collation c
 ON refclassid = 'pg_collation'::regclass AND refobjid = c.oid
 WHERE c.collversion <> pg_collation_actual_version(c.oid)
 ORDER BY 1, 2;

After handling, then:

Refresh collation version (pg10+):

ALTER COLLATION name REFRESH VERSION

II. Unofficial Workaround Solutions

I haven’t made a complete solution here, just some thoughts.

Handling partition table data written to wrong partition:

Partition key is int/bigint/float, no relation to collation, can be ignored.

Partition key is time partition, if timestamp, can be ignored. If varchar or other character types, depends on the situation.

Partition key is character type, refer to “a” and “-” sorting (pgconf Collation Challenges Sorting It Out). But note the following points:

If querying data, don’t query from the parent table; it might crash or fail to return results.
There’s no simple detection solution.

Handling primary key/unique key conflicts.
Handling fdw sort range anomaly issues.
Unknown problems.

ref
#

https://wiki.postgresql.org/wiki/Locale_data_changes

https://wiki.postgresql.org/wiki/Collations

pgconf Collation Challenges Sorting It Out

PFCONF Collations from A to Z

http://www.unicode.org/reports/tr10/tr10-34.html

https://sourceware.org/glibc/wiki/Release/2.28

https://www.postgresql.org/docs/18/sql-altercollation.html

https://www.postgresql.org/docs/18/sql-alterdatabase.html

https://www.postgresql.org/docs/17/locale.html#LOCALE-PROVIDERS

A Brief Review of Logical Replication in Oracle, MySQL, and PostgreSQL

Sun, 30 Nov 2025 00:00:00 +0000

PostgreSQL Logical Replication
#

（https://www.pgconf.asia/JA/2017/wp-content/uploads/sites/2/2017/12/D2-A7-EN.pdf）

PostgreSQL places all logical decoding related matters entirely within the database’s replication slots for management — an all-inclusive approach. Early versions had somewhat limited logical replication support, but in recent major versions, logical replication has been one of the primary functional improvements.

Advantages of the PG approach:

Very flexible: it exposes the logical decoding interface to users, with multiple types of decoding methods available.
Users can subscribe to only the data they need based on their requirements.

Disadvantages of the PG approach:

The number of concepts to learn and the learning cost are relatively higher compared to MySQL. Just the basic concepts — publication, subscription, walsender, replication slots, output plugins, etc. — I believe many people haven’t fully grasped their definitions and relationships.
Does the hardest work and takes the hardest hits. All logical decoding problems are exposed within the database: WAL backlog, large transactions, long transactions, reorder transaction sorting, privilege issues, streaming transmission — these are all problems PG has to deal with.

MySQL’s binlog
#

(https://blog.fasterinfo.top/6243.html)

MySQL places all decoded logical data locally — in binlog files. The approach is simple. MySQL’s binlog is roughly equivalent to PostgreSQL with full-table logical replication enabled and written locally.

Advantages of the MySQL approach:

Simple and straightforward: MySQL doesn’t expose the logical decoding interface directly to users. Instead, it provides already-decoded files directly to users, who don’t need to care about how parsing works — just read the binlog files.
Mature ecosystem. I personally believe MySQL’s mature ecosystem is closely tied to binlog. During the internet era, PG’s logical replication was still weak, while binlog was extremely simple. Downstream parsing of binlog to put data onto other platforms became a common pattern.

Disadvantages of the MySQL approach:

All data must be decoded; no customizable subscription. Poor flexibility.
Two-phase commit. Because MySQL’s primary-standby replication heavily depends on binlog, binlog data must be fully flushed to binlog files at commit time. A single commit must write two (or two kinds of) logs — binlog and redolog. Dual log writes are one of MySQL’s eternal pain points.

Oracle Logical Replication
#

（https://www.oracle-scn.com/oracle-goldengate-integrated-capture/）

Oracle itself does have logical Data Guard functionality, but virtually no one uses it. Here we’ll only discuss LogMiner. The Oracle database itself provides an interface like LogMiner for parsing logs (e.g., OGG integrated capture mode), but has zero replication link management itself — it relies on third-party tools to create and manage replication links.

Advantages of the Oracle approach:

Only provides a parsing interface, no replication link management. For the database itself, this is very hassle-free.
Pay and you get a solution. Just buy the powerful OGG directly. Don’t say Oracle hasn’t provided a logical replication solution — we not only have one, it’s powerful and highly recognized.

Disadvantages of the Oracle approach:

Relies on third-party software to manage replication links.

In summary, PG’s logical replication is an all-in-one, do-everything approach — very much in the open-source, technical spirit. MySQL’s approach is simple, crude, but effective — somewhat “one-step-to-finish.” Oracle’s approach is: provide an interface and leave everything else to third parties, but from the customer’s perspective, there is a mature solution available.

CXL and PolarDB-CXL

Sun, 30 Nov 2025 00:00:00 +0000

Paper: Unlocking the Potential of CXL for Disaggregated Memory in Cloud-Native Databases

SIGMOD best paper: https://sigmod.org/sigmod-awards/sigmod-best-paper-award/

CXL and PolarDB-CXL
#

What is CXL
#

CXL: An open industry standard, a high-speed interconnect specification formulated by the CXL Consortium (founded in 2019 by tech giants Intel, AMD, ARM, etc.). It represents the evolutionary direction of computing architecture. Currently at CXL 4.0.

Feature	CXL 1.0/1.1	CXL 2.0	CXL 3.0/3.1	CXL 4.0 (latest)
Release	March/Sept 2019	October 2020	August 2022 / November 2023	November 2025
Base Protocol	PCIe 5.0 (32 GT/s)	PCIe 5.0 (32 GT/s)	PCIe 6.0 (64 GT/s)	PCIe 7.0 (128 GT/s)
Max Bandwidth	1TB/s	1TB/s	2TB/s	4TB/s+
Topology Scale	Point-to-point / simple star	Single switch (≤32 nodes)	Multi-level Fabric (4096 nodes)	Ultra-large-scale Fabric

From my research, two descriptions of CXL left the deepest impression:

Memory as a Service
Near-memory computing and expansion

CXL switch: A switching chip, physical hardware. Many vendors are working on industrial implementations. The paper specifically references products from XConn Tech: CXL 2.0 switch. Note that as of November 22, 2025, XConn only has CXL 2.0 switches, no 3.0 products. However, there are products on the market supporting 3.0+ standards, such as Panmnesia CXL 3.2 Fabric Switch.

PolarCXLMem: According to the paper, “the first CXL-switch-based disaggregated memory system.” But the paper also states “we leverage the world’s first CXL switch[50]” — specifically referring to the XConn tech CXL 2.0 switch — and then says “PolarCXLMem is the first CXL-switch-based disaggregated memory.” This can be interpreted in two ways:

The first disaggregated memory system based on CXL switches
The first disaggregated memory system based on XConn tech CXL 2.0 switches

PolarDB-CXL: The paper doesn’t actually use this term, but the industry uses it. It represents “integrate PolarCXLMem into the multi-primary version of PolarDB, known as PolarDB-MP” — essentially “the CXL-upgraded version of PolarDB-MP.” The paper repeatedly uses lengthy phrases but never uses the term polardb-cxl. For convenience, this article uses polardb-cxl to represent its essential meaning.

RDMA vs CXL
#

PolarDB-MP uses RDMA architecture, while PolarDB-CXL uses CXL architecture:

(https://medium.com/@anan.mirji/cxl-switch-vs-rdma-a-technical-comparison-for-high-performance-interconnects-6aaa031cde31)

RDMA architecture is a cross-host distributed interconnect architecture, while CXL architecture is a single-host expanded interconnect architecture.

Key differences:

Dimension	RDMA Architecture	CXL Architecture
Topology	Multi-host + network switch distributed arch	Single-host + CXL switch expanded arch
Communication	Network (InfiniBand/RoCE)	PCIe bus (CXL based on PCIe physical layer)
Core Components	RDMA NIC (dedicated NIC)	CXL Controller, CXL Switch
Resource Ownership	“Remote resources” across independent hosts	“Expanded resources” within the host architecture

CXL’s Advantages
#

CXL’s advantages over RDMA:

Low latency: CXL connects to host or device memory via PCIe; RDMA requires protocol interface conversion between InfiniBand and PCIe.

Instruction support: CXL provides native load/store instructions, allowing the CPU to directly manipulate remote CXL device memory as if it were local memory. RDMA requires reading from remote memory to local memory, processing locally, then writing back to remote memory.

Simplified applications: RDMA requires special interfaces and drivers, needing professionals to design complex programs; CXL provides transparent memory space, greatly simplifying application design.

Memory fusion: CXL 3.0 supports physical hardware-level memory pooling.

Problems with PolarDB-MP and the value CXL provides:

CXL’s critique of MP:

Memory pages are 4-16K, so even when only a small amount of data transfer is needed, data must move between local and shared memory, causing read/write amplification.
Maintaining local memory adds extra memory overhead, reducing throughput.
Recovery is very time-consuming.
RDMA is far better than TCP/IP, but under high concurrency, it suffers from “doorbell register implicit contention” and “cache thrashing” issues.
The database itself must maintain shared memory.

Benefits CXL brings:

Eliminates the “shared memory - local memory” hierarchical memory structure, also eliminating the maintenance overhead and read/write amplification. Because CXL load/store to local memory is fast enough, it allows directly storing all buffer pages.
Uses cache lines (64B) as the minimum transfer unit between CPU cache and main memory, rather than PolarDB-MP’s 4K pages.
Saves main memory. DRAM costs are very high, roughly 40-50% of server/rack costs.
Simplifies system design. Minimal modifications to existing systems are important for commercial database stability.
PolarRecv: An instant recovery system built on CXL. After a database crash, data and metadata remain on CXL, allowing direct reads of consistent state from CXL memory, so recovery is very fast. (This seems similar to how PG’s page cache helps fast startup after a crash.)

DRAM vs RDMA vs CXL:

When data volume is small, RDMA has significantly higher latency than CXL; with larger data, RDMA’s latency improves slightly. Local DRAM access is slightly better than CXL access.

Overall, CXL memory access latency is slightly higher than DRAM but better than RDMA.

Regarding CXL’s higher latency vs DRAM, the paper explains: “database buffer pool operations are more sensitive to bandwidth than latency” — for database memory, bandwidth matters more than latency.

Custom Rack
#

Self-developed physical prototype rack. The left rack integrates two CXL switch-enabled clusters, each connected to memory devices and hosts; the right rack integrates one CXL switch connected to memory devices and hosts.

PolarCXLMem
#

The CXL 2.0 switch supports memory pooling, but the drivers don’t fully support it, so PolarCXLMem still designed its own CXL memory allocation and usage — it’s not fully transparent. PolarCXLMem processes CXL memory into a multi-tenant model, with different host nodes allocated different CXL memory regions.

PolarCXLMem characteristics:

Nodes have their own CXL memory regions; different nodes’ CXL memory does not overlap.
The buffer pool is allocated at database startup (by the CXL mem manager in the diagram) and does not change during runtime.
The memory unit structure in CXL mem is a block, which stores page data and page metadata, including: id (page id), lock state (whether the page is locked for update), prev/next (LRU doubly-linked list), lsn (latest log sequence number of the page).
Free list / in-use list is used for LRU.

Question: PG’s page header has lsn, starting free space pointer, prune xid, etc. What does PolarDB-CXL’s page header structure look like?

PolarRecv
#

PolarDB-MP was designed based on RDMA, where data pages are written locally, and the disaggregated shared memory doesn’t contain the latest version of data pages. This means after a host crash, you must scan and apply all redo log files (the paper says redo, not WAL) or pages from a small amount of shared memory.

CXL switches have independent power, so even if the host crashes, the latest data remains in CXL memory. PolarRecv leverages this to dramatically speed up database recovery after host crashes.

However, while CXL switch memory is transparent and persistent, directly using it after a crash still requires handling these issues:

LRU lists may be inconsistent at crash time
B-tree SMO (B-tree structure changes), such as index splits, may be inconsistent at crash time
Pages being updated at crash time may be inconsistent
The redo log buffer uses local DRAM. When the redo log hasn’t been flushed to disk at crash time, the page LSN in the CXL buffer pool may be greater than the LSN in the redo log file, directly violating the ARIES principle

PolarRecv’s design strategies:

Use mutex to protect the LRU structure. The mutex lock state indicates whether LRU was being modified at crash time. If so, LRU must be rebuilt; if not, use the LRU directly from CXL memory.
During B-tree SMO, a mini-transaction protects index pages. This mini-transaction is a two-phase lock corresponding to page locks. It’s only flushed to the redo log when the mini-transaction commits. So during recovery, if an index page is found with a write lock, recover from the redo logs.
PolarCXL’s read/write locks are stored in CXL memory. If a write lock still exists, it means the update was in an intermediate state at crash time and not completed. In this case, honestly read the page from the redo log file rather than reading an inconsistent page from CXL memory.
During recovery, first obtain the maximum LSN from the redo log, then check the lock and LSN of pages in CXL memory. If a page’s LSN in CXL memory is greater than the max LSN, rebuild the page using redo log information rather than using the CXL memory version.

Memory Fusion
#

Because PolarCXLMem is designed based on the CXL 2.0 switch, and CXL 3.0 supports memory fusion, memory fusion design is still needed. Since each node’s buffer pool is placed in isolation in PolarCXLMem, CXL 2.0’s memory fusion is achieved through DBP metadata management — each buffer pool only stores the page’s CXL memory address, not the page itself.

To understand this diagram, you need to distinguish between CXL memory, DBP, and local buffer:

CXL memory is the physical hardware, CXL mem itself.
DBP is a region carved out of CXL for managing memory fusion services.
Local metadata buffer contains local buffer metadata and part of CXL.

Also understand that for each page in the buffer pool, there are two flags:

invalid: After another node writes to the page, the current node needs to invalidate its local CPU cache.
removal: When a page moves from the in-use list to the free list, all nodes must set the removal flag.

Memory fusion page access flow:

The requested page is not in the local page metadata buffer: 1.1 Allocate a new meta record from the free list, and provide invalid and removal addresses to the memory fusion service via RPC.

The requested page is in the local page metadata buffer: 2.1 First check the removal flag. If removal is set, it means the memory fusion service has already reclaimed the page, and a new memory address must be requested from the memory fusion service via RPC. 2.2 Then check the invalid flag. If invalid is set, it means the page has been modified by another node, and the CPU cache must be invalidated to ensure consistency.

Fusion consistency:

Since CXL 2.0 doesn’t have memory fusion, CPU caches aren’t automatically updated. PolarCXL implements multi-node concurrent write control through page-level locks.

Nodes must acquire read/write locks to read/write pages. When one node is writing to a page, other nodes cannot read or write that page. After a node finishes writing, it must also:

Flush the CPU cache to CXL mem (cache line flush) to ensure CXL mem has the latest page version.
Set the invalid flag to ensure other nodes don’t read stale page versions from their CPU caches.

Memory fusion summary:

CXL 2.0 itself supports incomplete memory fusion, meaning the database layer still needs to design a memory fusion scheme. Memory pages are accessed via CXL addresses, rather than local/remote access to entire pages as in the RDMA approach. The local CPU cache needs the database layer to flush it to ensure node data access consistency — this is a hard limitation. This also means cross-node updates still use exclusive page-level locks (the RDMA approach also uses exclusive page-level locks).

Performance Evaluation
#

Multi-Node Read/Write
#

Benchmarking with 12 instances on a 192 vCPU host, comparing RDMA (PolarDB-MP) vs CXL (PolarDB-MP with PolarCXLMem) performance:

Point queries:

Range queries:

Read-write:

Point queries: Read amplification is most severe for point queries. CXL’s bandwidth consumption is 3-4x lower than RDMA. When reaching 3 nodes, RDMA bandwidth is already saturated — adding more nodes doesn’t improve bandwidth.
Range queries: Read amplification is less severe. Only at >4 nodes does it reach the bandwidth ceiling of 11GB/s, while CXL can still scale linearly with nodes.
Read-write: Performance is similar to range queries, just with smaller differences.

PolarRecv Recovery Time
#

vanilla: Refers to the general approach, probably similar to PG reading from local cache or disk (possibly polar redo).
RDMA-based: Refers to PolarDB-MP where some data can be read from disaggregated shared storage.
PolarRecv: Refers to continuing to read most data from CXL, with only a small amount of partial pages needing recovery from redo files.

The paper discusses recovery time in 2 phases: startup/recovery and reaching pre-crash load levels. Read-only doesn’t need recovery — as long as there’s data, you can start and take load. When writes exist, recovery is needed, and the advantage of continuing to read from CXL memory becomes apparent. The difference between 1-minute, 2-minute, and 4-minute recovery times is significant — it could be the difference between business being nearly imperceptible and noticeably impacted.

Shared Data Updates
#

The focal point of distributed database performance combat is updates to shared data. After PolarDB-MP crushed Taurus-MM, PolarDB-CXL also crushed PolarDB-MP:

At 0% shared data, the RDMA-based solution just accesses local buffers, and PolarDB-CXL just treats CXL as a memory pool. Even so, CXL-based still performs better, mainly due to the read/write amplification and bandwidth ceiling issues of the RDMA-based solution mentioned earlier.

From the performance comparison chart above, it’s clear that PolarDB-CXL significantly outperforms PolarDB-MP. The data is very clear. However, note that when shared data >60%, PolarDB-CXL’s performance improvement becomes less significant, mainly because:

Page-level locks become the bottleneck.
As lock contention intensifies, processes enter sleep states, and frequent context switching further exacerbates resource contention.

Summary
#

PolarDB-CXL advantages:

Eliminates RDMA’s “local-remote” hierarchical memory structure design.
Resolves RDMA’s read/write amplification problem.
Provides a CXL-based memory pool.
PolarRecv, based on CXL persistent memory, enables faster database crash recovery.
Benchmarking shows PolarDB-MP CXL outperforms PolarDB-MP RDMA.

PolarDB-CXL disadvantages:

Cross-node updates still use page-level locks, which remain the main performance bottleneck in shared data update scenarios.
The CXL 2.0 switch seems a bit dated — by the time the paper was published, switch devices supporting 3.2 were already available, and CXL 4.0 was announced in November 2025. We can predict future databases built on newer CXL standard switch devices.
The paper quality isn’t actually as high as the MP paper — it mainly revolves around solutions for the CXL 2.0 switch physical hardware, which differs from the extensive database-layer design found in the PolarDB-MP paper.

Original link: https://lastdba.com/2025/11/30/论文精读polar-db-cxl2025-sigmod最佳工业论文/

Paper Deep Read: PolarDB-MP | 2024 SIGMOD Best Industrial Paper

Sun, 30 Nov 2025 00:00:00 +0000

Paper: PolarDB-MP: A Multi-Primary Cloud-Native Database via Disaggregated Shared Memory

SIGMOD best paper: https://sigmod.org/sigmod-awards/sigmod-best-paper-award/

Foreword and Abstract
#

The paper opens with the problem: primary-replica architecture’s write throughput is limited by the primary. Shared-nothing architecture offers scalable multi-primary clusters that can solve the single-primary limitation, but this architecture suffers performance bottlenecks due to distributed transaction overhead. Recently, shared-storage-based cloud-native multi-primary databases have emerged, but under high-conflict scenarios, they face high conflict resolution costs and low data fusion efficiency.

So the problem is: single-primary primary-replica, shared-nothing, and shared-storage cloud-native multi-primary architectures all have their own issues.

This paper proposes PolarDB-MP, a novel multi-primary cloud-native database combining disaggregated shared memory with shared storage. (Since multi-primary cloud-native databases already exist, it needs to be “novel.”)

PolarDB-MP’s basic characteristics:

All nodes can equally access all data, allowing transactions to be processed independently on a single node, without traditional distributed transaction mechanisms.
Shared storage: PolarStore and PolarFS, or other compatible shared storage solutions.
Built on disaggregated shared memory.
Low-latency communication via RDMA (Remote Direct Memory Access).
LLSN (Local Logical Sequence Number): Used to establish partial order for WAL logs generated by different nodes, combined with custom recovery strategies to ensure consistency and efficiency during abnormal recovery.
Core component PMFS (Polar Multi-Primary Fusion Server) responsible for:
- Transaction Fusion — transaction ordering and visibility management
- Buffer Fusion — distributed shared buffer mechanism
- Lock Fusion — cross-node concurrency control

Classification
#

The classification is mainly to understand PolarDB-MP’s historical position and the “first” qualifier:

PolarDB-MP is the first multi-primary cloud-native database that utilizes disaggregated shared memory and shared storage for transaction coordination and buffer fusion

Competitor Weaknesses
#

Shared-nothing products: The paper doesn’t call out individual products, just one line: transactions accessing across multiple partitions require significant additional overhead for distributed transactions.

Oracle:

Expensive distributed lock management
Expensive network overhead
Reliance on sophisticated hardware (alien tech)
Difficult to migrate to cloud, or higher TCO (including maintenance and labor costs) compared to cloud-native databases after migration

AWS Aurora-MM:

Uses optimistic transaction model; high transaction abort rates under conflicts
In some scenarios, 4-node throughput is lower than single-node

Huawei Taurus-MM:

Pessimistic transaction model. Relies on page storage and log replay to ensure cache consistency, with high overhead in concurrency control and data synchronization.
Under 50% shared data read-write workload, 8 nodes only achieve 1.5x single-node performance improvement

The Oracle critique here is mainly plausible-sounding trash talk, while Aurora-MM and Taurus-MM have original vendor citations:

Aurora-MM “in some scenarios, 4-node throughput is lower than single-node”
Taurus-MM “under 50% shared data read-write workload, 8 nodes only achieve 1.5x single-node performance improvement”

Transaction Fusion
#

Transaction Fusion Overview
#

How does multi-primary ensure consistent data views?

Snapshot isolation is a common MVCC implementation. A characteristic of snapshot isolation is that queries or transactions must maintain their consistent data view during execution. But in multi-primary architecture, local nodes cannot guarantee consistent data views due to remote data updates.

To solve this, general multi-primary shared-storage architectures introduce global transaction mechanisms (Aurora-MM or Taurus-MM). PolarDB-MP introduces an innovative technique — transaction fusion within PMFS. Each node only maintains local transaction information, which can be accessed by other nodes via RDMA. In contrast to global transactions, transaction fusion is decentralized.

Local Transactions and TIT Table
#

Each node in PolarDB-MP maintains a small amount of memory to store local transaction information (accessible by other nodes via RDMA). This local transaction information is stored in the transaction Information Table (TIT).

TIT table contents:

Transaction object pointer
Commit timestamp (CTS) assigned by the global timestamp coordinator (TSO)
version, representing different transactions in the same slot
ref, indicating whether this transaction is being waited on by other transactions for lock release (probably PLock or RLock)

How Transactions Proceed
#

When a transaction begins, a local transaction id (presumably txid) is assigned, and the TIT slot stores the transaction object pointer, ref initialized to 0, and CTS initialized to CSN_INIT.

PolarDB-MP uses a global transaction ID to identify a transaction: global transaction ID = (node_id, trx_id, slot_id, version). The global transaction ID does not include CTS. To know the commit order of transactions, such as when constructing a transaction visibility view, you need to go through the global transaction ID, via RDMA, to the target node to find CTS (similar to PG’s pg_xact_commit_timestamp() function, which finds the corresponding transaction commit time from local files using the transaction id).

If trx_id is the transaction ID in PG, then node_id + trx_id can identify the global uniqueness of a transaction, or node_id + slot_id + version could also work to some extent (when slot id is not reused, e.g., at a given moment it uniquely identifies a transaction). Of course, the extra information combined is also unique. After all, this information is key to PolarDB-MP’s transaction fusion implementation.

Each transaction constructs a visibility view using the global transaction ID and CTS. The visibility view concept is consistent with PG: the current read view can read data rows committed before the read view, and the latest version rows.

Accessing Remote CTS
#

Since CTS is local (in TIT or on the local filesystem), obtaining the reading transaction’s CTS is an interesting task:

1.1 If a row’s CTS is CSN_INIT/CTS_INIT, meaning the transaction is still active, return the maximum CTS to indicate it’s invisible to all transactions except itself.

If a row’s CTS is not CSN_INIT/CTS_INIT, meaning the transaction has committed, and it’s in the local TIT, directly return CTS.
If a row has no CTS, obtain CTS via the row’s g_trx_id.

2.1 If the transaction belongs to the local node (g_trx_id has node id), read from local filesystem to local TIT.

2.2 If the transaction doesn’t belong to the local node, read from remote filesystem to remote TIT via RDMA.

3.1 If slot.version != g_trx_id.version, the transaction must have committed, so the row is definitely visible to all transactions. Return minimum CTS to indicate visibility to all transactions.

3.2 If slot.version = g_trx_id.version, refer to 1.1, 1.2.

PolarDB-MP’s transaction visibility concept is very similar to PG’s, except PG uses txid instead of CTS to indicate transaction ordering and doesn’t need to consider remote access.

Row Update Transactions
#

Additionally, row updates are also very similar:

When PolarDB-MP updates a row, besides updating the data itself, it must also:

Update the row’s global transaction ID (g_trx_id) (if it’s an in-row update, then it modifies PG’s row header).
Update the row’s CTS. (The paper doesn’t specify whether this is in the row header or filesystem. If similar to PG, it should be in the commit_ts directory on the filesystem. Polar not confirmed.)

Questions About Transaction Fusion (Things I Didn’t Understand)
#

g_trx_id is row metadata written to disk. If nodes are added or removed, does the node_id in the data row’s g_trx_id need updating? If not, which node should the row be loaded into when read next time?

A new row’s CTS is stored on local node A. If another node B updates this row, is the new CTS on node A or B?

“assigned a read view, which consists of its own g_trx_id and the current CTS.” Do read-only transactions also get assigned a g_trx_id when constructing a read view?

Without a doubt, a parameter like track_commit_timestamp must be forcibly enabled.

If there are many writes on node A and reads on node B, B’s reads will access A’s TIT data via RDMA — does this generate significant network IO? Should this be considered when designing read-write separation or multi-node reads and writes? The original paper might answer this — “Multi-primary architectures inherently require synchronizing large amounts of data and messages between nodes to support concurrent access across multiple nodes. As network technology develops (InfiniBand, RDMA) and achieves commercial deployment, the network bottleneck becomes less significant.”

Global timestamps could become a bottleneck in distributed systems. PolarDB-SCC is a shared-storage-based timestamp solution that appears to perform well. Due to time constraints, I’ll set this aside for now.

Buffer Fusion
#

Buffer Fusion Introduction
#

Each node in PolarDB-MP can update any data page, leading to substantial data transfer. Buffer Fusion’s distributed buffer pool (DBP) is designed to solve this problem. Each node has a local buffer pool (LBP), which is a subset of DBP.

How Buffer Fusion Works
#

LBP has two new metadata items for pages:

valid: whether the page has been updated by another node
r_addr: pointer to the page in DBP

When accessing a page from LBP, the current node must first check if the page is valid. If invalid, it must access DBP via r_addr. After DBP stores a new version of the page, buffer fusion invalidates all remote pages. In LBP, dirty pages are periodically flushed to DBP in the background or after releasing the PLock lock.

Page access steps:

1.1 If the page is in LBP and valid, access directly. 1.2 If the page is in LBP and invalid, access DBP via RDMA. 2. If the page is in neither LBP nor DBP, read from shared storage. 3. The page is loaded from a node into LBP and registered in DBP.

PolarDB’s buffer fusion key component is disaggregated shared memory. It appears to be a/group of physical hardware or an integrated component built on top of it, separate from compute nodes. This differs significantly from memory in traditional distributed systems.

It’s also different from transaction fusion: transaction fusion requires accessing remote nodes with the same architecture, while buffer fusion doesn’t require accessing remote nodes with the same architecture — it separately accesses the disaggregated shared storage component.

Questions About Buffer Fusion (Things I Didn’t Understand)
#

Disaggregated shared memory seems like a component separate from standard hosts — so what exactly is it?

Lock Fusion
#

Lock Types in Lock Fusion
#

Buffer fusion solves how nodes access remote data; lock fusion solves concurrent access control.

Buffer fusion has two types of locks:

page-locking (PLock): Similar to latches, controlling atomic access and internal structure consistency. Single-node page access doesn’t use PLock.
row-locking (RLock): Responsible for cross-node transaction control, following the two-phase lock protocol.

PLock Access Flow
#

(The paper doesn’t say where lock fusion occurs. Since PLock is a page-level latch and page fusion happens on shared memory, I’ll assume lock fusion also occurs on shared memory, as this is easier to understand.)

Before updating/reading a page, the local lock manager checks whether the local node already holds the corresponding X/S PLock (or higher-level lock). 1.1 If yes, execute in place. 1.2 If no, acquire PLock through Lock Fusion.
Lock fusion checks for conflicts before responding; if a conflict exists, the request waits.
When PLock is released by a node, it notifies Lock Fusion, which updates PLock’s state and notifies other nodes to continue their operations.

PLock Lazy Releasing
#

According to the PLock access flow above, a PLock is immediately released after local operations complete. This may not be optimal — according to temporal locality: “a data item or instruction accessed at a given time is likely to be accessed again in the near future.” Lazy releasing minimizes PLock lock RPC access load.

The principle is simple: PLock is not immediately released after use on the local node; it’s only released when ref reaches 0.

When other nodes need PLock, Lock Fusion also sends negotiation messages to intervene when the local node is holding the lock; the local node must communicate with Lock Fusion rather than autonomously handling PLock. Lock Fusion uses a “first-in-first-out” strategy to resolve cross-node lock ownership, again until the local node’s ref = 0, at which point other nodes can acquire the lock.

Lazy releasing is an effective distributed lock solution, balancing local lock optimization with global lock allocation.

RLock Overview
#

RLock uses the global transaction ID for determination (similar to PG). According to the transaction fusion content, the global transaction ID contains node id, transaction id, slot id, version. So when a local node reads a row, it can directly obtain the lock information on the row, know where the lock is (node id), and know if the lock is active.

There are two interesting points about determining transaction activity:

From the transaction fusion flow of accessing remote CTS: if the transaction’s CTS is a valid value, or the transaction is in the same slot in TIT but not the same version, the transaction has definitely committed, so no need to check activity. If the source transaction is not active, there’s no need to wait for locks — proceed directly.
PG has the concept of a minimum active transaction ID, which also exists in PolarDB-MP. If the transaction ID on the row is less than the global minimum active transaction ID, the source transaction must have also committed (or rolled back).

How RLock Works
#

Local rows are handled locally; only conflicts are processed in Lock Fusion; cross-node row locks require RLock. “The transaction ID in the row functions as a lock indicator. So this protocol only supports exclusive (X) lock. The shared (S) lock on a row is not supported in PolarDB-MP, but it’s acceptable.” Only truly conflicting exclusive locks need RLock; shared locks don’t need RLock.

T30 reads the row from shared storage and can determine from the row’s metadata (g_trx_id) that the transaction is active and which node it’s on.
T30 remotely adjusts T10’s transaction ref.
T30 sends a wait status to the Lock Fusion service.
Lock Fusion adds wait information to the wait info table.
T10 finishes execution and notifies Lock Fusion.
Lock Fusion checks the wait info table, then notifies T30 it can continue.

Questions About Lock Fusion (Things I Didn’t Understand)
#

“when attempting to update a row, it must already hold an X PLock lock on the page containing the row”

Updating also requires holding an exclusive PLock on the page, meaning updates on the same page block each other — doesn’t this affect concurrency? Locally, there shouldn’t be such behavior; PG doesn’t have page-exclusive locks for update scenarios.

In the “Logs ordering and recovery” chapter, there are two statements: “Thanks to the PLock design, only one transaction can update a page at a time” and “When a page is updated across two nodes, one node pushes its updated page to the DBP before releasing the PLock, allowing the next node to retrieve it from the DBP.”

Yes, during cross-node data updates, there are page-level exclusive locks.

PMFS Summary (Hot Take)
#

PMFS (Polar Multi-Primary Fusion Server) is the core component implementing PolarDB-MP’s multi-primary distributed system. Among its features, the global transaction ID design is ingenious — it transforms PG’s transaction ID into one containing node information, transaction id, and transaction fusion’s slot and version information, placed in the row header. This has several benefits:

Directly accessing a row reveals the row’s version ordering.
Directly accessing a row reveals which node updated it.
Directly accessing a row reveals whether cross-node locks may exist.
Uses minimum active transactions to reduce conflict determination.
Uses global transaction ID information to achieve distributed retrieval of transaction commit timestamps (CTS).

Additionally:

Buffer fusion and lock fusion in PMFS appear highly dependent on the shared memory component.
RDMA is omnipresent throughout.

Log Ordering
#

Partial Order
#

First, WAL is generated on each node without any concurrency control mechanism — each writes independently to shared storage. Each node’s LSN is sequential for that node, but across multiple nodes, WAL records don’t exhibit global ordering.

But is global ordering needed when writing WAL records?

From the paper, most of the time it’s not needed.

Only one case requires guaranteed global ordering during writing: cross-node updates to the same page.

However, according to the PMFS lock fusion mechanism, cross-node updates to the same page are exclusive. Lock fusion can ensure the ordering of cross-node page updates.

Recovery Ordering
#

Since LLSNs from cross-node writes come from multiple nodes and are likely not in order, recovery needs to be done in order. Reading all WAL records and sorting by LLSN is a simple approach, but massive sorting is very resource-intensive.

PolarDB-MP proposes segment-wise sorting of LLSN — each segment is called a chunk, with chunk boundaries called LLSN bounds. PolarDB-MP can guarantee that an LLSN bound is always less than the next bound, then sort LLSNs within each chunk.

Questions About Log Ordering (Things I Didn’t Understand)
#

“utilizing redo (write-ahead) logs for data recovery and undo logs for rolling back uncommitted changes”

PolarDB-MP has undo log files? What is this undo for?

I didn’t see anything particularly special about LLSN; the paper doesn’t detail its structure. LSN seems sufficient — maybe there are differences regarding global transaction IDs.

Evaluation
#

Read-only operations are all local, so adding nodes linearly increases throughput. If read-write/write-only data is well-partitioned and doesn’t cross nodes, it’s also nearly linear.

The problem lies in shared data across read-write/write-only nodes, which is the ultimate test of distributed database performance.

The paper directly compares against Huawei’s Taurus-MM. The conclusion: PolarDB-MP’s cross-node write performance is indeed significantly better.

Nitpicking
#

The paper mentions Taurus-MM’s performance improvement under 8-node shared data in two places, but the data is inconsistent:

The eight-node cluster only improves the throughput by 1.8× compared to the single-node version in the read-write workload with 50% shared data.

the throughput of Taurus-MM’s eight-node cluster is approximately 1.8× that of a single node under the SysBench write-only workload with 30% shared data, illustrating the trade-offs and challenges in optimizing multi-primary cloud databases

Sometimes 30% shared data, sometimes 50% — not very rigorous. The original Taurus MM paper says 50%:

Summary
#

Not much to summarize — see the Foreword and Abstract and PMFS Summary sections.

Original link: https://lastdba.com/2025/11/30/论文精读polar-db-mp2024-sigmod最佳工业论文/

Case: From Inaccurate DISTINCT to the Principles of DISTINCT Estimation

Sun, 19 Oct 2025 00:00:00 +0000

Problem Description
#

The n_distinct statistic was severely inaccurate.

This problem appeared across multiple databases. For example:

A table with 200 million rows and a true DISTINCT count of 8 million had a statistics DISTINCT value of only 40,000.

Analysis
#

Sampling Model
#

The default default_statistics_target=100 means 30,000 rows are sampled from 30,000 pages.

analyze verbose tablzl1;
INFO: 00000: analyzing "public.tablzl1"
LOCATION: do_analyze_rel, analyze.c:332
INFO: 00000: "tablzl1": scanned 30000 of 22963751 pages, containing 1061942 live rows and 3953 dead rows; 30000 rows in sample, 812872389 estimated total rows
LOCATION: acquire_sample_rows, analyze.c:1340

Note “scanned 30000” and “30000 rows in sample”.

DISTINCT Estimation Algorithm
#

The DISTINCT estimation algorithm in analyze.c:

			/*----------
			 * Estimate the number of distinct values using the estimator
			 * proposed by Haas and Stokes in IBM Research Report RJ 10025:
			 *		n*d / (n - f1 + f1*n/N)
			 * where f1 is the number of distinct values that occurred
			 * exactly once in our sample of n rows (from a total of N),
			 * and d is the total number of distinct values in the sample.
			 * This is their Duj1 estimator; the other estimators they
			 * recommend are considerably more complex, and are numerically
			 * very unstable when n is much smaller than N.
			 *
			 * In this calculation, we consider only non-nulls. We used to
			 * include rows with null values in the n and N counts, but that
			 * leads to inaccurate answers in columns with many nulls, and
			 * it's intuitively bogus anyway considering the desired result is
			 * the number of distinct non-null values.
			 *
			 * We assume (not very reliably!) that all the multiply-occurring
			 * values are reflected in the final track[] list, and the other
			 * nonnull values all appeared but once. (XXX this usually
			 * results in a drastic overestimate of ndistinct. Can we do
			 * any better?)
			 *----------
			 */
			int			f1 = nonnull_cnt - summultiple;
			int			d = f1 + nmultiple;
			double		n = samplerows - null_cnt;
			double		N = totalrows * (1.0 - stats->stanullfrac);
			double		stadistinct;

n*d / (n - f1 + f1*n/N)

n = number of sample rows (rows scanned)
d = number of distinct values found in the sample
f1 = number of values appearing exactly once in the sample
N = total number of rows in the table

Algorithm paper: https://hugepdf.com/download/download-extended-version-of-this-paper_pdf

The paper is rather dense, so let’s work through some assumptions to understand this DISTINCT algorithm:

Assume all values appear exactly once, and the table is large (n « N), so f1 = d, n/N ≈ 0

d*d / (d - d + d*0) = d²/0 — this would evaluate to -1.

Assume all values appear exactly once, and the table is small (n = N), so f1 = d, n/N = 1

n*d / (n - d + d*1) = d — the sampled distinct count, which equals the number of sampled rows.

Assume no values appear exactly once in the sample, i.e., f1 = 0

n*d / (n - f1 + f1*n/N) = n*d / n = d — just the distinct count in the sample.

If a column is populated by inserting several rows of the same value, then several rows of another value, like:

11, 2, 2, 2, 2, 3, 3, 3, …

3.1 Small table, all 30,000 rows sampled, true distinct = 10,000 (assumed): estimated distinct = d = 10,000

3.2 Large table, sample contains both repeating values and singletons (some repeating values only have one row captured), i.e., n = 30,000, n/N ≈ 0

n*d / (n - f1 + f1*n/N) = n*d / (n - f1) = 30000*d/(30000-f1) — the larger the distinct count in the sample, the larger the estimated distinct; the larger the number of singletons, the larger the estimated distinct.

Summary:

DISTINCT estimation is directly related to the distinct count and singleton count in the sample
If the singleton count = 0, then larger samples yield larger estimated distinct values

Verification
#

Since the default maximum sample size is 30,000 rows, for tables larger than this, the estimator is likely to underestimate DISTINCT. Note: the data should not have too many unique values.

Testing a table with different sample sizes:

Table: reltuples = 800 million, relpages = 20 million, size = 175GB, true column distinct = 100 million

target statistics	pages sampling ratio (approx)	tuples sampling ratio (approx)	n_distinct	execution time
50	0.00075	0.00001875	60k	2s
100	0.0015	0.0000375	110k	5s
1000	0.015	0.000375	1.03M	58s
3000	0.045	0.001125	2.68M	3min 1s
10000	0.15	0.00375	6.75M	7min 21s

(maximum target statistics is 10000)

A rough conclusion: n_distinct and ANALYZE execution time grow proportionally with the sample size.

n_distinct grows with sample size, while pages and tuples estimates remain consistently accurate.

Solution
#

For extremely large tables, consider partitioning or optimizing based on actual SQL patterns.

You can also adjust the statistics target. The default default_statistics_target=100 means 30,000 rows from 30,000 pages.

Temporary fix:

set default_statistics_target=3000;
analyze tab1;

Long-term fix:

alter table tab1 alter column col1 set STATISTICS 3000;

Notes:

Column-level statistics target has the highest priority, overriding default_statistics_target
Maximum statistics target is 10000
The table’s sampling target is determined by the maximum column target:

	/*
	 * Determine how many rows we need to sample, using the worst case from
	 * all analyzable columns. We use a lower bound of 100 rows to avoid
	 * possible overflow in Vitter's algorithm. (Note: that will also be the
	 * target in the corner case where there are no analyzable columns.)
	 */
	targrows = 100;
	for (i = 0; i < attr_cnt; i++)
	{
		if (targrows < vacattrstats[i]->minrows)
			targrows = vacattrstats[i]->minrows;
	}
	for (ind = 0; ind < nindexes; ind++)
	{
		AnlIndexData *thisdata = &indexdata[ind];

		for (i = 0; i < thisdata->attr_cnt; i++)
		{
			if (targrows < thisdata->vacattrstats[i]->minrows)
				targrows = thisdata->vacattrstats[i]->minrows;
		}
	}

If ANALYZE collects more or fewer rows than expected, check pg_statistic for per-column stattarget settings:

select attrelid::regclass,attname,attstattarget from pg_attribute where attrelid = 'tab1'::regclass and attstattarget not in (-1,0);

Summary
#

For large tables where columns are non-unique but have high distinct counts (a realistic scenario), the sampling algorithm underestimates the DISTINCT value, and this is positively correlated with the sampling ratio. The default sampling ratio is too small for large tables. You can increase it, but even the maximum is not that large.

Case Study: Performance Degradation After Adding an Index and the Generic Plan

Sat, 13 Sep 2025 00:00:00 +0000

Problem Description
#

An index was added the night before, and the next morning the CPU was maxed out. The problematic SQL was easy to locate — just one query. The SQL was running for over 30 seconds, but the day before it only took about 3 seconds, so we needed to examine the before-and-after execution plan changes.

Only the key parts of the execution plan are shown below.

Execution plan before adding the index:

 -> Nested Loop (cost=19.92..2259694.20 rows=265822 width=33)
 -> Index Scan using uk_lzl_task on lzl_task t (cost=0.29..20007.99 rows=195 width=24)
 Filter: ((created_by)::text = 'LIUZHILONG62'::text)
 -> Append (cost=19.63..11337.15 rows=14842 width=57)
 -> Bitmap Heap Scan on lzl_202501 cc_1 (cost=19.63..3053.69 rows=1467 width=66)
 Recheck Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Bitmap Index Scan on lzl_202501_task_no_idx (cost=0.00..19.27 rows=1594 width=0)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 -> Bitmap Heap Scan on lzl_202502 cc_2 (cost=21.67..3066.85 rows=1604 width=66)
 Recheck Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Bitmap Index Scan on lzl_202502_task_no_idx (cost=0.00..21.27 rows=1605 width=0)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 -> Index Scan using lzl_202503_task_no_idx on lzl_202503 cc_3 (cost=0.43..1362.61 rows=1637 width=57)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202504_task_no_idx on lzl_202504 cc_4 (cost=0.43..604.64 rows=1795 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202505_task_no_idx on lzl_202505 cc_5 (cost=0.43..445.30 rows=1450 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202506_task_no_idx on lzl_202506 cc_6 (cost=0.43..583.94 rows=1675 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202507_task_no_idx on lzl_202507 cc_7 (cost=0.43..633.45 rows=1973 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202508_task_no_idx on lzl_202508 cc_8 (cost=0.43..619.43 rows=1720 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))
 -> Index Scan using lzl_202509_task_no_idx on lzl_202509 cc_9 (cost=0.42..893.03 rows=1521 width=56)
 Index Cond: ((task_no)::text = (t.task_no)::text)
 Filter: ((created_date > '2025-01-07 09:00:00'::timestamp without time zone) AND (created_date < '2025-09-03 12:56:44.973'::timestamp without time zone))

The created_date time range searches for data within 1 year. The index added the night before was on created_date.

Execution plan after adding the index:

 -> Hash Join (cost=63.37..23740.82 rows=191 width=33)
 Hash Cond: ((cc.task_no)::text = (t.task_no)::text)
 -> Append (cost=0.00..23376.98 rows=114435 width=58)
 Subplans Removed: 28
 -> Index Scan using idx_lzltab_202501_created_date on lzltab_202501 cc_1 (cost=0.43..1450.59 rows=8958 width=66)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202502_created_date on lzltab_202502 cc_2 (cost=0.43..1822.73 rows=7405 width=66)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202503_created_date on lzltab_202503 cc_3 (cost=0.43..1430.03 rows=7917 width=57)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202504_created_date on lzltab_202504 cc_4 (cost=0.43..2412.44 rows=11041 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202505_created_date on lzltab_202505 cc_5 (cost=0.43..2260.73 rows=13381 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202506_created_date on lzltab_202506 cc_6 (cost=0.43..3930.10 rows=17832 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202507_created_date on lzltab_202507 cc_7 (cost=0.43..3878.77 rows=21786 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202508_created_date on lzltab_202508 cc_8 (cost=0.43..4736.72 rows=22033 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Index Scan using idx_lzltab_202509_created_date on lzltab_202509 cc_9 (cost=0.42..627.09 rows=1893 width=56)
 Index Cond: ((created_date > $1) AND (created_date < $2))
 -> Hash (cost=63.03..63.03 rows=27 width=24)
 -> Bitmap Heap Scan on ai_outbound_call_task t (cost=2.99..63.03 rows=27 width=24)
 Recheck Cond: ((created_by)::text = ($3)::text)
 -> Bitmap Index Scan on idx_ai_call_task_c (cost=0.00..2.99 rows=27 width=0)
 Index Cond: ((created_by)::text = ($3)::text)

The new execution plan switched from using the task_no index to using the created_date index, and changed from a Nested Loop to a Hash Join. The cost dropped from 2,259,694 to 23,740 — a 100x reduction. However, the actual execution time increased by roughly 10x.

Problem Diagnosis
#

Let’s work through three questions to analyze and diagnose the issue:

Why did the optimizer suggest the created_date index?
Why did it end up using the new index?
Why is the estimated row count very small even though the actual execution time is very long?

Why Did the Optimizer Suggest the created_date Index?
#

If we directly substitute the parameters from the PostgreSQL log into the SQL text, the execution plan is actually the good one — the one that runs in 3 seconds using the task_no index. The optimization engineer also ran it this way and found it to be fine. But in production, this wasn’t the execution plan that was used.

Even when we force PostgreSQL not to use the task_no index, the optimizer chooses a sequential scan rather than the created_date index:

 Hash Cond: (((cc.task_no)::text || ''::text) = (t.task_no)::text)
 -> Append (cost=0.00..2794425.58 rows=22238757 width=57)
 -> Seq Scan on lzltab_202501 cc_1 (cost=0.00..193060.05 rows=1585238 width=66)
 Filter: ((created_date > '2025-01-08 11:00:00'::timestamp without time zone) AND (created_date < '2025-09-04 08:31:43'::timestamp without time zone))
 -> Seq Scan on lzltab_202502 cc_2 (cost=0.00..178567.54 rows=1480969 width=66)
 Filter: ((created_date > '2025-01-08 11:00:00'::timestamp without time zone) AND (created_date < '2025-09-04 08:31:43'::timestamp without time zone))
 -> Seq Scan on lzltab_202503 cc_3 (cost=0.00..191073.34 rows=1583356 width=57)

This is very strange: no matter how we ran it ourselves, we couldn’t get it to use the bad created_date index. So how did production end up using it?

The answer lies in bind variables — it was likely a generic plan.

Characteristics of the generic plan:

When plan_cache_mode = auto, PostgreSQL compares the generic plan cost against the average cost of the first five hard parses (custom plans). If the generic plan has a lower cost, it is used and subsequent executions skip hard parsing; otherwise, every execution undergoes hard parsing (see the source function choose_custom_plan).
What the generic plan looks like has nothing to do with the actual bind variable values.

This is easy to reproduce using bind variables via PREPARE/EXECUTE:

PREPARE sql1(timestamp without time zone,timestamp without time zone,text) AS
SELECT COUNT(*)
xxxxxxx...;

=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 367.220 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 254.386 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 235.343 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 234.110 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 233.570 ms
=# EXECUTE sql1('2025-01-08 11:00:00','2025-09-04 08:31:43','LIUZHILONG62'); 
 count 
-------
 12016
(1 row)

Time: 70678.344 ms (01:10.678) -- 6th execution is significantly slower


=# select * from pg_prepared_statements\gx -- pg14 supports pg_prepared_statements
generic_plans | 1
custom_plans | 5

The first 5 hard parses (custom plans) all executed quickly. The 6th execution used the generic plan, which used the created_date index — this was the exact production failure plan, which was extremely slow.

So while the optimization suggestion to use the created_date index was somewhat problematic, when you substituted bind variables with actual values and ran EXPLAIN, the execution plan was correct. In production, however, the application used bind variables, and the generic plan kicked in — causing the failure.

Why Is the Estimated Row Count Small But the Actual Execution Time Very Long?
#

The failing execution plan has a problem: the estimated cost is too small, and the estimated rows are too few.

 -> Index Scan using idx_lzltab_202501_created_date on lzltab_202501 cc_1 (cost=0.43..1450.59 rows=8958 width=66)
 Index Cond: ((created_date > $1) AND (created_date < $2))

From a business logic perspective, this looks abnormal. The created_date condition spans multiple partitions, and since created_date is the partition key, WHERE created_date >= xx AND <= yy must be contiguous. The selectivity on a sub-partition should always be 1, meaning rows should equal the sub-partition row count — several million, not several thousand.

At first I thought it was a statistics issue, but the statistics were fairly accurate — the historical partition data for 202501 hadn’t changed.

Since this is a generic plan issue, we need to examine the generic plan cost estimation by reading the source code. Cost estimation is more complex, but rows estimation is relatively easier to understand and locate.

static double
calc_rangesel(TypeCacheEntry *typcache, VariableStatData *vardata,
			 const RangeType *constval, Oid operator)
{
...
		else
		{
			/* with any other operator, empty Op non-empty matches nothing */
			selec = (1.0 - empty_frac) * hist_selec;
		}
	}

	/* all range operators are strict */
	selec *= (1.0 - null_frac);

range_select = (1 - null_frac) * histogram_selectivity. The range histogram selectivity looks at the histogram buckets hit by the range plus any matching MCV entries. However, we don’t need to compute all this for this case.

Because the generic plan does not look at the histogram:

/*
 * rangesel -- restriction selectivity for range operators
 */
Datum
rangesel(PG_FUNCTION_ARGS)
{
...
	/*
	 * If we got a valid constant on one side of the operator, proceed to
	 * estimate using statistics. Otherwise punt and return a default constant
	 * estimate. Note that calc_rangesel need not handle
	 * OID_RANGE_ELEM_CONTAINED_OP.
	 */
	if (constrange)
		selec = calc_rangesel(typcache, &vardata, constrange, operator);
	else
		selec = default_range_selectivity(operator);
...
}

calc_rangesel is the selectivity calculation function that takes constant values (used above). The else branch calls default_range_selectivity, which does not pass any constants.

/*
 * Returns a default selectivity estimate for given operator, when we don't
 * have statistics or cannot use them for some reason.
 */
static double
default_range_selectivity(Oid operator)
{
	switch (operator)
	{
 ...
		case OID_RANGE_CONTAINS_ELEM_OP:
		case OID_RANGE_ELEM_CONTAINED_OP:

			/*
			 * "range @> elem" is more or less identical to a scalar
			 * inequality "A >= b AND A <= c".
			 */
			return DEFAULT_RANGE_INEQ_SEL;
	}
}

The default range selectivity define:

/* default selectivity estimate for range inequalities "A > b AND A < c" */
#define DEFAULT_RANGE_INEQ_SEL	0.005

Let’s verify this against the production row estimate:

 select reltuples::bigint*0.005 from pg_class where relname='lzltab_202501'\gx
-[ RECORD 1 ]------
?column? | 8958.350

This matches the actual estimated rows of 8958:

idx_lzltab_202501_created_date on lzltab_202501 cc_1 (cost=0.43..1450.59 rows=8958 width=66)

So the new execution plan’s inaccurate estimate is because the generic plan uses a default selectivity of 0.005.

Summary
#

Why Does the Generic Plan Exist, and the Problem with Soft Parsing
#

It’s easier to think of the generic plan as a “DEFAULT estimate plan.”

Why does the generic plan always seem to have problems?

Let’s trace the reasoning chain:

The generic plan exists to reduce hard parsing, i.e., to enable soft parsing.
If we don’t hard-parse every execution, we can reuse an execution plan without passing specific parameter values.
If we don’t pass parameters and directly use an execution plan, that plan must be generated in advance.

Ways to generate an execution plan in advance:

A parameter-less execution plan (the generic plan)
Reuse an execution plan generated from the first few executions with parameters (PostgreSQL doesn’t have this)

If we use a generic plan, it can be inaccurate, for example:

1. Data skew (e.g., a particular MCV has a very high frequency, like WHERE a = 1 but a = 1 appears extremely often). This heavily depends on what the parameter value actually is, but the generic plan receives no parameters, so the plan cannot be accurate.
1. Evenly distributed data where selectivity still cannot be accurately calculated (e.g., WHERE a > $1 AND a < $2). Without knowing the range, no one can compute the selectivity. The generic plan receives no parameters, so the plan cannot be accurate.

If we reused plans from the first few parameterized executions (which PostgreSQL doesn’t do), they could also be inaccurate:

Data skew: the first few parameter values may not be representative, and they would heavily influence what the subsequent fixed plan looks like.

Categories of Generic Plan Estimation Problems
#

Because the comparison requires 5 custom plans first, generic plan problems can be divided into two categories:

The first 5 SQL executions are not representative. This is closely tied to the first 5 execution plans and depends on data skew and whether the first 5 parameter values are representative.
The generic plan itself is problematic. Due to data skew or the inability to accurately compute selectivity for evenly distributed data, the generic plan itself is inefficient.

Optimization Recommendations
#

Based on this case, generic plan issues can appear on partitioned tables. The partition key is contiguous, and selectivity when scanning all partitions should be 1, but the generic plan uses 0.005, which can easily lead to a “full index scan” scenario.

So during optimization, we need to consider more:

Avoid creating too many indexes that confuse the optimizer.
Eliminate generic plan interference. Use EXECUTE to truly run the query 6 times.
At the session level, set plan_cache_mode = 'force_generic_plan' or set plan_cache_mode = 'force_custom_plan' to compare execution plans. Or, on pg16+, use EXPLAIN (GENERIC_PLAN) to compare.

Syntax reference:

--prepare/excute
PREPARE sql1(text) AS
SELECT COUNT(*) FROM LZL where a=$1;

EXECUTE sql1('zzz'); -- run 6 times first
EXPLAIN EXECUTE sql1('zzz');

select * from pg_prepared_statements -- view prepared statement info, current session only

-- Compare execution plans by setting session parameters before EXPLAIN EXECUTE
set plan_cache_mode='force_generic_plan'
set plan_cache_mode='force_custom_plan'
-- Directly view generic plan, pg16+
explain (GENERIC_PLAN) xx 

Query Conflicts: From a Static Table Conflict to Its Root Cause

Sat, 13 Sep 2025 00:00:00 +0000

Problem Symptoms
#

The Symptom
#

A static historical table with no updates whatsoever — yet queries on the same-city standby consistently hit query conflicts:

ERROR: 40001: canceling statement due to conflict with recovery
DETAIL: User query might have needed to see row versions that must be removed.
LOCATION: ProcessInterrupts, postgres.c:3197
Time: 30534.973 ms (00:30.535)

Why a Query Conflict on a Static Table Matters
#

My understanding was that a static table should never experience conflicts (this understanding was wrong — I’ll explain later).

The official documentation lists the conflict cases:

Access Exclusive locks taken on the primary server, including both explicit LOCK commands and various DDL actions, conflict with table accesses in standby queries.
Dropping a tablespace on the primary conflicts with standby queries using that tablespace for temporary work files.
Dropping a database on the primary conflicts with sessions connected to that database on the standby.
Application of a vacuum cleanup record from WAL conflicts with standby transactions whose snapshots can still “see” any of the rows to be removed.
Application of a vacuum cleanup record from WAL conflicts with queries accessing the target page on the standby, whether or not the data to be removed is visible.

LOCK, DDL, drop tablespace, drop database — definitely none of those.

Vacuum — none either, confirmed by pg_stat_all_tables.last_autovacuum and WAL vacuum records.

The official documentation’s explanation stops there. I carefully verified that none of the above applied.

Extrapolating from existing knowledge, perhaps other scenarios could kill the xmin held by a standby query’s snapshot. For example, in-page pruning removes xmin from rows on a page — if the standby query’s snapshot still depends on those xmins, theoretically a conflict could occur. But a page belongs to a specific table, and querying only one table holds only snapshots and xmins on that table. So, theoretically, in-page pruning on table A should not cause a query conflict on table B (this understanding was also wrong — I’ll explain later).

PG’s official documentation on query conflict scenarios is fairly vague and doesn’t explain well why a static table can experience conflicts. Even combining it with my own extrapolations, there shouldn’t be a conflict. But I noticed this pattern seemed to exist on many instances, so it was worth investigating.

Root Cause Analysis
#

Since the startup process kills the query, checking the startup process’s pstack should reveal the conflict function:

$ pstack 212012
#0 0x00002b283f63d783 in __select_nocancel () from /lib64/libc.so.6
#1 0x00000000008fcf5a in pg_usleep (microsec=<optimized out>) at pgsleep.c:56
#2 0x0000000000787905 in WaitExceedsMaxStandbyDelay (wait_event_info=134217762) at standby.c:208
#3 ResolveRecoveryConflictWithVirtualXIDs (waitlist=0x2398a50, reason=reason@entry=PROCSIG_RECOVERY_CONFLICT_SNAPSHOT, wait_event_info=wait_event_info@entry=134217762, report_waiting=report_waiting@entry=true) at standby.c:276
#4 0x0000000000787b33 in ResolveRecoveryConflictWithVirtualXIDs (report_waiting=true, wait_event_info=134217762, reason=PROCSIG_RECOVERY_CONFLICT_SNAPSHOT, waitlist=<optimized out>) at standby.c:333
#5 ResolveRecoveryConflictWithSnapshot (latestRemovedXid=<optimized out>, node=...) at standby.c:329
#6 0x00000000004c8ffe in heap_xlog_clean (record=0x2366978) at heapam.c:7764
#7 heap2_redo (record=0x2366978) at heapam.c:8917
#8 0x0000000000519e55 in StartupXLOG () at xlog.c:7411
#9 0x000000000072f211 in StartupProcessMain () at startup.c:204
#10 0x00000000005286b1 in AuxiliaryProcessMain (argc=argc@entry=2, argv=argv@entry=0x7ffeb7e39d70) at bootstrap.c:450
#11 0x000000000072c369 in StartChildProcess (type=StartupProcess) at postmaster.c:5494
#12 0x000000000072eb54 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x232edb0) at postmaster.c:1407
#13 0x00000000004892cf in main (argc=3, argv=0x232edb0) at main.c:210

XLOG_HEAP2_CLEAN
#

void
heap2_redo(XLogReaderState *record)
{
	uint8		info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;

	switch (info & XLOG_HEAP_OPMASK)
	{
		case XLOG_HEAP2_CLEAN:
			heap_xlog_clean(record);
			break;

Only when the redo is XLOG_HEAP2_CLEAN does it enter the next function heap_xlog_clean.

PG 18 no longer has XLOG_HEAP2_CLEAN (it was actually removed around PG15 — this article only looks at versions 13 and 18), but the define can still be found in heapam_xlog.h:

//pg13
#define XLOG_HEAP2_CLEAN		0x10
#define XLOG_HEAP2_FREEZE_PAGE	0x20
#define XLOG_HEAP2_CLEANUP_INFO 0x30

//pg18
 * There's no difference between XLOG_HEAP2_PRUNE_ON_ACCESS,
 * XLOG_HEAP2_PRUNE_VACUUM_SCAN and XLOG_HEAP2_PRUNE_VACUUM_CLEANUP records.
 * They have separate opcodes just for debugging and analysis purposes, to
 * indicate why the WAL record was emitted.
 */
#define XLOG_HEAP2_PRUNE_ON_ACCESS		0x10
#define XLOG_HEAP2_PRUNE_VACUUM_SCAN	0x20
#define XLOG_HEAP2_PRUNE_VACUUM_CLEANUP	0x30

I pulled out PG18’s source because PG13 (our production version) has zero explanation for these CLEAN xl_info macros, making them hard to understand. Since PG18 renamed the macros to something more intuitive and added comments, we can use PG18’s source to understand PG13’s — to figure out what this WAL record does.

All three opcodes are fundamentally PRUNE-related WAL records. From the names, PRUNE_ON_ACCESS looks like pruning triggered by access, while the other two are tied to VACUUM operations.

When checking with pg_waldump, rmgr: Heap2 CLEAN remxid records appear every few seconds, with highly varied filenodes and no relation to the static table:

$ pg_waldump 00000001000012FE00000001 |tail -200|egrep -i heap2
pg_waldump: fatal: error in WAL record at 12FE/F34F138: invalid resource manager ID 50 at 12FE/F34F168
rmgr: Heap2 len (rec/tot): 61/ 3520, tx: 0, lsn: 12FE/0F346ED0, prev 12FE/0F346EA0, desc: CLEAN remxid 1983744188, blkref #0: rel 1663/88121/1083807 blk 617606 FPW
rmgr: Heap2 len (rec/tot): 66/ 66, tx: 0, lsn: 12FE/0F34BC60, prev 12FE/0F34BC30, desc: CLEAN remxid 1984090598, blkref #0: rel 1663/88121/504681 blk 1447147

This matches our symptom pattern: no vacuum activity, but PRUNE is happening, leading into heap_xlog_clean → ResolveRecoveryConflictWithSnapshot and the rest of the conflict machinery.

The PRUNE action producing rmgr: Heap2 CLEAN remxid WAL records will be demonstrated later via testing.

Let’s finish the source code analysis first.

ResolveRecoveryConflictWithSnapshot
#

void
ResolveRecoveryConflictWithSnapshot(TransactionId latestRemovedXid, RelFileNode node)
{
	VirtualTransactionId *backends;

	/*
	 * If we get passed InvalidTransactionId then we do nothing (no conflict).
	 *
	 * This can happen when replaying already-applied WAL records after a
	 * standby crash or restart, or when replaying an XLOG_HEAP2_VISIBLE
	 * record that marks as frozen a page which was already all-visible. It's
	 * also quite common with records generated during index deletion
	 * (original execution of the deletion can reason that a recovery conflict
	 * which is sufficient for the deletion operation must take place before
	 * replay of the deletion record itself).
	 */
	if (!TransactionIdIsValid(latestRemovedXid))
		return;

	backends = GetConflictingVirtualXIDs(latestRemovedXid,
										 node.dbNode);

	ResolveRecoveryConflictWithVirtualXIDs(backends,
										 PROCSIG_RECOVERY_CONFLICT_SNAPSHOT,
										 WAIT_EVENT_RECOVERY_CONFLICT_SNAPSHOT,
										 true);
}

There are several types of query conflicts. ResolveRecoveryConflictWithSnapshot lives up to its name — it’s a snapshot conflict.

GetConflictingVirtualXIDs finds which backends conflict with the snapshot. ResolveRecoveryConflictWithVirtualXIDs handles the actual conflict resolution and timeout.

GetConflictingVirtualXIDs
#

GetConflictingVirtualXIDs is the key function that determines whether a backend’s virtual transaction ID triggers a query conflict. It requires a bit of brainpower.

Prerequisite knowledge for understanding this function:

limitXmin is latestRemovedXid — the CLEAN remxid from WAL, the xid that needs to be cleaned up (I read remxid as “remove xid”). /*limitXmin is supplied as either latestRemovedXid, or InvalidTransactionId*/
PGPROC contains current process info: backend id, database id, lock info, and much more
PGXACT contains the transaction info for the snapshot held by the current process. It’s lighter — the key field is xmin, the lowest xid the current process considers still running
C’s || rule: if either operand is true (non-zero), the result is true (1)
TransactionIdIsValid means xid != 0 — 0 is meaningless

Key function GetConflictingVirtualXIDs explained:

VirtualTransactionId *
GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
	...
	for (index = 0; index < arrayP->numProcs; index++) // iterate all local processes
	{
		int			pgprocno = arrayP->pgprocnos[index];
		PGPROC	 *proc = &allProcs[pgprocno]; // process's PGPROC
		PGXACT	 *pgxact = &allPgXact[pgprocno]; // process's PGXACT

		/* Exclude prepared transactions */
		if (proc->pid == 0) // prepared transactions have no owning process — can't handle
			continue;

		if (!OidIsValid(dbOid) || // global tables have dbOid=0 which is invalid — satisfies condition
			proc->databaseId == dbOid) // only process current database. Cross-db is different — no transaction conflict at all.
		{
			/* Fetch xmin just once - can't change on us, but good coding */
			TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin); // pgxact->xmin is the minimum xid of transactions held by this process. UINT32_ACCESS_ONCE is just for atomic access protection — the xmin logic is unchanged

			/*
			 * We ignore an invalid pxmin because this means that backend has
			 * no snapshot currently. We hold a Share lock to avoid contention
			 * with users taking snapshots. That is not a problem because the
			 * current xmin is always at least one higher than the latest
			 * removed xid, so any new snapshot would never conflict with the
			 * test here.
			 */
			if (!TransactionIdIsValid(limitXmin) || // limitXmin=0 possible? At least latestRemovedXid can't be — I can't think of a scenario where WAL would log an invalid xid
				(TransactionIdIsValid(pxmin) && !TransactionIdFollows(pxmin, limitXmin))) // TransactionIdIsValid(pxmin) is also not really needed. !TransactionIdFollows(pxmin, limitXmin) means pxmin <= limitXmin
			{
				VirtualTransactionId vxid;

				GET_VXID_FROM_PGPROC(vxid, *proc);
				if (VirtualTransactionIdIsValid(vxid))
					vxids[count++] = vxid;
			}
		}
	}

The critical line is !TransactionIdFollows(pxmin, limitXmin).

So the core logic for determining query conflicts is:

The primary’s cleaned remxid >= the standby query’s snapshot-held minimum xid → conflict.
Only kills queries in the current database; global system tables (no database) are killed indiscriminately.

This means: even if the pruned table on the primary has nothing to do with the table being queried on the standby, a conflict CAN occur!!!

In-Page Pruning
#

Now that the conflict logic is clear, we still need to understand where the WAL CLEAN records come from. That requires looking at how PRUNE is triggered.

From README.HOT on when pruning and defragmentation occur — “When can/should we prune or defragment?”:

The currently planned heuristic is to prune and defrag when first accessing a page that potentially has prunable tuples

Prune and defragment are indeed two distinct concepts, but they often happen together.

Prune: updating line pointers to shorten HOT chains, but doesn’t free space
Defragment: reclaiming space from dead line pointers and tuples after pruning

We cannot prune or defragment unless we can get a “buffer cleanup lock” on the target page; otherwise, pruning might destroy line pointers that other backends have live references to, and defragmenting might move tuples that other backends have live pointers to

The page must be under a “buffer cleanup lock” for prune or defragment to occur.

The worst-case consequence of this is only that an UPDATE cannot be made HOT but has to link to a new tuple version placed on some other page, for lack of centralized space on the original page.

A typical scenario: a HOT update spills to another page (easy to test).

space reclamation happens during tuple retrieval when the page is nearly full (<10% free) and a buffer cleanup lock can be acquired. This means that UPDATE, DELETE, and SELECT can trigger space reclamation, but often not during INSERT … VALUES because it does not retrieve a row.

SELECT/UPDATE/DELETE that scan rows can trigger space reclamation. INSERT typically won’t, since it doesn’t retrieve rows.

Clearly, after prune or defragment, the corresponding xids should be reclaimed. From the README we can see that HOT updates can reproduce prune/defragment, generating CLEAN WAL records. See [Test: Pure UPDATE Produces In-Page Pruning](## Test: Pure UPDATE Produces In-Page Pruning).

Testing
#

The tests below only observe whether conflicts occur, whether CLEAN WAL records appear, or whether page line pointers are updated — without distinguishing prune vs. defragment. In many cases both are triggered together; distinguishing them is tedious and maybe best left for later. The focus here is whether CLEAN WAL records appear.

Helper SQL:

--sql for test
--heap_page_items
select t_ctid,lp,
case lp_flags
when 0 then '0:LP_UNUSED'
when 1 then 'LP_NORMAL'
when 2 then 'LP_REDIRECT'
when 3 then 'LP_DEAD'
end as lp_flags,
t_xmin,t_xmax,t_field3 as t_cid, raw_flags, info.combined_flags,substring(t_data,0,40)
from heap_page_items(get_raw_page('lzl',0)) item,
LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2) info
order by lp;

--heap header
select * from page_header(get_raw_page('lzl',0));

--bt_page_items
SELECT itemoffset, ctid, itemlen, nulls, vars, dead, htid, tids[0:2] AS some_tids FROM bt_page_items('idxlzl',1);

--create table
create table lzl(a char(2000));
create index idxlzl on lzl(a);
insert into lzl values('z');
update lzl set a=md5(random()::text); -- non-hot
update lzl set a='z'; -- hot

--force index scan
set enable_seqscan =off;
set enable_indexonlyscan=off;

--open an RR transaction to hold a snapshot for observation
BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;

Test: Cross-Table Query Conflict
#

primary	standby
create table lzl(a bigint primary key);
insert into lzl values(1);
	BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
	select 1;
update lzl set a=2;
	no blocking
vacuum lzl;
	#3 ResolveRecoveryConflictWithVirtualXIDs (waitlist=0x277c340, reason=reason@entry=PROCSIG_RECOVERY_CONFLICT_SNAPSHOT, wait_event_info=wait_event_info@entry=134217762, report_waiting=report_waiting@entry=true) at standby.c:276 #4 0x0000000000787b33 in ResolveRecoveryConflictWithVirtualXIDs (report_waiting=true, wait_event_info=134217762, reason=PROCSIG_RECOVERY_CONFLICT_SNAPSHOT, waitlist=) at standby.c:333 #5 ResolveRecoveryConflictWithSnapshot (latestRemovedXid=, node=…) at standby.c:329 #6 0x00000000004c8ffe in heap_xlog_clean (record=0x273a258) at heapam.c:7764

Conclusion: As long as a query exists, it has a snapshot, and a snapshot has a snapshot xmin. Even if the queried table is completely unrelated, a query conflict CAN occur.

Test: Vacuum Produces In-Page Pruning
#

Pruning occurs, conflicts occur. Example omitted — not relevant to this case.

Test: UPDATE Produces In-Page Pruning
#

--HOT, off-page update triggers defragment
--An 8k heap page stores 4-2xx rows. Here we size rows so 4 fit and remain HOT — the next update spills off-page
create table lzl(a char(2000));
create table idxlzl on lzl(a);
insert into lzl values('z');
update lzl set a='z'; --hot
update lzl set a='z'; --hot
update lzl set a='z'; --hot

--heap page: 4 rows, all HOT:
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags |
--------+----+-----------+----------+----------+-------+----------------------------------------------------------------------------------------------------------+----------------+----------
 (0,2) | 1 | LP_NORMAL | 34954161 | 34954162 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_HOT_UPDATED} | {} | \x501f000
 (0,3) | 2 | LP_NORMAL | 34954162 | 34954163 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f000
 (0,4) | 3 | LP_NORMAL | 34954163 | 34954164 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f000
 (0,4) | 4 | LP_NORMAL | 34954164 | 0 | 0 | {HEAP_HASVARWIDTH,HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f000
(4 rows)

--index: only one entry:
 itemoffset | ctid | itemlen | nulls | vars | dead | htid | some_tids
------------+-------+---------+-------+------+------+-------+-----------
 1 | (0,1) | 48 | f | t | f | (0,1) | [null]

--One more update triggers off-page update
update lzl set a='z'; --page full, can't HOT

--HOT chain changed. LP changed
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags |
--------+----+-------------+----------+----------+--------+--------------------------------------------------------------------------------------+----------------+---------------------------
 [null] | 1 | LP_REDIRECT | [null] | [null] | [null] | [null] | [null] | [null]
 (0,2) | 2 | LP_NORMAL | 34954165 | 0 | 0 | {HEAP_HASVARWIDTH,HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020202020
 [null] | 3 | 0:LP_UNUSED | [null] | [null] | [null] | [null] | [null] | [null]
 (0,2) | 4 | LP_NORMAL | 34954164 | 34954165 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020202020
(4 rows)

--index: still only one entry, unchanged:
 itemoffset | ctid | itemlen | nulls | vars | dead | htid | some_tids
------------+-------+---------+-------+------+------+-------+-----------
 1 | (0,1) | 48 | f | t | f | (0,1) | [null]

The next update doesn’t go to a new page — instead, in-page pruning happens first, freeing space on the same page, so the row is written locally. This saves a page access.

WAL produces CLEAN remxid, confirming that a query conflict can occur:

rmgr: Heap2 len (rec/tot): 62/ 62, tx: 0, lsn: 3DB/F8017348, prev 3DB/F8017310, desc: CLEAN remxid 34954177, blkref #0: rel 1663/5893914/5893920 blk 0
rmgr: Heap len (rec/tot): 2070/ 2070, tx: 34954178, lsn: 3DB/F8017388, prev 3DB/F8017348, desc: HOT_UPDATE off 4 xmax 34954178 flags 0x10 ; new off 2 xmax 0, blkref #0: rel 1663/5893914/5893920 blk 0

Conclusion: UPDATE statements can produce in-page pruning and can cause query conflicts.

Test: Hint-Bit Writeback Producing In-Page Pruning?
#

primary	standby
wal_log_hints=on
truncate table lzl;
insert into lzl values(‘z’);
	BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;
	select * from lzl;
delete from lzl where a=‘z’;
checkpoint;
select * from lzl;
–WAL contains FPI_FOR_HINT
	–no query conflict

Standby pageinspect:

 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags | substring
--------+----+-----------+----------+----------+-------+------------------------------------------------------------------------------+----------------+-------------------------------------------------
 (0,1) | 1 | LP_NORMAL | 34954229 | 34954230 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_KEYS_UPDATED} | {} | \x501f00007a202020202020202020202020202020202020
(1 row)

Conclusion: WAL log hints only sync hint bits and don’t affect xmin/xmax. No CLEAN or similar records are produced, so hint-bit writeback does NOT cause query conflicts.

Test: SELECT Produces In-Page Pruning
#

SELECT normally doesn’t cause pruning, but it does when the page is nearly full: https://www.modb.pro/db/1683648157451362304

Testing pruning on a full page:

-- Same table as before, 4 HOT rows, nearly full
insert into lzl values('z');
update lzl set a='z';
update lzl set a='z';
update lzl set a='z';

--page at this point:
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags |
--------+----+-----------+----------+----------+-------+----------------------------------------------------------------------------------------------------------+----------------+---------------------
 (0,2) | 1 | LP_NORMAL | 34954232 | 34954233 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_HOT_UPDATED} | {} | \x501f00007a20202020
 (0,3) | 2 | LP_NORMAL | 34954233 | 34954234 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020
 (0,4) | 3 | LP_NORMAL | 34954234 | 34954235 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_UPDATED,HEAP_HOT_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020
 (0,4) | 4 | LP_NORMAL | 34954235 | 0 | 0 | {HEAP_HASVARWIDTH,HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020
(4 rows)

-- A SELECT
select * from lzl;

--page now shows in-page pruning:
 t_ctid | lp | lp_flags | t_xmin | t_xmax | t_cid | raw_flags | combined_flags | sub
--------+----+-------------+----------+--------+--------+---------------------------------------------------------------------------------------+----------------+---------------------------------------
 [null] | 1 | LP_REDIRECT | [null] | [null] | [null] | [null] | [null] | [null]
 [null] | 2 | 0:LP_UNUSED | [null] | [null] | [null] | [null] | [null] | [null]
 [null] | 3 | 0:LP_UNUSED | [null] | [null] | [null] | [null] | [null] | [null]
 (0,4) | 4 | LP_NORMAL | 34954235 | 0 | 0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID,HEAP_UPDATED,HEAP_ONLY_TUPLE} | {} | \x501f00007a20202020202020202020202020

Conclusion: SELECT can produce in-page pruning and can cause query conflicts.

Test: Shared Table Cross-Database Query Conflict
#

Shared tables are global. Earlier in GetConflictingVirtualXIDs we saw that global tables are killed indiscriminately. Let’s test.

Shared table info:

Source definition: IsSharedRelation
Source check: shared ? InvalidOid : MyDatabaseId;
Table: pg_class.relisshared
Directory: global/

Querying pg_class.relisshared directly is easier:

select relname,relkind,relisshared from pg_class where relisshared is true and relkind='r';
 relname | relkind | relisshared
-----------------------+---------+-------------
 pg_authid | r | t
 pg_subscription | r | t
 pg_database | r | t
 pg_db_role_setting | r | t
 pg_tablespace | r | t
 pg_auth_members | r | t
 pg_shdepend | r | t
 pg_shdescription | r | t
 pg_replication_origin | r | t
 pg_shseclabel | r | t

pg_authid stores role/user info. Testing with a password change:

--Test: on the primary, in a non-business database
create user lzl;
alter user lzl with password '1'; --run several times

CLEAN remxid appears:

rmgr: Heap len (rec/tot): 76/ 76, tx: 34954264, lsn: 3DB/F808D0F8, prev 3DB/F808D0B8, desc: HOT_UPDATE off 67 xmax 34954264 flags 0x20 ; new off 66 xmax 0, blkref #0: rel 1664/0/1260 blk 0
rmgr: Transaction len (rec/tot): 82/ 82, tx: 34954264, lsn: 3DB/F808D148, prev 3DB/F808D0F8, desc: COMMIT 2025-09-12 14:40:56.680782 CST; inval msgs: catcache 11 catcache 10
rmgr: Heap2 len (rec/tot): 60/ 60, tx: 0, lsn: 3DB/F808D1A0, prev 3DB/F808D148, desc: CLEAN remxid 34954264, blkref #0: rel 1664/0/1260 blk 0
rmgr: Heap2 len (rec/tot): 60/ 60, tx: 34954265, lsn: 3DB/F808D1E0,

The standby business database’s select 1 query was killed.

Conclusion: Shared tables can cause cross-database query conflicts.

That said, these shared system tables rarely see heavy updates in normal operations.

Conclusions
#

Developer Perspective
#

Query conflicts can be completely unrelated to the table being queried — meaning a fully static table CAN experience conflicts.

Cross-database means different business domains and data. Cross-database does NOT cause query conflicts. The one exception is shared tables, but these are just a handful of system tables that rarely see updates.

For developers, focus on:

Retry on failure: Standby queries can be killed — retrying is essential, and retries may succeed
Query duration: Longer queries are more likely to be killed
Alternative standbys: Consider using a different standby with lower disaster-recovery requirements

Operations Perspective
#

Since query conflicts can come from “all directions,” a simple long-running single-table query can be killed by in-page pruning on a completely different, frequently-updated table. You can increase max_standby_streaming_delay to reduce conflict probability.

However, max_standby_streaming_delay trades off against WAL apply — a longer delay means WAL application is paused. This parameter’s value directly represents the maximum possible standby replication lag (it can’t cap lag from network or other factors).

Query freshness: Prolonged WAL apply pauses mean the standby data lags significantly (WAL may already be on the standby’s disk), affecting data freshness requirements for other standby queries.
RTO: If the primary suffers a disaster and failover is needed, the standby must apply accumulated WAL. If apply delay stretches to hours, it may violate minute-level RTO SLAs.

So tuning max_standby_streaming_delay is a delicate exercise requiring consideration of the standby’s role, query freshness requirements, and even geography.

Parameters on the Control File and Primary-Standby Parameter Mismatch Issues

Mon, 25 Aug 2025 00:00:00 +0000

PARAMETER_CHANGE and Database Parameters on the Control File
#

Some PG parameters affect the standby’s operation. These parameters are not only in the configuration file but also written to the control file. Whenever parameters change, they are written to WAL and update the control file.

The standby redoes the PARAMETER_CHANGE WAL record and writes to the standby’s control file. PARAMETER_CHANGE WAL record:

rmgr: XLOG len (rec/tot): 54/ 54, tx: 0, lsn: 27F/800001C0, prev 27F/80000148, desc: PARAMETER_CHANGE max_connections=3000 max_worker_processes=20 max_wal_senders=10 max_prepared_xacts=0 max_locks_per_xact=1024 wal_level=logical wal_log_hints=off track_commit_timestamp=on

XLOG_PARAMETER_CHANGE records these 8 parameters, which can also be viewed directly from the control file:

$ pg_controldata |grep setting
wal_level setting: logical
wal_log_hints setting: on
max_connections setting: 1000
max_worker_processes setting: 20
max_wal_senders setting: 10
max_prepared_xacts setting: 0
max_locks_per_xact setting: 1024
track_commit_timestamp setting: on

These parameters are all from the primary, even if this control file belongs to the standby.

The startup process checks 6 parameters via the CheckRequiredParameterValues function. One parameter wal_level must be >= replica. The other 5 parameters — max_connections, max_worker_processes, max_wal_senders, max_prepared_transactions, max_locks_per_transaction — are checked for primary vs standby sizing. If the standby has a smaller value, recovery is paused. If you increase the primary’s parameters directly, the standby will crash. The PG log:

FATAL,22023,"hot standby is not possible because max_connections = 2000 is a lower setting than on the master server (its value was 3000)",,,,,"WAL redo at 27F/800001C0 for XLOG/PARAMETER_CHANGE: max_connections=3000 max_worker_processes=20 max_wal_senders=10 max_prepared_xacts=0 max_locks_per_xact=1024 wal_level=logical wal_log_hints=off track_commit_timestamp=on",,,,"","startup"

6 of the 8 parameters can seriously affect standby operation. The other 2 parameters — wal_log_hints, track_commit_timestamp — are not immediately checked by the startup process. All 8 parameters being synchronized to the control file serve their own purposes.

wal_log_hints Primary-Standby Mismatch
#

Changes to wal_log_hints are recorded in WAL logs. Although not checked by the startup process, pg_rewind does check it:

perform_rewind(...)
{
...
	/*
	 * Target cluster need to use checksums or hint bit wal-logging, this to
	 * prevent from data corruption that could occur because of hint bits.
	 */
	if (ControlFile_target.data_checksum_version != PG_DATA_CHECKSUM_VERSION &&
		!ControlFile_target.wal_log_hints)
	{
		pg_fatal("target server needs to use either data checksums or \"wal_log_hints = on\"");
	}

Since wal_log_hints is WAL-related, it doesn’t make sense for pg_rewind to check whether the standby’s wal_log_hints is enabled — it should check whether the primary’s wal_log_hints is enabled. Therefore, PG synchronizes the wal_log_hints parameter to the standby’s control file, which is very reasonable.

wal_log_hints primary-standby mismatch test:

select * from t1;
checkpoint;
update t1 set b='eee'; -- observation point 1
checkpoint; -- ignore this online checkpoint wal record
select * from t1; -- observation point 2

-- observation action
pg_waldump 000000020000027F0000000A|tail -10

-- observing option
select t_ctid,lp,
case lp_flags
when 0 then '0:LP_UNUSED'
when 1 then 'LP_NORMAL'
when 2 then 'LP_REDIRECT'
when 3 then 'LP_DEAD'
end as lp_flags,
t_xmin,t_xmax,t_field3 as t_cid, raw_flags, info.combined_flags,substring(t_data,0,40)
from heap_page_items(get_raw_page('t1',0)) item,
LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2) info
order by lp;

on, on:

-- Observation point 1:
rmgr: Heap len (rec/tot): 85/ 208, tx: 11140182, lsn: 27F/5000CC38, prev 27F/5000CBC0, desc: HOT_UPDATE off 3 xmax 11140182 flags 0x10 ; new off 4 xmax 0, blkref #0: rel 1663/7472552/7472597 blk 0 FPW
rmgr: Transaction len (rec/tot): 46/ 46, tx: 11140182, lsn: 27F/5000CD08, prev 27F/5000CC38, desc: COMMIT 2025-07-21 18:28:13.292397 CST
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/5000CD38, prev 27F/5000CD08, desc: RUNNING_XACTS nextXid 11140183 latestCompletedXid 11140182 oldestRunningXid 11140183
-- Observation point 2:
rmgr: XLOG len (rec/tot): 51/ 171, tx: 0, lsn: 27F/58000110, prev 27F/580000D8, desc: FPI_FOR_HINT , blkref #0: rel 1663/7472552/7472597 blk 0 FPW
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/580001C0, prev 27F/58000110, desc: RUNNING_XACTS nextXid 11140183 latestCompletedXid 11140182 oldestRunningXid 11140183

off, off:

-- Observation point 1:
rmgr: Heap len (rec/tot): 85/ 225, tx: 11140183, lsn: 27F/580003C8, prev 27F/58000390, desc: HOT_UPDATE off 4 xmax 11140183 flags 0x10 ; new off 5 xmax 0, blkref #0: rel 1663/7472552/7472597 blk 0 FPW
rmgr: Transaction len (rec/tot): 46/ 46, tx: 11140183, lsn: 27F/580004B0, prev 27F/580003C8, desc: COMMIT 2025-07-21 18:33:18.192146 CST
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/580004E0, prev 27F/580004B0, desc: RUNNING_XACTS nextXid 11140184 latestCompletedXid 11140183 oldestRunningXid 11140184
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/58000518, prev 27F/580004E0, desc: RUNNING_XACTS nextXid 11140184 latestCompletedXid 11140183 oldestRunningXid 11140184
-- Observation point 2:

on, off:

-- Observation point 1:
rmgr: Heap len (rec/tot): 85/ 274, tx: 11140186, lsn: 27F/58000C18, prev 27F/58000BA0, desc: HOT_UPDATE off 7 xmax 11140186 flags 0x10 ; new off 8 xmax 0, blkref #0: rel 1663/7472552/7472597 blk 0 FPW
rmgr: Transaction len (rec/tot): 46/ 46, tx: 11140186, lsn: 27F/58000D30, prev 27F/58000C18, desc: COMMIT 2025-07-21 18:40:17.638691 CST
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/58000D60, prev 27F/58000D30, desc: RUNNING_XACTS nextXid 11140187 latestCompletedXid 11140186 oldestRunningXid 11140187
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/58000D98, prev 27F/58000D60, desc: RUNNING_XACTS nextXid 11140187 latestCompletedXid 11140186 oldestRunningXid 11140187
-- Observation point 2:
rmgr: XLOG len (rec/tot): 51/ 236, tx: 0, lsn: 27F/58000E48, prev 27F/58000DD0, desc: FPI_FOR_HINT , blkref #0: rel 1663/7472552/7472597 blk 0 FPW
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/58000F38, prev 27F/58000E48, desc: RUNNING_XACTS nextXid 11140187 latestCompletedXid 11140186 oldestRunningXid 11140187

off, on:

-- Observation point 1:
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/58001108, prev 27F/58001090, desc: RUNNING_XACTS nextXid 11140187 latestCompletedXid 11140186 oldestRunningXid 11140187
rmgr: Heap len (rec/tot): 85/ 289, tx: 11140187, lsn: 27F/58001140, prev 27F/58001108, desc: HOT_UPDATE off 8 xmax 11140187 flags 0x10 ; new off 9 xmax 0, blkref #0: rel 1663/7472552/7472597 blk 0 FPW
rmgr: Transaction len (rec/tot): 46/ 46, tx: 11140187, lsn: 27F/58001268, prev 27F/58001140, desc: COMMIT 2025-07-21 18:44:08.550109 CST
rmgr: Standby len (rec/tot): 54/ 54, tx: 0, lsn: 27F/58001298, prev 27F/58001268, desc: RUNNING_XACTS nextXid 11140188 latestCompletedXid 11140186 oldestRunningXid 11140187; 1 xacts: 11140187
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/580012D0, prev 27F/58001298, desc: RUNNING_XACTS nextXid 11140188 latestCompletedXid 11140187 oldestRunningXid 11140188
-- Observation point 2:

Test summary:

FPI_FOR_HINT is produced when hint bits are written back; SELECT queries can produce FPI_FOR_HINT.
Regardless of the standby setting (on or off), when the primary is on, FPI_FOR_HINT will be produced.

Additional Knowledge: What is XLOG_RUNNING_XACTS
#

XLOG_RUNNING_XACTS is one type of RM_STANDBY_ID:

/*
 * XLOG message types
 */
#define XLOG_STANDBY_LOCK			0x00
#define XLOG_RUNNING_XACTS			0x10
#define XLOG_INVALIDATIONS			0x20

XLOG_STANDBY_LOCK: Records acquisition and release of AccessExclusiveLock, used by standby nodes to recognize lock states.

XLOG_RUNNING_XACTS: Running-xacts snapshots used for building snapshots to ensure transaction consistency.

XLOG_INVALIDATIONS: INVALIDATIONS messages for synchronizing metadata information to local backends.

 * standbydefs.h
 *	 Frontend exposed definitions for hot standby mode.

RM_STANDBY_ID is an rmgr specifically defined for hot standby read-only standbys. For local instance recovery and logical decoding scenarios that need WAL, RM_STANDBY_ID is essentially meaningless to them.

Observing WAL records during transaction commit:

command	wal record
begin;
select * from txid_current(); –11140191
commit;	rmgr: Transaction len (rec/tot): 46/ 46, tx: 11140191, lsn: 27F/80000538, prev 27F/80000500, desc: COMMIT 2025-07-23 11:16:10.872724 CST
	rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 27F/80000568, prev 27F/80000538, desc: RUNNING_XACTS nextXid 11140192 latestCompletedXid 11140191 oldestRunningXid 11140192

The transaction ID itself — commit or abort — is synchronized by rmgr: Transaction. Snapshots are synchronized via rmgr: Standby RUNNING_XACTS.

track_commit_timestamp Primary-Standby Mismatch
#

track_commit_timestamp: the startup process activates the standby’s commit_ts functionality upon receiving the corresponding WAL, primarily for viewing xid commit times on the standby:

/*
 * Activate or deactivate CommitTs' upon reception of a XLOG_PARAMETER_CHANGE
 * XLog record during recovery.
 */
void
CommitTsParameterChange(bool newvalue, bool oldvalue)
{
	/*
	 * If the commit_ts module is disabled in this server and we get word from
	 * the primary server that it is enabled there, activate it so that we can
	 * replay future WAL records involving it; also mark it as active on
	 * pg_control. If the old value was already set, we already did this, so
	 * don't do anything.
	 *
	 * If the module is disabled in the primary, disable it here too, unless
	 * the module is enabled locally.
	 *
	 * Note this only runs in the recovery process, so an unlocked read is
	 * fine.
	 */
	if (newvalue)
	{
		if (!commitTsShared->commitTsActive)
			ActivateCommitTs();
	}
	else if (commitTsShared->commitTsActive)
		DeactivateCommitTs();
}

track_commit_timestamp primary-standby mismatch test:

Initial state: primary=on, standby=on. Both can use committed_xact and similar functions.
primary=off (restart primary), standby=on (no change). Both cannot use committed_xact and similar functions.

After modifying and restarting the primary, standby replication remains normal, but committed_xact and similar functions are unusable:

$ select * from pg_last_committed_xact();
ERROR: 55000: could not get commit timestamp data
HINT: Make sure the configuration parameter "track_commit_timestamp" is set on the primary server.
LOCATION: error_commit_ts_disabled, commit_ts.c:385

$ show track_commit_timestamp
-> ;
 track_commit_timestamp
------------------------
 on
(1 row)

Time: 0.198 ms
$ \q
## pg_controldata |grep track_commit_timestamp
track_commit_timestamp setting: off

PG14+ Pause Recovery
#

PG14 improved the behavior when primary parameter changes cause standby crashes. When parameters don’t meet conditions, instead of the read-only standby directly crashing, it now only pauses replication. See RecoveryRequiresIntParameter.

Pause recovery on a hot standby server if the primary changes its parameters in a way that prevents replay on the standby (Peter Eisentraut)

Previously the standby would shut down immediately

Testing PG14 parameter changes causing standby replication interruption:

2025-07-23 19:46:31.337 CST,,,141823,,6880ca5f.229ff,14,,2025-07-23 19:41:19 CST,1/0,0,LOG,00000,"recovery has paused","If recovery is unpaused, the server will shut down.","You can then restart the server after making the necessary configuration changes.",,,"WAL redo at 281/78324BE8 for XLOG/PARAMETER_CHANGE: max_connections=2000 max_worker_processes=20 max_wal_senders=10 max_prepared_xacts=0 max_locks_per_xact=1024 wal_level=logical wal_log_hints=on track_commit_timestamp=on",,,,"","startup",,0

Since replication has already stopped, changing the primary’s parameters back won’t help — the standby can’t apply subsequent changes and update the control file. So you must modify the standby’s parameters and restart (the log hint is also quite clear).

Summary of the 8 Parameters
#

When any of the 8 parameters are modified on the primary and the primary is restarted, the local control file is updated. If parameters have changed, the updated parameters are written to WAL and synchronized to downstream. The downstream redoes this PARAMETER_CHANGE WAL record, updating its local control file. The standby then determines whether primary-standby replication or other functions are available based on certain conditions.

8 Parameters Written to Control File	Check	If not, standby (PG13-)	If not, standby (PG14+)
wal_level	!=minimal	Cannot sync, fundamental	Cannot sync, fundamental
max_connections	primary <= standby	hot standby shutdown	hot standby pause replication
max_worker_processes	primary <= standby	hot standby shutdown	hot standby pause replication
max_wal_senders	primary <= standby	hot standby shutdown	hot standby pause replication
max_prepared_transactions	primary <= standby	hot standby shutdown	hot standby pause replication
max_locks_per_transaction	primary <= standby	hot standby shutdown	hot standby pause replication
wal_log_hints	pg_rewind prerequisite (either data checksums or wal_log_hints = on)	Doesn’t affect standby sync	Doesn’t affect standby sync
track_commit_timestamp	Enable/disable standby commit_ts functionality	Doesn’t affect standby sync	Doesn’t affect standby sync

Special thanks to: Gao Changjun

PostgreSQL DDL Pitfalls and Clever Solutions

Sat, 19 Jul 2025 00:00:00 +0000

Save it, use it freely, no need to ask.

May be updated, may not be.

Feedback welcome — pick it apart if you can.

This article was originally published in Chinese on lastdba.com.

Case: GRANT Authorization Causes Walsender to Hang

Thu, 26 Jun 2025 00:00:00 +0000

Symptoms
#

The walsender’s LSN stopped advancing. The stack trace showed it was stuck in pathman’s invalidate_psin_entries_using_relid, with the relid constantly changing and the walsender CPU pegged at 100%.

pstack 121327
#0 hash_seq_search (status=status@entry=0x7fffaadf8330) at dynahash.c:1441
#1 0x00002ba3b40ec728 in invalidate_psin_entries_using_relid (relid=relid@entry=42319501) at src/relation_info.c:251
#2 0x00002ba3b40ecb3d in forget_status_of_relation (relid=relid@entry=42319501) at src/relation_info.c:232
#3 0x00002ba3b40fcc96 in pathman_relcache_hook (arg=<optimized out>, relid=42319501) at src/hooks.c:934
#4 0x000000000087168a in LocalExecuteInvalidationMessage (msg=0x3a391c8) at inval.c:595
#5 0x000000000071d50e in ReorderBufferExecuteInvalidations (rb=0x1b63ff8, txn=0x1be5f58, txn=0x1be5f58) at reorderbuffer.c:2238
#6 ReorderBufferCommit (rb=0x1b63ff8, xid=xid@entry=4285897514, commit_lsn=405674661986920, end_lsn=<optimized out>, commit_time=commit_time@entry=799377897828299, origin_id=origin_id@entry=0, origin_lsn=origin_lsn@entry=0) at reorderbuffer.c:1819
#7 0x0000000000712d18 in DecodeCommit (xid=4285897514, parsed=0x7fffaadf8630, buf=0x7fffaadf87f0, ctx=0x1a359e8) at decode.c:637
#8 DecodeXactOp (ctx=0x1a359e8, buf=buf@entry=0x7fffaadf87f0) at decode.c:245
#9 0x00000000007130b2 in LogicalDecodingProcessRecord (ctx=0x1a359e8, record=0x1a35c80) at decode.c:114
#10 0x0000000000733662 in XLogSendLogical () at walsender.c:2885
#11 0x0000000000735942 in WalSndLoop (send_data=send_data@entry=0x733620 <XLogSendLogical>) at walsender.c:2287
#12 0x0000000000736692 in StartLogicalReplication (cmd=0x1846c68) at walsender.c:1213
#13 exec_replication_command (cmd_string=cmd_string@entry=0x181a288 "START_REPLICATION SLOT \"lzl_logical_rep\" LOGICAL 170F5/7C3EAE78 (\"proto_version\" '1', \"publication_names\" 'lzl_logical_rep')") at walsender.c:1640
#14 0x0000000000774e91 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x1866478, dbname=0x18662b8 "lzldb", username=<optimized out>) at postgres.c:4325
#15 0x0000000000485989 in BackendRun (port=<optimized out>, port=<optimized out>) at postmaster.c:4526
#16 BackendStartup (port=0x18635b0) at postmaster.c:4210
#17 ServerLoop () at postmaster.c:1739
#18 0x0000000000702f08 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x1814da0) at postmaster.c:1412
#19 0x000000000048660a in main (argc=3, argv=0x1814da0) at main.c:210

## Second execution, same stack, different relid
pstack 121327
#0 hash_seq_search (status=status@entry=0x7fffaadf8330) at dynahash.c:1441
#1 0x00002ba3b40ec728 in invalidate_psin_entries_using_relid (relid=relid@entry=26560221) at src/relation_info.c:251
#2 0x00002ba3b40ecb3d in forget_status_of_relation (relid=relid@entry=26560221) at src/relation_info.c:232
#3 0x00002ba3b40fcc96 in pathman_relcache_hook (arg=<optimized out>, relid=26560221) at src/hooks.c:934
#4 0x000000000087168a in LocalExecuteInvalidationMessage (msg=0x39f1f68) at inval.c:595
...

Analysis
#

The changing relid showed that the walsender was still running, not dead. The LSN was not advancing, so we analyzed the LSN position to see what the transaction was doing.

If the slot information was still available, we could look up the restart LSN via the slot view to find the WAL position. If not, we could use the LSN from the stack trace to identify the WAL log.

Using pg_waldump to inspect WAL log entries, filtering by xid:

rmgr: Heap len (rec/tot): 961/ 961, tx: 4285897514, lsn: 170F5/7DFE3470, prev 170F5/7DFE3430, desc: UPDATE+INIT off 2 xmax 4285897514 flags 0x00 ; new off 1 xmax 0, blkref #0: rel 1663/17662/1259 blk 8443, blkref #1: rel 1663/17662/1259 blk 7327

...
rmgr: Transaction len (rec/tot): 1778325/1778325, tx: 4285897514, lsn: 170F5/7E1F4268, prev 170F5/7E1F4220, desc: COMMIT 2025-05-01 09:24:57.828299 CST; inval msgs: catcache 22 catcache 22 catcache 22 catcache 22 catcache 50 catcache 49 catcache 50 catcache 49 catcache 50 catcache 49 catcache 50 catcache 49 catcache 50 catcache 49 catcache 50 
...
 relcache 48813261 relcache 48813255 relcache 51030741 relcache 48813252 relcache 50737247 relcache 48813246 relcache 48813243 relcache 48813237 relcache 50737241 relcache 48813234 relcache 48813224 relcache 49379811 relcache 48813216 relcache 48813210 relcache 45452775

The transaction for rel 1663/17662/1259 had 180,000 records. The last record was inval msgs: ~70,000 catcache entries and ~30,000 relcache entries.

rel 1663/17662/1259 is pg_class. Querying by xmin reveals the affected tables and commit time:

select xmin,xmax,pg_xact_commit_timestamp(xmin),relname from pg_class where xmin='4285897514'::xid order by relname desc ;
 xmin | xmax | pg_xact_commit_timestamp | relname 
------------+------+-------------------------------+---------------------------------------------
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | v$session
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | tmp_20230801_id_seq
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | tmp_20230801
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | test_param
 4285897514 | 0 | 2025-05-01 09:24:57.828299+08 | test_20240105
 ...
 
 select count(*) from pg_class where xmin='4285897514'::xid ;
 count 
-------
 18523

 select count(*) from pg_class ;
 count 
--------
 139138

Checking the pglog by timestamp:

2025-05-01 09:24:59.837 CST,"postgres","lzldb",61418,"[local]",6812cd65.efea,3,"DO",2025-05-01 09:24:53 CST,549/0,0,LOG,00000,"duration: 6036.275 ms statement: 
...
 EXECUTE 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO r_lzldbdata_qry';
...
 END;
 $$",,,,,,,,,"psql","client backend"

We can basically confirm that the GRANT operation was the culprit. GRANT updates relacl in pg_class, and at least 18,000 relations had their permissions updated. Updates to pg_class trigger invalidation messages, and the massive number of invalidation messages were being processed slowly in the walsender process.

Reproduction
#

-- Create a logical replication slot, any kind will do
select pg_create_logical_replication_slot('logical_test','test_decoding');
pg_recvlogical -h 127.0.0.1 -p 7997 -d lzldb -U repuser --slot=logical_test --start -f recv.sql &

-- Create many tables
DO $$
BEGIN
 FOR i IN 1..20000 LOOP
 EXECUTE format(
 'CREATE TABLE IF NOT EXISTS table_%s ( 
 col1 varchar(10)
 )', 
 lpad(i::text, 5, '0') -- Generate 5-digit numbered table names
 );
 END LOOP;
END $$;

-- Single GRANT
grant select on all tables in schema public to r_lzldb_qry;

-- Perfectly reproduced
postgres@lzlhost:~/lzl/grant]$ pstack 172862
#0 hash_seq_search (status=status@entry=0x7ffd664be280) at dynahash.c:1444
#1 0x00002ad31235e728 in invalidate_psin_entries_using_relid (relid=relid@entry=1002857) at src/relation_info.c:251
#2 0x00002ad31235eb3d in forget_status_of_relation (relid=relid@entry=1002857) at src/relation_info.c:232
#3 0x00002ad31236ec96 in pathman_relcache_hook (arg=<optimized out>, relid=1002857) at src/hooks.c:934
#4 0x000000000087168a in LocalExecuteInvalidationMessage (msg=0x2ad3c3f61a88) at inval.c:595
#5 0x000000000071d50e in ReorderBufferExecuteInvalidations (rb=0x17e5698, txn=0x180d698, txn=0x180d698) at reorderbuffer.c:2238


[postgres@lzlhost:~/lzl/grant]$ pstack 172862
#0 0x0000000000891d0c in hash_seq_search (status=status@entry=0x7ffd664be280) at dynahash.c:1441
#1 0x00002ad31235e728 in invalidate_psin_entries_using_relid (relid=relid@entry=1011110) at src/relation_info.c:251
#2 0x00002ad31235eb3d in forget_status_of_relation (relid=relid@entry=1011110) at src/relation_info.c:232
#3 0x00002ad31236ec96 in pathman_relcache_hook (arg=<optimized out>, relid=1011110) at src/hooks.c:934


-- relid keeps changing
-- CPU pegged at 100%:
ps -eo pid,%cpu,%mem|grep 172862
172862 99.3 0.0

-- Takes about 2 hours to catch up

Accelerating Walsender by Removing Pathman
#

Since the database wasn’t actually using pathman partitioned tables but had the extension installed, we tried bypassing the pathman hook to speed up walsender processing.

drop extension pg_pathman;
grant update on all tables in schema public to r_lzldb_upd;

[postgres@lzlhost~/lzl/grant]$ pstack 133460
#0 hash_seq_search (status=status@entry=0x7ffe292d5c90) at dynahash.c:1418
#1 0x000000000087f228 in RelfilenodeMapInvalidateCallback (arg=<optimized out>, relid=1034036) at relfilenodemap.c:64
#2 0x000000000087168a in LocalExecuteInvalidationMessage (msg=0x2b9699795768) at inval.c:595
#3 0x000000000071d50e in ReorderBufferExecuteInvalidations (rb=0x195a358, txn=0x1a6ff38, txn=0x1a6ff38) at reorderbuffer.c:2238
#4 ReorderBufferCommit (rb=0x195a358, xid=xid@entry=328684387, commit_lsn=8016890875224, end_lsn=<optimized out>, commit_time=commit_time@entry=799851538975691, origin_id=origin_id@entry=0, origin_lsn=origin_lsn@entry=0) at reorderbuffer.c:1819

## Completed within 20 seconds

Even without commenting out pg_pathman from shared_preload_libraries, there was a dramatic improvement — walsender went from 2 hours to 20 seconds.

This seemed odd at first — without commenting shared_preload_libraries, the hook should still run. Source analysis revealed the reason: the very first step of the hook checks for the pathman config table; if it doesn’t exist, it skips pathman’s invalidation logic entirely, so execution completes quickly:

/*
 * Invalidate PartRelationInfo cache entry if needed.
 */
void
pathman_relcache_hook(Datum arg, Oid relid)
{
	Oid pathman_config_relid;

	/* See cook_partitioning_expression() */
	if (!pathman_hooks_enabled)
		return;

	if (!IsPathmanReady())
		return;

...

	/*
	 * Invalidation event for PATHMAN_CONFIG table (probably DROP EXTENSION).
	 * Digging catalogs here is expensive and probably illegal, so we take
	 * cached relid. It is possible that we don't know it atm (e.g. pathman
	 * was disabled). However, in this case caches must have been cleaned
	 * on disable, and there is no DROP-specific additional actions.
	 */
	pathman_config_relid = get_pathman_config_relid(true);
	if (relid == pathman_config_relid)
	{
		delay_pathman_shutdown();
	}

	/* Invalidation event for some user table */
	else if (relid >= FirstNormalObjectId)
	{
		/* Invalidate PartBoundInfo entry if needed */
		forget_bounds_of_rel(relid);

		/* Invalidate PartStatusInfo entry if needed */
		forget_status_of_relation(relid);

		/* Invalidate PartParentInfo entry if needed */
		forget_parent_of_partition(relid);
	}
}

get_pathman_config_relid fetches the pathman_config table. drop extension pg_pathman removes the pathman_config table from the database, so the source code never enters the forget_* logic.

There are other ways to accelerate walsender processing: setting pg_pathman.enable=off causes IsPathmanReady() to return false and bail out immediately. Or, most directly, comment out pg_pathman from shared_preload_libraries and restart the instance (this is instance-level, not database-level).

Improvements in PG14
#

PG14.0 release notes:

Allow logical decoding to more efficiently process cache invalidation messages (Dilip Kumar) This allows logical decoding to work efficiently in presence of a large amount of DDL.

https://www.postgresql.org/docs/release/14.0/

Patch:

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d7eb52d71

Comment from PG14’s ReorderBufferAddInvalidations:

We require to record it in form of the change so that we can execute only the required invalidations instead of executing all the invalidations on each CommandId increment.

Comparing PG14 vs PG13, ReorderBufferCommit underwent a major rewrite.

In PG13, transaction processing logic was directly in the ReorderBufferCommit function:

				case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
					Assert(change->data.command_id != InvalidCommandId);

					if (command_id < change->data.command_id)
					{
						command_id = change->data.command_id;

						if (!snapshot_now->copied)
						{
							/* we don't use the global one anymore */
							snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
																txn, command_id);
						}

						snapshot_now->curcid = command_id;

						TeardownHistoricSnapshot(false);
						SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);

						/*
						 * Every time the CommandId is incremented, we could
						 * see new catalog contents, so execute all
						 * invalidations.
						 */
						ReorderBufferExecuteInvalidations(rb, txn);
					}

In PG14, the main logic moved to ReorderBufferReplay -> ReorderBufferProcessTXN.

ReorderBufferProcessTXN introduced a new case REORDER_BUFFER_CHANGE_INVALIDATION branch to execute invalidations from the reorder buffer:

				case REORDER_BUFFER_CHANGE_INVALIDATION:
					/* Execute the invalidation messages locally */
					ReorderBufferExecuteInvalidations(
												 change->data.inval.ninvalidations,
												 change->data.inval.invalidations);
					break;

The logic after ReorderBufferExecuteInvalidations is largely the same. The main differences between PG13 and PG14’s ReorderBufferCommit:

ReorderBufferCommit is no longer the primary transaction processing function; the call stack is deeper
A new case REORDER_BUFFER_CHANGE_INVALIDATION branch was added, separated from REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, to handle invalidations independently
The per-command_id invalidation processing logic was removed

Root Cause and Solutions
#

The root cause of the walsender hang was a bulk GRANT operation that updated many rows in pg_class, triggering a massive number of invalidation messages. A statement like GRANT privs ON ALL TABLES IN SCHEMA public TO role1 executes as multiple commands within a single transaction in PostgreSQL. In PG13, logical replication processes invalidation messages per-command, invoking each hook’s inval hash table processing. In this scenario, pathman’s hook was particularly slow at processing the inval hash table, causing replication lag.

Conditions for pathman-induced slowness (all must apply):

PG13 or earlier
Bulk GRANT
pathman extension installed (whether used or not)
Logical replication slot active

Even after removing pathman, significant CPU time was still spent in functions like RelfilenodeMapInvalidateCallback. In PG13 testing, the processing time difference between with and without pathman was hours vs. minutes.

Other untested but community-mentioned scenarios (all must apply):

PG13 or earlier
Bulk DDL / TRUNCATE / DCL / DROP PUBLICATION
Logical replication slot active

Short-term fix: If pathman tables are not in use, drop the extension or unload the pathman shared library; restart the replication slot.

Long-term fix: Upgrade to PG14+ (tested — extremely fast with no lag).

#

References
#

https://www.postgresql.org/message-id/flat/17716-1fe42e7b44fc2f25%40postgresql.org

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d7eb52d71

Linux Memory Advanced

Thu, 19 Jun 2025 00:00:00 +0000

(For memory basics, refer to Linux Memory Analysis; this article covers memory knowledge above that foundation)

Memory Basic Concepts
#

buddy
#

The process of buddy system allocating and merging pages is omitted.

Easily overlooked knowledge points:

The prerequisite for buddy merging two blocks of the same size is that their physical addresses are contiguous
The merge algorithm is iterative: after merging at the current level, it will automatically attempt to merge larger blocks. This means compactd is not strictly required for merging

page table & PTE
#

page table and PTE are actually two different concepts, and they are easily confused because both generally refer to page tables. Below is relevant knowledge about page table and PTE[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]

PTE stores the physical address of the page frame
“page table” and “Page Table” are different concepts: “page table” refers to the pages that maintain the mapping between linear addresses and physical addresses, while “Page Table” refers to pages in the upper-level page table
pte_t, pmd_t, pud_t, pgd_t describe Page Table Entry, Page Middle Directory entry, Page Upper Directory entry, and Page Global Directory entry respectively
PTE is Page Table Entry

If you only look at the size of the pagetable used by the MMU to cache virtual-to-physical memory mappings, confusing pagetable with PTE doesn’t make much difference. However, if you need to go deep into page table directories, you need to separate the two concepts.

TLB
#

Each level of the page table is stored in memory. To complete a single virtual-to-physical address translation, all four page tables corresponding to the current virtual address must be found. This means a single memory IO requires looking up the page table in memory 4 times just for virtual-to-physical address translation. Translation Lookaside Buffers (TLB) are caches specifically designed to accelerate virtual-to-physical address translation.

Regarding the TLB’s location, it is usually in the L1 cache (some say it’s in registers or L2, which likely depends on the CPU architecture; for now, just consider it as CPU cache, distinct from main memory)¹:

In modern processors, the L1 cache is typically divided into multiple parts, including data cache dTLB and instruction cache iTLB. Frequently modifying page tables leads to increased main memory accesses, causing the CPU to frequently flush the TLB cache[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]. The TLB also has a finite size; improving TLB hit rate can reduce accesses to the main memory pagetable. Using huge pages can reduce PTEs by three orders of magnitude, greatly reducing TLB misses.[^ 《深入理解Linux进程和内存》 (Understanding Linux Processes and Memory)].

TLB information:

#cpuid -l
 L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax):
 L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx):
...
 L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax):

Observing TLB hit rate:

perf stat -e dTLB-loads,dTLB-load-misses,iTLB-loads,iTLB-load-misses -I 10000 -p $PM_PID 

During memory reclamation, TLB misses do increase, but it’s hard to establish a causal relationship. TLB miss is just one observation metric for the MMU — TLB is part of MMU.

Reverse Mapping
#

The general principles of PFRA (Page Frame Reclaiming Algorithm)[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]:

First, release “harmless” pages. Start by reclaiming harmless pages in the pagecache — pages not occupied by any process
All pages of user-mode processes are candidates for reclamation. FRPA will gradually deprive user-mode pages with longer sleep times of their page frames
Cancel the mapping of all page table entries for a shared page frame, then reclaim that shared page frame
Only reclaim “unused” pages

One of PFRA’s goals is to be able to release shared page frames. The process of quickly locating all page table entries pointing to the same page frame is called reverse mapping.

Reverse mappings for shared

Anonymous pages
File-mapping pages

Basic tricks of page frame reclaiming

LRU lists
Free cheapest pages first
Unmap all at once
Etc²

Huge Pages
#

Enabling huge pages provides certain performance improvements for specific application workloads. In PostgreSQL, enabling huge pages on large-memory instances also offers some performance gains and even some stability benefits.

Why are huge pages better?³:

Reduced TLB pressure
Reduced pagetable size in main memory
Huge pages are physically contiguous. Contiguous physical memory access is better than non-contiguous physical memory access
When using these kinds of larger pages, higher level pages can directly map them, with no need to use lower level page entries[^ kernel.org,mm pagetables]

However, using huge pages brings management challenges:

Huge pages need to be pre-allocated
Huge page size must be calculated in advance to avoid memory waste

Two ways for processes to use huge pages:

The first is by using shmget() to setup a shared region backed by huge pages
the second is the call mmap() on a file opened in the huge page filesystem

C Library and System Calls
#

The middle layer between kernel space and user space is the system call layer. Application Programming Interfaces (APIs) and system calls are different. Applications call APIs implemented in user space to program, rather than directly executing system calls. In the UNIX world, the most common system call layer is the POSIX standard (Portable Operation System Interface of UNIX). The POSIX standard targets APIs, not system calls. The Linux operating system’s API is typically provided in the form of C standard libraries, such as libc. The C standard library provides implementations for most POSIX APIs.[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)]

C app->C lib->system calls->OS->hardware⁴:

Common C library and system calls:

malloc,free=>C lib

mmap、brk、munmap=>system calls

Page Fault Exception
#

Page fault exceptions (or page fault interrupts) need to distinguish two cases: exceptions caused by programming errors; and physical page allocation behavior triggered by using virtual address space where physical page frames haven’t been allocated yet.[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]

Exceptional page fault: Segment Fault — each virtual memory area has associated permissions. If a process accesses a memory area outside its valid range, or illegally accesses a memory area, or accesses a memory area in an incorrect manner, the processor reports a page fault exception. In severe cases, it reports a “Segment Fault” and terminates the process[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)].
Normal page fault: System calls like mmap and brk manage virtual memory; they don’t directly allocate physical memory. Virtual memory system call functions only establish the process address space. Virtual memory is visible in user space, but no mapping between virtual memory and physical memory has been established. When a process accesses virtual memory where no mapping has been established, a page fault interrupt is triggered.[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)]

Page faults are also divided into two types:

minor fault: the page fault was handled without blocking the current process, and a page frame was allocated
major fault: the page fault forced the current process to sleep (likely because filling the page frame with data from disk took time). A page fault that blocks the current process is a major fault[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]

Copy-On-Write (COW)
#

When the fork system call is executed, the child process and parent process have independent process address spaces but share physical memory resources, including process context, process stack, memory information, file descriptors, directories, resource limits, etc. Only the parent process’s page table needs to be copied to the child process. At this point, sharing is read-only. When writing is needed (when running their respective tasks), data is copied, giving the parent and child processes their own copies.[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)]

For PostgreSQL’s multi-process model, fork itself isn’t heavy — you may only need to worry about page tables — but the various tasks that come after fork will trigger copy-on-write to create the child process’s own resource copies.

Note the distinction between copy-on-write and page fault exceptions: copy-on-write refers to resources not being allocated to the child process at fork time; page fault exceptions refer to physical memory allocation occurring for this process, unrelated to fork.

mmap, brk & Shared Memory Mapping Area, Heap Area
#

The functions and memory address regions used by mmap and brk are different:

mmap is used to manage shared memory, corresponding to the shared memory mapping area
brk is used to manage private memory, corresponding to the heap area

Linear address region functions:

mmap: The mapping area expands top-down. The mmap mapping area and heap expand toward each other until they exhaust the remaining space in the virtual address space. This structure facilitates the C runtime library’s use of the mmap mapping area and heap for memory allocation.
Stack: Stores local variables and function parameters during program execution, grows from high addresses to low addresses
Heap: Dynamic memory allocation area, managed through functions like malloc, new, free, and delete
BSS (Uninitialized Variables): Stores uninitialized global variables and static variables
Data: Stores global variables and static variables with predefined values in source code
Text (Code): Stores read-only program execution code, i.e., machine instructions.

Shared memory mapping area and heap area⁵:

Real postmaster heap and shared memory mapping:

cat /proc/1063005/smaps |grep -E "\-s|heap"
022a4000-022ee000 rw-p 00000000 00:00 0 [heap]
7fef6019e000-7fef601a5000 rw-s 00000000 00:17 21 /dev/shm/PostgreSQL.1291978332
7fef601a5000-7fef6098b000 rw-s 00000000 00:01 1052 /dev/zero (deleted) #this is shared buffers
7fef6e238000-7fef6e239000 rw-s 00000000 00:01 10 /SYSV0011f702 (deleted)

You can see the heap and shared memory area addresses roughly match.

VM
#

Linux kernel virtual memory subsystem

Directory: cd /proc/sys/vm/

compact
#

concept & param
#

Memory compaction is a mechanism in the Linux kernel for solving memory fragmentation problems. It improves the allocation and compaction efficiency of large contiguous memory pages by merging free physical pages.

Parameter	Function	Default/Range
`compact_memory`	Manually trigger a global memory compaction operation	Write 1 to trigger
`compaction_proactiveness`	Controls the frequency of proactive compaction	Parameter available since 4.x. 0-100 (default 20)
`compact_unevictable_allowed`	Whether to allow compaction of unreclaimable pages (e.g., `mlock` locked memory)	Parameter available since 4.x. 0 (disable) or 1 (allow)
`defrag_mode`	Controls the trigger strategy for memory defragmentation	Parameter available since 4.x. 0-3. 0 disables automatic compaction; 1 defers passive compaction. Default in 3.10 is 1
`extfrag_threshold`	Threshold for triggering compaction when large memory blocks are insufficient	0-1000 (default 500)

There are 3 compaction modes (depending on kernel version support):

Passive compaction: extfrag_threshold addresses “already occurred” fragmentation problems — triggered when a process requests large memory blocks and finds them insufficient.
Proactive compaction: compaction_proactiveness proactively controls compaction aggressiveness, optimizing “not yet occurred” but potential fragmentation risks.
Manual compaction: compact_memory.

extfrag_threshold is the Linux kernel parameter controlling passive compaction. When the kernel fails to allocate high-order contiguous physical memory (e.g., huge pages), it determines the failure cause via the fragmentation index:

-1: Allocation succeeded (watermark satisfied)
0: Failed due to insufficient memory
1000: Failed due to fragmentation

View specific values via /sys/kernel/debug/extfrag/extfrag_index. The output is a floating-point number (e.g., 0.854), but the actual range is magnified 1000x, so 0.854 corresponds to an actual value of 854:

cat /sys/kernel/debug/extfrag/extfrag_index |grep Normal
Node 0, zone Normal -1.000 -1.000 -1.000 -1.000 -1.000 -1.000 -1.000 -1.000 -1.000 0.995 0.998 

If extfrag_threshold=600, compaction is triggered when the fragmentation index > 600. extfrag_index is quite useful and can assist buddy in observing fragmentation issues.

dirty
#

concept & param
#

Dirty page flushing is somewhat similar to memory reclamation and is also divided into asynchronous and synchronous:

Asynchronous flushing: performed by background threads like pdflush/flush/kdmflush; application writes are not affected
Synchronous flushing: directly blocks the application process; the process that initiated the write operation flushes the dirty pages itself

Parameter Name	Description	Default
dirty_background_bytes	Background async flush threshold, in bytes	0 (disabled)
dirty_background_ratio	Background async flush threshold, as percentage	10%
dirty_bytes	Synchronous flush threshold, in bytes	0 (disabled)
dirty_ratio	Synchronous flush threshold, as percentage	20-40%
dirty_expire_centisecs	Maximum lifetime of dirty pages in memory	3000 (30s)
dirty_writeback_centisecs	Frequency of kernel periodic dirty page state checks	500 (5s)

xxx_bytes and xxx_ratio parameters are mutually exclusive.

Example parameters and flowchart:

dirty_background_bytes 0
dirty_background_ratio 10
dirty_bytes 0
dirty_ratio 40
dirty_expire_centisecs 3000
dirty_writeback_centisecs 500

%% Dirty page flushing flow diagram integrating time parameters
graph TD
 A[App writes generate dirty pages] --> B{Check interval reached?<br>dirty_writeback_centisecs every 5s}
 B -- No --> D{Expired dirty pages exist?<br> dirty_expire_centisecs>30s}
 B -- Yes --> C{Dirty page threshold check}
 C --> E[Dirty page ratio? dirty_background_ratio>10% ]
 C --> F[Dirty page ratio? dirty_ratio> 40%]
 E -- Trigger --> G[Background async flush]
 F -- Trigger --> H[Synchronous flush]
 D -- Trigger --> G
 G --> I[Dirty pages written to disk]
 H --> I[Dirty pages written to disk] 
 I --> J[Free memory space]

The configuration principles for dirty page flush parameters are basically the same as PostgreSQL dirty page flush parameters. Setting them too low causes overly frequent flushing — the same dirty page may be written to disk multiple times, wasting IO. Setting them too high may cause IO storms.

Observing Dirty Pages
#

Monitoring dirty pages:

ps -eo pid,%cpu,%mem,wchan,args,state|grep kdmflush|grep -E -w -v "S" #Observe async flush process state
cat /proc/vmstat| grep -E -w "nr_dirty|nr_writeback" #vmstat dirty, page count
cat /proc/meminfo |grep -i dirty #meminfo dirty, KB

Testing dirty pages with dd:

 grep -E "nr_dirty_threshold|nr_dirty_background_threshold" /proc/vmstat | awk '{printf "%s: %.2fGB\n", $1, ($2*4)/1048576}'
nr_dirty_threshold: 141.28GB
nr_dirty_background_threshold: 35.32GB

dd if=/dev/zero of=testfile bs=8k count=128000 # cache io

Failed test (same result after multiple tests):

No RUNNING kdmflush process observed
Dirty pages were flushed before reaching 35GB or 30S threshold

Timestamp	nr_dirty	nr_dirty(GB)	Trend Simulation
17:00:18	2,757	0.01052	▍
17:00:19	336,199	1.282	████▌
17:00:25	1,984,867	7.574	██████████████▍
17:00:32	4,252,177	16.22	████████████████████
17:00:33	3,699,227	14.11	█████████████████▊
17:00:38	170,865	0.652	▎
17:00:46	2,865,814	10.93	█████████▋
17:00:54	4,721,827	18.01	██████████████████████
17:00:55	3,876,509	14.79	██████████████████
17:01:03	835,097	3.186	██▊

os dirty != pg dirty
#

With pg fsync=on, data writes go through the OS pagecache before specific blocks are written to disk. PostgreSQL has its own dirty pages, and the OS also has dirty pages. What’s the relationship between the two?

## Observation commands
cat /proc/meminfo |grep -E -w "Dirty" # OS dirty pages

select isdirty,pinning_backends,count(*) from pg_buffercache where isdirty is true group by isdirty,pinning_backends; # PG dirty pages

checkpoint;
begin;
--Observe
insert into tlzl select generate_series(1,1000000);
--Observe
commit;
--Observe
checkpoint;
--Observe

Test results:

stage	dirty in pg	OS dirty
Clean state	0	0.02-2M fluctuating
After insert completion	200M	Rose to 1.7G, then dropped to 20KB
After commit	200M	0.02-2M fluctuating
After checkpoint flush	0	0.02-2M fluctuating

When the insert data size is increased, OS dirty rises during insert, rising to the GB level and then fluctuating.

PG dirty has some relation to OS dirty but they’re not entirely correlated. When PG inserts data, OS dirty does rise, but after the OS flushes its own dirty pages, PG’s dirty pages remain dirty. Preliminary judgment: dirty pages in shared memory are unrelated to OS dirty. It’s yet to be determined whether the OS dirty increase comes from PG’s private memory dirty pages.

swappiness
#

Controls the kernel’s bias toward reclaiming memory from the anonymous memory pool or the page cache. Essentially, it controls whether swapping anonymous pages or reclaiming file pages imposes a lower cost for the upper-layer application. For example, for compute-oriented applications using more dynamic allocation or private memory, a lower swappiness should be set; for data-dependent applications, a higher swappiness should be set to reduce the impact of flushing file pages on data access. However, all of this depends on the efficiency of swap IO and filesystem IO⁶. It all sounds ideal, but when swapping occurs, it very likely means performance degradation.

swappiness=0
#

When swappiness=0, the kernel will only swap when memory reaches the high watermark⁷. The specific strategy also relates to the kernel version and NUMA. What can be confirmed is that swappiness=0 does not mean swap is disabled — swapoff -a is what disables the swap functionality.

#Check if swap is enabled
swapon --show
free -h |grep Swap
cat /proc/swaps
grep -E 'swap|none' /etc/fstab
cat /proc/meminfo|grep Swap

#Monitor whether swapping is occurring
cat /proc/vmstat|grep swp
sar -W 1

inconsistent swap behavior
#

The OS-level /proc/sys/vm/swappiness has little-to-no effect on the swap behavior of cgroups v1 systems (has little-to-no effect on the swap). This issue can lead to inconsistent swap behavior⁸.

Occurrence conditions (all must be true):

vm.swappiness != cgroups memory.swappiness
cgroups v1

Cause:

systemd creates cgroups early during startup, before sysctl.service loads /etc/sysctl.conf. vm.swappiness cannot constrain cgroup memory.swappiness. The issue is: when the OS swap behavior and cgroup behavior differ, which one takes effect?

Solutions:

for cgroup v1, set vm.swappiness = all cgroups memory.swappiness
for cgroup v1, many solutions available, see https://access.redhat.com/solutions/6785021
Use cgroup v2. v2 adds the vm.force_cgroup_v2_swappiness parameter, which disables cgroup’s memory.swappiness

memory overcommitment
#

concept & param
#

Linux does not reserve physical memory for every virtual address; instead, it allocates memory only when actually needed. Overcommitment can limit the total virtual memory size that all processes can request. When the requested memory exceeds the defined physical memory size, it’s called overcommit.

There are three overcommit policy parameters: overcommit_memory, overcommit_ratio/overcommit_kbytes

The overcommit_memory parameter controls the overcommitment policy:

0 (default): Heuristic overcommitment policy, allows slight overcommit. CommitLimit = physical memory + swap.
1: No overcommit check
2: Strict limit, prohibits exceeding CommitLimit

graph TD
 A[Memory allocation request] --> B{Overcommit mode}
 B -->|Mode 0: Heuristic| C["Allow moderate virtual memory overcommit"]
 B -->|Mode 1: Unlimited| D["Virtual memory commits unconstrained"]
 B -->|Mode 2: Strict| E["Virtual memory total ≤ CommitLimit"]
 C --> F[Allocate physical pages on demand at runtime]
 D --> G[May exhaust physical memory + Swap]
 E --> H[Enforce virtual memory total control]

When overcommit_memory=2, only one of the overcommit_ratio and overcommit_kbytes parameters takes effect. The CommitLimit is calculated as follows: $$ CommitLimit = (RAM - huge page memory) × \frac{overcommit_ratio}{100} + SwapTotal $$ or $$ CommitLimit = (RAM - huge page memory) + overcommit_kbytes + SwapTotal $$ Interesting overcommit accounting⁹ — mmap, brk, fork are all accounted for, which clearly affects PostgreSQL:

Status
------
o	We account mmap memory mappings
o	We account mprotect changes in commit
o	We account mremap changes in size
o	We account brk
o	We account munmap
o	We report the commit status in /proc
o	Account and check on fork
o	Review stack handling/building on exec
o	SHMfs accounting
o	Implement actual limit enforcement

Reserve Memory and Overcommit
#

user_reserve_kbytes: When overcommit_memory=2, physical memory reserved for ordinary user processes. When system memory is severely insufficient, it ensures ordinary users can still perform basic operations (like starting new processes, handling memory allocation requests). Default is min(3% of the current process size, 128M). When set to 0, a single process can allocate (all free memory - admin_reserve_kbytes)

admin_reserve_kbytes: Physical memory reserved for users with CAP_SYS_ADMIN privileges (typically root user), ensuring admin recovery capability — reserved physical memory ensuring the system administrator can log in and execute commands. Default is min(3% memory, 8MB). When using strict overcommit mode, it’s best to increase this parameter.

$ cat user_reserve_kbytes 
131072
$ cat admin_reserve_kbytes 
8192

Observing Overcommit
#

grep -E 'CommitLimit|Committed_AS' /proc/meminfo
sar -r 1

$ grep -E 'CommitLimit|Committed_AS' /proc/meminfo
CommitLimit: 203103492 kB
Committed_AS: 252170700 kB

$ sar -r 1
07:32:35 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
07:32:37 PM 25472180 370249056 93.56 14588 274485956 252242936 62.91 233866528 103568816 12924
07:32:38 PM 25471904 370249332 93.56 14588 274487888 252242740 62.91 233851748 103570136 11180

Metric meanings:

meminfo CommitLimit: CommitLimit calculated from physical memory, Swap, and overcommit parameters
meminfo Committed_AS: Total virtual memory currently requested by all processes
sar -r kbcommit = Committed_AS
sar -r %commit = kbcommit / total physical memory

smaps or status can also show total requested virtual memory, but directly summing smaps/status total virtual memory double-counts shared library files and mapped files (like mmap), while Committed_AS only counts memory requested via mmap, brk, fork, etc., and does not double-count shared memory. The two have different calculation scopes. For total virtual memory, just look at Committed_AS or kbcommit.

watermark
#

Parameter Name	Description	Introduced	Default	Unit/Range
min_free_kbytes	Defines the minimum free memory the system reserves, directly affecting the watermarks `watermark[min]` calculation, ensuring the system retains enough memory for critical operations when memory is tight	Early kernel versions		KB
watermark_scale_factor	Globally adjusts the memory watermark gap (`high-low` and `low-min`)	Linux kernel 4.x (exact minor version unknown)	`10` (0.1% physical memory)	Max `3000` (30% physical memory)
watermark_boost_factor	Temporarily raises the high watermark (`high`), triggering aggressive memory reclamation to reduce fragmentation	Linux kernel 4.x (exact minor version unknown)	`15000` (i.e., 1.5x original high watermark)

min_free_kbytes
#

## Calculate total min and other values from zoneinfo
cat /proc/zoneinfo | grep -E -w "min|low|high"|grep -E -v "high:"| awk '
/min/ { total_min += $2 }
/low/ { total_low += $2 }
/high/ { total_high += $2 }
END {
 printf "Total min: %d KB\nTotal low: %d KB\nTotal high: %d KB\n",
 total_min * 4, total_low * 4, total_high * 4;
}'
Total min: 15828844 KB
Total low: 19786048 KB
Total high: 23743260 KB

#Current system min value
cat min_free_kbytes
15828849

Because there are other zones, the total min across all zones is approximately equal to min_free_kbytes. The Normal zone’s min is definitely slightly smaller than min_free_kbytes; you only need to focus on the Normal zone:

## Normal zone min, low, high settings; page=4k
cat /proc/zoneinfo | grep -A 50 Normal | grep -E "min|low|high"
 min 3931615
 low 4914518
 high 5897422

Before Linux kernel 4.6, min, low, and high had a fixed ratio, and you could only change low and high values by setting min_free_kbytes. min:low:high = 1:1.25:1.5.

Problems with the fixed ratio:

Ideally, you’d want to raise low to more proactively trigger kswapd async reclamation and lower min to reduce direct reclaim. Before 4.6, you could only indirectly adjust low/high by adjusting min, using min to adjust kswapd’s delta working buffer. For example:

	kswapd async reclamation working buffer (low-min)	kswapd async reclamation workload (high-low)
min=1GB, low=1.25GB, high=1.5GB	0.25GB	0.25GB
min=10GB, low=12.5GB, high=15GB	2.5GB	2.5GB

Raising min is done to raise low and high.

An excessively low min value causes kswapd to not have time to asynchronously reclaim more memory before direct reclaim triggers. An excessively high min not only wastes memory but also causes more frequent reclamation activity, resulting in higher sys CPU usage. The default difference between low and min in Linux indeed seems a bit small.

watermark_scale_factor
#

Wouldn’t it be great if you could directly adjust min, low, and high? Sorry, the Linux kernel doesn’t support that (Android has extra_free_kbytes). But…

Since Linux kernel 4.x, the watermark_scale_factor parameter was added, allowing adjustment of the ratios between parameters — the ratio is no longer fixed. Its default value is 10, corresponding to 0.1% of memory (10/10000), with a maximum of 3000. When set to 1000, it means the difference between “low” and “min”, and between “high” and “low”, will both be 10% of memory size (1000/10000).

0.1% is clearly too small — for 1TB of memory, the scale is only 1GB.

watermark_boost_factor
#

watermark_boost_factor is used to optimize external memory fragmentation. It temporarily raises the zone’s watermark, i.e., zone->watermark_boost, thereby raising the zone’s high watermark (WMARK_HIGH). This allows kswapd to reclaim more memory, making it easier for the memory compaction module (compactd kernel thread) to merge large blocks of contiguous physical memory. The default value of watermark_boost_factor is 15000, meaning the original high watermark is temporarily raised to 150%. Setting this to 0 disables the mechanism for temporarily raising zone watermarks¹⁰

oom
#

The OOM Killer is a kernel module, not a process.

Parameter Name	Description	Default
panic_on_oom	Controls system behavior when OOM occurs: 0: Don’t trigger panic, start OOM Killer 1: Trigger panic and halt 2: Trigger panic then attempt memory release	0
oom_kill_allocating_task	Whether to preferentially kill the process that triggered OOM (rather than traversing the process tree to select the optimal target): 0: Disabled 1: Enabled	0
oom_dump_tasks	Whether to dump all task information when OOM occurs (for post-mortem analysis): 0: Disabled 1: Enabled	1

oom_score
#

When OOM occurs, the system needs to decide which process to kill based on the OOM score. Each user process has 3 OOM score interface files:

-rw-r--r-- 1 postgres postgres 0 May 24 16:39 /proc/63766/oom_adj
-r--r--r-- 1 postgres postgres 0 May 24 16:39 /proc/63766/oom_score
-rw-r--r-- 1 postgres postgres 0 May 24 16:39 /proc/63766/oom_score_adj

oom_score is a dynamically calculated OOM score by the system, influenced at least by:

Many child processes: +points
Long-running: -points
Low nice value: +points (nice value represents process CPU time slice priority. Lower nice values mean higher priority, more CPU time slice allocation)
Direct hardware access: -points¹¹

In addition to the Linux-calculated OOM score, adjustments (adj) can be manually applied. oom_adj is from earlier Linux kernel versions; it’s best to adjust OOM scores through the oom_score_adj interface file.

Parameter/File	Purpose	Example Values
oom_score	Kernel-calculated raw score (dynamic)	0~1000
oom_score_adj	User-defined adjustment value, directly affects final score	-1000~1000; -1000 equivalent to disabling OOM
oom_adj (legacy)	Legacy adjustment parameter, range -17~15	-17~15

lowmem_reserve_ratio
#

Besides min_free_kbytes, there’s another minimum memory reserve parameter that can cause process memory allocation failures, but their functions differ significantly.

lowmem_reserve_ratio is a key kernel parameter used to protect low-end memory (DMA, DMA32) from being excessively consumed by high-end memory allocation requests. lowmem_reserve_ratio is just a coefficient, not a directly usable number; the kernel calculates the reserved page count for each zone.

#Default values below
cat /proc/sys/vm/lowmem_reserve_ratio 
256 256 32

Memory zones are ordered by priority from low to high: DMA → DMA32 → Normal → HighMem. Allocation requests from higher-priority zones can “borrow” memory from lower-priority zones, but must reserve a certain proportion of memory for use by the lower-priority zones.

cat /proc/zoneinfo |grep -Ew "Node 0|protection|free"
Node 0, zone DMA
 pages free 3976
 protection: (0, 2484, 386430, 386430)
Node 0, zone DMA32
 pages free 415741
 protection: (0, 0, 383946, 383946)
Node 0, zone Normal
 pages free 5658528
 protection: (0, 0, 0, 0)

For example, DMA’s protection indicates:

0: Allocation from this zone, no cross-zone allocation restrictions
2484: Pages DMA reserves for DMA32 zone allocations
386430: Pages DMA reserves for Normal zone allocations
386430: Reserved extension field, meaningless in this context

Based on these settings:

When DMA32 zone requests memory from DMA zone, 3976 > 2484, it may succeed
When Normal zone requests memory from DMA zone, 3976 < 386430, it will not succeed
Requests from lower zones to higher zones are not subject to this restriction

misc
#

A few more related parameters; those with less relevance are not listed:

Parameter	Purpose
nr_hugepages	Number of huge pages
~~nr_overcommit_hugepages~~	Overcommit of huge pages; The maximum is nr_hugepages + nr_overcommit_hugepages
~~nr_hugepages_mempolicy~~	NUMA-localized huge page allocation
~~hugetlb_shm_group~~	Shared memory permission control
~~hugetlb_optimize_vmemmap~~	Restructure huge page metadata management model, reducing memory usage of huge page metadata (struct page). Supported since Linux kernel 5.13
max_map_count	Limits the maximum number of memory mapping regions (VMA) a single process can have, default 65530
zone_reclaim_mode	Memory reclamation policy under NUMA, e.g., allocating memory from other nodes
stat_interval	VM stat refresh frequency, default 1 second
vfs_cache_pressure	Parameter for VFS (Virtual File System) cache reclamation pressure, mainly affecting the aggressiveness of kernel reclaiming dentry and inode caches
page-cluster	Swap readahead, swaps multiple pages to swap partition at once. Default 3, i.e., 8 pages at once

OS Memory Observation and Calculation
#

/proc/meminfo, /proc/vmstat, /proc/zoneinfo all contain memory information, much of it duplicative. I won’t list the differences — a glance tells you what’s what.

free available Calculation (Unfinished)
#

General direction: (NR_FREE_PAGES + NR_FILE_PAGES - NR_SHMEM + NR_SWAP_PAGES + NR_SLBA_RECLAIMABLE - TOTALRESERVE_PAGES - root reserved memory)

The kernel has its own estimated available memory. Directly calculating the available value using a formula is difficult to get exactly right:

## Not very accurate, don't use
cat /proc/meminfo |grep -Ew "MemFree|Active\(file\)|Inactive\(file\)|SwapFree|SReclaimable|nr_shmem|Shmem" |awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2} NR==4 {d=$2} NR==5 {e=$2} NR==6 {f=$2 ;print (a+b+c+d-e+f)}' ;
cat /proc/meminfo |grep -Ew "MemAvailable";

inactive_anon + active_anon != anon
#

Why?

Primary: Shmem separately counts shared memory pages. nr_anon_pages does not include shared memory pages, while nr_inactive_anon and nr_active_anon include anonymous shared memory pages
Secondary: anon includes some Unevictable pages (Mlocked is a subset of Unevictable)
Other minor statistical differences have little impact

A rough but relatively accurate formula: nr_inactive_anon + nr_active_anon + nr_unevictable - nr_shmem

## Applicable under huge pages; not applicable under NUMA
## /proc/meminfo, /proc/zoneinfo, /proc/vmstat can all be used for calculation

#/proc/vmstat
echo -n "anon_computed : ";cat /proc/vmstat|egrep -w "nr_inactive_anon|nr_active_anon|nr_unevictable|nr_shmem"| awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2} NR==4 {d=$2; print (a+b+c-d)}' ;\
echo -n "anon_real : ";cat /proc/vmstat|egrep -w "nr_anon_pages"|awk '{print $2}'
anon_computed : 15776924
anon_real : 15772671

##/proc/zoneinfo Normal
echo -n "anon_normal_computed : "; cat /proc/zoneinfo |grep Normal -A 50|egrep -w "nr_inactive_anon|nr_active_anon|nr_unevictable|nr_shmem"| awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2} NR==4 {d=$2; print (a+b+c-d)}' ;\
echo -n "anon_normal_real : "; cat /proc/zoneinfo |grep Normal -A 50|egrep -w "nr_anon_pages"|awk '{print $2}'
anon_normal_computed : 15711170
anon_normal_real : 15707402

cache Calculation
#

The buff/cache shown in the free command can be calculated from file pages or cache itself:

echo -n "filepage+shmem: ";cat /proc/meminfo |grep -Ew "Buffers|Active\(file\)|Inactive\(file\)|Shmem|SReclaimable"| awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2} NR==4 {d=$2} NR==5 {e=$2 ;print (a+b+c+d+e)}';\
echo -n "cached: ";cat /proc/meminfo |grep -Ew "Buffers|Cached|SReclaimable" | awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2 ;print (a+b+c)}';\
free -k;

#Execution results:
filepage+shmem: 289417584
cached: 289419156
 total used free shared buff/cache available
Mem: 395721236 79633516 26668564 84704912 289419156 178501152
Swap: 5242876 0 5242876

Controversy: Does shmem Count as cache?
#

Clearly, the calculation above includes shmem in cache. Theoretically, shmem shouldn’t be part of cache.

In fact, the kernel community has discussed thisWhy is Shmem included in Cached in /proc/meminfo?, wanting to remove shared memory from cache:

> -	cached = global_node_page_state(NR_FILE_PAGES) -
> -			total_swapcache_pages() - i.bufferram;
> +	cached = global_node_page_state(NR_FILE_PAGES) -
> +			total_swapcache_pages()
> +			- i.bufferram - i.sharedram;

But modifying this involves forward compatibility concerns. The question comes down to: which is more important — forward compatibility or improving the accuracy of a piece of information?

Currently, there’s no good resolution; that’s the status quo.

The email thread also discusses some interesting things:

Another point of view is that everything in tmpfs is part of the page
cache and can be written out to swap

- Dirty: total amount of RAM used to buffer data to be written on
permanent storage (dirty). Gets converted to Cached when write is
complete. (Actually I would call this "Buffers" but Dirty is okay, too.)
- Cached: total amount of RAM used to improve *performance* that can be
*immediately dropped* without any data-loss – note that this includes
all untouched RAM backed by swap.
- Shared: total amount of RAM shared between multiple process that
cannot be freed even if any single process gets killed. (If this is even
possible to know - note that this would *only* contain COW pages in
practice. We already have Committed_AS which is about as good for real
world heuristics.)

cache does not include dirty pages, and can be directly dropped without data loss
tmpfs is swapout

Shared memory appears to be swapout, which is clearly different from cache pages that can be directly dropped. PostgreSQL’s shared memory clearly cannot be directly dropped.

So for PostgreSQL, the fact that cache contains shared memory is quite important — don’t assume by default that it doesn’t.

Memory Page Statistics Often Don’t Add Up
#

When calculating memory pages, some calculations don’t add up. Summary of reasons:

shmem is counted in cache
Cannot see file-mapped and anonymous-mapped pages within shmem
nr_anon_pages does not include shared memory pages, while nr_inactive_anon and nr_active_anon include anonymous shared memory pages
VM and cgroup have slightly different statistical scopes

cgroup v1
#

cgroup Memory Management
#

cgroup can observe and limit the usage of anonymous pages, file pages, swap cache, and kernel memory. Each memcg has its own independent LRU; there is no concept of a GLOBAL LRU.

cgroup memory management differs from cgroup CPU management. A task can request lots of CPU work; reaching the cgroup CPU limit can extend execution time to handle it. However, the memory a task occupies is working memory — a task uses the same physical memory.

Key differences between cgroup CPU and memory management:

Memory must be managed through reuse and reclamation; a task’s working memory is truly occupied and cannot be used by other tasks. CPU is managed through time allocation; other tasks or cgroups can use it.
Memory needs to be instantly available; CPU works through time slicing — time can be dispersed.
CPU control’s core is time allocation; Memory Control’s core is page counting.

The core of the design is a counter called the page_counter. The page_counter tracks the current memory usage and limit of the group of processes associated with the controller

Memory Control’s core is page counting, meaning it’s not that physical pages are statically assigned. The memory allocated this time, when released back to free after use, most likely won’t be the same physical page next time¹².

Physical pages know which cgroup they belong to:

				+--------------------+
				| mem_cgroup |
				| (page_counter) |
				+--------------------+
				 / ^ \
				/ | \
 +---------------+ | +---------------+
 | mm_struct | |.... | mm_struct |
 | | | | |
 +---------------+ | +---------------+
 |
 + --------------+
 |
 +---------------+ +------+--------+
 | page +----------> page_cgroup|
 | | | |
 +---------------+ +---------------+

mm_struct represents virtual memory. Each virtual memory knows which cgroup it belongs to; each physical page can point to page_cgroup, meaning it knows which cgroup this physical memory belongs to¹².

cgroup Parameters and Metrics
#

cgroup uses interface files for configuration and viewing memory usage.

Directory: cd /sys/fs/cgroup/memory/xxx/

Kernel memory and mem+swap can have separate settings or usage viewing:

memory.kmem.xxx #kernel mem
memory.memsw.xxx #mem+swap

Below, we only look at mem-related items.

Interface files can be divided into three categories:

Read-only — show usage, permissions: -r--r--r--
Read-write — control parameters, permissions: -rw-r--r--
Other — special settings, permissions: other

Specific meanings are as follows, with important parameters highlighted:

Type	Interface File	Meaning
Read-only	`memory.numa_stat`	NUMA-dimensional memory stats
Read-only	`memory.stat`	Important, the primary memory usage interface file with many metrics; analyzed separately below
Read-only	`memory.usage_in_bytes`	usage_in_bytes is affected by the method and doesn’t show ’exact’ value of memory. Not recommended for viewing cgroup memory usage
Read-only	`memory.failcnt`	Number of times memory usage exceeded `memory.limit_in_bytes`, cumulative
Read-write	`cgroup.clone_children`	Controls whether child cgroups inherit parent configuration
Read-write	`cgroup.procs`	Used to manage process groups (process IDs, PIDs) in the current cgroup. For multi-process PostgreSQL, this means writing all PG processes, including management processes and backends, into the `procs` file
Read-write	`tasks`	Used to manage threads (thread IDs, TIDs) in the current cgroup. When writing a process PID to `cgroup.procs`, all its thread TIDs are automatically added to `tasks`
Read-write	`notify_on_release`	Controls whether a release operation is triggered when the last task (process or thread) in the cgroup exits. Would only be enabled for container management; traditional cgroup management keeps it disabled by default. Cgroups should be preserved after database restart
Read-write	memory.move_charge_at_immigrate	Deprecated in v2. Charge attribution rules when migrating cgroups
Read-write	memory.use_hierarchy	Whether parent cgroup limits child cgroups
Read-write	memory.limit_in_bytes	cgroup memory upper limit
Read-write	memory.soft_limit_in_bytes	Reclaim the portion exceeding the soft limit
Read-write	memory.max_usage_in_bytes	cgroup usage peak, an observation metric
Read-write	memory.oom_control	oom_kill_disable 1 — disable OOM under_oom 0 — whether currently in OOM state
Read-write	memory.swappiness	cgroup-level swappiness
Other	memory.force_empty	Write only; writing `0` forces release of all cgroup memory
Other	cgroup.event_control	Event notification interface, listens for memory pressure events, requires programming. Often used with memory.pressure_level
Other	memory.pressure_level	Memory pressure notification level

Using a PG instance to explain the meaning of various metrics in memory.stat.

This PG instance is configured as:

shared_memory_type=mmap
shared_buffers=64GB
approximately 800 clients, running

cat memory.stat

cache 345587761152 						 #page cache!!!
rss 27332608 #Anonymous and swap cache memory size. Note: differs from OS process RSS; clearly doesn't include PG shared memory
rss_huge 0 #of bytes of anonymous transparent hugepages. Note: anonymous huge pages
mapped_file 61491769344 #File shared memory size; includes PG shared memory here
swap 0 #On swap partition
pgpgin 389395357 #rss+cache charged pages
pgpgout 305016672 #rss+cache uncharged pages
pgfault 1954040341 #Omitted
pgmajfault 17 #Omitted
inactive_anon 165728256 #anonymous and swap cache memory on inactive LRU
active_anon 61549518848 #anonymous and swap cache memory on active LRU list
inactive_file 138240962560 #file-backed on inactive LRU list
active_file 145658613760 #file-backed memory on active LRU list
unevictable 0 #Unreclaimable memory
hierarchical_memory_limit 408021893120 #
hierarchical_memsw_limit 9223372036854771712 #
total_xxx #hierarchical 

Roughly (ignoring swap), cache+rss = inactive_anon+active_anon+inactive_file+active_file.

These values are quite convoluted. cache+rss doesn’t have a straightforward correspondence with [in]active_anon/file, and mapped_file (shared memory) is hard to categorize, making it easy to get confused. Combining various documentation and testing, I hand-rolled the following script:

#cginfo_lzl
echo -n "shared_mem_mapped : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "mapped_file"| awk '{print $2 / 1024 / 1024 /1024 }' ;\
echo -n "shared_mem_anon : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "rss|inactive_anon|active_anon"| awk 'NR==1 {a=$2} NR==2 {b=$2} NR==3 {c=$2; print (b + c -a)/1024/1024/1024}' ;\
echo -n "pagecache : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "cache"| awk '{print $2 / 1024 / 1024 /1024 }' ;\
echo -n "pagecache_cache-share : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "cache|mapped_file"| awk 'NR==1 {a=$2} NR==2 {b=$2; print (a - b)/1024/1024/1024}';\\n
echo -n "file_total : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "inactive_file|active_file"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "anon_total : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "inactive_anon|active_anon"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_used_rss+map : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "rss|mapped_file"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_mem_file+rss+map : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "inactive_file|active_file|rss|mapped_file"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_mem_rss+cache : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "rss|cache"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_mem_anon+file : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "inactive_file|active_file|inactive_anon|active_anon"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "total_memsw : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.stat|egrep -w "rss|cache|swap"| awk '{sum += $2} END {print sum /1024/1024/1024}';\\
echo -n "hard_limit : ";cat /sys/fs/cgroup/memory/$PGNAME/memory.limit_in_bytes| awk '{print $1 / 1024 / 1024 /1024 }'

#Database with shared_buffers=2GB
shared_mem_mapped : 1.69063
shared_mem_anon : 1.69828
pagecache : 5.94717
pagecache_cache-share : 4.25654
file_cache : 4.24889
anon_cache : 3.23096
total_used_rss+map : 3.2233
total_mem_file+rss+map : 7.47219
total_mem_rss+cache : 7.47984
total_mem_anon+file : 7.47984
total_memsw : 7.47984
hard_limit : 8

Differences Between cgroup RSS and Process RSS
#

#shared_buffers= 64GB, all PG process RSS sorted
ps -eo pid,ppid,rss,args |grep `cat $PGDATA/postmaster.pid|head -1`|sort -k3 -rn
 97632 97627 61103720 postgres: lzlinst: checkpointer 
 97633 97627 59045152 postgres: lzlinst: background writer 
 97627 1 2322820 /paic/postgres/base/11.3/bin/postgres -D /paic/pg6888/data
 97637 97627 85116 postgres: lzlinst: pgsentinel 
 97697 97627 19620 postgres: lzlinst: dbmgr users [local] idle
 97634 97627 17932 postgres: lzlinst: walwriter 
250063 97627 14508 postgres: lzlinst: dbmon postgres [local] idle
 97636 97627 13220 postgres: lzlinst: stats collector 
248777 97627 11576 postgres: lzlinst: dbmon postgres [local] idle
 97635 97627 2980 postgres: lzlinst: autovacuum launcher 
 97638 97627 2376 postgres: lzlinst: logical replication launcher 
 97630 97627 1592 postgres: lzlinst: logger 
250185 39130 972 grep --color=auto 97627

Generally, the PG processes with the highest RSS values are checkpointer and bgwriter, because RSS represents actual memory used, including shared memory, and these two processes that flush shared buffer dirty pages occupy the most. Backends with excessive data queries may also have higher RSS values, but this is usually caused by data extracts or slow full-scan queries.

Why is postmaster’s RSS so small? Because postmaster itself doesn’t need to do much shared_buffer operations; it only needs to open up the shared memory virtual address space and fork it for other processes to use.

PM’s child processes have the same shared memory address but not necessarily the same RSS:

$ cat /proc/97632/smaps |grep -A 3 "zero" #checkpointer
2b4fd87cf000-2b60a2143000 rw-s 00000000 00:04 15925397 /dev/zero (deleted)
Size: 70411728 kB
Rss: 61087812 kB
Pss: 31429895 kB
$ cat /proc/97633/smaps |grep -A 3 "zero" #bgwriter
2b4fd87cf000-2b60a2143000 rw-s 00000000 00:04 15925397 /dev/zero (deleted)
Size: 70411728 kB
Rss: 59043388 kB
Pss: 29394787 kB
$ cat /proc/97627/smaps |grep -A 3 "zero" #postmaster
2b4fd87cf000-2b60a2143000 rw-s 00000000 00:04 15925397 /dev/zero (deleted)
Size: 70411728 kB
Rss: 2318408 kB
Pss: 1741764 kB

Above, checkpointer and bgwriter occupy the most RSS, and most of their RSS is shared memory. These two processes almost evenly split the entire actually-used shared memory, while postmaster doesn’t use much. PM and all its forked child processes have the same shared memory virtual address.

But cgroup RSS is only a few tens of MB, far less than process RSS:

cat /sys/fs/cgroup/memory/lzlinst/memory.stat |egrep -w "rss|mapped_file"
rss 88997888
mapped_file 52963262464

You can see that PG shared memory is not in the cgroup stat RSS. cgroup RSS doesn’t count file pages or shared file pages.

linux kernel¹²:

Only anonymous and swap cache memory is listed as part of ‘rss’ stat. This should not be confused with the true ‘resident set size’ or the amount of physical memory used by the cgroup.

Process vs. cgroup memory statistics differences¹³:

Memory	Single Process	Process `cgroup(memcg)`
`cache`	None	`PageCache`
`mapped_file`	None	`file_rss + shmem_rss`
`RSS`	`anon_rss + file_rss ＋ shmem_rss`	`anon_rss`

For PostgreSQL, the RSS in stat does not include file map shared memory. The PG official documentation describes mmap as anonymous shared memory:

Possible values are mmap (for anonymous shared memory allocated using mmap), sysv (for System V shared memory allocated via shmget)

cgroup counts PG mmap memory as mapped_file.

Observing sysv and huge page scenarios, summary of PG’s memory.stat metrics:

RSS in stat does not include file map shared memory. Observation shows that regardless of mmap or sysv, RSS does not contain PG shared memory
Similarly, rss_huge also does not include file map shared huge page memory. Observation shows that even with huge pages enabled, stat does not contain PG shared memory
Without huge pages, PG shared memory (mmap or sysv) is all counted under memory.stat mapped_file; with huge pages, it’s in none of the stat metrics, including rss_huge

Where Exactly Is mapped_file?
#

mapped_file is in cache, and also in inactive_anon+active_anon
mapped_file can also be anonymous; both mmap and sysv are counted here

#Database with shared_buffers=2GB
shared_mem_mapped : 1.69063
shared_mem_anon : 1.69828
pagecache : 5.94717
pagecache_cache-share : 4.25654
file_cache : 4.24889
anon_cache : 3.23096
total_used_rss+map : 3.2233
total_mem_file+rss+map : 7.47219
total_mem_rss+cache : 7.47984
total_mem_anon+file : 7.47984
total_memsw : 7.47984
hard_limit : 8

soft_limit_in_bytes
#

Soft limit (memory.soft_limit_in_bytes) is a non-enforced constraint in cgroup memory management. When a cgroup’s memory usage exceeds the soft limit, the system does not immediately force memory reclamation. Instead, it will preferentially reclaim the excess memory of that cgroup when global memory pressure is high (e.g., when overall system free memory is insufficient).

Trigger condition: Global memory pressure (e.g., insufficient system free memory).
Call path: kswapd → balance_pgdat → check cgroup soft limits → trigger reclamation.
Reclamation target: Preferentially reclaim memory pages from cgroups exceeding their soft limits.

+-------------------+ Global memory pressure detection +-------------------+
| kswapd thread | ------------------------------------> | balance_pgdat |
+-------------------+ +-------------------+
 |
 | Traverse memory zones and check
 v
 +---------------------------+
 | Check each cgroup's soft |
 | limit usage |
 +---------------------------+
 |
 | Trigger reclamation for over-limit cgroups
 v
 +---------------------------+
 | Page reclamation (LRU list |
 | scanning, etc.) |
 +---------------------------+

The soft_limit_in_bytes mechanism is very similar to high. In v2, soft_limit_in_bytes has been deprecated, replaced by three new parameters: min, low, and high.

Impact of Overselling on pagecache
#

To be discussed later

cg oom
#

Normally, if sharedbuffer = 1/4 of cg mem, then without counting private memory, pagecache can reach up to 3/4 of cg mem. Generally, normal business private memory usage won’t be very high. If cg mem is full, memory can be reclaimed from cg pagecache (this is direct memory reclamation; AliOS has implemented async background reclamation: Memcg Background Async Reclamation). So the best way to test cg oom is to use sessions that consume lots of private memory rather than stress testing.

Test case:

#Observe score
-r--r--r-- 1 postgres postgres 0 May 24 16:39 /proc/63766/oom_score
rss # whichever command you like

## A SQL that can consume lots of private memory, many union alls create many plan nodes
psql -d lzldb -tX -c "create table lzl1(col1 varchar(1));"
psql -tX -c "\o sqltext.sql" -c "
SELECT 'select col1 from lzl1' || ' union all'
FROM generate_series(1, 100000)
UNION ALL
SELECT 'select col1 from lzl1;'
FROM generate_series(1, 1);
" 

#Adjust stack parameter otherwise SQL will be aborted
psql -d lzldb -c "set max_stack_depth=1024000" -f sqltext.sql

cg oom off:

wchan shows OOM information, even an oom score, but the process won’t be killed by the OOM killer

## vm oom enabled; 0: don't trigger panic, start OOM Killer
$ cat /proc/sys/vm/panic_on_oom 
0

## cg oom disabled; 1: disable oom
$ cat /sys/fs/cgroup/memory/$PGNAME/memory.oom_control
oom_kill_disable 1
under_oom 0

$ ps -eo user,ppid,pid,state,%cpu,%mem,stime,wchan:14,args,rss,vsz,sig_block |grep `head -1 $PGDATA/postmaster.pid` |grep -v grep 
postgres 19005 870 D 0.0 0.0 10:54 mem_cgroup_oom postgres: pg3ymhp2: lzluser 7216 2807460 0000000000000000
postgres 19005 3417 S 0.0 0.0 10:55 pipe_wait postgres: pg3ymhp2: lzluser 22944 2808540 0000000000000000
postgres 19005 13069 D 0.0 0.0 11:10 mem_cgroup_oom postgres: pg3ymhp2: lzluser 11944 2808348 0000000000000000
postgres 19005 13104 D 0.0 0.0 11:10 mem_cgroup_oom postgres: pg3ymhp2: lzluser 12224 2808348 0000000000000000
postgres 19005 14352 D 0.0 0.0 11:10 mem_cgroup_oom postgres: pg3ymhp2: lzluser 11680 2808348 0000000000000000

cat /sys/fs/cgroup/memory/$PGNAME/memory.oom_control
oom_kill_disable 1
under_oom 1

cat /proc/97994/oom_score
11

shared_mem_mapped : 2.00019
shared_mem_anon : 2.0023
pagecache : 2.0023
pagecache_cache-share : 0.00211334
file_cache : 0
anon_cache : 8
total_used_rss+map : 7.99789
total_mem_file+rss+map : 7.99789
total_mem_rss+cache : 8
total_mem_anon+file : 8
total_memsw : 8
hard_limit : 8

Currently, it appears that PG processes may also crash when unable to allocate memory. For example, if walwriter crashes, it can cause all other processes to crash.

cg oom on:

User processes are killed due to high OOM score, sent kill -9. Most PG processes crash; postmaster reset_shared() then automatically restarts other processes. Both message and dmesg show out-of-memory information:

#cg oom enabled
oom_kill_disable 0

pg log:
2025-05-29 19:10:45.945 CST,,,198877,,6838374d.308dd,4,,2025-05-29 18:30:37 CST,,0,LOG,00000,"server process (PID 236413) was terminated by signal 9: Killed","Failed process was running: select col1 from lzl1 union all

message:
May 29 19:10:45 lzlhost kernel: Memory cgroup stats for /t1lzldb: cache:8392988KB rss:8384228KB rss_huge:0KB mapped_file:7458316KB swap:0KB inactive_anon:1310184KB active_anon:15467032KB inactive_file:0KB active_file:0KB unevictable:0KB
May 29 19:10:45 lzlhost kernel: Memory cgroup out of memory: Kill process 236413 (postgres) score 497 or sacrifice child

dmesg:
[Thu May 29 18:26:27 2025] Memory cgroup stats for /t1lzldb: cache:8392988KB rss:8384228KB rss_huge:0KB mapped_file:7458316KB swap:0KB inactive_anon:1310184KB active_anon:15467032KB inactive_file:0KB active_file:0KB unevictable:0KB
[Thu May 29 18:26:27 2025] Memory cgroup out of memory: Kill process 236413 (postgres) score 497 or sacrifice child
[Thu May 29 18:26:27 2025] Killed process 236413 (postgres) total-vm:18828736kB, anon-rss:8328252kB, file-rss:2328kB, shmem-rss:1832kB

Management differences between cg oom on and off for PG databases:

on: cg oom killer will kill processes with high OOM score, typically user processes
off: cg oom killer won’t start. PG processes will hang — they may recover on their own, but PG’s critical processes (like walwriter) might crash due to insufficient memory, and the instance may still go down.

Note: this is cg oom, not vm oom. System-level vm oom is determined by the system-level vm overcommit mechanism.

cg v1 Problems
#

No cg pagetable statistics
No cg slab statistics
No cg hugepage statistics (hugepages are not charged, not just not counted)
No cg async/sync page reclamation statistics
cg RSS and process RSS have different statistical scopes
shmem statistics are messy

What’s New in V2
#

V2 Officially released in Linux 4.5 (March 2016)¹⁴.

cgroup v2 memory management improvements and changes:¹⁵

cg mem interface file	vs v1	Meaning
memory.current	Reworked	Current memory usage. Removes the less useful usage_in_bytes
memory.min	New	Different from VM’s min/low/high. VM watermarks are about remaining OS memory; cg v2 watermarks are about cg memory used. memory.min is a hard memory protection value, default 0. Even when the system has no reclaimable memory, memory at or below this boundary won’t be reclaimed¹⁶
memory.low	New	Best-effort memory protection value, default 0. System preferentially reclaims memory from unprotected cgroups. If still insufficient, reclaims memory between memory.min and memory.low.
memory.high	New	Memory reclamation warning threshold, default max. When cgroup memory usage reaches high, triggers synchronous memory reclamation for this cgroup and children, trying to keep memory below high
memory.max	Reworked	Equivalent to memory.limit_in_bytes
memory.reclaim	Reworked	Active reclamation interface file. v1 only had memory.force_empty for forced clearing
memory.peak	Reworked	Equivalent to max_usage_in_bytes; exceeding peak triggers cg oom killer
memory.oom.group	New	Controls whether cg OOM killer terminates the entire cgroup (1) or just a single process (0). Default 0. If oom_score_adj=-1000, process won’t be killed
memory.events	New	Reports memory-related events
memory.stat	Reworked	Many changes, analyzed separately
memory.zswap.current, memory.zswap.max, memory.zswap.writeback	New	Zswap is a compressed swap mechanism in the Linux kernel. Through compressing memory pages awaiting swap, it reduces disk I/O operations, improving system performance. Its core idea is to compress swap data that would have been written to disk and temporarily store it in memory, only writing data to physical swap devices (like swap partitions or files) when necessary
soft_limit_in_bytes	Removed
memory.oom_control	Removed	This means v2 cannot directly disable cg oom killer; however, fine-grained memory management can be achieved through min/low/high settings and event memory notifications

v2 cg mem management advantages:

Compared to v1, v2 has simpler and clearer hierarchical management
v1 only had OOM kill or freeze; v2 has more means to control memory size (such as memory.min/low/high)
v2 makes it easier to control burst loads¹⁷
Removes the interface file for directly disabling cg oom killer
Adds memory_hugetlb_accounting

memory.stat:

Parameter	Meaning	v1 Counterpart
anon	Anonymous pages	active_anon+inactive_anon
file	File pages, including tmpfs	active_file+inactive_file
kernel (npn)	Total kernel memory, including kernel_stack, pagetables, percpu, vmalloc, slab, and other kernel memory usage.	New
kernel_stack	Memory occupied by kernel stacks.	New
pagetables	page tables	New
sec_pagetables	Secondary page tables, suitable for VMs, GPU devices, network acceleration cards, and other hardware resource isolation scenarios	New
percpu (npn)	Memory size used for per-cpu kernel data structures	New
sock (npn)	network transmission buffers	New
vmalloc (npn)	vmalloc	New
shmem	Including tmpfs, shm, shared anonymous mmap	New
zswap	Memory consumed by zswap compression itself	New
zswapped	Amount of user memory zswapped	New
file_mapped	mmap() size	Somewhat similar to v1 mapped_file, though mapped_file includes tmpfs, shm
file_dirty		Same as v1 dirty
file_writeback		Same as v1 writeback
swapcached		Same as v1 swapcached
anon_thp	Anonymous pages in transparent huge pages	New
file_thp	File pages in transparent huge pages	New
shmem_thp	Transparent huge pages for shm, tmpfs, anonymous mmap	New
inactive_anon, active_anon, inactive_file, active_file, unevictable		Same as v1
slab_reclaimable	As the name suggests	New
slab_unreclaimable	As the name suggests	New
slab (npn)	As the name suggests	New
workingset_refault_anon, workingset_refault_file, workingset_activate_anon, workingset_activate_file, workingset_restore_anon, workingset_restore_file, workingset_nodereclaim	Refaulted page statistics	New
pswpin (npn)	swap in	Same as v1 pgpgin
pswpout (npn)	swap out	Same as v1 pgpgout
pgscan (npn)	scanned pages (in an inactive LRU list)	New
pgsteal (npn)	Reclaimed memory	New
pgscan_kswapd (npn)	As the name suggests	New
pgscan_direct (npn)	As the name suggests	New
pgscan_khugepaged (npn)	Pages scanned by the transparent huge page daemon	New
pgscan_proactive (npn)	Pages scanned proactively	New
pgsteal_kswapd (npn), pgsteal_direct (npn), pgsteal_khugepaged (npn), pgsteal_proactive (npn)	As the name suggests; pgsteal\* corresponds to pgscan\*	New
pgfault (npn)	As the name suggests	Same as v1 pgfault
pgmajfault (npn)	As the name suggests	Same as v1 pgmajfault
pgrefill (npn)	Pages scanned in active LRU	New
pgactivate (npn)	Pages moved to active LRU	New
pgdeactivate (npn)	Pages moved to inactive LRU	New
pglazyfree (npn)	Pages whose release is deferred when under memory pressure	New
pglazyfreed (npn)	Reclaimed lazyfree pages	New
swpin_zero,swpout_zero	zero-filled pages; during Swap In, when the kernel detects page content is all zeros (Zero-filled), marks the page as “zero page” in metadata, skipping disk I/O	New
zswpin,zswpout,zswpwb	zswap-related pages	New
thp_fault_alloc (npn), thp_collapse_alloc (npn), thp_swpout (npn), thp_swpout_fallback (npn)	Transparent huge page-related pages	New
numa_pages_migrated (npn), numa_pte_updates (npn), numa_hint_faults (npn)	NUMA-related pages; also memory.numa_stat exists	New
pgdemote_kswapd, pgdemote_direct, pgdemote_khugepaged, pgdemote_proactive	Unclear what demote means	New
hugetlb	Huge pages	New

v2 cg mem observation advantages:

Adds slab, pagetable, pgscank/pgscand/pgsteal, and huge page info — none of which v1 had
More observation metrics related to specific features, such as sock, vmalloc, transparent huge pages, zswap compression interactions, swap_zero zero-fill interactions, etc.
Shared memory shmem and file_mapped metrics are separated

wchan
#

Waiting Channel, name of the kernel function in which the process is sleeping

Generally, you should check the wchan of processes in D state to see what kernel function the process is waiting on.

-: Running tasks will display a dash (’-’) in this column

poll_schedule_timeout: Common for PM, usually in running state

zz ***Fri May 2 04:50:10 CST 2025
postgres 141378 1 19 0.5 0.4 70585180 2322876 poll_schedule_timeout S 21:06:18 00:02:40 /paic/postgres/base/11.3/bin/postgres -D /paic/pg6888/data
zzz ***Fri May 2 04:50:43 CST 2025
postgres 141378 1 19 0.5 0.4 70585180 2322876 - R 21:06:18 00:02:42 /paic/postgres/base/11.3/bin/postgres -D /paic/pg6888/data

futex_wait_queue_me: Common for SLEEP processes. Occasionally D state

postgres 455358 141378 19 4.7 1.0 70590684 5349576 futex_wait_queue_me S 03:01:12 00:02:47 postgres: t1lzldb: lzl test3 30.181.32.3(39801) COMMIT

hugetlb_fault: Only seen when huge pages are first loaded and load starts up

do_last: Function in the VFS (Virtual File System) path resolution logic, responsible for handling the last component of a file path (such as filename or symbolic link) and triggering actual file operations

lock_page_killable: Lock a physical memory page in an interruptible manner. “Interruptible” means the process is allowed to respond to fatal signals like SIGKILL while waiting for the page lock

rpc_wait_bit_killable: This function relates to the Remote Procedure Call (RPC) mechanism, used in the kernel to wait for changes to certain bit flags

wait_on_page_bit: Wait for changes to page flag states (e.g., PG_locked, PG_writeback)

blkdev_issue_flush: Block device layer cache flush function. Possible call chain: user calls fsync() → file system (e.g., ext4) submits relevant dirty pages to the block device layer → calls blkdev_issue_flush() to ensure device cache is flushed

on_proc_exit: Register cleanup functions for process exit

ima_file_check: Belongs to the IMA (Integrity Measurement Architecture) subsystem, used to verify file integrity during file access; typically involved with open() calls

flush_work: Wait for task completion

call_rwsem_down_write_failed: When attempting to acquire a write lock (down_write()) fails, this function handles write lock contention and waiting logic. It uses spin or sleep mechanisms to make the current process wait for lock release (rwsem: read-write semaphore)

get_request: Appears when iowait is high. Gets a free request structure (struct request) from the block device request queue. If the queue is full (device processing speed insufficient), the thread waits until a request is available

lookup_slow: Slow path for VFS (Virtual File System) path resolution

/**
 * lookup_fast - do fast lockless (but racy) lookup of a dentry
 * @nd: current nameidata
 *
 * Do a fast, but racy lookup in the dcache for the given dentry, and
 * revalidate it. Returns a valid dentry pointer or NULL if one wasn't
 * found. On error, an ERR_PTR will be returned.
 */
 static struct dentry *lookup_fast(struct nameidata *nd)

/* Fast lookup failed, do it the slow way */
static struct dentry *__lookup_slow(const struct qstr *name,
				 struct dentry *dir,
				 unsigned int flags)

static struct dentry *lookup_slow(const struct qstr *name,
				 struct dentry *dir,
				 unsigned int flags)
{
	struct inode *inode = dir->d_inode;
	struct dentry *res;
	inode_lock_shared(inode);
	res = __lookup_slow(name, dir, flags);
	inode_unlock_shared(inode);
	return res;
}

lookup_fast and lookup_slow both search for dentries and return them. lookup_fast searches in the dentry cache; if it fails, lookup_slow is used.

Stress testing with huge pages enabled, no direct memory reclamation, the following events occurred:

lock_page: Appears when iowait is high. When the kernel attempts to lock a memory page, if the page is already locked by another thread/process, the current thread enters a waiting state.

vx_svar_sleep_unlock, vx_ilock, vx_bc_biowait, vx_dio_physio, vx_rwsleep_lock:

vx is a journaling file system developed by Veritas (now owned by Symantec and subsequently spun off as Veritas Technologies), designed for high-performance, high-availability large-scale data storage, primarily targeting enterprise application scenarios. Like xfs and ext4, it is a type of file system.

pipe_wait: When a process attempts to read from or write to a pipe, if the pipe buffer is full (write operation) or empty (read operation), the current thread enters sleep state, waiting for buffer state changes

pipe_write: Entry function for pipe write operations. When the buffer is full, the thread sleeps in this function, waiting for writable space

congestion_wait: When the block device I/O queue is congested (e.g., request queue full or device processing delayed), the kernel uses this function to briefly sleep the thread

wait_iff_congested: Checks whether the block device queue is congested and enters brief sleep if so. Similar to congestion_wait but more lightweight, typically used in memory reclamation or dirty page writeback paths

mem_cgroup_oom_synchronize: When usage_in_bytes reaches limit_in_bytes, marks oom_control.under_oom=1. Whether the OOM killer kernel module is activated depends on oom_control.oom_kill_disable

mem_cgroup_oom: Same as mem_cgroup_oom_synchronize

rmap_walk
#

One of PFRA’s goals is to reclaim shared page frames. To achieve this, the Linux 2.6 kernel can quickly locate all page table entries pointing to the same page frame — this process is called reverse mapping[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)].

When a page frame already referenced by one process is inserted into another process’s page table entries (fork), rmap_walk should also occur

zcat hostlzl_ps_25.04.08.0900.dat.gz|egrep "\-D /dirlzl/pg5998/data|zzz"|less
zzz ***Tue Apr 8 09:10:50 CST 2025
postgres 209987 1 19 0.2 0.5 70247548 2117844 poll_schedule_timeout S 22:17:21 00:01:56 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
zzz ***Tue Apr 8 09:11:20 CST 2025
postgres 209987 1 19 0.2 0.5 70247548 2117844 poll_schedule_timeout S 22:17:21 00:01:56 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
zzz ***Tue Apr 8 09:13:08 CST 2025
postgres 209987 1 19 0.2 0.5 70247548 2117844 - D 22:17:21 00:01:57 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
postgres 225076 209987 19 1.6 0.0 70247548 1720 rmap_walk D 09:11:51 00:00:01 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
postgres 224924 209987 19 0.7 0.0 70247548 1728 rmap_walk D 09:11:46 00:00:00 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
postgres 224817 209987 19 0.5 0.0 70247548 1720 try_to_unmap_file D 09:11:44 00:00:00 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
zzz ***Tue Apr 8 09:19:16 CST 2025
postgres 209987 1 19 0.3 0.5 70247548 2117884 poll_schedule_timeout S 22:17:21 00:02:00 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data
postgres 250875 209987 19 0.0 0.0 70247548 2208 - R 09:19:17 00:00:00 /dirlzl/postgres/base/postgressqlbin/postgresdb -D /dirlzl/pg5998/data
zzz ***Tue Apr 8 09:19:48 CST 2025
postgres 209987 1 19 0.3 0.5 70247548 2117884 poll_schedule_timeout S 22:17:21 00:02:01 /dirlzl/postgres/base/postgressql/bin/postgresdb -D /dirlzl/pg5998/data

try_to_unmap_file
#

The try_to_unmap_file() function calls try_to_unmap_cluster(), and try_to_unmap_cluster() scans all page table entries corresponding to linear addresses in that linear region, attempting to clear them[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]. try_to_unmap_file() performs reverse mapping of mapped pages. Note: reverse mapping means finding all VMAs through the page table and reclaiming shared physical page frames.

page_referenced
#

referenced and active are used to control page activity level and are used in page reclamation. When refcount=0, it indicates free pages or pages about to be released[^《奔跑吧 Linux内核入门篇（第2版）》 (Running Linux Kernel: Beginner’s Guide 2nd Edition)].

In kernel.org doc’s Object-Based Reverse Mapping, there is a description of the page_referenced() function³:

page_referenced() which checks all PTEs that map a page to see if the page has been referenced recently

page_referenced() calls page_referenced_obj() which is the top level function for finding all PTEs within VMAs that map the page.

If a page is mapped and it is referenced through the mapping, index hash table, this bit is set. It is used during page replacement for moving the page around the LRU lists

In short, page_referenced() finds all PTEs’ VMAs that map a page through the page frame. This is also a reverse mapping process.

Linux introduced two page flags, PG_active and PG_referenced, to identify the activity level of pages, thereby deciding how to move pages between two lists (active LRU and inactive LRU).

PG_active is used to indicate whether the page is currently active — if this bit is set, the page is active. PG_referenced is used to indicate whether the page has been accessed recently — each time the page is accessed, this bit is set.

page_referenced(): When the operating system performs page reclamation, each time a page is scanned, this function is called to set the page’s PG_referenced bit. If a page’s PG_referenced bit is set but the page is not accessed again within a certain time, its PG_referenced bit will be cleared.¹⁸.

Memory Observation Metrics
#

View basic memory settings:

Observe memory metrics:

Some Questions
#

Do kswapd and Direct Memory Reclamation Execute Together?
#

Yes. If it’s watermark-triggered memory reclamation, pgscand is often accompanied by pgscank; the reverse is not necessarily true. If both pgscank and pgscand are frequent, consider adjusting memory reclamation watermarks, increasing the delta to prevent it from being quickly breached.

However, there’s another case: when fragmentation rate is high and free memory is still plentiful, blocking memory compaction may be directly triggered with pgscand but no pgscank at all. In this case, adjusting watermarks won’t help. Consider enabling huge page memory and increasing shared buffer hit rate to reduce frequent pagecache allocation that fragments memory.

Impact of Oversized pagetable on Memory Reclamation
#

An oversized pagetable increases the cost and time of reverse mapping. During direct memory reclamation, reverse mapping is needed to find all processes’ virtual address spaces (VMAs), then cancel the VMA page table mappings of all processes. This means: the more processes, the larger the pagetable, and the slower the memory reclamation.

The more PostgreSQL processes, the larger the pagetable; the larger shared buffer, the larger the pagetable.

Enabling huge page memory can reduce pagetable size by 500x (4k=>2M), not only freeing up memory but also improving memory reclamation efficiency.

How Large Should shared buffers Be?
#

sharedbuffers = 1/4 cgmem seems to have become an industry standard, but the actual situation is far more complex. Theoretically, reducing sharedbuffers a bit can increase pagecache a bit, actually slightly increasing total cache size. Increasing sharedbuffers a bit slightly reduces total cache size but improves sharedbuffer hit rate somewhat. Clearly, making sharedbuffers too large is bad, and making it too small is also bad. If sharedbuffers is too small, PG’s own working memory becomes too small, effectively offloading memory management to the OS — OS pagecache reclamation will also affect performance. If sharedbuffers is too large, not only is pagecache squeezed, but PG’s dirty page flushing impact must also be considered, especially for write-heavy scenarios where corresponding bgwriter parameters need adjustment.

From rough stress testing:

Without huge pages, shared buffers = min(1/4 MEM, 20GB)
With huge pages, shared buffers = min(1/4 MEM, 60GB)

Is the Difference Between Processes and Threads Really Not That Big?
#

Any Linux kernel material will say that the difference between processes and threads is not significant. Whether creating a process or a thread, the kernel uses the same function, kernel_clone, to implement it. The only difference lies in the parameters passed. The fork and clone system calls are roughly the same[^ 《深入理解Linux进程和内存》 (Understanding Linux Processes and Memory)]:

Dimension	Process	Thread
childID	Each process has an independent `pid` (process ID)	Each thread has a `tid` (thread ID), but the thread’s `pid` is the same as its process’s `pid`.
Address Space	Each process has an independent address space (`mm_struct`), including memory, stack, etc.	Threads share the address space of their process; all threads’ `mm_struct` points to the same address space.
File System	Each process has its own `fs_struct`, including file descriptors, mount points, etc.	Threads share their process’s `fs_struct`; all threads’ file descriptors and mount points are the same as the process.

Compared to processes, threads are only slightly “lighter”. Overall, the similarities between processes and threads outweigh their differences.

However, when the number of processes increases, the difference becomes significant, especially for multi-process applications like PostgreSQL:

Each process has its own VMA, so more address spaces need to be maintained
Each process has its own pagetable, so pagetables consume more memory
Multiple processes increase TLB flush overhead, while threads do not
Process switching requires more context switch overhead, while threads do not
Inter-process communication (IPC) is less efficient, while threads can directly share memory without IPC communication issues

You could say: processes and threads don’t differ much at creation time, but multi-process management and multi-thread management differ greatly.

Why Does the Standby Have PG-Level Dirty Pages?
#

The standby’s WAL replay mechanism itself generates dirty pages, and the standby also flushes dirty pages. You can view standby dirty pages through pg_buffercache. The standby’s dirty pages are different from the primary’s — standby dirty data is also just regular relations. You can also observe that the standby’s checkpoint/bgwriter/backend dirty flushing is different from the primary’s.

Why Is File Cache Higher on Some Databases and Lower on Others?
#

Generally, databases with high data dispersion have more file cache. Simple slow SQL queries are unlikely to maintain high file cache levels long-term. A slow SQL query accessing lots of data might briefly raise filecache, but after a while, these file pages’ reference count drops, becoming inactive file pages, and memory can reclaim this portion. However, frequent data dispersion — such as when an index’s correlation approaches 0 (like a UUID primary key) — results in decent SQL performance but high reads, potentially generating frequent physical IO and loading too many pages into filecache. Even changes in business patterns can cause a large amount of shared buffer swapping in and out, significantly impacting performance.

PG Processes and Shared Memory Mapping
#

#Without huge pages: /dev/zero (deleted)
cat /proc/102208/smaps |egrep "rw\-s" -A 1
2aefd8901000-2aefd8902000 rw-s 00000000 00:04 1202061313 /SYSV00001000 (deleted)
Size: 4 kB
--
2aefd8918000-2aefd898f000 rw-s 00000000 00:13 4084862058 /dev/shm/PostgreSQL.1008001451
Size: 476 kB
--
2aefe2605000-2b00ad129000 rw-s 00000000 00:04 4084864418 /dev/zero (deleted)

#With huge pages: /anon_hugepage (deleted)
cat /proc/29091/smaps |egrep "rw\-s" -A 1
2aaaaac00000-2ac3a2c00000 rw-s 00000000 00:0e 215471503 /anon_hugepage (deleted)
Size: 104726528 kB
--
2b48dfe93000-2b48dfe94000 rw-s 00000000 00:04 88604727 /SYSV00001000 (deleted)
Size: 4 kB
--
2b48dfeab000-2b48dff22000 rw-s 00000000 00:12 215515747 /dev/shm/PostgreSQL.1123685558
Size: 476 kB

Child process page tables are all copied from the parent process; parent and child processes therefore share the same page frames[^ 《深入理解Linux内核》 (Understanding the Linux Kernel)]. So whether it’s the postmaster or backend processes (any process forked from postmaster), they all map the same shared memory address in their virtual memory — their addresses and Size in smaps are equal.

Why Do All PG Processes Have /dev/zero as the Largest Segment in Virtual Memory?
#

There are two main ways to implement anonymous page mapping with mmap: one is by setting the MAP_ANONYMOUS flag with fd=-1, and the other is by opening the /dev/zero device file and passing the resulting file descriptor to mmap. These two methods are functionally equivalent.

PG shared buffers use the /dev/zero device mapping to implement anonymous shared pages, which is why you typically see PG processes having a large proportion of their virtual memory address space as /dev/zero.

References
#

[Understanding the Linux Kernel]: Understanding the Linux Kernel: Memory Addressing, Memory Management, Address Space Management, Page Frame Reclamation

[Understanding Linux Processes and Memory]: Understanding Linux Processes and Memory: CPU Hardware Principles, Process and Thread Comparison

[Running Linux Kernel: Beginner’s Guide 2nd Edition]: Running Linux Kernel: Beginner’s Guide 2nd Edition: System Calls, Memory Management

My 2024 Year-End Summary

Sat, 11 Jan 2025 00:00:00 +0000

As a DBA
#

2023 was a year of comprehensive PostgreSQL learning for me, and 2024 has been a year of comprehensive PostgreSQL operations. There’s actually a lot of material I really want to dive into but haven’t had the time. This year was mainly case analysis — I could only supplement my foundational knowledge here and there.

Mid-year there was a discussion about “will DBAs be eliminated in the cloud era.” This discussion left a deep impression on me. I thought about many things afterward — why do others seem to have so few things to deal with while I, as a DBA, have so much? I even went into cloud computing groups to debate about it, and I actually gained something from it. Different perspectives lead to unexpected conclusions. The conclusion of the debate may boil down to just one thing: DBAs are providing 1510 emotional value to their leaders.

Right or wrong, you can see reflections on the DBA profession in many of my articles this year. Continuing down the traditional DBA path is certainly a dead end. Today’s DBAs lean more toward business data layer operations, or moving up to architecture design. Positions for expert DBAs focused purely on databases are actually very few.

READING
#

Let me reiterate why I’m so devoted to reading (I said this in 2023 too…):

The value brought by reading is immeasurable in the short term
Reading brings a pleasant sense of intellectual enrichment
Learning is a belief. Yuval Harari has a view: believing in science is actually also a form of faith. I choose to believe in this faith, at least in 2024 and the foreseeable future.

My book list roughly falls into three categories: PostgreSQL, broader technical scope, and extracurricular. Some are in Chinese, some in English. Some are physical books, some electronic.

This year, let me continue with a reading list ranking. Horizontal comparison across different categories is a bit of a stretch, so let’s compare within categories. Once again, note: these book lists are for books I aimed to “finish cover to cover.” Books used as references don’t count here.

2024 PostgreSQL Book List (ranked by preference):

“PostgreSQL Database Kernel Analysis” — clear thinking and framework, though the version is a bit old
“Quickly Mastering PostgreSQL Version New Features” — this should be my favorite PostgreSQL book this year, because it has zero fluff throughout, a pleasure to read
“The Internals of PostgreSQL” — I originally wanted to put this first, but since there’s a free online version at interdb, I wouldn’t even recommend buying this book. It’s ranked here because interdb is so excellent — its substitute is enshrined here as a deity
“The Way of PostgreSQL: From Apprentice to Expert, 2nd Edition” — very detailed but also very long. I recommend skimming through quickly to find the key points without lingering too long
“PostgreSQL Technical Internals: Transaction Processing Deep Dive” — transactions are the foundation of PostgreSQL, and also the foundation of my source code journey
“PostgreSQL in Action” — the practical examples are well worth referencing
“PostgreSQL 16 Administration Cookbook” — not recommended. The table of contents framework looks good, but the content is hollow. Don’t waste time on this book.

2024 Broader Technical Scope Book List (ranked by preference):

“DDIA-v2: Designing Data-Intensive Applications (2nd Edition)” — so good I don’t know where to begin. So excellent that I specially wrote reading notes (my only book notes article this year). I wish I had found it sooner.
“A Brief History of Databases” — reading history truly brings insight. The story of databases begins here. Some technical things become clearer in hindsight.
“ITIL 4 and DevOps Service Management Certification Guide (2nd Edition)” — a classic in IT service management. It elevated my understanding of the operations role — how did these things so closely tied to my work come about? Which parts don’t match reality, and why weren’t they applied? You can grasp many things from it.
“Cloud Native Kubernetes” — hardcore, another track entirely
“Docker Deep Dive” — decent for understanding containers and container history. The container knowledge itself isn’t actually that much.
“Brother Bird’s Linux Private Kitchen” — sorry, I genuinely hadn’t read this classic. Came to catch up. The writing approach is well worth learning from. The drawback is that much of it isn’t useful for my role.
“Machine Learning” — ranked here not because the book is bad, but because it’s very hard to understand. I gave up about a quarter of the way through. This book showed me the upper limits of my intelligence, and I’m sad about it.
“Building a Vector Database from Scratch” — if you want to read source code, go to GitHub
“Deep Understanding of Go Language” — understood nothing at all

2024 Extracurricular Book List (ranked by preference):

“Cancer Ward” — I finished this early in the first half of the year. While reading it, I felt: barring surprises, this book would rank first this year. Nobel Prize in Literature, well-deserved.
“Intimate Relationships” — understanding relationships with lovers, friends, and bosses. Academic paper style, solid, I like it.
“Does God Play Dice? A History of Quantum Physics” — setting aside everything else, the writing style provides immense emotional value, making me want to keep reading. I finished it in just a few days.
“The Worlds I See” — AI pioneer Fei-Fei Li’s autobiography. The story of a girl who grew up in Chengdu venturing into the melting pot of America, eventually leading Google AI, while also narrating the history of AI development.
“21 Lessons for the 21st Century” — the final installment of Yuval Harari’s trilogy. I loved the first two books, but this one felt just okay. At least it brought closure.
“The Old Man and the Sea” — hard to evaluate. I like its temperament, but not its content.
“The Wandering Earth” — this is a collection of Liu Cixin’s short stories. One day at the library, I bought it because of the first short story. After buying it, I found the other short stories to be very boring and childish. I felt cheated.
“Journey to the West” — hot take: they can’t even explain Tang Sanzang’s background properly. A mess, completely confused. I gave up after a little bit. (My evaluation of “Romance of the Three Kingdoms” last year was very high.)

Blog and WeChat Official Account
#

2024 Published Articles:

PostgreSQL technical: 21
Other technical: 2
Book notes: 1
Useless articles: 1

I only wrote 25 articles this year, a noticeable decrease from last year.

WeChat Official Account followers: 600. Though not many, I believe every single one has good taste 😸

Writing technical articles is actually quite tiring — it takes far more time than one would imagine. However, you genuinely learn things during the writing process, and the sense of accomplishment from completing a piece is real. Since I feel responsible for my articles, I won’t write recklessly about things I don’t understand. As for errors arising from misunderstandings, that’s actually normal. No one can guarantee that their future self won’t criticize their current self — just write correctly for the current state.

In terms of writing content this year, I gave up writing reading notes for extracurricular books. I wrote quite a few last year, but writing reading notes takes a lot of time with very low value. Low emotional value tasks naturally get abandoned. In fact, my writing content varies each year. Currently, PostgreSQL database technical articles are the only constant — other types aren’t as stable. This is normal. The blog was originally meant for database writing. If there’s no application scenario for other domains, I won’t touch them again after the brief exploratory period.

One more complaint: domestic blogging platforms only care about article quantity, which is completely at odds with my writing style. Each of my articles is tens of thousands of hand-typed characters. I’m a quality-over-quantity blogger. So I can’t be bothered anymore — I’m planning to abandon CSDN in 2025 and just post on GitHub and my WeChat Official Account.

I’ve been writing on CSDN since 2017. When I first started blogging, there weren’t many good blog hosting platforms. Looking at CSDN now: community interaction is zero, and the vast majority of articles on it are terrible. Even I don’t want to find CSDN articles myself. It’s like a first love of 7-8 years — sometimes you just have to break up.

2024 Publication Channels:

CSDN Blog: https://liuzhilong.blog.csdn.net
Modb.pro: liuzhilong62
GitHub: https://github.com/liuzhilong62/blogs
WeChat Official Account: 破斯特贵斯库儿

Expected 2025 Channels:

GitHub: https://github.com/liuzhilong62/blogs
WeChat Official Account: 破斯特贵斯库儿
Other platforms: we’ll see

Final Thoughts
#

I seem to talk about work-learning balance every year… Due to a dramatic increase in workload this year, there was even a period where I couldn’t study at all. Balance has been shattered. Not having time to study is unacceptable to me, so I later adjusted my daily schedule (thanks to “Atomic Habits” — I absolutely love this book), and finally managed to squeeze out some study time. Actually, as long as no one’s around, learning efficiency is high.

I’ve collected some quotes I resonated with this year:

Don’t let others become dependencies in your task chain –heisenberg.liu
Plans that require execution are generally simple plans –heisenberg.liu
Things not implemented equal things not done –somebody
Solve problems yourself instead of waiting for others to reply –somebody
Important things should be done immediately — waiting even a moment means they won’t get done –somebody
Don’t do repetitive low-value tasks. Think more about the context behind this requirement –heisenberg.liu
Don’t pan for gold in shit. Find ways to get quality information sources –somebody
SREs need the ability to configure optimal default parameters and the ability to modify these parameters in bulk –“Enterprise Cloud Computing”
The more miscellaneous tasks you do, the more miscellaneous tasks come your way –heisenberg.liu
SREs spend 50% of time on operations and 50% on development –“Enterprise Cloud Computing”
Premature optimization is the root of all evil. Premature code abstraction is also the root of all evil –somebody
The speed at which the human brain receives knowledge is limited –somebody
If someone won’t let you read, leave that person or leave that environment –heisenberg.liu
Teams that build knowledge bases are slackers –somebody
The value of a standard is determined by the customer –“ITIL 4”
Heroism: working long hours and troubleshooting alone. Long working hours also lead to burnout with the work itself. Those who want to be heroes are only interested in their own achievements and turn a deaf ear to team collaboration –“ITIL 4”
Not all problems need root cause analysis. It depends on the frequency of occurrence and the scope of the failure –“ITIL 4”

Looking back at the plans I set for myself in 2023: only 2 items total, and I completed neither. KPI achievement rate: 0% 😄

Combining agile operations, agile project management, and OKR thinking: setting a full-year plan for myself at the beginning of the year is simply unreasonable. Looking back at last year and the year before, some of my plans emerged mid-way and won priority battles over other tasks. And some tasks simply couldn’t be completed — this should be a normal state. So, I won’t set too many flags for myself.

2025 Plan:

Continue some things
Think about how to produce output
Master another track
PostgreSQL… haven’t figured out what more to do
Find a way to resume fitness

PostgreSQL Ops Experience 2024

Wed, 08 Jan 2025 00:00:00 +0000

This article focuses on common PostgreSQL operations issues — rare edge cases that surface once every two or three years are out of scope.

It’s primarily a technical ops summary, aiming for clarity and quick applicability. Deep dives at the source-code level are deliberately avoided.

SQL Performance & Execution Plans
#

Sudden Execution Plan Changes
#

PostgreSQL does not support optimizer hints natively, and the community has made it clear it never will. The PG community’s stance is roughly: “Our optimizer is perfect. If the current plan isn’t good enough, it’s because the developer doesn’t understand optimization.”

Regardless of what the PG community thinks, sudden execution plan regressions happen all the time in production, and we don’t have the rich, native plan-binding mechanisms that Oracle provides. This is a real challenge for production operations. For example: one morning, a sensitive query suddenly changes its plan, runtime jumps from 0.1s to 1s, and due to some concurrency the database CPU gets hammered — the business notices immediately. Without plan-binding tools, our only two rapid recovery options are: 1) collect statistics, or 2) scale up CPU.

A question about rapid recovery: does collecting statistics always help? A good DBA can identify where the optimizer went wrong, but can’t instantly conjure up a complete correct plan — especially for complex queries. Collecting statistics essentially hands the optimization problem back to the optimizer, trusting it to get it right. While this sounds a bit shaky, in PostgreSQL it actually works most of the time. (For scenarios where collecting stats is known to be useless, see the “ORDER BY LIMIT Problem” section.)

Why do execution plans suddenly change and regress?

Plans are cost-based, costs rely on statistics, and statistics are always lagging
Sufficiently complex SQL has a huge number of possible execution paths, and the optimizer picks the lowest-cost one
PG exposes many optimizer parameters to tune for local hardware (e.g., seq_page_cost, effective_cache_size). These can nudge the optimizer’s preferences but are very low-level. While there’s theoretical tuning headroom, changing them has system-wide effects. After go-live, adjusting these is extremely high-risk. The very existence of these parameters hints that no plan can be 100% perfect, because the optimizer’s reasoning depends on its environment

Even mighty Oracle, with its arsenal of plan-stabilization features, can’t guarantee 100% problem-free SQL — because SQL, data, statistics, bind variables, etc. are all dynamic.

For PG users, we’re not there yet, but we can work on making plans more stable:

Don’t join too many tables. More tables mean more possible plans — to the point where PG GEQO stops enumerating all plans, reducing the chance of finding the optimal one
Don’t write overly complex SQL. Keep in mind SQL may come from ORM frameworks rather than hand-written queries. Framework-generated SQL is often optimized for a goal with little regard for brevity or readability, making it very hard to tune
Don’t create indexes indiscriminately — have a clear goal. Random indexes confuse the optimizer
Tune per-table statistics collection thresholds via autovacuum_analyze_scale_factor (see “Delayed Statistics Collection”)
Use pg_hint_plan to give the optimizer hints

pg_hint_plan
#

pg_hint_plan is a third-party extension that uses hints to guide the optimizer toward the correct plan.

What pg_hint_plan supports:

Specifying scan methods (e.g., index scan), join methods (NL/HASH/MERGE), join order, memoize, estimated row counts, parallelism, and GUC parameters
Binding hints to SQL via hint_plan.hints without modifying the application SQL text

pg_hint_plan limitations:

Usage restrictions with subqueries, foreign tables, CTEs, views, PL/pgSQL, etc.
compute_query_id treats hints as comments and ignores them
Unknown bugs

While this extension is actively maintained, I haven’t found large-scale production deployment cases yet. We’ve also encountered issues in limited production use where hints don’t take effect — possibly related to JDBC plan caching — but it’s hard to draw firm conclusions.

In short: pg_hint_plan is a good tool, but large-scale production deployment is still TBD. I recommend waiting and watching. You can trial it, but don’t become dependent on it.

Delayed Statistics Collection
#

Statistics are the foundation of SQL optimization. PG statistics aren’t particularly complex, but many people still don’t fully understand them.

The three key views for PG statistics: pg_class, pg_stat_all_tables, pg_stats

-- pg_class: pages and tuples
select relname,relpages,reltuples::bigint from pg_class where relname='lzlpg'\gx
-[ RECORD 1 ]------
relname | lzlpg
relpages | 187501
reltuples | 6000032

-- pg_stat_all_tables: live tuples, dead tuples, last analyze time
 select relname,n_live_tup,n_dead_tup,last_analyze,last_autoanalyze from pg_stat_all_tables where relname='lzlpg'\gx
-[ RECORD 1 ]----+------------------------------
relname | lzlpg
n_live_tup | 6000032
n_dead_tup | 0
last_analyze | 2025-01-04 15:54:44.553057+08
last_autoanalyze | [null]

-- pg_stats: per-column statistics — understand every field
 select * from pg_stats where tablename='lzlpg' and attname='a'\gx
-[ RECORD 1 ]----------+-------
schemaname | public
tablename | lzlpg
attname | a
inherited | f
null_frac | 0
avg_width | 70
n_distinct | -1
most_common_vals | [null]
most_common_freqs | [null]
histogram_bounds | [null]
correlation | [null]
most_common_elems | [null]
most_common_elem_freqs | [null]
elem_count_histogram | [null]

Stale statistics are very likely to cause execution plan changes and SQL performance issues. Check last_autovacuum and last_autoanalyze in pg_stat_all_tables to determine if collection is lagging.

Why tune it? Because the default autovacuum_analyze_scale_factor is 0.1, meaning statistics are only collected when data changes exceed 10%. For a 1-billion-row table, that’s 100 million rows — possibly far too infrequent.

Evaluate whether to tune per-table autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor based on: whether it’s a core business table, number of joins, query complexity, access frequency, month-boundary issues, data skew, etc. The goal: increase collection frequency to reduce plan-regression risk without wasting resources on excessive vacuuming.

What value should you set? An example:

For a monthly table (or monthly partition) with queries hitting the current day’s data: with autovacuum_analyze_scale_factor = 0.1, the table gets analyzed almost daily for the first ~10 days, but may skip analysis around day 12. At that point statistics can cross a boundary and plans may degrade. To ensure analysis continues through days 10–31 of the month, set autovacuum_analyze_scale_factor below 0.03. I recommend autovacuum_analyze_scale_factor = 0.02.

Parameter tuning reference (consider your table’s data model!):

Parameter	Default	Recommended
`autovacuum_vacuum_scale_factor`	`0.2`	`0.04`
`autovacuum_analyze_scale_factor`	`0.1`	`0.02`

The Optimizer May Choose a Non-Primary-Key Index
#

Intuitively, a primary key should have the best selectivity, but the optimizer may still choose something else.

-- Reproduction commands
create table t1(a char(1000) primary key,b char(1000));
insert into t1 select md5(g::text),md5(g::text) from generate_series(1,10000) g;
create index idxa on t1(a);
create index idxb on t1(b);
analyze t1;

explain (analyze,buffers) select * from t1 where a='qwer' and b='qwer';
explain (analyze,buffers) select * from t1 where a='qwer' and b||''='qwer';

-- Columns a and b have identical selectivity, but the optimizer picks the regular index, not the PK
explain (analyze,buffers) select * from t1 where a='qwer' and b='qwer';
 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Index Scan using idxb on t1 (cost=0.41..5.43 rows=1 width=2008) (actual time=0.045..0.046 rows=0 loops=1)
 Index Cond: (b = 'qwer'::bpchar)
 Filter: (a = 'qwer'::bpchar)
 Buffers: shared hit=3

-- Force the PK path — cost is only marginally higher
explain (analyze,buffers) select * from t1 where a='qwer' and b||''='qwer';
 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Index Scan using idxa on t1 (cost=0.41..5.44 rows=1 width=2008) (actual time=0.079..0.079 rows=0 loops=1)
 Index Cond: (a = 'qwer'::bpchar)`
 Filter: (((b)::text || ''::text) = 'qwer'::text)
 Buffers: shared read=3

Even though columns a and b have the same type and selectivity, the optimizer picks the regular index over the PK. The PK path costs 0.01 more.

Why does this matter?

With the current data distribution, picking the regular index is harmless. But once data changes, the two index plans can diverge dramatically:

alter table t1 set (autovacuum_enabled ='off');
insert into t1 select md5(g::text),'repeat' from generate_series(20001,30000) g;

-- b='repeat' has terrible selectivity, but the b index is still chosen
explain (analyze,buffers) select * from t1 where a='qwer' and b='repeat';
 QUERY PLAN
--------------------------------------------------------------------------------------------------------------
 Index Scan using idxb on t1 (cost=0.41..5.43 rows=1 width=2008) (actual time=15.823..15.824 rows=0 loops=1)
 Index Cond: (b = 'repeat'::bpchar)
 Filter: (a = 'qwer'::bpchar)
 Rows Removed by Filter: 10000
 Buffers: shared hit=2511

-- Compare with the PK plan
 explain (analyze,buffers) select * from t1 where a='qwer' and b||''='repeat';
 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Index Scan using idxa on t1 (cost=0.41..5.44 rows=1 width=2008) (actual time=0.041..0.041 rows=0 loops=1)
 Index Cond: (a = 'qwer'::bpchar)
 Filter: (((b)::text || ''::text) = 'repeat'::text)
 Buffers: shared hit=3

Even with poor real selectivity, the optimizer sticks with the regular index — but efficiency is far worse (shared hit=2511 vs. shared hit=3). For latency-sensitive queries or larger data volumes, this becomes a real production problem.

Solutions:

Manually collect statistics; increase collection frequency
Use pg_hint_plan
Rewrite the SQL to prevent it from using the regular index

The ORDER BY LIMIT Problem
#

ORDER BY with LIMIT is a well-known issue with plenty of write-ups and case studies online (see my post ORDER BY LIMIT 10 Is Slower Than ORDER BY LIMIT 100).

The root cause: the optimizer currently can’t estimate where data sits in the table relative to the index order. If matching rows happen to be near the end of the table, the scan reads far more data than expected before returning the LIMIT rows. Note this isn’t limited to ORDER BY + LIMIT — any operation involving sorted output + LIMIT can hit it: GROUP BY + LIMIT, DISTINCT + LIMIT, merge joins, etc.

Solutions:

Rewrite the SQL: add an expression to prevent using the sort-column index (including PK), e.g., order by ''||col1 limit xxx
Create a composite index: a composite index on (sort_column + index_column) may be chosen by the optimizer and is generally more efficient than an index on the sort column alone. This approach doesn’t require changing the SQL

Table Bloat
#

Something Blocking Dead Tuple Cleanup
#

Putting aside autovacuum configuration issues and edge cases, the common blockers are:

Long-running transactions. Note: a long transaction on a different table also blocks dead-tuple reclamation. Read-only queries cause this too.
Replication slots. Lagging or defunct replication slots cause this.

Both are relatively easy to solve: 1) terminate the long-transaction session, 2) drop the replication slot, or have the consumer analyze why consumption is so slow.

High-Concurrency UPDATE Causing Table Bloat
#

Unlike something blocking vacuum, this is about dead tuples being generated faster than vacuum can clean them up. Typically, such tables show high pg_stat_all_tables.n_tup_upd. If table bloat requires repack, assess whether write volume is high enough to make repeated manual repack a losing game. In that case, tune the table/index fillfactor.

For the underlying principles, see this post From Painfully Slow Unique Index Scans to Index Bloat. I’ll summarize the conclusions here:

fillfactor basics:

fillfactor acts as a high-water mark for tables or indexes. During INSERT, once a page reaches its fillfactor line, new rows go to the next page. The purpose is to reserve space for UPDATEs so they don’t constantly seek new pages.

While both tables and indexes have fillfactor with the same goal (accommodating UPDATEs), the details differ significantly:

Tables: If a page still has free space, an UPDATE can stay within the same page — no new page needed, no need to find another page with space. More importantly, thanks to PG’s HOT (Heap-Only Tuple) feature, in-page updates don’t touch indexes, naturally slowing index bloat
Indexes: Different rows or out-of-page updates of the same row generate new index entries. Reserving space in index pages via fillfactor greatly reduces index page splits

Of course, fillfactor settings are tightly coupled with the workload. If data is append-only like logs with zero updates, fillfactor=100 for both tables and indexes is perfectly fine. But most business tables see updates, so fillfactor shouldn’t be 100. With frequent UPDATEs, it should be even lower.

Yet PG’s defaults are:

Table default: fillfactor=100
Index default: fillfactor=90

Recommended settings:

alter table lzlpg set (fillfactor=60);
alter index lzlpg_pkey set (fillfactor=70);
-- These commands only affect new pages; existing pages need repack

-- Repack:
1. Check for long transactions; resolve them first
2. nohup pg_repack -d lzldb --table lzlpg -p 6666 -no-kill-backend > pgrepack_lzlpg_log.log 2>&1 &

Long Transaction Problems
#

Long transactions don’t have a huge amount of theory behind them — monitor and handle promptly — but they absolutely deserve their own section.

Long transactions cause many problems:

Unreleased locks → application blocking
WAL not recycled → disk alerts
Dead tuples not cleaned → SQL performance degradation
Various other bizarre performance issues linked to long transactions
…

Long transactions in PostgreSQL are far more damaging than in Oracle or MySQL. They must be strictly managed.

Subtransaction Problems
#

“Subtransactions are basically cursed. Rip em out.”

Subtransactions cause many problems and are a frequent pain point in the industry.

Industry experience reports:

Waiting for Postgres 17: Configurable SLRU cache sizes for increased performance

Subtransactions-overflow-and-the-performance-cliff

Why we spent the last month eliminating PostgreSQL subtransactions

Where subtransactions come from:

PL/pgSQL functions containing a block with an exception clause
savepoints
JDBC + autosave=always (default autosave=never)
ODBC

Note: OGG uses an ODBC driver, and ODBC cannot disable subtransactions.

GaussDB’s ODBC can disable subtransactions via ForExtensionConnector.

So we can advise applications to keep subtransactions under 64, but we can’t easily advise against using OGG, since migrating off Oracle often depends on OGG-based data sync tools.

Subtransaction problem scenarios and symptoms:

1(+) long transaction + subtransaction overflow + high concurrency → severe performance drop
Subtransaction overflow (64+) → noticeable performance dip
Subtransaction overflow (64+) + multixact → severe performance drop
1(+) long transaction + 1(+) subtransaction → severe query performance drop on read replicas

Improvements in PG17:

SLRU manages transaction relationships for clog, multixact, subtrans, etc. in shared memory. Relevant source definitions:

/* Number of SLRU buffers to use for subtrans */
#define NUM_SUBTRANS_BUFFERS	32 // 32 SLRU pages in shared memory

/*
 * Each backend advertises up to PGPROC_MAX_CACHED_SUBXIDS TransactionIds
 * for non-aborted subtransactions of its current top transaction. These
 * have to be treated as running XIDs by other backends.
 *
 * We also keep track of whether the cache overflowed (ie, the transaction has
 * generated at least one subtransaction that didn't fit in the cache).
 * If none of the caches have overflowed, we can assume that an XID that's not
 * listed anywhere in the PGPROC array is not a running transaction. Else we
 * have to look at pg_subtrans.
 */
#define PGPROC_MAX_CACHED_SUBXIDS 64	// Overflow at 64+, per backend

PG17 SLRU improvements: New GUC parameter to configure SLRU slot count; split the existing single centralized SLRU lock into multiple bank locks.

Improvement effect:

(https://www.pgevents.ca/events/pgconfdev2024/sessions/session/53/slides/27/SLRU%20Performance%20Issues.pdf)

Subtransaction handling strategies:

Dev standards: Don’t use savepoints; consider ON CONFLICT for write conflicts
Dev standards: Don’t use exception blocks
Dev standards: Ensure JDBC does not have autosave=always enabled
Monitoring: Targeted monitoring of pg_stat_slru
Monitoring: Targeted monitoring of SAVEPOINT and EXCEPTION
CDC standards: Use ODBC (and OGG or other ODBC-based tools) with care; split transactions, cap subtransactions per large transaction at 50K
Upgrade: Move to PG17

Concurrency & Performance
#

Snapshot and Concurrency Parameter Tuning
#

Parameter	Type	Default	Recommended	Requires Restart
`old_snapshot_threshold`	cpu	-1 (community)	-1	Yes
`max_parallel_workers_per_gather`	cpu	2	0	No

old_snapshot_threshold easily causes performance problems when enabled — there’s plenty of material online. Even though it requires a restart, I strongly recommend keeping it disabled.

max_parallel_workers_per_gather auto-enables parallelism for large queries, but parallelism of 2 won’t give a proportional 2x speedup. This parameter is best used in specific scenarios, like explicitly setting parallel workers for batch jobs. Since no restart is needed, it’s a quick change.

Will disabling old_snapshot_threshold cause problems?

No. This parameter exists to limit long transactions — which do damage performance in PG — but the parameter itself causes performance issues, defeating the purpose.

Long transactions can be handled via several mechanisms:

Long transaction monitoring. This is the most important, and monitoring is fairly mature.
Set statement_timeout (default 0)
Set transaction_timeout (default 0, available in PG17+)
Set lock_timeout (default 0; recommended at session level for DDL)
Set idle_in_transaction_session_timeout (default 0; we set it to 2h)
Set idle_session_timeout (default 0; not relevant here)

High-Concurrency Commits Causing LWLOCK:WALWrite
#

Case Study: Intermittent Slow INSERT … VALUES

Key takeaways:

There’s only one IO:WALWrite, but there can be dozens of LWLOCK:WALWrite waiters
You can’t directly see the LWLOCK blocking chain, but from the source code we know LWLOCK:WALWrite is waiting on IO:WALWrite
In high-concurrency small-transaction scenarios, increasing WAL buffer size theoretically doesn’t help much

What problems does this cause?

Concurrent writes block, write latency increases, active sessions may spike
High-concurrency small transactions can’t saturate disk IO

Solutions:

Distribute concurrent writes across time
Batch commits at the application level
Analyze and try to reduce FPI (see FPI section)
Group commit (TBD)

WAL & Latency
#

FPI and Checkpoint Parameters
#

PG generates WAL FPI (Full Page Images) the first time a page is touched after a checkpoint. So more frequent checkpoints → higher probability of FPI.

Checkpoint frequency is controlled by two parameters:

checkpoint_timeout
max_wal_size

Principle:

(Egor Rogov, PostgreSQL 14 Internals)

max_wal_size defaults to 1GB, which is too small for high-load databases. Generally, you should increase this parameter to reduce FPI.

checkpoint_timeout defaults to 5 minutes, which seems reasonable.

FPI and Random Writes
#

Even with longer checkpoint intervals, FPI problems may persist. Check whether the workload involves UUID-based random writes. You may need to switch to sequences or another UUID scheme.

Finding the specific index:

Check if FPI is severe

--stats=record is handy

pg_waldump -z --stats=record 00000001000001860000001B

Sort which relations have the most FPWs

pg_waldump 00000001000001860000001B|grep FPW|awk -F ':' '{print $7}'|awk '{print $2}'|sort -n|uniq -c |sort -r|head -10

Logical Replication & Replication Slots
#

Logical replication has many issues and is a key optimization area for the community — nearly every major version brings significant improvements.

Logical Replication and Replication Slots Basics

Spill Problem
#

Analysis of PG Startup Logic and Spill-Induced Slow Startup

Spill key takeaways:

Spill occurs when logical decoding can’t fit transaction data in memory, so it writes to disk. Spill files contain transaction information
Each walsender has independent decoding, so each logical replication subscriber has its own spill
Large transactions produce large spill files, typically few in number
Subtransaction spill produces one spill file per subtransaction

Versions:

PG12 and earlier: hard-coded 4096 changes
PG13 added logical_decoding_work_mem to adjust memory and reduce spill probability
PG14+ supports streaming replication
Streaming also requires certain conditions to trigger, so even with streaming, spilling can still occur
PG17 added debug_logical_replication_streaming to force streaming

WALSender Blocking Shutdown
#

PG Shutdown Logic and WALSender Blocking Shutdown Analysis

In reality, any process that doesn’t exit can block shutdown. The question is which ones are most likely to cause trouble. From the shutdown code flow, archiver and walsender are frequent blockers because during shutdown they attempt a final archive or log transmission.

If shutdown is stuck on walsender, try kill (not kill -9) — the checkpoint hasn’t finished yet, and a forced shutdown leaves an inconsistent state. Even for forced shutdown, prefer pg_ctl stop -D $PGDATA -m i over raw kill -9
If shutdown is stuck on archiver, kill -9 is fine — the checkpoint is already complete and the database is in a consistent state

Partitioned Tables
#

Partitioned Table Basics

PG’s partitioned tables have unique characteristics that developers generally don’t fully understand without study, leading to many pitfalls.

Index Mismatch Between Parent and Child Partitions
#

Due to non-standard partition creation, many indexes are created directly on child tables (which should not be done), and the “create index on all children + attach” workflow is skipped. The result: the parent table has no index or no effective index. Since the parent has no data, this doesn’t directly impact queries — but when new partitions are created, they only inherit the parent’s indexes, so new child tables end up missing indexes.

Fixing parent-table missing indexes is fairly straightforward: see The Correct Way to Create Partition Indexes

-- Create an invalid index ONLY on the parent. Fast, but blocks subsequent DML — watch for long transactions
CREATE INDEX IDX_DATECREATED ON ONLY lzlpartition1(date_created);
-- Create the index CONCURRENTLY on each child partition. Slow, but doesn't block DML — watch for long DML transactions that could cause the operation to fail
create index concurrently idx_datecreated_202302 on lzlpartition1_202302(date_created);
-- Attach all indexes. Fast, no business blocking
 ALTER INDEX idx_datecreated ATTACH PARTITION idx_datecreated_202302;

Fixing a missing primary key on the parent is harder: see Adding Primary Keys and Unique Indexes to Partitioned Tables

Adding a primary key on the parent acquires AccessExclusiveLock, blocking everything. Creating an index on a partitioned table is slow, and the PK then causes further blocking. There’s currently no low-impact way to add a PK on a partitioned table. Workarounds: “attach a unique index + NOT NULL constraint”, schedule extended downtime for the partition table while the index builds, or use a third-party sync tool to populate a new table that already has the PK.

Abusing the DEFAULT Partition
#

Default Partition Overgrowth Causing Prolonged Blocking During CREATE TABLE ... PARTITION OF

The root cause is simple: when adding a new partition, the DDL must validate that data in the DEFAULT partition doesn’t conflict with the new partition’s range. This scans a large amount of data in the DEFAULT partition, and the new partition creation never completes. Blocking then cascades — business queries and writes stall.

DEFAULT partition abuse is a widespread problem! The community PG doesn’t provide interval partitioning. If a developer forgets to create a partition, data silently lands in DEFAULT with no error or alert. Day after day, the DEFAULT partition grows enormous — and then the next schema change causes an outage.

You can’t leave an oversized DEFAULT partition as-is forever. Even though ATTACH can avoid the blocking problem, you still need to defuse this bomb eventually.

DEFAULT partition data handling — Plan 1:

DETACH the default partition, create proper partitions, then re-insert DEFAULT data into the partitioned table
If needed, after detach and creating proper partitions, create an empty DEFAULT partition to maintain business continuity
Note: DETACH (unlike ATTACH) requires an AccessExclusiveLock on the parent. PG14 supports DETACH CONCURRENTLY, but not for DEFAULT partitions

DEFAULT partition data handling — Plan 2:

DETACH the default partition, create proper partitions, then ATTACH the detached DEFAULT table as a regular child partition — careful with range boundaries
If needed, after detach and creating proper partitions, create an empty DEFAULT partition to maintain business continuity
Note: DETACH (unlike ATTACH) requires an AccessExclusiveLock on the parent. PG14 supports DETACH CONCURRENTLY, but not for DEFAULT partitions

DEFAULT partition data handling — Plan 3:

Create a new table, sync all data via DTS
Rename tables

Plan 3 looks the crudest, but it’s the one I personally recommend most. If you have 5 instances to fix, a surgical approach is fine. If you have 200 instances, the labor cost makes DTS the practical winner.

Missing SELECT Privileges on Partitions Causing Abnormal Plans
#

If a user lacks SELECT privilege on a child partition, their queries can’t access that partition’s statistics, leading to bad execution plans. Partitions created via CREATE TABLE ... PARTITION OF normally don’t carry SELECT grants — but data is accessible through the parent — so this is a widespread issue.

Solutions:

Have the cloud platform handle it automatically
Enforce dev standards requiring SELECT grants on child partitions

High-Concurrency Full Partition Scans and LWLock:lockmanager
#

This is another very common problem!

I recommend reading the AWS documentation, which explains it clearly: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/wait-event.lw-lock-manager.html

Symptoms:

Spiking active sessions
Severe LWLock:lockmanager wait events
Database performance cliff

Trigger conditions:

Query scans multiple partitions
That query has high concurrency

Key takeaways:

The fastpath lock mechanism is designed for quick access to “weak locks”, improving database concurrency
fastpath works for lock levels ≤ 3 — i.e., SELECT, SELECT FOR xxx, and DML (lock modes below ShareUpdateExclusiveLock — levels 1, 2, 3 can use fastpath). It’s meant to benefit normal operations
FP_LOCK_SLOTS_PER_BACKEND: a local process holds at most 16 fastpath locks; beyond that, it must acquire locks in shared memory, producing LWLock:lockmanager
Not just tables — every accessed index also acquires a lock
This problem isn’t tightly coupled to partition count — even a modest number of partitions can trigger LWLock:lockmanager and degrade performance

Let’s calculate: with a partitioned table having 1 primary key and 2 regular indexes, how many partitions exhaust the fastpath?

16 / (3 indexes + 1 table) - 1 parent = 3 child partitions

Yes — a full scan across just 3 partitions can already trigger LWLock:lockmanager waits.

For a regular table, 16 indexes would similarly exhaust fastpath.

Solutions:

For not-too-large tables, merge partitions into a regular table
Add partition key filter conditions to queries
Reduce indexes (not very practical, since partition count alone can exceed 16)

The hard part:

In Oracle-to-PG migrations, Oracle supports global indexes, so primary keys and unique indexes don’t need to include the partition key. In PG, they must include the partition key.

PK example:

idxlzl(primarykey) --oracle
idxlzl(primarykey,partitionkey) --pg

A common query pattern:

select col from tlzl where primarykey=12345;

Should you push the application to add a partition filter here? It’s a tough sell. The resistance is: “I already passed the primary key — what more do you want? If I knew everything, why would I query the database?”

In this case, the only recommendation is to convert the partitioned table to a regular table. I haven’t found a better solution.

Memory
#

Excessive Objects Leading to Oversized relcache
#

Key takeaways:

relcache stores relation metadata: OID, pg_class info, partitions, subtransactions, row-level security policies, statistics, index metadata, access methods, etc.
Each session has its own (rel)cache for system catalog data (metadata, etc.)
Normally this cache is small. When the catalog is huge and a session accesses all of it, the cache can become very large
Cache management is simple: no eviction mechanism, no limit (though there are invalidation messages)
Closing the session releases the cache

Solutions:

Reduce the number of objects — especially check whether partition child tables are excessive
Set aggressive connection-pool disconnection parameters so business connections recycle more frequently

Memory Fragmentation
#

Recommended commands:

cat /proc/meminfo|grep whatyouneed
cat /proc/buddyinfo

## cgroup memory
/opt/cgtools/cginfo -t perf -s mem

# Pay attention to pgscand/s (direct memory reclaim) — values in the tens of thousands indicate a problem
sar -B -s "08:00:00" -e "09:00:00"
# min_free_kbytes setting:
cat /proc/sys/vm/min_free_kbytes
# Total physical memory usage of all processes:
grep Pss /proc/[1-9]*/smaps | awk '{total+=$2}; END {printf "%d kB\n", total }'
# PSS memory for a specific process:
cat /proc/90875/smaps |grep Pss |awk '{sum+=$2 };END {print sum/1024}'
# RSS memory for a specific process:
cat /proc/68729/smaps |grep Rss |awk '{sum+=$2 };END {print sum/1024}'
# Private memory for a specific process:
cat /proc/90875/smaps|sed '/zero/,/VmFlags/d' |grep Private |awk '{sum+=$2 };END {print sum/1024}'

min_free_kbytes:

(https://vivani.net/2022/06/14/linux-kernel-tuning-page-allocation-failure/)

When free memory is low, the kswapd daemon is woken to free pages:

pages_low: when free pages fall below pages_low, buddy allocator wakes kswapd and the kernel begins swapping pages to disk
pages_min: when free pages reach pages_min, reclamation pressure is high — the zone urgently needs free pages. The allocator performs synchronous kswapd work, sometimes called direct reclaim
pages_high: once kswapd is awake and freeing pages, the kernel considers the zone “balanced” only when free pages reach pages_high. At pages_high, kswapd goes back to sleep. Free pages above pages_high means the zone is in an ideal state

vm.min_free_kbytes (the pages_min watermark) is an extremely important OS parameter. Too low a value prevents effective memory reclamation, potentially causing system crashes and service interruptions. Too high a value increases reclaim activity, causing allocation delays that can immediately trigger OOM.

Optimization results:

After increasing min_free_kbytes + deploying off-peak drop-cache jobs, problems have decreased significantly.

Why increase min_free_kbytes?

This is used to force the Linux VM to keep a minimum number of kilobytes free. The VM uses this number to compute a watermark[WMARK_MIN] value for each lowmem zone in the system. Each lowmem zone gets a number of reserved free pages based proportionally on its size.

Source: kernel.org docs

The point of raising min_free_kbytes isn’t to raise the min watermark and trigger direct reclaim more often — it’s because the low watermark couldn’t be tuned before Linux 7. The only way to raise low proportionally was to raise min, making asynchronous reclaim trigger earlier and giving direct reclaim a buffer window.

Red Hat 8 added two memory parameters to improve reclaim: watermark_scale_factor can raise watermarks without touching min_free_kbytes.

Recommend enabling huge pages:

Huge pages perform better when PG requests contiguous memory
Huge pages also help reduce page cache size
shared_buffers can use huge pages; requires Huge_pages=on and OS-level huge pages enabled
Instances with huge pages enabled in production show better performance and fewer problems
AWS huge pages standard: enabled by default for all instance classes except certain test tiers, and cannot be disabled

Huge_pages parameter is turned on by default for all DB instance classes other than t3.medium, db.t3.large, db.t4g.medium, db.t4g.large instance classes. You can’t change the huge_pages parameter value or turn off this feature in the supported instance classes of Aurora PostgreSQL.

cgroup and Host Memory Mismatch
#

When cgroup memory hits its limit, kswapd prioritizes reclaiming pages within the cgroup. With cloud VM instance types and cgroup configurations, the host may have free memory above watermarks while the cgroup is under pressure. The host-level pages_low doesn’t trigger asynchronous reclaim for either host or cgroup memory. Eventually, direct reclaim fires to satisfy the cgroup’s DB memory demand.

The root cause: cgroups lack independent free-page memory management.

The only fix: increase the cgroup memory limit, overcommitting the host more aggressively so the host reaches pages_low sooner.

shared_buffer and pagecache
#

PG uses a double-buffer mechanism — no direct IO yet.

Double buffer: DB shared_buffers (one layer of shared memory) + OS pagecache (another layer). In real deployments, pagecache is typically far larger than shared_buffers. And pagecache counts against cgroup mem but isn’t reflected in cgroup memory monitoring…

Bottom line: leave plenty of memory for pagecache. Don’t make shared_buffers excessively large (20GB seems sufficient for most cases). Only increase it if you clearly observe buffer-mapping-related wait events.

work_mem Cannot Cap Hash Join / Hash Aggregate Memory
#

hash_mem_multiplier limits memory for hash-based operations (hash join, hash agg, etc.), capping at hash_mem_multiplier * work_mem. The default is 2.

Before PG13, work_mem was tunable, but there was no way to limit how many hash operations a single query could use. PG13 added this multiplier. In other words, pre-13, it was very hard to cap hash-table memory.

In a PG12- production environment, I found a single session consuming 300GB of memory — the culprit was the lack of hash-table limits combined with a plan that incorrectly used hash tables.

Other Issues
#

Exclusive Backup and Startup Issues
#

Normally, when the database stops and restarts, the startup position comes from pg_controldata’s LSN. But if there’s a backup_label file in PGDATA, the startup LSN is read from backup_label.

What problems does this cause?

Disk snapshots taken directly on the data directory may include the label file. If the database is large and the backup took a long time, restart can be very slow
Bigger problem: after a production shutdown from certain causes, restart takes forever. The root cause is the startup LSN coming from the backup rather than controldata

Version changes:

PG13:

pg_start_backup() pg_stop_backup()

Supports exclusive and non-exclusive modes; exclusive is the default. Exclusive mode creates backup_label in the data directory at start and cleans it at stop. Non-exclusive mode doesn’t create the label at start; it returns the label info at stop.

PG15:

pg_backup_start() pg_backup_stop()

Function names changed, and exclusive backup mode was removed. No backup_label is written at backup start; instead it’s written to the backup area at backup stop.

pg_stat_activity Unqueryable
#

Symptom:

pg_stat_activity hangs and can’t be queried.

pstack at the time:

#0 pgstat_read_current_status () at pgstat.c:3642
#1 0x0000000000727181 in pgstat_read_current_status () at pgstat.c:2788
#2 pgstat_fetch_stat_numbackends () at pgstat.c:2789
#3 0x000000000083f2ee in pg_stat_get_activity (fcinfo=0x25c2d98) at pgstatfuncs.c:575
#4 0x000000000065058f in ExecMakeTableFunctionResult (setexpr=0x25b1d28, econtext=0x25b1c48, argContext=<optimized out>, expectedDesc=0x2545218, randomAccess=false) at execSRF.c:234
#5 0x00000000006609dc in FunctionNext (node=node@entry=0x25b1b38) at nodeFunctionscan.c:94
#6 0x000000000065110c in ExecScanFetch (recheckMtd=0x660700 <FunctionRecheck>, accessMtd=0x660720 <FunctionNext>, node=0x25b1b38) at execScan.c:133

Analysis:

The code location is clear — stuck in an infinite loop after st_changecount becomes odd.

Triggers: OOM (reproducible), abnormal backend exit (possible), terminate (maybe). None of these guarantee the issue, though.

Community thread didn’t reach a conclusion. Currently the trigger probability appears low.

Solution: restart the database.

Connection and Connection Pooling Issues
#

IO Error Messages
#

IO errors typically mean the application is using a connection that’s already been closed. This happens often, and diagnosing it is difficult because the entire chain involves many components and broad domain knowledge. Here’s a brief summary.

Known active-disconnection scenarios:

hikari maxLifetime

Symptom: session lifetime matches the parameter. Possible cause: the application holds an explicit transaction with an uncommitted SELECT, the pool closes the session, and the app gets io error; could not rollback or similar.

pg.datasouce.maxLifetime

druid timeout

Symptom: connection drops after SQL execution exceeds 20s.

spring.datasource.dynamic.druid.socketTimeout=20000
spring.datasource.dynamic.druid.connectTimeout=20000
Change to:
spring.datasource.socketTimeout=3600000
spring.datasource.connectTimeout=3600000

Application Horizontal Scaling vs. Database Connection Limits
#

Horizontal application scaling meets PG connection bottlenecks:

HikariCP is now Spring Boot’s default connection pool. With the proliferation of Spring Boot and microservices, HikariCP usage is widespread. Every pod scaled out increases database connection count. The maximumPoolSize stays the same per pod, but more nodes mean more total connections. From existing node count, added node count, and current total connections, you can proportionally calculate how many idle connections will be added.

Applications can scale horizontally without state, but databases cannot. PG’s connection limit is max_connections. Unchecked application scaling can saturate idle connections. Tuning max_connections is painful because it requires a database restart.

PG connection upper limit:

Also, even with unlimited horizontal scaling, max_connections should adjust with instance class — but there’s a real ceiling. In any database, idle connections degrade performance as they increase.

Refer to AWS’s approach: max_connections is tied to instance class, with a maximum of 5000, LEAST({DBInstanceClassMemory/9531392}, 5000). This reduces manual connection ops and provides a reasonable ceiling.

PG Shutdown Logic and Walsender Blocking Shutdown Analysis

Sat, 04 Jan 2025 00:00:00 +0000

Walsender Blocking Shutdown Symptoms
#

Production shutdown log output:

2024-12-06 17:00:02.036 CST,,,447560,,65693cde.6d448,1320,,2023-12-01 09:54:38 CST,,0,LOG,00000,"received fast shutdown request",,,,,,,,,"","postmaster"
2024-12-06 17:00:02.295 CST,,,447560,,65693cde.6d448,1322,,2023-12-01 09:54:38 CST,,0,LOG,00000,"background worker ""logical replication launcher"" (PID 448996) exited with exit code 1",,,,,,,,,"","postmaster"
2024-12-06 17:00:10.627 CST,,,448990,,65693ce0.6d9de,213833,,2023-12-01 09:54:40 CST,,0,LOG,00000,"checkpoint complete: wrote 426844 buffers (5.1%); 0 WAL file(s) added, 0 removed, 5 recycled; write=91.427 s, sync=0.055 s, total=91.508 s; sync files=761, longest=0.028 s, average=0.001 s; distance=2197531 kB, estimate=2680783 kB",,,,,,,,,"","checkpointer"
2024-12-06 17:00:10.628 CST,,,448990,,65693ce0.6d9de,213834,,2023-12-01 09:54:40 CST,,0,LOG,00000,"shutting down",,,,,,,,,"","checkpointer"
...
--checkpointer finished checkpoint and is in shutting down state, pm has not exited

--160s later pm receives immediate shutdown, triggered by health check script
2024-12-06 17:02:43.348 CST,,,447560,,65693cde.6d448,1323,,2023-12-01 09:54:38 CST,,0,LOG,00000,"received immediate shutdown request",,,,,,,,,"","postmaster"
2024-12-06 17:02:43.370 CST,"logicaluser","lzldb",283840,"10.33.77.159:39865",6751a2dc.454c0,7,"idle",2024-12-05 20:55:56 CST,89/847309655,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"
2024-12-06 17:02:43.370 CST,"logicaluser","lzldb",157641,"10.33.77.159:39407",67408354.267c9,7,"idle",2024-11-22 21:12:52 CST,9/3193590104,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"
2024-12-06 17:02:43.370 CST,"logicaluser","lzldb",157916,"10.33.77.159:57038",67408356.268dc,7,"idle",2024-11-22 21:12:54 CST,115/3293293502,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"
2024-12-06 17:02:43.370 CST,"repuser","",164392,"30.151.40.19:41641",66b25869.28228,3,"streaming 42D3B/1732C5F0",2024-08-07 01:07:53 CST,296/0,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"standby_6666","walsender"
2024-12-06 17:02:43.371 CST,,,447560,,65693cde.6d448,1324,,2023-12-01 09:54:38 CST,,0,LOG,00000,"archiver process (PID 448994) exited with exit code 2",,,,,,,,,"","postmaster"
2024-12-06 17:02:43.371 CST,"logicaluser","lzldb",57755,"10.33.77.159:38918",67125534.e19b,7,"idle",2024-10-18 20:31:48 CST,243/902018192,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"
2024-12-06 17:02:43.372 CST,"logicaluser","lzldb",157915,"10.33.77.159:43433",67408356.268db,7,"idle",2024-11-22 21:12:54 CST,60/3248014863,0,WARNING,57P02,"terminating connection because of crash of another server process","The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.","In a moment you should be able to reconnect to the database and repeat your command.",,,,,,,"Debezium Streaming","walsender"


--pm finished shutting down
2024-12-06 17:02:57.534 CST,,,447560,,65693cde.6d448,1325,,2023-12-01 09:54:38 CST,,0,LOG,00000,"database system is shut down",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.536 CST,,,211844,,6752bdf3.33b84,1,,2024-12-06 17:03:47 CST,,0,LOG,00000,"ending log output to stderr",,"Future log output will go to log destination ""csvlog"".",,,,,,,"","postmaster"

17:00:02 postmaster receives fast shutdown

17:00:10 checkpoint completed, checkpointer stopped

17:02:43 postmaster receives immediate shutdown

17:02:43 1 physical and 5 logical replication walsenders stopped

17:02:57 postmaster stopped

17:03:49 postmaster receives startup task

From the above, it’s clear that walsender was blocking the shutdown.

Shutdown and Signals
#

Before diving into source code, we need to understand signals and signal registration in PG.

Common Signals in PG
#

OS signals:

$ kill -l
 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
...

Common signals used in PG:

-1 or -SIGHUP: Hangup signal. In PG, typically tells the process to reload configuration.
-2 or -SIGINT: Interrupt signal (usually Ctrl+C). In PG, usually corresponds to cancel command.
-3 or -SIGQUIT: In PG, usually means forced exit (die).
-9 or -SIGKILL: Unconditional termination signal.
-15 or -SIGTERM: Termination signal, the signal used by pg_terminate_backend. In PG, usually means graceful exit.
-10 or -SIGUSR1: Custom signal.
-12 or -SIGUSR2: Custom signal.
-17 or SIGCHLD: Signal used by the pm process. When a child process exits, pm receives this signal to trigger child process reaping.

The specific meaning of signals registered by each type of PG process can be found by reading the respective process source code.

Shutdown Defined by pg_ctl
#

There are several ways to shut down a PG database. At the bottom level, they all boil down to sending a signal to the postmaster process.

signal	pg_ctl	Meaning
`SIGTERM`	Smart Shutdown	Disallow new connections, but allow existing sessions to finish their work normally. Only shuts down after all sessions terminate.
`SIGINT`	Fast Shutdown	Server disallows new connections and sends SIGTERM to all existing child processes, aborting current transactions and exiting quickly. Waits for almost all child processes (some are not needed) to exit, then shuts down.
`SIGQUIT`	Immediate Shutdown	Sends SIGQUIT to all child processes and waits for them to terminate. If any child process has not terminated within 5 seconds, they are sent SIGKILL.

Note: pg_ctl has no parameter for sending SIGKILL (kill -9), but you can send SIGKILL directly to pm — though it’s definitely not recommended. When sending SIGKILL to pm, pm won’t do any cleanup of child processes, shared memory, or semaphores. Since SIGQUIT to pm has fallback logic for SIGKILL-ing child processes, SIGQUIT to pm basically guarantees pm will stop.

In the source code, there are only 3 shutdown states, corresponding to shutdown modes:

/* Startup/shutdown state */
#define			NoShutdown		0
#define			SmartShutdown	1
#define			FastShutdown	2
#define			ImmediateShutdown	3

These states appear frequently in shutdown routine source code, generally checked via the Shutdown variable:

Shutdown >= FastShutdown

pm Signals
#

When pm receives the corresponding signal, it handles it accordingly:

void
PostmasterMain(int argc, char *argv[])
{...
	pqsignal_pm(SIGHUP, SIGHUP_handler);	/* reread config file and have
											 * children do same */
	pqsignal_pm(SIGINT, pmdie); /* send SIGTERM and shut down */
	pqsignal_pm(SIGQUIT, pmdie);	/* send SIGQUIT and die */
	pqsignal_pm(SIGTERM, pmdie);	/* wait for children and shut down */
	pqsignal_pm(SIGALRM, SIG_IGN);	/* ignored */
	pqsignal_pm(SIGPIPE, SIG_IGN);	/* ignored */
	pqsignal_pm(SIGUSR1, sigusr1_handler);	/* message from child process */
	pqsignal_pm(SIGUSR2, dummy_handler);	/* unused, reserve for children */
	pqsignal_pm(SIGCHLD, reaper);	/* handle child termination */

pmdie: The three shutdown signals call the pmdie function. pmdie is the key shutdown function, analyzed in detail below.
reaper: During shutdown, handles child process exit cleanup. When a child process exits, it sends SIGCHLD to pm, which enters reaper to clean up the child. Each child process cleanup has its own logic — for instance, normal exit of the checkpointer process checks whether archiver and walsender have completed their respective tasks.
sigusr1, sigusr2: sigusr1_handler is the standard routine for SIGUSR1. Each child process handles SIGUSR1 differently. SIGUSR2 is entirely custom per child process; some child processes don’t even register this signal.

Walsender Signals
#

When a child process is forked, it first registers signals.

WalSndSignals registers signals for the walsender process:

/* Set up signal handlers */
void
WalSndSignals(void)
{
	/* Set up signal handlers */
	pqsignal(SIGHUP, SignalHandlerForConfigReload);
	pqsignal(SIGINT, StatementCancelHandler);	/* query cancel */
	pqsignal(SIGTERM, die);		/* request shutdown */
	pqsignal(SIGQUIT, quickdie);	/* hard crash time */
	InitializeTimeouts();		/* establishes SIGALRM handler */
	pqsignal(SIGPIPE, SIG_IGN);
	pqsignal(SIGUSR1, procsignal_sigusr1_handler);
	pqsignal(SIGUSR2, WalSndLastCycleHandler);	/* request a last cycle and
												 * shutdown */
}

Note SIGUSR1 and SIGUSR2.

Checkpointer Signals
#

CheckpointerMain registers checkpointer signals:

void
CheckpointerMain(void)
{
...
 //checkpointer blocks SIGTERM, the actual stop signal is SIGUSR2
	pqsignal(SIGHUP, SignalHandlerForConfigReload);
	pqsignal(SIGINT, ReqCheckpointHandler); /* request checkpoint */
	pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */
	pqsignal(SIGQUIT, SignalHandlerForCrashExit);
	pqsignal(SIGALRM, SIG_IGN);
	pqsignal(SIGPIPE, SIG_IGN);
	pqsignal(SIGUSR1, procsignal_sigusr1_handler);
	pqsignal(SIGUSR2, SignalHandlerForShutdownRequest);

Note SIGUSR1 and SIGUSR2, and also note that checkpointer does not register SIGTERM.

Shutdown Source Code Analysis
#

pm Signal Handling and State Machine
#

The pmdie function handles different postmaster signals, including SIGCHLD sent by child processes to pm and shutdown signals sent by pg_ctl. The main logic of pm signal handling is converting the signal into a pmState state machine state transition, then entering PostmasterStateMachine for processing.

pmdie:

/*
 * pmdie -- signal handler for processing various postmaster signals.
 */
static void
pmdie(SIGNAL_ARGS)
{
	int			save_errno = errno;

...

	switch (postgres_signal_arg)
	{
		case SIGTERM://Smart Shutdown
			...
			if (pmState == PM_RUN)
				connsAllowed = ALLOW_SUPERUSER_CONNS;
			...
 //smart shutdown does not process pmstate, hands directly to state machine
			//at this point normal pmState = PM_RUN
			PostmasterStateMachine();
			break;

		case SIGINT://Fast Shutdown
			...
			else if (pmState == PM_RUN ||
					 pmState == PM_HOT_STANDBY)
			{
				/* Report that we're about to zap live client sessions */
				ereport(LOG,
						(errmsg("aborting any active transactions")));
				pmState = PM_STOP_BACKENDS;
			}

			//Fast Shutdown transitions pmstate to PM_STOP_BACKENDS
 //then hands to state machine
			PostmasterStateMachine();
			break;

		case SIGQUIT://Immediate Shutdown
			...

			TerminateChildren(SIGQUIT);//abort all children with SIGQUIT, wait for them to exit
			pmState = PM_WAIT_BACKENDS;

			/* set stopwatch for them to die */
			AbortStartTime = time(NULL);

 //Immediate Shutdown transitions pmstate to PM_WAIT_BACKENDS
 //process children before entering state machine
 //first interrupt children with SIGQUIT, wait for them to exit
 //then use SIGKILL on remaining children
 //finally non-consistent exit
			PostmasterStateMachine();
			break;
	}
...
}

Before entering the state machine handler, let’s look at the postmaster states:

typedef enum
{
	PM_INIT,					/* postmaster starting */
	PM_STARTUP,					/* waiting for startup subprocess */
	PM_RECOVERY,				/* in archive recovery mode */
	PM_HOT_STANDBY,				/* in hot standby mode */
	PM_RUN,						/* normal "database is alive" state */
	PM_STOP_BACKENDS,			/* need to stop remaining backends */
	PM_WAIT_BACKENDS,			/* waiting for live backends to exit */
	PM_SHUTDOWN,				/* waiting for checkpointer to do shutdown
								 * ckpt */
	PM_SHUTDOWN_2,				/* waiting for archiver and walsenders to
								 * finish */
	PM_WAIT_DEAD_END,			/* waiting for dead_end children to exit */
	PM_NO_CHILDREN				/* all important children have exited */
} PMState;

Since shutdown normally happens from the running state, we only need to focus on states at PM_RUN and below.

PostmasterStateMachine execution has a sequential logic:

/*
 * Advance the postmaster's state machine and take actions as appropriate
 *
 * This is common code for pmdie(), reaper() and sigusr1_handler(), which
 * receive the signals that might mean we need to change state.
 */
static void
PostmasterStateMachine(void)
{
	//smart shutdown, pmState should be PM_RUN at this point
	if (pmState == PM_RUN || pmState == PM_HOT_STANDBY)
	{
	...

		if (connsAllowed == ALLOW_NO_CONNS)
		{
 //After all normal backends exit, transition pmState to PM_STOP_BACKENDS
			if (CountChildren(BACKEND_TYPE_NORMAL) == 0)
				pmState = PM_STOP_BACKENDS;
		}
	}

 //PM_STOP_BACKENDS stops some core child processes, some will continue running
 //autovacuum, bgwriter, walwriter, startup, walreceiver will stop
 //walsender, checkpointer, archiver, stats, and syslogger will keep running
 //smart shutdown later phase enters this logic, fast shutdown enters directly
	if (pmState == PM_STOP_BACKENDS)
	{
		...		
		//Note this line about walsender!
		/* Signal all backend children except walsenders */
		SignalSomeChildren(SIGTERM,
						 BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND);
		/* and the autovac launcher too */
		if (AutoVacPID != 0)
			signal_child(AutoVacPID, SIGTERM);
		/* and the bgwriter too */
		if (BgWriterPID != 0)
			signal_child(BgWriterPID, SIGTERM);
		/* and the walwriter too */
		if (WalWriterPID != 0)
			signal_child(WalWriterPID, SIGTERM);
		/* If we're in recovery, also stop startup and walreceiver procs */
		if (StartupPID != 0)
			signal_child(StartupPID, SIGTERM);
		if (WalReceiverPID != 0)
			signal_child(WalReceiverPID, SIGTERM);
		/* checkpointer, archiver, stats, and syslogger may continue for now */

 //Transition pmState from PM_STOP_BACKENDS to PM_WAIT_BACKEND
 //PM_WAIT_BACKEND means waiting for backends to exit
		pmState = PM_WAIT_BACKENDS;
	}

	/*
	 * If we are in a state-machine state that implies waiting for backends to
	 * exit, see if they're all gone, and change state if so.
	 */
 //
 //smart shutdown, fast shutdown later phase enters this logic
 //immediate shutdown when entering state machine, directly enters this logic
	if (pmState == PM_WAIT_BACKENDS)
	{
 //During crash recovery and immediate shutdown, checkpointer needs proper exit
 //archiver, stats, and syslogger don't need handling since they don't touch shared memory
 //Walsenders also don't need handling; they exit after checkpoint record is written, just like archiver
		if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
			StartupPID == 0 &&
			WalReceiverPID == 0 &&
			BgWriterPID == 0 &&
			(CheckpointerPID == 0 ||
			 (!FatalError && Shutdown < ImmediateShutdown)) &&
			WalWriterPID == 0 &&
			AutoVacPID == 0)
		{
			if (Shutdown >= ImmediateShutdown || FatalError)
			{
				//ImmediateShutdown waits for dead end processes to finish
				pmState = PM_WAIT_DEAD_END;

				/*
				 * We already SIGQUIT'd the archiver and stats processes, if
				 * any, when we started immediate shutdown or entered
				 * FatalError state.
				 */
			}
			else
			{
 //smart, fast shutdown goes here
 //regular child processes have all exited, now notify checkpointer to do shutdown checkpoint
				Assert(Shutdown > NoShutdown);
				//If checkpointer process doesn't exist, start one
				if (CheckpointerPID == 0)
					CheckpointerPID = StartCheckpointer();
				/* And tell it to shut down */
				if (CheckpointerPID != 0)
				{
 //Send SIGUSR2 to Checkpointer
 //pmState = PM_SHUTDOWN
					signal_child(CheckpointerPID, SIGUSR2);
					pmState = PM_SHUTDOWN;
				}
				else
				{
 //Failing to start Checkpointer is a serious problem
					FatalError = true;
					pmState = PM_WAIT_DEAD_END;

					/* Kill the walsenders, archiver and stats collector too */ 
 //Comment says kill walsender, but it actually doesn't; at least not via SIGQUIT
					SignalChildren(SIGQUIT);
					if (PgArchPID != 0)
						signal_child(PgArchPID, SIGQUIT);
					if (PgStatPID != 0)
						signal_child(PgStatPID, SIGQUIT);
				}
			}
		}
	}
	
 //The pmdie function and state machine function won't create PM_SHUTDOWN_2 state, but reaper will
 //When reaper handles checkpointer exit, it sets pmState = PM_SHUTDOWN_2; at the end of reaper, it enters the state machine function, which is here
	if (pmState == PM_SHUTDOWN_2)
	{
		/*
		 * PM_SHUTDOWN_2 state ends when there's no other children than
		 * dead_end children left. There shouldn't be any regular backends
		 * left by now anyway; what we're really waiting for is walsenders and
		 * archiver.
		 */
 //PM_SHUTDOWN_2 essentially waits for walsender and archiver
 //only changes pmState
		if (PgArchPID == 0 && CountChildren(BACKEND_TYPE_ALL) == 0)
		{
			pmState = PM_WAIT_DEAD_END;
		}
	}

	if (pmState == PM_WAIT_DEAD_END)
	{
 //PM_WAIT_DEAD_END means BackendList is completely empty
		if (dlist_is_empty(&BackendList) &&
			PgArchPID == 0 && PgStatPID == 0)
		{
			/* These other guys should be dead already */
			Assert(StartupPID == 0);
			Assert(WalReceiverPID == 0);
			Assert(BgWriterPID == 0);
			Assert(CheckpointerPID == 0);
			Assert(WalWriterPID == 0);
			Assert(AutoVacPID == 0);
			/* syslogger is not considered here */
			pmState = PM_NO_CHILDREN;
		}
	}

 //PM_NO_CHILDREN is the last shutdown state, meaning normal shutdown can proceed
	if (Shutdown > NoShutdown && pmState == PM_NO_CHILDREN)
	{
		if (FatalError)
		{
			ereport(LOG, (errmsg("abnormal database system shutdown")));
 //Abnormal pm exit
			ExitPostmaster(1);
		}
		...

		 	//Normal pm exit
			ExitPostmaster(0);
		}
	}
	...
}

reaper is the process reaping function. When a child process exits, it sends SIGCHLD to pm, and pm cleans up the process via the reaper function. Each process type — backend, startup, checkpointer, etc. — has its own cleanup flow.

Here we only look at checkpointer cleanup. Also, reaper has no cleanup logic for walsender:

		if (pid == CheckpointerPID)
		{
			CheckpointerPID = 0;
 //Checkpointer exited normally, and pmState is PM_SHUTDOWN: waiting for checkpoint completion
			if (EXIT_STATUS_0(exitstatus) && pmState == PM_SHUTDOWN)
			{
				/*
				 * OK, we saw normal exit of the checkpointer after it's been
				 * told to shut down. We expect that it wrote a shutdown
				 * checkpoint. (If for some reason it didn't, recovery will
				 * occur on next postmaster start.)
				 *
				 * At this point we should have no normal backend children
				 * left (else we'd not be in PM_SHUTDOWN state) but we might
				 * have dead_end children to wait for.
				 *
				 * If we have an archiver subprocess, tell it to do a last
				 * archive cycle and quit. Likewise, if we have walsender
				 * processes, tell them to send any remaining WAL and quit.
				 */
				Assert(Shutdown > NoShutdown);

				//Wake archiver for the last time
				if (PgArchPID != 0)
					signal_child(PgArchPID, SIGUSR2); //pgarch SIGUSR2=pgarch_waken_stop

 //Wake walsender for the last time
				SignalChildren(SIGUSR2);//walsender SIGUSR2=WalSndLastCycleHandler
 
 //Here PM_SHUTDOWN_2 is set
 //At this point Checkpointer has exited normally; we should wait for pgarch and walsender to finish their last task
 //This is PM_SHUTDOWN_2 state
 pmState = PM_SHUTDOWN_2;

				...
			}
			else
			{
 //checkpointer abnormal exit is considered a crash
				HandleChildCrash(pid, exitstatus,
								 _("checkpointer process"));
			}

			continue;
		}
...
	//At the end reaper still enters the state machine function
	PostmasterStateMachine();

...
}

Checkpointer and Walsender Process Exit
#

Checkpointer main loop handling requests and shutdown:

void
CheckpointerMain(void)
{
	/*
	 * Loop forever
	 */
	for (;;)
	{
		bool		do_checkpoint = false;
		int			flags = 0;
		pg_time_t	now;
		int			elapsed_secs;
		int			cur_timeout;

		/* Clear any already-pending wakeups */
		ResetLatch(MyLatch);

		/*
		 * Process any requests or signals received recently.
		 */
 //Process recent sync requests and signals
		AbsorbSyncRequests();
		HandleCheckpointerInterrupts();

Checkpointer shutdown function:

/*
 * Process any new interrupts.
 */
static void
HandleCheckpointerInterrupts(void)
{
 ...
	if (ShutdownRequestPending)
	{
		/*
		 * From here on, elog(ERROR) should end with exit(1), not send control
		 * back to the sigsetjmp block above
		 */
		ExitOnAnyError = true;
		
		ShutdownXLOG(0, 0);//This writes the shutdown checkpoint
		
		proc_exit(0);//Normal exit code 0
	}
}

Checkpointer exit needs to wait for ShutdownXLOG to complete.

ShutdownXLOG:

/*
 * This must be called ONCE during postmaster or standalone-backend shutdown
 */
void
ShutdownXLOG(int code, Datum arg)
{
...

	//Here's the checkpointer "shutting down" log, usually always seen
	ereport(IsPostmasterEnvironment ? LOG : NOTICE,
			(errmsg("shutting down")));

	/*
	 * Signal walsenders to move to stopping state.
	 */
 //Initialize walsender stopping
	WalSndInitStopping();

 //Wait for all walsenders to be in stopping state
	WalSndWaitStopping();

	if (RecoveryInProgress())
		CreateRestartPoint(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_IMMEDIATE);
	else
	{
		/*
		 * If archiving is enabled, rotate the last XLOG file so that all the
		 * remaining records are archived (postmaster wakes up the archiver
		 * process one more time at the end of shutdown). The checkpoint
		 * record will go to the next XLOG file and won't be archived (yet).
		 */
		if (XLogArchivingActive() && XLogArchiveCommandSet())
			RequestXLogSwitch(false);
		//This is the shutdown checkpoint creation function
		CreateCheckPoint(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_IMMEDIATE);
	}
	ShutdownCLOG();
	ShutdownCommitTs();
	ShutdownSUBTRANS();
	ShutdownMultiXact();
}

Checkpointer notifies all walsenders to begin stopping:

/*
 * Signal all walsenders to move to stopping state.
 *
 * This will trigger walsenders to move to a state where no further WAL can be
 * generated. See this file's header for details.
 */
void
WalSndInitStopping(void)
{
	int			i;

	for (i = 0; i < max_wal_senders; i++)
	{
		WalSnd	 *walsnd = &WalSndCtl->walsnds[i];
		pid_t		pid;

		SpinLockAcquire(&walsnd->mutex);
		pid = walsnd->pid;
		SpinLockRelease(&walsnd->mutex);

		if (pid == 0)
			continue;

		SendProcSignal(pid, PROCSIG_WALSND_INIT_STOPPING, InvalidBackendId);
	}
}

Walsender receives the signal via the SendProcSignal function, with signal SIGUSR1:

/*
 * SendProcSignal
 *		Send a signal to a Postgres process
 *
 * Providing backendId is optional, but it will speed up the operation.
 *
 * On success (a signal was sent), zero is returned.
 * On error, -1 is returned, and errno is set (typically to ESRCH or EPERM).
 *
 * Not to be confused with ProcSendSignal
 */
int
SendProcSignal(pid_t pid, ProcSignalReason reason, BackendId backendId)
{
	else
	{
		/*
		 * BackendId not provided, so search the array using pid. We search
		 * the array back to front so as to reduce search overhead. Passing
		 * InvalidBackendId means that the target is most likely an auxiliary
		 * process, which will have a slot near the end of the array.
		 */
		int			i;

		for (i = NumProcSignalSlots - 1; i >= 0; i--)
		{
			slot = &ProcSignal->psh_slot[i];

			if (slot->pss_pid == pid)
			{
				/* the above note about race conditions applies here too */

				/* Atomically set the proper flag */
				slot->pss_signalFlags[reason] = true;
				/* Send signal */
				return kill(pid, SIGUSR1);
			}
		}
	}

	errno = ESRCH;
	return -1;
}

Walsender’s SIGUSR1 registration:

	pqsignal(SIGUSR1, procsignal_sigusr1_handler);
	pqsignal(SIGUSR2, WalSndLastCycleHandler);	/* request a last cycle and
												 * shutdown */

sigusr1 classifies handling by signal reason:

/*
 * procsignal_sigusr1_handler - handle SIGUSR1 signal.
 */
void
procsignal_sigusr1_handler(SIGNAL_ARGS)
{
...
	if (CheckProcSignal(PROCSIG_WALSND_INIT_STOPPING))
		HandleWalSndInitStopping();
...
}

The handler for PROCSIG_WALSND_INIT_STOPPING is HandleWalSndInitStopping:

/*
 * Handle PROCSIG_WALSND_INIT_STOPPING signal.
 */
void
HandleWalSndInitStopping(void)
{
	Assert(am_walsender);

	/*
	 * If replication has not yet started, die like with SIGTERM. If
	 * replication is active, only set a flag and wake up the main loop. It
	 * will send any outstanding WAL, wait for it to be replicated to the
	 * standby, and then exit gracefully.
	 */
	if (!replication_active)
		kill(MyProcPid, SIGTERM);
	else
		got_STOPPING = true;//If walsender is active, initstopping just sets a flag for the main loop to handle
}

The “main loop” mentioned in the comment is somewhat ambiguous. Walsender has a main loop ServerLoop, but in reality only the loop in WalSndWaitForWal has checks for got_STOPPING.

The WalSndWaitForWal function is the main loop for walsender waiting for new WAL records. Since WAL records are initially generated in memory, walwriter flushes them based on certain conditions, not all the time. WalSndWaitForWal compares the currently sent LSN with the flushed LSN to determine whether new WAL needs to be sent. In other words, unflushed WAL is not transmitted; only flushed WAL is passed downstream.

WalSndWaitForWal code segment about stopping:

/*
 * Wait till WAL < loc is flushed to disk so it can be safely sent to client.
 *
 * Returns end LSN of flushed WAL. Normally this will be >= loc, but
 * if we detect a shutdown request (either from postmaster or client)
 * we will return early, so caller must always check.
 */
static XLogRecPtr
WalSndWaitForWal(XLogRecPtr loc)
{
 ...
	for (;;)
	{
 ...
 //After receiving got_STOPPING, do one flush of WAL
 //This is necessary! Because walwriter may have already shut down at this point, WAL may not be flushed yet
		if (got_STOPPING)
			XLogBackgroundFlush();

		/* Update our idea of the currently flushed position. */
		if (!RecoveryInProgress())
			RecentFlushPtr = GetFlushRecPtr();
		else
			RecentFlushPtr = GetXLogReplayRecPtr(NULL);

 //Break out of the for loop
 //After getting new RecentFlushPtr, still need to send
		if (got_STOPPING)
			break;
 ...
	}
	/* reactivate latch so WalSndLoop knows to continue */
	SetLatch(MyLatch);
	return RecentFlushPtr;
}

Back to walsender main loop: WalSndLoop(XLogSendLogical):

/* Main loop of walsender process that streams the WAL over Copy messages. */
static void
WalSndLoop(WalSndSendDataCallback send_data)
{
...
	for (;;)
	{
		/* Clear any already-pending wakeups */
		ResetLatch(MyLatch);
		...

		//Process replies from downstream
		ProcessRepliesIfAny();

		/*
		 * If we have received CopyDone from the client, sent CopyDone
		 * ourselves, and the output buffer is empty, it's time to exit
		 * streaming.
		 */
 //Exit loop when streaming is done
		if (streamingDoneReceiving && streamingDoneSending &&
			!pq_is_send_pending())
			break;

 //If output buffer has pending data, send it
		if (!pq_is_send_pending())
			send_data();
		else
			WalSndCaughtUp = false;

		/* Try to flush pending output to the client */
		if (pq_flush_if_writable() != 0)
			WalSndShutdown();//Downstream not writable, downstream closed, normal walsender shutdown, exit code 0

		/* If nothing remains to be sent right now ... */
		if (WalSndCaughtUp && !pq_is_send_pending())
		{
			/*
			 * If we're in catchup state, move to streaming. This is an
			 * important state change for users to know about, since before
			 * this point data loss might occur if the primary dies and we
			 * need to failover to the standby. The state change is also
			 * important for synchronous replication, since commits that
			 * started to wait at that point might wait for some time.
			 */
 //Data transmission is done, but commit info still needs to be sent
			if (MyWalSnd->state == WALSNDSTATE_CATCHUP)
			{
				ereport(DEBUG1,
						(errmsg("\"%s\" has now caught up with upstream server",
								application_name)));
				WalSndSetState(WALSNDSTATE_STREAMING);
			}

 //Received SIGUSR2, meaning shutdown checkpoint is done.
 //Send the shutdown checkpoint record, wait for completion, then exit
			if (got_SIGUSR2)
				WalSndDone(send_data);//exit code 0
		}
		
		...
	}
}

Let’s return to checkpointer’s ShutdownXLOG logic. The above only analyzed WalSndInitStopping(). After this signal is sent to walsender, WalSndWaitStopping executes to wait for walsender.

As long as any walsender hasn’t exited, this is an infinite loop that won’t return:

/*
 * Wait that all the WAL senders have quit or reached the stopping state. This
 * is used by the checkpointer to control when the shutdown checkpoint can
 * safely be performed.
 */
void
WalSndWaitStopping(void)
{
	for (;;)
	{
		int			i;
		bool		all_stopped = true;

		for (i = 0; i < max_wal_senders; i++)
		{
			WalSnd	 *walsnd = &WalSndCtl->walsnds[i];

			SpinLockAcquire(&walsnd->mutex);

			if (walsnd->pid == 0)
			{
				SpinLockRelease(&walsnd->mutex);
				continue;
			}

			if (walsnd->state != WALSNDSTATE_STOPPING)
			{
				all_stopped = false;
				SpinLockRelease(&walsnd->mutex);
				break;
			}
			SpinLockRelease(&walsnd->mutex);
		}

		/* safe to leave if confirmation is done for all WAL senders */
		if (all_stopped)
			return;

		pg_usleep(10000L);		/* wait for 10 msec */
	}
}

Finally, combined with the comments in walsender.c:

 * If the server is shut down, checkpointer sends us
 * PROCSIG_WALSND_INIT_STOPPING after all regular backends have exited. If
 * the backend is idle or runs an SQL query this causes the backend to
 * shutdown, if logical replication is in progress all existing WAL records
 * are processed followed by a shutdown. Otherwise this causes the walsender
 * to switch to the "stopping" state. In this state, the walsender will reject
 * any further replication commands. The checkpointer begins the shutdown
 * checkpoint once all walsenders are confirmed as stopping. When the shutdown
 * checkpoint finishes, the postmaster sends us SIGUSR2. This instructs
 * walsender to send any outstanding WAL, including the shutdown checkpoint
 * record, wait for it to be replicated to the standby, and then exit.

After all regular backends have exited, checkpointer sends PROCSIG_WALSND_INIT_STOPPING to walsenders
Walsender may enter the stopping state
Only after all walsenders enter stopping state does checkpointer perform the shutdown checkpoint
After the shutdown checkpoint completes, pm sends SIGUSR2 to walsender, which sends any remaining WAL including the shutdown checkpoint record itself, waits for standby to complete, then exits

Shutdown Flow Diagram
#

After going through the source code, it felt like I understood but also didn’t — needed a shutdown flowchart to clarify.

Summary of the fast shutdown flow:

(High resolution: https://www.processon.com/view/link/6778a73a04a8344b9502637a)

PG manages shutdown logic through signals, per-process main loops, PM state machine, and the pmdie process reaping function
Also note: signals themselves are asynchronous. If you need to wait for the result of signal processing in a target process, you typically need other synchronization mechanisms (pipes, semaphores, shared memory, etc.). PG mainly relies on process dependencies and whether processes exit normally to determine if signals were properly handled.
pgarch and walsender are treated as the same type of process, handled differently from others (walwriter, bgwriter). pgarch and walsender need to do an additional “last task”. The signal for the “last task” is typically defined as SIGUSR2.
Checkpointer’s normal exit depends on pgarch and walsender exiting normally.
pgarch’s last task is the final archive. So archiving can affect shutdown.
Walsender’s second-to-last task is delivering the final WAL, and its last task is delivering the checkpoint shutdown info. These tasks require downstream reply messages, so walsender can affect shutdown.

Test Reproduction
#

Test: Reproducing Walsender Blocking Shutdown
#

After fast stop shutdown, walsender can block the shutdown.

Tested various scenarios to reproduce walsender blocking shutdown. Currently, the following conditions together make it easier to trigger abnormal shutdown:

One walsender for publication/subscription
One walsender for DTS
Large number of subtransactions causing replication slot spill

This three-in-one scenario doesn’t represent the only scenario; it’s just one that was easier to reproduce after testing many.

--Reproduction commands (not extremely stable reproduction)
1.Create table
--pg
create table lzlpg(id bigserial primary key,a char(2000),b char(2000),c char(2000));
--oracle
create table lzl.lzloracle(id number primary key ,a char(2000),b char(2000),c char(2000)) tablespace FADATA;
2.Set up 2 logical replication links (1 pub/sub, 1 DTS to oracle)

3.Reduce logical_decoding_work_mem
logical_decoding_work_mem=1MB

4.Write large amounts of data (recommended: subtransaction spill)
--Insert one row at a time, each insert as a subtransaction
echo "begin;">subtx.sql
for i in {1..500000}
do
 echo "savepoint p$i;">>subtx.sql
 echo "insert into lzlpg(column1,column2,column3) select 'a','b','c';">>subtx.sql
done

nohup psql -d lzl -f subtx.sql &

5.Stop the database before writing completes
pg_ctl stop -D $PGDATA -m fast

At this point, with fast shutdown, the database is in an incomplete shutdown state:

~/lzl/slot]$ ps -axjf|grep 110402
150696 64964 64961 146782 pts/42 64961 S+ 6001 0:00 \_ grep --color=auto 110402
 1 110402 110402 110402 ? -1 Ss 6001 0:00 /myhost/postgres/base/rasesql1.5.6/bin/postgres -D /myhost/pg8094/data
110402 110599 110599 110599 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: logger 
110402 117803 117803 117803 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: checkpointer 
110402 117807 117807 117807 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: stats collector 
110402 118563 118563 118563 ? -1 Rs 6001 3:29 \_ postgres: lzlpg: walsender lzl 127.0.0.1(62971) idle
110402 222918 222918 222918 ? -1 Rs 6001 2:59 \_ postgres: lzlpg: walsender dtssync 30.181.46.203(57218) idle

Walsender, checkpointer, postmaster are all still there; logger and stats haven’t exited either.

The control file state is in production: meaning running in production, indicating the local shutdown checkpoint by checkpointer didn’t complete:

~/lzl/slot]$ pg_controldata|grep -i state
Database cluster state: in production

Checkpointer stack:

pstack 117803
#0 0x00002b879fe0b983 in __select_nocancel () from /lib64/libc.so.6
#1 0x00000000008fd04a in pg_usleep (microsec=microsec@entry=10000) at pgsleep.c:56
#2 0x00000000007610c8 in WalSndWaitStopping () at walsender.c:3209
#3 0x000000000051fa86 in ShutdownXLOG (code=code@entry=0, arg=arg@entry=0) at xlog.c:8596
#4 0x00000000007215ff in HandleCheckpointerInterrupts () at checkpointer.c:566
#5 CheckpointerMain () at checkpointer.c:343
...

At this point, checkpointer is stuck in WalSndWaitStopping, meaning checkpointer is waiting for walsender processes to enter stopping state.

Walsender stack at this point:

#0 0x00000000007484fb in ReorderBufferLargestTXN (rb=<optimized out>) at reorderbuffer.c:2345
#1 ReorderBufferCheckMemoryLimit (rb=0x2b8808b94118) at reorderbuffer.c:2390
#2 ReorderBufferQueueChange (rb=0x2b8808b94118, xid=<optimized out>, lsn=1676456602544, change=change@entry=0x2b87a229f408) at reorderbuffer.c:649
#3 0x000000000073ec99 in DecodeTruncate (buf=<optimized out>, buf=<optimized out>, ctx=<optimized out>) at decode.c:872
#4 DecodeHeapOp (buf=0x7ffda7d35180, ctx=0x2b87a224b118) at decode.c:455
#5 LogicalDecodingProcessRecord (ctx=0x2b87a224b118, record=<optimized out>) at decode.c:126
#6 0x000000000075f502 in XLogSendLogical () at walsender.c:2886
#7 0x0000000000761822 in WalSndLoop (send_data=send_data@entry=0x75f4c0 <XLogSendLogical>) at walsender.c:2287
...

Walsender is stuck in the transaction spill function. (Why it’s stuck is still unclear!!!)

Checkpointer process is blocked in WalSndWaitStopping:

/*
 * Wait that all the WAL senders have quit or reached the stopping state. This
 * is used by the checkpointer to control when the shutdown checkpoint can
 * safely be performed.
 */
void
WalSndWaitStopping(void)
{
	for (;;)
	{
		int			i;
		bool		all_stopped = true;

		for (i = 0; i < max_wal_senders; i++)
		{
			WalSnd	 *walsnd = &WalSndCtl->walsnds[i];

			SpinLockAcquire(&walsnd->mutex);

			if (walsnd->pid == 0)
			{
				SpinLockRelease(&walsnd->mutex);
				continue;
			}

			if (walsnd->state != WALSNDSTATE_STOPPING)
			{
				all_stopped = false;
				SpinLockRelease(&walsnd->mutex);
				break;
			}
			SpinLockRelease(&walsnd->mutex);
		}

		/* safe to leave if confirmation is done for all WAL senders */
		if (all_stopped)
			return;

		pg_usleep(10000L);		/* wait for 10 msec */
	}
}

From the code and stack, it’s clear the condition walsnd->state != WALSNDSTATE_STOPPING is hit, causing the infinite loop.

Test: Handling the Mid-Shutdown State
#

The above is an awkward mid-shutdown state. Besides kill -9, there are other better ways to achieve consistent shutdown:

Solution 1: Shut down the downstream process
Solution 2: Send SIGTERM to walsender

Solution 1 test:

When the downstream exits, walsender will also exit:

static void
ProcessRepliesIfAny(void)
{...
				/*
				 * 'X' means that the standby is closing down the socket.
				 */
			case 'X':
				proc_exit(0);

For pub/sub, execute the following on the subscriber side; even if the upstream is in mid-shutdown state, this will cause walsender to exit:

\c lzldb
alter SUBSCRIPTION sub_lzl disable;

However, this depends on the downstream’s own handling; we can’t always quickly shut down the downstream receiver process of DTS and other sync tools.

Solution 2 test:

Since walsender registers the SIGTERM signal, and the select pg_terminate_backend($walsender_pid) command run while the database is running also sends SIGTERM to walsender, theoretically just sending SIGTERM to walsender should handle this, without needing kill -9.

Command:

kill -SIGTERM 62834
#same as kill -15 62834
#same as kill 62834

After normal kill, pm and all other processes exit completely.

Check the control file and WAL log to confirm consistent shutdown:

pg_controldata database state changed from in production to shut down — consistent shutdown:

$ pg_controldata|grep -i state
Database cluster state: shut down

The last record in the WAL log is CHECKPOINT_SHUTDOWN:

pg_waldump 000000010000018600000012|tail -1
pg_waldump: fatal: error in WAL record at 186/915D7920: invalid record length at 186/915D7998: wanted 24, got 0
rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 186/915D7920, prev 186/915D78A8, desc: CHECKPOINT_SHUTDOWN redo 186/915D7920; tli 1; prev tli 1; fpw true; xid 0:13431045; oid 3808147; multi 3; offset 6; oldest xid 485 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 494/13431044; oldest running xid 0; shutdown

Test: Reproducing Only Primary Having CHECKPOINT_SHUTDOWN
#

A phenomenon in the production environment was that the local WAL had a shutdown checkpoint but the standby didn’t. In production, an immediate stop was performed during mid-shutdown, and then startup failed.

At the time, the last 2 WAL records on primary and standby looked something like:

#Primary WAL:
CHECKPOINT_ONLINE
CHECKPOINT_SHUTDOWN
#Standby WAL:
CHECKPOINT_ONLINE

Reproduction commands:

## 1. First reproduce walsender blocking shutdown
(skipped)
## 2. Check the last WAL record
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 188/307ABE00, prev 188/307ABDC8, desc: RUNNING_XACTS nextXid 13432445 latestCompletedXid 13432444 oldestRunningXid 13432445
## 3. pg_ctl stop -D $PGDATA -m i
## 4. Check last WAL record
Unchanged, same as 2
## 5. pg_ctl start -D $PGDATA
## 6. Check last two WAL records
rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 188/307ABE00, prev 188/307ABDC8, desc: RUNNING_XACTS nextXid 13432445 latestCompletedXid 13432444 oldestRunningXid 13432445 #same as 2
rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 188/307ABE38, prev 188/307ABE00, desc: CHECKPOINT_SHUTDOWN redo 188/307ABE38; tli 1; prev tli 1; fpw true; xid 0:13432445; oid 3832732; multi 3; offset 6; oldest xid 485 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 494/13432444; oldest running xid 0; shutdown #CHECKPOINT_SHUTDOWN appears

From this reproduction, CHECKPOINT_SHUTDOWN is actually done during startup!

This matches the production sequence: 1. fast shutdown didn’t complete 2. immediate shutdown 3. startup failed.

Question 1: When during startup is CHECKPOINT_SHUTDOWN done?

Question 2: When is CHECKPOINT_ONLINE triggered? From reproduction appearances, occasionally fast shutdown results in the last WAL record being CHECKPOINT_ONLINE.

Question 1 analysis:

Doing a shutdown checkpoint at startup easily suggests the startup process. Since we’ve previously analyzed the startup process flow, we can directly locate the function StartupXLOG:

/*
 * This must be called ONCE during postmaster or standalone-backend startup
 */
void
StartupXLOG(void)
{...
	if (InRecovery) //Since it was a shutdown stop, instance recovery is needed
	{
		/*
		 * Perform a checkpoint to update all our recovery activity to disk.
		 *
		 * Note that we write a shutdown checkpoint rather than an on-line
		 * one. This is not particularly critical, but since we may be
		 * assigning a new TLI, using a shutdown checkpoint allows us to have
		 * the rule that TLI only changes in shutdown checkpoints, which
		 * allows some extra error checking in xlog_redo.
		 *
		 * In fast promotion, only create a lightweight end-of-recovery record
		 * instead of a full checkpoint. A checkpoint is requested later,
		 * after we're fully out of recovery mode and already accepting
		 * queries.
		 */
		if (bgwriterLaunched) //This if is clearly for standby streaming replication
		{...
		}
		else //Primary startup goes here
			CreateCheckPoint(CHECKPOINT_END_OF_RECOVERY | CHECKPOINT_IMMEDIATE);
	}

Doing a shutdown checkpoint is intentional, mainly for TLI logic code robustness
Whenever it’s not a consistent shutdown, a shutdown checkpoint is performed during startup

So, doing -m i forced shutdown and then starting up will also produce CHECKPOINT_SHUTDOWN — self-tested.

Question 2 analysis:

Tested multiple times, occasionally seen. Speculation: it just happened that before shutdown, checkpoint conditions were met and an online checkpoint was triggered — pure coincidence.

Considering that after a failed database shutdown, whether it’s a script, HA, or manual intervention, forced shutdown may be done, it’s recommended to do at least one checkpoint before shutdown.

Test: Impact of Archiving on Shutdown
#

While analyzing the shutdown code, I also found that after the checkpointer process exits, reaper for checkpointer sends SIGUSR2 to pgarch for its last archive and exit:

static void
reaper(SIGNAL_ARGS)
{...
		if (pid == CheckpointerPID)
		{
			CheckpointerPID = 0;
			if (EXIT_STATUS_0(exitstatus) && pmState == PM_SHUTDOWN)
			{...

				/* Waken archiver for the last time */
				if (PgArchPID != 0)
					signal_child(PgArchPID, SIGUSR2);
 ...
			}
...

And pm’s exit depends on all processes except syslogger having exited:

	if (pmState == PM_WAIT_DEAD_END)
	{
		if (dlist_is_empty(&BackendList) &&
			PgArchPID == 0 && PgStatPID == 0)
		{
			/* These other guys should be dead already */
			Assert(StartupPID == 0);
			Assert(WalReceiverPID == 0);
			Assert(BgWriterPID == 0);
			Assert(CheckpointerPID == 0);
			Assert(WalWriterPID == 0);
			Assert(AutoVacPID == 0);
			/* syslogger is not considered here */
			pmState = PM_NO_CHILDREN;
		}
	}

So in production, slow archiving was also found to affect shutdown.

Reproduction commands:

#Configure archiving
archive_mode = on
archive_command = '/bin/false ;sleep 1000'#Set archiving to always fail with sleep to bypass NUM_ARCHIVE_RETRIES logic

#Shutdown
pg_ctl stop -D $PGDATA -m fast

Processes after shutdown:

$ ps -axjf|grep 61470
88406 88405 68705 pts/48 88405 S+ 6001 0:00 \_ grep --color=auto 61470
61470 61470 61470 ? -1 Ss 6001 0:00 /myhost/postgres/base/rasesql1.5.6/bin/postgres -D /myhost/pg8094/data
61772 61772 61772 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: logger 
63880 63880 63880 ? -1 Ss 6001 0:00 \_ postgres: lzlpg: archiver archiving 000000010000018800000007

Since the checkpointer here has already fully stopped, the database is in a consistent state, so using kill -9 on archiver is fine.

One-Sentence Summary
#

Q1: Why didn’t shutdown complete?

Walsender blocked shutdown. Checkpointer sent SIGUSR1 to walsender and infinitely waited for all walsender processes to enter stopping state; checkpointer got stuck at this step.

The shutdown eventually completed due to -m i forced shutdown.

Q2: Is there a graceful way to shut down from the mid-shutdown state caused by walsender blocking?

Yes. Send SIGTERM (i.e. kill, or kill -15, kill -SIGTERM) to all walsenders. Afterwards, checkpointer and postmaster will complete a clean shutdown.

Walsender registers the SIGTERM signal at startup, and testing shows no scenario where it can’t be handled.

SIGTERM is also the signal sent by pg_terminate_backend(pid), and it’s the command that should be executed to stop walsender during a standard shutdown.

Q3: Why did primary and standby differ by exactly one shutdown checkpoint?

3.1 Explanation for both primary and standby having CHECKPOINT_ONLINE:

The primary triggering CHECKPOINT_ONLINE was purely coincidental
Since the physical walsender was still there, this WAL record was transmitted to the standby

3.2 Explanation for only primary having CHECKPOINT_SHUTDOWN:

This CHECKPOINT_SHUTDOWN was done during primary startup
Since the primary hadn’t fully started, this WAL record wasn’t transmitted to the standby

Q4: Why does archiver block shutdown?

When reaping the checkpointer process, pm tells archiver to do one last archive, and pm depends on all processes except syslogger having exited. So if the last archive is slow or has issues, it blocks shutdown. Archive failure won’t — the archiver process exits quickly on failure.

Q5: Which processes can block shutdown?

Actually, any process not exiting can block shutdown. The question is which ones are more likely to cause trouble. From the shutdown code flow, archiver and walsender commonly block shutdown because they perform a last archive or log transmission during the shutdown phase.

References
#

https://www.postgresql.org/docs/current/server-shutdown.html https://wiki.postgresql.org/wiki/Signals postgres.c postmaster.c walsender.c xlog.c checkpointer.c startup.c pgarch.c

PG Startup Logic and Spill-Caused Slow Startup Analysis

Sat, 04 Jan 2025 00:00:00 +0000

Problem Symptom — Slow Startup
#

Version: PG 13.2

Database startup was slow. The startup process was reading spill files, and the filenames kept changing. Checking the spill files was also very slow — ls -l eventually showed 8 million spill files.

Why Tens of Millions of Spill Files?
#

WAL Segment and LSN Meaning
#

LSN
#

LSN is a 64-bit bigint. An LSN actually looks like 42D3B/1732C540 (hex). Before the slash / is the 32-bit logical log number, and after the / are 32 bits split into segment number + block number + intra-block offset. These 4 parts are:

32 bits	8 bits	11 bits	13 bits
Logical log number	Log segment number	Block number	Intra-block offset

Intra-block offset 8192 = 2^13

Block number = 16M (default WAL segment size) / 8192

WAL Segment
#

A WAL filename consists of 3 groups of hex digits.

Taking the 8k WAL file 0000000300042D3B00000002 as example:

32 bits	32 bits	32 bits
timeline	Logical log number	Log segment number
00000003	00042D3B	00000002

It can be seen that an LSN can locate a WAL filename and the offset position within the file.

Among these, the part before the LSN slash / is the logical log number, and the 8-bit log segment number after the slash / will be used below.

Spill Filename Conversion
#

Replication slot name: logical_ex2209_rep

Spill filename: xid-407989064-lsn-42D1E-20000000.spill

42D1E is not a complete LSN and cannot be directly used with pg_walfile_name to locate a WAL filename. 42D1E is a logical log number. If we directly filter WAL files containing 42D1E in the name, we find 16 WAL files.

Can we locate the WAL log segment number from the number 20000000 to pinpoint the exact file?

Spill filename generation:

/*
 * Given a replication slot, transaction ID and segment number, fill in the
 * corresponding spill file into 'path', which is a caller-owned buffer of size
 * at least MAXPGPATH.
 */
static void
ReorderBufferSerializedPath(char *path, ReplicationSlot *slot, TransactionId xid,
 XLogSegNo segno)
{
 XLogRecPtr recptr;

 XLogSegNoOffsetToRecPtr(segno, 0, wal_segment_size, recptr);

 snprintf(path, MAXPGPATH, "pg_replslot/%s/xid-%u-lsn-%X-%X.spill",
 NameStr(MyReplicationSlot->data.name),
 xid,
 (uint32) (recptr >> 32), (uint32) recptr);
}

The pg_replslot/%s and xid-%u-lsn parts are easy to understand — just the replication slot name and xid. The recptr needs a closer look at its definition:

/*
 * Pointer to a location in the XLOG. These pointers are 64 bits wide,
 * because we don't want them ever to overflow.
 */
typedef uint64 XLogRecPtr;

XLogSegNoOffsetToRecPtr calculates the LSN from the WAL log segment number, segment size, and offset:

#define XLogSegNoOffsetToRecPtr(segno, offset, wal_segsz_bytes, dest) \
 (dest) = (segno) * (wal_segsz_bytes) + (offset)

XLogRecPtr is the LSN! So:

(uint32) (recptr >> 32) takes the first 32 bits of LSN, (uint32) recptr) takes the last 32 bits.

The first 32 bits of LSN is what we saw as the first half of LSN, lsn-42D1E. The last 32 bits of LSN actually contain more information; here we only need the first few bits of the last 32 bits — the segment number.

Since the passed-in offset=0 and we also have segno, we don’t actually need the intra-block offset information to calculate the dest value. The real value of wal_segsz_bytes is 128M = 128*1024*1024. Converting the formula in XLogSegNoOffsetToRecPtr:

segno= dest/(128*1024*1024)
-- Convert hex 20000000
segno= x'20000000'::int/(128*1024*1024)
segno= 4

From this formula we can derive the log segment number segno, which lets us locate the WAL file number.

So the spill filename xid-407989064-lsn-42D1E-20000000.spill corresponds to the WAL file:

Logical log number=42D1E, segment number=04:

ls 42D1E*04
0000000200042D1E00000004

pg_waldump shows xid 407989064 inside.

In practice, the WAL size is also fixed after instance creation, i.e. (128*1024*1024) is a constant, so segno is absolutely correlated with (uint32) recptr, but not equal to it. This means that switching to a new WAL log file creates a new spill file.

Summary of spill file generation rules:

Same transaction id: if it spans multiple WAL files, it produces multiple spills. E.g., a large transaction without subtransactions spanning 3 WAL files produces 3 spill files.
Different transaction ids produce different spills. E.g., 10 million subtransactions produce 10 million spill files.

Spill filename structure xid-407989064-lsn-42D1E-20000000.spill:

xid	First 32 bits of LSN; i.e., WAL logical log number	Converted from WAL log segment number; not equal to segment number
xid-407989064	lsn-42D1E	20000000

## Recovered environment

[postgres]$ ll |head -100
total 40000276
-rw------- 1 postgres postgres 184 Dec 6 15:20 state
-rw------- 1 postgres postgres 196 Dec 6 13:25 xid-407989064-lsn-42D1E-0.spill
-rw------- 1 postgres postgres 208 Dec 6 13:25 xid-407989064-lsn-42D1E-20000000.spill
...
-rw------- 1 postgres postgres 540 Dec 6 16:44 xid-407989064-lsn-42D2A-D0000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989065-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989066-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989068-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989070-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989072-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989076-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989079-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989080-lsn-42D1D-C8000000.spill
-rw------- 1 postgres postgres 152 Dec 6 13:09 xid-407989082-lsn-42D1D-C8000000.spill

[postgres@lzlhost /myhost/liuzhilong/pg_replslot/logical_ex9e15_rep]$ ll |awk '{print $9}'|awk -F '-' '{print $2}'|sort|uniq -c|wc -l
10000003
[postgres@lzlhost /myhost/liuzhilong/pg_replslot/logical_ex9e15_rep]$ ll |wc -l
10000070

So in the actual environment we saw 10,000,070 files, with 10,000,003 distinct xids among them — meaning 1 parent transaction spanning about 70 WAL files, with this parent transaction having 10 million subtransactions.

Replication Slot Spill Testing
#

--Pub/sub replication link setup
logical_decoding_work_mem = 64MB #pg_ctl reload
wal_segment_size =128 MB

--source
CREATE TABLE replication_table (
 id BIGSERIAL PRIMARY KEY,
 column1 char(2000),
 column2 char(2000),
 column3 char(2000)
);
create publication pub_test for table replication_table ;

--dest
CREATE TABLE replication_table (
 id BIGSERIAL PRIMARY KEY,
 column1 char(2000),
 column2 char(2000),
 column3 char(2000)
);

CREATE SUBSCRIPTION sub_test
CONNECTION 'host=127.0.0.1 port=8094 dbname=lzl user=lzl password=qwer'
PUBLICATION pub_test;

--source
select * from pg_replication_slots;

Large Transaction, No Subtransactions, Replicated Table Spill Test
#

--Create a large transaction, don't commit yet
begin;
insert into replication_table(column1,column2,column3) 
select 'a','b','c' from generate_series(1,1000000) g;

--Replication slot spill
 ll
total 331924
-rw------- 1 postgres postgres 184 Dec 9 20:22 state
-rw------- 1 postgres postgres 88226964 Dec 9 20:22 xid-5074343-lsn-163-38000000.spill
-rw------- 1 postgres postgres 119698488 Dec 9 20:22 xid-5074343-lsn-163-40000000.spill

After the large transaction commits, wait for consumption until replication lag is 0, and the spill files disappear:

M=# select pid,usename,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,reply_time from pg_stat_replication;
 pid | usename | sent_lsn | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | reply_time 
--------+---------+--------------+--------------+--------------+--------------+-----------+-----------+------------+------------------------------
 163525 | lzl | 163/4996E1C8 | 163/4996E1C8 | 163/4996E1C8 | 163/4996E1C8 | [null] | [null] | [null] | 2024-12-09 20:25:35.14769+08
(1 row)

M=# select pid,usename,pg_wal_lsn_diff(pg_current_wal_lsn(),sent_lsn) diff_sent_mb,pg_wal_lsn_diff(pg_current_wal_lsn(),write_lsn) diff_write_mb,pg_wal_lsn_diff(pg_current_wal_lsn(),flush_lsn) diff_flush_mb,pg_wal_lsn_diff(pg_current_wal_lsn(),replay_lsn) diff_replay_mb,pg_walfile_name_offset(sent_lsn) sentoffset,pg_walfile_name_offset(write_lsn) writeoffset,pg_walfile_name_offset(flush_lsn) flush_lsn from pg_stat_replication;
 pid | usename | diff_sent_mb | diff_write_mb | diff_flush_mb | diff_replay_mb | sentoffset | writeoffset | flush_lsn 
--------+---------+--------------+---------------+---------------+----------------+-------------------------------------+-------------------------------------+-------------------------------
 163525 | lzl | 0 | 0 | 0 | 0 | (000000010000016300000009,26665416) | (000000010000016300000009,26665416) | (000000


[/mypg/pg8094/data/pg_replslot/sub_test]$ ll
total 357392
-rw------- 1 postgres postgres 184 Dec 9 20:23 state
-rw------- 1 postgres postgres 88226964 Dec 9 20:22 xid-5074343-lsn-163-38000000.spill
-rw------- 1 postgres postgres 137696328 Dec 9 20:23 xid-5074343-lsn-163-40000000.spill
-rw------- 1 postgres postgres 26076708 Dec 9 20:23 xid-5074343-lsn-163-48000000.spill
[/mypg/pg8094/data/pg_replslot/sub_test]$ ll
total 4
-rw------- 1 postgres postgres 184 Dec 9 20:25 state2666
(1 row)

Large Transaction, No Subtransactions, Non-Replicated Table Spill Test
#

--source: create an unrelated table for writing data
CREATE TABLE no_replication_table (
 id BIGSERIAL PRIMARY KEY,
 column1 char(2000),
 column2 char(2000),
 column3 char(2000)
);

--Create a large transaction, don't commit yet
begin;
insert into no_replication_table(column1,column2,column3) 
select 'a','b','c' from generate_series(1,1000000) g;

--Spill
[postgres@lzldb:MYINST:8094 /mypg/pg8094/data/pg_replslot/sub_test]$ ll
total 357492
-rw------- 1 postgres postgres 184 Dec 9 20:09 state
-rw------- 1 postgres postgres 107511456 Dec 9 20:08 xid-5074106-lsn-163-28000000.spill
-rw------- 1 postgres postgres 137698804 Dec 9 20:09 xid-5074106-lsn-163-30000000.spill
-rw------- 1 postgres postgres 4308444 Dec 9 20:09 xid-5074106-lsn-163-38000000.spill

Large Transaction, Subtransactions, Non-Replicated Table Spill Test
#

## One insert per row, each insert as one subtransaction
echo "begin;">subtx.sql
for i in {1..1000000}
do
 echo "savepoint p$i;">>subtx.sql
 echo "insert into no_replication_table(column1,column2,column3) select 'a','b','c';">>subtx.sql
done


nohup psql -d lzl -f subtx.sql &

#During execution, observed 800k+ spill files
[/myhost/pg8094/data/pg_replslot/sub_test]$ ll |wc -l
823749
[/myhost/pg8094/data/pg_replslot/sub_test]$ ll |head -10
total 1099532
-rw------- 1 postgres postgres 184 Dec 9 21:10 state
-rw------- 1 postgres postgres 1236 Dec 9 21:10 xid-5519686-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519687-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519688-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519689-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519690-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519691-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519692-lsn-163-70000000.spill
-rw------- 1 postgres postgres 252 Dec 9 21:09 xid-5519693-lsn-163-70000000.spill

Analysis of Slow Database Startup
#

Startup Process Startup Flow Analysis
#

Here we parse the startup flow frame by frame using the call stack:

11: main: Nothing to say.

10: PostmasterMain:

Before the main loop, it first calls the startup flow StartupPID = StartupDataBase(); which essentially calls StartChildProcess(StartupProcess):

#define StartupDataBase()		StartChildProcess(StartupProcess)

9: StartChildProcess: Forks a process. This process is the auxiliary process for starting postmaster; normal child process startup goes through this logic, forking at this step. The input AuxProcType=StartupProcess.

8: AuxiliaryProcessMain:

Since MyAuxProcType=StartupProcess, it goes through the StartupProcessMain flow, which is different from child processes like walsender, walwriter, bgwriter. The startup process itself exists for crash recovery WAL reading, but it does many other things:

	switch (MyAuxProcType)
	{
		case CheckerProcess:
			/* don't set signals, they're useless here */
			CheckerModeMain();
			proc_exit(1);		/* should never return */

		case BootstrapProcess:

			/*
			 * There was a brief instant during which mode was Normal; this is
			 * okay. We need to be in bootstrap mode during BootStrapXLOG for
			 * the sake of multixact initialization.
			 */
			SetProcessingMode(BootstrapProcessing);
			bootstrap_signals();
			BootStrapXLOG();
			BootstrapModeMain();
			proc_exit(1);		/* should never return */

		case StartupProcess: //Here here here here
			/* don't set signals, startup process has its own agenda */
			StartupProcessMain();
			proc_exit(1);		/* should never return */

		case BgWriterProcess:
			/* don't set signals, bgwriter has its own agenda */
			BackgroundWriterMain();
			proc_exit(1);		/* should never return */

		case CheckpointerProcess:
			/* don't set signals, checkpointer has its own agenda */
			CheckpointerMain();
			proc_exit(1);		/* should never return */

		case WalWriterProcess:
			/* don't set signals, walwriter has its own agenda */
			InitXLOGAccess();
			WalWriterMain();
			proc_exit(1);		/* should never return */

		case WalReceiverProcess:
			/* don't set signals, walreceiver has its own agenda */
			WalReceiverMain();
			proc_exit(1);		/* should never return */

		default:
			elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
			proc_exit(1);
	}

7: StartupProcessMain: Mainly to call StartupXLOG().

6: StartupXLOG:

Function comment:

This must be called ONCE during postmaster or standalone-backend startup

StartupXLOG is always called by postmaster regardless, whether crash shutdown or consistent shutdown:

	switch (ControlFile->state)
	{
 ...
			case DB_IN_PRODUCTION:
			ereport(LOG,
					(errmsg("database system was interrupted; last known up at %s",
							str_time(ControlFile->time))));
			break;

This matches the log output. Here’s the shutdown and startup log:

2024-12-06 17:02:57.534 CST,,,447560,,65693cde.6d448,1325,,2023-12-01 09:54:38 CST,,0,LOG,00000,"database system is shut down",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.536 CST,,,211844,,6752bdf3.33b84,1,,2024-12-06 17:03:47 CST,,0,LOG,00000,"ending log output to stderr",,"Future log output will go to log destination ""csvlog"".",,,,,,,"","postmaster"
2024-12-06 17:03:49.536 CST,,,211844,,6752bdf3.33b84,2,,2024-12-06 17:03:47 CST,,0,LOG,00000,"starting PostgreSQL 13.2 (RaseSQL 1.3) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39.0.1), 64-bit",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.537 CST,,,211844,,6752bdf3.33b84,3,,2024-12-06 17:03:47 CST,,0,LOG,00000,"listening on IPv4 address ""0.0.0.0"", port 7284",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.539 CST,,,211844,,6752bdf3.33b84,4,,2024-12-06 17:03:47 CST,,0,LOG,00000,"listening on Unix socket ""/tmp/.s.PGSQL.7284""",,,,,,,,,"","postmaster"
2024-12-06 17:03:49.557 CST,,,211995,,6752bdf5.33c1b,1,,2024-12-06 17:03:49 CST,,0,LOG,00000,"database system was interrupted; last known up at 2024-12-06 17:00:10 CST",,,,,,,,,"","startup"

So, after shutdown, the control file recorded the database state as in production:

Database cluster state: in production

The in production state means the database is running, not a normal shutdown state — indicating that at the time of shutdown, it was not a consistent shutdown.

Continuing with the key code about fsync:

	/*----------
	 * If we previously crashed, perform a couple of actions:
	 *
	 * - The pg_wal directory may still include some temporary WAL segments
	 * used when creating a new segment, so perform some clean up to not
	 * bloat this path. This is done first as there is no point to sync
	 * this temporary data.
	 *
	 * - There might be data which we had written, intending to fsync it, but
	 * which we had not actually fsync'd yet. Therefore, a power failure in
	 * the near future might cause earlier unflushed writes to be lost, even
	 * though more recent data written to disk from here on would be
	 * persisted. To avoid that, fsync the entire data directory.
	 */
	if (ControlFile->state != DB_SHUTDOWNED &&
		ControlFile->state != DB_SHUTDOWNED_IN_RECOVERY)
	{
		RemoveTempXlogFiles();
		SyncDataDirectory();
	}

Here, because the control file state is not a normal shutdown, it enters the if-block and calls SyncDataDirectory() for fsync persistence.

StartupXLOG does many many things. Among those related to spill, besides SyncDataDirectory(), there’s also StartupReorderBuffer():

	/*
	 * Initialize replication slots, before there's a chance to remove
	 * required resources.
	 */
	StartupReplicationSlots();

	/*
	 * Startup logical state, needs to be setup now so we have proper data
	 * during crash recovery.
	 */
	StartupReorderBuffer();

StartupReorderBuffer is also called. It calls ReorderBufferCleanupSerializedTXNs to clean up spill files in all slot directories (but does not delete directories or state files):

/*
 * Delete all data spilled to disk after we've restarted/crashed. It will be
 * recreated when the respective slots are reused.
 */
void
StartupReorderBuffer(void)
{
	DIR		 *logical_dir;
	struct dirent *logical_de;

	logical_dir = AllocateDir("pg_replslot");
	while ((logical_de = ReadDir(logical_dir, "pg_replslot")) != NULL)
	{
		if (strcmp(logical_de->d_name, ".") == 0 ||
			strcmp(logical_de->d_name, "..") == 0)
			continue;

		/* if it cannot be a slot, skip the directory */
		if (!ReplicationSlotValidateName(logical_de->d_name, DEBUG2))
			continue;

		/*
		 * ok, has to be a surviving logical slot, iterate and delete
		 * everything starting with xid-*
		 */
		ReorderBufferCleanupSerializedTXNs(logical_de->d_name);
	}
	FreeDir(logical_dir);
}

5: SyncDataDirectory:

The function comment is very important:

/*
 * Issue fsync recursively on PGDATA and all its contents.
 *
 * We fsync regular files and directories wherever they are, but we
 * follow symlinks only for pg_wal and immediately under pg_tblspc.
 * Other symlinks are presumed to point at files we're not responsible
 * for fsyncing, and might not have privileges to write at all.
 *
 * Errors are logged but not considered fatal; that's because this is used
 * only during database startup, to deal with the possibility that there are
 * issued-but-unsynced writes pending against the data directory. We want to
 * ensure that such writes reach disk before anything that's done in the new
 * run. However, aborting on error would result in failure to start for
 * harmless cases such as read-only files in the data directory, and that's
 * not good either.
 *
 * Note that if we previously crashed due to a PANIC on fsync(), we'll be
 * rewriting all changes again during recovery.
 *
 * Note we assume we're chdir'd into PGDATA to begin with.
 */

fsync all data directory files to persist them
This action only happens during the startup phase
This action ensures the data directory is fully persistent before the database starts running

The body of SyncDataDirectory recursively walks directories and fsyncs (with some special handling for symlinks):

	walkdir(".", datadir_fsync_fname, false, LOG);
	if (xlog_is_symlink)
		walkdir("pg_wal", datadir_fsync_fname, false, LOG);
	walkdir("pg_tblspc", datadir_fsync_fname, true, LOG);

4: walkdir: Recurse to .

3: walkdir: Recurse to ./pg_replslot

2: walkdir: Recurse to ./pg_replslot/slotname

1: lstat: C library call. walkdir not only does fsync (via the callback datadir_fsync_fname), the walkdir function body also does lstat to get file info such as inode, file size, last modification time, etc. — similar to the Linux stat command.

0: _lxstat: C library call.

Startup logic summary:

PG starts an auxiliary process startup to help with startup. Unlike common child processes (walwriter, bgwriter, checkpointer, etc.), it’s always started during the startup process and does many things.
StartupXLOG is always called during startup, whether or not the database was consistently shut down.
Only in a non-normal shutdown state does SyncDataDirectory get triggered.
SyncDataDirectory fsyncs all data files for persistence and checks stat info for all data files.
fsync ensures data file consistency before startup; stat is probably to verify files are normal and readable (before the startup process starts, only the readability of the datadir directory was verified).
Regardless of shutdown state, StartupReorderBuffer is always called and cleans up spill files for all replication slots.

When Is the Ready State?
#

After the startup process finishes its work, the database is not yet in ready state. When the pmState state machine changes state, the reaper process reaping function is called. The reaper function itself does some recovery or startup work after a child process exits. The pmState state machine records the state as PM_STARTUP, which controls the startup/shutdown state.

Last steps of PostmasterMain:

	StartupPID = StartupDataBase();
	Assert(StartupPID != 0);
	StartupStatus = STARTUP_RUNNING;
	pmState = PM_STARTUP; //State machine changes state

	/* Some workers may be scheduled to start now */
	maybe_start_bgworkers();

	status = ServerLoop();

	/*
	 * ServerLoop probably shouldn't ever return, but if it does, close down.
	 */
	ExitPostmaster(status != STATUS_OK);

	abort();					/* not reached */
}

The core startup flow of PostmasterMain goes to reaper to handle the normal exit of the startup process.

PMState comment:

/*
 * We use a simple state machine to control startup, shutdown, and
 * crash recovery (which is rather like shutdown followed by startup).
 *
 * After doing all the postmaster initialization work, we enter PM_STARTUP
 * state and the startup process is launched. The startup process begins by
 * reading the control file and other preliminary initialization steps.
 * In a normal startup, or after crash recovery, the startup process exits
 * with exit code 0 and we switch to PM_RUN state. 

PMState is passed and processed via signals. After the startup process exits, reaper is activated to reap the process.

reaper function handling the startup child process’s normal exit:

		if (pid == StartupPID)
		{
			StartupPID = 0;
...

			/*
			 * Startup succeeded, commence normal operations
			 */
			StartupStatus = STARTUP_NOT_RUNNING; //Transition from STARTUP_RUNNING to STARTUP_NOT_RUNNING
			FatalError = false; //After none of the above ifs are hit, it's not fatal
			AbortStartTime = 0;
			ReachedNormalRunning = true;
			pmState = PM_RUN; //State machine transitions from PM_STARTUP to PM_RUN
			connsAllowed = ALLOW_ALL_CONNS;

			/*
			 * Crank up the background tasks, if we didn't do that already
			 * when we entered consistent recovery state. It doesn't matter
			 * if this fails, we'll just try again later.
			 */
 //Below: starting core child processes
			if (CheckpointerPID == 0)
				CheckpointerPID = StartCheckpointer();
			if (BgWriterPID == 0)
				BgWriterPID = StartBackgroundWriter();
			if (WalWriterPID == 0)
				WalWriterPID = StartWalWriter();

			/*
			 * Likewise, start other special children as needed. In a restart
			 * situation, some of them may be alive already.
			 */
 //Below: starting non-core child processes
			if (!IsBinaryUpgrade && AutoVacuumingActive() && AutoVacPID == 0)
				AutoVacPID = StartAutoVacLauncher();
			if (PgArchStartupAllowed() && PgArchPID == 0)
				PgArchPID = pgarch_start();
			if (PgStatPID == 0)
				PgStatPID = pgstat_start();

			/* workers may be scheduled to start now */
			maybe_start_bgworkers();
		 //At this point it's officially ready to accept connections
			/* at this point we are really open for business */
			ereport(LOG,
					(errmsg("database system is ready to accept connections")));

			/* Report status */
			AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_READY);
#ifdef USE_SYSTEMD
			sd_notify(0, "READY=1");
#endif

			continue;
		}

The “database system is ready to accept connections” message is right here.

Checkpointer, bgwriter, walwriter, autovacuum, arch (if present), stats — all these processes need to be started. At this stage, launching these processes doesn’t have to return success; they can be retried later in ServerLoop or on the next reaper execution. Only the startup process must start and complete all related tasks in one shot:

	if (pid < 0)
	{
		/* in parent, fork failed */
		int			save_errno = errno;

		errno = save_errno;
		switch (type)
		{
			case StartupProcess:
				ereport(LOG,
						(errmsg("could not fork startup process: %m")));
				break;
			case BgWriterProcess:
				ereport(LOG,
						(errmsg("could not fork background writer process: %m")));
				break;
			case CheckpointerProcess:
				ereport(LOG,
						(errmsg("could not fork checkpointer process: %m")));
				break;
			case WalWriterProcess:
				ereport(LOG,
						(errmsg("could not fork WAL writer process: %m")));
				break;
			case WalReceiverProcess:
				ereport(LOG,
						(errmsg("could not fork WAL receiver process: %m")));
				break;
			default:
				ereport(LOG,
						(errmsg("could not fork process: %m")));
				break;
		}

		/*
		 * fork failure is fatal during startup, but there's no need to choke
		 * immediately if starting other child types fails.
		 */
		if (type == StartupProcess)
			ExitPostmaster(1);
		return 0;
	}

Spill File Generation Logic Across Versions
#

Spill in all versions spills the largest transaction. Here we focus on when spilling happens.

PG12: pg12 hard-codes 4096 changes:

static const Size max_changes_in_memory = 4096;

/*
 * Check whether the transaction tx should spill its data to disk.
 */
static void
ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
	/*
	 * TODO: improve accounting so we cheaply can take subtransactions into
	 * account here.
	 */
	if (txn->nentries_mem >= max_changes_in_memory)
	{
		ReorderBufferSerializeTXN(rb, txn);
		Assert(txn->nentries_mem == 0);
	}
}

PG13: Spills when exceeding logical_decoding_work_mem memory size:

static void
ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
...
	while (rb->size >= logical_decoding_work_mem * 1024L)
	{
		/*
		 * Pick the largest transaction (or subtransaction) and evict it from
		 * memory by serializing it to disk.
		 */
		txn = ReorderBufferLargestTXN(rb);

		ReorderBufferSerializeTXN(rb, txn);
...
}

PG14: Adds streaming transfer ReorderBufferStreamTXN:

static void
ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
...
	while (rb->size >= logical_decoding_work_mem * 1024L)
	{
		/*
		 * Pick the largest transaction (or subtransaction) and evict it from
		 * memory by streaming, if possible. Otherwise, spill to disk.
		 */
		if (ReorderBufferCanStartStreaming(rb) &&
			(txn = ReorderBufferLargestTopTXN(rb)) != NULL)
		{...
			ReorderBufferStreamTXN(rb, txn);
		}
		else
		{...
			ReorderBufferSerializeTXN(rb, txn);
		}
...
}

Although PG14 has streaming replication, triggering it requires certain conditions:

/* Returns true, if the streaming can be started now, false, otherwise. */
static inline bool
ReorderBufferCanStartStreaming(ReorderBuffer *rb)
{
	LogicalDecodingContext *ctx = rb->private_data;
	SnapBuild *builder = ctx->snapshot_builder;

	/* We can't start streaming unless a consistent state is reached. */
	if (SnapBuildCurrentState(builder) < SNAPBUILD_CONSISTENT)
		return false;

	/*
	 * We can't start streaming immediately even if the streaming is enabled
	 * because we previously decoded this transaction and now just are
	 * restarting.
	 */
	if (ReorderBufferCanStream(rb) &&
		!SnapBuildXactNeedsSkip(builder, ctx->reader->EndRecPtr))
		return true;

	return false;
}

	/*
	 * Found a point after SNAPBUILD_FULL_SNAPSHOT where all transactions that
	 * were running at that point finished. Till we reach that we hold off
	 * calling any commit callbacks.
	 */
	SNAPBUILD_CONSISTENT = 2

Additional streaming trigger conditions:

Condition 1: All transactions covered by the snapshot have completed (presumably committed or rolled back)
Condition 2: The context is private data (does this mean two links to one table won’t trigger streaming?)
Condition 3: Transactions in the snapshot are not skippable (probably some special transactions can be skipped)

PG15: Similar to 14, just cleaner functions with less nesting.

PG16: About the same.

PG17: About the same, adds DEBUG_LOGICAL_REP_STREAMING_IMMEDIATE to force streaming.

Key points to remember:

PG12 and earlier: hard-coded 4096 changes
PG13: adds logical_decoding_work_mem parameter, allowing memory size adjustment to reduce spill probability
PG14 and later: supports streaming replication
Triggering streaming also requires certain conditions, so even with streaming, spills can still happen
PG17: adds debug_logical_replication_streaming parameter to force streaming

Spill File Cleanup Logic
#

Startup-time spill cleanup is just one scenario. There’s also walsender startup cleanup and drop slot cleanup.

Walsender Startup Cleanup
#

ReorderBufferCleanupSerializedTXNs is called during database startup (before walsender has started) and during walsender startup (while the database is running). Note these are different scenarios, though they call the same function. From the function comment, it’s meant to “remove leftover serialized reorder buffers” — i.e., clean up spill files.

/*
 * Remove any leftover serialized reorder buffers from a slot directory after a
 * prior crash or decoding session exit.
 */
static void
ReorderBufferCleanupSerializedTXNs(const char *slotname)
{
	DIR		 *spill_dir;
	struct dirent *spill_de;
	struct stat statbuf;
	char		path[MAXPGPATH * 2 + 12];

	sprintf(path, "pg_replslot/%s", slotname);

	/* we're only handling directories here, skip if it's not ours */
	if (lstat(path, &statbuf) == 0 && !S_ISDIR(statbuf.st_mode))
		return;

	spill_dir = AllocateDir(path);
	while ((spill_de = ReadDirExtended(spill_dir, path, INFO)) != NULL)
	{
		/* only look at names that can be ours */
 //Only compare first 3 characters
		if (strncmp(spill_de->d_name, "xid", 3) == 0)
		{
			snprintf(path, sizeof(path),
					 "pg_replslot/%s/%s", slotname,
					 spill_de->d_name);

			if (unlink(path) != 0)
				ereport(ERROR,
						(errcode_for_file_access(),
						 errmsg("could not remove file \"%s\" during removal of pg_replslot/%s/xid*: %m",
								path, slotname)));
		}
	}
	FreeDir(spill_dir);
}

Two things to note about the above cleanup logic:

Cleans files whose names start with “xid”. Obviously, the state file is not cleaned.
Uses unlink to clean, one file at a time. This can help us devise a faster startup scheme.

Database Startup Cleanup
#

During database startup, a startup process is forked to clean slots. The cleanup function is the same one walsender calls: ReorderBufferCleanupSerializedTXNs.

One more difference: after walsender restarts, it only cleans spills for the current slot with the same name; whereas during database startup, all slot spills are cleaned sequentially.

Database startup process, while-loop sequential cleanup logic:

void
StartupReorderBuffer(void)
{
	DIR		 *logical_dir;
	struct dirent *logical_de;

	logical_dir = AllocateDir("pg_replslot");
	while ((logical_de = ReadDir(logical_dir, "pg_replslot")) != NULL)
	{	//Exclude . and ..
		if (strcmp(logical_de->d_name, ".") == 0 ||
			strcmp(logical_de->d_name, "..") == 0)
			continue;
		//Validate slot name
		/* if it cannot be a slot, skip the directory */
		if (!ReplicationSlotValidateName(logical_de->d_name, DEBUG2))
			continue;

		/*
		 * ok, has to be a surviving logical slot, iterate and delete
		 * everything starting with xid-*
		 */
		ReorderBufferCleanupSerializedTXNs(logical_de->d_name);
	}
	FreeDir(logical_dir);
}

The while loop calls ReorderBufferCleanupSerializedTXNs, and after that, the logic is the same as walsender startup cleanup.

Manual Cleanup via pg_drop_replication_slot
#

The drop slot cleanup logic is different from the automatic spill file cleanup — it does not call ReorderBufferCleanupSerializedTXNs.

Drop slot flow:

pg_drop_replication_slot(PG_FUNCTION_ARGS) -> ReplicationSlotDrop(const char *name, bool nowait) -> ReplicationSlotDropAcquired(void) -> ReplicationSlotDropPtr

ReplicationSlotDropPtr’s slot cleanup logic is also interesting:

/*
 * Permanently drop the replication slot which will be released by the point
 * this function returns.
 */
static void
ReplicationSlotDropPtr(ReplicationSlot *slot)
{
	char		path[MAXPGPATH];
	char		tmppath[MAXPGPATH];

	/*
	 * If some other backend ran this code concurrently with us, we might try
	 * to delete a slot with a certain name while someone else was trying to
	 * create a slot with the same name.
	 */
	LWLockAcquire(ReplicationSlotAllocationLock, LW_EXCLUSIVE);

	/* Generate pathnames. */
	sprintf(path, "pg_replslot/%s", NameStr(slot->data.name));
	sprintf(tmppath, "pg_replslot/%s.tmp", NameStr(slot->data.name));

	/*
	 * Rename the slot directory on disk, so that we'll no longer recognize
	 * this as a valid slot. Note that if this fails, we've got to mark the
	 * slot inactive before bailing out. If we're dropping an ephemeral or a
	 * temporary slot, we better never fail hard as the caller won't expect
	 * the slot to survive and this might get called during error handling.
	 */
	if (rename(path, tmppath) == 0) //rename file
	{
		/*
		 * We need to fsync() the directory we just renamed and its parent to
		 * make sure that our changes are on disk in a crash-safe fashion. If
		 * fsync() fails, we can't be sure whether the changes are on disk or
		 * not. For now, we handle that by panicking;
		 * StartupReplicationSlots() will try to straighten it out after
		 * restart.
		 */
 //fsync persistence
		START_CRIT_SECTION();
		fsync_fname(tmppath, true);
		fsync_fname("pg_replslot", true);
		END_CRIT_SECTION();
	}
...

	/*
	 * If removing the directory fails, the worst thing that will happen is
	 * that the user won't be able to create a new slot with the same name
	 * until the next server restart. We warn about it, but that's all.
	 */
 
	if (!rmtree(tmppath, true))
		ereport(WARNING,
				(errmsg("could not remove directory \"%s\"", tmppath)));
	/*
	 * We release this at the very end, so that nobody starts trying to create
	 * a slot while we're still cleaning up the detritus of the old one.
	 */
	LWLockRelease(ReplicationSlotAllocationLock);
}

Drop slot doesn’t directly unlink files in the slot directory. Instead, it first renames the slotname/ directory to slotname.tmp/, then unlinks the files inside, and finally removes the slotname.tmp/ directory itself.

In this, rmtree also loops to unlink files.

Accelerated Startup Scheme After Replication Slot Spill
#

Deleting 10 million spill files is obviously very slow, but directly moving the directory (mv) is extremely fast. However, direct mv requires attention to the name after the move and the state file, as well as knowing which source code step the mv bypasses.

mv Naming Notes
#

Since it was an abnormal shutdown, the startup process will execute SyncDataDirectory to fsync and stat all data files — this is hard to bypass. After SyncDataDirectory completes, it starts handling replication slots. When handling slots, it calls StartupReorderBuffer() -> ReorderBufferCleanupSerializedTXNs to fully clean up spill files.

Before entering cleanup, ReplicationSlotValidateName validates the slot name. We can exploit ReplicationSlotValidateName to trick the startup process into skipping the ReorderBufferCleanupSerializedTXNs process.

ReplicationSlotValidateName rules:

bool
ReplicationSlotValidateName(const char *name, int elevel)
{
...

	for (cp = name; *cp; cp++)
	{ //Key rule here
		if (!((*cp >= 'a' && *cp <= 'z')
			 || (*cp >= '0' && *cp <= '9')
			 || (*cp == '_')))
		{
			ereport(elevel,
					(errcode(ERRCODE_INVALID_NAME),
					 errmsg("replication slot name \"%s\" contains invalid character",
							name),
					 errhint("Replication slot names may only contain lower case letters, numbers, and the underscore character.")));
			return false;
		}
	}
	return true;
}

Valid slot names only contain a-z, 0-9, _.

So when renaming, it’s recommended to add a dot .:

Recommended: slotname.bak, slotname.20241215, etc.
Not recommended: slotnamebackup, slotname20241215, slotname_bak, etc.
Not recommended: .tmp suffix — slot names with .tmp have special meaning.

After renaming, you need to create the directory and copy the state file, otherwise the slot will behave strangely on startup (e.g., duplicate slot names, auto-generated slot names, inability to delete slots, downstream unable to start the replication link, etc.).

Recommended mv operations summarized:

cd pg_replslot
mv slotname slotname.bak 
mkdir slotname
cp slotname.bak/state slotname/

Startup Time Comparison
#

Compare startup speed across different source code flows to see if manual mv/rm acceleration is actually meaningful.

Reference source logic principles:

Normal shutdown: goes through fsync and stat
Abnormal shutdown: goes through fsync and stat
Valid mv: rename slot directory to .bak, skip unlink
Invalid mv: rename slot directory to _bak, spill files start with xid, goes through unlink

Since actual spill files would be too slow, I manually created fake slot directories and spill files: 50 slots total, 400k spills per slot, 20 million spills total, to test startup time (using cp directory is much faster than cp or dd files).

#	Test Plan	Startup Time
1	Normal shutdown; no fsync/stat, no unlink	0.1 seconds
2	Normal shutdown, invalid mv; no fsync/stat, unlink	11 min 41 sec
3	Abnormal shutdown, valid mv; fsync/stat, no unlink	4 min 35 sec
4	Abnormal shutdown, invalid mv; fsync/stat, unlink	32 min 2 sec
5	Abnormal shutdown, rm (create slot dir, keep state)	13 min 4 sec

Comparing plans 3 and 5, theoretically in the scenario at hand, a valid mv could achieve startup in about 4 minutes, while rm would take about 13 minutes. (This is a rough comparison; the recovery environment already showed some differences.)

Book Notes — Designing Data-Intensive Applications (2nd Edition)

Fri, 20 Sep 2024 00:00:00 +0000

DDIA-v2 Chinese edition: https://github.com/Vonng/ddia/tree/v2

After finishing DDIA-v2, I couldn’t put it down. Everything data-related is explained with such clarity — why is it like this? What’s the current state? What problems does this have? The observations and ideas are incredibly incisive and concise. Even the nautical-chart-style diagrams at the start of each chapter are fascinating.

Note: This article is essentially a collection of excerpts from the original work, with almost none of my own thoughts or ideas. I’ve simply plucked out the parts I love most. Some knowledge I’ve already mastered and some topics too remote are skipped!

Ch1: Trade-offs in Data System Architecture
#

OLTP & OLAP
#

The distinction between OLTP and analytics is not always clear-cut, but the following table lists some typical characteristics:

Attribute	Transactional Systems (OLTP)	Analytical Systems (OLAP)
Primary read pattern	Point queries (fetch individual records by key)	Aggregation over a large number of records
Primary write pattern	Create, update, and delete individual records	Bulk import (ETL) or event stream
Human user example	End users of web/mobile applications	Internal analysts, for decision support
Machine use example	Check whether an action is authorized	Detect fraud/abuse patterns
Query type	Fixed set of queries, predefined by the application	Analysts can issue arbitrary queries
Data representation	Latest state of data (current point in time)	History of events over time
Dataset size	GB, TB	TB, PB

A data warehouse is a separate database where analysts can query freely without affecting OLTP operations. Data warehouses typically store data in a very different way from OLTP databases, optimized for the query types common in analytics. The process of getting data into the data warehouse is called Extract–Transform–Load (ETL). Some database systems offer Hybrid Transaction/Analytical Processing (HTAP), aiming to enable both OLTP and analytics in a single system without ETL from one system to another. Despite the existence of HTAP, the separation between transactional and analytical systems remains common due to their differing goals and requirements. In particular, it is considered good practice for each business system to have its own database, resulting in hundreds of independent operational databases; on the other hand, an enterprise typically has only one data warehouse, allowing business analysts to combine data from several business systems in a single query. A data lake is a centralized data repository that holds any data potentially useful for analysis, sourced from business systems through ETL processes. Unlike a data warehouse, a data lake contains only files and imposes no specific file format or data model. Data warehouses typically use the relational data model and are queried via SQL. A data lakehouse goes beyond a standalone data warehouse by enabling typical data warehouse workloads (SQL queries and business analytics) as well as data science/machine learning workloads to run directly on files in the data lake. This architecture is called a data lakehouse. It requires a query execution engine and a metadata (e.g., schema management) layer to extend the file storage of the data lake. Apache Hive, Spark SQL, Presto, and Trino are examples of this approach.

Cloud Services vs. Self-Hosting
#

The pros and cons of cloud services: Using cloud services, rather than running comparable software yourself, is essentially outsourcing the operation of that software to a cloud provider. There are strong arguments both for and against using cloud services.

Advantages:

When you use the cloud, you still need an operations team, but outsourcing basic system administration can free your team to focus on higher-level problems.
Cloud services are especially valuable if your system load varies significantly over time. If you provision machines to handle peak load but those computing resources sit idle most of the time, your system becomes less cost-effective.
Compared to physical machines, cloud instances can be provisioned faster and come in a wider variety of sizes.

Disadvantages:

The biggest drawback of cloud services is that you have no control over them.
If you already have experience setting up and operating the required systems and your load is fairly predictable (i.e., the number of machines you need won’t fluctuate dramatically), it is typically cheaper to buy your own machines and run the software yourself.
If the service lacks a feature you need, your only option is to politely ask the vendor whether they’ll add it; you usually can’t implement it yourself.
If the service goes down, you can only wait for it to recover.
If you use the service in a way that triggers a bug or causes performance issues, it’s very difficult to diagnose the problem. With software you run yourself, you can obtain performance metrics and debugging information from the business system to understand its behavior, and you can inspect server logs. But with vendor-hosted services, you typically don’t have access to this internal information.
Moreover, if the service shuts down or becomes unacceptably expensive, or if the vendor decides to change its product in a way you don’t like, you’re at their mercy — continuing to run an old version of the software is usually not an option, so you’ll be forced to migrate to another service. This risk can be mitigated if there are alternative services offering compatible APIs, but for many cloud services, there is no standard API, which increases switching costs and makes vendor lock-in a real problem.
Latency-critical applications such as high-frequency trading require complete control over hardware, making the cloud a poor choice for such businesses.

Cloud-Native
#

Category	Self-Hosted Systems	Cloud-Native Systems
Transactional/OLTP	MySQL, PostgreSQL, MongoDB	AWS Aurora, Azure SQL DB Hyperscale, Google Cloud Spanner
Analytical/OLAP	Teradata, ClickHouse, Spark	Snowflake, Google BigQuery, Azure Synapse Analytics

The key idea behind cloud-native services is not only to use computing resources managed by the business system but also to build on top of lower-level cloud services to create higher-level services. For example:

Object storage services like Amazon S3, Azure Blob Storage, and Cloudflare R2 store large files. They provide a more limited API than a typical filesystem (basic file reads and writes), but their advantage is hiding the underlying physical machines: the service automatically distributes data across many machines, so you don’t need to worry about running out of disk space on any single machine. Even if some machines or their disks fail entirely, no data is lost.
Many other services are in turn built on top of object storage and other cloud services: for example, Snowflake is a cloud-based analytical database (data warehouse) that relies on S3 for data storage, and some services are further built on top of Snowflake.

Cloud-native systems are typically multi-tenant, meaning they don’t provision separate machines for each customer. Instead, data and computation from several different customers are handled by the same service on shared hardware. Multi-tenancy enables better hardware utilization, easier scalability, and simpler management for cloud providers.

Operations in the Cloud Era
#

Traditionally, the people managing an organization’s server-side data infrastructure were called database administrators (DBAs) or system administrators (sysadmins). In recent years, many organizations have attempted to integrate software development and operations roles into a single team jointly responsible for backend services and data infrastructure; the DevOps philosophy has guided this trend. Site Reliability Engineers (SREs) represent Google’s implementation of this philosophy.

The DevOps/SRE philosophy emphasizes:

Automation — preferring repeatable processes over one-off manual tasks,
Preferring ephemeral virtual machines and services over long-running servers,
Promoting frequent application updates,
Learning from incidents,
Preserving organizational knowledge about systems even as individual personnel come and go.

The operations team at an infrastructure company focuses on the details of reliably delivering services to a large number of customers, while the customers of the service spend as little time and energy on infrastructure as possible. Beyond the traditional need for capacity planning, adopting cloud services may be easier and faster than running your own infrastructure. While the cloud is changing the role of operations, the need for operations remains urgent.

Ch2: Defining Non-Functional Requirements
#

Hardware and Software Faults
#

In large-scale systems, hardware faults happen frequently enough that they become part of normal system operation:

About 2-5% of disk hard drives fail each year; in a storage cluster with 10,000 disks, we can therefore expect on average one disk failure per day.
About 0.5-1% of solid-state drives (SSDs) fail each year. Uncorrectable errors occur about once per drive per year.
About one in 1,000 machines has a CPU core that occasionally computes incorrect results.
Data in RAM can also be corrupted, due to random events like cosmic rays or permanent physical defects. Additionally, certain pathological memory access patterns can flip bits with high probability.
Other hardware components such as power supplies, RAID controllers, and memory modules also fail.
An entire data center can become unavailable (e.g., due to power outages or network misconfiguration) or even permanently destroyed (e.g., fire or flood).

Software faults are often unpredictable and, because they are correlated across nodes, can cause more system failures than hardware faults:

A bug that causes all application server instances to crash upon receiving a specific bad input. For example, the leap second on June 30, 2012, caused many applications to hang simultaneously due to a bug in the Linux kernel.
A runaway process that exhausts some shared resource — CPU time, memory, disk space, or network bandwidth.
A service that the system depends on becomes slow, unresponsive, or starts returning incorrect responses.
Cascading failures, where a small fault in one component triggers a fault in another, which triggers further faults.

Operational configuration errors are the leading cause of service outages, while hardware faults (server or network) account for only 10-25% of service outages.

Scalability Principles
#

A good general principle for scalability is to decompose the system into small components that can operate relatively independently. This is the basic principle behind microservices. However, the challenge lies in knowing where to draw the line between things that belong together and things that should be separate.

If a single-machine database can do the job, it may be preferable to a complex distributed setup. A system with five services is simpler than one with fifty services. Good architecture often involves a mix of approaches.

Operations
#

An operations team is critical to keeping software systems running smoothly. The typical responsibilities of a good operations team include (and go beyond) the following:

Monitoring system health and quickly restoring service when it degrades.
Tracking down the causes of problems, such as system failures or performance degradation.
Keeping software and platforms up to date, including security patches.
Understanding interactions between systems to avoid damaging changes before they cause harm.
Anticipating future problems and addressing them before they occur (e.g., capacity planning).
Establishing good practices for deployment, configuration, and management, and writing supporting tools.
Performing complex maintenance tasks, such as migrating applications from one platform to another.
Maintaining system security during configuration changes.
Defining workflows to make operations predictable and maintain production environment stability.
Preserving organizational knowledge about systems as personnel come and go.

Good operability means easier day-to-day work, allowing the operations team to focus on high-value tasks. Data systems can make routine tasks easier in various ways:

Providing good monitoring with visibility into the system’s internal state and runtime behavior.
Offering good support for automation, integrating the system with standardized tools.
Avoiding dependence on a single machine (allowing machines to be taken down for maintenance while the overall system continues running uninterrupted).
Providing good documentation and an easy-to-understand operational model (“if you do X, Y will happen”).
Providing good default behavior but also allowing administrators to freely override defaults when needed.
Self-healing when possible, but also allowing administrators to manually control system state when needed.
Predictable behavior, minimizing surprises.

Some aspects of operations can and should be automated, but setting up correctly functioning automation in the first place still depends on humans.

Systems with too strong an individual stamp cannot succeed. When the initial design is complete and relatively stable, the real testing begins as different people test it in their own ways. — Donald Knuth

Ch3: Data Models and Query Languages
#

Most applications are built by layering one data model on top of another.

As an application developer, you observe the real world (with people, organizations, goods, actions, money flows, sensors, etc.) and model it in terms of objects or data structures and APIs that manipulate those data structures. These structures are typically specific to your application.
When you want to store these data structures, you express them in a general-purpose data model, such as JSON or XML documents, tables in a relational database, or vertices and edges in a graph. These data models are the subject of this chapter.
The engineers who build your database software decided on a way to represent that JSON/relational/graph data as bytes in memory, on disk, or on the network. This representation may allow the data to be queried, searched, manipulated, and processed in various ways. We’ll discuss these storage engine designs in a later chapter.
At an even lower level, hardware engineers have figured out how to represent bytes in terms of electric currents, light pulses, magnetic fields, and so on.

SQL & NoSQL
#

Databases can execute declarative queries in parallel across multiple CPU cores and machines, without you needing to worry about how to implement that parallelism. Implementing such parallel execution yourself in hand-coded algorithms would be an enormous undertaking.

The relational model, despite being half a century old, remains an important data model for many applications — especially in data warehousing and business analytics, where relational star or snowflake schemas and SQL queries are ubiquitous. However, in other domains, several alternatives to relational data have become popular:

The document model targets use cases where data comes in the form of self-contained JSON documents and relationships between documents are rare.
The graph data model goes in the opposite direction, targeting use cases where anything can be related to everything, and queries may need to traverse multiple hops to find data of interest (this can be expressed using recursive queries in Cypher, SPARQL, or Datalog).
The dataframe generalizes relational data into a large number of columns, building a bridge between databases and the multidimensional arrays that form the foundation of most machine learning, statistical data analysis, and scientific computing.

Databases also tend to expand into adjacent domains by adding support for other data models: for example, relational databases have added support for document data in the form of JSON columns, document databases have added relational-like joins, and support for graph data in SQL is gradually improving.

Ch4: Storage and Indexing
#

Hash Indexes
#

Key-value stores are quite similar to the dictionary type found in most programming languages, which is typically implemented using a hash map or hash table.

Generally, the hash map of a hash index is kept entirely in memory. Data values can use more space than available memory because the required portion can be loaded from disk with a single disk seek.

Drawbacks of hash indexes:

In principle, a hash map can be maintained on disk. Unfortunately, disk-based hash maps struggle to perform well. They require a large amount of random-access I/O, are expensive to grow when exhausted, and require tedious logic to resolve hash collisions.
Range queries are inefficient. For example, you can’t easily scan all keys between kitty00000 and kitty99999 — you must look up each key individually in the hash map.

B-Tree Indexes
#

B-tree indexes have been around since 1970 and are widely accepted and used in the industry. This section is familiar to most readers — skipped.

SSTables & LSM Trees
#

In hash indexes, the order of key-value pairs doesn’t matter. But we can require that the sequence of key-value pairs be sorted by key. This format is called a Sorted String Table, or SSTable.

Compared to log segments using hash indexes, SSTables have several major advantages:

Even if the file is larger than available memory, merging segments remains simple and efficient. The approach is like the one used in merge sort algorithms: you start reading multiple input files side by side, look at the first key in each file, copy the lowest key (according to the sort order) to the output file, and repeat. This produces a new merged segment file, also sorted by key.

To find a particular key in the file, you no longer need to keep an index of all keys in memory. You still need an in-memory index to tell you the offsets for some of the keys, but it can be sparse: one key per several kilobytes of segment file is sufficient, because several kilobytes can be scanned very quickly.

Using these data structures, you can insert keys in any order and read them back in sorted order. Now we can make our storage engine work as follows:

When a new write comes in, add it to an in-memory balanced tree data structure (e.g., a red-black tree). This in-memory tree is sometimes called a memtable.
When the memtable becomes larger than some threshold (typically a few megabytes), write it out to disk as an SSTable file. This can be done efficiently because the tree already maintains key-value pairs sorted by key. The new SSTable file becomes the most recent segment of the database. While that SSTable is being written to disk, new writes can continue on a new memtable instance.
When a read request comes in, first try to find the key in the memtable, then in the most recent on-disk segment, then in the next older segment, and so on.
From time to time, run a merging and compaction process in the background to combine segment files and discard overwritten or deleted values.

The algorithm described here is essentially the technique used by LevelDB and RocksDB, key-value storage engine libraries designed to be embedded in other applications. Similar storage engines are used in Cassandra and HBase, and all of them were inspired by Google’s Bigtable paper (which introduced the terms SSTable and memtable).

In-Memory Databases
#

In-memory databases: As RAM becomes cheaper, the argument that RAM costs more per GB is eroding. Many datasets are not that large, so keeping them entirely in memory is quite feasible, including potentially distributed across multiple machines. This has led to the development of in-memory databases. Losing data when restarting a computer may be acceptable. Durability can also be achieved through special hardware (e.g., battery-backed RAM), by writing a change log to disk, by periodically writing snapshots to disk, or by replicating the in-memory state to other machines.

The typical in-memory database Redis provides weak durability through asynchronous writes to disk. Other in-memory databases include Memcached, VoltDB, MemSQL, Oracle TimesTen, and RAMCloud.

Counterintuitively, the performance advantage of in-memory databases does not come from avoiding disk reads. Instead, they are faster because they avoid the overhead of encoding in-memory data structures into on-disk data structures.

Materialized Views and OLAP
#

Think of SQL functions like COUNT, SUM, AVG, MIN, or MAX. If the same aggregations are used by many different queries, it may be wasteful to process the raw data each time. Why not cache some of the most frequently used counts or sums? One way to create such a cache is a Materialized View.

When the underlying data changes, a materialized view needs to be updated because it is a denormalized copy of the data. The database can do this automatically, but such updates make writes more expensive, which is why materialized views are not commonly used in OLTP databases. In read-heavy data warehouses, they may make more sense because warehouses don’t have many small, frequent updates.

The advantage of a materialized data cube is that it can make certain queries extremely fast because they have already been effectively precomputed. For example, if you want to know the total sales per store, you just look at the total along the appropriate dimension without scanning millions of rows of raw data.

The disadvantage of a data cube is that it lacks the flexibility of querying raw data. For example, there is no way to compute what proportion of sales came from items costing over $100, because price is not one of the dimensions. Therefore, most data warehouses try to keep as much raw data as possible and use aggregate data (like data cubes) only as a performance boost for certain queries.

Column-Oriented Storage
#

The idea behind column-oriented storage is simple: instead of storing all the values from one row together, store all the values from each column together. Column-oriented storage is easiest to understand in the relational data model, but it applies equally to non-relational data. For example, Parquet is a column-oriented storage format that supports a document data model based on Google’s Dremel.

These optimizations (column compression, sorting, etc.) make sense in data warehouses, where the workload consists mainly of large read-only queries run by analysts. Column-oriented storage, compression, and sorting all help read those queries faster. However, their drawback is that writes become more difficult.

Ch5: Encoding and Evolution
#

REST vs. RPC
#

Servers expose APIs over the network, and clients can connect to servers to make requests to those APIs. The API exposed by a server is called a service. Download data via GET requests, submit data to the server via POST requests.

When a service uses HTTP as the underlying communication protocol, it can be called a web service. There are two popular approaches to web services: REST and SOAP. REST is not a protocol but a design philosophy based on HTTP principles. APIs designed according to REST principles are called RESTful.

Remote Procedure Calls (RPC) are very different from local function calls:

Local function calls are predictable and succeed or fail based only on parameters under your control. Network requests are unpredictable: requests or responses may be lost due to network problems, or the remote machine may be slow or unavailable.
A local function call either returns a result, throws an exception, or never returns (because it enters an infinite loop or the process crashes). A network request has another possible outcome: it may return with no result due to a timeout.
And so on.

REST seems to be the dominant style for public APIs, while RPC frameworks mainly focus on requests between services owned by the same organization, typically within the same data center.

Ch6: Replication
#

Replication logs, failover, single-leader mode — the content is relatively straightforward. Skipped.

Multi-Leader Replication
#

Multi-leader replication is often a retrofitted feature in many databases, so it frequently has subtle configuration pitfalls and often interacts unexpectedly with other database features. For example, auto-increment primary keys, triggers, and integrity constraints can all cause trouble. Therefore, multi-leader replication is often considered dangerous territory and should be avoided whenever possible.

However, multi-leader replication does have certain advantages, such as distributing write I/O, disaster recovery, and reducing network overhead in multi-region deployments (local writes), etc.

Write conflicts: The biggest problem with multi-leader replication is the potential for write conflicts, and resolving them is quite tricky. In principle, conflict detection could be made synchronous — i.e., wait for writes to be replicated to all replicas before telling the user the write succeeded. But this may defeat the purpose of multi-leader: if you want synchronous conflict detection, you might as well just use single-leader replication.

Resolving multi-leader write conflicts:

Avoid conflicts. For example, have the application control that users only edit their own data.
Converge to consistency:
- Last Write Wins (LWW). Write by timestamp — may result in data loss.
- Priority writes. Higher-priority writes win — may result in data loss.
- Extra code. Preserve conflict information and write custom conflict resolution code.

Real-time collaborative editing applications allow multiple people to edit a document simultaneously — Etherpad and Google Docs are mature examples. Databases are still very young in the area of multi-leader writes.

Multi-leader write conflicts in databases are mostly resolved or avoided at the application level. The following are relatively mature areas of write conflict research for reference:

Conflict-free Replicated Data Types (CRDTs) are data structures such as sets, maps, ordered lists, and counters that can be concurrently edited by multiple users and resolve conflicts automatically in a reasonable way. Some CRDTs have been implemented in Riak 2.0.
Mergeable Persistent Data Structures explicitly track history, similar to the Git version control system, and use three-way merge functions (whereas CRDTs use two-way merges).
Operational Transformation (OT) is the conflict resolution algorithm behind collaborative editing applications like Etherpad and Google Docs. It is designed specifically for concurrent editing of ordered lists, such as lists of characters that make up a text document.

Ch7: Partitioning
#

Range Partitioning and Hash Partitioning
#

The drawback of range partitioning is that certain access patterns can lead to hot spots. If the primary key is a timestamp, partitions correspond to time ranges, and all writes will go to the same partition (i.e., today’s partition), which may become overloaded with writes while other partitions sit idle. You can use something other than the timestamp as the first part of the primary key to scatter the hot spot, but the drawback is that range queries won’t benefit.

Hash partitioning can mitigate the risk of skew and hot spots. For the purpose of partitioning, the hash function doesn’t need to be a cryptographically strong algorithm. The drawback of hash partitioning is that by partitioning by key hash, we lose a great property of key-range partitioning: the ability to efficiently execute range queries.

Hash partitioning can help reduce hot spots. But it cannot eliminate them entirely. For example, on a social media site, a celebrity user with millions of followers doing something can trigger a storm. This event can cause a large number of writes to the same key (the key might be the celebrity’s user ID or the ID of the action being commented on). Hash strategies don’t help here, because the hash of two identical IDs is still the same. If a primary key is very hot, a simple workaround is to add a random number at the beginning or end of the primary key. Just a two-digit decimal random number can scatter the primary key into 100 different primary keys, thus stored in different partitions. In any case, it’s about scattering hot spots, and you need to consider side effects such as the impact on range queries.

Ch8: Transactions
#

ACID, BASE
#

ACID is actually a very old definition. Due to the later discovery of many “anomalies,” a system claiming to guarantee ACID can’t actually articulate what exactly it guarantees. Whatever the case, ACID remains deeply ingrained — it represents the most fundamental principles of transactions. Conversely, systems that don’t meet the ACID criteria are sometimes called BASE, which stands for Basically Available, Soft State, and Eventual Consistency. BASE is a concept commonly mentioned in the NoSQL world.

The definition of BASE is even fuzzier than ACID. A simple, easy-to-understand, easy-to-remember theory of BASE: BASE (which means “alkali” in chemistry) is the opposite of ACID (which means “acid”).

You can think of it simply this way:


Relational databases	Non-relational databases
Transactions	No transactions
ACID	BASE
SQL	NoSQL

Atomicity and isolation within ACID are relatively easy to understand. The concept of consistency is actually quite vague and doesn’t seem closely related to the database itself. A quote in the book is very classic:

Joe Hellerstein pointed out that in Härder and Reuter’s paper, “the C in ACID” was “tossed in to make the acronym work,” and at the time, nobody cared much about consistency.

And the definition of isolation is very fuzzy. The industrial practice of serializability has also been stagnant. Transaction isolation can be described as “a mess,” but if serializability is a panacea, why does no one use it? Refer to this article: The History of Transactions and SSI

Anomalies in non-serializable isolation levels generally only manifest under high concurrency; databases with low concurrency rarely encounter problems.
When anomalies do occur, some applications may not notice them or may detect them but find them unimportant.
Data may be anomalous, but the application may simply return an error and enter an anomaly-handling routine.
Cost is too high. Not only is the development cost of serializable isolation levels high for databases, but applications also need adaptation costs for serializability. Just understanding this complex theory is no easy task.
Higher isolation levels lose some performance. Massive rework may be thankless; applications need to choose between “high concurrency” and “no anomalies.”
Businesses develop based on mechanisms, not rules. Businesses have somewhat adapted to the anomalies of weaker isolation levels, especially Read Committed.

Summed up in one sentence: It’s not like it’s unusable!

Pessimistic and Optimistic Transaction Models
#

Two-phase locking is a so-called pessimistic concurrency control mechanism: it is based on the principle that if something might go wrong (e.g., another transaction holding a lock), it’s better to wait until the situation is safe before proceeding. It’s like a mutex used to protect data structures in multi-threaded programming.

In a sense, serial execution could be called the ultimate in pessimism: for the duration of each transaction, each transaction holds an exclusive lock on the entire database (or a partition of the database). As compensation for the pessimism, we make each transaction execute very fast, so the “lock” is only held for a short time.

In contrast, Serializable Snapshot Isolation is an optimistic concurrency control technique. In this context, optimistic means that if there is potential danger, the transaction is not blocked — instead, it continues executing, hoping everything will turn out fine. When a transaction wants to commit, the database checks whether anything bad happened (i.e., whether isolation was violated); if so, the transaction is aborted and must be retried. Only serializable transactions are allowed to commit. If there is a lot of contention (i.e., many transactions trying to access the same objects), performance suffers because a large proportion of transactions need to be aborted. If the system is already near maximum throughput, the additional load from retried transactions can worsen performance.

Ch9: Distributed Systems
#

Clocks
#

Clocks are critically important in distributed systems — they can directly affect the visibility, isolation, and correctness of transactions. In reality, reading a precise point in time is meaningless (from a quantum theory perspective, there is no concept of an absolute point in time; the actual situation is even more complex). Spanner’s Google TrueTime API reports a confidence interval for the local clock. The confidence interval reports an extremely short and trustworthy time range rather than a time point. For example, if you have two confidence intervals, each containing the earliest and latest possible timestamps ($A = [A_{earliest}, A_{latest}]$, $B=[B_{earliest}, B_{latest}]$), and these two intervals do not overlap (i.e., $A_{earliest} < A_{latest} < B_{earliest} < B_{latest}$), then B definitely happened after A — there is no doubt. Only when the intervals overlap are we uncertain about the order in which A and B occurred. To ensure that transaction timestamps reflect causality, Spanner deliberately waits for the length of the confidence interval before committing a read-write transaction. To keep the clock uncertainty as small as possible, Google deploys a GPS receiver or atomic clock in every data center, allowing clocks to be synchronized to within about 7 milliseconds. Logical clocks are based on incrementing counters rather than oscillating quartz crystals. Logical clocks only measure the relative ordering of events.

Real time may not exist. Responsiveness trumps everything. For most server-side data processing systems, real-time guarantees are uneconomical or unsuitable. Therefore, these systems must endure pauses and clock instability in non-real-time environments.

Ch10: Consistency and Consensus
#

All the problems we’ve assumed are possible: packets in the network can be lost, reordered, duplicated, or arbitrarily delayed; clocks are at best approximate; and nodes can pause (e.g., due to garbage collection) or crash at any time.

CAP
#

The formal definition of the CAP theorem is limited to a very narrow scope — it only considers one consistency model (linearizability) and one type of fault (network partitions, or nodes that are alive but disconnected from each other). It doesn’t discuss anything about network delays, dead nodes, or other trade-offs. Therefore, despite CAP’s historical influence, it has no practical value for designing systems.

Distributed Transactions and Consensus
#

All the consensus protocols discussed so far internally use a leader in some form, but they don’t guarantee that the leader is unique. Instead, they make a weaker guarantee: the protocol defines an epoch number (called ballot number in Paxos, view number in Viewstamped Replication, and term number in Raft) and ensures that within each epoch, the leader is unique. Whenever the current leader is thought to be dead, a vote begins among the nodes to elect a new leader. This election is assigned an incrementing epoch number, so epoch numbers are totally ordered and monotonically increasing. If there is a conflict between leaders from two different epochs (perhaps because the previous leader hadn’t actually died), the leader with the higher epoch number prevails. Designing algorithms that robustly cope with unreliable networks remains an open research problem.

Ch11: Batch Processing
#

Services (online systems) Services wait for requests or instructions from clients to arrive. Upon receiving one, the service attempts to process it as quickly as possible and sends back a response. Response time is typically the primary performance metric for services, and availability is usually very important (if clients can’t reach the service, users may receive error messages).
Batch processing systems (offline systems) A batch processing system takes a large amount of input data, runs a job to process it, and produces some output data. This often takes a while (from minutes to days), so typically no user is waiting for the job to finish. Instead, batch jobs typically run periodically (e.g., once a day). The primary performance metric for batch jobs is typically throughput (the time needed to process input of a certain size). This chapter discusses batch processing.
Stream processing systems (near-real-time systems) Stream processing sits between online and offline (batch) processing, so it is sometimes called near-real-time or nearline processing. Like batch processing systems, stream processing consumes inputs and produces outputs (without needing to respond to requests). However, stream jobs operate on events shortly after they occur, whereas batch jobs wait for a fixed set of input data. This difference gives stream processing systems lower latency compared to batch processing systems.

The batch processing algorithm MapReduce, published in 2004, was (perhaps over-enthusiastically) called “the algorithm that made Google’s massive scalability possible.” MapReduce is a fairly low-level programming model.

MapReduce and Distributed File Systems
#

Compared to the query optimizer of a relational database, Unix tools, despite their simplicity, are still remarkably useful. The biggest limitation of Unix tools is that they can only run on a single machine — this is where tools like Hadoop came in. MapReduce is somewhat like Unix tools but distributed across thousands of machines. Like Unix tools, it’s fairly crude but surprisingly effective. MapReduce jobs read and write files on a distributed file system. In Hadoop’s implementation of MapReduce, this file system is called HDFS (Hadoop Distributed File System), an open-source implementation of the Google File System (GFS). Besides HDFS, there are various other distributed file systems such as GlusterFS and the Quantcast File System (QFS). Object storage services like Amazon S3, Azure Blob Storage, and OpenStack Swift are similar in many ways.

To create a MapReduce job, you need to implement two callback functions, Mapper and Reducer, which behave as follows:

Mapper The Mapper is called once on each input record. Its job is to extract key-value pairs from the input record. For each input, it can generate any number of key-value pairs (including none). It does not retain any state from one input record to the next, so each record is processed independently.
Reducer The MapReduce framework takes the key-value pairs produced by the Mapper, collects all values belonging to the same key, and iteratively calls the Reducer over this set of values. The Reducer can produce output records (e.g., the count of occurrences of the same URL).

Using the MapReduce programming model, the physical network communication aspects of computation (getting data from the right machines) are separated from the application logic (processing the data after obtaining it). This separation contrasts sharply with the typical use of databases, where requests to fetch data from the database frequently appear within application code. Because MapReduce handles all network communication, it also frees application code from worrying about partial failures, such as the crash of another node: MapReduce can transparently retry failed tasks without affecting application logic.

Another common pattern of “putting related data together” is grouping records by some key (like the GROUP BY clause in SQL). The simplest way to implement this grouping operation with MapReduce is to set up the Mapper so that the key-value pairs it generates use the desired grouping key. The partitioning and sorting process then directs all records with the same partition key to the same Reducer.

Hadoop vs. Distributed Databases
#

As we’ve seen, Hadoop is somewhat like a distributed version of Unix, where HDFS is the file system and MapReduce is a peculiar implementation of Unix processes (always running the sort utility between the Map and Reduce phases). We’ve seen how various join and grouping operations can be implemented on top of these primitives.

When the MapReduce paper was published, it was — in a sense — not new. All the processing and parallel join algorithms we discussed in earlier sections had already been implemented over a decade earlier in so-called massively parallel processing (MPP) databases. Examples include the Gamma database machine, Teradata, and Tandem NonStop SQL, which were pioneers in this area.

The biggest difference is that MPP databases focus on executing analytical SQL queries in parallel across a set of machines, whereas the combination of MapReduce and a distributed file system is more like a general-purpose operating system that can run arbitrary programs.

Diversity of Processing Models
#

Having only two processing models, SQL and MapReduce, is not enough — more diverse models are needed! And due to the openness of the Hadoop platform, implementing a whole range of approaches is feasible, something that was impossible within the monolithic MPP database paradigm. Traditionally, MPP databases met the needs of business intelligence analytics and business reporting, but this is only one of many domains that use batch processing. In the years since MapReduce became popular, execution engines for distributed batch processing have matured significantly.

Ch12: Stream Processing
#

Skipped.

Event Sourcing
#

Event sourcing is a powerful data modeling technique: from the application’s perspective, it’s more meaningful to record user actions as immutable events rather than recording the effects of those actions in a mutable database. Event sourcing is similar to the chronicle data model. Like change data capture, event sourcing involves storing all changes to application state as a log of change events. Applications using event sourcing need to pull the event log (representing the data written to the system) and transform it into application state suitable for display to users. The current state is derived from the event log.

Ch13: The Future of Data Systems
#

Lambda Architecture
#

If batch processing is used to reprocess historical data and stream processing is used for recent updates, how do we combine the two? The Lambda Architecture is one proposal for this. The core idea of the Lambda Architecture is to record incoming data by appending immutable events to an ever-growing dataset, similar to event sourcing. In the Lambda approach, the stream processor consumes events and quickly produces an approximate update to the view; the batch processor later uses the same set of events and produces a corrected version of the derived view.

Unix evolved pipelines and files that are just byte sequences, while databases evolved SQL and transactions. Which approach is better? Of course, it depends on what you want. Unix is “simple” because it’s a fairly thin wrapper around hardware resources; relational databases are “simpler” because a short declarative query can leverage a lot of powerful infrastructure (query optimization, indexes, join methods, concurrency control, replication, etc.) without requiring the query author to understand the implementation details. I interpret the NoSQL movement as a desire to apply Unix-like low-level abstractions to the domain of distributed OLTP data storage.

Separation of Application Code and State
#

In theory, a database could be a deployment environment for arbitrary application code, much like an operating system. In practice, however, they are poorly suited to this goal. They don’t meet the requirements of modern application development, such as dependency and package management, version control, rolling upgrades, evolvability, monitoring, metrics, calls to network services, and integration with external systems. I believe it makes sense to have some parts of the system specialized for persistent data storage and other parts specialized for running application code. The two can interact while remaining independent. The trend is to separate stateless application logic from state management (databases): don’t put application logic into the database, and don’t put persistent state into the application.

I assert that in most applications, integrity is far more important than timeliness. Violating timeliness may be confusing and annoying, but violating integrity can be catastrophic.

Problems Introduced by Algorithms
#

Bias and discrimination: For example, in racially segregated areas, a person’s ZIP code, or even their IP address, is a strong indicator of race. Given this, it seems absurd to believe that an algorithm can somehow take biased data as input and produce fair and unbiased output. Yet this view often seems to lurk among advocates of data-driven decision-making — an attitude satirized as “machine learning is like money laundering for bias.” Predictive analytics systems simply extrapolate from the past; if the past was discriminatory, they codify that discrimination into rules.
Responsibility and accountability: Automated decision-making raises questions about responsibility and accountability. If a person makes a mistake, they can be held accountable, and those affected by the decision can appeal. Algorithms also make mistakes, but when they do, who is responsible?
Privacy and surveillance: Let’s do a thought experiment. Try replacing the word data with surveillance and see if common phrases still sound as nice. For example: “In our surveillance-driven organization, we collect real-time surveillance streams and store them in our surveillance warehouse. Our surveillance scientists use advanced analytics and surveillance processing to gain new insights.”

Blind faith in the supremacy of data-driven decisions is not just delusional — it’s genuinely dangerous. As data-driven decision-making becomes more prevalent, we need to figure out how to make algorithms more accountable and transparent, how to avoid reinforcing existing biases, and how to fix them when they inevitably err.

Users barely know what data they’re giving us, what data goes into the database, and how the data is retained and processed — most privacy policies are ambiguous, stringing users along without coming clean. If users don’t understand what will happen to their data, they can’t give any meaningful consent. For users who disagree with surveillance, the only truly viable alternative is simply not to use the service. But this choice isn’t truly free either: if a service is so popular that it is “considered a necessity for basic social participation by most,” then expecting people to opt out is unreasonable — using it is effectively mandatory.

Summary
#

Since software and data have such an enormous impact on the world, we engineers must remember that we have a responsibility to work toward the kind of world we want: a world that respects people, that respects humanity. I hope we can work together toward that goal.

PostgreSQL CLOG Files and Standby Synchronization Analysis

Tue, 03 Sep 2024 00:00:00 +0000

Among all relational databases, PostgreSQL’s CLOG is a very special type of log. CLOG’s existence is inseparable from PostgreSQL’s MVCC mechanism. Some basic knowledge about transaction IDs and CLOG won’t be covered in this article. If interested, please refer to CLOG and Hint Bits. This article focuses on the structure of CLOG files, manually locating transaction states, and the CLOG WAL log synchronization mechanism, to further understand PostgreSQL’s CLOG.

CLOG Segment
#

CLOG Directory
#

To distinguish from regular logs, PostgreSQL 10 renamed the CLOG and WAL directories ¹:

pg9.6	pg10
pg_clog	pg_xact
pg_xlog	pg_wal

Don’t get confused — I was also troubled by pg_xlog and pg_xact for a while…

CLOG Segment Name
#

CLOG is also managed by SLRU, and CLOG file naming is also in slru.c:

#define SlruFileName(ctl, path, seg) \
	snprintf(path, MAXPGPATH, "%s/%04X", (ctl)->Dir, seg)

%04X means hexadecimal (X), width of 4, zero-padded on the left (04). Example CLOG filenames:

[pg_xact]$ ll
-rw------- 1 postgres postgres 262144 Aug 15 16:29 03C0
-rw------- 1 postgres postgres 262144 Aug 19 23:04 03C1
...

TransactionID and CLOG Location Conversion
#

CLOG only stores transaction ID status, not the transaction ID itself. Through the TransactionID itself, you can directly locate the CLOG file and the position within the file. Before that, we need to understand some fundamentals.

Transaction States Stored in CLOG
#

There are only 4 transaction states:

typedef int XidStatus;

#define TRANSACTION_STATUS_IN_PROGRESS		0x00
#define TRANSACTION_STATUS_COMMITTED		0x01
#define TRANSACTION_STATUS_ABORTED		0x02
#define TRANSACTION_STATUS_SUB_COMMITTED	0x03

Transaction states are only: in progress, committed, aborted, subtransaction committed. Note that transaction IDs don’t have an “not started” state — as soon as a transaction ID is allocated in the database, that transaction has definitely already started. Conversely, transaction IDs not yet allocated in the database (actually a few — see the extend CLOG section below) correspond to in_progress status in CLOG. Four transaction states actually only need 2 bits to store. So 1 byte (8 bits) can store 4 transaction states, and 1 page (8k) can hold 8KB*4=32768 transaction states. These are all defined in the source code:

 * Defines for CLOG page sizes. A page is the same BLCKSZ as is used
 * everywhere else in Postgres.
// CLOG page size = BLCKSZ = 8k (default)
#define CLOG_BITS_PER_XACT	2 							 // One transaction state occupies 2 bits
#define CLOG_XACTS_PER_BYTE 4 							 // 1 byte can hold 4 transaction states
#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE) // 1 page can hold 32768 transaction states, 8KB*4=32768
#define CLOG_XACT_BITMASK ((1 << CLOG_BITS_PER_XACT) - 1) // Transaction status bitmask = ((1<<2)-1) = 3, expressed in binary as 11

#define SLRU_PAGES_PER_SEGMENT	32 // 1 segment has 32 pages

Summary:

1 CLOG segment has 32 pages
1 CLOG page is 8k (typically)
1 byte has 4 transaction states
1 transaction state occupies 2 bits

CLOG Segment/Page/Byte Conversion
#

Finding which CLOG segment a transaction ID corresponds to is not easy — it’s hidden in the comments:

 * Note: because TransactionIds are 32 bits and wrap around at 0xFFFFFFFF,
 * CLOG page numbering also wraps around at 0xFFFFFFFF/CLOG_XACTS_PER_PAGE,
 * and CLOG segment numbering at
 * 0xFFFFFFFF/CLOG_XACTS_PER_PAGE/SLRU_PAGES_PER_SEGMENT
// segment number = xid/CLOG_XACTS_PER_PAGE/SLRU_PAGES_PER_SEGMENT = xid/32768/32 // Which CLOG segment the transaction ID corresponds to, xid/32768/32, needs to be converted to hex

Mapping transaction ID to page, byte, etc. is clearer ²:

#define TransactionIdToPage(xid)	((xid) / (TransactionId) CLOG_XACTS_PER_PAGE) // Which CLOG page the transaction ID corresponds to, xid/32768
#define TransactionIdToPgIndex(xid) ((xid) % (TransactionId) CLOG_XACTS_PER_PAGE) // The offset within the above page, xid%32768
#define TransactionIdToByte(xid)	(TransactionIdToPgIndex(xid) / CLOG_XACTS_PER_BYTE) // Which byte in the page the transaction ID corresponds to, (xid%32768)/4
#define TransactionIdToBIndex(xid)	((xid) % (TransactionId) CLOG_XACTS_PER_BYTE)		// Which bit index in the above byte (note: bit index, not the bit itself), xid%4

Generally (with 8k BLCKSZ), 1 CLOG segment has 32 pages; 1 CLOG segment has 328k bytes, i.e., CLOG file size is fixed at 256K; 1 CLOG segment can hold 432*8k transaction states.

[pg_xact]$ ll # 256k CLOG segment
-rw------- 1 postgres postgres 262144 Aug 15 16:29 03C0
-rw------- 1 postgres postgres 262144 Aug 19 23:04 03C1
...

CLOG Bit Conversion
#

The functions for setting CLOG bits and getting CLOG bits (corresponding to TransactionIdSetStatusBit and TransactionIdGetStatus) both have the following code to obtain which two bits in the CLOG the transaction ID corresponds to:

	int			bshift = TransactionIdToBIndex(xid) * CLOG_BITS_PER_XACT;
	char	 *byteptr;
...
	byteptr = XactCtl->shared->page_buffer[slotno] + byteno;
	curval = (*byteptr >> bshift) & CLOG_XACT_BITMASK;

bshift represents the right-shift position, where TransactionIdToBIndex=xid%4, CLOG_BITS_PER_XACT=2, CLOG_XACT_BITMASK=3 (binary: 11). The key code for getting CLOG bits curval = (*byteptr >> bshift) & CLOG_XACT_BITMASK can be understood in two parts:

*byteptr >> bshift means right-shifting the pointer by 0, 2, 4, or 6 bits
& CLOG_XACT_BITMASK is simply taking the last two bits after the right shift (00&11=00, 01&11=01, 10&11=10, 11&11=11)

So, calculating the position of a transaction ID’s state within a byte:

xid%4=0: takes bits 7 and 8
xid%4=1: takes bits 5 and 6
xid%4=2: takes bits 3 and 4
xid%4=3: takes bits 1 and 2

Note: the transaction ID state’s bit positions within a byte are taken in reverse order, not sequentially forward. Byte and page positions are taken in sequential increasing order.

Manually Calculating Transaction ID Position in CLOG File
#

If we want to manually locate a transaction in CLOG using hexdump, we need to calculate three elements: <CLOG segment number, offset within segment in bytes, offset on byte in bit index>. (This references the approach in “PostgreSQL Database Kernel Analysis” but with some differences ³)

Before calculating, you also need to understand:

CLOG segment file numbers are in hexadecimal
hexdump is in hexadecimal, each line holds 16 bytes, i.e., each line holds 16*CLOG_XACTS_PER_BYTE=16*4=64 transaction states
hexdump -s xxx is in byte units

The following SQL can calculate the position of a transaction ID in CLOG:

-- CLOG segment number
-- %4294967296 represents transaction ID wraparound, /(8192*4*32) represents the maximum number of transactions a segment file can contain, to_hex converts to hex for filename, lpad left-pads to 4 digits
select lpad(upper(to_hex(txid_current()%4294967296/(8192*4*32))),4,'0') as clog_segmentno;

-- Offset within segment in bytes
-- %4294967296 represents transaction ID wraparound, %(8192*32*4) takes the remaining transaction IDs, /4 converts to byte units
select txid_current()%4294967296%(8192*32*4)/4 as in_clog_offset_bytes;

-- Offset on byte in bit index
-- %4294967296 represents transaction ID wraparound, %4 takes the bit index within the byte
select txid_current()%4294967296%4 as in_byte_offset_bitindex;


-- Or a single SQL
select lpad(upper(to_hex(txid_current()%4294967296/(8192*4*32))),4,'0') as clog_segmentno,txid_current()%4294967296%(8192*32*4)/4 as in_clog_offset_bytes,txid_current()%4294967296%4 as in_byte_offset_bitindex;

Practical simulation — computing a transaction ID’s state in CLOG:

begin;
 select lpad(upper(to_hex(txid_current()%4294967296/(8192*4*32))),4,'0') as clog_segmentno,txid_current()%4294967296%(8192*32*4)/4 as in_clog_offset_bytes,txid_current()%4294967296%4 as in_byte_offset_bitindex;
 clog_segmentno | in_clog_offset_bytes | in_byte_offset_bitindex 
----------------+----------------------+-------------------------
 0002 | 63196 | 3
rollback;
checkpoint; 

Rollback is used to roll back the transaction, mainly for easier observation, since most transactions are committed. Checkpoint is to ensure the CLOG page is flushed — otherwise the CLOG page might still be in the CLOG buffer and not yet written to the CLOG segment file.

cd pg_xact/
 hexdump -C 0002 -s 63196 -n 1 -v
0000f6dc 95 |.|
0000f6dd

-- Convert hex to binary
> select 'x96'::bit(8);
 bit 
----------
 10010110

When xid%4=3, take bits 1 and 2. So the bit value for this rolled-back transaction is 10, where 10 represents TRANSACTION_STATUS_ABORTED.

Why CLOG Usually Contains Many 55s and U’s?
#

In a typical transactional database CLOG file, a direct hexdump looks like this:

hexdump -C 0001 -v|head -10
00000000 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000010 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000020 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000030 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000040 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000050 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000060 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000070 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000080 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|
00000090 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 |UUUUUUUUUUUUUUUU|

Because the committed transaction state = 01 = TRANSACTION_STATUS_COMMITTED. When 4 consecutive transactions in a byte are all committed, it becomes 01010101.

Binary: 01010101, hex: 55
Hex 55 in ASCII is ‘U’, so when visually examining CLOG files you can generally see many U’s
Occasionally some bytes are not 55 or U because in production environments some transactions occasionally haven’t completed or use subtransactions. The committed state of subtransactions in CLOG is 0x03.

Shared CLOG Buffer
#

The number of CLOG shared buffers is easy to understand:

/*
 * Number of shared CLOG buffers.
 *
 * On larger multi-processor systems, it is possible to have many CLOG page
 * requests in flight at one time which could lead to disk access for CLOG
 * page if the required page is not found in memory. Testing revealed that we
 * can get the best performance by having 128 CLOG buffers, more than that it
 * doesn't improve performance.
 *
 * Unconditionally keeping the number of CLOG buffers to 128 did not seem like
 * a good idea, because it would increase the minimum amount of shared memory
 * required to start, which could be a problem for people running very small
 * configurations. The following formula seems to represent a reasonable
 * compromise: people with very low values for shared_buffers will get fewer
 * CLOG buffers as well, and everyone else will get 128.
 */
Size
CLOGShmemBuffers(void)
{
	return Min(128, Max(4, NBuffers / 512));
}

Translation: Testing has shown that 128 CLOG buffers provide the best performance — more than that doesn’t improve performance. However, because some database configurations are too small, 128 CLOG buffers seems a bit large, so it takes 1/512 of the shared_buffers count. In other words: Number of CLOG buffers = 1/512 shared_buffer, minimum is 4, maximum is 128. Note: these are all buffer counts, not sizes!

How large is a single buffer? CLOG buffer is managed by SLRU, and each SLRU page is 8k:

A page is the same BLCKSZ as is used everywhere

We can glimpse the size of shared CLOG buffer from the perspective of CLOG SLRU initialization:

/*
 * Initialization of shared memory for CLOG
 */
Size
CLOGShmemSize(void)
{
	return SimpleLruShmemSize(CLOGShmemBuffers(), CLOG_LSNS_PER_PAGE);
}

The passed CLOGShmemBuffers() is 4~128, and the passed CLOG_LSNS_PER_PAGE = 1024 bytes (with 8k pages). SimpleLruShmemSize initializes SLRU shared memory:

Size
SimpleLruShmemSize(int nslots, int nlsns)
{
	Size		sz;

	/* we assume nslots isn't so large as to risk overflow */
	sz = MAXALIGN(sizeof(SlruSharedData));
	sz += MAXALIGN(nslots * sizeof(char *));	/* page_buffer[] */
	sz += MAXALIGN(nslots * sizeof(SlruPageStatus));	/* page_status[] */
	sz += MAXALIGN(nslots * sizeof(bool));	/* page_dirty[] */
	sz += MAXALIGN(nslots * sizeof(int));	/* page_number[] */
	sz += MAXALIGN(nslots * sizeof(int));	/* page_lru_count[] */
	sz += MAXALIGN(nslots * sizeof(LWLockPadded));	/* buffer_locks[] */

	if (nlsns > 0)
		sz += MAXALIGN(nslots * nlsns * sizeof(XLogRecPtr));	/* group_lsn[] */

	return BUFFERALIGN(sz) + BLCKSZ * nslots;
}

SLRU uses some arrays to store SLRU metadata and control information. The sz size is all roughly data type * buffer count, and these are generally not very large. The main initialized memory is BLCKSZ * nslots, i.e., 8k * (4~128) = (32k~1M). So we can roughly estimate that the shared CLOG buffer size is around 1M.

CLOG WAL: Types, Writing, and Redo
#

When writing CLOG, is CLOG WAL log also written? If so, wouldn’t that mean lost CLOG could be restored by reapplying WAL logs to recover transaction states? Let’s explore the CLOG WAL writing and redo source code with these questions in mind.

Extend CLOG
#

ZeroCLOGPage writes WAL. ZeroCLOGPage(pageno, true) is actually only called by ExtendCLOG:

/*
 * Make sure that CLOG has room for a newly-allocated XID.
 *
 * NB: this is called while holding XidGenLock. We want it to be very fast
 * most of the time; even when it's not so fast, no actual I/O need happen
 * unless we're forced to write out a dirty clog or xlog page to make room
 * in shared memory.
 */
void
ExtendCLOG(TransactionId newestXact)
{
	int			pageno;

	/*
	 * No work except at first XID of a page. But beware: just after
	 * wraparound, the first XID of page zero is FirstNormalTransactionId.
	 */
	if (TransactionIdToPgIndex(newestXact) != 0 &&
		!TransactionIdEquals(newestXact, FirstNormalTransactionId))
		return;

	pageno = TransactionIdToPage(newestXact); // CLOG page number converted from TransactionId

	LWLockAcquire(XactSLRULock, LW_EXCLUSIVE);

	/* Zero the page and make an XLOG entry about it */
	ZeroCLOGPage(pageno, true);

	LWLockRelease(XactSLRULock);
}

ZeroCLOGPage mainly calls WriteZeroPageXlogRec:

/*
 * Write a ZEROPAGE xlog record
 */
static void
WriteZeroPageXlogRec(int pageno)
{
	XLogBeginInsert();
	XLogRegisterData((char *) (&pageno), sizeof(int));
	(void) XLogInsert(RM_CLOG_ID, CLOG_ZEROPAGE);
}

WriteZeroPageXlogRec is writing a WAL record, with type “RM_CLOG_ID, CLOG_ZEROPAGE”. Using waldump, you can view CLOG_ZEROPAGE. Its proportion is generally very small:

pg_waldump -z 000000010000056B00000018 --stat=record
Type N (%) Record size (%) FPI size (%) Combined size (%)
---- - --- ----------- --- -------- --- ------------- ---
...
CLOG/ZEROPAGE 1 ( 0.00) 30 ( 0.00) 0 ( 0.00) 30 ( 0.00)
...

Extending CLOG page is always in page units. In fact, at the end of a CLOG segment you can easily see 00s:

hexdump 03C2
0000000 5555 5555 5555 5555 5555 5555 5555 5555
*
001bb30 5555 5555 0055 0000 0000 0000 0000 0000
001bb40 0000 0000 0000 0000 0000 0000 0000 0000 
* ## The end of the CLOG file is all zeros
001c000

Truncate CLOG
#

Besides extending CLOG, there’s also truncating CLOG. Truncate CLOG is called during vacuum. When called, it writes a truncate CLOG WAL record and flushes the WAL record to disk:

/*
 * Remove all CLOG segments before the one holding the passed transaction ID
 *
 * Before removing any CLOG data, we must flush XLOG to disk, to ensure
 * that any recently-emitted FREEZE_PAGE records have reached disk; otherwise
 * a crash and restart might leave us with some unfrozen tuples referencing
 * removed CLOG data. We choose to emit a special TRUNCATE XLOG record too.
 * Replaying the deletion from XLOG is not critical, since the files could
 * just as well be removed later, but doing so prevents a long-running hot
 * standby server from acquiring an unreasonably bloated CLOG directory.
 *
 * Since CLOG segments hold a large number of transactions, the opportunity to
 * actually remove a segment is fairly rare, and so it seems best not to do
 * the XLOG flush unless we have confirmed that there is a removable segment.
 */
void
TruncateCLOG(TransactionId oldestXact, Oid oldestxid_datoid)
{
	int			cutoffPage;

	/*
	 * The cutoff point is the start of the segment containing oldestXact. We
	 * pass the *page* containing oldestXact to SimpleLruTruncate.
	 */
	// What's written to WAL is the CLOG position, which is the CLOG page number converted from oldestXact
	cutoffPage = TransactionIdToPage(oldestXact); 

.....
	/*
	 * Write XLOG record and flush XLOG to disk. We record the oldest xid
	 * we're keeping information about here so we can ensure that it's always
	 * ahead of clog truncation in case we crash, and so a standby finds out
	 * the new valid xid before the next checkpoint.
	 */
	// WriteTruncateXlogRec writes the corresponding WAL record and flushes it to disk
	WriteTruncateXlogRec(cutoffPage, oldestXact, oldestxid_datoid);
	
	// After WAL is written, actually execute the CLOG segment truncation
	/* Now we can remove the old CLOG segment(s) */
	SimpleLruTruncate(XactCtl, cutoffPage);
}

WriteTruncateXlogRec writes a WAL record with RMGR as RM_CLOG_ID and info as CLOG_TRUNCATE:

/*
 * Write a TRUNCATE xlog record
 *
 * We must flush the xlog record to disk before returning --- see notes
 * in TruncateCLOG().
 */
static void
WriteTruncateXlogRec(int pageno, TransactionId oldestXact, Oid oldestXactDb)
{
	XLogRecPtr	recptr;
	xl_clog_truncate xlrec;

	xlrec.pageno = pageno;
	xlrec.oldestXact = oldestXact;
	xlrec.oldestXactDb = oldestXactDb;

	XLogBeginInsert();
	XLogRegisterData((char *) (&xlrec), sizeof(xl_clog_truncate));
	recptr = XLogInsert(RM_CLOG_ID, CLOG_TRUNCATE);
	XLogFlush(recptr);
}

After generating CLOG WAL records, the redo recovery routine is also needed:

/*
 * CLOG resource manager's routines
 */
void
clog_redo(XLogReaderState *record)
{
...
	// When redo info type is CLOG_ZEROPAGE, place the read redo information in memory, then write to the CLOG page file
	if (info == CLOG_ZEROPAGE)
	{
		int			pageno;
		int			slotno;

		memcpy(&pageno, XLogRecGetData(record), sizeof(int));

		LWLockAcquire(XactSLRULock, LW_EXCLUSIVE);

		slotno = ZeroCLOGPage(pageno, false);
		SimpleLruWritePage(XactCtl, slotno); 
		Assert(!XactCtl->shared->page_dirty[slotno]);

		LWLockRelease(XactSLRULock);
	}
	// When redo info type is CLOG_TRUNCATE, place the read redo information in memory, confirm the page is deletable (write page if not), then truncate the segment
	else if (info == CLOG_TRUNCATE)
	{
		xl_clog_truncate xlrec;

		memcpy(&xlrec, XLogRecGetData(record), sizeof(xl_clog_truncate));

		/*
		 * During XLOG replay, latest_page_number isn't set up yet; insert a
		 * suitable value to bypass the sanity test in SimpleLruTruncate.
		 */
		XactCtl->shared->latest_page_number = xlrec.pageno;

		AdvanceOldestClogXid(xlrec.oldestXact);

		SimpleLruTruncate(XactCtl, xlrec.pageno);
	}
	else
		elog(PANIC, "clog_redo: unknown op code %u", info);
}

What the CLOG redo routine does:

When redo info type is CLOG_ZEROPAGE: finds a suitable slot (evict if necessary), performs writability checks based on the read redo information (actually the CLOG page number), then writes the page to the CLOG file
When redo info type is CLOG_TRUNCATE: based on the read redo information (actually the CLOG page number), confirms the page is deletable (write page if not available), then truncates the CLOG segment

CLOG Synchronization Summary
#

CLOG has only two types of WAL logs, neither containing transaction status information. They are only triggered when extending CLOG pages and truncating CLOG segments, and the written WAL record is just a CLOG page number. CLOG’s WAL log RMGR type has only one: RM_CLOG_ID. This type has only two info codes: CLOG_ZEROPAGE, CLOG_TRUNCATE.

/* XLOG stuff */
#define CLOG_ZEROPAGE 0x00
#define CLOG_TRUNCATE 0x10

CLOG WAL synchronization summary: The standby database is essentially not synchronizing CLOG information — it’s only synchronizing some CLOG file expansion and deletion information.

However, the standby’s CLOG file clearly does have status information, and the standby obviously needs this information for visibility checking. How is the transaction status in CLOG synchronized?

Transaction ID WAL: Types, Writing, and Redo
#

The WAL for rmgr=CLOG doesn’t contain transaction status. Does the standby not synchronize CLOG transaction information? No — WAL logs do contain transaction ID status information, and CLOG is also updated:

-- Roll back a transaction, commit a transaction
> begin;
BEGIN
> select txid_current();
 txid_current 
--------------
 1817254
(1 row)

> rollback;
ROLLBACK
> begin;
BEGIN
> select txid_current();
 txid_current 
--------------
 1817258
(1 row)

> commit;
COMMIT
> checkpoint;
CHECKPOINT

-- pg_waldump to view transaction ID status in logs
[datalzl/pg_wal]$ pg_waldump ../../pg_wal/000000010000007300000008|grep -E "1817254|1817258"
rmgr: Transaction len (rec/tot): 34/ 34, tx: 1817254, lsn: 73/400ED210, prev 73/400ED1E0, desc: ABORT 2024-08-01 14:41:26.017612 CST
rmgr: Transaction len (rec/tot): 46/ 46, tx: 1817258, lsn: 73/400EEB08, prev 73/400EEAD8, desc: COMMIT 2024-08-01 14:41:37.042545 CST
pg_waldump: fatal: error in WAL record at 73/400F7C78: invalid record length at 73/400F7F88: wanted 24, got 0

The WAL records the status of transaction IDs (1817254, 1817258), recorded as ABORT and COMMIT respectively; rmgr is Transaction. Transaction ID status is in WAL logs, but does PostgreSQL write it to the standby’s CLOG? Obviously, we need to find this redo information. Based on previous experience, clog_redo represents the WAL redo source code for rmgr=CLOG. Searching the source for _redo should find the WAL redo source code for rmgr=Transaction. Searching… in xact.c we find the function xact_redo, which mainly calls xact_redo_commit and xact_redo_abort, clearly corresponding to WAL log application logic for committed and rolled-back transactions respectively.

void
xact_redo(XLogReaderState *record)
{
	uint8		info = XLogRecGetInfo(record) & XLOG_XACT_OPMASK;

	/* Backup blocks are not used in xact records */
	Assert(!XLogRecHasAnyBlockRefs(record));

	if (info == XLOG_XACT_COMMIT)
	{
	...
		xact_redo_commit(&parsed, XLogRecGetXid(record),
						 record->EndRecPtr, XLogRecGetOrigin(record));
	}
...
	else if (info == XLOG_XACT_ABORT)
	{
	...
		xact_redo_abort(&parsed, XLogRecGetXid(record));
	}
...
	}
	else
		elog(PANIC, "xact_redo: unknown op code %u", info);
}

Taking commit as an example:

static void
xact_redo_commit(xl_xact_parsed_commit *parsed,
				 TransactionId xid,
				 XLogRecPtr lsn,
				 RepOriginId origin_id)
{
...

	if (standbyState == STANDBY_DISABLED)
	{
		/*
		 * Mark the transaction committed in pg_xact.
		 */
		TransactionIdCommitTree(xid, parsed->nsubxacts, parsed->subxacts);
	}
	else // standby logic
	{
	...
		/*
		 * Mark the transaction committed in pg_xact. We use async commit
		 * protocol during recovery to provide information on database
		 * consistency for when users try to set hint bits. It is important
		 * that we do not set hint bits until the minRecoveryPoint is past
		 * this commit record. This ensures that if we crash we don't see hint
		 * bits set on changes made by transactions that haven't yet
		 * recovered. It's unlikely but it's good to be safe.
		 */
		// Mark transaction committed in pg_xact
		TransactionIdAsyncCommitTree(xid, parsed->nsubxacts, parsed->subxacts, lsn);

...
}

It looks like TransactionIdAsyncCommitTree is the function we’re looking for that writes to CLOG.

To verify the redo logic for transaction commit information in WAL, let’s set three breakpoints on the standby’s startup process, then execute begin;select txid_current();commit; on the source database to commit a transaction, and see if the standby’s startup process hits the functions we want to see when doing redo:

(gdb) bt
#0 TransactionIdAsyncCommitTree (xid=xid@entry=1818665, nxids=0, xids=0x0, lsn=lsn@entry=495398394064) at transam.c:274
#1 0x000000000050c139 in xact_redo_commit (parsed=parsed@entry=0x7ffda52c0fc0, xid=1818665, lsn=495398394064, origin_id=<optimized out>) at xact.c:5805
#2 0x000000000050ffa3 in xact_redo (record=0x2b5ff2434038) at xact.c:5962
#3 0x0000000000519ea5 in StartupXLOG () at xlog.c:7411
#4 0x000000000072f301 in StartupProcessMain () at startup.c:204
#5 0x0000000000528701 in AuxiliaryProcessMain (argc=argc@entry=2, argv=argv@entry=0x7ffda52c6ef0) at bootstrap.c:450
#6 0x000000000072c459 in StartChildProcess (type=StartupProcess) at postmaster.c:5494
#7 0x000000000072ec44 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x2b5ff242d1c0) at postmaster.c:1407
#8 0x000000000048931f in main (argc=3, argv=0x2b5ff242d1c0) at main.c:210
(gdb) info b
Num Type Disp Enb Address What
1 breakpoint keep y 0x000000000050c060 in xact_redo_commit at xact.c:5753
 breakpoint already hit 43 times
2 breakpoint keep y 0x0000000000508190 in TransactionIdCommitTree at transam.c:262
3 breakpoint keep y 0x00000000005081a0 in TransactionIdAsyncCommitTree at transam.c:274
 breakpoint already hit 1 time

The breakpoint TransactionIdAsyncCommitTree is hit, and xid=1818665, which is the transaction ID just committed on the source database. This confirms the code logic we visually traced is correct. So, the standby database’s CLOG transaction ID status is synchronized by WAL with rmgr=Transaction.

Summary
#

CLOG only stores transaction ID status, not the transaction ID itself
Transaction status in CLOG files can be manually located via the transaction ID
WAL for rmgr=CLOG only extends and cleans up CLOG files, it does not update transaction status
WAL for rmgr=Transaction updates CLOG transaction status

References
#

“Quickly Mastering PostgreSQL Version New Features”, p24 ↩︎
Yan Shuli, PostgreSQL CLOG Analysis https://www.modb.pro/db/606433 ↩︎
“PostgreSQL Database Kernel Analysis”, Chapter 7, p380-390 ↩︎

PostgreSQL Case Study: Analysis of Abnormally Long Planning Time

Wed, 21 Aug 2024 00:00:00 +0000

Problem Analysis Overview
#

The database kept OOMing. Analysis revealed the issue was in query plan generation: planning time ~1 second, planning shared hits ~1 million. After thorough investigation, the root cause was identified as bloat in the statistics base table pg_statistic. On the first SQL execution of a session — due to a CatCacheMiss — the backend accessed and cached an excessive amount of dead tuple data from pg_statistic. Application connections always spawned new sessions, and the combined memory usage across multiple backends was too large, leading to OOM.

Below is the detailed analysis process.

Problem Symptoms
#

A certain database kept OOMing and restarting. After investigation, we found that while the number of concurrent sessions wasn’t high, each session’s memory footprint was quite large. The total memory exceeded the cgroup memory limit, causing OOM.

We could preliminarily rule out the following causes:

Not caused by excessive metadata. Too many objects (typically too many partitions) would cause sessions to cache excessive metadata. This database didn’t have that many objects.
Not caused by SQL execution plan issues. Sorting/hash operations might use too much memory. This database didn’t fit that scenario — the SQL in question was a simple sequential scan.

During the investigation, we discovered that any simple SQL query in this database took a very long time to execute, and Planning Buffers showed about 1 million hits:

 explain (analyze,buffers,timing) select * from lzlinfo limit 1;
 QUERY PLAN 
--------------------------------------------------------------------------------------------------------------------
 Limit (cost=0.00..1.02 rows=1 width=71) (actual time=0.011..0.012 rows=1 loops=1)
 Buffers: shared hit=1
 -> Seq Scan on lzlinfo (cost=0.00..480.73 rows=473 width=71) (actual time=0.010..0.010 rows=1 loops=1)
 Buffers: shared hit=1
 Planning:
 Buffers: shared hit=1127312 -- Abnormal planning shared hit
 Planning Time: 947.038 ms -- Abnormal planning time
 Execution Time: 0.035 ms
(8 rows)

Running the same SQL a second time, the planning time was normal.

Problem Investigation Process
#

Printing Execution Plan Statistics
#

We enabled logging for each phase of the execution plan:

set log_parser_stats =on;
set log_planner_stats =on;
set log_executor_stats =on;

Then ran the SQL. The log output was as follows:

2024-08-13 10:02:33.936 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,13,"idle",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"PARSER STATISTICS","! system usage stats:
! 0.000046 s user, 0.000046 s system, 0.000091 s elapsed
! [0.001661 s user, 0.001661 s system total]
! 4660 kB max resident size
! 0/0 [0/8] filesystem blocks in/out
! 0/36 [0/996] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/0 [5/0] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
2024-08-13 10:02:33.938 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,14,"EXPLAIN",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"PARSE ANALYSIS STATISTICS","! system usage stats:
! 0.001459 s user, 0.000000 s system, 0.001464 s elapsed
! [0.003146 s user, 0.001687 s system total]
! 5972 kB max resident size
! 0/0 [0/8] filesystem blocks in/out
! 0/325 [0/1324] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/0 [5/0] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
2024-08-13 10:02:33.938 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,15,"EXPLAIN",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"REWRITER STATISTICS","! system usage stats:
! 0.000001 s user, 0.000000 s system, 0.000001 s elapsed
! [0.003177 s user, 0.001687 s system total]
! 5972 kB max resident size
! 0/0 [0/8] filesystem blocks in/out
! 0/0 [0/1324] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/0 [5/0] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
2024-08-13 10:02:34.644 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,16,"EXPLAIN",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"PLANNER STATISTICS","! system usage stats:
! 0.539964 s user, 0.164083 s system, 0.705718 s elapsed
! [0.543248 s user, 0.165770 s system total]
! 745072 kB max resident size -- Abnormal point
! 0/0 [0/8] filesystem blocks in/out
! 0/184803 [0/186157] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/1 [5/1] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
2024-08-13 10:02:34.644 CST,"postgres","lzldb",85532,"[local]",66babe8c.14e1c,17,"EXPLAIN",2024-08-13 10:01:48 CST,4/713,0,LOG,00000,"EXECUTOR STATISTICS","! system usage stats:
! 0.540248 s user, 0.164170 s system, 0.706088 s elapsed
! [0.543532 s user, 0.165857 s system total]
! 745596 kB max resident size
! 0/0 [0/8] filesystem blocks in/out
! 0/184898 [0/186252] page faults/reclaims, 0 [0] swaps
! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
! 0/1 [5/1] voluntary/involuntary context switches",,,,,"explain (analyze,buffers) select *,1 from lzlinfo
"

During the planner phase, memory usage skyrocketed and elapsed time also spiked. This pinpointed the issue to the planner phase within the overall planning stage. There wasn’t much else actionable from the stats.

strace Tracing
#

strace -p 76419

strace: Process 76419 attached
epoll_wait(4, [{EPOLLIN, {u32=15422552, u64=15422552}}], 1, -1) = 1
recvfrom(9, "Q\0\0\0\262explain (analyze,buffers) s"..., 8192, 0, NULL, NULL) = 179
lseek(5, 0, SEEK_END) = 8192
brk(NULL) = 0xfed000
brk(0x100e000) = 0x100e000
brk(NULL) = 0x100e000
brk(NULL) = 0x100e000
brk(0x1007000) = 0x1007000
brk(NULL) = 0x1007000
mmap(NULL, 270336, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b7806b0c000
open("base/17076/16678", O_RDWR) = 7
lseek(7, 0, SEEK_END) = 0
open("base/17076/46160", O_RDWR) = 12
lseek(12, 0, SEEK_END) = 7667712
open("base/17076/46168", O_RDWR) = 13
lseek(13, 0, SEEK_END) = 188416
open("base/17076/46170", O_RDWR) = 14
lseek(14, 0, SEEK_END) = 188416
mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b78c1b36000
brk(NULL) = 0x1007000
brk(0x102c000) = 0x102c000
brk(NULL) = 0x102c000
brk(NULL) = 0x102c000
brk(0x1025000) = 0x1025000
brk(NULL) = 0x1025000
lseek(12, 0, SEEK_END) = 7667712
open("pg_stat_tmp/pgss_query_texts.stat", O_RDWR|O_CREAT, 0600) = 15
pwrite64(15, "explain (analyze,buffers) select"..., 172, 93934) = 172
pwrite64(15, "\0", 1, 94106) = 1
close(15) = 0
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\250\3\0\0\264B\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 936, 0, NULL, 0) = 936
sendto(8, "\2\0\0\0\10\1\0\0\264B\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 264, 0, NULL, 0) = 264
sendto(8, "\2\0\0\0\10\1\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 264, 0, NULL, 0) = 264
sendto(8, "\16\0\0\0H\0\0\0\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0"..., 72, 0, NULL, 0) = 72
sendto(9, "T\0\0\0#\0\1QUERY PLAN\0\0\0\0\0\0\0\0\0\0\31\377\377\377\377"..., 826, 0, NULL, 0) = 826
recvfrom(9, 0xd2b4e0, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(4, 

Although there were many shared hits, strace didn’t reveal much. strace showed the session only opened 4 data files. Using fd and oid2name to look up the data files, they turned out to be: the table, two indexes on the table, and pathman_config:

From database "lzldb":
 Filenode Table Name
--------------------------------------
 46170 ix_name
 46168 pk_lzlinfo
 46160 lzlinfo
 16678 pathman_config

These objects are not large, so it didn’t look like oversized tables (or indexes) were the cause.

perf
#

(No screenshot — use your imagination.)

The perf flame graph showed ~40% of the time spent on the heap_hot_search_buffer stack.

gdb
#

Using heap_hot_search_buffer as a clue, after multiple gdb sessions, we set the following breakpoints to investigate:

b relation_open
b get_relation_info
b RelationCacheInvalidateEntry 
b get_relname_relid
b AcceptInvalidationMessages
b RelationClearRelation
b pg_hint_plan_planner
b heap_hot_search_buffer

When breakpoints first hit, there was a lot of noise — they were normal logic. But later, after execution reached a certain point, only heap_hot_search_buffer kept hitting:

Breakpoint 15, heap_hot_search_buffer (tid=tid@entry=0x2313c60, relation=0x2b2141663910, buffer=17045, snapshot=snapshot@entry=0x228a058, heapTuple=heapTuple@entry=0x23273d0, 
 all_dead=all_dead@entry=0x7ffce272e28f, first_call=true) at heapam.c:1503
1503 in heapam.c
(gdb) 
Continuing.
...

Breakpoint 15, heap_hot_search_buffer (tid=tid@entry=0x2313c60, relation=0x2b2141663910, buffer=96708, snapshot=snapshot@entry=0x228a058, heapTuple=heapTuple@entry=0x23273d0, 
 all_dead=all_dead@entry=0x7ffce272e28f, first_call=true) at heapam.c:1503
1503 in heapam.c

Most arguments passed to heap_hot_search_buffer remained unchanged — including the addresses of relation and heapTuple — only the buffer parameter changed, indicating it was scanning the same relation.

heapTuple contained table OID information. Let’s print it:

(gdb) p *heapTuple
$46 = {
 t_len = 968, 
 t_self = {
 ip_blkid = {
 bi_hi = 0, 
 bi_lo = 7211
 }, 
 ip_posid = 5
 }, 
 t_tableOid = 2619, -- This is useful
 t_data = 0x2b2155fced00

heap_hot_search_buffer was called with OID=2619. Looking up 2619 in pg_class, it’s pg_statistic:

select oid,relname from pg_class where oid in (2619)
 oid | relname 
-------+----------------------------------
 2619 | pg_statistic

Accessing the statistics base table is expected — PG needs statistics to estimate costs when generating candidate execution plans.

pg_statistic Bloat
#

Now that we’ve pinpointed pg_statistic, let’s check its condition:

> \dt+ pg_statistic
 List of relations
 Schema | Name | Type | Owner | Persistence | Size | Description 
------------+--------------+-------+----------+-------------+---------+-------------
 pg_catalog | pg_statistic | table | postgres | permanent | 1036 MB | 


> select * from pg_class where relname='pg_statistic'\gx
-[ RECORD 1 ]-------+------------------------------------------------
oid | 2619
relname | pg_statistic
relnamespace | 11
reltype | 12016
reloftype | 0
relowner | 10
relam | 2
relfilenode | 2619
reltablespace | 0
relpages | 132481
reltuples | 4655

pg_statistic is 1GB — certainly oversized. 132,481 blocks but only 4,655 rows — this is clearly table bloat. But even with bloat, does accessing statistics really require caching the entire pg_statistic table? Logically, no — you only need the statistics for the specific table. And indeed, PG accesses pg_statistic through its primary key index pg_statistic_relid_att_inh_index. From the call stack below, we can see the composite primary key fields being passed:

bt
...
#6 0x000000000086edbc in SearchCatCacheMiss (cache=cache@entry=0x226ba80, nkeys=nkeys@entry=3, hashValue=hashValue@entry=853716409, hashIndex=hashIndex@entry=57, v1=v1@entry=18767, v2=v2@entry=1, 
 v3=v3@entry=0, v4=v4@entry=0) at catcache.c:1368
#7 0x000000000086fa82 in SearchCatCacheInternal (v4=0, v3=<optimized out>, v2=<optimized out>, v1=<optimized out>, nkeys=3, cache=0x226ba80) at catcache.c:1299
#8 SearchCatCache3 (cache=0x226ba80, v1=v1@entry=18767, v2=v2@entry=1, v3=v3@entry=0) at catcache.c:1183
#9 0x0000000000880d70 in SearchSysCache3 (cacheId=cacheId@entry=58, key1=key1@entry=18767, key2=key2@entry=1, key3=key3@entry=0) at syscache.c:1145
#10 0x0000000000874092 in get_attavgwidth (relid=relid@entry=18767, attnum=1) at lsyscache.c:2991
#11 0x00000000006a2d46 in set_rel_width (root=root@entry=0x2326600, rel=rel@entry=0x21e8418) at costsize.c:5516 
...

The call passes relid=relid@entry=18767, attnum=1:

 select ctid,starelid,staattnum from pg_statistic where starelid=18767;
 ctid | starelid | staattnum 
------------+----------+-----------
 (132657,6) | 18767 | 1
 (132657,7) | 18767 | 2
 (132657,8) | 18767 | 3
 (132657,9) | 18767 | 4
 (132658,1) | 18767 | 5
 (132658,2) | 18767 | 6
 (132658,3) | 18767 | 7
 (132658,4) | 18767 | 8
 (132658,5) | 18767 | 9
 (132658,6) | 18767 | 10
 -- lzlinfo has 10 columns total, each with a staattnum entry

From the ctid, we can see this data actually lives in just 2 blocks.

Now let’s access pg_statistic via the composite primary key index. Even with data in only 2 blocks, it took 1 second to access with ~1 million (1,141,568) shared hits:

 explain (analyze,buffers,timing,verbose) select ctid,starelid from pg_statistic where starelid=18767;
 QUERY PLAN 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Index Scan using pg_statistic_relid_att_inh_index on pg_catalog.pg_statistic (cost=0.41..103.31 rows=23 width=10) (actual time=105.416..1035.723 rows=10 loops=1)
 Output: ctid, starelid
 Index Cond: (pg_statistic.starelid = '18767'::oid)
 Buffers: shared hit=1141568 -- Abnormal
 Planning:
 Buffers: shared hit=8
 Planning Time: 0.102 ms
 Execution Time: 1035.802 ms

Accessing 10 rows in pg_statistic via the index resulted in ~1M shared hits — roughly matching the ~1M planning shared hits from the original SQL. (Note: Planning Time here is minimal, meaning the issue is not in plan generation per se, but in the data access during planning.)

Index Dead Tuples
#

If vacuum hasn’t truly “run properly”, index dead tuples still point to dead heap tuples.

Refer to: From Very Slow Unique Index Scans to Index Bloat

autovacuum Not Reclaiming Dead Tuples
#

With such severe table bloat, shouldn’t autovacuum have reclaimed it?

select * from pg_stat_all_tables where relname='pg_statistic'\gx
-[ RECORD 1 ]-------+------------------------------
relid | 2619
schemaname | pg_catalog
relname | pg_statistic
seq_scan | 1 	 -- Very few sequential scans on pg_statistic
seq_tup_read | 4655
idx_scan | 28715508 -- Many index scans on pg_statistic
idx_tup_fetch | 25150245
n_tup_ins | 46
n_tup_upd | 1292143 -- Lots of updates
n_tup_del | 14
n_tup_hot_upd | 138448
n_live_tup | 4655
n_dead_tup | 1496776
n_mod_since_analyze | 1292203
n_ins_since_vacuum | 0
last_vacuum | [null]
last_autovacuum | 2024-08-16 20:34:15.045022+08 -- Note: autovacuum timestamp is recent
last_analyze | [null]
last_autoanalyze | [null]
vacuum_count | 0
autovacuum_count | 144170
analyze_count | 0
autoanalyze_count | 0

Actually, autovacuum was constantly running on pg_statistic, but the worker process may not have been visible because it finished quickly (having nothing to actually reclaim) and went back to naptime:

show autovacuum_naptime ;
 autovacuum_naptime 
--------------------
 1min

It naps every 1 minute, and the logs show autovacuum info printed every 1 minute as well:

2024-08-16 21:05:15.267 CST,,,41080,,66bf4e87.a078,1,,2024-08-16 21:05:11 CST,27/166839,0,LOG,00000,"automatic vacuum of table ""lzldb.pg_catalog.pg_statistic"": index scans: 0
pages: 0 removed, 132685 remain, 1 skipped due to pins, 0 skipped frozen
tuples: 0 removed, 1501745 remain, 1497090 are dead but not yet removable, oldest xmin: 119329380
buffer usage: 265443 hits, 0 misses, 0 dirtied
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s
system usage: CPU: user: 0.53 s, system: 0.17 s, elapsed: 3.38 s
WAL usage: 1 records, 0 full page images, 233 bytes",,,,,,,,,"","autovacuum worker"
2024-08-16 21:05:17.474 CST,,,41080,,66bf4e87.a078,2,,2024-08-16 21:05:11 CST,27/166844,136438968,LOG,00000,"automatic analyze of table ""lzldb.public.lzlinfo"" system usage: CPU: user: 2.02 s, system: 0.00 s, elapsed: 2.08 s",,,,,,,,,"","autovacuum worker"
"

1497090 are dead but not yet removable — although autovacuum was triggered, it didn’t reclaim any dead tuples at all. 1,497,090 dead tuples remained uncleaned.

Investigating who held oldest xmin: 119329380, we quickly identified a replication slot:

select * from pg_replication_slots;
 slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn | wal_status | safe_wal_size 
-----------------+----------+-----------+--------+----------+-----------+--------+------------+--------+--------------+--------------+---------------------+------------+---------------
 slotslotlostname | pgoutput | logical | 17076 | lzldb | f | f | [null] | [null] | 119329380 | 3F9/105A4970 | 3F9/105F8778 | extended | [null]

The slot’s catalog_xmin=119329380 matched the vacuum’s oldest xmin: 119329380.

active=f indicated that the replication link was already broken.

Fixing the Problem
#

Drop the replication slot:

select pg_drop_replication_slot('slotslotlostname');
 pg_drop_replication_slot 
--------------------------

Then manually vacuum or wait 1 minute for autovacuum.

Finally, open a brand-new session to verify the fix:

## psql
psql (13.2)
Type "help" for help.

> \c lzldb
You are now connected to database "lzldb" as user "postgres".
> explain (analyze,buffers,timing) select * from lzlinfo limit 1;
 QUERY PLAN 
---------------------------------------------------------------------------------------------------------------------
 Limit (cost=0.00..8.04 rows=1 width=71) (actual time=0.023..0.025 rows=1 loops=1)
 Buffers: shared hit=1
 -> Seq Scan on lzlinfo (cost=0.00..3802.73 rows=473 width=71) (actual time=0.018..0.018 rows=1 loops=1)
 Buffers: shared hit=1
 Planning:
 Buffers: shared hit=2578
 Planning Time: 9.605 ms
 Execution Time: 0.098 ms

Planning time dropped from ~1 second to ~10 ms, and planning shared hits dropped from ~1M to ~2K. The problem was basically resolved.

Case Summary
#

The replication link broke and the replication slot wasn’t cleaned up in time, leading to bloat in the pg_statistic statistics base table. This caused each backend to be very slow when loading statistics for the first time and to read excessive pages into its local cache. Each backend’s cache exceeded normal levels (~2GB), and with multiple backends this led to OOM.

The problem itself is simple — it was just the investigation that was convoluted. In short: bloat in the base table pg_statistic caused excessive data access during the plan generation phase. Metadata base table bloat can cause other tricky problems too — until next time.

Book Notes — Average Is Over

Tue, 13 Aug 2024 00:00:00 +0000

Why I Read This Book
#

In the final pages of Elon Musk, the author briefly introduced two books by economist Tyler Cowen: The Great Stagnation and Average Is Over. The Great Stagnation is about why America’s development has stalled over the past 40 years — something I’m naturally not that interested in. But Average Is Over is not a study of history; it’s a perspective on future development, especially the impact of AI on human life.

I’ve always been interested in what human life will look like in the future. Recently, OpenAI has been hot, and it feels like the AI era is upon us. What changes will AI bring to our lives and work? Will social structures shift? Which jobs will gradually disappear? Which jobs will benefit?

Chess
#

The book spends a large portion (nearly half) discussing chess and computer programs. You can tell the author is definitely a chess enthusiast — he’s deeply knowledgeable about chess history and its evolution. Reading this section always reminds me of The Queen’s Gambit. If it weren’t for that show, I wouldn’t have known chess had rapid formats or that the Soviet Union was the world’s strongest chess nation. The author also uses chess to explore the influence of computer programs on the game.

This influence goes beyond AlphaGo defeating the world’s strongest human Go player — the “beating the brightest human minds” kind of impact. It also includes how early chess programs changed the way humans learn chess. In the early days of chess, before computers took off, people could only learn chess from other people. A beginner couldn’t often play against a chess master. But as computer programs became widespread, they were adopted en masse. Chess programs could teach you, you could play against them, and you could even set the difficulty level. This was incredibly convenient for beginners. Without us even noticing, computer programs quietly reshaped our lives. In the future, we will increasingly collaborate with AI.

Polarization
#

Once AI is widely deployed, many aspects of our lives will change. AI is unlikely to revolutionarily overturn the social structure of rich and poor; the reality that a tiny minority controls the vast majority of wealth may intensify further. The middle class is perhaps the most vulnerable stratum. Many middle-class workers perform partially intellectual but repetitive work — exactly AI’s sweet spot. The book argues that the value of middle-class work isn’t actually that great and may be relatively easily replaced. Disparities in basic assets will widen the gap in wealth accumulation — in other words, differences in starting capital will amplify differences in asset accumulation. In this age, that sentence is easy to understand.

The book approaches wealth distribution from an American perspective, but it’s easy to map onto the Chinese context. China’s economic development over recent decades has been truly remarkable — the dividends of population and infrastructure construction, a phase all developed nations went through. But the introduction of market economics and the passage of time have been accompanied by growing wealth inequality. Let’s leave it there… I don’t want to write anything too sensitive…

The Rising Cost of Learning
#

The cost of learning keeps rising. This doesn’t refer to the cost of tuition or training courses, but the difficulty of learning or mastering a profession. The word “inventor” — I suspect many people haven’t heard it in a long time. Our impression of the term is still stuck in the Edison era. Back then, individuals could invent things on their own; they just needed some relatively advanced knowledge in their field and a bit of brainpower. “Inventing” didn’t seem that hard. But as time passed, we rarely hear the word “inventor” anymore. It’s not that humans have stopped inventing — it’s that what people invent now is almost always the work of a team, many people, often requiring cross-disciplinary collaboration among multiple specialists. The cost of “inventing” things keeps rising because the knowledge required to master a field grows ever larger and more complex. It’s unrealistic for one person to master an entire industry; people tend to specialize in narrower domains — and even a narrow domain is enough for a lifetime of study.

Academia today faces this exact situation. A relatively successful paper typically requires experts from various fields to use their specialized knowledge to verify the correctness of one small segment of a proof. The book gives a classic example: if a mathematician proves a conjecture in mathematics, there may be only a handful of people in the entire world who can truly understand what the mathematician is proving. Most of them may only understand one section of the content — and even the mathematician themselves may only say: “I might be right.” We have no way to verify the correctness of the proof.

Human knowledge is becoming increasingly complex. Scientists now tend to, and increasingly do, hand calculations and experiments over to machines. Humanity seems to have reached a tipping point: our brains are nearly incapable of understanding this knowledge anymore. From a biological perspective, the human brain necessarily has a limit. The processing speed of the human brain can’t remotely keep up with machines.

Self-Learning
#

Even as learning costs rise, education will become ever more important in the future. The education system may change. Since time will be more precious in the future world and learning resources will be easier to access, people will lean toward online learning and self-directed learning. At the same time, this makes self-drive even more critical.

As an IT professional, I have a deep appreciation for self-learning. This industry is intensely competitive; if you don’t keep learning, you’re basically on the brink of obsolescence. But highly effective learning is also reflected in your salary. Our parents’ generation relied on assigned jobs and could work in one position for decades without major changes. People back then just thought about working, not obsessively self-improving and chasing certifications. Times have truly changed. How many people, like me, are still writing articles at 11 PM? I’m even baffled by industries where you don’t need to keep learning after graduation — just how backward are they? You graduate university in your early twenties and still have decades to learn. It would be utterly strange to just stagnate there. Of course, I don’t like cutthroat competition, but I like standing still even less — especially in an age where just sitting on a stool spacing out causes the wealth gap to widen.

Finally
#

The cover and illustrations for this post are all AI-generated. I just typed in “goodbye age of mediocrity” and the AI produced astonishing images. I don’t know exactly which industries or professions will disappear in the future, but at the very least, illustrators are going to have a hard time surviving in the AI era.

AI has already invaded the IT domain. As a DBA, which of our work patterns will be replaced? That’s a question worth pondering. Whatever happens, in this age, only learning can keep you competitive. I hope none of us will be the “disappearing shoulder pole porter.”

Book Notes — Educated and Atomic Habits

Tue, 13 Aug 2024 00:00:00 +0000

I’ve actually wanted to write about these two books for a long time. I love reading, but I absolutely detest writing. Maintaining a blog is practically a miracle for me. I love reading because I love and believe in the power of education. As for how much I hate writing, let me tell you a little story.

I Hate Writing Essays
#

My dislike for writing is practically innate. Since elementary school, I never wrote diaries or essays. Every winter and summer break homework required a daily diary entry — I never wrote a single one. I still remember when school started and I had to turn in summer homework. The teacher threatened that if I didn’t finish it, I couldn’t attend class. I still wrote nothing and just sat in the classroom as usual. Later, for some assignment, our homeroom teacher had to submit examples of correcting typically flawed students — only 4 or 5 kids in the class were selected. One was bad at sports, one had a temper problem… and I was the one bad at writing! And the task was to write an essay about correcting that flaw! I can’t write — why would you make me write an essay about fixing my inability to write??? I dragged it out for two weeks. All the other students turned theirs in. I couldn’t squeeze out a single word. The homeroom teacher personally coached me. She said: when you walk down the street, you can turn anything you see into a sentence. See a blue sky? You can form a sentence in your mind: “The sky is cloudless for miles.” Practicing sentence construction regularly can help with writing. You could also think of other ways to fix your writing aversion. Another week passed, and I wrote down exactly what she told me, verbatim. I could see the frustration in her eyes. Later, in middle school, I cleverly befriended the Chinese class representative so she’d leave my name off the missing-homework list. That’s how I dodged three years of middle school. Then in high school, I once awkwardly wrote an essay my own way and scored 30 out of 60 — a devastating blow. So for every monthly exam, I simply didn’t write the essay. I figured: it’s just mock exams, not the Gaokao — I’ll just forfeit those 60 points. Finally, for the actual Gaokao and the two mock exams before it, I crammed Qu Yuan and Li Bai into essay templates like eight-legged essays. I found there was nothing you couldn’t cram them into, and I muddled through the Gaokao essay hurdle. College? No need to mention it — my hand had forgotten how to hold a pen.

Yes, with this peculiar writing psychology, I hated essays. But after entering the workforce, I gradually understood: the dullest pencil is better than the sharpest memory. No matter how many books you read, you need to internalize them. The pressures of ambition, family, and work forced me to change. Whether I needed this skill or not didn’t seem to matter — if society needs it, I should try to adapt. Writing not only pushes you forward, it’s also a way to record growth, to record life. Even my own technical articles — years later, I still have to come back and read them carefully, review them carefully.

Reading Originals Is Good for Body and Mind
#

I read both Educated and Atomic Habits in their original English. I had a bit of an English foundation, and since I was preparing for graduate school entrance exams at the time, I wanted to improve my English reading — so I chose English originals. At first, reading English originals was quite difficult. Many words were unfamiliar, and I’d look them up and annotate them in the book. Progress was painfully slow. But as I read deeper, there were fewer and fewer annotations. It wasn’t that I quickly memorized many new words — rather, some important words appear repeatedly throughout a book, while others that appear rarely don’t affect comprehension. Also, at the start you don’t know what the book is about, so comprehension latency is high. Later, once you know where it’s headed, reading naturally speeds up. For instance, the word “ridge” appeared very frequently early on, and I eventually remembered it. Some similar words I still can’t remember, but I know they’re some kind of geographic term — summit, valley, ridge — and even without remembering them precisely, it doesn’t stop me from reading. That’s how English originals work: difficult at first, faster the further you go.

Educated
#

The Chinese title of Educated is You Should Fly Like a Bird to Your Mountain — I really want to complain about this title. “Educated” is the spiritual essence of the entire book, and near the end, the author drives it home with “call it educated” — absolutely brilliant. This Chinese title is like shit, completely missing the book’s essence. The author’s personal experience is legendary: a child who walked out of some corner of the American mountains, who through sheer effort studied all the way to Cambridge. Her father was uneducated, anti-social, lacking basic physics knowledge — his ignorance led to family members getting injured or even disabled. He disapproved of children going to school, even believing education was government brainwashing. Countless absurd behaviors. Her brother also had personality issues — he shoved her head into a filthy toilet and made her beg for mercy, then the next day acted like nothing happened and continued being her “good brother”… Later, the author found her way out through education, and in the end, she didn’t want to return to that valley. Reading the ending always reminds me of my own experience. Of course, I didn’t have such an extreme environment, nor such a legendary journey, but I feel like I can understand — after being educated, family interactions somehow feel unnatural. It’s not about getting cocky after university — the generation gap is real. I deeply believe in the importance of education. If my family hadn’t sold everything they had to fully support my education, our circumstances would never have changed. If you’ve truly been mired in poverty, you know how fierce the desire to escape it is — and education is almost the only way out for people like us. Educated is a great book: clear prose, comfortable sentence structure, suited to modern reading rhythms, a gripping story, a profound theme. It’s an excellent choice as your first English original.

Atomic Habits
#

Atomic Habits — I’ve forgotten exactly how I found this book, but it changed my understanding of behavior. Building good habits isn’t actually that hard; most people just don’t know how. Many have said: I’ll read X books in a few months, run Y kilometers, lose Z pounds — but they rarely follow through. Building good habits requires genuinely liking the habit, changing your mindset, reducing the friction of the action, putting obstacles farther away, forming reward mechanisms, and so on. When you want to become a certain kind of person, don’t focus on how to become that person — think about what that kind of person does, and learn to do it. For example, quitting smoking: if your brain thinks you’re “in the process of quitting,” it’s very hard. If someone offers you a cigarette and you say “I’m quitting,” a few words from them might get you to smoke. But if you genuinely believe you’re someone who “doesn’t smoke” — note, this must be your authentic inner belief — when someone offers you a cigarette, you’ll simply say “I don’t smoke,” and you probably won’t have to smoke it. Some small details: say you want to build a habit of reading at night — you need to break the habit of scrolling on your phone. Move your books from the bookshelf to your bedside for easier access. Put your phone at the foot of the bed, making getting up the barrier to grabbing the phone — this makes it easier to reach for the book instead of the phone. If picking up the book is still hard, reframe your thinking: “reading” as an action may feel difficult, but break it down — “pick up the book” or “open to the first page” becomes your mental target. The startup action for reading is simple and easy to complete. After reading the first page, think about what comes next — and in reality, once you’ve read the first page, it’s hard not to read the second. Of course, there are many more excellent suggestions for building good habits and shedding bad ones — every word is a gem, thoroughly engaging. After reading Atomic Habits, whenever I want a certain habit, I first consider the book’s guidance, then plan how to implement it — rather than acting on impulse.

Finally
#

At last — these two books have had an enormous impact on me. One is a legendary autobiography; the other is a behavior-transforming book. Neither is the kind of work you forget shortly after reading. They’re perfect starter books for cultivating a reading habit, especially for those wanting to read English originals. I really don’t recommend Pride and Prejudice or One Hundred Years of Solitude — yes, they’re classics, but their impact on the reader is quite low, and they were written so long ago that some vocabulary and grammar are too archaic, making them unsuitable for first-time English readers. Looking at this through the lens of Atomic Habits: reading these English classics is not only more difficult but also lacks immediate personal benefit, making it hard to form a habit.

Book Notes — Elon Musk

Tue, 13 Aug 2024 00:00:00 +0000

Gifted
#

Musk’s ancestors, driven by a love of adventure, emigrated from America to South Africa. His maternal grandfather even flew a plane from Africa to Australia. Musk was born in South Africa and showed astonishing memory and brilliance from an early age. His mother, Maye Musk, told his teacher: “My son is a genius.” The teacher replied, “Yes, every mother says that.” Maye: “No, I mean he really is a genius.” As a child, Musk sometimes seemed “slow to react.” His mother said when people talked to him, he’d give no response at all. She thought something was wrong with his brain and even took him to a doctor. But later she discovered Musk was simply immersed in his own world of thought. As a child, Musk could even finish reading the entire library’s collection and then ask the library to get more books…

Arriving in America
#

Due to the less-than-ideal environment in South Africa, Musk, approaching university age, executed a two-step jump. He first went to university in Canada, then to the United States for his master’s. Upon finally reaching America, Musk immersed himself in Silicon Valley’s work environment. The tech industry desperately needed young people like him — brilliant and relentless. And Silicon Valley’s tech atmosphere and culture of freely exercising one’s talents let Musk dive in completely.

Zip2 and PayPal
#

Soon Musk founded Zip2, essentially a corporate version of online maps. While we’re now very familiar with online maps, the US internet industry was just getting started back then — this was all novel stuff. After many twists and turns, Zip2 did grow. Personally, I think Zip2’s model would have struggled to survive long-term without pivoting toward online maps or something like Yelp. Eventually, some sucker bought Zip2 for $300 million, instantly turning Musk into a multimillionaire and Silicon Valley tech tycoon. You could actually tell — Zip2 was deeply divided internally, had directional problems, and Musk didn’t have absolute decision-making power. He probably wanted out long ago.

Before leaving Zip2, Musk was already planning and recruiting for online payments. At that time, the world didn’t even have anything like Alipay… Musk believed traditional finance was too conservative and that there was enormous opportunity to change the industry model. But many bankers didn’t believe internet finance could work, because internet finance couldn’t handle network security issues — after all, the slightest error in finance could have enormous consequences. Initially, the company Musk founded wasn’t PayPal but X.com, which later merged with PayPal and kept the latter’s name. Early on, X.com suffered massive attacks but survived. Their security mechanisms at the time had a significant influence on the later online payments industry. PayPal was later acquired by eBay, netting Musk hundreds of millions of dollars — another huge payday.

SpaceX and Tesla
#

Zip2 and PayPal were, for Musk, validation of his industry sensitivity and business acumen — though some questioned his execution and decision-making abilities, i.e., his CEO chops. As always, Musk viewed these industries as too conservative and old-fashioned. Musk loved recruiting extremely capable top university graduates and disliked hiring seasoned, conservative-minded industry veterans. He ran both companies simultaneously, and for a long time, neither company produced any product at all. And, as you’d imagine, rocket-building burns through money like nothing else. After several failed rocket launches, Musk deployed his signature skill: fire… And just as the financial crisis hit and no one wanted to invest, he poured his entire personal fortune into both companies. After several failures, SpaceX’s Falcon rocket finally achieved the feat of being the first private company to successfully launch a satellite, landing a $1 billion NASA contract. Tesla, after shamelessly asking early Roadster customers for more money (because developing such a radically new-concept EV cost far more than projected), finally produced a finished vehicle and built out a highway EV charging network and an electric car factory. After simultaneously succeeding with two industry-disrupting companies, no one questioned Musk’s ability anymore.

For Humanity
#

Musk’s success is inseparable from his excellent qualities: sensitivity to future technology, rapid comprehension of new industries, talent identification, a free and open tech and market environment, long working hours and execution… But the things people dislike include his ruthlessness toward employees — some loyal, devoted people, fired just like that. As a worker myself, I deeply understand the feeling of giving your all without recognition from the company. Reading this book, I could even feel how American capitalists truly exploit workers. Once, an employee missed a company gathering because he didn’t want to miss his daughter’s birth. Musk emailed him an angry tirade: do you want to wallow in domestic trivialities or work relentlessly to change the world? The guy just didn’t want to miss his daughter’s birth.

A few years ago, reading Steve Jobs, I thought: how could someone be so obsessive? But that exact kind of person changed the mobile industry and brought about the smartphone revolution. Jobs was way too formidable. After reading Elon Musk, I now feel Musk is even stronger than Jobs. Tesla, SpaceX, SolarCity — they’re all oriented toward humanity’s future. The future world seems to have started its engines; you can see it slowly arriving.

Musk’s Mars plan finally seems to have glimpsed some dawn. For decades, the American space industry had nearly stagnated. He brought a new model and once again made aerospace a hot field. But there are also uncertainties. If a crewed launch explodes and causes casualties, SpaceX could plunge back into the abyss. And if Tesla discovers a serious defect requiring a mass recall, the stock price would crash.

If you could be the first human to set foot on Mars, would you do it? Musk has thought about it, and he truly could become that person. But Musk wouldn’t do it. The book’s original words: I want to go, but I don’t have to. The point is to enable many people to go to Mars. It would be like the head of Boeing being a test pilot — for space exploration, that’s unwise. Even never going to space is fine. The point is to extend the lifespan of humanity as much as possible.

Working for humanity — this theme truly stirs the heart. I’ve played Civilization VI for days and nights on end, from stick-wielding primitives to igniting rockets, all for that moment of launch, when humanity becomes an interplanetary species and builds a new home on Mars!

Book Notes — Rich Dad Poor Dad

Tue, 13 Aug 2024 00:00:00 +0000

Rich Dad Poor Dad. I used to scoff at this kind of book. It’s the type of success-literature you see displayed at bookstore entrances, looking insubstantial at a glance, very unreliable — the kind of thing that seems to prey on people at the bottom of society who dream of getting rich quick but can never actually apply the book’s advice due to their own circumstances or environment. Besides, smart people don’t read such uncultured books, right? The title is tacky as hell!

I spent a period watching Lao Gao and Xiao Mo on Bilibili, and one episode talked about this book, making it sound mystical and mysterious. Plus, it’s a global bestseller, so I bought it to see just how magical it really was.

Started Reading E-Books
#

I’m a die-hard fan of paper books. I love the feeling of finishing an entire book and placing it on the shelf to collect — that “this whole bookshelf is my knowledge” feeling. I wasn’t really into e-books; they give a “read it and it’s gone” vibe. Three reasons brought me back to e-books:

I recently deleted all my go-to time-killing apps and needed an app that wasn’t so brain-numbing — something I could open first when pulling out my phone.
E-books are just more convenient than paper books; you can pull them out and read anytime.

Making use of fragmented subway commute time. Back when I was preparing for graduate school entrance exams, I made a detailed daily schedule that included subway time. Since I’d already built the habit of studying on the subway, I didn’t want to give it up. I recommend my summary of the graduate exam experience: How I Got Into Wuhan University’s Part-Time Graduate Program

I slightly adjusted my old plan — no need to grind vocabulary as intensely anymore — so I swapped in e-book reading. I split subway time into two blocks: morning commute and evening commute.

In the morning, my mind is clear and my mental state is good, so I read technical e-books. These require slow reading, sometimes stopping to think. This is goal-driven reading.

In the evening, my mind is foggy (not really foggy, more often it’s a headache), so I read lighter books — like “extracurricular” books such as Rich Dad Poor Dad. These books aren’t hardcore in content, so I read fast and enjoyably, with a bit of a dopamine-driven reading feel.

Poor Dad and Rich Dad
#

Author Robert Kiyosaki grew up in Hawaii. His biological father was a highly educated government education official — the “Poor Dad.” His best friend’s father was a high school dropout with extraordinary financial intelligence — the “Rich Dad.” Poor Dad had higher education but worried daily about loans and bills, while Rich Dad spent every day directing people to create wealth for him. The book has a classic line: “The poor work for money; the rich make money work for them.”

The mindset of making money: As a child, the author and his good friend came up with many ways to earn money. They once used toothpaste tubes to counterfeit coins — not knowing at the time that making money was illegal — and were stopped by adults. Later, they gathered free books from stores, set up a little library in a spot, and earned money by renting books to neighborhood kids. They stopped after attracting some local unsavory characters. Rich Dad admired their money-making behavior. He believed the difference between the rich and the poor is that the rich are always thinking about how to make money, while the poor are only thinking about how to find a good job.
On taxes: The poor pay far more in taxes than the rich. Rich Dad paid more taxes than Poor Dad, but Rich Dad’s income was vastly higher. When the US president decided to raise taxes on the rich, they only raised taxes on the salaried middle class — the truly rich were unaffected. The rich have many ways to legally avoid taxes, by understanding how to use the law. For example, the author says in real estate: if you sell a house, the income is heavily taxed, but if you swap houses, there’s no tax. The rich can use this statute to invest in real estate and legally avoid taxes. But the poor can’t escape income tax — the more you earn, the more tax you pay.
On investing: Investing requires cultivating knowledge in accounting, finance, and law. In short, if you want to make money, you need to develop your financial intelligence. When you earn a big sum, you should start the next investment rather than buying consumer goods. Think of the tables, chairs, jars, bottles, clothes, and household items in our homes — we pay a relatively high price for them, but the moment they’re bought, their value drops to near zero. This isn’t to say don’t buy things — but consider investing your money first, then consider zero-return consumption.

Finally
#

The book says our education system cultivates people’s ability to work, not their ability to make money. I strongly agree with this statement, but I still believe in the power of education. The author isn’t telling people to skip education — education is also very important for making money. We need to understand the basic operating principles and rules of this world, and that can help us find suitable ways to make money.

Given the author’s family background at the time, they may have been relatively poor compared to Rich Dad, but for truly poor people, their family conditions were far from poor. I feel my own circumstances haven’t yet reached the point where I can fully devote myself to investment and money-making. If one day I have some spare money and my management, social, and decision-making skills reach a certain level, I might look for ways to make money. For now, I can’t think that far — gotta code well and fill the holes first.

The Learning Pyramid in the book left a deep impression on me. You really retain very little from passive learning. That’s why I persist in writing frequently, including book notes like these. Problems I encountered during years of late-night database maintenance — I still remember them vividly. Hands-on experience truly creates the deepest memories. That said, hands-on experience is unpredictable and rare; reading and self-learning are the lowest-cost, easiest-to-form habits, and the most cost-effective way to improve ability. They’re not that “passive.” The Learning Pyramid’s “passive” refers to knowledge being received by the subject; “active” refers to the subject outputting knowledge. This also strengthens my motivation to record and share — whether technical or non-technical.

Book Notes — Sapiens: A Brief History of Humankind

Tue, 13 Aug 2024 00:00:00 +0000

This is a book I spent a long time reading. It’s thick, covers an enormous range of topics, and tackling the original English edition was challenging. But thankfully, I finally finished it — today (February 2023). A real sense of accomplishment.

Sapiens: A Brief History of Humankind is a grand history book that comprehensively introduces the development of human civilization. I’ve always enjoyed learning about human history, immersing myself in its weight and the vitality of civilizational progress.

The Cognitive Revolution and Fiction
#

Conventional views of human history hold that humanity’s first major evolution or revolution was learning to use tools. Like in 2001: A Space Odyssey, where apes bang bones together as the iconic BGM plays — but that’s science fiction. Sapiens argues that humanity’s first major revolution was the Cognitive Revolution, the key distinction between humans and animals. Learning to walk upright didn’t just free our hands — more importantly, it freed our minds. Four-legged running animals never evolved the way we did because harsh natural environments demanded stronger bodies and limbs for speed. Walking upright obviously makes you slower, so group living and tool use compensated. But group living and tools aren’t unique to Sapiens — many animals live in groups, and chimpanzees use tools too. What set Sapiens apart was learning to manufacture weapons, boats, and sustain much larger social groups. They walked from Africa to the Middle East, to Europe, battling the physically stronger Neanderthals and ultimately taking their territory. They reached the Far East, crossed the Bering Strait into the Americas, and even sailed to Australia. This ability to craft complex tools and communicate at unprecedented levels — that’s what the Cognitive Revolution brought.

Neanderthals themselves have gone extinct, but recent research shows the vast majority of humans carry a small amount of Neanderthal DNA — except for indigenous Africans. This suggests Neanderthals weren’t entirely wiped out by Sapiens; a small number interbred with Sapiens and their genes spread across the world. This is also key evidence supporting the Out-of-Africa theory of human origins.

The book gives a classic example of the Cognitive Revolution: imagine a lion by the river. One Sapiens sees it and tells others. The others then construct in their minds the idea that “there is a lion by the river” — even though they don’t know for certain whether one is actually there. The prerequisite is that Sapiens had to learn to conceive of things that aren’t immediately present. More importantly, once they mastered this skill, language, fiction, lies, power, social structures followed… Neanderthals clearly exchanged far less information than Sapiens.

The Cognitive Revolution had an enormous impact on civilizational development. It allowed the construction of things that don’t actually exist — gods, religions, power, money, social structures, dynasties… Take a company, for example. A company is really a social construct; it doesn’t actually exist in the physical world. A company can be a stack of 4A paper with a stamp in a document bag — but that’s just paper. Employees believe the company exists because their minds believe it does. Everyone believes it exists, but the company itself is a fiction in human minds — the entity “company” does not exist in the real world.

Money
#

How did money come about? In a world without money, stable social structures gave rise to barter trade. But as the variety of traded goods increased, the number of equivalent exchange equations grew exponentially. When trading shoes for a rake, it’s a simple one-for-one swap. Add a donkey, and you have three exchange equations: shoes-rake, rake-donkey, shoes-donkey. As goods multiply, the number of exchange equations becomes a combinatorial explosion — and that’s not even accounting for multi-item trades. Then an intermediary — money — appeared and solved the problem instantly. Everything only needed to be equated with money. Money served as the universal equivalent for all goods, and the convenience of trade improved dramatically. Early forms of money were diverse, with shells being the most common. If shells were too easy to obtain, someone could buy up everything in the market, so shell-based monetary civilizations were typically inland. Since people carried money in their pockets to buy things or hoarded it at home, and worried that too-easy acquisition of currency would disrupt markets, gold — rare, resistant to decay, difficult to mine — became humanity’s primary currency for long periods. In ancient Europe, many kings minted gold coins bearing their portraits or logos, resulting in a vast variety of European gold coins. Ancient China was somewhat different: starting with shells unearthed at Sanxingdui, then bronze coins during the Spring and Autumn and Warring States periods, then gold, silver, copper, and paper money (jiaozi) across dynasties. China didn’t stick to gold like Europe did, mainly because the population was too large and gold reserves too small, making gold too valuable — they needed other metals to create a monetary gradient to smooth trade across different scales.

There’s another great insight in the book: money and religion both have a certain transmissibility. Money and religion are essentially no different — they are both human constructs, fictions. Their only difference: religion tells you what you should believe, while money tells you what others believe.

Columbus and Zheng He
#

The Age of Discovery, the early Industrial Revolution. Europeans were passionate about exploring the world’s unknown territories. After Europeans learned the Earth was round, lacking good surveying tools, Columbus set sail westward from Europe aiming for India. They crossed the Atlantic and reached a landmass, encountered the locals, thought they’d reached India, and called them “Indians.” To this day, “Indian” in the United States carries both meanings. Europeans realized the world still had many corners untouched (at least by relatively modern civilization). They redrew world maps, filling unknown regions with sea monsters and leviathans. These maps are still widely used in video games — for instance, Civilization VI uses sea monster maps for unexplored territory, waiting to be discovered. Europeans eagerly sought new lands, and soon South America, New Zealand, Australia, and countless small islands were discovered and claimed. Where local civilizations were too far behind — the Aztec, Native American, Māori, Tasmanian civilizations — they were brutally massacred, their lands occupied by white settlers.

When the Aztec civilization encountered Spaniards clad in gleaming iron armor and wielding sharp iron swords, they thought those men were gods. They couldn’t comprehend such hard clothing and weapons — they must have been sent by the gods. And then they were deceived and slaughtered by “higher civilization.”

Zheng He’s ships were called “dragon boats” (the original text says this, even includes illustrations — I think it may be a mistake, or Westerners assumed any ship with a dragon figurehead was a dragon boat). They were several times larger than Columbus’s ships and set sail one to two centuries earlier. Zheng He’s fleet, with far superior technology, discovered new lands but didn’t occupy them — they traded with the locals. The book argues that Europeans were more adventurous and aggressive, thus ushering in the Age of Discovery. It seems Europeans hold a relatively friendly view of the Ming Dynasty. In Civilization VI, only three Chinese leaders appear: Qin Shi Huang, Wu Zetian, and Zhu Di — and only Zhu Di of the Ming Dynasty is the “tall build” development-focused leader.

Closing
#

Retracing the development of human civilization lets us understand where we came from, what we’re doing now, and explore where we’re headed. This love for the subject is also why I enjoy strategy games like Civilization VI and Humankind. When you plant rice, domesticate horses, mine salt, iron, coal, oil, uranium… there’s a thrill of human progress.

I’d like to close with a quote from Civilization VI, a game I’ve played for over 400 hours: “From the first stirrings of life beneath the water… to the great beasts of the Stone Age… to man taking his first upright steps, you have come far. Now begins your greatest quest.”

Getting Started with pg_rewind

Tue, 13 Aug 2024 00:00:00 +0000

What is pg_rewind?
#

pg_rewind is a PostgreSQL-provided tool. When the timelines of two PG instances diverge, pg_rewind can synchronize them. (For example, the primary is running, the standby failover has been running for a while — at this point the primary and standby timelines have diverged.)

pg_rewind compares the sizes of files between the source and target, then copies differing files from source to target, including configuration files. However, it does not compare unchanged files, so pg_rewind runs efficiently on large databases with few changes.

pg_rewind can be used after a standby failover: even if the standby has been running independently for some time, it can be pulled back to the same state as the primary and become a standby again.

During execution, pg_rewind compares the divergence point between primary (source) and standby (target), and transmits the primary’s WAL logs after the divergence point to the standby. Therefore, if the primary’s WAL after the divergence point is also lost, rewind won’t copy nonexistent WAL logs, and the standby will still fail to become a standby. The solution is to use restore.

!!! When using pg_rewind, back up the target instance. pg_rewind directly overwrites the target database’s files. If rewind fails, the target database may be unable to start.

Using pg_rewind
#

After a primary-standby switchover, the old primary continues running, causing timeline inconsistency. The old primary cannot start as a standby for the new primary.

When attempting to start the standby, a timeline error appears:

LOG: entering standby mode
FATAL: requested timeline 2 is not a child of this server's history
DETAIL: Latest checkpoint is at 0/6000028 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/4000098.
LOG: startup process (PID 22321) exited with exit code 1
LOG: aborting startup due to startup process failure
LOG: database system is shut down

At this point, rewind is needed to realign the primary and standby.

Configure pg_hba on the current primary Set up login permissions for the pg_rewind user to access the source database. hba changes require a database restart.

vi $source/pg_hba.conf
host all pg 172.17.100.150/32 trust

pg_rewind requires a high-privilege user. Newer PG versions allow granting privileges; older versions should use a superuser. My environment is PG 9.6, so I use the OS superuser directly.

wal_log_hints = on parameter configuration Append wal_log_hints = on to the target database’s postgres.conf, then start and shut down the target database once (at this point the primary is running and the standby is shut down).

vi $dest/postgres.conf

wal_log_hints = on

Execute pg_rewind

[pg@lzl pg96data_sla]$ /pg/pg96/bin/pg_rewind --target-pgdata /pg/pg96data_pri --source-server='host=172.17.100.150 port=5433 user=pg password=oracle dbname=postgres'
servers diverged at WAL position 0/4000098 on timeline 1
rewinding from last common checkpoint at 0/4000028 on timeline 1
Done!

Configure standby parameters Modify IP, port, directory, etc. in postgres.conf and recovery.conf. pg_rewind also copies configuration files over.

[pg@lzl pg96data_pri]$ mv recovery.done recovery.conf
[pg@lzl pg96data_pri]$ vi recovery.conf
[pg@lzl pg96data_pri]$ vi postgres.conf

Start the standby

[pg@lzl pg96data_pri]$ /pg/pg96/bin/pg_ctl -D /pg/pg96data_sla -l /pg/pg96data_sla/server.log start 
server starting
[pg@lzl pg96data_sla]$ psql -p5433 postgres
psql (9.6.17)
postgres=# \x
Expanded display is on.
postgres=# select * from pg_stat_replication ;
-[ RECORD 1 ]----+------------------------------
pid | 24766
usesysid | 16384
usename | lzl
application_name | walreceiver
client_addr | 172.17.100.150
client_hostname | 
client_port | 47345
backend_start | 2021-07-30 07:44:05.582546+00
backend_xmin | 
state | streaming
sent_location | 0/4033790
write_location | 0/4033790
flush_location | 0/4033790
replay_location | 0/4033790
sync_priority | 0
sync_state | async

Common Issues
#

pg_rewind Error 1
#

could not fetch remote file "global/pg_control": ERROR: must be superuser to read files
Failure, exiting

Solution: Use a high-privilege user.

postgres=# \du
 List of roles
 Role name | Attributes | Member of 
-------------+------------------------------------------------------------+-----------
 lzl | Replication | {}
 pg | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 rewind_user | | {}

The pg user is the built-in superuser that comes with the PG server, matching the PG installation user. The OS installation user certainly has permission to modify pg_control.

/pg/pg96/bin/pg_rewind --target-pgdata /pg/pg96data_pri --source-server='host=172.17.100.150 port=5433 user=pg password=oracle dbname=postgres'

pg_rewind Error 2
#

could not connect to server: FATAL: no pg_hba.conf entry for host "172.17.100.150", user "rewind_user", database "postgres"
Failure, exiting

No pg_hba.conf entry configured for the connection. Solution: Configure pg_hba for the user, e.g.:

host all pg 172.17.100.150/32 trust

pg_rewind Error 3
#

[pg@lzl pg96data_sla]$ /pg/pg96/bin/pg_rewind --target-pgdata /pg/pg96data_pri --source-server='host=172.17.100.150 port=5433 user=pg password=oracle dbname=postgres'

target server needs to use either data checksums or "wal_log_hints = on"

Root causes:

full_page_writes (enabled by default)
wal_log_hints must be set to on, or PG must have checksums enabled at initdb time.

Solution: Add wal_log_hints = on to the target database’s postgres.conf, then start and shut down the target database once (the target was already shut down — it must be started and shut down again for the parameter to take effect).

vi postgres.conf # add to target database config

wal_log_hints = on

Restart the target database to apply:

[pg@lzl pg96data_sla]$ /pg/pg96/bin/pg_ctl -D /pg/pg96data_pri -l /pg/pg96data_pri/server.log start 
server starting
[pg@lzl pg96data_sla]$ /pg/pg96/bin/pg_ctl -D /pg/pg96data_pri -l /pg/pg96data_pri/server.log stop
waiting for server to shut down.... done

References
#

https://www.postgresql.org/docs/9.6/app-pgrewind.html

How I Got Into Wuhan University's Part-Time Master's Program

Tue, 13 Aug 2024 00:00:00 +0000

Why Did I Want to Pursue a Part-Time Master’s?
#

To improve my academic credentials. My undergraduate degree is from an ordinary university. A higher degree can add a bit of competitiveness in my career.
I once submitted my resume to a state-owned enterprise and was completely ghosted. But a colleague with better academic credentials in the same office got through. So for state-owned enterprises, higher education is the knock on the door.
To make up for failing the graduate entrance exam as a senior and revive the dream of graduate studies.
Learning is never wrong — this is my creed.

Differences Between Full-Time and Part-Time Graduate Programs
#

Study Mode
#

Full-time means you quit your job; part-time allows you to keep working. This basically locks in part-time as the only option for most working people.

Exam Scope
#

Full-time exams are more demanding, usually covering four subjects: advanced math, graduate English, politics, and a specialized course. Part-time exams are less demanding, covering two subjects: the Management Comprehensive Exam (middle school math, logic, writing) and graduate English. Except for English, which is similar to the full-time version, the management comprehensive exam content is much easier than the full-time track (more on this later).

Research Direction
#

Full-time graduate programs lean toward research, emphasizing learning and research output, cultivating students’ learning and research abilities.

Part-time programs lean toward enhancing students’ management skills, delivering management-oriented talent to society.

These two directions are quite different.

Social Recognition
#

Full-time graduate degrees certainly carry more recognition than part-time ones. After all, the bar is higher, the study pressure is greater, it’s the mainstream path, and society recognizes it more. But part-time degrees do carry recognition too — many schools have explicitly stated they treat both equally (on paper). Most importantly, part-time graduate students hold dual certificates (degree certificate and diploma).

As for employment, it depends on the employer. Some positions only require any graduate degree, while others may explicitly state “full-time graduate degree required.” But for those who can’t quit their jobs to pursue higher education, part-time is practically the only path. In summary: Part-time also grants dual certificates, but part-time recognition < full-time recognition.

How to Choose a Major
#

Consider your job nature, career aspirations, and your wallet.

HR, finance, or corporate executive: MBA

Technical roles or engineering management: MEM

Civil servant or public administration: MPA

Accounting: MPAcc. There are a few other niche options — search online.

From a financial perspective, tuition varies by school but generally follows similar ranges. Taking Sichuan University as an example: MEM costs about 15,000 yuan per year, MBA about 150,000 yuan per year, MPA roughly similar to MEM.

For an IT professional like me, with a thin wallet and not seeing myself as an executive, MEM is the better fit.

How to Choose a School
#

Since the study difficulty is relatively low and social recognition isn’t as high as full-time, I recommend choosing a prestigious local university. 211 and 985 universities are strongly recommended — just pick one you like. Many 985 universities set their admission cutoff at the national line, so I personally feel that non-211/985 schools aren’t worth applying to. If the scores are the same, why not choose a better school?

Of course, some 985 universities set their own cutoff lines. You’ll need to check the school’s department website for historical admission scores. For example, Sichuan University sets its own line every year, typically 20–30 points above the national line.

How Does the Exam Work?
#

Exam Content
#

The exam is divided into the preliminary exam and the re-examination. The preliminary exam is in late December; the re-examination is in March.

The preliminary exam is a written test. After registering for the exam and selecting a test venue, you take it in late December — finished in one day, each session 3 hours.

The re-examination is an interview. A few schools add a written component, but since the pandemic, it’s all been online interviews — rarely do you need to write anything during the interview.

Preliminary exam content (Management Comprehensive Exam):

Re-examination content:

Early Interview
#

The early interview means the school arranges an interview before the preliminary exam — effectively moving the re-examination earlier. Once you pass the early interview, you only need to reach the national line on the preliminary exam. Under the normal process, you’d need to exceed the school’s own cutoff line.

Early interviews are only offered by some schools. For example, Tsinghua has an early-admission interview; Sichuan University doesn’t. You’ll need to check the official website of your target school.

If you pass the early interview, the pressure on the preliminary exam is indeed lighter.

How to Register
#

The two most important websites in the graduate exam process are your target school’s official website and the China Graduate Admission Website (研招网, YZW).

Before registering, check the master’s program catalog for your target school and major. For example, part-time engineering management should be selected as follows:

Should You Sign Up for a Training Course?
#

Many people wonder whether to sign up for a training course. Signing up feels too expensive — what if you don’t pass? Not signing up means you don’t know how to study, or studying feels too exhausting.

I have some authority on this question, because I did sign up for one.

I saw a training course online and asked about the price — 8,000 yuan. On top of that, there was an information gap: I didn’t know what the exam covered, how to study, how to register, which school to apply to, or where to search for this information. (Searching “part-time graduate” on Baidu immediately yields nothing but ads.) Plus, I was genuinely determined to study at the time. So I fell into this trap…

What Did the Training Course Give Me?
#

First, a pile of study materials — study methods, past exam papers, and so on. Other than the English vocabulary list, which I immediately started memorizing, I barely touched anything else. I printed the past exam papers too, but I never looked at them — not even by the time the exam was over. They looked useful, but in reality, you can search past papers on Taobao and find plenty of officially published versions with detailed explanations — far more useful and less straining on the eyes. And that vocabulary list — strangely, the one the course gave me didn’t match the one in Zhang Jian’s Yellow Book. I memorized the course’s vocabulary for a long time, only to find that some common exam words weren’t in the list. Later I switched to Zhang Jian’s Yellow Book vocabulary and it felt much better.

Beyond study materials, the most important component was live-streamed lectures, typically 8–10 PM — two hours of teaching and ten minutes of Q&A.

The live lectures were useful, especially logic and math. Just listening to those two subjects essentially eliminated the need to buy extra books to thoroughly study the fundamentals of math and logic — you only needed to do the post-class exercises and practice problems. I barely listened to the English classes; I mostly self-studied. Personally, I felt that listening to English lectures was very inefficient and a waste of time — better to memorize more words and do more reading exercises. I only listened to the last two sessions of English writing, which were extremely useful. More on English writing later (with practical tips). Finally, don’t fantasize about asking the teacher questions — these online classes have many students, and the Q&A time is only about ten minutes. My questions were never picked.

Benefits of a training course: Convenience of learning. From a working person’s perspective, you’re already working overtime a lot. Coming home exhausted, expecting yourself to spread out materials and study like it’s the gaokao — too hard. But if it’s a lecture, you just sit on the sofa and watch the livestream. That’s much easier. Saves time. No need to laboriously make a study plan and constantly adjust it. Listening to lectures is also easier than reading through a thick textbook on your own. Essentially, a training course is trading money for time.

The good learning state of classmates motivates you. You’re not studying alone with no idea how others are doing.

The pitfalls of training courses: Quality varies widely. The institution I signed up with was Shangde. I didn’t research them beforehand — they were quite mediocre. Some of their programs are contract-based: they refund if you don’t pass, but there were traps in the contract, and no one got refunds. Our group had many people fighting for refunds. Also, personal information leaks — basically every student received refund scam calls. Even I, who passed, got five or six such calls.

Teacher quality is uneven. Some teachers were excellent; others seemed like they were just coasting. Some explanations were outright misleading. In my class, the math and logic teachers were especially good, English was garbage, and writing was misleading…

Don’t fantasize that a training course will train you into a great candidate. The course is only an aid — it mainly depends on you. From the day I started planning for the exam until the preliminary exam was over, I had basically zero weekends — every one was spent in the library or a café. I declined every social gathering.

So, should you sign up for a training course?

I think if you meet all of the following conditions, you can consider it:

Enough resolve. Since you’ve paid, don’t let it go to waste. I also don’t recommend refund-based programs that give you an escape route.
Enough money. Online courses start at a few thousand yuan — my 8,000 yuan can serve as a reference. In-person courses are more expensive but offer face-to-face tutoring.
Unable to bridge the information gap. The information gap may prevent you from planning your own study schedule. If the information gap is what drives you toward a course, I suggest looking at others’ study plans and successful Bilibili uploaders’ cases. The biggest source of information is always the school’s official website.

If you have the time, energy, insufficient funds, or decent learning ability, you absolutely don’t need to sign up. In that case, making a study plan that suits you is especially important.

The Preliminary Exam
#

Before the preliminary exam is over, focus only on preparing for the preliminary exam. Generally speaking, you can prepare for the re-examination content after the preliminary exam is done. Preparing for the preliminary exam is the core of your studies, the most energy-consuming and competitive phase — this is where success is decided.

How to Prepare for the Preliminary Exam — Study Plan
#

To prepare for the preliminary exam, you need a study plan that fits you, and you need to throw yourself into it completely.

The study plan is extremely, extremely, extremely important. You need to first examine yourself — what are your strengths and weaknesses, your circumstances, which subjects you’re unfamiliar with, and which ones need long-term study.

Everyone’s situation is different. Let me first share my study plan — you can reference my approach to customizing a plan and my study methods. Because the pressure isn’t as high (compared to full-time), I strongly recommend starting in July or August. Starting too late means not enough time; starting too early makes it easy to slack off. Total study time should be 5–6 months. But if your English is really poor, start memorizing words a few months earlier.

My personal conditions:

Not enough time — often worked overtime until 9 PM, weekends generally off. Commute by subway, two hours total both ways.
Math almost completely forgotten, never touched logic before, Chinese writing has been terrible since childhood, decent English vocabulary, reading comprehension fine, English writing completely unable.

Given the study pressure and my personal conditions, my plan needed to be:

Memorize words. English is definitely the most time-consuming — it requires sustained, long-term vocabulary memorization. Before all other studying, memorize English II vocabulary. Since morning memory retention is best, I memorized words on the subway to work every day and also on weekend mornings. From August until the preliminary exam.
English reading. Actually, once you’ve memorized the words, reading comprehension is easy. English II doesn’t have many long, complex sentences — if you know all the words, reading is no problem. But I personally enjoy English reading, so I scheduled daily reading of English originals. I can’t say it had a huge impact, but it wasn’t useless — consider it a supplement to exam prep. Importantly, hobbies make habits easier to form.
Math and logic have similar study difficulty. Even though I was completely clueless, they were relatively easy to learn (the concepts are simple; the exam itself is another story — but more on that later). I studied math or logic from 8 PM to 10 PM on weekday evenings (mostly attending lectures — if you don’t have lectures, buy materials and self-study). This is also long-term study: early phase learning concepts, late phase practicing. Since I couldn’t always leave work on time, sometimes I had to use the evening commute and the one-hour lunch break to complete the daily math and logic tasks. (Never fall behind — one day of delay leads to a huge backlog.)
Chinese writing. Prepare about one month before the exam — late November or early December. Look at writing materials and try writing yourself. Don’t aim for perfection — the main thing is to express the core idea clearly. Trust me, during the exam you’ll absolutely be writing in frantic cursive.
English writing. Prepare about one month before the exam. Remember: absolutely, absolutely do not memorize model essays. Not only is it brutally hard to memorize them, they’re nearly impossible to adapt. Memorize 2–3 templates before the exam, then practice with past exam topics using the templates. You only need to swap in words — no situation where you pick up the pen and can’t write a single word.

So, my weekly plan:

This arrangement felt quite suitable for me — it fully utilized fragmented time and made good use of weekends. The first 3–4 months build the foundation: English’s foundation is vocabulary, logic and math’s foundation is concepts. Weekly study on workdays could be consolidated and practiced on weekends. The final 1–1.5 months are mainly for writing and getting the feel of past papers.

Recommended Study Materials
#

English: Zhang Jian’s Yellow Book. Just buy the vocabulary book and past exam papers. Baicizhan (vocabulary app), iReading, WeChat public account: 考研英语外刊 (Graduate English Foreign Journals).
English writing: I don’t recommend any purchasable writing guide. Use universal templates; don’t memorize model essays.
Math: Chen Jian’s Math High Score Guide, past exam answer keys.
Logic: Zhonggong’s Logic Easy Pass, past exam answer keys.
Management comprehensive writing: Buy a popular one — they’re all not great. No need to master writing too deeply.

Don’t buy practice problem books — buy past papers directly. The quality of existing practice problems doesn’t compare at all to past papers. For English, no need to buy practice books — directly buy past papers. For math and logic, beyond the built-in exercises that come with foundational study, don’t buy extra practice problem books. I did math practice problems for a short period — very time-consuming and ineffective. The key for math and logic is to solidify the fundamentals, cover all the concepts, then do past papers and review the answer explanations. In short, immersive study time (like weekends) should only be spent on past papers. Do the last 20 years’ worth of papers, then cycle through them again. (20 sets of past papers — only 2 per weekend — takes over two months to finish one round; by the time you redo them, you’ve largely forgotten the earlier ones.) Always save the most recent two years’ papers untouched — use them for timed self-testing two weeks before the exam.

English Study
#

Vocabulary
#

Morning vocabulary memorization, rain or shine (it works especially well on the subway…). Baicizhan — some people like using it. I used it early on too, but I found it ineffective. It covers the entire vocabulary pool; after a year you may not even complete one full cycle, and you’ve long forgotten what you studied earlier. So I stopped using it later. I strongly recommend my personal vocabulary method.

Everyone’s vocabulary is different. At the start, you must go through all the words once (graduate exam vocabulary is about 5,000 words) and pull out the ones you don’t know onto a vocabulary list. Since carrying a vocabulary notebook on the subway is slightly awkward, I put it on my phone.

I have a dedicated vocabulary photo album:

When memorizing, open it, zoom in — effectively covering the definitions while memorizing.

Cycle through like this. At first I did 2 pages a day, advancing 1 page a day. Later, 4 pages a day, advancing 4 pages a day. No matter what, cycle through — memorize until you can cover the definition and know the word’s meaning. For words easily confused, add them to the list, take another photo, and update the album.

Before my preliminary exam, I had cycled through these words six or seven times. Basically, aside from beyond-syllabus words, there was nothing I didn’t know.

Reading
#

English total score: 100. Reading ability components = Cloze 10 points + Reading Comprehension 40 points + New Question Type Reading 10 points = 60 points. No matter how poor your English ability, reading comprehension cannot be weak. My reading ability mostly came from daily foreign journal reading, such as iReading and 考研英语外刊. About 20 minutes a day, light study pressure. (Actually, it’s mainly vocabulary — if you know the words, sentences are easy to understand.)

iReading: “Love the World” foreign journals, one passage a day. Under Reading Plan → More Collections → Foreign Journals → Love the World, subscribe to the monthly issues, one a day. Relatively easy, good for early-stage reading improvement.
WeChat public account: 考研英语外刊, one passage a day. Updated daily. This account is very well done, highly recommended. Just harder — good for later-stage challenge. If you don’t fully understand, that’s fine; I sometimes couldn’t fully grasp it either, since the difficulty is a bit high.

Read along with morning vocabulary memorization. If short on time, you can also read on the way home.

Long and complex sentences: English II doesn’t have many. Some people dedicate time specifically to studying them. If you want to study long sentences specifically, I especially recommend Liu Xiaoyan’s Long Sentences video series (just search on video sites — they’re all free). It’s very engaging and well-organized, easy to stick with. As for me, I only watched the simple sentence part of Liu Xiaoyan’s course and stopped. Because, first, I found that as long as you know the words, you basically understand the sentences; second, the videos are too long and numerous, taking up too much study time.

Writing
#

Writing is divided into short composition (letter or notice, 10 points) and long composition (data analysis essay — bar chart/pie chart analysis, 15 points).

Again, do NOT memorize model essays. Before the exam I bought a writing book and memorized 10 model essays — truly, truly excruciating to memorize, and impossible to adapt. After memorizing the model essays, the first time I attempted an English writing question, I couldn’t write a single word — no exaggeration.

The most valuable thing in my training course was the English templates. Using the templates, I worked through all the past years’ English writing topics — every single one could be adapted. The number of words to swap in doesn’t exceed twenty; you just need to be able to write simple sentences. Here are the templates:

Short composition template — Letter:

Dear Sir or Madam,

 I am an undergraduate who majos in Applied English in this/a university.I am writing this letter for the purpose of doing sth.

 1.It,first an formost,is my idea that not only ... but also

 2.Then more importantly,so ... that...

 3.The last on I must point out is that 简单句,which could be accepted by the majority of 人/.

 So It is the very moment for me to do ...,And I am looking forward to your reply.

 yours truly,
 xxx.

Where “doing sth” includes:

感谢信:expressing my genuine gratitude for your kind help
建议信:making some suggestions concerning sth.
投诉信:making my complaints concerning sth.
祝贺信:show my sincere congratulations to you because 句子
道歉信:offer my sincere apology to you because 句子
邀请信:invite you to participate in 活动 on behalf of 某人/组织
通知信:have 某人 informed that 句子

The letter template works for all types of letters.

Besides letters, the short composition may — with low probability — test notices. The notice format differs from letters.

Short composition template — Notice:

 Notice

 In an effort to do sth,I woud like to offer you some detailed information about it.

 The 活动 will be held in the school auditorium at 7 p.m.,next Saturday,December 28th and the requirements for sth. are listed as follows.

 主段内容同书信...

 If you have any questions,please feel free to send on email to
studentsunion@123.com or call 1234567.We are looking forward to your participation.

The long composition essentially only involves analyzing bar charts and pie charts. The data falls into two categories: comparing magnitudes and comparing trends. Only the first paragraph differs between the two; the latter two paragraphs are the same.

Long composition template:

 (比大小首段)The diagram clearly shows/illustrates/d that 句子/词组(the purposes of/attitudes toward/the proportions of) among participants/respondents in a certain college.
Based on the data offered,one can distinctly see that 对象1 ranks the first/highest among all the categories,accounting for 数据1.Next are 对象2 and 对象3 with 数据2 and
数据3 respectively ,while 对象4 only constitutes 数据4.
 (比趋势首段)The diagram clearly illustrates how 话题 changed during the past several years.Based on the data provided,one can distinctly see that the number of 对象1 rose/fell significantly/slightly/gradually from 数据 in 年 to 数据 in 年,while that the number of 对象2 experienced a gradual/significant increase/decrease during the same period,reaching 数据 in 年.
 From my standpoint,there are two fundamental factors that are responsible for this scence.To begin with,the first contributing factor is that 句子.In addition,another important factor that cannot be ignored is that 句子.
 In view of the analysis above,we can conclude that it is of little surprise to see this phenomenon in the current era.Therefore,it can be predicted that 名词词组/动词ing will still take up a large share in the future.

Writing without templates — relying on your own ability — is extremely difficult. Getting an ultra-high score with templates is hard, but getting 70–80% of the score is no problem, and the upfront investment is basically zero. After memorizing the templates, just write through all the past years’ writing topics once.

Math and Logic Study
#

Math questions:
#

Logic questions:
#

Math and logic are both multiple choice — nothing special to say. Early phase: learn concepts. Later phase: improve speed.

Math and logic have a huge number of concepts. The early study phase takes 3–4 months, 2–3 hours a day, to learn all the concepts. After mastering the concepts, practice with past papers and review answer explanations. In the final month, practice with a stopwatch to improve speed: math questions within 70 minutes, logic within 60 minutes.

Chinese Writing Study
#

The management comprehensive essay is divided into Argument Validity Analysis and Argumentative Essay.

Argument Validity Analysis is essentially nitpicking — get a writing guide and look through it; it’s not hard. You need to find the logical flaws in a lengthy passage of material. When writing, find four problem points. If you haven’t studied it, you might struggle to find them; after studying, finding four points is fairly easy. Don’t worry about naming the flaws precisely — overgeneralization, equivocation, false dichotomy, etc. Just write “xxx does not lead to xxx.”

The Argumentative Essay mainly involves interpreting a short passage. The key is not to misinterpret the theme. Finding the theme is also challenging at first — look at more sample materials to get a feel for it; generally you can locate the theme. The standard structure is introduction-body-conclusion. I recommend using “individual – enterprise – nation” as the framework (intro – individual – enterprise – nation – conclusion, 5 paragraphs total). Pick a few tried-and-tested points to plug in. Some students with strong writing skills write argumentative essays using other approaches — I certainly admire that. But writing time is extremely limited. Unless you’re naturally gifted with lightning-fast thinking, I recommend using a formulaic approach. Finishing the essay is the top priority.

Exam Time Strategy
#

Yes, you need to strategize the exam timing too. Trust me 100% — you will not finish the management comprehensive exam. It’s the most time-crunched exam I’ve ever taken. You know you can solve the problems, but you have no time to compute.

On exam day: morning — management comprehensive, 3 hours. Afternoon — English, 3 hours.

English: 3 hours, relatively little content, no need for repeated recalculation — time is completely sufficient. When I finished, I had 50 minutes left and left early.

Management comprehensive: 3 hours, absolutely not enough. In my pre-exam self-timed simulations, I consistently took 4 hours. During the real exam, for math — any question over 3 minutes, skip immediately. If it feels computationally heavy, skip immediately. For logic — absolutely cannot use your usual analytical approach. Speed-read the question (logic questions have colossal amounts of text), look at the options, pick whatever feels right. Logic questions requiring computation: temporarily abandon, come back later if time permits. For writing — read the prompt and start writing immediately. Write as fast as you possibly can (both essays combined no more than 1 hour). Every 2 minutes saved could rescue a multiple-choice question.

You don’t have to finish all the multiple-choice (do fill in the answer sheet completely though), but you MUST finish the writing. So the question order is important. Many people do the essays first, then multiple-choice. I did math first, then essays, then logic. Either way, don’t leave writing for the end. In the real exam, both essays must be finished within 55 minutes — 1,500 words total, plus reading the prompt and brainstorming. Try it once and you’ll know how impossibly short the time is.

The last 20 minutes: fill in the answer sheet. After filling it in, continue solving problems.

The Re-examination
#

Basic Information About the Re-examination
#

If you’ve made it past the national line or the school’s own cutoff — congratulations, you’ve completed 90% of the journey. The remaining 10% is the re-examination. The re-examination has a mandatory elimination rate (required by national policy), typically around 70–80% passing rate. Since elimination must exist, some people get cut every year. If you don’t prepare, you’re very likely to be among them. Here’s a joke: I got cut from Sichuan University’s re-examination~

Re-examination timeframe: mid-to-late March each year.

Score release: mid-March.

Content: spoken English, specialized knowledge, comprehensive interview, politics (Sichuan University: open-book politics, no need to prepare. Wuhan University: closed-book written politics…)

Since the pandemic, re-examinations have been online interviews — no written test environment. Experts ask questions; you answer.

So you have three months to prepare for the re-examination. Conveniently, the preliminary exam ends late December, scores aren’t out yet, and February is Chinese New Year — realistically, most people start preparing only when scores are released. Take me as a cautionary example: I received the re-examination notice on March 21, the re-examination was on March 27 — I had six days to prepare, including spoken English and engineering management, which I’d never touched before… So it was embarrassing: I couldn’t answer a single one of the examiner’s English questions, couldn’t answer a single specialized question. Cut from the re-examination.

Post-Adjustment (Tiaoji)
#

When I found out I’d failed Sichuan University’s re-examination, my mood plummeted. But — every cloud has a silver lining. The adjustment (tiaoji) process was my lifesaver. While searching for adjustment schools, I found Wuhan University.

The China Graduate Admission Website has a dedicated adjustment window, giving students who failed their initial re-examination three more interview opportunities. You can fill in three preferences — three schools to apply to. Since each school has different re-examination dates and requirements, preparing for all of them is very hard. I focused mainly on preparing for Wuhan University’s adjustment. The adjustment, of course, also involves a re-examination — essentially, schools that haven’t filled their enrollment quotas run the process again, giving students who weren’t admitted in the first round another chance.

Adjustment window: late March to early April.

How to Prepare for the Re-examination?
#

The re-examination is also highly competitive. Lazy people like me are not uncommon… But no matter what, you’ve already invested over half a year — you can’t let it go down the drain. (I almost did…) For non-specialist students like me, the hardest parts of the re-examination are spoken English and specialized knowledge. From score release to the re-examination, you have about one week (while still working!), so learning from scratch is impossible. Based on my experience, the following approaches, in descending order of importance:

Find seniors who’ve been through it and get past re-examination materials (discreetly — sharing re-examination materials externally is prohibited) and course materials. See if anyone you know is at that school, or find groups on forums or Tieba.
Search Bilibili for common graduate re-examination questions. Summarize them and memorize.
Buy the school’s recommended reference books (usually course materials). They’re thick; you won’t finish them.

Finally, the most important thing: mock re-examination. Summarize potential English questions, specialized questions, and comprehensive interview questions, then find a partner to act as the examiner for a mock interview.

There are other re-examination requirements — keep an eye on department updates and your email: score weightings, interview process, dual-camera setup, interview schedule, document preparation, etc.

The End
#

The 2022 national preliminary exam line was 185. My preliminary score was 210 (English 80, Management Comprehensive 130). Here’s my re-examination acceptance notice ^_^

Good luck to all working-student-warriors battered by society but still holding onto your dreams — may your graduate exam go smoothly. You’ve got this!!!

OGG Oracle-to-PostgreSQL Sync — Hands-On Steps

Tue, 13 Aug 2024 00:00:00 +0000

Source DB: Oracle (11.2.0.4) 192.168.10.141 Target DB: PGSQL (10.12) 192.168.10.128 OGG software version: (19.1.0.0.4) OGG download: Oracle GoldenGate Downloads glibc issue handling: https://www.cnblogs.com/hxlasky/p/16779047.html

1. Install OGG Software on Source and Target
#

Source:

A. Configure response file: oggcore.rsp

oracle.install.responseFileVersion=/home/oracle/oggcore.rsp
INSTALL_OPTION=ORA11g
SOFTWARE_LOCATION=/oracle/ogg
START_MANAGER=false
MANAGER_PORT=7809
DATABASE_LOCATION=/oracle/db/11.2.0.4
INVENTORY_LOCATION=/oracle/oraInventory
UNIX_GROUP_NAME=oinstall

B. Silent install OGG

./runInstaller -silent -nowait -responseFile /home/oracle/oggcore.rsp

oracle@szgtsp431-or@ecsdb>./runInstaller -silent -nowait -responseFile /home/oracle/oggcore.rsp
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 32405 MB Passed
Checking swap space: must be greater than 150 MB. Actual 2048 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2020-08-14_08-57-27AM. Please wait ...
You can find the log of this install session at:
 /oracle/oraInventory/logs/installActions2020-08-14_08-57-27AM.log
Successfully Setup Software.
The installation of Oracle GoldenGate Core was successful.
Please check '/oracle/oraInventory/logs/silentInstall2020-08-14_08-57-27AM.log' for more details.

2. Set Database to Archive Mode
#

oracle@szgtsp431-or@ecsdb>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Fri Aug 14 09:06:34 2020
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination /oracle/oradata/archivelog
Oldest online log sequence 19
Next log sequence to archive 21
Current log sequence 21

3. Enable Force Logging and Minimum Supplemental Logging
#

alter database force logging;
alter database add supplemental log data;
alter system switch logfile;

Verify force logging and minimum supplemental logging enabled:

select force_logging,supplemental_log_data_min from v$database;

4. Set enable_goldengate_replication Parameter
#

alter system set enable_goldengate_replication=true scope=both;

If RAC, all nodes must be modified:

alter system set enable_goldengate_replication=true scope=both sid='*';

5. Create OGG User, Tablespace, and Grant Privileges
#

create tablespace tbs_ogg datafile '/oracle/oradata/datafile/tbs_ogg01.dbf' size 100M;
create user goldengate identified by 123456 default tablespace tbs_ogg temporary tablespace temp;
grant create session,alter session to goldengate;
grant alter system to goldengate;
grant resource to goldengate;
grant connect to goldengate;
grant select any dictionary to goldengate;
grant flashback any table to goldengate;
grant select any table to goldengate;
grant select any table to goldengate;
grant insert any table to goldengate;
grant update any table to goldengate;
grant delete any table to goldengate;
grant select on dba_clusters to goldengate;
grant execute on dbms_flashback to goldengate;
grant create table to goldengate;
grant create sequence to goldengate;
grant alter any table to goldengate;
grant dba to goldengate;
grant lock any table to goldengate;

6. Enable Table-Level Supplemental Logging
#

To sync table data from specific schemas, enable supplemental logging on those tables.

Check supplemental logging:

SELECT owner, table_name, log_group_name, log_group_type,
 decode(always, 'ALWAYS', 'Unconditional', NULL, 'Conditional') always
FROM dba_log_groups
ORDER BY owner, table_name, log_group_name;

Enable supplemental logging during low-activity window:

oracle@szgtsp431-or@ecsdb>ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Oct 17 2019 23:13:12
Operating system character set identified as US-ASCII.
Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
GGSCI (szgtsp431-or) 1> dblogin userid goldengate,password 123456
Successfully logged into database.
GGSCI (szgtsp431-or as goldengate@ecsdb) 2> add trandata ecs.*
2020-08-14 09:13:54 INFO OGG-15132 Logging of supplemental redo data enabled for table ECS.DEPT.
2020-08-14 09:13:54 INFO OGG-15133 TRANDATA for scheduling columns has been added on table ECS.DEPT.
2020-08-14 09:13:54 INFO OGG-15135 TRANDATA for instantiation CSN has been added on table ECS.DEPT.
2020-08-14 09:13:54 INFO OGG-15132 Logging of supplemental redo data enabled for table ECS.INFO.
...

Verify all supplemental logging added:

select * from (
 select owner,table_name from dba_tables where owner in ('BGLWT')
 minus
 select owner,table_name from dba_log_groups)
order by owner,table_name;
-- no rows selected = all table-level supplemental logging added successfully

7. Configure Manager Process
#

oracle@szgtsp431-or@ecsdb>ggsci
...
GGSCI (szgtsp431-or) 1> dblogin userid goldengate,password 123456
Successfully logged into database.
GGSCI (szgtsp431-or as goldengate@ecsdb) 2> create subdirs
Creating subdirectories under current directory /home/oracle
...
GGSCI (szgtsp431-or as goldengate@ecsdb) 3> edit param mgr
PORT 7809
DYNAMICPORTLIST 7810-7980
PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 3
PURGEDDLHISTORY MINKEEPDAYS 7, MAXKEEPDAYS 10
LAGREPORTHOURS 1
LAGINFOMINUTES 30
LAGCRITICALMINUTES 45

8. Configure Extract Process
#

GGSCI (szgtsp431-or as goldengate@ecsdb) 7> add extract extecs, tranlog, threads 1,begin now
EXTRACT added.
GGSCI (szgtsp431-or as goldengate@ecsdb) 8> add exttrail ./dirdat/lt, extract extecs
EXTTRAIL added.
GGSCI (szgtsp431-or as goldengate@ecsdb) 9> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING 
EXTRACT STOPPED EXTECS 00:00:00 00:00:38 
GGSCI (szgtsp431-or as goldengate@ecsdb) 10> edit param extecs
EXTRACT extecs
SETENV (ORACLE_HOME = "/oracle/db/11.2.0.4")
SETENV (ORACLE_SID = "ecsdb")
USERID goldengate, PASSWORD 123456
EXTTRAIL ./dirdat/lt
TRANLOGOPTIONS EXCLUDEUSER goldengate
TRANLOGOPTIONS DBLOGREADER
DBOPTIONS ALLOWUNUSEDCOLUMN
FETCHOPTIONS USESNAPSHOT, USELATESTVERSION, MISSINGROW REPORT
STATOPTIONS REPORTFETCH
WARNLONGTRANS 1h, CHECKINTERVAL 10m
DYNAMICRESOLUTION
DISCARDFILE ./dirrpt/extecs.dsc, APPEND, MEGABYTES 1024
DISCARDROLLOVER AT 6:00
REPORTROLLOVER AT 6:00
REPORTCOUNT EVERY 1 MINUTES, RATE
DDL INCLUDE MAPPED
DDLOPTIONS ADDTRANDATA, REPORT
DDLOPTIONS NOCROSSRENAME, REPORT
TABLE ECS.*;

9. Configure Pump Process
#

GGSCI (szgtsp431-or as goldengate@ecsdb) 11> add extract deliecs, exttrailsource ./dirdat/lt
EXTRACT added.
GGSCI (szgtsp431-or as goldengate@ecsdb) 12> add rmttrail ./dirdat/rt, extract deliecs, megabytes 500
RMTTRAIL added.
GGSCI (szgtsp431-or as goldengate@ecsdb) 13> edit param deliecs
EXTRACT deliecs
PASSTHRU
DYNAMICRESOLUTION
RMTHOST 192.168.10.100, MGRPORT 7809
RMTTRAIL ./dirdat/rt
DISCARDFILE ./dirrpt/deliecs.dsc, APPEND, MEGABYTES 1024
DISCARDROLLOVER AT 6:00
REPORTCOUNT EVERY 1 MINUTES, RATE
REPORT AT 0:00
REPORT AT 1:00
...
REPORT AT 23:00
REPORTROLLOVER AT 00:00
STATOPTIONS RESETREPORTSTATS
TABLE ECS.*; 

10. Start Extract Process
#

GGSCI (szgtsp431-or as goldengate@ecsdb) 20> start extecs
Sending START request to MANAGER ...
EXTRACT EXTECS starting
GGSCI (szgtsp431-or as goldengate@ecsdb) 21> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING 
EXTRACT STOPPED DELIECS 00:00:00 00:06:06 
EXTRACT RUNNING EXTECS 00:00:00 00:00:01 

11. Configure Target OGG Software
#

A. Upload OGG software and extract B. Configure OGG environment variables

[pgsql@szgtsp428-or ~]$ vi .bash_profile
## .bash_profile
## Get the aliases and functions
if [ -f ~/.bashrc ]; then
 . ~/.bashrc
fi
## User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
export PGHOME=/usr/local/pgsql
export PGDATA=/data/pgsql
export OGG_HOME=/data/ogg
export PATH=$PATH:$PGHOME/bin:$OGG_HOME
LD_LIBRARY_PATH=$PGHOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib:$OGG_HOME/lib
export LD_LIBRARY_PATH
export ODBCINI=/home/pgsql/odbc.ini
export DD_ODBC_HOME=/data/ogg

[pgsql@szgtsp428-or ~]$ ggsci
Oracle GoldenGate Command Interpreter for PostgreSQL
Version 19.1.0.0.200714 OGGCORE_19.1.0.0.0OGGBP_PLATFORMS_200628.2141
Linux, x64, 64bit (optimized), PostgreSQL on Jun 29 2020 03:59:15
Operating system character set identified as UTF-8.
Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
GGSCI (szgtsp428-or) 1>

12. Create Database and Table on Target
#

ecsdb=# \l
 List of databases
 Name | Owner | Encoding | Collate | Ctype | Access privileges 
-----------+----------+----------+-------------+-------------+-------------------
 ecsdb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | 
 postgres | pgsql | UTF8 | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | pgsql | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/pgsql +
 | | | | | pgsql=CTc/pgsql
 template1 | pgsql | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/pgsql +
 | | | | | pgsql=CTc/pgsql
(4 rows)

ecsdb=# \d
 List of relations
 Schema | Name | Type | Owner 
--------+--------------+-------+----------
 public | student_info | table | postgres
(1 row)

ecsdb=# select * from student_info;
 id | name | address 
----+------+---------
 1 | Zhang San | Guangzhou
 2 | Li Si | Shenzhen
 3 | Wang Wu | Shanghai
 4 | Zhao Liu | Beijing
 5 | Sun Qi | Wuhan
 6 | A Da | Chengdu
 7 | A Er | Nanjing
(7 rows)

13. Configure Target Manager Process and Start
#

[pgsql@szgtsp428-or ogg]$ ggsci
Oracle GoldenGate Command Interpreter for PostgreSQL
...
GGSCI (szgtsp428-or) 1> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER STOPPED 
GGSCI (szgtsp428-or) 2> create subdirs
Creating subdirectories under current directory /data/ogg
...
GGSCI (szgtsp428-or) 3> edit param mgr
port 7809
GGSCI (szgtsp428-or) 4> info all
...
GGSCI (szgtsp428-or) 5> start mgr
Manager started.
GGSCI (szgtsp428-or) 7> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING 

Now start the pump process on source (deliecs):

oracle@szgtsp431-or@ecsdb>ggsci
...
GGSCI (szgtsp431-or) 1> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING 
EXTRACT ABENDED DELIECS 00:00:00 01:06:41 
EXTRACT RUNNING EXTECS 00:00:00 00:00:07 
GGSCI (szgtsp431-or) 2> start deliecs
Sending START request to MANAGER ...
EXTRACT DELIECS starting
GGSCI (szgtsp431-or) 3> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING 
EXTRACT RUNNING DELIECS 00:00:00 01:06:55 
EXTRACT RUNNING EXTECS 00:00:00 00:00:01 

14. Target PostgreSQL Parameter Adjustment
#

wal_level = logical #minimal, replica, or logical
max_replication_slots = 10 #max number of replication slots
max_wal_sender = 10 #maximum number of wal sender processes
wal_receiver_status_interval=10s #optional, keep the system default
wal_sender_timeout = 60s #optional, keep the system default
track_commit_timestamp=off #optional, keep the system default

Restart PostgreSQL after adjusting parameters:

[pgsql@szgtsp428-or pgsql]$ pg_ctl stop -D /data/pgsql/ -l /data/pgsql/logfile
waiting for server to shut down.... done
server stopped
[pgsql@szgtsp428-or pgsql]$ pg_ctl start -D /data/pgsql/
waiting for server to start.... done
server started

15. Data Source Configuration (odbc.ini)
#

[ODBC Data Sources]
PGDSN=DataDirect 10.12 PostgreSQL Wire Protocol
postgres=DataDirect 10.12 PostgreSQL Wire Protocol
scott=DataDirect 10.12 PostgreSQL Wire Protocol
[ODBC]
IANAAppCodePage=4
InstallDir=/data/ogg
[PGDSN]
Driver=/data/ogg/lib/GGpsql25.so
Description=DataDirect 10.12 PostgreSQL Wire Protocol
Database=ecsdb
HostName=127.0.0.1
PortNumber=5432
LogonID=postgres
Password=123456

16. Connection Test
#

[pgsql@szgtsp428-or ~]$ cd /data/ogg
[pgsql@szgtsp428-or ogg]$ ggsci
...
GGSCI (szgtsp428-or) 1> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING 
GGSCI (szgtsp428-or) 2> dblogin sourcedb pgdsn userid postgres, password postgres
2020-08-14 11:35:01 INFO OGG-03036 Database character set identified as UTF-8. Locale: en_US.UTF-8.
2020-08-14 11:35:01 INFO OGG-03037 Session character set identified as UTF-8.
Successfully logged into database.

17. Configure and Start Replicat Process on Target
#

Add checkpoint table:

GGSCI (szgtsp428-or) 1> dblogin sourcedb pgdsn userid postgres, password 123456
Successfully logged into database.
GGSCI (szgtsp428-or as postgres@pgdsn) 2> add checkpointtable public.chkt
Successfully created checkpoint table public.chkt.

Configure replicat:

GGSCI (szgtsp428-or as postgres@pgdsn) 34> edit param repl
REPLICAT repl
SOURCEDEFS ./dirdef/student_info.def
SETENV (PGCLIENTENCODING = "UTF8")
SETENV (ODBCINI="/home/pgsql/odbc.ini")
SETENV (NLS_LANG="AMERICAN_AMERICA.AL32UTF8")
targetdb pgdsn userid postgres, password 123456
DISCARDFILE ./dirrpt/repl.dsc, purge
MAP ecs.student_info, TARGET public.student_info;

GGSCI (szgtsp428-or as postgres@pgdsn) 36> add replicat repl,exttrail ./dirdat/rt,checkpointtable public.chkt
REPLICAT added.
GGSCI (szgtsp428-or as postgres@pgdsn) 38> start repl
Sending START request to MANAGER ...
REPLICAT REPL starting

GGSCI (szgtsp428-or as postgres@pgdsn) 55> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING 
REPLICAT RUNNING REPL 00:00:00 00:00:08

18. Test Verification
#

First, create matching table structure on target:

create table student_info (id int primary key, name varchar(100), address varchar(100));

Then initialize data:

Configure extinit process on source:

GGSCI (szgtsp431-or as goldengate@ecsdb) 17> edit param extinit
EXTRACT extinit
userid goldengate, PASSWORD 123456
REPORTCOUNT EVERY 30 MINUTES, RATE
DISCARDFILE ./dirrpt/extinit.dsc, APPEND, MEGABYTES 1024
RMTHOST 192.168.10.100,MGRPORT 7809, compress
RMTTASK replicat,GROUP replinit
TABLE ecs.student_info;

GGSCI (szgtsp431-or as goldengate@ecsdb) 18> ADD EXTRACT extinit, SOURCEISTABLE
EXTRACT added.

Configure replinit process on target:

GGSCI (szgtsp428-or as postgres@pgdsn) 28> edit param replinit
REPLICAT replinit
targetDB pgdsn, USERID postgres, PASSWORD 123456
discardfile ./dirrpt/replinit.dsc, PURGE
SOURCEDEFS ./dirdef/student_info.def
Map ecs.student_info,target public.student_info;

GGSCI (szgtsp428-or as postgres@pgdsn) 29> add replicat repinit, SPECIALRUN
REPLICAT added.

Start Oracle-to-PG data initialization:

GGSCI (szgtsp431-or as goldengate@ecsdb) 9> start extinit
Sending START request to MANAGER ...
EXTRACT EXTINIT starting

Target: (view initialization row count via View report replicat)

Check both sides:

Source (Oracle):

SQL> select * from student_info;
 ID NAME ADDRESS
---------- ---------- ----------
Zhang San Guangzhou
Li Si Shenzhen
Wang Wu Shanghai
Zhao Liu Beijing
Sun Qi Wuhan
A Da Chengdu
A Er Nanjing
A San Beijing
rows selected.

Target (PostgreSQL):

ecsdb=# select * from student_info;
 id | name | address 
----+------+---------
 1 | Zhang San | Guangzhou
 2 | Li Si | Shenzhen
 3 | Wang Wu | Shanghai
 4 | Zhao Liu | Beijing
 5 | Sun Qi | Wuhan
 6 | A Da | Chengdu
 7 | A Er | Nanjing
 8 | A San | Beijing
(8 rows)

Insert data on source:

SQL> insert into ecs.student_info values (10,'aa','bb');
1 row created.
SQL> commit;
Commit complete.

Check target — data synchronized successfully.

Original link: https://lastdba.com/2024/08/13/ogg搭建oracle-pg同步实操步骤/

OGG PostgreSQL-to-Oracle Sync — Hands-On Steps

Tue, 13 Aug 2024 00:00:00 +0000

OGG software version: (19.1.0.0.4) Oracle version: 11.2.0.4 PG version: pg10 OGG download: https://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

glibc issue handling: https://www.cnblogs.com/hxlasky/p/16779047.html

1. Create Database and Table on Source
#

[root@node2 ~]# su - postgres
Last login: Tue Jul 21 21:08:52 CST 2020 on pts/0
[postgres@node2 ~]$ pg_ctl -D /opt/pgsql_data -l logfile start
waiting for server to start.... done
server started
postgres=# create database test
postgres=# \c lzldb
postgres=# create table tab1(id int primary key,name varchar(20))

2. Create Database and Table on Target
#

sqlplus / as sysdba
SQL> create table ORALZL.tab1(id number primary key,name varchar2(20));

3. Extract and Install OGG for PostgreSQL
#

-- Unlike OGG for Oracle, OGG for PG only needs extraction. Oracle version requires running runInstaller.
[postgres@node1 ~]$ id postgres
uid=54323(postgres) gid=54330(postgres) groups=54330(postgres)
[postgres@node1 ~]$ exit
logout
[root@node1 ~]# mkdir /ogg
[root@node1 ~]# chown -R postgres /ogg
[root@node1 ~]# chmod -R 755 /ogg
[root@node1 ~]#
[root@node1 soft]# ls -l

total 240744

-rw-r--r--. 1 root root 87028695 Jul 22 02:51 19100200714_ggs_Linux_x64_PostgreSQL_64bit.zip
[root@node1 soft]# chmod 777 19100200714_ggs_Linux_x64_PostgreSQL_64bit.zip
[root@node1 soft]# unzip 19100200714_ggs_Linux_x64_PostgreSQL_64bit.zip
Archive: 19100200714_ggs_Linux_x64_PostgreSQL_64bit.zip
 inflating: ggs_Linux_x64_PostgreSQL_64bit.tar 
 inflating: OGG-19.1.0.0-README.txt 
 inflating: release-notes-oracle-goldengate_19.1.0.200714.pdf 
[root@node1 soft]# chmod 777 ggs_Linux_x64_PostgreSQL_64bit.tar
[root@node1 soft]# su - postgres
[postgres@node1 ~]$ cd /soft
[postgres@node1 soft]$ tar -xf ggs_Linux_x64_PostgreSQL_64bit.tar -C /ogg

4. Configure PG User Environment Variables
#

Source PG:

[postgres@node1 ~]$ cat .bash_profile
## .bash_profile
## Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
## User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export GGHOME=/ogg
export PG_DATA=/opt/pgsql/pgsql/bin
export PATH=$PG_DATA:$PATH
export PG_HOME=/opt/pgsql/pgsql
export LD_LIBRARY_PATH=$PG_HOME/lib:$LD_LIBRARY_PATH:$GGHOME/lib
export ODBCINI=/home/postgres/odbc.ini
export DD_ODBC_HOME=/ogg
export PATH
[postgres@node1 ~]$ source .bash_profile

5. Configure Manager Process
#

[postgres@node1 ~]$ cd /ogg
[postgres@node1 ogg]$ ./ggsci
Oracle GoldenGate Command Interpreter for PostgreSQL
Version 19.1.0.0.200714 OGGCORE_19.1.0.0.0OGGBP_PLATFORMS_200628.2141
Linux, x64, 64bit (optimized), PostgreSQL on Jun 29 2020 03:59:15
Operating system character set identified as UTF-8.
Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
GGSCI (node1) 2> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER STOPPED 

GGSCI (node1) 3> create subdirs
Creating subdirectories under current directory /ogg
Parameter file /ogg/dirprm: created.
Report file /ogg/dirrpt: created.
Checkpoint file /ogg/dirchk: created.
Process status files /ogg/dirpcs: created.
SQL script files /ogg/dirsql: created.
Database definitions files /ogg/dirdef: created.
Extract data files /ogg/dirdat: created.
Temporary files /ogg/dirtmp: created.
Credential store files /ogg/dircrd: created.
Masterkey wallet files /ogg/dirwlt: created.
Dump files /ogg/dirdmp: created.
GGSCI (node1) 4> edit params mgr
GGSCI (node1) 5> view params mgr
port 7809
GGSCI (node1) 6> start mgr
Manager started.
GGSCI (node1) 7> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING 

6. Adjust Source PostgreSQL Parameters
#

[postgres@node1 ogg]$ vi /opt/pgsql_data/postgresql.conf
wal_level = logical #minimal, replica, or logical
max_replication_slots = 10 #max number of replication slots
max_wal_sender = 10 #maximum number of wal sender processes
wal_receiver_status_interval=10s #optional, keep the system default
wal_sender_timeout #optional, keep the system default
track_commit_timestamp #optional, keep the system default
wal_receiver_status_interval=10s
wal_sender_timeout = 60s
track_commit_timestamp=off

Restart source PostgreSQL after adjustment:

[postgres@node1 ogg]$ pg_ctl -D /opt/pgsql_data -l logfile stop
[postgres@node1 ogg]$ pg_ctl -D /opt/pgsql_data -l logfile start

7. Configure OGG for PG Data Source
#

cd /home/postgres/
vi odbc.ini
[ODBC Data Sources]
PGDSN=DataDirect 7.1 PostgreSQL Wire Protocol
postgres=DataDirect 7.1 PostgreSQL Wire Protocol
scott=DataDirect 7.1 PostgreSQL Wire Protocol
[ODBC]
IANAAppCodePage=4
InstallDir=/ogg
[PGDSN]
Driver=/ogg/lib/GGpsql25.so
Description=DataDirect 7.1 PostgreSQL Wire Protocol
Database=test
HostName=192.168.1.112
PortNumber=5432
LogonID=postgres
Password=postgres

8. Connection Test
#

[postgres@node1 ~]$ cd /ogg
[postgres@node1 ogg]$ ./ggsci
--dblogin sourcedb pgdsn userid pg, password 123456
GGSCI (node1) 1> dblogin sourcedb pgdsn userid postgres, password postgres
2020-07-22 03:10:44 INFO OGG-03036 Database character set identified as UTF-8. Locale: en_US.UTF-8.
2020-07-22 03:10:44 INFO OGG-03037 Session character set identified as UTF-8.
Successfully logged into database.
GGSCI (node1 as postgres@pgdsn) 2>

9. Enable Table-Level Supplemental Logging
#

Source:

GGSCI (node1) 3> dblogin sourcedb pgdsn userid postgres, password postgres
2020-07-22 03:21:01 INFO OGG-03036 Database character set identified as UTF-8. Locale: en_US.UTF-8.
2020-07-22 03:21:01 INFO OGG-03037 Session character set identified as UTF-8.
Successfully logged into database.
GGSCI (node1 as postgres@pgdsn) 4> add trandata public.tab1 --If table has primary key, this step can be skipped
Logging of supplemental log data is enabled for table public.tab1. REPLICA IDENTITY was DEFAULT and is changed to FULL
GGSCI (node1 as postgres@pgdsn) 5>
GGSCI (node1 as postgres@pgdsn) 5> info trandata public.tab1
Logging of supplemental log data is enabled for table public.t1 with REPLICA IDENTITY set to FULL

10. Register Extract Process on PG
#

Registering an extract process on PG essentially creates a replication slot. The output plugin defaults to test_decoding.

 GGSCI (node1 as postgres@pgdsn) 6> Register Extract ext_pg
 2020-07-22 03:25:27 INFO OGG-25355 Successfully created replication slot 'ext_pg_2947c06e0ea2ec74' for EXTRACT group 'EXT_PG' in database 'test'.

11. Configure Extract and Pump Processes
#

Configure extract process:

 edit param ext_pg

SETENV ( PGCLIENTENCODING = "UTF8" )
 SETENV (NLS_LANG="AMERICAN_AMERICA.AL32UTF8")
 extract ext_pg
 SETENV (ODBCINI="/home/pg/odbc.ini" ) 
 SOURCEDB pgdsn, USERID pg, PASSWORD 123456
 exttrail ./dirdat/st
 TABLE PUBLIC.TAB1;
 ----GETTRUNCATES ### This feature on PostgreSQL 10.12: ERROR OGG-25541 GETTRUNCATES is not valid. PostgreSQL supports TRUNCATE capture from version 11.

Note: PG to Oracle cannot sync TRUNCATE commands.

Configure pump process:

 extract pump_pg
 SETENV (ODBCINI="/home/pg/odbc.ini" )
 RMTHOST 172.17.100.150, MGRPORT 7809, compress
 numfiles 10000
 RMTTRAIL ./dirdat/rt
 TABLE PUBLIC.TAB1;

12. Add Trail and Start Extract/Pump
#

ADD extract ext_pg, TRANLOG,BEGIN now
 add exttrail ./dirdat/st,extract ext_pg,megabytes 500
 add extract pump_pg,exttrailsource ./dirdat/st
 add rmttrail ./dirdat/rt,extract pump_pg,megabytes 500
 start ext_pg
 start pump_pg

13. Configure defgen
#

If table structures are consistent, you can configure ASSUMETARGETDEFS.

edit param defgen

DEFSFILE ./dirdef/tab1.def, PURGE
 SOURCEDB pgdsn, USERID pg, PASSWORD 123456
 TABLE PUBLIC.tab1;

Generate table definition file:

defgen paramfile /oggpg/dirdef/tab1.prm

Copy the defgen file to the target’s dirdef directory.

14. Verify Trail Delivery on Target
#

[oracle@lzl dirdat]$ cd dirdat
 [oracle@lzl dirdat]$ ll
 -rw-r----- 1 pg pg 1439 Feb 28 11:02 rt000000000

15. Register Extract Process on PG
#

Registering an extract process on PG creates a replication slot:

GGSCI (node1 as postgres@pgdsn) 6> Register Extract ext_pg
2020-07-22 03:25:27 INFO OGG-25355 Successfully created replication slot 'ext_pg_2947c06e0ea2ec74' for EXTRACT group 'EXT_PG' in database 'test'.

16. Configure Oracle User Environment Variables
#

export ORACLE_BASE=/oracle/app/oracle
 export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1
 export ORACLE_SID=oralzl
 export OGG_HOME=/oggfororacle
 export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH
 export TNS_ADMIN=$ORACLE_HOME/network/admin
 export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$OGG_HOME:$ORACLE_HOME/lib32:/lib/usr/lib:/usr/local/lib

17. Configure Oracle Listener and TNS
#

OGG for Oracle defaults to using TNS_ADMIN’s tns. You can also manually configure during extract configuration, e.g.: USERID goldengate@127.0.0.1:1521/oralzl, PASSWORD 123456

18. Install OGG for Oracle on Target
#

Download OGG software. Configure oggcore.rsp file:

oracle.install.responseFileVersion=/home/oracle/oggcore.rsp
INSTALL_OPTION=ORA11g
SOFTWARE_LOCATION=/ogg
START_MANAGER=false
MANAGER_PORT=7809
DATABASE_LOCATION=/oracle/db/11.2.0.4
INVENTORY_LOCATION=/oracle/oraInventory
UNIX_GROUP_NAME=oinstall

Silent install OGG:

./runInstaller -silent -nowait -responseFile /home/oracle/oggcore.rsp

19. Oracle Database User and Privileges
#

create user goldengate identified by "123456";
 grant create session,alter session to goldengate;
 grant alter system to goldengate;
 grant resource to goldengate;
 grant connect to goldengate;
 grant select any dictionary to goldengate;
 grant flashback any table to goldengate;
 grant select any table to goldengate;
 grant select any table to goldengate;
 grant insert any table to goldengate;
 grant update any table to goldengate;
 grant delete any table to goldengate;
 grant select on dba_clusters to goldengate;
 grant execute on dbms_flashback to goldengate;
 grant create table to goldengate;
 grant create sequence to goldengate;
 grant alter any table to goldengate;
 grant dba to goldengate;
 grant lock any table to goldengate;

20. Target Manager Process
#

edit param mgr

PORT 7809
 DYNAMICPORTLIST 7810-7980
 PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 3
 PURGEDDLHISTORY MINKEEPDAYS 7, MAXKEEPDAYS 10
 LAGREPORTHOURS 1
 LAGINFOMINUTES 30
 LAGCRITICALMINUTES 45

start mgr

21. Configure Replicat Process on Target
#

GGSCI (node2) 8> dblogin userid goldengate@127.0.0.1:1521/oralzl,password 123456
GGSCI (node2 as postgres@pgdsn) 9> add checkpointtable goldengate.chkt
Successfully created checkpoint table public.chkt.

Replicat process:

 edit param rep_pg
REPLICAT rep_pg
 USERID goldengate@127.0.0.1:1521/oralzl, PASSWORD 123456
 SOURCEDEFS ./dirdef/tab1.def
 MAP public.tab1, TARGET oralzl.tab1; 
add replicat rep_pg,exttrail ./dirdat/rt,checkpointtable goldengate.chkt
 start rep_pg

22. Test Sync
#

[postgres@node1 ~]$ psql
postgres=# \c lzldb
test=# \d tab1;
​ Table "public.tab1"
 Column | Type | Collation | Nullable | Default 
--------+-----------------------+-----------+----------+---------
 id | integer | | not null | 
 name | character varying(20) | | | 
Indexes:
 "t1_pkey" PRIMARY KEY, btree (id)


lzldb=# insert into t2 values(1,'lzl1') ;
INSERT 0 1
lzldb=# select * from t2;
 id | name 
----+------
 1 | lzl1


[postgres@node2 ~]$sqlplus / as sysdba
SQL> select * from oralzl.tab1; 
​ id name 
---------- ----------
​ 1 lzl1 

Original link: https://lastdba.com/2024/08/13/ogg搭建pg-oracle同步实操步骤/

PostgreSQL Logical Replication

Tue, 13 Aug 2024 00:00:00 +0000

What is Logical Replication
#

PostgreSQL logical replication is based on logical decoding, which parses WAL log streams into a specified format for output. The subscriber node receives the parsed data and applies it.

Logical replication differs from streaming replication (physical replication) which is based on instance-level primary-standby where the physical structures are identical. Logical replication can selectively replicate at the table level. Logical Replication in official documentation specifically refers to the “publish-subscribe” model. In fact, many tools can use logical decoding for heterogeneous database data synchronization.

pg9.4’s pglogical plugin can support logical replication (https://github.com/2ndQuadrant/pglogical), and pg10 onwards natively supports logical replication.

Logical replication can be used for database upgrades, heterogeneous data migration, table-level data synchronization links, subscribing to data streams, etc.

Logical Decoding
#

Logical decoding can parse table data changes in WAL logs into row data streams or SQL text. These row data streams or SQL text can be consumed by other types of databases or software. The specific parsing format is determined by the output plugin.

Replication Slots
#

In logical replication, a replication slot represents a data change stream. Like physical replication slots, logical replication slots also ensure that after an abnormal replication interruption, the related WAL logs are not deleted, so that WAL log parsing can continue after replication reconnects. A database can have multiple replication slots. Each replication slot has only one output plugin, and each replication slot represents one replication link. Replication slots are essentially used to manage replication links. Unlike streaming replication which can function without replication slots, logical replication must have replication slots.

Output Plugin
#

The output plugin converts WAL log information into the format required by the replication slot. PostgreSQL has some built-in output plugins and additional ones can be added through plugins. Each logical replication slot has an output plugin for WAL-related parsing work.

Output plugins use callback functions to manage parsing. For example, OUTPUT_PLUGIN_BINARY_OUTPUT and OUTPUT_PLUGIN_TEXTUAL_OUTPUT are used to set whether the out_type is binary or text. There are also callback functions to notify the plugin of transaction data changes and sort transactions. Callback functions of course don’t need to be used manually; some built-in output plugins are already packaged.

Each output plugin has some different parsing behaviors and output formats.

Several Common Output Plugins
#

test_decoding: This is a sample output plugin, essentially the raw form of an output plugin. Official documentation says it’s a template, but it can still parse. This output plugin comes with PostgreSQL but needs to be compiled in contrib.

pgoutput: The default output plugin for the publish-subscribe model. In publish-subscribe, the walsender process uses this output plugin to logically decode WAL logs.

decoder_raw: Parses into SQL text format. This is not included with PostgreSQL; compile it yourself: https://github.com/michaelpq/pg_plugins/tree/main/decoder_raw

wal2json: This output plugin converts WAL log information into JSON format.

Other output plugins can be referenced at: https://wiki.postgresql.org/wiki/Logical_Decoding_Plugins

Some domestic vendors have also made their own output plugins.

Relationship between several output plugins and logical replication plugins:

pgoutput, test_decoding, and wal2json have been introduced above.

pglogical was the predecessor of pglogical replication in pg9.4.

BDR was developed by 2ndQuadrant, supporting bidirectional replication and DDL synchronization with more powerful features. BDR 3.0 onwards became closed-source.

Functions and Tools for Manually Receiving Parsed Data
#

pg_logical_slot_get_changes(): Displays parsed data and consumes it.

pg_logical_slot_peek_changes(): Displays parsed data without consuming it.

pg_recvlogical: A tool included with PostgreSQL that can consume data within a replication slot, equivalent to the downstream of logical replication. The corresponding physical WAL receiving tool is pg_receivewal.

Logical Decoding Test 1: Observing data parsing with 2 different output plugins
#

-- Create two logical replication slots using logical_test and logical_raw respectively

lzldb=# select pg_create_logical_replication_slot('logical_test','test_decoding');

 pg_create_logical_replication_slot 

------------------------------------

 (logical_test,0/1756F50)

lzldb=# select pg_create_logical_replication_slot('logical_raw','decoder_raw');

 pg_create_logical_replication_slot 

------------------------------------

 (logical_raw,0/1756F88)

-- Only the upstream is created, slot is in f state

lzldb=# select * from pg_replication_slots;

 slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn | wal_status | safe_wal_size 

--------------+---------------+-----------+--------+----------+-----------+--------+------------+------+--------------+-------------+---------------------+------------+---------------

 logical_test | test_decoding | logical | 16385 | lzldb | f | f | | | 558 | 0/1766878 | 0/17668B0 | reserved | 

 logical_raw | decoder_raw | logical | 16385 | lzldb | f | f | | | 557 | 0/1756F50 | 0/1756F88 | reserved | 

-- Create a table

lzldb=# create table tdecoder222(a int,b varchar(10));

CREATE TABLE

-- Attempt to get this DDL

lzldb=# SELECT * FROM pg_logical_slot_get_changes('logical_raw', NULL, NULL, 'include-xids', '0');

ERROR: option "include-xids" = "0" is unknown

CONTEXT: slot "logical_raw", output plugin "decoder_raw", in the startup callback

lzldb=# SELECT * FROM pg_logical_slot_get_changes('logical_test', NULL, NULL, 'include-xids', '0');

 lsn | xid | data 

-----------+-----+--------

 0/17669C8 | 558 | BEGIN

 0/1776778 | 558 | COMMIT

-- We can see that decoder_raw didn't parse the DDL at all, and logical_test only got the DDL transaction without the DDL statement itself, essentially not parsing the DDL

-- Insert a row

lzldb=# insert into tdecoder222 values(1,'lzl');

INSERT 0 1

lzldb=# select * from pg_logical_slot_peek_changes('logical_test',null,null);

 lsn | xid | data 

-----------+-----+---------------------------------------------------------------------------

 0/1776890 | 560 | BEGIN 560

 0/1776890 | 560 | table public.tdecoder222: INSERT: a[integer]:1 b[character varying]:'lzl'

 0/1776900 | 560 | COMMIT 560

lzldb=# select * from pg_logical_slot_peek_changes('logical_raw',null,null);

 lsn | xid | data 

-----------+-----+----------------------------------------------------------

 0/1776890 | 560 | INSERT INTO public.tdecoder222 (a, b) VALUES (1, 'lzl');

-- test_decoding parsed the transaction

-- decoder_raw parsed the transaction into SQL statements

This test allows us to conclude:

Replication slots in f state still parse, waiting for downstream consumption
Each output plugin has some different parsing behaviors and output formats

Logical Decoding Test 2: Using pg_recvlogical to receive logically decoded data, simulating a logical replication link
#

-- Configure passwordless login

[pg@lzl ~]$ vi .pgpass

[pg@lzl ~]$ cat .pgpass

lzl:5410:lzldb:pg:pg

[pg@lzl ~]$ chmod 0600 .pgpass

-- Start pg_recvlogical

[pg@lzl ~]$ pg_recvlogical -h lzl -p 5410 -d lzldb -U pg --slot=logical_raw --start -f recv.sql &

[pg@lzl ~]$ ps -ef|grep recv|grep -v grep

pg 7747 7355 0 21:40 pts/3 00:00:00 pg_recvlogical -h lzl -p 5410 -d lzldb -U pg --slot=logical_raw --start -f recv.sql

lzldb=# insert into tdecoder222 values(2,'qwe');

INSERT 0 1

lzldb=# update tdecoder222 set b='asd' where a=2;

UPDATE 1

[pg@lzl ~]$ tail -2f recv.sql

INSERT INTO public.tdecoder222 (a, b) VALUES (2, 'qwe');

-- update was not correctly parsed

-- Add a primary key to the table

lzldb=# alter table tdecoder222 add primary key(a);

ALTER TABLE

lzldb=# insert into tdecoder222 values(100,'lzl1');

INSERT 0 1

lzldb=# insert into tdecoder222 values(200,'lzl2');

INSERT 0 1

lzldb=# update tdecoder222 set b='lzlupdate' where a=200;

UPDATE 1

[pg@lzl ~]$ tail -3f recv.sql

INSERT INTO public.tdecoder222 (a, b) VALUES (100, 'lzl1');

INSERT INTO public.tdecoder222 (a, b) VALUES (200, 'lzl2');

UPDATE public.tdecoder222 SET a = 200, b = 'lzlupdate' WHERE a = 200;

– After adding a primary key, update was correctly parsed by decoder_raw – Without a primary key, it won’t be correctly parsed. This is related to replica identity, which will be introduced later.

Prerequisites for Logical Replication
#

1. Parameters
#

1.1 Basic Required Parameters

wal_level. Takes effect after restart, default is replica. The wal_level parameter must be logical. logical does not change WAL to logical; it means that on top of supporting physical replication (replica), the necessary information for logical decoding is added. Since pg9.6, there are only minimal, replica, and logical, with information content increasing successively.
max_replication_slots. Takes effect after restart, default value below pg9.6 is 0, pg10 and above is 10. 10 is generally sufficient. Like physical replication, logical replication generally also uses replication slots. PostgreSQL backups and physical replication can both occupy replication slot counts.

1.2 Source-side Required Parameters

max_wal_senders. Takes effect after restart, default 10. Sender process count limit. The publisher’s sender transmits the parsed logs. Generally, one logical replication slot corresponds to one sender and one worker. This is similar to physical replication, where one physical replication slot corresponds to one sender and one receiver.

1.3 Target-side Required Parameters

max_worker_processes. Takes effect after restart, default 8. Worker process count limit. Parallel processes (parallel queries, parallel statistics collection, etc., limited by max_parallel_workers), logical replication worker processes (max_logical_replication_workers), and some other programs that need to fork workers are all related to this parameter. It should be set to max_parallel_workers + logical replication apply workers + other background workers.
max_logical_replication_workers. Takes effect after restart, default 4. Logical replication worker process count, including logical replication apply worker processes and table sync worker processes.
max_sync_workers_per_subscription. Takes effect after reload, default 2. Sync worker processes when adding new tables to logical replication. Currently, one table has only one parallel.
The above three parameters are tiered: max_sync_workers_per_subscription < max_logical_replication_workers < max_worker_processes. In short, there must be workers available.

2. Permissions
#

Replication user permissions. Logical replication users need replication privileges.

ALTER ROLE <usename> WITH REPLICATION;

HBA access restrictions, allowing downstream to access the database using the replication user.

host lzldb user1 172.17.100.150/32 md5

For the publish-subscribe model, CREATE permission on the database or superuser permission is needed.

When creating a publication, for table only, at least the table owner with CREATE permission is needed. All other publications require superuser.

When creating a subscription, superuser is required.

grant create on database lzl1db to owner1; or

alter user replicate1 superuser;

Additionally, read or write permissions on tables during replication are also necessary.

Logical Synchronization Between PostgreSQL Instances — Publish and Subscribe
#

PostgreSQL’s built-in logical replication is based on the publish-subscribe model. The publish-subscribe model does not parse into SQL for application.

Publication
#

A publisher can have multiple publications, and each publication can have multiple tables.
When publishing, you can specify:

for table — publishes certain tables. New tables need to be explicitly added with ALTER PUBLICATION ADD TABLE. At minimum, the table owner is needed to create this publication.

for all tables — publishes all tables under the database. New tables are automatically published. Superuser is required to create this publication.

for all tables in schema — publishes all tables under the schema. New tables are automatically published. Superuser is required to create this publication. Supported starting from pg15.

Publications by default include INSERT, UPDATE, DELETE, and TRUNCATE. You can also specify to replicate only certain commands. DDL is not synchronized. (Official documentation verbatim. This means truncate is not considered DDL in PostgreSQL — leaving this as a topic for later research. Truncate is DDL in MySQL and Oracle.)
Only base tables can be published; temporary tables, foreign tables, views, sequences, etc. cannot be published. Partitioned table publishing is related to PostgreSQL version and partition attributes. pg15 defaults to publishing all partitions of a partitioned table.
publish_via_partition_root. Supported from pg13. This publication parameter indicates whether partitioned tables use partitions for filtering (false, default) or use the parent partition for row filtering. If set to true, heterogeneous partitioned table logical replication is supported, such as partitioned table to regular table replication. truncate replication is not possible when true.

Subscription
#

A subscription has only one publisher but can subscribe to multiple publications on the publisher.
A subscriber can have multiple subscriptions, each receiving data from one replication slot.
One subscription corresponds to one replication slot, which is on the publisher side.
When creating or deleting a subscription, the replication slot is automatically created or deleted on the publisher by default.
Creating a subscription requires superuser.
DDL is not synchronized; tables must already be created.
Existing data is synchronized by default, via COPY snapshot to the subscriber.
Synchronization can be paused and resumed with ALTER SUBSCRIPTION sub1 {ENABLE|DISABLE}.
When a publication adds new tables, refresh is needed on the subscriber side: alter subscription sub1 refresh publication.
Schema names, table names, and column names must be consistent between publication and subscription. Column types can differ (as long as implicit conversion succeeds). Column order can be different.
Subscriptions also have some attributes, such as binary transfer, streaming, synchronous commit, two-phase commit, etc.

logical replication launcher is used to start the subscriber-side worker processes and only exists at startup.

/*-------------------------------------------------------------------------
...
 * IDENTIFICATION

 * src/backend/replication/logical/launcher.c
...
 * NOTES
 * This module contains the logical replication worker launcher which
 * uses the background worker infrastructure to start the logical
 * replication workers for every enabled subscription.
 *-------------------------------------------------------------------------

 */

Publish-Subscribe Related Views
#

pg_publication; – View publications. Publications themselves are stateless; replication slots are stateful, so there’s no pg_stat_publication.

pg_publication_tables – View published tables, simple and clear.

pg_publication_rel – View published tables, all IDs.

pg_stat_subscription – View subscription status, pid is the worker process pid.

pg_subscription – View subscriptions.

pg_subscription_rel – View subscription tables. There’s no pg_subscription_tables. Additionally, this view can show the sync status of individual tables under a subscription, which the replication slot view cannot do.

\dRp list replication publications

\dRs list replication subscriptions

Creating a Publication and Subscription
#

Using a dedicated replication user replicate1, create a publication and subscription in the database lzldb to implement logical replication of table trep1.

`Role`	`Host IP`	`Port`	`Database`	`Schema`	`Table`	`Replication User`	`Version`
`Publisher`	`172.17.100.150`	`5410`	`lzldb`	`public`	`trep1`	`replicate1`	`pg13`
`Subscriber`	`172.17.100.150`	`5412`	`lzlbd`	`public`	`trep1`	`replicate1`	`pg13`

Creating the Publication
#

# Modify postgres.conf, wal_level parameter takes effect after restart

wal_level=logical 

# Modify pg_hba.conf file, takes effect after reload

host lzldb replicate1 172.17.100.150/32 md5

-- Create replication user and grant privileges

create user replicate1 with password 'replicate1';

alter user replicate1 with replication;

grant create on database lzldb to replicate1;

-- Create the table to be replicated and grant privileges to the replication user

\c lzldb replicate1 -- If the replication user is not the table owner, should grant select on trep1 to replicate1

create table trep1(a int primary key,b char(10));

insert into trep1 values(1,'abc') 

-- Create publication, superuser can also be used

\c lzldb replicate1

create publication pub_lzl1 for table trep1;

-- View publication. \dRp or pg_publication
lzldb=# select * from pg_publication;

 oid | pubname | pubowner | puballtables | pubinsert | pubupdate | pubdelete | pubtruncate | pubviaroot 

-------|----------|----------|--------------|-----------|-----------|-----------|-------------|-----------
 16400 | pub_lzl1 | 16392 | f | t | t | t | t | f

Creating the Subscription
#

-- Create table definition

create table trep1(a int primary key,b char(10));

-- Use superuser to create subscription

CREATE SUBSCRIPTION sub_test

CONNECTION 'host=172.17.100.150 port=5410 dbname=lzldb user=replicate1 password=replicate1'

PUBLICATION pub_lzl1;

lzlbd=# select * from pg_subscription; -- View subscription. \dRs or pg_subscription

 oid | subdbid | subname | subowner | subenabled | subconninfo | subslotname | subsynccommit | subpublications 

-------|---------|----------|----------|------------|--------------------------------------------------------------------------------|-------------|---------------+-----------------
 16394 | 16384 | sub_test | 10 | t | host=172.17.100.150 port=5410 dbname=lzldb user=replicate1 password=replicate1 | sub_test | off | {pub_lzl1}

lzlbd=# select * from trep1; -- Verify existing data has been synchronized

 a | b 

---+------------
 1 | abc

Publish-Subscribe Model Test 1: Truncate Synchronization
#

lzldb=# truncate table trep1;

TRUNCATE TABLE

lzldb=# select * from trep1;

 a | b 

---+---
(0 rows)

lzlbd=# select * from trep1; -- In publish-subscribe mode, truncate is synchronized

 a | b 

---+---
(0 rows)

Publish-Subscribe Model Test 2: Adding New Table Synchronization
#

-- Under an existing publish-subscribe, add a new table synchronization. lzldb is publisher, lzlbd is subscriber

lzldb=# create table tab_pk(a int,b varchar(10));

CREATE TABLE

lzldb=# alter table tab_pk add primary key(a);

ALTER TABLE

lzldb=# alter publication pub_lzl1 add table tab_pk;

ALTER PUBLICATION

-- After adding a table on the publisher, refresh must be executed on the subscriber. Refresh defaults to synchronizing existing data

lzlbd=# alter subscription sub_test refresh publication; 

ALTER SUBSCRIPTION

lzlbd=# select * from pg_subscription_rel ;

 srsubid | srrelid | srsubstate | srsublsn 

---------+---------+------------+-----------
 16394 | 16389 | r | 0/15F2898
 16394 | 16400 | d | 

-- Subscription state codes: i = initializing, d = copying data, s = synchronized, r = ready (normal replication)

-- At this point, table tab_pk data has not been synchronized because the subscriber's replication user lacks query permission on the table

lzldb=# grant select on tab_full to replicate1;

GRANT

lzlbd=# select * from pg_subscription_rel ;

 srsubid | srrelid | srsubstate | srsublsn 

---------+---------+------------+-----------
 16394 | 16389 | r | 0/15F2898
 16394 | 16400 | r | 0/172D830

-- Subscription is in ready state, new table synchronization complete

Replica Identity
#

Replica identity is written into WAL logs to identify a row of data. Whether it’s publish-subscribe or third-party logical sync tools, they all need to locate rows in the table to identify which row downstream the update or delete affects.

PostgreSQL supports 4 replica identity modes.

default(d): Default identity for non-system tables. Uses primary key if the table has one; if no primary key, it’s nothing.
index(i): Uses a non-null unique index as the identity. Must be non-null and unique to identify a row. If only unique, there can be multiple null values. You can also explicitly specify the primary key in index mode.
full(f): Uses all columns of the row as the identity. Full mode increases WAL log volume.
nothing(n): Default mode for system tables. No identity; update and delete cannot affect downstream.

-- View table's replica identity:
select relname,relreplident from pg_class where relname='tabname1';
-- When a table's replica identity is i, check if the index is the replica identity:
\d tabname
select rel.relname,idx.indisreplident from pg_index idx ,pg_class rel where idx.indexrelid=rel.oid and relname='idx_1';

Modify table replica identity:

ALTER TABLE tab1 REPLICA IDENTITY { DEFAULT | USING INDEX index_name | FULL | NOTHING };

Replica Identity Test 1: Setting a non-null unique index as replica identity for a table without a primary key
#

lzldb=# create table tab_idx(a int,b varchar(10));

CREATE TABLE

lzldb=# select relname,relreplident from pg_class where relname='tab_idx';

 relname | relreplident 

---------+--------------
 tab_idx | d

lzldb=# create unique index idx_1 on tab_idx(b);

CREATE INDEX

lzldb=# alter table tab_idx alter b set not null; -- The index used as replica identity must be a non-null unique index

ALTER TABLE

lzldb=# select rel.relname,idx.indisreplident from pg_index idx ,pg_class rel where idx.indexrelid=rel.oid and relname='idx_1';

 relname | indisreplident 

---------+----------------
 idx_1 | f

lzldb=# alter table tab_idx REPLICA IDENTITY using index idx_1; -- Modify table's replica identity

ALTER TABLE

lzldb=# select rel.relname,idx.indisreplident from pg_index idx ,pg_class rel where idx.indexrelid=rel.oid and relname='idx_1';

 relname | indisreplident 

---------+----------------
 idx_1 | t

lzldb=# \d tab_idx -- pg_index or \d to view index replica identity. \d can only display explicitly modified index replica identity

 Table "public.tab_idx"

 Column | Type | Collation | Nullable | Default 

--------+-----------------------+-----------+----------+---------
 a | integer | | | 
 b | character varying(10) | | not null | 

Indexes:
 "idx_1" UNIQUE, btree (b) REPLICA IDENTITY

Replica Identity Test 2: Full mode — can duplicate rows be synchronized normally?
#

-- Execute the following on the publisher

lzldb=# create table tab_full (a int,b varchar(10)); -- Add table sync without primary key and non-null index

CREATE TABLE

lzldb=# insert into tab_full values(1,'abc'); -- Insert 5 identical rows

INSERT 0 1

lzldb=# grant select on tab_full to replicate1;

GRANT

lzldb=# alter publication tab_full add table tab_pk;

ALTER PUBLICATION

--

lzlbd=# alter subscription sub_test refresh publication; 

ALTER SUBSCRIPTION

lzlbd=# select ctid,* from tab_full ;

 ctid | a | b 

-------+---+-----
 (0,1) | 1 | abc
 (0,2) | 1 | abc
 (0,3) | 1 | abc
 (0,4) | 1 | abc
 (0,5) | 1 | abc

lzldb=# delete from tab_full where ctid='(0,2)';

ERROR: cannot delete from table "tab_full" because it does not have a replica identity and publishes deletes

HINT: To enable deleting from the table, set REPLICA IDENTITY using ALTER TABLE.

lzldb=# update tab_full set a=2 where ctid='(0,5)';

ERROR: cannot update table "tab_full" because it does not have a replica identity and publishes updates

HINT: To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.

-- When the table's replica identity is d(default), without a primary key it's nothing. nothing cannot replicate delete and update.

lzldb=# alter table tab_full replica identity full;

ALTER TABLE

lzldb=# delete from tab_full where ctid='(0,2)'; -- After setting replica identity to full, delete succeeds

DELETE 1

lzlbd=# select ctid,* from tab_full ; --

 ctid | a | b 

-------+---+-----
 (0,2) | 1 | abc
 (0,3) | 1 | abc
 (0,4) | 1 | abc
 (0,5) | 1 | abc

lzldb=# update tab_full set a=2 where ctid='(0,5)'; 

UPDATE 1

lzldb=# select ctid,* from tab_full; 

 ctid | a | b 

-------+---+-----
 (0,1) | 1 | abc
 (0,3) | 1 | abc
 (0,4) | 1 | abc
 (0,6) | 2 | abc

lzlbd=# select ctid,* from tab_full ; 

 ctid | a | b 

-------+---+-----
 (0,3) | 1 | abc
 (0,4) | 1 | abc
 (0,5) | 1 | abc
 (0,6) | 2 | abc

– This example proves 3 points: – 1. When replica identity is d(default), it defaults to primary key; if no primary key, it’s nothing. – 2. nothing cannot replicate delete and update. – 3. Duplicate data in full mode can also be normally logically replicated. Although the ctid of data rows differs, the replication goal is still achieved.

Third-Party Synchronization Software
#

Third-party synchronization software already has relatively mature solutions and is widely used, such as OGG, DTS, KTL, etc.

These sync tools are very flexible. They can achieve true heterogeneous synchronization, from PostgreSQL databases to different databases or Kafka, big data consumption platforms, etc.

Of course, they can also sync from other architecture data platforms to PostgreSQL databases, such as the now common Oracle to PostgreSQL sync scenario.

Since we’re mainly discussing the PostgreSQL database itself, when PostgreSQL acts as the downstream target, it’s just some data write issues with very few problems. There won’t be logical decoding, replication slot issues, etc. So this small section won’t discuss PostgreSQL as a heterogeneous sync target. We’ll only observe and summarize scenarios where PostgreSQL acts as the upstream syncing to heterogeneous databases. These third-party tools generally utilize PostgreSQL’s own logical decoding, specify their own output plugin, and automatically create replication slots and replication links. Some tools automatically create subscriptions, while others only have replication slots without subscriptions.

Having already understood logical decoding, output plugins, replication slots, replica identity, and prerequisites for replication, let’s simulate a PostgreSQL to Oracle sync by directly configuring the prerequisites and starting synchronization.

Creating OGG Sync from PostgreSQL to Oracle
#

Software Installation:

ogg for oracle: Oracle GoldenGate 21.3.0.0.0 for Oracle on Linux x86-64

ogg for pg: Oracle GoldenGate 21.3.0.0.0 for PostgreSQL on Linux x86-64

oracle: 11.2.0.4

pg: 13.10

Installation steps:

OGG installation and deployment won’t be introduced here. I followed the article’s installation steps step by step. Installation article reference: https://liuzhilong.blog.csdn.net/article/details/129252320?spm=1001.2014.3001.5502

Sync architecture diagram:

lzldb=# select * from pg_replication_slots where slot_name='ext_pg_5d4b1d39f7494f79';

-[ RECORD 1 ]-------+------------------------
slot_name | ext_pg_5d4b1d39f7494f79
plugin | test_decoding -- OGG defaults to using test-decoding
slot_type | logical
datoid | 16385
database | lzldb
temporary | f
active | t -- As long as OGG extract is running, the replication slot is active
active_pid | 3509
xmin | 
catalog_xmin | 591
restart_lsn | 0/17F3E38
confirmed_flush_lsn | 0/17F4020
wal_status | reserved
safe_wal_size | 

select * from pg_stat_replication
-[ RECORD 2 ]----+------------------------------
pid | 3509
usesysid | 10
usename | pg
application_name |GoldenGateCapture
client_addr | 127.0.0.1
client_hostname | 
client_port | 43665
backend_start | 2023-02-28 15:12:17.350469+08
backend_xmin | 
state | streaming
sent_lsn | 0/17F4140
write_lsn | 0/17F4020
flush_lsn | 0/17F4020
replay_lsn | 
write_lag | 
flush_lag | 
replay_lag | 
sync_priority | 0
sync_state | async
reply_time | 2023-02-28 16:39:44.986625+08

-- replay_lsn has no value
-- Even lag has no value

Logical Replication Monitoring
#

An important method for logical replication lag monitoring is checking lag from the replication software. Without that, you can only check from the replication slot view. The replication slot view provides quite a lot of information, such as whether the replication slot is active directly indicating whether the replication link is syncing.

The replication slot view is very important for logical replication monitoring. Some additional monitoring for publish-subscribe was introduced earlier. Here we focus on broader logical replication monitoring.

pg_replication_slots
#

The replication slot view shows information about each replication slot and some slot statuses. Manually created slots or slots automatically created by tools and subscriptions are all displayed here.

slot_name	Replication slot name
plugin	Output plugin name for logical replication slots. If empty, it’s a physical replication slot
slot_type	physical or logical
datoid	Database ID for logical replication slot
database	Database for logical replication slot
temporary	Whether it’s a temporary replication slot. Temporary slots are not written to disk and are automatically deleted when the session ends. pg_basebackup uses temporary slots by default
active	Replication slot status: t or f. If f, you should quickly consider restarting the replication link or deleting it, as it may block WAL log deletion and fill up the primary database disk. This is related to the max_slot_wal_keep_size parameter
active_pid	walsender PID using this replication slot. Only present when the slot status is t
xmin	Minimum transaction ID the slot needs to hold
catalog_xmin	Minimum catalog transaction ID the slot needs to hold
restart_lsn	LSN position of WAL the slot needs to retain to ensure downstream consumer’s required WAL won’t be cleaned. max_slot_wal_keep_size parameter is the maximum WAL size the slot needs to retain. Beyond this value, WAL can also be deleted. Default -1 means never cleaned. This value represents the LSN position after the downstream’s latest checkpoint consumption and can help locate replication link lag
confirmed_flush_lsn	LSN confirmed received by the logical replication downstream. Empty for physical replication slots
wal_status	`Status of WAL claimed by this replication slot` reserved: the slot reserves WAL, WAL hasn’t exceeded max_wal_size (auto-checkpoint interval) extended: the slot reserves WAL, WAL has exceeded max_wal_size but the slot still retains it. WAL in this state is still within wal_keep_size or max_slot_wal_keep_size unreserved: the slot no longer retains needed WAL, WAL will be deleted at next checkpoint lost: WAL needed by the slot has been cleaned, slot is invalid. `The last two states are seen only when max_slot_wal_keep_size is non-negative. This is easy to understand, since max_slot_wal_keep_size is the criterion for whether WAL can be deleted. Without a mechanism to delete slot WAL, unreserved and lost states wouldn't appear.` `If restart_lsn is NULL, this field is null. Also easy to understand — if there's no WAL LSN, you can't know the WAL retention position or judge whether WAL has exceeded wal_keep_size or max_slot_wal_keep_size.`
safe_wal_size	Number of WAL bytes that can be written before WAL files would be deleted. If this value is negative or zero, it means max_slot_wal_keep_size has been exceeded, and WAL files will be deleted as soon as a checkpoint occurs, requiring the standby using this slot to be rebuilt

pg_stat_replication
#

Rather than replication status, it’s more accurate to call it walsender status. This view shows the status of each walsender, one record per walsender.

If present in pg_replication_slots but not in pg_stat_replication, the walsender is gone; logical replication is down; pg_replication_slots active should be f.
If absent in pg_replication_slots but present in pg_stat_replication, this is physical replication without a replication slot.

You can have replication stat info without a replication slot. Replication slots with walsenders also need this view because it reveals more replication status info than pg_replication_slots.

So when the replication slot hasn’t failed, pg_stat_replication is very important for monitoring logical replication lag.

pid	walsender PID, same as pg_replication_slots active_pid
usesysid	User OID connected to this walsender, i.e., the downstream’s replication user OID
usename	Username connected to this walsender
application_name	Downstream application name. If subscription, it’s the subscription name. If pg_recvlogical, it’s pg_recvlogical
client_addr	Downstream IP. If empty, it’s a local socket connection
client_hostname	Downstream hostname
client_port	Downstream port. If -1, it’s a local socket connection
backend_start	Backend start time, i.e., when downstream connected to walsender
backend_xmin	Standby’s xmin when hot_standby_feedback is enabled. This is clearly for physical replication
state	States are relatively easy to understand. startup: walsender starting. catchup: walsender catching up with primary logs. streaming: walsender has caught up with primary logs, normal replication state. backup: walsender sending backup, this state appears for walsender used for backup. stopping: walsender stopping
sent_lsn	LSN sent
write_lsn	LSN written to disk by downstream
flush_lsn	LSN flushed to disk by downstream
replay_lsn	LSN replayed by downstream
write_lag	Log lag between primary flush wal and downstream write
flush_lag	Log lag between primary flush wal and downstream flush
replay_lag	Log lag between primary flush wal and downstream relay
sync_priority	Synchronization priority
sync_state	Synchronization state
reply_time	Last reply time

Relationship between sent_lsn, write_lsn, flush_lsn, replay_lsn
#

The above nicely shows the hierarchical relationship of sent_lsn, write_lsn, flush_lsn.

These monitoring metrics look very much like streaming replication. For logical replication, sent_lsn, write_lsn, flush_lsn also generally have values.

However, when logical replication doesn’t know what the downstream is, the replay log replay action may not exist, so logical replication may not have replay_lsn.

But one thing is confirmed effective: sent_lsn.

After reviewing pg_replication_slots and pg_stat_replication view monitoring, we find that neither shows log parsing delay; at most, you can see log transmission delay.

pg_stat_replication_slots
#

This view has been available since pg14. It specifically monitors logical replication slot status and can additionally monitor spill status. For pg13, you can only check the pg_replslot directory. Spill will be introduced later.

Logical Replication Slot Transaction Snapshots and pg_logical Directory
#

The transaction snapshots needed by replication slots are persisted to disk. The source code is in snapbuild.c.

void
SnapBuildSerializationPoint(SnapBuild *builder, XLogRecPtr lsn)
{
if (builder->state < SNAPBUILD_CONSISTENT)
SnapBuildRestore(builder, lsn);
else
SnapBuildSerialize(builder, lsn);
}

Snap persistence has two behaviors: one is restore, loading from disk to memory; the other is serialize, persisting from memory to disk.

Transaction snapshot persistence:

SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
...
sprintf(path, "pg_logical/snapshots/%X-%X.snap",
(uint32) (lsn >> 32), (uint32) lsn);
...
else if (ret == 0)
{
/*
 * somebody else has already serialized to this point, don't overwrite
 * but remember location, so we don't need to read old data again.
 *
 * To be sure it has been synced to disk after the rename() from the
 * tempfile filename to the real filename, we just repeat the fsync.
 * That ought to be cheap because in most scenarios it should already
 * be safely on disk.
 */
fsync_fname(path, false);
fsync_fname("pg_logical/snapshots", true);

builder->last_serialized_snapshot = lsn;
goto out;
}

Transaction snapshot loading into memory:

SnapBuildRestore(SnapBuild *builder, XLogRecPtr lsn)
...
if (builder->state == SNAPBUILD_CONSISTENT)
return false;

sprintf(path, "pg_logical/snapshots/%X-%X.snap",
(uint32) (lsn >> 32), (uint32) lsn);

fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);

The transactions needed by logical replication slots, before being committed, store dirty transaction data and unconsumed data under pg_logical/snapshots/. After committing data or starting the replication slot, data is handed to reorderbuffer; or after cleaning the replication slot, the data is released.

My environment has a long-unused slot with restart_lsn at 0/1776858:

postgres=# select slot_name,plugin,slot_type,database,active,restart_lsn from pg_replication_slots where slot_name='logical_test';
 slot_name | plugin | slot_type | database | active | restart_lsn 
--------------+---------------+-----------+----------+--------+-------------
 logical_test | test_decoding | logical | lzldb | f | 0/1776858

The oldest snapshot under pg_logical/snapshots/ is it:

[pg@lzl snapshots]$ ll
total 300
-rw------- 1 pg pg 144 Feb 23 20:41 0-1776858.snap
-rw------- 1 pg pg 144 Feb 23 20:44 0-1776900.snap
-rw------- 1 pg pg 144 Feb 23 20:45 0-1776938.snap

Delete unwanted replication slot:

select pg_drop_replication_slot('logical_test');

After a few minutes, snap is deleted:

[pg@lzl snapshots]$ ll 0-1776858.snap
ls: cannot access 0-1776858.snap: No such file or directory

Logical Decoding Working Memory and Spill to pg_replslot
#

logical_decoding_work_mem
#

Before pg13, logical decoding would retain at most 4096 changes in memory (max_changes_in_memory hardcoded). Beyond 4096 changes, transaction data would be written to disk.

pg13 introduced the logical_decoding_work_mem parameter. Working memory used by logical decoding. All walsender decoding uses this shared memory area. If the data held by logical decoding exceeds this memory value, it’s written to disk. Logical decoding working memory size defaults to 64MB.

Related ReorderBuffer and Spill
#

Description in reorderbuffer.c:

 * This module gets handed individual pieces of transactions in the order
 * toplevel transaction sized pieces. When a transaction is completely
 * reassembled - signaled by reading the transaction commit record - it
 * will then call the output plugin (cf. ReorderBufferCommit()) with the
 * individual changes. The output plugins rely on snapshots built by
 * snapbuild.c which hands them to us.

When a transaction commits, reorderbuffer can receive transaction entries and sort them, then send data changes to the output plugin for output. The output plugin relies on snapshots built by snapbuild.c, which are handed to reorderbuffer.

/*
 * Maximum number of changes kept in memory, per transaction. After that,
 * changes are spooled to disk.
 *
 * The current value should be sufficient to decode the entire transaction
 * without hitting disk in OLTP workloads, while starting to spool to disk in
 * other workloads reasonably fast.
 *
 * At some point in the future it probably makes sense to have a more elaborate
 * resource management here, but it's not entirely clear what that would look
 * like.
 */
int logical_decoding_work_mem;
static const Size max_changes_in_memory = 4096; /* XXX for restore only */

When parsed data exceeds logical_decoding_work_mem, it’s written to disk. max_changes_in_memory is hardcoded at 4096, now only used to trigger disk restore. In pg12 source, there’s no int logical_decoding_work_mem, and subsequent serialization was also judged based on max_changes_in_memory.

In pg13, Disk serialization source code starts from line 2333. When parsed data in memory exceeds logical_decoding_work_mem, the largest transaction is spilled to disk. ReorderBufferLargestTXN(rb) finds the largest transaction. ReorderBufferSerializeTXN(rb, txn) persists this transaction. The immediately following code is ReorderBufferSerializeTXN():

/*
 * Spill data of a large transaction (and its subtransactions) to disk.
 */
static void
ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
dlist_iter subtxn_i;
dlist_mutable_iter change_i;
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;

elog(DEBUG2, "spill %u changes in XID %u to disk",
 (uint32) txn->nentries_mem, txn->xid);

/* do the same to all child TXs */
...

At debug2 level, spill logs are output:

/*
 * Given a replication slot, transaction ID and segment number, fill in the
 * corresponding spill file into 'path', which is a caller-owned buffer of size
 * at least MAXPGPATH.
 */
static void
ReorderBufferSerializedPath(char *path, ReplicationSlot *slot, TransactionId xid,
XLogSegNo segno)
{
XLogRecPtr recptr;

XLogSegNoOffsetToRecPtr(segno, 0, wal_segment_size, recptr);

snprintf(path, MAXPGPATH, "pg_replslot/%s/xid-%u-lsn-%X-%X.spill",
 NameStr(MyReplicationSlot->data.name),
 xid,
 (uint32) (recptr >> 32), (uint32) recptr);
}

Persisted to pg_replslot/replication_slot_name/xid-%u-lsn-%X-%X.spill.

Similarly, besides serialize, there’s also restore:

/*
 * Restore a number of changes spilled to disk back into memory.
 */
static Size
ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn,
TXNEntryFile *file, XLogSegNo *segno)
{
Size restored = 0;
XLogSegNo last_segno;
...
while (restored < max_changes_in_memory && *segno <= last_segno)
{
int readBytes;
ReorderBufferDiskChange *ondisk;

...
/*
 * Read the statically sized part of a change which has information
 * about the total size. If we couldn't read a record, we're at the
 * end of this file.
 */
ReorderBufferSerializeReserve(rb, sizeof(ReorderBufferDiskChange));
readBytes = FileRead(file->vfd, rb->outbuf,
 sizeof(ReorderBufferDiskChange),
 file->curOffset, WAIT_EVENT_REORDER_BUFFER_READ);
...
/*
 * ok, read a full change from disk, now restore it into proper
 * in-memory format
 */
ReorderBufferRestoreChange(rb, txn, rb->outbuf);
restored++;
}

return restored;
}

ReorderBufferRestoreChanges() just does judgment and looping (restored++), calling ReorderBufferRestoreChange():

static void
ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
 char *data)
{
...
/*
 * Update memory accounting for the restored change. We need to do this
 * although we don't check the memory limit when restoring the changes in
 * this branch (we only do that when initially queueing the changes after
 * decoding), because we will release the changes later, and that will
 * update the accounting too (subtracting the size from the counters). And
 * we don't want to underflow there.
 */
ReorderBufferChangeMemoryUpdate(rb, change, true,
ReorderBufferChangeSize(change));
}

Looking at ReorderBufferRestoreChanges(), its while loop judgment is restored < max_changes_in_memory, and restored starts at 0. It will loop 4096 times. There’s a comment in ReorderBufferRestoreChange explaining that although restore isn’t based on memory limit, it still needs to update memory usage to prevent underflow. Meaning: since I just restored it, don’t spill it again in a nested fashion. (It feels a bit odd — clearly judging by memory limit would be better rather than hardcoding the restore loop count.)

Interpreting the logical decoding process based on source code:

xtransaction snap preserves the metadata needed for parsing locks. When the replication slot is inactive or the transaction is uncommitted, snap persists to pg_logical/snapshots/%restart_lsn.snap. After the replication slot restarts or the transaction commits, the transaction snap metadata on disk is read into memory and sent to reorderbuffer for WAL parsing, sorted by transaction start order. If logical decoding data fills up the logical_decoding_work_mem memory area, change entries persist the largest transaction to pg_replslot/slot_name/xid-%u-lsn-%X-%X.spill, send other in-memory transactions to the output plugin for format conversion, and finally send the decoded information to the downstream.

In fact, we can see that long transactions and large transactions can make the entire logical replication link very slow. Large transactions are preferentially spilled to disk, then loaded back from disk to memory after the transaction completes.

Summary
#

Logical replication is managed through replication slots: one replication slot, one walsender process, one output plugin.
The output plugin determines the output form of logically decoded data, specified when creating the replication slot.
Replica identity priority recommendation: primary key -> non-null unique index -> full.
The publish-subscribe model is PostgreSQL’s built-in logical replication, using pgoutput by default. Publications can be used independently.
The publisher process is walsender, and the subscriber process is worker. Pay attention to their respective process parameters.
There are many third-party logical replication tools; they generally use PostgreSQL’s logical decoding system.
For monitoring replication links, pay attention to pg_replication_slots and pg_stat_replication.
The pg_logical directory stores transaction parsing metadata snaps, waiting for transaction commit before parsing.
The pg_replslot directory stores transaction information exceeding logical_decoding_work_mem, called spill.

References
#

Book: 《PostgreSQL实战》

Official Documentation:

PostgreSQL: Documentation: 15: Chapter 49. Logical Decoding

PostgreSQL: Documentation: 15: 49.1. Logical Decoding Examples

PostgreSQL: Documentation: 15: pg_recvlogical

PostgreSQL: Documentation: 14: 52.81. pg_replication_slots

PostgreSQL: Documentation: 13: 19.6. Replication

PostgreSQL: Documentation: 13: 48.6. Logical Decoding Output Plugins

PostgreSQL: Documentation: 15: 31.1. Publication

PostgreSQL: Documentation: 15: 31.2. Subscription

PostgreSQL: Documentation: 15: CREATE PUBLICATION

Highly Recommended:

https://www.pgconf.asia/JA/2017/wp-content/uploads/sites/2/2017/12/D2-A7-EN.pdf

Logical replication internals | Select * from Adrien

An Overview of Logical Replication in PostgreSQL - Highgo Software Inc.

Discussing Logical Decoding from Real Cases

Long-Troubling Logical Decoding Anomalies

Monitoring replication: pg_stat_replication - CYBERTEC

Other References:

https://zhuanlan.zhihu.com/p/311496301

A Guide to PostgreSQL Change Data Capture - DZone

Change data capture in Postgres: How to use logical decoding and wal2json - Microsoft Community Hub

PgSQL · The Secrets of PostgreSQL Logical Streaming Replication Technology · Database Kernel Monthly · KanCloud

Analyzing PostgreSQL Logical Replication Principles - CSDN Blog

http://pigsty.cc/zh/blog/2021/03/03/postgres逻辑复制详解/

Logical replication and logical decoding - Azure Database for PostgreSQL - Flexible Server | Microsoft Learn

PostgreSQL Streaming Replication

Tue, 13 Aug 2024 00:00:00 +0000

What is PostgreSQL Streaming Replication?
#

Streaming Replication is a method for transmitting WAL logs introduced in PostgreSQL 9.0. As soon as the primary database generates a log, it is immediately passed to the standby database. Before PostgreSQL 9.0, PostgreSQL could only transfer WAL logs one at a time (log shipping), and the standby database lagged behind the primary by at least one WAL log.

PostgreSQL Streaming Replication Processes
#

wal sender: The wal sender exists on the primary database. The wal sender process transmits the WAL between the primary’s latest LSN and the standby’s latest LSN to the standby. wal receiver: The wal receiver exists on the standby database. The wal receiver process transmits the standby’s latest LSN to the primary. The wal receiver receives WAL data passed by the wal sender and writes it to WAL logs. startup: The standby instance recovery process. It replays WAL logs on the standby database.

pg 16776 14632 0 13:33 ? 00:00:00 postgres: wal sender process lzl 172.17.100.150(13338) streaming 0/3002D30

pg 16775 15329 0 13:33 ? 00:00:00 postgres: wal receiver process streaming 0/3002D30
pg 15330 15329 0 10:26 ? 00:00:00 postgres: startup process recovering 000000010000000000000003

PostgreSQL Streaming Replication Principles
#

PostgreSQL streaming replication is primarily divided into two phases: the instance recovery phase and the primary-standby synchronization phase. Instance Recovery Phase: When a PostgreSQL database crashes abnormally, upon startup, PostgreSQL replays all WAL logs after the last checkpoint before the crash (this is the same principle as instance recovery in Oracle, MySQL, and other relational databases — the goal is to bring the database to a consistent state). When setting up a PostgreSQL standby database, the primary is generally not shut down. At this point, the backup taken from the primary is in an inconsistent state, and the startup process performs instance recovery when the standby starts. Primary-Standby Synchronization Phase: The wal receiver process transmits the standby’s latest LSN to the primary. The wal sender transmits the WAL between the primary’s latest LSN and the standby’s latest LSN to the wal receiver. The wal receiver receives the WAL and writes it to disk, and the startup process replays the WAL logs on the standby.

Synchronous and Asynchronous
#

PostgreSQL primary-standby has 5 modes, controlled by the synchronous_commit parameter. The essence of the synchronous_commit parameter is to control when the primary commits. remote_apply: The primary commits only after all standby databases have applied the WAL. This mode is synchronous — the primary and standby are consistent. Data that can be queried on the primary can definitely also be queried on the standby. In this mode there is no primary-standby lag, but it affects the primary commit time because the primary commit needs to wait for network transmission and standby application time.

The meaning of synchronous_commit has two scenarios: with and without standby databases (when synchronous_standby_names is empty or non-empty):

When synchronous_standby_names is non-empty: remote_apply: The standby has applied the WAL, only then can the primary commit. In this mode the primary and standby are synchronous. on: default. The primary commits when both primary and standby WAL have been written to disk. Similar to semi-synchronous, no data will be lost. remote_write: The primary commits when the standby has received the WAL and written the WAL log to the filesystem cache. At this point the standby has received the WAL but hasn’t flushed it to disk yet. If the OS crashes, data will be lost. local: The primary commits when its WAL is flushed to disk. This mode is asynchronous — the primary doesn’t need to confirm the standby’s status before committing. off: The primary can commit without its own WAL being flushed to disk. There is a risk of data loss. Not recommended.

When synchronous_standby_names is empty: (When synchronous_standby_names is empty, only on and off are effective for synchronous_commit. If set to remote_apply, remote_write, or local, they are still treated as on.) on: default. The database WAL must be written to disk before a transaction can commit. off: The primary can commit without its own WAL being flushed to disk. There is a risk of data loss. Not recommended.

Primary-Standby Synchronization Relationship

Primary-Standby Reliability

Failover
#

When the primary crashes, the standby needs to initiate failover, at which point the standby becomes the new primary. PostgreSQL does not provide a method to detect failures, but it does provide a method to activate the primary. (Typically, third-party tools call the PostgreSQL activation method, while primary-standby monitoring, primary crash detection, connection switching, etc. are not handled by PostgreSQL itself.) PostgreSQL provides 2 methods to activate a standby as the primary: the trigger_file file and the pg_ctl promote command. (In PostgreSQL 12 and later, trigger_file becomes promote_trigger_file.) Both trigger_file and pg_ctl promote can complete the task of activating the standby with a single command. The difference is that trigger_file requires the trigger_file configuration to be written in recovery.conf in advance. Using trigger_file for primary-standby switchover (pg_ctl promote has the same effect and is simpler):

Configure trigger_file in the standby’s recovery.conf
Shut down the primary
touch trigger_file to start the old standby as the new primary
Configure recovery.conf to start the old primary as the new standby
Observe the new and old primary/standby databases

Failover Example: Environment: Primary 172.17.100.150 5432 Standby 172.17.100.150 5433

1. Configure trigger_file in standby recovery.conf

$ cat recovery.conf|grep trigger
trigger_file = '/pg/pg96data_sla/trigger.kenyon'
$ ll /pg/pg96data_sla/trigger.kenyon
ls: cannot access /pg/pg96data_sla/trigger.kenyon: No such file or directory

Simply configure the trigger file path in recovery.conf. The trigger file won’t appear until it’s created.

Add configuration to standby postgres.conf

max_wal_senders = 6 #max_wal_senders is the maximum number of sender processes, default is 0, so the standby must configure this before switchover
hot_standby=on #Enable query functionality on standby

2. Shut down the primary

$ pg_ctl stop -D /pg/pg96data_pri -m fast
waiting for server to shut down.... done
server stopped

(Check if primary WAL has been fully applied by the standby: pg9.6- cd pg_xlog; pg 10+ cd pg_wal)

ls -ltr|tail -n 1 |awk '{print $NF}'|while read xlog;do pg_xlogdump $xlog;done

Look for the keyword “shutdown” in the standby’s WAL

3. touch to activate standby (or pg_ctl promote -D /pg/pg96data_sla)

$ touch /pg/pg96data_sla/trigger.kenyon

At this point recovery.conf becomes recovery.done

4. Set up primary as standby Configure the new standby’s recovery.conf file. You can directly copy from the old standby and modify the IP and directory.

vi $新备库/recover.conf
standby_mode = on
primary_conninfo = 'host=172.17.100.150 port=5433 user=lzl password=lzl'
recovery_target_timeline = 'latest'

Configure postgres.conf, write hot_standby = on to enable queries on the standby

vi $新备库/postgres.conf
hot_standby = on

Start the new standby

/pg/pg96/bin/pg_ctl -D /pg/pg96data_pri -l /pg/pg96data_pri/server.log start

5. Check primary and standby

postgres=# \x
Expanded display is on.
postgres=# select * from pg_stat_replication ;
-[ RECORD 1 ]----+------------------------------
pid | 24766
usesysid | 16384
usename | lzl
application_name | walreceiver
client_addr | 172.17.100.150
client_hostname | 
client_port | 47345
backend_start | 2021-07-30 07:44:05.582546+00
backend_xmin | 
state | streaming
sent_location | 0/4033790
write_location | 0/4033790
flush_location | 0/4033790
replay_location | 0/4033790
sync_priority | 0
sync_state | async

pg_basebackup
#

pg_basebackup is PostgreSQL’s built-in backup tool for performing base backups. pg_basebackup can be used for PITR and also for constructing log-shipping standby and streaming standby. It is PostgreSQL’s physical backup tool. https://liuzhilong.blog.csdn.net/article/details/119533506

pg_rewind
#

pg_rewind can be used as a maintenance tool for PostgreSQL primary-standby setups. When the timelines of two PostgreSQL instances diverge, pg_rewind can synchronize between the instances. (For example, if the standby is running after failover while the primary was still running, the timelines of primary and standby will have diverged.) https://liuzhilong.blog.csdn.net/article/details/119250794

Replication Slots
#

What are PostgreSQL Replication Slots? In a primary-standby architecture, if the standby hasn’t received WAL logs yet but the primary has already deleted them, such lag cannot be automatically recovered. Replication slots ensure that the primary won’t delete WAL logs that haven’t been transmitted to the standby yet. Without replication slots, you might need to use wal_keep_size/wal_keep_segments and archive_command to ensure WAL logs aren’t deleted, but this approach always retains too many WAL files and cannot guarantee that WAL won’t be deleted when lag is significant. This is exactly why replication slots were created. However, replication slots may cause the primary to never delete WAL (e.g., if the standby has crashed), causing disk space to fill up. In this case, max_slot_wal_keep_size is needed to set an upper limit on WAL file retention.

Replication Slot Parameters: max_slot_wal_keep_size: When replication slots are in use, this parameter defines the maximum size of WAL files in the pg_wal directory. The default value is -1, meaning there is no upper limit on the size of WAL files retained by the primary for the standby. wal_keep_segments/wal_keep_size: PostgreSQL 12 and below use wal_keep_segments, PostgreSQL 13 and above use wal_keep_size. Ensures that WAL files under pg_wal are not deleted. Without replication slots, WAL files exceeding this size may be deleted, potentially causing the standby to be unable to catch up. If set too large, it may cause the directory to grow excessively. The default is 0, meaning WAL files are not retained. If WAL is deleted, the following error may occur: ERROR: requested WAL segment xxxx has already been removed At this point the standby can only hope for archives; otherwise, it must be rebuilt. primary_slot_name: Sets the slot name, indicating that the PostgreSQL primary-standby setup uses replication slots. So enabling PostgreSQL replication slots requires at least the following configuration: primary_conninfo = ‘host=172.17.100.150 port=5433 user=lzl password=lzl’ primary_slot_name = ‘pg_slot_lzl’ max_replication_slots: The maximum number of replication slots. Takes effect upon restart. If there aren’t enough replication slots, the standby will fail to start. This value should be set relatively high. In PostgreSQL versions below 9.6, the default is 0; in PostgreSQL 10 and above, it’s 10.

Creating PostgreSQL Replication Slots

1. Set max_replication_slots on the primary Primary: (my PostgreSQL version is 9.6) max_replication_slots=10 Add to postgres.conf and restart the primary

2. Create replication slot Create replication slot:

postgres=# SELECT * FROM pg_create_physical_replication_slot('pg_slot_lzl');
 slot_name | xlog_position 
-------------+---------------
 pg_slot_lzl |

View replication slot

postgres=# SELECT slot_name, slot_type, active FROM pg_replication_slots;
 slot_name | slot_type | active 
-------------+-----------+--------
 pg_slot_lzl | physical | f

3. Set primary_slot_name on the standby primary_slot_name = 'pg_slot_lzl' Add to recovery.conf and restart the standby

4. Check replication slot

postgres=# select *,pg_xlogfile_name(restart_lsn)as current_xxlog from pg_replication_slots;
 slot_name | plugin | slot_type | datoid | database | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn | current_xxlog 
-------------+--------+-----------+--------+----------+--------+------------+------+--------------+-------------+---------------------+--------------------------
 pg_slot_lzl | | physical | | | t | 12802 | | | 0/A002340 | | 00000002000000000000000A
--pg_xlogfile_name(restart_lsn) to view current WAL log info

Query Conflicts
#

What are Query Conflicts? The standby may encounter the following error during queries: ERROR：canceling statement due to conflict with recovery

Why do conflicts occur? Let’s think carefully. For example, if the standby is executing a query based on a certain table (this query could be from an application or a manual connection), and the primary executes a drop table operation, this operation is written to WAL logs and transmitted to the standby for application. To ensure data consistency, PostgreSQL will inevitably replay the data quickly, at which point the drop table and select will conflict, as shown below:

Conflict scenarios: The above only introduces one type of query conflict. To summarize, there are several situations:

Primary exclusive locks (including explicit LOCK commands and various DDL operations)
Primary vacuum cleaning up dead tuples — if the standby is using those tuples, a conflict will occur
Primary drops the tablespace that the standby query is using
Primary drops the database that the standby is using

Consider a primary-only scenario: Scenario 1: A session issues a drop table and finds that a select statement is currently executing. The session can only wait for the select to complete its transaction. Scenario 2: A session issues a vacuum or automatic background vacuum — it won’t conflict with current database queries because vacuum won’t clean up tuples that are in use.

The standby’s handling is different. Because the primary doesn’t know the standby’s transaction status, and the standby needs to stay consistent with the primary, this is why “query conflicts” occur.

Query Conflict Parameters hot_standby_feedback: This is the most frequently mentioned parameter in the topic of query conflicts. Let’s explore it in detail below. Suppose, without a standby, Session 1 queries a row of data, Session 2 deletes that data and commits. Then Session 2 performs a vacuum. We know this vacuum won’t delete that row because Session 1’s transaction still needs to use that tuple, so it won’t be cleaned up. What about in a primary-standby setup? How does the primary know that the standby is still querying when it’s about to perform a vacuum? This is the purpose of this parameter. After setting hot_standby_feedback, the standby will periodically notify the primary of the minimum active transaction ID (xmin) value, so the primary vacuum process won’t clean up tuples with values greater than xmin. This parameter helps reduce conflicts but cannot completely avoid them. If you think about it carefully, this parameter only reduces conflicts caused by the primary vacuuming dead tuples — it cannot resolve conflicts caused by exclusive locks. Or conflicts caused by network interruptions: if the network between primary and standby is interrupted, the standby cannot send the xmin value to the primary normally. If the interruption is long enough, the primary will still clean up useless tuples during this period, and after the network recovers, the vacuum conflict described above may occur. It’s worth noting that the hot_standby_feedback parameter won’t override the value limited by the old_snapshot_threshold parameter on the primary. The old_snapshot_threshold parameter limits the infinite expansion of dead tuples. When transaction information exceeds the old_snapshot_threshold limit, cleanup will still occur.

max_standby_streaming_delay: The waiting time before the standby cancels a query due to a conflict caused by receiving WAL stream logs. Setting this parameter means that when a conflict occurs, the standby query won’t be immediately canceled but will wait for a period before throwing an error if it hasn’t finished. The value can be set based on the expected runtime of potential long transactions on the standby.

max_standby_archive_delay: The waiting time before the standby cancels a query due to a conflict caused by processing archived WAL logs. Similar to the parameter above.

vacuum_defer_cleanup_age: Specifies the number of transactions by which vacuum delays cleaning up dead tuples. Vacuum will delay clearing invalid records. The number of deferred transactions is set through vacuum_defer_cleanup_age. That is, vacuum and vacuum full operations won’t immediately clean up recently deleted tuples.

You can view conflict occurrences through the pg_stat_database and pg_stat_database_conflicts views.

Other Related Parameters
#

Transmission Parameters max_wal_senders: The maximum number of services that can fetch WAL using wal sender, i.e., the maximum number of standby databases + basebackup clients. PostgreSQL 9.6 defaults to 0; PostgreSQL 10 and later default to 10. wal_send_timeout: Interrupt replication after WAL transmission fails for xx seconds. When the standby crashes or the network is interrupted for a long time, WAL will no longer attempt transmission. Default is 60. 0 means never interrupt replication. track_commit_timestamp: Record transaction timestamps. Default is off.

Primary Parameters synchronous_standby_names: Configured on the primary. The standby replication list. There are several forms (s1, s2, s3 represent the standby’s application_name, configured in recovery.conf): synchronous_standby_names=‘s1’ means the primary can commit when s1 standby returns. synchronous_standby_names=‘FIRST 2 (s1,s2,s3)’ means the primary can commit when the first two of the three standbys (s1 and s2) return. synchronous_standby_names=‘ANY 2 (s1,s2,s3)’ means the primary can commit when any two of the three standbys return. synchronous_standby_names=’’ means matching any host — the primary can commit when any host returns. wal_level: WAL log level. This parameter determines how much information is written to WAL logs. The default is replica, which supports replication and WAL archiving while also supporting standby read-only queries. minimal: Other than records needed for instance crash recovery, nothing else is recorded. For example, CREATE TABLE AS, CREATE INDEX, CLUSTER, COPY can be skipped. The log information recorded in this mode is insufficient to support WAL archiving and streaming replication. logical: Adds additional information on top of replica to support logical decoding. This mode increases WAL log volume, especially for databases with many UPDATE and DELETE operations. Before PostgreSQL 9.6, there were also archive and hot_standby modes, which map to the current replica mode. synchronous_commit: As discussed earlier, 5 modes, each with pros and cons. archive_mode: archive_mode = on enables archiving. archive_command: Archiving command. PostgreSQL archiving directly calls operating system commands. Can be a simple cp command to the backup side. listen_addresses: Listening addresses. ‘’ means listen on all IPs. Default is local.

Standby Parameters hot_standby: on enables standby read-only queries. primary_conninfo: The connection string for the standby to connect to the primary. E.g., primary_conninfo = ‘host=172.17.100.150 port=5432 user=lzl password=lzl’. trigger_file/promote_trigger_file: The trigger file for activating the standby. Before PostgreSQL 12 it’s called trigger_file; PostgreSQL 12 and later use promote_trigger_file. Both trigger_file and pg_ctl promote can activate the standby with a single command, as demonstrated earlier. wal_receiver_create_temp_slot: When there is no slot, temporarily create one (named after primary_slot_name). Default is off.

References:
#

《The Way of PostgreSQL》(修炼之道) https://www.postgresql.org/docs/current/warm-standby.html

https://www.postgresql.org/docs/13/high-availability.html

https://www.postgresql.org/docs/current/runtime-config-replication.html

https://www.postgresql.org/docs/13/runtime-config-wal.html

https://www.postgresql.org/docs/current/app-pgbasebackup.html

https://www.postgresql.org/docs/current/hot-standby.html#HOT-STANDBY-CONFLICT

https://cloud.tencent.com/developer/article/1555354

https://www.modb.pro/db/29737

https://wiki.postgresql.org/wiki/Streaming_Replication

https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/

https://www.cybertec-postgresql.com/en/the-synchronous_commit-parameter/

https://blog.csdn.net/m15217321304/article/details/88850146

https://blog.51cto.com/lishiyan/2460518?source=dra

A Brief Analysis of Linux Memory

Mon, 12 Aug 2024 00:00:00 +0000

Basic Memory Concepts
#

Operating system memory is very important and fairly complex. Many knowledge points need to be mastered to further analyze program issues. Since this is the first comprehensive and systematic exposure to OS memory, the goal is to understand Linux memory concepts thoroughly and at a low level without diving deep into principles, so this chapter will also try to avoid Linux source code knowledge.

Physical Memory and Virtual Memory
#

(https://en.wikipedia.org/wiki/Memory_address)

Physical Memory: Physical memory is the actual hardware memory present in a computer system, typically in the form of RAM (Random Access Memory).

Virtual Memory: Virtual memory is a linear region that has not been allocated actual physical memory. Programs think they have a larger address space than the actual physical memory. The implementation of virtual memory allows programs to access a larger address range than physical memory without requiring all data to be present in physical memory simultaneously. The kernel releases physical pages by releasing linear regions, finding the corresponding physical pages, and releasing them all.

Memory Management Unit (MMU): A hardware component responsible for converting virtual addresses used by programs into physical addresses where data is actually stored in physical memory. The MMU’s primary task is to perform address mapping.

Page Table: A page table is a data structure used to store the mapping between virtual address space and physical address space. When a program attempts to access virtual memory, the MMU determines the corresponding physical address by querying the page table.

System call flow: https://users.cs.utah.edu/~aburtsev/cs5460/lectures/lecture19-memory-management/lecture19-memory-management.pdf

(The image is a bit blurry, the topmost text is “User Space|Kernel Space”)

User programs can only access the kernel system through C libraries or system calls; user programs cannot directly access the kernel system
The kernel system accesses physical memory through the MMU; it accesses disks and other external devices through drivers
The virtual memory system (VM Subsystem in the figure above) includes buddy, slab algorithms, etc.

User Space and Kernel Space
#

The process virtual address space is divided into user space and kernel space.

User Space:

The space where user processes run in memory
This portion of space is protected, and the system prevents other processes from accessing it (except for shared memory)
However, kernel processes can directly access user processes

Kernel Space:

Kernel space is the space used by kernel processes
In kernel space, the operating system’s kernel code runs with higher privilege levels, allowing direct access to system hardware, process management, file system operations, etc.

Context Switching:

When a user program needs to access system services or perform operations requiring higher privileges, a context switch from user space to kernel space is triggered.
Context switching is an operating system mechanism for saving and restoring program state, ensuring no data loss occurs when switching between user programs and the kernel.

The division between user space and kernel space is to provide security isolation, preventing user programs from directly affecting critical parts of the operating system. Early operating systems and DOS did not distinguish between kernel and user space, so a single program’s error or malicious behavior could affect the entire system.

(https://www.zhihu.com/tardis/zm/art/66794639?source_id=1003)

32-bit systems: Total 4GB address space, 3G UserSpace | 1G KernelSpace

64-bit systems: Total 256TB address space, 128T UserSpace | 128T KernelSpace

2^32=4GB, 2^64=16777216TB, why does a 64-bit system only have 256TB address space?

The 64-bit computing wiki has an explanation. In short, 256TB (256 × 1024^4 bytes) of memory addresses is sufficient, and currently and in the imaginable future there won’t be 16EB (16 × 1024^6 bytes) of memory.

Process Virtual Address Space
#

Each process typically has its own independent virtual memory space. Virtual memory is an abstract concept that provides each running process with an address space that appears continuous and private, making each process feel like it has the entire computer system’s full memory.

Process virtual address space layout:

(https://www.sohu.com/a/392831824_467784)

The mmap mapping region expands from top to bottom, and the mmap mapping region and heap expand relative to each other until the remaining area in the virtual address space is exhausted. This structure facilitates the C runtime library’s use of the mmap mapping region and heap for memory allocation.
Stack: Stores local variables and function parameters during program execution, growing from high addresses to low addresses
Heap: Dynamic memory allocation area, managed through functions like malloc, new, free, and delete
BSS (Uninitialized Variables): Stores uninitialized global variables and static variables
Data: Stores global variables and static variables with predefined values in source code
Text: Stores read-only program execution code, i.e., machine instructions

Process virtual address space distribution and mapping:

(https://velog.io/@mysprtlty/%EA%B0%80%EC%83%81-%EB%A9%94%EB%AA%A8%EB%A6%AC%EC%99%80-%EA%B0%80%EC%83%81-%EC%A3%BC%EC%86%8C-%EA%B3%B5%EA%B0%84)

Shared Memory
#

As mentioned earlier, the user space in the virtual address space cannot be accessed by other user processes. If multi-process user access to the same memory data is implemented through the kernel area, context switching cannot be avoided. Multi-process applications clearly need inter-process access, so a method that directly allows user processes to access the same physical memory emerged — this is shared memory.

Shared memory is one of the mechanisms for implementing IPC (Inter Process Communication), with other methods including message queues and semaphores.

(https://www.geeksforgeeks.org/inter-process-communication-ipc/)

Since it is inherently multiple virtual memory address spaces corresponding to one physical memory address space, you just need to point a segment in the address spaces of two processes to the same physical memory.

(https://www.softprayog.in/programming/interprocess-communication-using-system-v-shared-memory-in-linux)

Shared memory (seems like) has many implementation methods. For example, PostgreSQL defaults to using mmap to implement shared memory, refer to the shared_memory_type parameter and Managing Kernel Resources. Other shared memory implementations can be found in this article: Song Baohua: The Best Shared Memory in the World (The Most Thorough Linux Shared Memory Article)

Page Table
#

The process virtual address space is per-process, while there is only one physical memory space. So how do you map and convert virtual memory and shared memory?

(https://courses.engr.illinois.edu/cs241/sp2014/lecture/09-VirtualMemory_II_sol.pdf)

The page table is where the correspondence between virtual memory addresses and physical memory addresses is stored. (There are concepts like MMU and TLB here — let’s simplify and just think of it as the virtual-to-physical memory conversion function (PAGING), and only look at the page table here). A page table consists of a set of Page Table Entries (PTEs), with each PTE storing the map between a virtual page and a physical page.

Although a single page table can implement memory-to-virtual-memory conversion, implementing it directly this way would consume too much memory for the page table itself.

(https://courses.engr.illinois.edu/cs241/sp2014/lecture/09-VirtualMemory_II_sol.pdf)

Therefore, the single page table needs to be subdivided: two-level page tables and four-level page tables.

Two-level page tables:

A two-level page table is a further subdivision of a single page table. 4G of space requires 4M of page tables to store the mapping table. If these 4M are divided into 1K pages (4K each), these 1K pages also need a table for management, which we call the page directory table. This page directory table has 1K entries, each 4 bytes, making the page directory table size 4K as well.

Four-level page tables:

For 64-bit systems, two-level page tables are insufficient; four-level page tables are needed.

(https://maodanp.github.io/2019/06/02/linux-virtual-space/)

Check page table size:

[pg@lzl 2345]$ cat /proc/meminfo |grep PageTables
PageTables: 46736 kB

NUMA
#

Uniform Memory Access (UMA): All CPUs have equivalent access time to memory. The problem with UMA is that multiple processors access memory through a single bus, increasing the load on the shared bus. Multiple processors contend for the memory controller causing conflicts. Additionally, the bus bandwidth is limited, leading to access delays.

Non-Uniform Memory Access (NUMA): A small group of CPUs access their own local memory together. When there are multiple groups of CPUs and their memory groups, each group of CPUs and memory constitutes a NUMA node.

UMA:

NUMA:

(https://users.cs.utah.edu/~aburtsev/cs5460/lectures/lecture19-memory-management/lecture19-memory-management.pdf)

Basic NUMA characteristics:

CPU access to local node memory is faster than remote
By default, Linux prioritizes allocating local memory on the CPU; the policy can be configured
Each node has its own memory structure
NUMA is not suitable for all scenarios; it requires adaptation by upper-layer applications

NUMA balancing: Achieves local access by automatically transferring tasks to remote CPUs or copying remote data to local memory. Enabled by default on Red Hat 7.

Transferring tasks or copying data itself consumes resources and can slow down tasks. This feature may not be suitable for some applications; for example, Oracle’s Exadata has targeted NUMA optimizations.

numactl: NUMA OS configuration tool.

numactl --show displays CPU and node information. Below is an example of 4 nodes with 64c 256g total, each node having 16c 64g:

available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 32 33 34 35 36 37 38 39
node 0 size: 65418 MB
node 0 free: 310 MB
node 1 cpus: 8 9 10 11 12 13 14 15 40 41 42 43 44 45 46 47
node 1 size: 65536 MB
node 1 free: 41 MB
node 2 cpus: 16 17 18 19 20 21 22 23 48 49 50 51 52 53 54 55
node 2 size: 65536 MB
node 2 free: 82 MB
node 3 cpus: 24 25 26 27 28 29 30 31 56 57 58 59 60 61 62 63
node 3 size: 65536 MB
node 3 free: 43 MB

Zone
#

NUMA divides CPUs and memory into multiple nodes (node 0, node 1, node 2…). In UMA structures, the CPU memory as a whole can be viewed as node 0.

In Linux, each node is represented by the data structure struct pglist_data, with the data type typedef pg_data_t. Each node is further divided into multiple zones. A zone’s data structure is zone_t, with the data type zone_struct. There are generally 3 types: ZONE_DMA, ZONE_NORMAL, ZONE_HIGHMEM, each with different functions.

(https://www.kernel.org/doc/gorman/html/understand/understand005.html)

Zone distribution and functions in 32-bit:

ZONE_DMA: (<16MB), Direct Memory Access (DMA), the ancient 16 MiB limit, includes ISA devices.
ZONE_DMA32: Since many devices encounter problems accessing memory that cannot be addressed with 32 bits, this zone was added in x86-64. This zone only exists in x86-64 architecture. (See ZONE_DMA32)
ZONE_NORMAL: (16MB to 896MB), ordinary memory domain that can be directly mapped to the kernel segment; most kernel operations take place in the NORMAL zone, this is the most important zone
ZONE_HIGHMEM: (>896MB), marks physical memory beyond the kernel segment, cannot be directly called by the kernel.

Zone distribution diagram for 32-bit and 64-bit:

https://users.cs.utah.edu/~aburtsev/cs5460/lectures/lecture19-memory-management/lecture19-memory-management.pdf

Note that zones are for physical memory. Virtual memory must switch from user mode to kernel mode before it can call physical memory. The following diagram shows the relationship between kernel addresses in virtual memory space and zones in physical address space:

(https://wr.informatik.uni-hamburg.de/_media/teaching/wintersemester_2014_2015/kp-1415-memory-management.pdf)

Inspect zones:

cat /proc/zoneinfo

cat /proc/buddyinfo

cat /proc/pagetypeinfo

$ cat /proc/buddyinfo 
Node 0, zone DMA 1 1 1 1 1 0 1 1 1 1 2 
Node 0, zone DMA32 688 2080 1420 995 596 357 278 241 276 32 133 
Node 0, zone Normal 195748 204074 161167 119070 70791 33578 9556 2070 1034 2533 7328 
Node 1, zone Normal 11705 51467 36752 21326 11343 7309 5024 3403 2597 3056 10898

Pages
#

Virtual memory and physical memory are divided into fixed-size segments, typically 4KB in size. So after virtual memory is divided, we have virtual pages, and after physical memory is divided, we have physical pages (PP or PF, Physical Page or Page Frame), also called page frames, also 4KB. The page frame represents the minimum unit of system memory.

Each page in the virtual address space can be mapped to a page frame in the physical address space through its descriptor.

Huge Pages / Transparent Huge Pages
#

Pages are the minimum unit of memory allocation (default 4K). When mapping and allocating a large number of contiguous pages, performance is poor. Huge Pages solve this problem. Huge pages are not only cheaper to allocate, but the page table is also relatively smaller. hugepagesz is 2 MB or 1 GB, defaulting to 2MB. Huge Pages were implemented starting from Red Hat 6.

Since manually managing huge pages is cumbersome, Red Hat 6 also provided automatic huge page management, i.e., Transparent Huge Pages.

In Oracle database management, huge pages are generally enabled for SGA use, while transparent huge pages are disabled. There is plenty of related material available for searching.

Similarly, PostgreSQL can also enable huge pages. Since databases generally occupy more operating system memory, enabling huge pages for databases can generally reduce memory allocation pressure.

File Pages & Page Cache / Anonymous Pages & Swap Cache
#

File pages can be mapped to files on disk. File system reads and writes use Page Cache as buffered IO. Dirty data is synced (or fsynced, etc.) to the corresponding disk periodically or when called. Page Cache is the memory area used to “boost” disk performance.

Correspondingly, pages without associated files are called Anonymous Pages, generally corresponding to heap and stack. When memory resources are tight, the kernel writes infrequently used anonymous page data to swap partitions or swap files.

In short:

Page cache corresponds to file mappings
Swap cache corresponds to anonymous pages

(https://www.slideshare.net/raghusiddarth/memory-management-in-linux-11551521?from_search=2)

The above page cache diagram is from the operating system’s perspective. Application (such as database) writes can also be non-delayed, or even bypass Page Cache.

Memory Allocation
#

Memory allocation is also very complex, involving many concepts. Two common memory allocation methods are buddy and slab.

Buddy
#

The buddy system is used for allocating contiguous memory pages. Each zone has its own buddy system. The buddy system divides large blocks of memory to respond to memory allocation requests, and due to its coalescing characteristics, it can reduce system memory fragmentation.

The buddy allocator divides memory into pages of powers of 2, with the maximum order being 10:

When a memory request is larger than the existing block size, the system splits the larger block into two equally sized buddy blocks. When memory is freed, the system attempts to merge adjacent buddy blocks into a larger block:

When freeing a page, the page is directly placed back into the free list. If the other half of the previously split page is also unallocated, they are combined into a double-sized page and given to the next larger list, and so on, until it can no longer be merged or has reached the top.

When higher-order pages are depleted due to continuous allocation, fragmentation issues arise when requesting higher-order pages:

After waiting for memory reclamation to succeed, buddy itself merges lower orders into higher orders, then allocates higher-order pages:

(The implementations of anti pages fragmentation in Linux kernel https://teawater.github.io/presentation/antif.pdf)

However, memory reclamation may also not keep up with allocation speed, so the buddy system is not always ideal.

Analysis example:

$ cat /proc/buddyinfo 
Node 0, zone DMA 0 0 0 1 2 1 1 0 1 1 3 
Node 0, zone DMA32 7 6 5 6 5 6 7 7 6 2 272 
Node 0, zone Normal 317681 38869 31620 19250 8931 2579 815 182 19 5 0 

The above contains 3 ZONEs: DMA, DMA32, Normal
Orders: 0 ~ 10, i.e., the count of each order in buddy. The maximum order of buddy is 10, i.e., 1024 pages, which is 4MB
For example, the 3rd column in the Normal row indicates there are 31620 blocks of 2^2 contiguous memory available
By extension, the further back, the more contiguous the space. The larger the number, the more contiguous space of that size there is. When large contiguous spaces are scarce, it indicates significant memory fragmentation
Additionally, summing everything up gives the current free memory

Judging memory fragmentation issues through buddyinfo:

#host 1
Node 0, zone Normal 317681 38869 31620 19250 8931 2579 815 182 19 5 0 
#host 2
Node 0, zone Normal 7321 7833 10885 8514 2311 1644 1663 1302 1141 7384 80675 

The above shows the memory conditions of two hosts. Comparing them, the host below has more contiguous memory, while the host above has memory fragmentation issues.

Slab
#

The slab allocator manages memory based on objects. The slab system is a memory allocation algorithm specifically designed for kernel memory. It works by dividing memory into fixed-size caches, where each slab contains a set of objects of the same type. When there is a memory request, the algorithm first checks if available objects exist in the appropriate slab cache. If they exist, the object is returned. If not, the algorithm allocates a new slab and adds it to the appropriate cache.

Objects of different sizes correspond to different slab caches:

(https://bootlin.com/doc/training/linux-kernel/linux-kernel-slides.pdf)

Although slab has different caches and objects, slab still uses physically contiguous memory:

(https://i.stack.imgur.com/wo8Gg.png)

Slab also has 3 implementation methods:

Memory Reclamation
#

Memory Reclamation Overview
#

When system memory pressure is high, memory reclamation is performed on each zone under pressure. Memory reclamation mainly targets anonymous pages and file pages.
For anonymous pages, during memory reclamation, some infrequently used anonymous pages are selected, written to the swap partition, and then released as free page frames to the buddy system.
For file pages, during memory reclamation, some infrequently used file pages are also selected:
- If the content saved in this file page is consistent with the corresponding file content on disk, this file page is a clean file page and does not need to be written back; it is directly released as a free page frame to the buddy system.
- If the data saved in the file page is inconsistent with the corresponding data in the file on disk, this file page is considered a dirty page. It must first be written back to the corresponding data location on disk, and then released as a free page frame to the buddy system.
After memory reclamation completes, the number of free page frames in the system increases, alleviating memory pressure. However, the reclamation process puts significant IO pressure on the system. Therefore, a threshold is set for each zone in the system. When the number of free page frames falls below this threshold, memory reclamation operations are performed. When the number of free page frames meets this threshold, the system does not perform memory reclamation operations.

Zone Watermarks and kswapd
#

(https://vivani.net/2022/06/14/linux-kernel-tuning-page-allocation-failure/)

When available memory is low, the kswapd daemon is awakened to free pages.

pages_low: When the number of available free pages falls below pages_low, the buddy allocator wakes up the kswapd process, and the kernel begins swapping pages out to disk.
pages_min: When the number of available pages reaches pages_min, the pressure of page reclamation work is relatively high because the memory zone urgently needs free pages. The allocator will execute kswapd work in a synchronous manner, sometimes called direct reclaim.
pages_high: Once kswapd is awakened and begins freeing pages, the kernel considers the zone “balanced” only when the number of available pages reaches pages_high. If the watermark reaches pages_high, kswapd will re-enter the sleep state. If free pages exceed pages_high, the kernel considers the zone state ideal.

Memory reclamation is performed on a per-zone basis. /proc/zoneinfo can display the values of min, low, and high.

vm.min_free_kbytes is the min_pages watermark, a very important OS parameter. Very low values prevent the system from effectively reclaiming memory, potentially leading to system crashes and service interruptions. Too high values increase system reclamation activity, causing allocation delays, which may lead the system to immediately enter an out-of-memory state.

Types of Memory Allocation and Reclamation
#

Fast Memory Allocation: Performed by the get_page_from_freelist() function, which obtains a suitable zone from the zonelist using the low threshold for allocation. If the zone has not reached the low threshold, fast memory reclamation is performed, and allocation is retried after fast memory reclamation.

Slow Memory Allocation: When fast allocation fails, meaning no zone in the zonelist obtained memory in fast allocation, the min threshold is used for slow allocation. During slow allocation, three main things happen: asynchronous memory compaction, direct memory reclamation, and light synchronous memory compaction. Finally, OOM allocation may occur depending on the situation. And after each of these operations, fast memory allocation is called once to attempt to obtain page frames.

(https://blog.csdn.net/weixin_35094083/article/details/116688112)

Different memory allocation paths trigger different memory reclamation methods. Zone memory reclamation is divided into two types:

Background Memory Reclamation (kswapd): When physical memory is tight, the kswapd kernel thread is awakened to reclaim memory. This memory reclamation process is asynchronous and does not block process execution.
Direct Memory Reclamation (direct reclaim): If background asynchronous reclamation cannot keep up with process memory application speed, direct reclamation begins. This memory reclamation process is synchronous and blocks process execution.

Memory Compaction
#

Memory compaction: see Memory Monitoring - /proc/pagetypeinfo section

LRU
#

For zone memory reclamation, it targets three things for reclamation: slab, pages in LRU lists, and buffer_head. Here we only discuss memory reclamation targeting LRU lists.

The main purpose of LRU lists is to sort pages, placing pages most deserving of reclamation at the back and pages least deserving of reclamation at the front. Then, during memory reclamation, scanning proceeds from back to front, attempting to reclaim scanned pages.

LRU list descriptor, containing 5 LRU lists: active/inactive anonymous page LRU lists, active/inactive file page LRU lists, and unevictable page list:

(https://lpc.events/event/11/contributions/896/attachments/793/1493/slides-r2.pdf)

For memory reclamation, it only processes the first 4 LRU lists: active anonymous page LRU list, inactive anonymous page LRU list, active file page LRU list, and inactive file page LRU list. After reclaiming enough page frames, it returns directly: fast memory reclamation and kswapd memory reclamation do this.

Global lruvec can be viewed through meminfo (understood as LRU areas):

## cat /proc/meminfo |grep -i active
Active: 597380 kB
Inactive: 601920 kB
Active(anon): 10896 kB
Inactive(anon): 117376 kB
Active(file): 586484 kB
Inactive(file): 484544 kB

In reality, there is more than one lruvec. cgroup and NUMA nodes each have their own lruvec, and global also has its own lruvec.

Drop Cache
#

Drop cache records which pages are caching file system data pages and writes data back to disk when pages are forcibly reclaimed, so they can be cached again on the next access.

Default value: vm.drop_caches = 0. By default, the Linux kernel does not automatically clear caches.
Setting /proc/sys/vm/drop_caches to 1: The kernel clears unused page cache.
Setting /proc/sys/vm/drop_caches to 2: The kernel releases memory used by dentry and inode. Dentry and inode are file system metadata structures used to store file and directory information.
Setting /proc/sys/vm/drop_caches to 3: Equivalent to 1+2, releases all unused caches.

When the kernel decides to reclaim certain caches, it checks whether the data in the cache is consistent with the data on disk. If the data is inconsistent, the kernel needs to write the data back to disk before reclaiming that cache. This process can cause IO spikes. When performing Drop Cache operations, it is recommended to avoid any important I/O operations as this may affect system performance.

Operation commands:

echo 3 > /proc/sys/vm/drop_caches # Flush cache
echo 0 > /proc/sys/vm/drop_caches # Restore default

Memory Monitoring
#

Without understanding basic memory knowledge, it is actually very difficult to interpret memory monitoring information. With the above memory fundamentals in place, let’s go through memory-related monitoring commands and tools one by one.

What’s in the /proc Directory?
#

/proc mainly contains process information and system information.

In the system information part, some are interfaces provided by Linux for system status, allowing you to view monitoring information at the entire operating system level, such as slabinfo, swaps, zoneinfo, buddyinfo.

The other part, process, contains running data and status information for each process. cd into the corresponding process directory to see the FDs held by the corresponding process and process memory information.

Processes also have threads. Thread information directory: /proc/[pid]/task/[tid]/, with content similar to the process directory.

For more proc information, refer to proc(5) — Linux manual page

/proc/meminfo
#

/proc/meminfo is the primary interface for understanding the current Linux system memory usage. The most commonly used commands like free, vmstat, ps obtain data through it. /proc/meminfo information is more comprehensive. Below we only list some common information. For detailed meanings, refer to the Red Hat documentation

# General memory information
cat /proc/meminfo | grep "Mem"
MemTotal: 994328 kB # Total memory size (minus some reserved and kernel)
MemFree: 66428 kB # Completely unused physical memory
MemAvailable: 207192 kB # Maximum available memory for starting a new application without using swap space

# IO buffers
cat /proc/meminfo | grep -e "Buffers" -we "Cached" 
Buffers: 12820 kB # IO buffers used by raw disk blocks, not exceeding 20MB
Cached: 254592 kB # Page cache size used by disks (includes tmpfs and shmem, excludes SwapCached)

# swap
cat /proc/meminfo | grep "Swap" 
SwapCached: 13936 kB # Swap cache contains anonymous memory pages determined to be swapped but not yet written to physical swap area
SwapTotal: 945416 kB # Total swap space size
SwapFree: 851064 kB # Remaining swap size

# lru active and inactive page counts (self-explanatory)
cat /proc/meminfo | grep -e "Active" -e "Inactive" 
Active: 194308 kB
Inactive: 553172 kB
Active(anon): 59024 kB
Inactive(anon): 437264 kB
Active(file): 135284 kB
Inactive(file): 115908 kB

# Dirty pages
cat /proc/meminfo | grep -e "Dirty" -e "Writeback" 
Dirty: 0 kB # Dirty pages not yet written
Writeback: 0 kB # Dirty pages being written
WritebackTmp: 0 kB # Temporary buffer for writebacks used by the FUSE module

# Map information
cat /proc/meminfo | grep -e "AnonPages" -e "Map"
AnonPages: 95296 kB # Mapped anonymous pages
Mapped: 153192 kB # Mapped file pages
DirectMap4k: 113336 kB # Mapped 4k kernel pages
DirectMap2M: 1900544 kB # Mapped 2M kernel pages
DirectMap1G: 0 kB # Mapped 1G kernel pages

# Shared memory
cat /proc/meminfo | grep "Shmem"
Shmem: 28920 kB # Total memory size of shmem and tmpfs
ShmemHugePages: 0 kB # Total huge page memory size of shmem and tmpfs
ShmemPmdMapped: 0 kB # Shared memory mapped into userspace with huge pages

# Kernel memory (note: slab is kernel)
cat /proc/meminfo | grep -ie "reclaim" -e "slab" -e "kernel"
KReclaimable: 35008 kB # Reclaimable memory allocated to kernel
Slab: 88752 kB # Slab cache
SReclaimable: 35008 kB # Reclaimable memory in slab cache
SUnreclaim: 53744 kB # Non-reclaimable memory in slab cache
KernelStack: 5988 kB # Kernel stack memory used by all tasks

# Allocatable memory (different meaning from MemAvailable)
## CommitLimit=[("total RAM pages" - "total huge TLB pages") * overcommit_ratio]/100 + "total swap pages"
## In short, MemAvailable watermark plus swap equals allocatable memory
cat /proc/meminfo | grep -ie "commit"
CommitLimit: 1442580 kB # Allocatable memory
Committed_AS: 3035924 kB # Estimated memory needed in current worst-case scenario

# Virtual memory
cat /proc/meminfo | grep -e "Vmalloc"
VmallocTotal: 34359738367 kB # Total allocated virtual memory size
VmallocUsed: 34780 kB # Total used virtual memory size
VmallocChunk: 0 kB # Largest contiguous virtual memory block

# Page table memory (self-explanatory)
cat /proc/meminfo | grep PageTables
PageTables: 4120 kB

# Huge page memory
cat /proc/meminfo | grep -i hugepage
AnonHugePages: 32768 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

/proc/buddyinfo
#

Due to its concise and easy-to-understand information, buddyinfo is the most commonly used method for judging memory fragmentation issues. See “Memory Allocation - Buddy section” for details.

$ cat /proc/buddyinfo 
Node 0, zone DMA 0 0 0 1 2 1 1 0 1 1 3 
Node 0, zone DMA32 7 6 5 6 5 6 7 7 6 2 272 
Node 0, zone Normal 317681 38869 31620 19250 8931 2579 815 182 19 5 0 

/proc/pagetypeinfo
#

pagetypeinfo first provides information about page block sizes. It provides the same type of information as buddyinfo but broken down by type and detailing the number of pages of each type.

Before understanding pagetypeinfo, you need to first understand memory compaction.

Suppose the memory in a zone looks like this:

White represents free memory, red represents used memory. The memory fragmentation above is already quite severe. If a request for memory of order 2 or higher is made at this point, it cannot be allocated. This is where memory compaction comes into play. The compaction algorithm marks movable pages and free pages lists on the existing zone.

The movable scanner scans from bottom to top, and the free scanner scans from top to bottom. The movable and free scanners will eventually meet at some point in the middle. Then, through page migration, used pages are moved to the top of the zone.

Two trigger methods for page compaction:

When allocating pages, if allocation fails at the LOW watermark, slow memory allocation is attempted, during which page compaction occurs
Page compaction can be started with echo x > /proc/sys/vm/compact_memory. After starting, the kernel thread kcompactd begins page defragmentation.

Because page data is migrated to new locations, there are no performance issues as severe as those caused by memory reclamation. Moreover, since the goal is clearer, the cost of obtaining contiguous pages is lower. Additionally, ANON page reclamation requires SWAP, while this does not.

Now let’s look at /proc/pagetypeinfo:

$ cat /proc/pagetypeinfo 
Page block order: 9
Pages per block: 512

... (DMA omitted)
Node 0, zone Normal, type Unmovable 870 530 391 157 103 41 9 2 1 0 0 
Node 0, zone Normal, type Movable 5886 9235 5728 4072 1561 324 115 41 12 4 13018 
Node 0, zone Normal, type Reclaimable 3 4 8 11 2 3 1 1 1 0 0 
Node 0, zone Normal, type HighAtomic 0 0 0 0 0 0 0 0 0 0 0 
Node 0, zone Normal, type CMA 0 0 0 0 0 0 0 0 0 0 0 
Node 0, zone Normal, type Isolate 0 0 0 0 0 0 0 0 0 0 0 

Different pages are classified as pageblocks. Each pageblock is divided into several lists based on its type. When allocating memory, pages are requested from the corresponding list based on the requested page type, and when freed, they return to the corresponding list based on their pageblock. Different pageblocks:

Unmovable: Pages that cannot be compacted
Movable: Pages that can be compacted
Reclaimable: Pages that can be reclaimed
HighAtomic: Pageblock added to mitigate fragmentation issues. Only higher-order and same-level requests can request pages from this pageblock
CMA: CMA stands for Contiguous Memory Allocator
Isolate: Pages will not be allocated; used to help isolate pages. When isolating pages, pageblocks are first set to isolate to prevent them from being freed

CMA appears to be another large topic, which can be simply understood as a supplement to the buddy system:

(Memory Journey — How to Improve CMA Utilization? https://ost.51cto.com/posts/10815)

smaps & maps & pmap
#

VSS/RSS/PSS/USS

When viewing the memory occupied by a process, there are commonly four forms: VSS/RSS/PSS/USS, mainly differing in memory calculation methodology.

(https://cloud.tencent.com/developer/article/1683708)

VSS (Virtual Set Size) is just a virtual space size, with little significance for actual memory usage.
RSS (Resident Set Size) is used for calculating the total memory occupied by a process, including shared memory size occupied by shared libraries. For example, if private memory size is N and shared memory size is M, then RSS = N + M. This can be misleading, because for large shared libraries like libc, shared by many processes, counting it all against one process is not scientific.
PSS (Proportional Set Size) is the actual physical memory occupied by a single process when running, including proportionally allocated shared library memory. If a shared library is used by N processes, the size proportionally allocated to PSS is 1/N. PSS calculates process memory more accurately, including exclusive memory plus the shared portion.
USS (Unique Set Size) is the physical memory exclusively occupied by a process, not including shared memory.

/proc/[pid]/maps

/proc/[pid]/maps can view the user space memory mappings of the process’s virtual memory.

[pg@lzl 2345]$ cat maps 
StartAddr-EndAddr Perms Offset Dev Inode Filename
00400000-00bae000 r-xp 00000000 fd:00 1093852 /pg/pg15.3/bin/postgres ---text segment
00dad000-00dc3000 rw-p 007ad000 fd:00 1093852 /pg/pg15.3/bin/postgres
00dc3000-00df5000 rw-p 00000000 00:00 0 
00f1e000-00f60000 rw-p 00000000 00:00 0 [heap] ---heap area
33a6000000-33a6022000 r-xp 00000000 fd:00 1976006 /lib64/ld-2.17.so
...
7fbe2ae09000-7fbe2ae0a000 rw-p 0000c000 fd:00 1975966 /lib64/libnss_files-2.17.so
7fbe2ae1b000-7fbe33ca7000 rw-s 00000000 00:04 12556 /dev/zero (deleted)
7fbe33ca7000-7fbe39b38000 r--p 00000000 fd:00 1181300 /usr/lib/locale/locale-archive
7fbe39b38000-7fbe39b3d000 rw-p 00000000 00:00 0 
7fbe39b46000-7fbe39b4d000 rw-s 00000000 00:10 12559 /dev/shm/PostgreSQL.3661351388
7fbe39b4d000-7fbe39b4e000 rw-s 00000000 00:04 32769 /SYSV0010c0b6 (deleted)
7fbe39b4e000-7fbe39b4f000 rw-p 00000000 00:00 0 
7fffe3933000-7fffe3948000 rw-p 00000000 00:00 0 [stack] --stack area
7fffe397d000-7fffe397e000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

(1) Start-End Address: The address range of this segment in virtual memory (2) Permissions: Permissions of this segment; r-read, w-write, x-execute, p-private (3) Offset: The offset of this segment mapping in the file (4) Device: The device number of the device where the mapped file resides, corresponding to vm_file->f_dentry->d_inode->i_sb->s_dev. Anonymous mappings have 0. fd is the major device number, 00 is the minor device number. (5) Inode: Corresponds to vm_file->f_dentry->d_inode->i_ino, matches the content displayed by ls -i, anonymous mappings have 0. (6) Mapped File Name: For named mappings, it’s the mapped file name. For anonymous mappings, it’s the role of this memory segment in the process.

Below is an analysis by Wenxin (it actually analyzed it correctly, this is a PostgreSQL postmaster process):

/proc/[pid]/smaps

The /proc/[pid]/smaps file is an extension based on /proc/[pid]/maps, providing more detailed information than the maps file in the same directory. Each VMA has the following series of data:

[pg@lzl 2345]$ cat smaps 
00400000-00bae000 r-xp 00000000 fd:00 1093852 /pg/pg15.3/bin/postgres
Size: 7864 kB --VSS memory
Rss: 408 kB --RSS memory
Pss: 140 kB --PSS memory
Shared_Clean: 404 kB --Shared, clean memory size
Shared_Dirty: 0 kB --Shared, dirty (i.e., modified) memory size
Private_Clean: 4 kB --Private, clean memory size
Private_Dirty: 0 kB --Private, dirty memory size
Referenced: 408 kB --Current page marked as referenced or containing anonymous mappings
Anonymous: 0 kB --Anonymous pages
AnonHugePages: 0 kB --Anonymous huge pages
Swap: 0 kB --Swapped-out memory size
KernelPageSize: 4 kB --Kernel page size
MMUPageSize: 4 kB --Page table page size
...
7fffe3933000-7fffe3948000 rw-p 00000000 00:00 0 [stack]
Size: 88 kB
Rss: 16 kB
Pss: 16 kB
...

Now we know that maps are the process’s memory mapping information, and smaps also includes the memory size of each mapping segment (VSS, RSS, PSS).

You can calculate a process’s memory usage by looking at PSS, RSS, etc. data in process smaps. Note the unit is KB.

Total physical memory usage of all processes:

grep Pss /proc/[1-9]*/smaps | awk '{total+=$2}; END {printf "%d kB\n", total }'

PSS memory of a specific process:

cat /proc/90875/smaps |grep Pss |awk '{sum+=$2 };END {print sum/1024}'

RSS memory of a specific process:

cat /proc/68729/smaps |grep Rss |awk '{sum+=$2 };END {print sum/1024}'

Private memory of a specific process:

cat /proc/90875/smaps|sed '/zero/,/VmFlags/d' |grep Private |awk '{sum+=$2 };END {print sum/1024}'

pmap

The pmap command parses the /proc/[pid]/maps and /proc/[pid]/smaps files. It has few parameters; -x means show more information.

[root@lzl ~]# pmap -x 2345
2345: /pg/pg15.3/bin/postgres -D /pg/1503data
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 7864 212 0 r-x-- postgres
0000000000dad000 88 12 12 rw--- postgres
0000000000dc3000 200 36 32 rw--- [ anon ]
0000000000f1e000 264 12 8 rw--- [ anon ]
00000033a6000000 136 108 0 r-x-- ld-2.17.so
...
00007fbe2ae09000 4 0 0 rw--- libnss_files-2.17.so
00007fbe2ae1b000 145968 4396 4396 rw-s- zero (deleted)
00007fbe33ca7000 96836 8 0 r---- locale-archive
00007fbe39b38000 20 16 16 rw--- [ anon ]
00007fbe39b46000 28 4 4 rw-s- PostgreSQL.3661351388
00007fbe39b4d000 4 0 0 rw-s- [ shmid=0x8001 ]
00007fbe39b4e000 4 4 4 rw--- [ anon ]
00007fffe3933000 84 16 16 rw--- [ stack ]
00007fffe397d000 4 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------ ------ ------
total kB 268896 5532 4540

The pmap output format is similar to /proc/[pid]/maps, with one line per VMA address, but includes VSS and RSS in addition to maps, allowing you to directly see the size used by each region of the process’s virtual memory, helping to quickly determine where the regions with more memory are.

If the [heap] in the address space is too large, it might be a heap memory leak. For another example, if the process address space contains too many VMAs (each line in maps can be understood as a VMA), it’s likely that the application called many mmaps without munmap. Or, continuously observing changes in the address space — if certain entries are continuously growing, there’s likely an issue there.

Analysis Example

From the host’s TOP memory view, a certain PostgreSQL backend process memory appears relatively high. Further analysis of map information is needed:

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
68729 postgres 20 0 5579004 5.116g 5.114g R 97.4 1.4 128:27.94 postgres: lzl: lzldb lzl 30.78.14.174(58067) DELETE 

Check this process’s Rss, Pss, Uss:

cat /proc/68729/smaps |grep Rss |awk '{sum+=$2 };END {print sum/1024}'
5422.67 ---5.4G Rss
cat /proc/68729/smaps |grep Pss |awk '{sum+=$2 };END {print sum/1024}'
467.957 ---467mb Pss
cat /proc/68729/smaps|sed '/zero/,/VmFlags/d' |grep Private |awk '{sum+=$2 };END {print sum/1024}'
179.605 ---179mb Uss

Rss-Uss=5.3G of shared memory. From Pss-Uss=290mb of proportional shared memory, we can roughly see that this backend is only a small portion of this shared memory proportion.

$ pmap -x 68729
68729: postgres: pdmp: pdmpdata pdmp 30.78.14.174(46252) DELETE
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 6084 2444 0 r-x-- postgres
0000000000bf0000 4 4 4 r---- postgres
0000000000bf1000 52 52 52 rw--- postgres
...
00002b7f65bfa000 5441216 5365444 5365444 rw-s- zero (deleted) --this part takes the most
00002b80b1daa000 48 0 0 r-x-- libnss_files-2.17.so
00002b80b1db6000 2044 0 0 ----- libnss_files-2.17.so
00002b80b1fb5000 4 4 4 r---- libnss_files-2.17.so
00002b80b1fb6000 4 4 4 rw--- libnss_files-2.17.so
00002b80b1fb7000 24 0 0 rw--- [ anon ]
00002b80ba001000 516 516 516 rw--- [ anon ]
00007fffe16f7000 132 88 88 rw--- [ stack ]
00007fffe175b000 8 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]

Diving deeper into smap analysis, we can directly locate the zero (deleted) part:

$ cat smaps 
00400000-009f1000 r-xp 00000000 fd:06 58726481 /paic/postgres/base/9.6.6/bin/postgres
...
2b7f65bfa000-2b80b1daa000 rw-s 00000000 00:04 72254 /dev/zero (deleted)
Size: 5441216 kB
Rss: 5365444 kB
Pss: 264618 kB
Shared_Clean: 0 kB
Shared_Dirty: 5365444 kB --shared dirty data
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 5364764 kB
Anonymous: 0 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
VmFlags: rd wr sh mr mw me ms sd 

From the above analysis, we can conclude: this is a PostgreSQL private process that has modified a large amount of data without flushing dirty pages. Its own private memory is not much; most is occupied in shared memory. This is likely a transaction in PostgreSQL that has modified a lot of data but hasn’t committed yet.

Additionally, /dev/zero (deleted) is explained in proc(5) — Linux manual page:

Although these entries are present for memory regions that were mapped with the MAP_FILE flag, the way anonymous shared memory (regions created with the MAP_ANON | MAP_SHARED flags) is implemented in Linux means that such regions also appear on this directory. Here is an example where the target file is the deleted /dev/zero one:
 lrw-------. 1 root root 64 Apr 16 21:33
 7fc075d2f000-7fc075e6f000 -> /dev/zero (deleted)

“Unofficial translation”: Anonymous pages and shared pages are represented by /dev/zero (deleted).

/proc/[pid]/status
#

status can view process state information, including some memory information.

[root@lzl 2345]# cat status 
Name: postgres ---the command running this thread
State: S (sleeping) ---process state
Tgid: 2345 ---Thread group ID (i.e., Process ID)
Pid: 2345 ---Thread ID
PPid: 1 ---PID of parent process.
...
VmPeak: 268964 kB ---virtual memory peak
VmSize: 268896 kB ---virtual memory current
VmLck: 0 kB
VmHWM: 13400 kB ---RSS peak
VmRSS: 5532 kB ---RSS current
VmData: 528 kB ---data segment
VmStk: 88 kB ---stack segment
VmExe: 7864 kB ---text segment
VmLib: 3100 kB ---shared library code segment
VmPTE: 136 kB ---Page table entries
VmSwap: 308 kB ---swap size
Threads: 1 ---number of threads in this process
....

Compared to maps, status has no mapping information. The memory data is more summarized, allowing for a more intuitive view of the size occupied by each segment of virtual memory.

View processes with the most SWAP usage:

for file in /proc/*/status ; do awk '/VmSwap|Name|^Pid/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 3 -n -r | head

cgroup memory
#

cgroup memory control is now very common. Some host parameters need to be set in cgroup. Memory settings and monitoring information are under /sys/fs/cgroup/memory/.

cginfo to view CGROUP memory allocation and usage: /opt/cgtools/cginfo -t perf -s mem

cginfo -t perf -s mem
==================== Cgroup Performance: memory ====================
DB_TYPE INSTANCE_NAME MEM_OOM MEM_FILE_GB MEM_MAP_GB MEM_USED_GB MEM_ALLO_GB ALLO_RATE MEM_GLOB_GB GLOB_RATE 
------- ------------- ------- ----------- ---------- ----------- ----------- --------- ----------- --------- 
postgres LZLDB 0 154.3 0.0 4.2 160.0 2.6% 375 1.1% 

View relatively detailed CGROUP memory usage status: /sys/fs/cgroup/memory/[group]/memory.stat

$ cat memory.stat 
...
total_cache 167791534080
total_rss 4006932480
total_rss_huge 0
total_mapped_file 11747328
total_swap 0
total_pgpgin 792754417976
total_pgpgout 792712474991
total_pgfault 477971874868
total_pgmajfault 97318
total_inactive_anon 1610874880
total_active_anon 2408255488
total_inactive_file 73446166528
total_active_file 94332768256
total_unevictable 0

smem
#

smem is a powerful tool for displaying memory usage. It reads information from smaps, meminfo, etc. under /proc and outputs summaries. smem can output overall and specific map memory conditions, which is very intuitive and can be analyzed from different dimensions. Overall, it’s a very useful tool for analyzing memory usage.

The repo can be downloaded directly. Basically, just extract and use it. For more usage, refer to smem memory reporting tool. Below are just simple examples:

View system memory usage -w:

[root@lzl ~]# smem -w -k
Area Used Cache Noncache 
firmware/hardware 0 0 0 
kernel image 0 0 0 
kernel dynamic memory 183.9M 84.0M 99.9M 
userspace memory 112.3M 62.2M 50.1M 
free memory 700.3M 700.3M 0 

View memory consumption per user -u:

[root@lzl ~]# smem -s pss -urk
User Count Swap USS PSS RSS 
oracle 25 85.2M 30.8M 95.7M 383.0M 
root 93 112.4M 38.5M 42.3M 86.2M 
pg 12 5.9M 1.6M 2.5M 5.9M 
mysql 1 169.7M 1.7M 1.7M 2.0M 

View memory consumption for a specific user -U:

[root@lzl ~]# smem -U pg -k
 PID User Command Swap USS PSS RSS 
 2345 pg /pg/pg15.3/bin/postgres -D 364.0K 124.0K 134.0K 228.0K 
 2352 pg postgres: logical replicati 636.0K 144.0K 161.0K 196.0K 
 ...

Filter a specific process -P (PROCESSFILTER, not pid):

[root@lzl ~]# smem -P postgres -p
 PID User Command Swap USS PSS RSS 
 2346 pg /pg/pg16.0/bin/postgres -D 0.01% 0.01% 0.01% 0.01% 
 2350 pg postgres: walwriter 0.01% 0.01% 0.01% 0.01% 
 ...

View process mapping and memory usage -m:

[root@lzl ~]# smem -P postgres -mpr -s pss
Map PIDs AVGPSS PSS 
<anonymous> 13 0.02% 0.24% 
[heap] 3 0.07% 0.20% 
/usr/lib64/libpython2.6.so.1.0 1 0.11% 0.11% 
/pg/pg15.3/bin/postgres 6 0.01% 0.06% 
/pg/pg16.0/bin/postgres 6 0.01% 0.06% 
/dev/zero 12 0.00% 0.03% 
[stack] 13 0.00% 0.02% 
...

smem is very intuitive for viewing process USS\PSS\RSS. However, there is one issue: smem cannot filter by pid, only by username or PROCESSFILTER. When a host has multiple database instances deployed, filtering by parent PID or child PID is not very friendly.

top
#

top can display system running status in real time. top can be quite fancy in its usage. Running top directly can also display a lot of information.

Sorting in top:

command sorted-field supported
M %MEM Yes
N PID Yes
P %CPU Yes
T TIME+ Yes

You can use %MEM to sort processes with higher memory usage. %MEM represents the RES memory percentage.

top - 23:38:01 up 3 days, 22:32, 2 users, load average: 1.12, 1.42, 1.09
Tasks: 198 total, 13 running, 183 sleeping, 0 stopped, 2 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1020348k total, 325848k used, 694500k free, 1352k buffers
Swap: 4128760k total, 635872k used, 3492888k free, 150288k cached

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18537 oracle 20 0 636m 24m 21m S 0.0 2.4 0:05.41 oracle
18533 oracle 20 0 638m 24m 21m S 0.0 2.4 0:02.01 oracle
...
18509 oracle 20 0 634m 4384 4036 S 0.0 0.4 0:01.93 oracle
 2639 root 20 0 729m 4052 1444 S 0.0 0.4 8:45.32 nautilus 

Memory-related interpretation:

Line 4: Memory usage information: physical memory amount, used memory, free memory, buffer memory Line 5: Swap partition information: available swap total, used swap total, free swap total, kernel cached amount

Line 6 (memory-related):

VIRT: VSS
RES: RSS (likely), anything occupying physical memory
SHR: Shared Memory Size. It will include shared anonymous pages and shared file-backed pages
%MEM: RSS percentage, a task’s currently resident share of available physical memory.

Additionally, don’t forget to look at the process status when checking memory.

S (example column 8) Process Status:

D = uninterruptible sleep. Indicates the process is waiting for an external event to complete, such as disk I/O operations or network requests. Usually, D processes cannot be directly terminated.
I = idle
R = running
S = sleeping
T = stopped by job control signal
t = stopped by debugger during trace
Z = zombie

The top command can see the host’s memory summary information. Process memory usage information includes RSS and SHR. A rough calculation of RES-SHR=USS can also calculate the private memory usage size. Additionally, you can see process status, so top -p to view basic memory information for a specific process is very useful.

free
#

free displays the host’s swap, total and remaining memory, all parsed from /proc/meminfo.

user@ubuntu:~$ free

 total used free shared buff/cache available
Mem: 8029356 794336 6297928 183384 937092 6816804
Swap: 0 0 0

total: Total usable memory (MemTotal and SwapTotal in /proc/meminfo). This includes the physical and swap memory minus a few reserved bits and kernel binary code.
used: Used or unavailable memory (calculated as total - available)
free: Unused memory (MemFree and SwapFree in /proc/meminfo) shared Memory used (mostly) by tmpfs (Shmem in /proc/meminfo)
buffers: Memory used by kernel buffers (Buffers in /proc/meminfo)
cache: Memory used by the page cache and slabs (Cached and SReclaimable in /proc/meminfo). Not just pagecache, but also SReclaimable slab!
buff/cache: Sum of buffers and cache
available: cache includes pagecache and SReclaimable, free includes mem free and swap free; while available includes pagecache and memory about to be reclaimed. Indicates available memory, but their calculation methods differ. In practical applications, due to cache existence, available is usually larger than free.

Page Cache: Page cache is primarily used as a cache for file data on the file system, especially when processes have read/write operations on files.

Buffer Cache: Buffer cache is primarily designed for caching blocks when the system reads/writes block devices.

ps aux
#

The biggest advantage of ps is analyzing process status (including memory) from the process perspective. Processes with [ ] flags in the COMMAND are kernel processes.

[pg@lzl ~]$ ps aux|head -1;ps aux|grep postgres
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
pg 2345 0.0 0.0 268896 236 ? Ss Jan01 0:03 /pg/pg15.3/bin/postgres -D /pg/1503data
pg 2353 0.0 0.0 269040 196 ? Ss Jan01 0:00 postgres: checkpointer
pg 2354 0.0 0.0 269032 160 ? Ss Jan01 0:02 postgres: background writer
pg 2356 0.0 0.0 269032 116 ? Ss Jan01 0:01 postgres: walwriter
pg 2357 0.0 0.0 270508 824 ? Ss Jan01 0:02 postgres: autovacuum launcher
pg 2358 0.0 0.0 270492 620 ? Ss Jan01 0:00 postgres: logical replication launcher
pg 29818 0.0 0.0 103372 868 pts/0 S+ 09:16 0:00 grep postgres

VSZ and RSS units are KB. Memory information is limited; VSZ has little value, RSS can be referenced, but there’s no PSS or USS type information, so not much can be analyzed.

ipcs
#

ipcs -m is a command for querying IPC (Interprocess Communication) shared memory resources. It’s quite useful when analyzing shared memory.

[pg@lzl ~]$ ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status 
0x0010c0b6 32769 pg 600 56 6 

Shared memory key value
Shared memory ID (shmid)
User who created this shared memory
Permissions (perms)
Created size (bytes)
Number of processes attached to this shared memory (nattach)
Shared memory status

When connecting a session to PostgreSQL, one more backend process appears:

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status 
0x0010c0b6 32769 pg 600 56 7 

nattch+1, indicating that the private backend process also shares a portion of the PG shared memory. At this point, the following diagram is understood more deeply:

(http://gauss.ececs.uc.edu/Courses/c4029/code/memory/virtual.pdf)

vmstat
#

vmstat is an abbreviation for Virtual Memory Statistics, and can monitor the operating system’s virtual memory, processes, and CPU activity. It provides statistics on the overall system situation; the shortcoming is that it cannot perform in-depth analysis of a specific process.

Useful parameter explanations:

vmstat [options] [delay [count]]
OPTIONS:
-a Display active and inactive memory
-m Display slabinfo
-s Display memory-related statistics and various system activity counts
-t Append timestamp to each line
-w Wide output mode. Without w, the output is narrow, reducing alignment issues

-bash-4.1$ vmstat -w 1 3
procs -------------------memory------------------ ---swap-- -----io---- --system-- -----cpu-------
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 0 1 661652 763348 324 76100 15 12 54 21 18 45 0 0 79 21 0
 2 1 661652 763340 304 75764 0 0 0 32 12 84 0 1 0 99 0
41 1 661652 760744 244 78300 228 0 3216 0 265 442 0 0 0 100 0

pidstat
#

pidstat is a command from the sysstat tool, used to monitor all or specified processes’ CPU, memory, threads, device IO, and other system resource usage.

Useful parameter explanations:

pidstat OPTIONS interval [ count ] 
-d :Report I/O statistics 
-u :Report CPU utilization
-r :Report page faults and memory utilization
-w :Report task switching activity
-p :pid[,...] 
-l :Display the process command name and all its arguments.

View memory status of a specific process:

-bash-4.1$ pidstat -r -l -p 2345
Linux 2.6.32-431.el6.x86_64 (lzl) 01/06/2024 _x86_64_ (1 CPU)

02:48:32 PM PID minflt/s majflt/s VSZ RSS %MEM Command
02:48:32 PM 2345 0.23 0.00 268896 240 0.02 /pg/pg15.3/bin/postgres -D /pg/1503data 

Various indicators are relatively easy to understand. VSZ, RSS — tired of talking about them.

minflt/s: Abbreviation for “minor page faults”, indicating the number of “minor page faults” that occur per second. A page fault occurs when a program tries to access a page that is not in physical memory. If the page is indeed in the swap area on disk, this is a minor page fault.
majflt/s: Abbreviation for “major page faults”, indicating the number of “major page faults” that occur per second. Unlike minor page faults, major page faults occur when a program tries to access a page that is not in physical memory and is also not in the swap area on disk.

sar
#

sar (System Activity Reporter) is currently one of the most comprehensive system performance analysis tools on Linux. It can report on various aspects of system activity, including: file read/write status, system call usage, disk I/O, CPU efficiency, memory usage, process activity, and IPC-related activity. The SAR tool is part of the sysstat software package.

(https://www.brendangregg.com/Perf/linux_observability_sar.png)

sar is very powerful. The man parameter introduction alone has over 1k lines. This article cannot possibly explain everything (being lazy).

Memory-related parameters:

sar OPTIONS interval [ count ] 
-B :Report paging statistics
-r :Report memory utilization statistics
-W :Report swapping statistics.
-H :Report hugepages utilization statistics
-s [ start_time ] ] [ -e [ end_time ] 

Example: sar view memory utilization sar -r 1 3

kbmemfree: This value is basically consistent with the free value in the free command, so it does not include buffer and cache space
kbmemused: This value is basically consistent with the used value in the free command, so it includes buffer and cache space
%memused: This value is kbmemused as a percentage of total memory (excluding swap)
kbbuffers: buffer in the free command
kbcached: cache in the free command
kbcommit: Memory needed to guarantee the current system, i.e., memory needed to ensure no overflow (RAM + swap)
%commit: This value is kbcommit as a percentage of total memory (including swap)

Example: sar view memory page status sar -B 1 3

pgpgin/s: Kilobytes paged in from disk or SWAP to memory per second
pgpgout/s: Kilobytes paged out from memory to disk or SWAP per second
fault/s: Number of page faults per second, i.e., sum of major and minor faults
majflt/s: Number of major faults per second
pgfree/s: Number of pages placed on the free queue per second
pgscank/s: Number of pages scanned by kswapd per second
pgscand/s: Number of pages directly scanned per second
pgsteal/s: Number of pages reclaimed from cache to meet memory needs per second
%vmeff: Pages stolen (pgsteal) as a percentage of total scanned pages (pgscank + pgscand) per second

Example: sar view swap information sar -W 1 3

Report explanation:

pswpin/s: Number of swap pages swapped in per second
pswpout/s: Number of swap pages swapped out per second

Example: sar view historical memory information sar -B -s "08:00:00" -e "10:00:00"

#Without -e, it shows information from the start time to now
$ sar -B -s "08:00:00"
09:45:01 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
09:46:01 PM 414429.37 395024.08 179478.63 0.07 352922.62 12003.78 4266.52 16269.42 99.99
09:47:01 PM 879907.08 337948.43 157970.97 0.02 402290.21 0.00 0.00 0.00 0.00
09:48:01 PM 772977.43 507343.30 150255.50 0.05 466742.08 0.00 5821.28 5821.27 100.00

Above, pgscank represents the speed at which the kswapd process intervenes in memory reclamation, and pgscand represents the speed of direct memory reclamation.

gcore
#

gcore is part of gdb and can generate a core dump file for a process.

Example: dump a PostgreSQL backend process:

[root@lzl ~] ps -ef|grep 8296
pg 8296 2345 0 09:41 ? 00:00:00 postgres: pg lzldb [local] idle 
[root@lzl ~] cat /proc/8296/smaps |grep Pss |awk '{sum+=$2 };END {print sum/1024}' 
0.351562
[root@lzl ~] cat /proc/8296/smaps |grep Rss |awk '{sum+=$2 };END {print sum/1024}'
0.445312
[root@lzl ~] cat /proc/8296/smaps|sed '/zero/,/VmFlags/d' |grep Private |awk '{sum+=$2 };END {print sum/1024}'
0.0078125

Process 8296’s USS is only 7.8 KB, RSS 445 KB. Dump memory:

gcore -o /tmp/dump 8296

Dumping takes some time, and the dumped file is relatively large, and it will hang the process.

[root@lzl 8296]# ls -lh /tmp/dump.8296 
-rw-r--r-- 1 root root 252M Jan 7 10:59 /tmp/dump.8296

gdb
#

gdb can view specific locations and content in memory.

Example: view PostgreSQL backend cached data:

Open a new session to query a partitioned table, keeping the session open:

[pg@lzl ~]$ psql
psql (15.3)
Type "help" for help.

postgres=> \c lzldb
You are now connected to database "lzldb" as user "pg".
lzldb=> select * from lzlpartition limit 1;
 appl_no | is_deleted | date_created | date_updated 
---------+------------+--------------+--------------
(0 rows)

Use pmap, smaps to view process memory usage and find the memory segment to dump:

[root@lzl 13393]# pmap -x 13393
13393: postgres: pg lzldb [local] idle 
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 7864 1204 0 r-x-- postgres
..
00007fbe2ae1b000 145968 2164 176 rw-s- zero (deleted) ---RSS takes the most here
00007fbe33ca7000 96836 0 0 r---- locale-archive
00007fbe39b38000 20 0 0 rw--- [ anon ]
00007fbe39b46000 28 0 0 rw-s- PostgreSQL.3661351388
00007fbe39b4d000 4 0 0 rw-s- [ shmid=0x8001 ]
00007fbe39b4e000 4 0 0 rw--- [ anon ]
00007fffe3933000 84 36 0 rw--- [ stack ]
00007fffe397d000 4 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]

[root@lzl 13393]# cat /proc/13393/smaps |grep -A 13 zero
7fbe2ae1b000-7fbe33ca7000 rw-s 00000000 00:04 12556 /dev/zero (deleted)
Size: 145968 kB
Rss: 2164 kB
Pss: 2164 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 1988 kB
Private_Dirty: 176 kB
Referenced: 2164 kB
Anonymous: 0 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB

gdb dump memory:

The starting position for dumping memory is the vm address in smaps + 0x:

[pg@lzl tmp]$ gdb
(gdb) attach 13393
(gdb) dump memory /tmp/delete.dump 0x7fbe2ae1b000 0x7fbe33ca7000

View the dump file:

You can simply view it through strings:

[root@lzl 13393]# strings /tmp/delete.dump|grep lzl|sort|uniq
...
 @lzlpartition_202301
lzlpartition_202301
lzlpartition_202301_appl_no_idx
lzlpartition_202301_date_created_idx
...
lzlpartition_202306
lzlpartition_202306_appl_no_idx
lzlpartition_202306_date_created_idx
 @lzlpartition_attach
lzlpartition_attach
 @nk_lzlpartition
nk_lzlpartition
select * from lzlpartition limit 1;

As long as the session queries a partitioned table, all partition and index metadata is cached in the backend process.

Note:

gdb attach [pid] will hang the process; do not execute casually
The dump file size equals VSS, generally much larger than RSS/PSS/USS

Memory Summary
#

References
#

Easily Break Through File I/O Bottlenecks: Memory-Mapped mmap Technology https://blog.51cto.com/u_15481245/6582927

Step by Step with Diagrams: Deep Understanding of Linux Physical Memory Management https://cloud.tencent.com/developer/article/2352771?areaId=106001

Systematically Learning Memory Management from a DBA’s Perspective https://mp.weixin.qq.com/s/CybzGP44dVWQN5hfFrVx7A

https://linux2me.wordpress.com/2017/09/15/linux-introduction-to-memory-management/

Memory management in Linux https://www.slideshare.net/raghusiddarth/memory-management-in-linux-11551521?from_search=2

Linux Performance Tunning Memory https://www.slideshare.net/shayc1/linux-performance-tunning-memory?from_search=4

How to Learn the Linux Kernel (Memory Chapter) https://mp.weixin.qq.com/s/lKKHH1MMiZbnIbDQt3-IAQ

https://courses.engr.illinois.edu/cs241/sp2014/lecture/09-VirtualMemory_II_sol.pdf

Linux Process Virtual Address Space https://maodanp.github.io/2019/06/02/linux-virtual-space/

Red Hat Official Documentation https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/chap-virtualization_tuning_optimization_guide-numa

Data Processing on Modern Hardware https://db.in.tum.de/teaching/ss21/dataprocessingonmodernhardware/MH_8.pdf?lang=de

Chapter 2 Describing Physical Memory https://www.kernel.org/doc/gorman/html/understand/understand005.html

Various command man pages

Linux Forced Memory Reclamation, Linux Memory Source Code Analysis - Memory Reclamation (Overall Process) https://blog.csdn.net/weixin_35094083/article/details/116688112

Memory Journey — How to Improve CMA Utilization? https://ost.51cto.com/posts/10815

The implementations of anti pages fragmentation in Linux kernel https://teawater.github.io/presentation/antif.pdf

T H E /proc F I L E S Y S T E M https://www.kernel.org/doc/Documentation/filesystems/proc.txt

The /proc/meminfo File in Linux https://www.baeldung.com/linux/proc-meminfo

the proc filesystem https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-proc-meminfo

Introduction and Usage of Linux /proc/{pid}/maps (Locating Memory Leaks) https://blog.csdn.net/mijichui2153/article/details/123934531

CPU and Memory Usage in Linux top Command https://blog.csdn.net/weixin_45030965/article/details/127693042

smem memory reporting tool https://www.selenic.com/smem/

Linux performance optimization https://feiyang233.club/post/linux/

gdb onlinedocs https://sourceware.org/gdb/current/onlinedocs/gdb

Linux_Core_Dumps https://averageradical.github.io/Linux_Core_Dumps.pdf

The Last DBA on Last DBA

A DBA's Perspective on the 0526 Approved Database List

TL;DR #

The Latest List #

The Floodgates Open #

Impact #

Bittersweet Reflections #

Reference #

UUID v4 and v7: Collision Incidents and Performance Benchmarks

TL;DR #

The UUID v4 Collision Incident #

Cause 1: Unreliable entropy sources #

Cause 2: Known npm uuid package bugs #

Cause 3: Linux kernel /dev/random race condition #

Cause 4: Go UUID library not checking return values #

Cause 5: Historical AMD CPU RNG defects #

UUID v7 Performance Comparison in PG 16 #

Test Conditions #

Performance Results #

Why Such a Big Difference #

Summary #

When PostgreSQL Becomes AI's Hands — Bruce Momjian's MCP Server in Practice

Theory Layer: Explaining the RAG → MCP Evolution Through Transformers (Slides 1-33) #

RAG vs MCP: In One Sentence #

“Word or MCP” — That Set of Vector Embedding Diagrams #

Is This Model Correct? #

Practice Layer: Two Working Demos (Slides 34-69) #

Demo 1: Letting ChatGPT Read a Real-World Geiger Counter #

Demo 2: Using PG as a Pretzel Shop Inventory System #

How Far from Production #

Between the Two Layers #

My Blog is Live

It’s Live! #

Highlights #

Reflections on Going Live #

What It Cost #

Finally #

Reference #

Case Study: Startup Failure and SysV Shared Memory

Problem Symptoms #

Error Analysis: pre-existing shared memory block #

Three Types of Shared Memory #

Source Code Analysis #

SysV shmem #

What Are shmkey and shmid? #

How PG Obtains the shmkey #

PG shmkey in Cloud Environments #

shmid Relationships #

Testing #

Reproducing the Production Issue #

Normal Crash Recovery (Successful Startup) #

Holding a File Descriptor But Not shmem #

Deleting postmaster.pid Then Failing to Start #

Stop a Different Instance, Start the Current One #

Error Analysis: lock file "postmaster.pid" already exists #

Summary #

DBA, Writing, Learning and the Future in the AI Era

Writing in the AI Era #

Reflections on Operations #

The Essence of Operations and AI Ops #

AIOps and Agent Research Results #

The Value of a DBA #

Is a DBA’s Value Just Being the Decision-Maker and Scapegoat? #

The Uniqueness of the Postgres DBA #

Why Keep Learning #

Why People Still Love Reading “Human-Written” Articles #

The Psychology of Preferring Imperfection #

Borrowing an Expert’s ATTENTION for Free #

Can AGI Solve All Problems? #

Refuting Musk #

The Mathematical Foundation for Why AI Cannot Solve Everything #

Conclusion #

ref #

PostgreSQL Operations Experience 2025

CPU #

Execution Plans #

Inaccurate DISTINCT Estimates #

Generic Plan Interference #

LWLock:Lockmanager Caused by Row Locks #

Idle Connections #

TL;DR
#

The Latest List
#

The Floodgates Open
#

Impact
#

Bittersweet Reflections
#

Reference
#

TL;DR
#

The UUID v4 Collision Incident
#

Cause 1: Unreliable entropy sources
#

Cause 2: Known npm uuid package bugs
#

Cause 3: Linux kernel /dev/random race condition
#

Cause 4: Go UUID library not checking return values
#

Cause 5: Historical AMD CPU RNG defects
#

UUID v7 Performance Comparison in PG 16
#

Test Conditions
#

Performance Results
#

Why Such a Big Difference
#

Summary
#

Theory Layer: Explaining the RAG → MCP Evolution Through Transformers (Slides 1-33)
#

RAG vs MCP: In One Sentence
#

“Word or MCP” — That Set of Vector Embedding Diagrams
#

Is This Model Correct?
#

Practice Layer: Two Working Demos (Slides 34-69)
#

Demo 1: Letting ChatGPT Read a Real-World Geiger Counter
#

Demo 2: Using PG as a Pretzel Shop Inventory System
#

How Far from Production
#

Between the Two Layers
#

It’s Live!
#

Highlights
#

Reflections on Going Live
#

What It Cost
#

Finally
#

Reference
#

Problem Symptoms
#

Error Analysis: `pre-existing shared memory block`
#

Three Types of Shared Memory
#

Source Code Analysis
#

SysV shmem
#

What Are shmkey and shmid?
#

How PG Obtains the shmkey
#

PG shmkey in Cloud Environments
#

shmid Relationships
#

Testing
#

Reproducing the Production Issue
#

Normal Crash Recovery (Successful Startup)
#

Holding a File Descriptor But Not shmem
#

Deleting postmaster.pid Then Failing to Start
#

Stop a Different Instance, Start the Current One
#

Error Analysis: `lock file "postmaster.pid" already exists`
#

Summary
#

Writing in the AI Era
#

Reflections on Operations
#

The Essence of Operations and AI Ops
#

AIOps and Agent Research Results
#

The Value of a DBA
#

Is a DBA’s Value Just Being the Decision-Maker and Scapegoat?
#

The Uniqueness of the Postgres DBA
#

Why Keep Learning
#

Why People Still Love Reading “Human-Written” Articles
#

The Psychology of Preferring Imperfection
#

Borrowing an Expert’s ATTENTION for Free
#

Can AGI Solve All Problems?
#

Refuting Musk
#

The Mathematical Foundation for Why AI Cannot Solve Everything
#

Conclusion
#

ref
#

CPU
#

Execution Plans
#

Inaccurate DISTINCT Estimates
#

Generic Plan Interference
#

LWLock:Lockmanager Caused by Row Locks
#

Idle Connections
#

Idle in Transaction
#

Memory
#

Memory Issues and Huge Pages
#

Notable Cgroup Knowledge
#

Unsolved Mysteries
#

Pay Attention to the OS
#

Pay Attention to Everything OS
#

Physical Reads
#