Cost-Based Optimization

SQL is a declarative language, which means that users only specify what they want to achieve, but not how to achieve it. For example, should the query SELECT * FROM mytable WHERE age > 50; perform a full table scan and apply a filter, or should it use an index (see the following blog post for more details about this)? The optimizer of the database management system is responsible for determining the best execution plan to execute a given query. During query planning, the optimizer generates multiple alternative plans. Many DBMSs perform cost-based optimization, where each plan is qualified with a cost estimate, a numerical value representing the estimated resource usage (e.g., CPU time, I/O operations) required to execute the plan. The optimizer then selects the plan with the lowest estimated cost as the final execution plan for the query.

To calculate the costs of the plan nodes, the optimizer uses a cost model that accounts for factors such as the number of rows predicted to be processed (based on statistics and selectivity estimates) and constants.

Query Plans in PostgreSQL

Using the EXPLAIN command in PostgreSQL, you can see the final chosen plan and its estimated total cost, and the costs of the individual plan nodes. For example, using EXPLAIN (VERBOSE, ANALYZE) SELECT * FROM test1 WHERE id = 5;, the query plan of the given select query is shown:

jan2=# EXPLAIN (VERBOSE, ANALYZE) SELECT * FROM test1 WHERE id = 5;
 QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------
 Index Only Scan using test1_pkey on public.test1  (cost=0.28..8.29 rows=1 width=4) (actual time=0.153..0.160 rows=1 loops=1)
   Output: id
   Index Cond: (test1.id = 5)
 Heap Fetches: 1
 Planning Time: 1.166 ms
 Execution Time: 0.284 ms
(6 rows)

The plan consists of only one Index Only Scan node, with an estimated total cost of 0.28..8.29, which means that the startup cost is 0.28 and the total cost is 8.29. The startup cost is the cost of getting the first row, while the total cost is the cost of getting all rows.

A more complex example with a join might look like this:

jan2=# EXPLAIN (VERBOSE, ANALYZE) SELECT * FROM test1 LEFT JOIN test2 ON (test1.id = test2.id);
 QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------
 Hash Left Join (cost=27.50..45.14 rows=1000 width=8) (actual time=0.625..1.422 rows=1000 loops=1)
   Output: test1.id, test2.id
 Inner Unique: true
   Hash Cond: (test1.id = test2.id)
 ->  Seq Scan on public.test1  (cost=0.00..15.00 rows=1000 width=4) (actual time=0.038..0.220 rows=1000 loops=1)
         Output: test1.id
 ->  Hash (cost=15.00..15.00 rows=1000 width=4) (actual time=0.571..0.572 rows=1000 loops=1)
         Output: test2.id
 Buckets: 1024  Batches: 1 Memory Usage: 44kB
 ->  Seq Scan on public.test2  (cost=0.00..15.00 rows=1000 width=4) (actual time=0.019..0.191 rows=1000 loops=1)
               Output: test2.id
 Planning Time: 3.436 ms
 Execution Time: 1.551 ms
(13 rows)

In this case, the plan consists of a Hash Left Join node with an estimated cost of 27.50..45.14. The plan also contains two Seq Scan nodes with estimated costs of 15.00..15.00, one for each of the test1 and test2 tables. Furthermore, a Hash node is used to build a hash table for test2; its cost remains 15.00..15.00.

Structure of a Query Plan

Like most database management systems, PostgreSQL uses a tree of plan nodes to organize the data processing. Each node represents a specific operation (e.g., scan, join, aggregate), requests data from its child nodes as input tuples (like an iterator), and provides the operation’s result as output. The child nodes usually read the data of tables, and the root node provides the final result of the query. The interface of the nodes is standardized, which means that the nodes can be easily combined to create different plans (see the open-next-close protocol). Most nodes just let tuples pass in a streaming manner and work on only one tuple at a time. However, some nodes, like the sort operations, have to read the entire input of the child node before they can emit the first output tuple.

Another representation of the plan is the following, which shows the plan nodes and their relationships in a graph format:

Trace Plan Alternatives using pg_plan_alternatives

For all queries, the optimizer considers multiple alternative plans during the planning phase. However, PostgreSQL does not provide a way to inspect these alternatives. This is where pg_plan_alternatives comes into play.

pg_plan_alternatives uses eBPF (Extended Berkeley Packet Filter) to instrument the PostgreSQL optimizer. eBPF allows loading custom programs into the kernel and attaching them to various events, such as function calls. By attaching an eBPF program to the add_path function of the PostgreSQL optimizer, the tool can capture all the alternative paths that are generated and considered. Paths are an early lightweight representation of a plan node during query planning. Such a path consists of an operator, the estimated costs, and the number of tuples the node is expected to process.

High-Level Architecture of pg_plan_alternatives

pg_plan_alternatives consists of three main components:

• An eBPF program that runs in kernel space.
• A user-space script, pg_plan_alternatives, that collects events emitted by the eBPF program and loads the eBPF program into the kernel.
• A visualization script, visualize_plan_graph, that takes the collected events and visualizes the alternative plans.

The pg_plan_alternatives Python script, which runs in user space, collects the data emitted by the eBPF program and prints the received events. These events can then be visualized using the visualize_plan_graph script, which is also part of the pg_plan_alternatives project. The visualization shows the alternative plans and their costs in a graph format, which makes it easier to understand the decision-making process of the optimizer.

Capturing Paths During Query Planning

Capturing the plans directly at the moment they are generated is necessary, since there is no point in time when all the alternative query plans are available in memory. The optimizer removes alternatives that are not promising directly using pfree and only keeps the most promising ones (e.g., those with lower estimated costs). When the query planning is done, a second probe is attached to the create_plan function, which is responsible for creating the final execution plan. This allows pg_plan_alternatives to determine which of the alternatives was finally chosen by the optimizer.

Handling Function Parameters in eBPF

A challenge is dealing with the function parameters in the eBPF program, since PostgreSQL structs like Path are opaque to eBPF. The function parameters are therefore pointers to opaque data structures, but the eBPF program needs to access certain fields (e.g., the type of the path or the costs).

There are three approaches to this problem:

• Copy PostgreSQL structs from the source code into the eBPF program. This is possible because both are written in C. However, eBPF supports a limited set of datatypes, and complex struct members (such as pointers to other structs) must be resolved and converted to simple data types, which requires considerable work.

• Hard-code the byte offsets of the fields. Like the previous approach, this makes the tool fragile and less robust to changes in the PostgreSQL codebase.

• The approach used by pg_plan_alternatives is to analyze the PostgreSQL binary and extract the offsets of the relevant struct fields using DWARF debug information. This debug information lets the plan tracer determine the byte offset of each field. These offsets are extracted by the user-space part of pg_plan_alternatives and provided to the eBPF program using #define directives. With these offsets, the eBPF program can locate the relevant fields and read the necessary information. Extracting offsets dynamically allows pg_plan_alternatives to adapt to different PostgreSQL versions without changing the eBPF program.

Insights from pg_plan_alternatives

The insights that pg_plan_alternatives provides can be used in several places.

• The PostgreSQL planner can be tuned using configuration parameters such as random_page_cost or cpu_tuple_cost. These parameters feed into the cost functions and influence the planner’s estimates; they should match the actual system environment so the optimizer can make good decisions. Using pg_plan_alternatives, you can see which alternatives are considered and how close their costs are.

• Extension developers who rewrite query plans during the planning phase should be aware of the alternatives considered by the optimizer, since their code must handle all relevant cases correctly. pg_plan_alternatives helps visualize the alternatives generated by the optimizer.

Examples

In this section examples of how pg_plan_alternatives can be used to gain insights into PostgreSQL’s query planning process. All examples below use two tables, each with 1000 rows and up-to-date statistics:

CREATE TABLE test1(id INTEGER PRIMARY KEY);
CREATE TABLE test2(id INTEGER PRIMARY KEY);

INSERT INTO test1 SELECT generate_series(1, 1000);
INSERT INTO test2 SELECT generate_series(1, 1000);

ANALYZE;

The pg_plan_alternatives tracer can be installed using the following command:

pip install pg_plan_alternatives

Simple SELECT Query

To inspect the alternative plans for a simple SELECT query, the query tracer can be started as follows:

sudo pg_plan_alternatives -x /home/jan/postgresql-sandbox/bin/REL_17_1_DEBUG/bin/postgres -n $(pg_config --includedir-server)/nodes/nodetags.h

The -x parameter specifies the path to the PostgreSQL binary that should be instrumented, while the -n parameter specifies the path to the nodetags.h header file, which contains the definitions of the plan node types. When no -p parameter is specified, the tool will trace all running processes for that binary. After starting the tracer, the following query can be executed:

SELECT * FROM test1;

The tracer should show an output as follows:

================================================================================
PostgreSQL Plan Alternatives Tracer
Binary: /home/jan/postgresql-sandbox/bin/REL_17_1_DEBUG/bin/postgres
Tracing all PostgreSQL processes
================================================================================

Received event: PID=3917080, Type=ADD_PATH, PathType=T_SeqScan
[20:14:54.116] [PID 3917080] ADD_PATH: T_SeqScan (startup=0.00, total=15.00, rows=1000, parent_rti=1, parent_oid=26144)
Received event: PID=3917080, Type=ADD_PATH, PathType=T_IndexOnlyScan
[20:14:54.118] [PID 3917080] ADD_PATH: T_IndexOnlyScan (startup=0.28, total=43.27, rows=1000, parent_rti=1, parent_oid=26144)
Received event: PID=3917080, Type=ADD_PATH, PathType=T_BitmapHeapScan
[20:14:54.118] [PID 3917080] ADD_PATH: T_BitmapHeapScan (startup=25.52, total=40.52, rows=1000, parent_rti=1, parent_oid=26144)
Received event: PID=3917080, Type=ADD_PATH, PathType=T_SeqScan
[20:14:54.118] [PID 3917080] ADD_PATH: T_SeqScan (startup=0.00, total=15.00, rows=1000, parent_oid=26144)
Received event: PID=3917080, Type=CREATE_PLAN, PathType=T_SeqScan
[20:14:54.118] [PID 3917080] CREATE_PLAN: T_SeqScan (startup=0.00, total=15.00) [CHOSEN]

The output already gives insights into the planning process. For example, the optimizer considered three different plans for scanning the test1 table: a SeqScan, an IndexOnlyScan, and a BitmapHeapScan. The costs of these plans are shown. When the CREATE_PLAN event is emitted, the SeqScan plan was chosen by the optimizer, which is also reflected in the EXPLAIN output of the query:

jan2=# EXPLAIN (VERBOSE, ANALYZE) SELECT * FROM test1;
 QUERY PLAN
-------------------------------------------------------------------------------------------------------------
 Seq Scan on public.test1  (cost=0.00..15.00 rows=1000 width=4) (actual time=0.119..0.291 rows=1000 loops=1)
   Output: id
 Planning Time: 0.855 ms
 Execution Time: 0.437 ms
(4 rows)

To visualize the alternatives, the tracer output can be formatted as JSON and stored in a file. To run pg_plan_alternatives in that mode, the following command can be used:

$ sudo pg_plan_alternatives -x /home/jan/postgresql-sandbox/bin/REL_17_1_DEBUG/bin/postgres -n $(pg_config --includedir-server)/nodes/nodetags.h -j -o examples/select.json

The SELECT SQL query should be repeated while the tracer is running to capture the events. After that, the visualize_plan_graph script can be used to visualize the alternatives:

$ visualize_plan_graph -i examples/select.json -o examples/select.svg --db-url psql://localhost/jan2 -v 

This produces an .svg file with the following content:

The graph shows all nodes considered for scanning the base relation test1 and their costs. The green T_SeqScan node is the one finally chosen by the optimizer, with costs of 0.00..15.00, while the gray-blue nodes are the alternatives considered but not chosen.

SELECT Query with WHERE Clause

The second example is a SELECT query with a WHERE clause. In this example, the pg_plan_alternatives tracer should be started as in the previous example. The query in this example is as follows:

SELECT * FROM test1 WHERE id = 5;

PostgreSQL will choose an Index Only Scan for this query, since there is an index on the id column. The EXPLAIN output of the query should look as follows:

jan2=# EXPLAIN (VERBOSE, ANALYZE) SELECT * FROM test1 WHERE id = 5;
 QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------
 Index Only Scan using test1_pkey on public.test1  (cost=0.28..8.29 rows=1 width=4) (actual time=0.153..0.160 rows=1 loops=1)
   Output: id
   Index Cond: (test1.id = 5)
 Heap Fetches: 1
 Planning Time: 1.166 ms
 Execution Time: 0.284 ms
(6 rows)

The output of the plan visualization should look as follows:

The same nodes as in the previous example are shown, but now the T_IndexOnlyScan node is the one that was chosen by the optimizer with costs of 0.28..8.29, while the T_SeqScan and T_BitmapHeapScan nodes are the alternatives that were considered but not chosen.

SELECT Query with ORDER BY Clause

The third example is a SELECT query with an ORDER BY clause. The example query is as follows:

SELECT * FROM test1 ORDER BY id;

The EXPLAIN output of the query shows that an Index Only Scan is used to scan the test1 table, which is also used to return the rows in the correct order.

EXPLAIN (VERBOSE, ANALYZE) SELECT * FROM test1 ORDER BY id;
 QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
 Index Only Scan using test1_pkey on public.test1  (cost=0.28..43.27 rows=1000 width=4) (actual time=0.192..5.385 rows=1000 loops=1)
   Output: id
 Heap Fetches: 1000
 Planning Time: 1.167 ms
 Execution Time: 5.579 ms
(5 rows)

However, this time the optimizer also considered producing the result by performing a Seq Scan and then sorting the output using a T_Sort node. This is followed by a T_Result node, an internal PostgreSQL node used for operations such as applying a projection. Since the costs of this path are higher than the Index Only Scan path, it was not chosen by the optimizer.

SELECT Query with GROUP BY Clause

The fourth example is a SELECT query with a GROUP BY clause. This time, a simple COUNT aggregation is performed, and a GROUP BY is applied on the id column. The example query is as follows:

SELECT id, COUNT(*) FROM test1 GROUP BY id;

The EXPLAIN output of the query shows that a HashAggregate node is used to perform the aggregation, which is fed by a Seq Scan node that scans the test1 table. The HashAggregate node has an estimated cost of 20.00..30.00.

jan2=# EXPLAIN (VERBOSE, ANALYZE) SELECT id, COUNT(*) FROM test1 GROUP BY id;
 QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
 HashAggregate  (cost=20.00..30.00 rows=1000 width=12) (actual time=3.171..4.009 rows=1000 loops=1)
   Output: count(*), id
 Group Key: test1.id
   Batches: 1 Memory Usage: 193kB
 ->  Seq Scan on public.test1  (cost=0.00..15.00 rows=1000 width=4) (actual time=0.176..0.769 rows=1000 loops=1)
         Output: id
 Planning Time: 2.297 ms
 Execution Time: 4.637 ms
(8 rows)

The plan visualization shows that the optimizer also considered aggregating on top of an Index Only Scan; sorting the result of the Seq Scan and then aggregating was also considered. Since the costs of these alternatives are higher than the chosen plan, they were not selected. The chosen plan has a total cost of 30.00, while the two alternatives have total costs of 58.28 and 82.33.

JOIN Query

The fifth example is a JOIN query, which joins the test1 and test2 tables on the id column. The query used in this example is:

SELECT * FROM test1 LEFT JOIN test2 ON (test1.id = test2.id);

The EXPLAIN output of the query shows that a Hash Left Join is used to perform the join, which is fed by two Seq Scan nodes that scan the test1 and test2 tables. The Hash Left Join node has an estimated cost of 27.50..45.14.

jan2=# EXPLAIN (VERBOSE, ANALYZE) SELECT * FROM test1 LEFT JOIN test2 ON (test1.id = test2.id);
 QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------
 Hash Left Join (cost=27.50..45.14 rows=1000 width=8) (actual time=0.625..1.422 rows=1000 loops=1)
   Output: test1.id, test2.id
 Inner Unique: true
   Hash Cond: (test1.id = test2.id)
 ->  Seq Scan on public.test1  (cost=0.00..15.00 rows=1000 width=4) (actual time=0.038..0.220 rows=1000 loops=1)
         Output: test1.id
 ->  Hash (cost=15.00..15.00 rows=1000 width=4) (actual time=0.571..0.572 rows=1000 loops=1)
         Output: test2.id
 Buckets: 1024  Batches: 1 Memory Usage: 44kB
 ->  Seq Scan on public.test2  (cost=0.00..15.00 rows=1000 width=4) (actual time=0.019..0.191 rows=1000 loops=1)
               Output: test2.id
 Planning Time: 3.436 ms
 Execution Time: 1.551 ms
(13 rows)

The plan visualization shows that the optimizer considered many alternatives for joining the tables. Most alternatives try joining test1 with test2 using different join algorithms (e.g., Nested Loop, Merge Join, or Hash Join) and different access paths for the base relations like Seq Scan, Index Only Scan, or Bitmap Heap Scan. The optimizer also considered flipping the join order and joining test2 with test1. However, the costs of these alternatives are higher than the chosen plan. For example, the Nested Loop Join using an Index Only Scan on test1 and a Seq Scan on test2 has an estimated cost of 27515.83, which is much higher than the chosen plan’s total cost of 45.14.

JOIN Query with WHERE Clause

The last example is a JOIN query with a WHERE clause. The example query is as follows:

SELECT * FROM test1 LEFT JOIN test2 ON (test1.id = test2.id) WHERE test1.id=123;

The EXPLAIN output indicates that a Nested Loop Left Join is used to perform the join, which is fed by an Index Only Scan on the test1 table and an Index Only Scan on the test2 table. The Nested Loop Left Join node has an estimated cost of 0.55..16.60.

jan2=# EXPLAIN (VERBOSE, ANALYZE) SELECT * FROM test1 LEFT JOIN test2 ON (test1.id = test2.id) WHERE test1.id=123;
 QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------
 Nested Loop Left Join (cost=0.55..16.60 rows=1 width=8) (actual time=0.183..0.189 rows=1 loops=1)
   Output: test1.id, test2.id
 Inner Unique: true
 ->  Index Only Scan using test1_pkey on public.test1  (cost=0.28..8.29 rows=1 width=4) (actual time=0.139..0.143 rows=1 loops=1)
         Output: test1.id
         Index Cond: (test1.id = 123)
 Heap Fetches: 1
 ->  Index Only Scan using test2_pkey on public.test2  (cost=0.28..8.29 rows=1 width=4) (actual time=0.032..0.032 rows=1 loops=1)
         Output: test2.id
         Index Cond: (test2.id = 123)
 Heap Fetches: 1
 Planning Time: 1.116 ms
 Execution Time: 0.336 ms
(13 rows)

The plan visualization shows that different ways to access the base relations and different join algorithms were considered by the optimizer. The optimizer also considers performing a Bitmap Heap Scan on test2 and materializing the result as input for the Nested Loop join.

Conclusion

In this article, the pg_plan_alternatives tool was discussed, which uses eBPF to trace the alternative query plans that are considered by the PostgreSQL optimizer during the planning phase. The tool consists of an eBPF program that runs in kernel space and a user-space script that collects the events emitted by the eBPF program and visualizes the alternatives. By using pg_plan_alternatives, you can gain insights into the decision-making process of the optimizer and tune system parameters accordingly. Furthermore, the basics of cost-based optimization and the structure of query plans in PostgreSQL were explained. In addition, several examples were shown to demonstrate how pg_plan_alternatives can be used to inspect the alternative plans for different types of queries. The tool is open-source and can be found on GitHub.

This article explains what spinlocks are and how they are implemented in PostgreSQL. It also describes how spinlocks can be monitored and demonstrates how my new pg_spinlock_tracer tool can be used to trace spinlock internals using eBPF.

What are Spinlocks?

When multiple processes need to access a shared resource, locks are used to ensure that only one process can modify the resource at a time. If a lock is not available, the waiting process is put to sleep until the lock can be acquired. This reduces CPU usage since the waiting process does not consume CPU cycles while sleeping. However, putting a process to sleep and waking it up again involves context switches, which take time and add latency to the operation. If the lock is expected to be held for a very short time, it may be more efficient for the waiting process to continuously check if the lock is available instead of sleeping. That is what spinlocks do: the lock spins in a loop, repeatedly checking the lock’s status until it can be acquired. Using a spinlock avoids the sleep/wakeup latency but can consume CPU cycles while spinning. If the hardware has only a few CPU cores, spinning can waste CPU cycles and lead to worse overall performance.

Implementation in PostgreSQL

The PostgreSQL implementation of spinlocks is mainly in src/include/storage/s_lock.h and src/backend/storage/lmgr/s_lock.c. The spinlock API provides four basic operations:

• SpinLockInit: Initializes a spinlock.
• SpinLockAcquire: Acquires a spinlock, blocking until it is available.
• SpinLockRelease: Releases a spinlock.
• SpinLockFree: Checks if a spinlock is free.

Note: SpinLockAcquire can also raise a FATAL error if the lock cannot be acquired within a certain time limit. In that case, the server terminates, performs recovery on restart, and becomes available again once recovery finishes.

Using Spinlocks

To use a spinlock, it must first be initialized using SpinLockInit.

slock_t mutex;
SpinLockInit(&mutex);

After initialization, the lock can be acquired and released as needed:

SpinLockAcquire(&mutex);
/* critical section */
SpinLockRelease(&mutex);

To determine if a spinlock is currently held by another process, the function SpinLockFree can be used:

if (!SpinLockFree(&mutex))
    /* lock is held by another process */

Spinlocks are used in several places in the PostgreSQL codebase, for example, to coordinate access in the write-ahead log (WAL) implementation or during checkpoints.

Implementation Details

The implementation is split into a platform-independent part and platform-specific parts. The platform-independent code in s_lock.c defines the API and higher-level behavior, while s_lock.h pulls in platform-specific assembly implementations depending on the target architecture.

Acquiring a Spinlock

To acquire a spinlock, PostgreSQL performs an atomic test-and-set (TAS) on the lock variable. The lock value is 0 when free and 1 when held. The TAS operation is atomic to avoid races where two processes both observe a free lock and try to acquire it simultaneously.

The platform-independent code for acquiring a lock looks as follows:

int
s_lock(volatile slock_t *lock, const char *file, int line, const char *func)
{
	SpinDelayStatus delayStatus;

	init_spin_delay(&delayStatus, file, line, func);

	while (TAS_SPIN(lock))
	{
		perform_spin_delay(&delayStatus);
	}

	finish_spin_delay(&delayStatus);

	return delayStatus.delays;
}

A struct SpinDelayStatus is used to track the number of spins and delays (this will be discussed in the next section). The platform-dependent macro TAS_SPIN performs the fast-path check and the actual test-and-set operation on the lock variable. As long as the lock is held by another process, TAS_SPIN returns 1 and the loop continues, calling perform_spin_delay before the next attempt. Once the lock becomes available, TAS_SPIN returns 0 and the loop terminates.

The implementation of TAS_SPIN for the x86-64 architecture looks as follows:

#define TAS_SPIN(lock)    (*(lock) ? 1 : TAS(lock))

static __inline__ int
tas(volatile slock_t *lock)
{
	slock_t		_res = 1;

	__asm__ __volatile__(
		"   lock         \n"
		"   xchgb  %0,%1 \n"
:		"+q"(_res), "+m"(*lock)
:		/* no inputs */
:		"memory", "cc");
	return (int) _res;
}

The macro TAS_SPIN first checks whether the lock variable is non-zero; if so, it returns 1 immediately without performing the atomic exchange. If the lock variable is 0, it calls TAS(lock) (which ultimately invokes tas) to perform the atomic test-and-set operation.

The tas function performs the atomic exchange using inline assembly. The lock prefix ensures the instruction is executed atomically across multiple CPU cores. The xchgb instruction swaps _res and the lock variable: _res starts at 1, so if the lock was free (0), the swap sets the lock to 1 and _res becomes 0 (success). If the lock was already 1, _res becomes 1 (failure to acquire). The function returns _res (0 on success, 1 on failure).

Spinlock Contention

When the lock cannot be acquired, the function perform_spin_delay is invoked. It implements an adaptive backoff and looks like this:

void
perform_spin_delay(SpinDelayStatus *status)
{
    [...]
	if (++(status->spins) >= spins_per_delay)
	{
		if (++(status->delays) > NUM_DELAYS)
			s_lock_stuck(status->file, status->line, status->func);

		if (status->cur_delay == 0) /* first time to delay? */
			status->cur_delay = MIN_DELAY_USEC;

		[...]
		pg_usleep(status->cur_delay);
		[...]

		/* increase delay by a random fraction between 1X and 2X */
		status->cur_delay += (int) (status->cur_delay *
			pg_prng_double(&pg_global_prng_state) + 0.5);

		/* wrap back to minimum delay when max is exceeded */
		if (status->cur_delay > MAX_DELAY_USEC)
			status->cur_delay = MIN_DELAY_USEC;

		status->spins = 0;
    }
}

On each invocation of the function, the number of spins is increased by one. If the number of spins exceeds a certain threshold (spins_per_delay), PostgreSQL sleeps for a few microseconds (pg_usleep) before the next attempt to acquire the lock. This turns PostgreSQL’s spinlocks into a hybrid approach (spin first, then sleep) and serves as a safety mechanism to prevent excessive CPU usage under high contention. It is only performed after a certain number of spins, which indicates that the lock was held for an extended period by another process.

Additionally, the delay is increased by a random fraction between 1X and 2X on every delay, which means that the delay increases exponentially with the number of delays. If the delay exceeds a certain maximum value (MAX_DELAY_USEC, 1000000 microseconds by default), it is wrapped back to a minimum value (MIN_DELAY_USEC, 1000 microseconds by default). This prevents the delay from growing indefinitely and ensures that the process will eventually wake up and try to acquire the lock again. The random fraction adds jitter, which can help reduce contention by preventing multiple processes from waking up and trying to acquire the lock at the same time.

If the number of delays exceeds NUM_DELAYS (default 1000), PostgreSQL calls s_lock_stuck, which raises a FATAL error indicating that the lock appears stuck.

Monitoring Spinlocks

Monitoring spinlocks and understanding spinlock contention can be crucial for diagnosing performance issues in PostgreSQL. In the following sections, an artificial spinlock contention is created and then observed using the pg_stat_activity view and the pg_spinlock_tracer tool.

Note: This example should not be executed on a production system, since it will cause the server to become unresponsive and may eventually terminate due to the FATAL error raised by s_lock_stuck.

Creating Artificial Spinlock Contention

To create such an artificial contention, two sessions to a database are opened. Afterward, two different tables are created:

CREATE TABLE data1 (id INT);
CREATE TABLE data2 (id INT);

Furthermore, a debugger is attached to the first session, and a breakpoint is set in ReserveXLogInsertLocation. This function is responsible for reserving space in the write-ahead log (WAL) for a new record. It uses a spinlock to coordinate access to the WAL insertion point. Afterward, the first session performs an INSERT statement, which will cause the process to acquire the spinlock in ReserveXLogInsertLocation and then wait at the breakpoint.

INSERT INTO data1 VALUES (1);

After the breakpoint is hit, the following statements should be executed in the debugger until the line SpinLockAcquire(&Insert->insertpos_lck); is executed.

In the second session, another INSERT statement is executed, which will also try to acquire the same spinlock in ReserveXLogInsertLocation and wait for the lock to be released by the first session.

INSERT INTO data2 VALUES (1);

Two different tables are used to ensure that the contention is on the spinlock in ReserveXLogInsertLocation and not on another lock related to the table access.

Using pg_stat_activity

The view pg_stat_activity of the cumulative statistics system provides information about the current activity of all sessions in the PostgreSQL server. Lock contention can also be seen in this view.

mydb=# SELECT pid, backend_start, wait_event_type, wait_event, state, query from pg_stat_activity;
   pid   |         backend_start         | wait_event_type |     wait_event      | state  |  query                                            
---------+-------------------------------+-----------------+---------------------+--------+-----------------------------
 2129513 | 2026-02-08 19:48:26.32229+01  |                 |                     | active | insert into data1 values(1);
 2129736 | 2026-02-08 19:49:00.578201+01 | Timeout         | SpinDelay           | active | insert into data2 values(1);
[...]

The output shows that the second session (PID 2129736) is waiting for a SpinDelay, which indicates that it is trying to acquire a spinlock but is currently delayed due to contention. More information about this view and the meaning of the different columns can be found in the documentation.

However, this view only provides a high-level overview of the lock contention and does not provide detailed information about the spinlock behavior, such as the number of spins and delays or the current delay value. For that, a more detailed tracing tool is needed.

Tracing Spinlocks with pg_spinlock_tracer

To trace spinlock contention in PostgreSQL, I implemented pg_spinlock_tracer as part of the pg-lock-tracer project. The tool uses eBPF to instrument the perform_spin_delay function and prints the contents of the SpinDelayStatus struct. For instance, it reports the number of spins and delays, the current delay, and the source location where the spinlock is being attempted.

Unlike the PostgreSQL view, pg_spinlock_tracer shows the internals of spinlock acquisition and contention, which can be useful for understanding behavior. A simple output of the tool looks as follows:

$ pg_spinlock_delay_tracer -x /home/jan/postgresql-sandbox/bin/REL_17_1_DEBUG/bin/postgres
[...]
13180680737869452 [Pid 1864403] SpinDelay spins=996 delays=939 cur_delay=566086 at ReserveXLogInsertLocation, xlog.c:1132
13180680737874986 [Pid 1864403] SpinDelay spins=997 delays=939 cur_delay=566086 at ReserveXLogInsertLocation, xlog.c:1132
13180680737880522 [Pid 1864403] SpinDelay spins=998 delays=939 cur_delay=566086 at ReserveXLogInsertLocation, xlog.c:1132
13180680737886009 [Pid 1864403] SpinDelay spins=999 delays=939 cur_delay=566086 at ReserveXLogInsertLocation, xlog.c:1132
13180681304189362 [Pid 1864403] SpinDelay spins=0 delays=940 cur_delay=661655 at ReserveXLogInsertLocation, xlog.c:1132
13180681304227806 [Pid 1864403] SpinDelay spins=1 delays=940 cur_delay=661655 at ReserveXLogInsertLocation, xlog.c:1132
13180681304241759 [Pid 1864403] SpinDelay spins=2 delays=940 cur_delay=661655 at ReserveXLogInsertLocation, xlog.c:1132
13180681304255150 [Pid 1864403] SpinDelay spins=3 delays=940 cur_delay=661655 at ReserveXLogInsertLocation, xlog.c:1132
[...]

The output shows that PID 1864403 (the second session to PostgreSQL) is trying to acquire a spinlock in ReserveXLogInsertLocation (xlog.c:1132). In the example, the process spins up to 999 times; once it reaches the threshold, it sleeps for cur_delay microseconds, and the spin counter is reset (visible as spins=0). The delay value then grows for subsequent attempts.

Conclusion

This article provided an overview of spinlocks in PostgreSQL, their implementation details, and how to observe spinlock contention. Spinlocks are a crucial part of PostgreSQL’s locking mechanism for short-term protection of shared resources. Understanding how they work and how to analyze contention can be valuable for diagnosing performance issues in PostgreSQL. The cumulative statistics system provides some insights into lock contention. The new pg_spinlock_tracer tool offers a more detailed view of the spinlock behavior and contention patterns.

This blog post explores the fundamentals of flame graphs and offers a few practical tips on utilizing them to identify and debug performance bottlenecks in PostgreSQL.

The content presented in this blog post is based on material found in other articles or blog posts, as well as in Brendan Gregg’s excellent book on system performance. Over the years, I have collected a number of commands in my lab notebook that I typically use when diagnosing PostgreSQL-related performance problems. I have shared these commands in several emails over the years, so I decided to write a whole blog post on this topic.

Flame Graphs

Flame graphs are based on data captured by a profiler. They aggregate call stacks to make it easier to see where a program spends most of its processing time. Without aggregation, it is difficult to see the big picture in the thousands (or more) of call stacks that a profiler collects.

When a flame graph is created, these call stacks are collapsed, and the time spent in similar call stacks is summed up. Based on this data, the flame graph is created. The idea behind this is as follows: the more time a program spends in a particular code path, the more often those call stacks will appear in the samples. Since the resulting graph consists of call stacks of different heights, and the stacks are usually colored in red to yellow tones, it looks like a flame.

Brendan Gregg states in ‘The Flame Graph’, ACM Queue, Vol 14, No 2:

A flame graph visualizes a collection of stack traces (aka call stacks), shown as an adjacency diagram with an inverted icicle layout.7 Flame graphs are commonly used to visualize CPU profiler output, where stack traces are collected using sampling.

An example flame graph looks like this:

In this flame graph, it can be seen that most time is spent in the ExecModifyTable function, which is part of the PostgreSQL executor. This function calls other functions like ExecInsert or ExecProcNode. ExecInsert is the one which takes the most time. So, when performing a performance analysis (and searching for functions that are worth optimizing), it is important to focus on the functions with a long bar on the x-axis. The longer the bar, the more time is spent in that function.

Creating a Flame Graph

As I primarily work on diagnosing performance issues in PostgreSQL, this blog post will focus on creating flame graphs for PostgreSQL. However, the presented methods can also be used to generate flame graphs for other applications written in C or Rust.

To profile a PostgreSQL backend process, the PID (Process ID) of the process needs to be known. To get the PID of the PostgreSQL backend process, you can use the following command:

mydb=# select pg_backend_pid();
 pg_backend_pid
----------------
        2112031
(1 row)

In this example, the PID is 2112031. Now, the data for the flame graph can be collected. Two ways to collect and process the data are available: using the perf tool or using the FlameGraph tool by Brendan Gregg. The first method is more straightforward since only one command is needed to collect and process the data. However, the second method is more flexible and allows for more advanced processing of the collected data. So, let’s start with the first FlameGraph method first.

Processing the Collected Data using FlameGraph

The FlameGraph tool can be found in the FlameGraph repository. To use it, you need to clone the repository:

git clone https://github.com/brendangregg/FlameGraph

This needs to be done only once. Afterward, the perf tool can be used to collect the data for the flame graph. Perf is a powerful profiler available for Linux systems.

The following command captures the call stacks of the PostgreSQL backend process with PID 2112031:

sudo perf record -a -g -F 111 -o data.perf -p 2112031

The parameter -a means that all CPUs are monitored, -g enables call graph recording, -F 111 sets the sampling frequency to 111 Hz, and -o data.perf specifies the output file. The frequency should be adjusted based on the monitored process (shorter workloads may require a higher frequency). In addition, the frequency should be set to a value that is not used by any cyclic task (e.g., a job that is executed every 100 ms) in the system. Otherwise, the profile will always or never run at the moment the cyclic task is executed. So, the value 111 Hz is a good choice, as it is not a common value for cyclic tasks.

Now, the workload that you want to profile should be executed. For example, the following query can be executed a few times to let the PostgreSQL backend process do some work:

INSERT INTO data (key, value) SELECT i, i::text FROM generate_series(1, 100000) i;

The \watch command of the psql client can be used to execute the query repeatedly to allow the backend process to do some work. After some time (usually when the workload that should be profiled is complete), the perf tool can be terminated using CTRL+C and used to process the collected data. The following command generates a text file with the call stacks:

perf script -i data.perf > data.stacks

This command reads the data from the data.perf file and writes the call stacks to the data.stacks file, which can then be used to generate a flame graph. However, before the flame graph can be generated, the data.stacks file needs to be processed and folded:

 ~/FlameGraph/stackcollapse-perf.pl data.stacks > data.folded

Folding means that the call stacks are aggregated so that the time spent in similar call stacks is summed up. Based on this data, the flame graph can be created:

~/FlameGraph/flamegraph.pl data.folded > data.svg

The resulting SVG can be found here for reference.

Processing the Data with ‘perf script’

perf script flamegraph -a -F 111 -p 2112031 sleep 60

This command unifies all the manual commands we have used so far. It collects the call stacks of the PostgreSQL backend process with PID 2112031 for 60 seconds, and then generates a flame graph afterward.

Unfortunately, this method does not work out-of-the-box on Debian based distributions.

Flame Graph template /usr/share/d3-flame-graph/d3-flamegraph-base.html does not exist. Please install the js-d3-flame-graph (RPM) or libjs-d3-flame-graph (deb) package, specify an existing flame graph template (--template PATH) or another output format (--format FORMAT).

The reason for the error is that the required template file is not available on Debian-based systems. This is a known issue and has been reported in the Debian bug tracker. Until the bug is fixed and the package is created for Debian, you can use the package from the Fedora repository. To download the package and convert it to a Debian package, use the following commands:

wget https://rpmfind.net/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/j/js-d3-flame-graph-4.0.7-10.fc42.noarch.rpm
apt-get install alien
fakeroot alien js-d3-flame-graph-4.0.7-10.fc42.noarch.rpm
sudo dpkg -i js-d3-flame-graph_4.0.7-11_all.deb

Afterwards, the perf script command can be used to generate a flame graph. Although the flame graph can be generated using the perf script command, I prefer to use the FlameGraph tool directly, as it also allows you to generate differential flame graphs (see below).

The resulting HTML page can be found here for reference.

Build Types

When profiling, it is important to decide which build type should be used. A highly optimized build is typically used in production and can also be profiled. However, many functions are inlined in such builds, which can lead to misleading results in the flame graph. For example, if a function is inlined, it will not appear in the flame graph, even if it consumes a significant amount of time. Alternatively, a debug build can be used for profiling. However, a debug build is not optimized and may exhibit different performance characteristics than a production build. Additionally, some functions might only be executed in a debug build (e.g., assertions), which can also lead to misleading results in the flame graph. Therefore, it is essential to be aware of the build type used for profiling and to interpret the results accordingly.

For some hard-to-catch performance problems, I have created profiles for both build types (optimized and debug) and compared the results.

When compiling a debug version of PostgreSQL, I use the following options:

CFLAGS="-ggdb -O0 -g3 -fno-omit-frame-pointer"

Different Types of Flame Graphs

Several types of flame graphs exist. In addition to the standard flame graph, the most common are off-CPU flame graphs and differential flame graphs. These will be discussed in the following sections.

On-CPU / Off-CPU Flame Graphs

Usually, the flame graphs show the time spent in the code while the CPU is executing it. However, sometimes it is also useful to see how much time a process spends waiting for resources (e.g., I/O operations). In this case, off-CPU flame graphs can be used. The profiler also adds the time to the call stacks when the process is not running on the CPU.

An off-CPU flame graph can be taken by using the offcputime binary. For example:

sudo offcputime-bpfcc -df -p 12345 > out.stacks

The parameter -d means that these should be a delimiter between kernel/user stacks, -f produces the needed folded format. 12345 must be replaced with the process ID of the process to be observed.

Afterward, the output can be processed as follows:

~/FlameGraph/flamegraph.pl --color=io --title="Off-CPU Time Flame Graph" --countname=us < out.stacks > out.svg

Differential Flame Graphs

Differential flame graphs are used to compare two different profiler runs. Usually, the first run is a baseline run, which is used to compare the second run (e.g., after a potential performance improvement).

Using a differential flame graph makes the differences between the runs more obvious. Otherwise, the width of the bars has to be compared, which is a hard task. To create a differential flame graph, the .folded files of the two runs need to be created first. Afterward, the script difffolded.pl from the FlameGraph tool can be used to create a differential flame graph:

~/FlameGraph/difffolded.pl data.folded data2.folded | ~/FlameGraph/flamegraph.pl > diff.svg

The resulting SVG file will clearly highlight the differences between the two profiler runs. For example:

Functions marked in red are slower in the second run, while functions marked in blue are faster.

Conclusion

This blog post provides an overview of how to create flame graphs for PostgreSQL (and other applications) using the perf tool and the FlameGraph tool. I frequently use flame graphs to gain insight into a program’s performance characteristics, identify potential bottlenecks, and find functions that are worth optimizing.

The methods presented work well with C or Rust code. If you want to profile a Java application, I highly recommend using the async-profiler tool, which is optimized for Java applications and provides similar functionality.

The query optimizer is responsible for finding the most efficient plan for a given query. The plan generator creates possible plans, which are then evaluated based on their costs. Afterward, the cheapest plan is chosen and executed. When the DBMS expects to return a large portion of the table, a full table scan can be more efficient than following the pointers in an index structure. However, it is hard to determine when the DBMS prefers one plan over another and when the switch between plans occurs.

In a few evenings, I implemented the plan explorer for PostgreSQL. It iterates over a search space and generates visualizations that show when the plan changes and how many tuples are expected versus the actual number returned. This blog post examines the “art” of query optimization. It discusses the plan explorer tool, the images the tool generates, and the insights the tool provides into the decisions made by the PostgreSQL query optimizer.

Plan Explorer

Plan Explorer is a tool that iterates over a two-dimensional search space and executes a given SQL query for each parameter combination. Based on the returned query plans (and actual query executions), various diagrams are generated. These diagrams have an artistic aspect, but they also provide valuable insights into the workings of the query optimizer and visually represent the decisions made (i.e., at which point PostgreSQL considers using a specific index).

The tool is based on the ideas of Picasso, but implemented as a modern web application.

The Picasso database query optimizer visualizer, Jayant R. Haritsa, Proceedings of the VLDB Endowment, Volume 3, Issue 1-2, September 2010

Server Mode

Initially, the tool was developed as a standalone website that does not require any server component. The WebAssembly build of PostgreSQL, PGlite, is used to run PostgreSQL within the user’s browser and extract the needed query plans.

In the most recent version of the tool, an optional server mode was added. Using this server mode, queries are sent to a REST endpoint on a web server. On the web server, a script can take the requested queries and forward them to an actual PostgreSQL server. The server component acts as a proxy for query execution. This is necessary because, due to security constraints, JavaScript running in a browser cannot open network connections to other hosts. So, to communicate with a real PostgreSQL server, the proxy component is needed. However, when an actual PostgreSQL server is queried, large datasets can be preloaded or custom extensions can be integrated into the database server. This allows the tool to run in these environments as well (e.g., a PostgreSQL extension developer could check their custom cost models and whether PostgreSQL picks up the desired query plans).

The architecture of the plan explorer is shown in the diagram below:

flowchart LR
    subgraph **Browser**
        UI["WebUI"]
        PGLite["PGlite (WebAssembly PostgreSQL)"]
        UI -- "SQL, Parameters" --> PGLite
        PGLite -- "Query Results, Plans" --> UI
    end
    subgraph "**Proxy Mode** (Optional)"
        Proxy["Proxy Server"]
        PG["PostgreSQL Database"]
        UI -- "HTTP (SQL/Params)" --> Proxy
        Proxy -- "SQL" --> PG
        PG -- "Results" --> Proxy
        Proxy -- "Results" --> UI
    end

A further feature of the server mode is that the queries can also be executed (EXPLAIN (ANALYZE)) instead of only being planned (EXPLAIN). When only the planning of the query is performed, the tool operates much faster and can quickly iterate over the given search space. On the other hand, if the query is actually executed, PostgreSQL provides further information, such as the actual number of returned tuples or the execution time.

Generated Drawings

The plan explorer tool creates several different drawings from one query. These drawings are:

• The used query plans
• The expected costs of the query
• The actual execution time of the query
• The estimated number of result tuples
• The actual number of result tuples
• The difference between the estimated and actual number of result tuples

For a database administrator, the different query plans and their distribution might be the most interesting output of the tool. For a developer who has implemented custom scan nodes and cost models, the other outputs may also be of interest. They enable database developers to determine if the cost models function as expected.

Example Query Discussion

In this section, the output of the tool for the following query is discussed. The query performs a self-join of the table data on the attribute key. Two predicates in the WHERE clause define filter conditions. The exact values for these conditions are determined by the search space. In this example, the search space for both dimensions is the interval [0; 50000] and steps of 1000 are used.

SELECT * FROM data d1 LEFT JOIN data d2 ON (d1.key = d2.key) WHERE d1.key > %%DIMENSION0%% AND d2.key > %%DIMENSION1%%;

The query is executed after the database is prepared with the following SQL commands:

CREATE TABLE data(key integer, value text);
INSERT INTO data (key, value) SELECT i, i::text FROM generate_series(1, 100000) i;
CREATE INDEX ON data(key);
ANALYZE data;

So, the table data consists of 100000 tuples and has an index on the attribute key.

The query plan explorer reveals that five different query plans are used by PostgreSQL for the given parameter combinations of the search space.

Query Plans

In the following two subsections, two of the five different query plans are discussed. Query plan 1 is the light blue one in the middle left side of the drawing. Query plan 2 is the area in the upper left corner of the drawing.

Query Plan 1

The first query plan has the following fingerprint. Fingerprints are used by the query plan explorer to classify query plans as identical (e.g., the parameters in some nodes can change but the structure of the query plan is the same): Hash Join > Seq Scan(d1) > Hash > Seq Scan(d2)

So, the query plan consists of a hash join. One child node of the hash join is a sequential scan and the other child is a hash operator which also reads the tuples of the data table. No index scans are used.

Note: One interesting optimization in PostgreSQL is that the executed join type is changed. In the query, a LEFT JOIN is performed. However, an INNER JOIN ("Join Type": "Inner",) is actually executed. A left join guarantees that every tuple of the left input relation is contained in the output. If no join partner is found, the tuple is added to the output and all attributes of the right relation are populated with NULL values. However, since a filter is applied on the key attribute of the right relation (d2.key > %%DIMENSION1%%), none of these artificially generated tuples would fulfill the filter condition (a tuple with a NULL attribute for key will always be eliminated by the filter). So, there is no need to generate these tuples that are removed later. Therefore, PostgreSQL changes the join type to an inner join and does not generate these tuples at all.

[
  {
    "Plan": {
      "Node Type": "Hash Join",
      "Parallel Aware": false,
      "Async Capable": false,
      "Join Type": "Inner",
      "Startup Cost": 3040,
      "Total Cost": 6205,
      "Plan Rows": 100000,
      "Plan Width": 18,
      "Actual Startup Time": 20.053,
      "Actual Total Time": 55.887,
      "Actual Rows": 100000,
      "Actual Loops": 1,
      "Inner Unique": false,
      "Hash Cond": "(d1.key = d2.key)",
      "Plans": [
        {
          "Node Type": "Seq Scan",
          "Parent Relationship": "Outer",
          "Parallel Aware": false,
          "Async Capable": false,
          "Relation Name": "data",
          "Alias": "d1",
          "Startup Cost": 0,
          "Total Cost": 1790,
          "Plan Rows": 100000,
          "Plan Width": 9,
          "Actual Startup Time": 0.006,
          "Actual Total Time": 10.072,
          "Actual Rows": 100000,
          "Actual Loops": 1,
          "Filter": "(key > 0)",
          "Rows Removed by Filter": 0
        },
        {
          "Node Type": "Hash",
          "Parent Relationship": "Inner",
          "Parallel Aware": false,
          "Async Capable": false,
          "Startup Cost": 1790,
          "Total Cost": 1790,
          "Plan Rows": 100000,
          "Plan Width": 9,
          "Actual Startup Time": 20.036,
          "Actual Total Time": 20.036,
          "Actual Rows": 100000,
          "Actual Loops": 1,
          "Hash Buckets": 131072,
          "Original Hash Buckets": 131072,
          "Hash Batches": 1,
          "Original Hash Batches": 1,
          "Peak Memory Usage": 5115,
          "Plans": [
            {
              "Node Type": "Seq Scan",
              "Parent Relationship": "Outer",
              "Parallel Aware": false,
              "Async Capable": false,
              "Relation Name": "data",
              "Alias": "d2",
              "Startup Cost": 0,
              "Total Cost": 1790,
              "Plan Rows": 100000,
              "Plan Width": 9,
              "Actual Startup Time": 0.003,
              "Actual Total Time": 9.906,
              "Actual Rows": 100000,
              "Actual Loops": 1,
              "Filter": "(key > 0)",
              "Rows Removed by Filter": 0
            }
          ]
        }
      ]
    },
    "Planning Time": 0.504,
    "Triggers": [],
    "Execution Time": 58.059
  }
]

Query Plan 2

The second query plan Hash Join > Seq Scan(d1) > Hash > Index Scan(d2) has a similar structure as the first one but with one exception: an index scan is used for the input of the hash operation ("Node Type": "Index Scan").

[
  {
    "Plan": {
      "Node Type": "Hash Join",
      "Parallel Aware": false,
      "Async Capable": false,
      "Join Type": "Inner",
      "Startup Cost": 2530.96,
      "Total Cost": 5297.79,
      "Plan Rows": 60183,
      "Plan Width": 18,
      "Actual Startup Time": 26.732,
      "Actual Total Time": 48.043,
      "Actual Rows": 60000,
      "Actual Loops": 1,
      "Inner Unique": false,
      "Hash Cond": "(d1.key = d2.key)",
      "Plans": [
        {
          "Node Type": "Seq Scan",
          "Parent Relationship": "Outer",
          "Parallel Aware": false,
          "Async Capable": false,
          "Relation Name": "data",
          "Alias": "d1",
          "Startup Cost": 0,
          "Total Cost": 1790,
          "Plan Rows": 100000,
          "Plan Width": 9,
          "Actual Startup Time": 0.004,
          "Actual Total Time": 9.898,
          "Actual Rows": 100000,
          "Actual Loops": 1,
          "Filter": "(key > 0)",
          "Rows Removed by Filter": 0
        },
        {
          "Node Type": "Hash",
          "Parent Relationship": "Inner",
          "Parallel Aware": false,
          "Async Capable": false,
          "Startup Cost": 1778.67,
          "Total Cost": 1778.67,
          "Plan Rows": 60183,
          "Plan Width": 9,
          "Actual Startup Time": 18.759,
          "Actual Total Time": 18.76,
          "Actual Rows": 60000,
          "Actual Loops": 1,
          "Hash Buckets": 65536,
          "Original Hash Buckets": 65536,
          "Hash Batches": 1,
          "Original Hash Batches": 1,
          "Peak Memory Usage": 2973,
          "Plans": [
            {
              "Node Type": "Index Scan",
              "Parent Relationship": "Outer",
              "Parallel Aware": false,
              "Async Capable": false,
              "Scan Direction": "Forward",
              "Index Name": "data_key_idx",
              "Relation Name": "data",
              "Alias": "d2",
              "Startup Cost": 0.29,
              "Total Cost": 1778.67,
              "Plan Rows": 60183,
              "Plan Width": 9,
              "Actual Startup Time": 0.05,
              "Actual Total Time": 11.329,
              "Actual Rows": 60000,
              "Actual Loops": 1,
              "Index Cond": "(key > 40000)",
              "Rows Removed by Index Recheck": 0
            }
          ]
        }
      ]
    },
    "Planning Time": 0.131,
    "Triggers": [],
    "Execution Time": 49.318
  }
]

Expected Total Cost and Actual Time

Also, drawings for the expected total cost and the actual time are shown. The first image shows that PostgreSQL assumes the highest costs in the lower left corner. This is the moment when both predicates of the WHERE condition let all tuples (> 0) pass. The costs decrease when one of the predicates can filter more tuples, and PostgreSQL needs to join fewer tuples. The lowest costs are in the upper right corner of the image. The lines have some irregularities since the query plan changes for the parameter combinations, which also affects the total costs of the query.

Closely related to the last image is the total execution time. It shows how long the DBMS actually needs to execute the query. The image also shows the trend that the query becomes faster when fewer tuples have to be processed. However, the times are a bit more scattered. This has two reasons: (i) in the current version of the tool, just a single execution of the query is performed and outliers are not handled (e.g., by taking the average value over multiple executions), and (ii) the query is short and changing the parameters have not a big impact on the actual execution time.

Expected and Actual Tuples

The following next three images show the expected, actual returned tuples and the difference between these two values. It can be seen by comparing the first two images that the pattern of the expected and the actual returned tuples is different and PostgreSQL actually performed a misprediction. The first drawing shows diagonal lines and the actual graph shows horizontal lines (i.e., changing one of the parameters leads to more tuples in the output independently from the other parameter) and PostgreSQL expects some correlation (so-called cross-column dependencies) between these parameters. See the function clauselist_selectivity() and its comment for more details. Using extended statistics might help to get a better prediction (this might be covered in a follow-up blog posts).

The last drawing shows the difference between these two graphs. Even if a DBMS user is not usually pleased that the number of result tuples has been incorrectly estimated (as this may result in non-optimal plans being used), this error rewards us with a nice-looking picture.

Conclusion

This blog post covers the basic functionality of plan explorer. The tool is now available as open source at GitHub. The tool iterates over a two-dimensional search space and executes a query for every parameter combination. Based on the information returned from the database system, the tool draws visualizations of query plans, estimated values like time and returned tuples, and actual values such as the number of returned tuples.

Apart from the artistic aspect of the drawings, the tool gives you insights into the query optimizer decisions of PostgreSQL and highlights mispredictions, such as an incorrect number of returned tuples. So, the tool can be used by anyone who wants to tune cost models or integrate their own cost models into PostgreSQL. The new server mode allows you to connect to real PostgreSQL installations running on another system. Therefore, queries on large (custom) data sets can be analyzed, as well as custom PostgreSQL extensions that are not yet available as a WebAssembly build and integrated into PGlite.

I was actively involved in web development roughly 20–25 years ago, when CGI, Perl, and early versions of PHP were popular. I have no idea how modern web development actually works. I always had some projects in mind that I wanted to create, but I never had the time to dig into one of the modern JavaScript frameworks like React. GitHub Copilot now seems like a way to create (web) applications just by describing the requirements (i.e., vibe coding) for an entire application.

This post describes my experience building a PostgreSQL query plan explorer using React and VS Code in two evenings—without writing a single line of code myself.

When I started with web development, building web applications was quite simple. You had HTML, the first version of CSS, some software, and the Common Gateway Interface (CGI) standard. To build a dynamic website, you just wrote plain HTML to stdout, like:

#!/usr/bin/perl

print "Content-type: text/html\n\n";
print "<html><head></head><body>Hello World</body></html>\n";

Since that time, the internet has evolved. CSS became increasingly popular, and I switched to PHP for web development. JavaScript, AJAX, and further technologies have changed the way websites work. In my career, I developed mostly backend components and switched to database internals after I finished university. So, I lost track of how modern web applications work.

In 2020, I worked in a management role and led a team of web developers. I learned a bit about recent technologies like React, GraphQL, and Node.js, but never wrote code myself. However, since then, I have wanted to build a modern web application to understand how this works today. But I never had enough time to look deeper into any of these technologies.

The moment the agent mode for GitHub Copilot was released, it became clear to me that I wanted to challenge myself and see if I could build a modern web application without writing a single line of code—just by knowing the requirements, knowing which technologies can be used, and relying on 25-year-old knowledge about web development.

The Project

Since I spend most of my day with database internals, it was clear that I wanted to build something DBMS-related. A few weeks ago, a friend of mine drew my attention to Picasso.

The Picasso database query optimizer visualizer, Jayant R. Haritsa, Proceedings of the VLDB Endowment, Volume 3, Issue 1-2, September 2010

Picasso Database Query Optimizer Visualizer

The Picasso database query optimizer visualizer creates a multi-dimensional search space and executes a database query with different parameters from this search space to determine the query plan used by the database system. The resulting query plans (e.g., an index scan and a full table scan) are fingerprinted and plotted in an output graph. Similar plans are shown with the same color. Thus, the tool visualizes the query plans used and their distribution, generating images from that information.

Plan Explorer

The idea was to build a tool similar to Picasso using React as a static website. The required PostgreSQL installation to execute SQL queries and run the query optimizer can be embedded directly into the browser using PGlite to get a standalone app without any need for a database server.

PGlite is a WASM (WebAssembly – bytecode that is directly loaded and executed in the browser) build of PostgreSQL that can be loaded and executed in any WASM-capable browser (which is supported by most browsers these days).

Query Plans in PostgreSQL

In PostgreSQL, you can get the query plan used by prefixing a query with EXPLAIN. If the keyword JSON is added as an option, the returned query plan is in JSON format, which is useful for processing the query plans with JavaScript.

For example, if you have the following table structure:

CREATE TABLE data(key integer, value text);
INSERT INTO data (key, value) SELECT i, i::text FROM generate_series(1, 100000) i;
CREATE INDEX ON data(key);

And you perform a SELECT statement on the table with the following WHERE clause key > ...

SELECT * FROM data WHERE key > ...;

PostgreSQL could use:

• An index scan to get the tuples that qualify.
• Or a full table scan and apply the filter on each of the scanned tuples.

The query optimizer determines the fastest query plan. When the query returns many tuples, the index scan is very costly. The index structure has to be traversed, and the referenced tuples have to be read from the table. Therefore, a lot of random I/O is performed, and traversing the index also adds overhead. If the query returns a large fraction of the table, it could be faster to perform a full table scan and apply a filter condition to each of the tuples.

In contrast, if only a small fraction of the table is returned by the query (i.e., the selectivity of the predicate in the WHERE clause is low), the overhead added by the index scan is less than applying the filter condition to all tuples. So, it is beneficial to use the index.

However, there is no easy way to determine when this change between the two query plans will happen. This tool will run the query with different WHERE clauses to answer this question. The example section shows how this decision will look for a concrete query.

Tool Features

So, the main tasks of the tool are:

• Iterating over a one- or two-dimensional search space.
• Letting PGlite generate the query plan for each parameter combination of the search space.
• Fingerprinting the returned query plans and finding similar query plans (i.e., plans with the same structure).
• Generating a clear visualization from the gathered data.

Using GitHub Copilot running in agent mode and GPT 4.1, I was able to build the desired tool in two evenings (roughly 2 × 2.5 hours) and a few small corrections in the days afterwards. I described the requirements (e.g., “the user should be able to determine a one or two-dimensional search space”, “dimension 1 of the search space should be optional”, “the tool should have a modern UI”).

The tool looks like this:

In the upper part of the tool, up to two dimensions, ranges, and steps can be defined. These describe the one- or two-dimensional search space that should be iterated by the tool. Afterwards, preparation steps to set up the database can be defined (e.g., creating tables and filling data). Then, the actual query with placeholders (%%DIMENSION0%% for the value of the first dimension and %%DIMENSION1%% for the value of the second dimension) can be defined.

A build of the tool can be found at https://jnidzwetzki.github.io/planexplorer/ if you want to try it yourself.

Example Query Plans

The tool provides useful insights into the decisions made by the query planner. In this section, the tool output for three different queries is discussed.

Select Using a Table Scan or an Index

In the query plan section, the case that PostgreSQL can pick an index scan if it is more efficient than a full table scan was already discussed. The following image shows the output of the tool, when dimension 0 (the first dimension) from 0 to 50000 in steps of 10000 is iterated and the query:

SELECT * FROM data WHERE key > ...;

was executed.

It can be seen in the produced image that after the value of 40000, the query plan changes (light blue vs. dark blue). This is also the expected behavior. When fewer tuples are returned by the query, it is beneficial to use the index.

Below is the visualization of the query plan, the actually used query plans are shown. The first one is the actual full table scan:

[
 {
    "Plan": {
      "Node Type": "Seq Scan",
      "Parallel Aware": false,
      "Async Capable": false,
      "Relation Name": "data",
      "Alias": "data",
      "Startup Cost": 0,
      "Total Cost": 1790,
      "Plan Rows": 100000,
      "Plan Width": 9,
      "Filter": "(key > 0)"
 }
 }
]

The second query plan is the index scan using the index on the key attribute.

[
 {
    "Plan": {
      "Node Type": "Index Scan",
      "Parallel Aware": false,
      "Async Capable": false,
      "Scan Direction": "Forward",
      "Index Name": "data_key_idx",
      "Relation Name": "data",
      "Alias": "data",
      "Startup Cost": 0.29,
      "Total Cost": 1769.94,
      "Plan Rows": 59896,
      "Plan Width": 9,
      "Index Cond": "(key > 40000)"
 }
 }
]

Changing the Random Page Costs

The next example shows when PostgreSQL changes from a table scan to an index scan as the selectivity of a predicate changes. A one-dimensional search space was used to execute the query.

However, the exact point at which PostgreSQL switches from one plan to another also depends on the set costs for page access. For instance, the setting random_page_cost has a default value of 4 and describes the costs that occur when a page is accessed in random order (in contrast to seq_page_cost with a default value of 1 when a page is accessed sequentially). In this experiment, the same query is performed, but a two-dimensional search space is used. The value for random_page_cost is changed from 0 to 8 in steps of 0.25. The result can be seen in the following image:

Light blue is again the query plan that uses the sequential (full table) scan and dark blue is the query plan that uses the index scan.

It can be seen that PostgreSQL uses the index scan much earlier when the random_page_cost is low (i.e., the penalty of following the pointers in the index and accessing pages in random order is lower). In contrast, when the random_page_cost is high, PostgreSQL starts to use the index scan much later.

Performing a Self-Join

The last example shows a more complex scenario with more than two different query plans. The example query now performs a self-join and has the same WHERE clause as the previous examples:

SET random_page_cost = %%DIMENSION1%%;
SELECT * FROM data d1 LEFT JOIN data d2 ON (d1.key = d2.key) WHERE d1.key > %%DIMENSION0%%;

Again, dimension 0 changes the selectivity of the WHERE clause and dimension 1 changes the random_page_cost.

The generated image shows that PostgreSQL now uses four different query plans to execute the query. Even in this new query, PostgreSQL can decide whether to use the index or not. Furthermore, the join order can be changed and further optimizations can be applied (however, the details will not be covered in this blog post).

Conclusion

I was able to build a modern web application that uses an in-browser version of PostgreSQL to visualize the query plans used for particular queries in just a few hours, despite having only minimal skills in modern web development. GitHub Copilot with GPT 4.1 did a very good job, and vibe coding really seems to be a viable approach for building simple web apps.

I definitely learned less than I would have by building this tool in plain React and reading all the needed tutorials. But I spent just a couple of hours on the problem and have a usable tool. Otherwise, this would have been a multi-week project and I would never have taken up this development.

The created tool is available at https://jnidzwetzki.github.io/planexplorer/ and could be used by the database (research) community to explore the generated query plans by PostgreSQL. It might also be a valuable tool for PostgreSQL extension developers who create their own operators and cost models and want to understand when a particular query plan is chosen by the query optimizer.

This blog post discusses some implementation details of snapshots.

Tuple Visibility

The following table is used in this article to illustrate how snapshots work in PostgreSQL.

CREATE TABLE temperature (
  time timestamptz NOT NULL,
  value float
);

Let’s insert the first record into this table. This is done by creating a new transaction, checking the current transaction ID (if assigned), inserting a new tuple, checking the transaction ID again, and committing the transaction.

BEGIN;

SELECT * FROM txid_current_if_assigned();
 txid_current_if_assigned
--------------------------

(1 row)

INSERT INTO temperature VALUES(now(), 4);

SELECT * FROM txid_current_if_assigned();
 txid_current_if_assigned
--------------------------
                  5062286
(1 row)

COMMIT;

An important observation in this example is that PostgreSQL only assigns a transaction ID to a transaction when data is modified. This optimization prevents unnecessary work and avoids transaction ID exhaustion. Even though the transaction ID is a 32-bit integer, it can eventually be exhausted. PostgreSQL handles this overflow by freezing tuples to manage transaction ID wraparounds properly.

The system attributes xmin and xmax determine the first and last transactions that can see a particular tuple. Additionally, the ctid attribute indicates the tuple’s position on the corresponding page. These attributes are displayed when explicitly mentioned in a SELECT statement:

SELECT xmin, xmax, ctid, * FROM temperature;
  xmin   | xmax |  ctid |             time              | value
---------+------+-------+-------------------------------+-------
 5062286 |    0 | (0,1) | 2024-04-02 22:06:03.035868+02 |     4
(1 row)

The output indicates that all transactions with a transaction ID >= 5062286 can see this tuple. When the tuple is deleted, the xmax value is updated with the transaction ID of the deleting transaction. The ctid value (0,1) means the tuple is the first on page 0. Now, let’s delete the tuple:

BEGIN;

DELETE FROM temperature;

SELECT * FROM txid_current_if_assigned();
 txid_current_if_assigned
--------------------------
                  5062291
(1 row)

COMMIT;

However, when a SELECT statement is executed, no rows are returned, even though the tuple has xmin and xmax values.

SELECT xmin, xmax, ctid, * FROM temperature;
 xmin | xmax | ctid | time | value
------+------+------+------+-------
(0 rows)

This behavior is due to the internal scanner. If a tuple is not visible in the current transaction snapshot, it is skipped. To retrieve these values, we need to use lower-level tools instead of a simple SELECT.

The pageinspect extension for PostgreSQL allows us to examine all tuples stored on a page and decode their internal flags and attributes. After loading the extension, we can inspect the pages of a relation.

-- Load the extension
CREATE EXTENSION pageinspect;

-- Get the tuples of the first page of the relation 'temperature'
SELECT lp, t_xmin, t_xmax FROM heap_page_items(get_raw_page('temperature', 0));

 lp | t_xmin  | t_xmax
----+---------+---------
  1 | 5062286 | 5062291

The output shows that the first tuple on page 0 (with ctid (0,1)) has a t_xmax value of 5062291, which matches the transaction ID that deleted the tuple. Thus, any transaction with a transaction ID greater than 5062291 will not see this tuple.

Snapshots

When PostgreSQL scans a table, a snapshot must be specified. The table_beginscan function takes the snapshot data as its second parameter:

static inline TableScanDesc table_beginscan(Relation rel,
    Snapshot snapshot, int nkeys, struct ScanKeyData *key)

Internal Data Structures

Typically, the transaction snapshot is used as a parameter for this function. The structure SnapshotData contains all the information that is part of a snapshot. In this blog post, we focus on the following attributes:

typedef struct SnapshotData
{
  [...]
	/*
	 * An MVCC snapshot can never see the effects of XIDs >= xmax. It can see
	 * the effects of all older XIDs except those listed in the snapshot. xmin
	 * is stored as an optimization to avoid needing to search the XID arrays
	 * for most tuples.
	 */
	TransactionId xmin;			/* all XID < xmin are visible to me */
	TransactionId xmax;			/* all XID >= xmax are invisible to me */

	/*
	 * For normal MVCC snapshot this contains the all xact IDs that are in
	 * progress, unless the snapshot was taken during recovery in which case
	 * it's empty. For historic MVCC snapshots, the meaning is inverted, i.e.
	 * it contains *committed* transactions between xmin and xmax.
	 *
	 * note: all ids in xip[] satisfy xmin <= xip[i] < xmax
	 */
	TransactionId *xip;
	uint32		xcnt;			/* # of xact ids in xip[] */
  [...]
}

The xmin field defines the oldest active transaction in the system. All transactions with a transaction ID lower than this value have already been committed. Thus, all tuples with a lower transaction ID should be visible in this snapshot. The xmax field contains the most recent transaction ID known to the snapshot. All tuples with a transaction ID greater than xmax are invisible in the current snapshot.

Why are the xip and xcnt fields needed? For transaction IDs between xmin and xmax, it must be determined whether the transaction was committed or in progress when the snapshot was created.

A DBMS processes queries from multiple users, who can start transactions at any time. The start and commit times of these transactions are not ordered. This means there might be transactions with a transaction ID larger than xmin that were already committed when the snapshot was created. However, some other transactions in the range [xmin, xmax] might still be uncommitted. Since the data of committed and uncommitted transactions must be handled properly, an array of transaction IDs xip of length xcnt is defined. It contains all transactions larger than xmin and lower than xmax that were in progress when the snapshot was taken.

Example

To illustrate this behavior, let’s perform a practical example using three transactions.

Transaction 1

BEGIN;

INSERT INTO temperature VALUES(now(), 5);

SELECT * FROM txid_current_if_assigned();
 txid_current_if_assigned
--------------------------
                  5062310
(1 row)

The first transaction inserts new data into the temperature table but remains uncommitted. The transaction has a transaction ID of 5062310.

Transaction 2

BEGIN;

INSERT INTO temperature VALUES(now(), 5);

SELECT * FROM txid_current_if_assigned();
 txid_current_if_assigned
--------------------------
                  5062311
(1 row)

The second transaction also inserts data into the same table but remains uncommitted. The transaction ID is 5062311.

Transaction 3

SELECT * FROM pg_current_snapshot();
 pg_current_snapshot
---------------------
 5062310:5062310:
(1 row)

The third transaction uses the pg_current_snapshot function to get the current snapshot. The output indicates that all changes by transactions with an ID lower than 5062310 are visible. Changes equal to or larger than transaction ID 5062310 are not visible, and no uncommitted transactions exist at this point.

So, what happened to the still-pending transactions 5062310 and 5062311? Since no further transactions have been committed in this demo system, PostgreSQL has not changed the current transaction ID. However, this can be changed:

SELECT * FROM pg_current_xact_id_if_assigned();
 pg_current_xact_id_if_assigned
--------------------------------

(1 row)

SELECT * FROM pg_current_xact_id();
 pg_current_xact_id
--------------------
            5062312
(1 row)

SELECT * FROM pg_current_snapshot();
       pg_current_snapshot
---------------------------------
 5062310:5062313:5062310,5062311
(1 row)

Unlike the pg_current_xact_id_if_assigned function, the pg_current_xact_id function forces the assignment of a transaction ID to the current transaction. In this case, the transaction ID is 5062312. Using this transaction ID also updates the snapshot.

The first value remains the same: all tuples modified by transactions with an ID lower than 5062310 are visible in the current snapshot. However, the upper limit (xmax) has changed. Now, all changes equal to or larger than 5062313 are not visible in the current snapshot. Since our transaction ID is 5062312, it makes sense that these changes should not be visible. What about the new part 5062310,5062311? This is the xip part of the snapshot, indicating that transactions 5062310 and 5062311 were uncommitted when the snapshot was taken. Therefore, these changes should also not be visible in the current snapshot. As soon as one of these transactions commits and we take a new snapshot, the transaction ID is removed from xip, and the changes become visible in the current snapshot.

Exporting Snapshots

Another interesting feature of PostgreSQL is the ability to export snapshots and load them in other sessions. A snapshot can be exported by calling the pg_export_snapshot function. The function returns the snapshot ID and creates a corresponding file in the pg_snapshots folder of the data directory.

BEGIN;

SELECT * FROM pg_export_snapshot();
 pg_export_snapshot
---------------------
 0000000C-000005F6-1
(1 row)

This file contains the same information as returned by pg_current_snapshot, which we discussed earlier. Additionally, it includes further information about the isolation level and the database ID.

$ cat ~/postgresql-sandbox/data/REL_15_1_DEBUG/pg_snapshots/0000000C-000005F6-1
vxid:12/1526
pid:1362769
dbid:706615
iso:1
ro:0
xmin:5062310
xmax:5062313
xcnt:2
xip:5062310
xip:5062311
sof:0
sxcnt:0
rec:0

This exported snapshot can be loaded into another transaction by calling SET TRANSACTION SNAPSHOT '0000000C-000005F6-1' to run with the same snapshot as the transaction that created it.

Snapshots and Transaction Isolation Level

Depending on the isolation level, the snapshot is taken when the transaction starts (Repeatable Read) or for every statement in the transaction (Read Committed). When a new snapshot is created for each statement inside a transaction, committed data from other transactions becomes visible in the current transaction. If only one snapshot is created for the entire transaction, the xmax value remains constant, no new data from transactions with a higher ID becomes visible, and reads are repeatable.

Summary

This blog post discusses the basics of multi-version concurrency control in PostgreSQL. It then introduces snapshots and explains how they control the visibility of tuples. The integration with the table scan API is also discussed.

Heavyweight locks are used to control access to tables. Lightweight locks (LWLocks) manage access to data structures, such as adding data to the write-ahead log (WAL). Row-level locks control access to individual tuples. For example, tuples need to be locked when executing an SQL statement like SELECT * FROM table WHERE i > 10 FOR UPDATE;. The tuples returned by the query are internally locked with an exclusive lock (LOCK_TUPLE_EXCLUSIVE). Another transaction attempting to lock the same tuples must wait until the first transaction releases the locks.

In this article, we discuss the tool pg_row_lock_tracer, which uses eBPF and UProbes to trace PostgreSQL’s row-locking behavior. The tool can be downloaded from the pg-lock-tracer project website.

This is the third article in a series about tracing PostgreSQL locks. The first article covers the tracing of heavyweight locks, and the second article focuses on LW locks.

Background

PostgreSQL implements four different row lock modes. These can be requested by adding FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, or FOR KEY SHARE to a SELECT statement. Additionally, operations like updates automatically acquire these locks before modifying a tuple. For example, when a transaction successfully performs a FOR UPDATE lock on a tuple, an update operation by another parallel transaction is blocked until the first transaction releases the lock. Row locks can be requested by calling the function heapam_tuple_lock.

Lock Types

Internally, these locks are called LockTupleKeyShare, LockTupleShare, LockTupleNoKeyExclusive, and LockTupleExclusive. They are defined in the enum LockTupleMode. These locks have varying strengths, and some are compatible (i.e., multiple transactions can hold locks simultaneously on the same row), while others are conflicting (i.e., only one lock can be held at a time, and a conflicting lock request must wait).

Lock Behavior

Users can specify various lock behaviors in addition to different lock modes. For instance, if a tuple is already locked and a second transaction requests a conflicting lock, the user can choose to skip the lock instead of waiting. The possible behaviors are defined in the enum LockWaitPolicy.

For example, the following SQL query acquires a LockTupleExclusive row lock if it does not conflict with existing locks. Any already locked tuples are skipped by the current transaction:

SELECT * FROM table WHERE i > 10 FOR UPDATE SKIP LOCKED;

A transaction that successfully acquires these locks can assume that no other transaction will modify the tuples in parallel. The returned values from the SELECT statement can then be processed, modified, and updated in subsequent UPDATE statements before being committed.

Lock Results

The possible outcomes of a lock operation are defined in the enum TM_Result. A lock can be granted (TM_Ok), or it may fail for various reasons: the tuple is invisible to the current snapshot (TM_Invisible), already modified by the same backend process (TM_SelfModified), updated (TM_Updated), or deleted (TM_Deleted). Additionally, if the lock is instructed not to wait, it may return TM_BeingModified if another transaction is currently modifying the tuple, or TM_WouldBlock if the lock would otherwise block.

pg_row_lock_tracer

pg_row_lock_tracer enables real-time tracing of PostgreSQL row-level locks using eBPF and UProbes. It also provides statistics about requested locks and their outcomes.

Download and Usage

The lock tracer can be installed via the Python package installer pip:

pip install pg-lock-tracer

Once installed, the locks of one or more running processes can be traced:

# Trace the row locks of the given PostgreSQL binary
pg_row_lock_tracer -x /home/jan/postgresql-sandbox/bin/REL_14_9_DEBUG/bin/postgres

# Trace the row locks of PID 1234
pg_row_lock_tracer -p 1234 -x /home/jan/postgresql-sandbox/bin/REL_14_9_DEBUG/bin/postgres

# Trace the row locks of PIDs 1234 and 5678
pg_row_lock_tracer -p 1234 -p 5678 -x /home/jan/postgresql-sandbox/bin/REL_14_9_DEBUG/bin/postgres

# Trace the row locks of PID 1234 with verbose output
pg_row_lock_tracer -p 1234 -x /home/jan/postgresql-sandbox/bin/REL_14_9_DEBUG/bin/postgres -v

# Trace the row locks and display statistics
pg_row_lock_tracer -x /home/jan/postgresql-sandbox/bin/REL_14_9_DEBUG/bin/postgres --statistics

A sample output of the tool looks as follows:

[...]
2783502701862408 [Pid 2604491] LOCK_TUPLE_END TM_OK in 13100 ns
2783502701877081 [Pid 2604491] LOCK_TUPLE (Tablespace 1663 database 305234 relation 313419) - (Block and offset 7 143) - LOCK_TUPLE_EXCLUSIVE LOCK_WAIT_BLOCK
2783502701972367 [Pid 2604491] LOCK_TUPLE_END TM_OK in 95286 ns
2783502701988387 [Pid 2604491] LOCK_TUPLE (Tablespace 1663 database 305234 relation 313419) - (Block and offset 7 144) - LOCK_TUPLE_EXCLUSIVE LOCK_WAIT_BLOCK
2783502702001690 [Pid 2604491] LOCK_TUPLE_END TM_OK in 13303 ns
2783502702016387 [Pid 2604491] LOCK_TUPLE (Tablespace 1663 database 305234 relation 313419) - (Block and offset 7 145) - LOCK_TUPLE_EXCLUSIVE LOCK_WAIT_BLOCK
2783502702029375 [Pid 2604491] LOCK_TUPLE_END TM_OK in 12988 ns

The tool’s output shows the tuples being locked, the type of locks used, and additional options such as LOCK_WAIT_BLOCK. It also includes the result of the lock operation (TM_OK).

When the --statistics option is used, the tool collects and displays statistics about the traced locks upon termination (e.g., after pressing CTRL+C):

Lock statistics:
================

Used wait policies:
+---------+-----------------+----------------+-----------------+
|   PID   | LOCK_WAIT_BLOCK | LOCK_WAIT_SKIP | LOCK_WAIT_ERROR |
+---------+-----------------+----------------+-----------------+
| 2604491 |       1440      |       0        |        0        |
+---------+-----------------+----------------+-----------------+

Lock modes:
+---------+---------------------+------------------+---------------------------+----------------------+
|   PID   | LOCK_TUPLE_KEYSHARE | LOCK_TUPLE_SHARE | LOCK_TUPLE_NOKEYEXCLUSIVE | LOCK_TUPLE_EXCLUSIVE |
+---------+---------------------+------------------+---------------------------+----------------------+
| 2604491 |          0          |        0         |             0             |         1440         |
+---------+---------------------+------------------+---------------------------+----------------------+

Lock results:
+---------+-------+--------------+-----------------+------------+------------+------------------+---------------+
|   PID   | TM_OK | TM_INVISIBLE | TM_SELFMODIFIED | TM_UPDATED | TM_DELETED | TM_BEINGMODIFIED | TM_WOULDBLOCK |
+---------+-------+--------------+-----------------+------------+------------+------------------+---------------+
| 2604491 |  1440 |      0       |        0        |     0      |     0      |        0         |       0       |
+---------+-------+--------------+-----------------+------------+------------+------------------+---------------+

Summary

pg_row_lock_tracer is a tool for tracing PostgreSQL row-level locks. It is available for download on GitHub. Using eBPF and UProbes, it enables real-time tracing of row lock activity. Like its related tools (pg_lock_tracer and pg_lw_lock_tracer), it is designed for debugging and analyzing lock behavior and performance issues.

This is the third article in a series about tracing PostgreSQL locks. The first part discusses a lock tracer for heavyweight locks, while the second part focuses on tracing LW locks.

The Elixir Cross Referencer is a source code indexer that provides a web interface and an API for quickly looking up symbols. I had used it several times while navigating the Linux source code and wondered what it would take to set up a local installation for PostgreSQL.

To my surprise, it was easier than expected. Elixir can be installed using Docker, and custom images for new projects can be created effortlessly. For instance, to create a new Docker image containing a copy of the PostgreSQL source code, the following commands need to be executed:

$ git clone https://github.com/bootlin/elixir.git

$ cd elixir

$ docker build -t elixir:postgresql-11-01-2024 --build-arg GIT_REPO_URL=https://github.com/postgres/postgres.git --build-arg PROJECT=postgresql . -f docker/debian/Dockerfile

The last command builds a new Docker image called elixir:postgresql-11-01-2024. This process takes some time to complete. The two build-arg parameters are sufficient to clone and index the PostgreSQL repository. Once the image is created, it should appear as an available image in the local Docker installation.

$ docker images

REPOSITORY                                                       TAG                     IMAGE ID       CREATED        SIZE
elixir                                                           postgresql-11-01-2024   fb993f66c1cc   2 hours ago    2.38GB

Next, a new container can be started using the image. I use the parameter -p 8081:80 to map port 80 of the container to port 8081 on my local system.

$ docker run elixir:postgresql-11-01-2024 -d -p 8081:80

Once the container is running, the PostgreSQL source code can be accessed by opening the URL http://172.17.0.2:8081/postgresql/latest/source.

If you want to customize the header of the Elixir installation, you can modify the file templates/header.html before building the Docker image. More information about customizing the image can be found in the project’s documentation.

⚠️ An updated and slightly revised version of this post is available in the Timescale company blog.

Used Environment

PostgreSQL is a database management system that uses vacuum operations to reclaim space from dead (e.g., updated or deleted) tuples. In this post, we will trace the vacuum calls and determine the needed time for the vacuum operations per table.

In the following examples, a PostgreSQL 14 server is used. The PostgreSQL binary is located at /home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres. In addition, the examples are executed in a database with these two tables:

CREATE TABLE testtable1 (
   id int NOT NULL,
   value int NOT NULL
);

CREATE TABLE testtable2 (
   id int NOT NULL,
   value int NOT NULL
);

Note: Depending on the used C compiler and applied optimizations, the symbols of internal (i.e., as static declared) functions could not be visible. In this case, uprobes can not be used to trace the function invocations. To address this issue, there are two possible solutions: (1) remove the static modifier from the function declaration and recompile PostgreSQL, or (2) create a full debug build of PostgreSQL.

Using funclatency-bpfcc to Trace Function Calls

Let’s explore the solutions that already exist before developing our tool to trace the vacuum operations. The tool funclatency-bpfcc is available for most Linux distributions (on Debian, it is contained in the package bpfcc-tools) and allows it to trace a function enter and exit and measure the function latency (i.e., the time the function needs to complete).

In PostgreSQL, the function vacuum_rel is invoked when a vacuum operation on a relation is performed. To trace these function calls with funclatency-bpfcc, the path of the PostgreSQL binary and the function name have to be provided. For instance:

$ sudo funclatency-bpfcc -r /home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel

Tracing 1 functions for "/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel"... Hit Ctrl-C to end.

Afterward, a eBPF program is loaded into the Linux kernel, a uprobe is defined on the function enter and one uprobe is defined on the function exit. The latency between these two events is measured and stored.

To execute some vacuum operations, we perform the following SQL statement in a second session:

database=# VACUUM FULL;
VACUUM FULL

This SQL statement triggers PostgreSQL to perform a vacuum operation of all tables of the currently open database. After the vacuum operations are done, the funclatency-bpfcc program can be stopped (by executing CTRL+C). This ends the observation of the binary and shows the recorded execution times on the terminal.

$ sudo funclatency-bpfcc -r /home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
[...]
^C
Function = b'vacuum_rel' [876997]
     nsecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 0        |                                        |
        64 -> 127        : 0        |                                        |
       128 -> 255        : 0        |                                        |
       256 -> 511        : 0        |                                        |
       512 -> 1023       : 0        |                                        |
      1024 -> 2047       : 0        |                                        |
      2048 -> 4095       : 0        |                                        |
      4096 -> 8191       : 0        |                                        |
      8192 -> 16383      : 0        |                                        |
     16384 -> 32767      : 0        |                                        |
     32768 -> 65535      : 0        |                                        |
     65536 -> 131071     : 0        |                                        |
    131072 -> 262143     : 0        |                                        |
    262144 -> 524287     : 0        |                                        |
    524288 -> 1048575    : 0        |                                        |
   1048576 -> 2097151    : 0        |                                        |
   2097152 -> 4194303    : 0        |                                        |
   4194304 -> 8388607    : 2        |*                                       |
   8388608 -> 16777215   : 13       |***********                             |
  16777216 -> 33554431   : 44       |****************************************|
  33554432 -> 67108863   : 7        |******                                  |
  67108864 -> 134217727  : 1        |                                        |

avg = 22765358 nsecs, total: 1525279002 nsecs, count: 67

Detaching...

The output contains the information that the function vacuum_rel was called 67 times and the average function time is 22765358 nsecs. In addition, a histogram of the function latency is printed. This gives a lot of helpful information, but it might be helpful to get the information which vacuum calls for which relation needs how much time. This is something that is not supported by this tool because it does not evaluate the parameters of the function (e.g., the OID of relation that the current function invocation should vacuum). However, this is something that we can do with bpftrace.

Tracing Function Entries

Let’s start with a very simple bpftrace program that prints a line once the vacuum_rel function is invoked in the PostgreSQL binary. bpftrace is called with the eBPF program that should be loaded into the Linux kernel. The eBPF programs that are passed to bpftrace have the following syntax:

<probe1> {
        <Actions>
}

[...]

<probeN> {
        <Actions>
}

The syntax to define a uprobe on a userland binary is: uprobe:library_name:function_name[+offset]. For instance, to define an uprobe on the function invocation of vacuum_rel in the binary /home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres and print the line Vacuum started, the following bpftrace call can be used:

$ sudo bpftrace -e '
uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel {
    printf("Vacuum started\n");
}
'

Attaching 1 probe...
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
[...]

As soon as the VACUUM FULL SQL statement in PostgreSQL is executed in another terminal session, the program starts to print the message on the screen. This is a good start, but we still have less information available than output by the existing tool funclatency-bpfcc. The latency of the function calls is missing.

Tracing Function Returns / Latency

To measure the latency of the function invocations, we need two things:

• We need to define a second probe that is invoked when the function observed returns; this can be done by a uretproble.
• The time between the function invocation and the return has to be measured.

A uretproble in bpftrace can be defined using the same syntax (uretprobe:binary:function) as the already defined uprobe. In addition, bpftrace allows it to create variables like associative arrays. We use such an array to capture the start time of a function invocation @start[tid] = nsecs;. The key of the array is the id of the current thread tid. So, multiple threads (and processes like in our case with PostgreSQL) can be traced simultaneously without overriding the last function invitation start time.

In the uretprobe we take the current time and subtract the time of the function invocation (nsecs - @start[tid]) to get the time the function call needs. In addition, we use a function predicate (/@start[tid]/) to let bpftrace know that we only want to execute the function body of the uretprobe as soon as this array value is defined. Using this predicate, we prevent handling a function return without seeing the function enter before (e.g., we start the bpftrace program in the middle of a running function call, and we get only the uretprobe invocation for this function call).

Note: Is it not guaranteed that the eBPF events are delivered and processed in-order by bpftrace. Especially when a function call is short and we have a lot of function invocations, the events could be processed out-of-order (e.g., we see two function enter events followed by two function return events). In this case, function latency observations with bpftrace become imprecise. To avoid this, we use VACUUM FULL calls instead of vacuum calls. These calls are much more expensive since they rewrite the table. Therefore, they take longer and can be reliably observed by bpftrace.

$ sudo bpftrace -e '
uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        printf("Performing vacuum\n");
        @start[tid] = nsecs;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{
        printf("Vacuum call took %d ns\n", nsecs - @start[tid]);
        delete(@start[tid]);
}
'

After running this bpftrace call and executing VACUUM FULL in a second session, we see the following output:

Attaching 2 probes...
Performing vacuum
Vacuum call took 37486735 ns
Performing vacuum
Vacuum call took 16491130 ns
Performing vacuum
Vacuum call took 32443568 ns
Performing vacuum
Vacuum call took 17959933 ns
[...]

For each call of the vacuum_rel in PostgreSQL, we measure the time the vacuum operation needs. However, it would be convenient if we could also trace the OID or the name of the relation that is vacuumed by the current vacuum operation. This requires the handling of the function parameters of the observed function.

Handle Function Parameters

The function vacuum_rel has the following signature in PostgreSQL 14. The first parameter is the Oid (an unsigned int) of the processed relation. The second parameter is a RageVar struct, which could contain the name of the relation. The third parameter is a VacuumParams struct, which contains additional parameters for the vacuum operation and the last parameter is a BufferAccessStrategy, which defines the access strategy of the used buffer.

static bool vacuum_rel(Oid relid,
        RangeVar *relation,
        VacuumParams *params,
        BufferAccessStrategy bstrategy 
)

Bpftrace allows it to access the function parameter using the keywords arg0, arg1, …, argN. To include the Oid in the output our logging, we need only to print the first parameter of the function.

$ sudo bpftrace -e '

uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        printf("Performing vacuum of OID %d\n", arg0);
        @start[tid] = nsecs;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{
        printf("Vacuum call took %d ns\n", nsecs - @start[tid]);
        delete(@start[tid]);
}
'

When the VACUUM FULL operation is executed again in a second terminal, the output looks as follows:

Attaching 2 probes...
[...]
Performing vacuum of OID 1153888
Vacuum call took 37486734 ns
Performing vacuum of OID 1153891
Vacuum call took 49535256 ns
Performing vacuum of OID 2619
Vacuum call took 39575635 ns
Performing vacuum of OID 2840
Vacuum call took 40683526 ns
Performing vacuum of OID 1247
Vacuum call took 14683600 ns
Performing vacuum of OID 4171
Vacuum call took 20587503 ns

To determine which Oid belongs to which relation, the following SQL statement can be executed:

blog=# SELECT oid, relname FROM pg_class WHERE oid IN (1153888, 1153891);
   oid   |  relname   
---------+------------
 1153888 | testtable1
 1153891 | testtable2
(2 rows)

The result shows that the Oids 1153888 and 1153891 belong to the tables testtable1 and testtable2, which we have created in one of the first sections of this article. These values belong to our test environment. In your environment, different Oids might be shown.

Handle Function Struct Parameters

So far, we have processed simple parameters with bpftrace (like Oids, which are unsigned integers). However, many parameters in PostgreSQL are structs. Furthermore, these structs can be handled in bpftrace programs as well.

The second parameter of the vacuum_rel function is a RangeVar struct. This struct is defined in PostgreSQL 14 as follows:

typedef struct RangeVar
{
	NodeTag	type;
	char *catalogname;
	char *schemaname;
	char *relname;
	[...]
}

To process the struct, the following bpftrace program can be used. Please note, that the internal NodeTag data type of PostgreSQL is replaced by a simple int. The NodeTag data type is an enum. Enums are backed by the integer data type in C. To handle this enum correctly, we could (1) also copy the enum definition into the eBPF program, or (2) we could replace it with a data type of the same length. To keep the bpftrace program simple, the second option is used here. The next three struct members are char pointer which contains the catalogname, the schema, and the name of the relation. The schemaname and the relname are the fields we are interested in. The struct contains more members, but these members are ignored to keep the example clear.

$ sudo bpftrace -e '
struct RangeVar
{
	int type;
	char *catalogname;
	char *schemaname;
	char *relname;
};

uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        printf("[PID %d] Performing vacuum of OID %d (%s.%s)\n", pid, arg0, str(((struct RangeVar*) arg1)->schemaname), str(((struct RangeVar*) arg1)->relname));
        @start[tid] = nsecs;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{
        printf("[PID %d] Vacuum call took %d ns\n", pid, nsecs - @start[tid]);
        delete(@start[tid]);
}
'

After the struct is defined, the members of the struct can be accessed as in a regular C program. For example: ((struct RangeVar*) arg1)->schemaname. In addition, we also print the process id (PID) of the program that has triggered the uprobe. This allows it to identify the process that has performed the vacuum operation.

When running the following SQL statements in a second terminal:

VACUUM FULL public.testtable1;
VACUUM FULL public.testtable2;

The bpftrace program shows the following output:

Attaching 2 probes...
[PID 616516] Performing vacuum of OID 1153888 (public.testtable1)
[PID 616516] Vacuum call took 23683600 ns
[PID 616516] Performing vacuum of OID 1153891 (public.testtable2)
[PID 616516] Vacuum call took 24240837 ns

The table names are extracted from the RangeVar data structure and shown in the output. However, this data structure is not always populated by PostgreSQL. The data structure might be empty when running VACUUM FULL without specifying a table name. Therefore, we use two single invocations with explicit table names to force PostgreSQL to populate this data structure.

Optimizing the Bpftrace Program Using Maps

The bpftrace programs we have developed so far use one or more printf statements directly. A printf call is slow and reduces the throughput the bpftrace program can monitor.

This can be optimized by storing the data in a map that is printed when bpftrace is stopped. To do this, we introduce three new maps @start, @oid, and @vacuum. The first two maps are populated in the uprobe event of the vacuum_rel function. The map @start contains the time when the probe is triggered, and the map @oid contains the oid of the parameter function.

When the function is left and the uretprobe is activated, the @vacuum map is populated. The key is the Oid and the value are the needed time to perform the vacuum operation. In addition, the keys of the first two maps are removed.

When bpftrace exits (i.e., by pressing CRTL+C), all populated maps are printed automatically. By using these three maps, we have separated the actual monitoring from the output; the expensive printf function is called after the monitoring is done.

In addition, in the following program, we use the two functions BEGIN and END that are called by bpftrace when the observation begins and ends.

$ sudo sudo bpftrace -e '

uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        @start[tid] = nsecs;
        @oid[tid] = arg0;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{

        @vacuum[@oid[tid]] = nsecs - @start[tid];
        delete(@start[tid]);
        delete(@oid[tid]);

}

BEGIN
{
        printf("VACUUM calles are traced, press CTRL+C to stop tracing\n");
}

END 
{
        printf("\n\nNeeded time in ns to perform VACUUM FULL per Oid\n");
}
'

After bpftrace is started, the first message is printed. After the program is stopped, the second message is printed. In addition, the content of the @vacuum map is printed. For each Oid, the needed time for the vacuum operations is shown.

VACUUM calles are traced, press CTRL+C to stop tracing
^C

Needed time in ns to perform VACUUM FULL per Oid

@vacuum[1153888]: 7526823
@vacuum[1153891]: 8462672
@vacuum[2613]: 10764797
@vacuum[2995]: 11429589
@vacuum[6102]: 11436539
@vacuum[12801]: 14373934
@vacuum[6106]: 14396012
@vacuum[3118]: 14507167
@vacuum[3596]: 14695385
@vacuum[12811]: 14871237
@vacuum[3429]: 15106778
@vacuum[3350]: 15158742
@vacuum[2611]: 15432053
@vacuum[3764]: 15534169
@vacuum[2601]: 16055863
@vacuum[3602]: 16128624
@vacuum[2605]: 16405419
@vacuum[2616]: 16914195
@vacuum[3576]: 17003920
[...]

Conclusion

This article provides a brief overview of eBPF. To trace the function latency of PostgreSQL vacuum calls, we used the tool funclatency-bpfcc. Additionally, we utilized bpftrace to create a tool that allows for more in-depth observation of the calls. Our bpftrace script also takes into account the parameters of the PostgreSQL vacuum_rel function, enabling us to monitor the vacuum time per relation.

This data structure is widely used in PostgreSQL code. Internally, so-called words of bits are used and store the information on which element is part of the set. For instance, this data structure supports efficient tests if an integer is part of the set (using the bms_is_member function), to add new values (using the bms_add_member, bms_add_members, or bms_add_range functions), or to iterate over the values (using the bms_next_member and bms_prev_member functions).

Dumping the Content of the Bitmapset

However, the content of this data structure is difficult to debug. The debugger does not show the stored content due to the lack of knowledge about the semantics of the bits. A lot of internal PostgreSQL data structures can be dumped using the pprint function. Unfortunately, the pprint function is unable to print the content of the Bitmapset.

For instance, when the GDB should print the content of the set, it looks as follows:

(gdb) print *node_state->unused_batch_states
$1 = {nwords = 1, words = 0x5588689773f0}

The output indicates that one word (consisting of 32 bits) is used to represent the stored values. Unfortunately, in the output, it can not be seen which values are stored exactly.

On the PostgreSQL developer mailing list was a patch discussed to introduce a function called bmsToString. This function can also be used to display the content of a Bitmapset. However, this function can be only called when PostgreSQL is running. When a core dump of a crashed PostgreSQL process is examined with GDB, the function cannot be used.

git a(gdb) call bmsToString(chunk_state->unused_batch_states)
$6 = 0x5588689a8818 "(b 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15)"

Because the Bitmapset data structure is used heavily inside of PostgreSQL and the database server has no reliable way to print the content during debugging, I have developed a GDB extension to solve this problem. This article presents a GDB extension, which provides a remedy and makes the content displayable in the debugger.

A GDB extension to show the content of the Bitmapset

The debugger GDB can be extended using python scripts. The Pretty Printing API can be used to develop Pretty Printer to analyze data structures and to improve the output of the debugger when they are displayed.

The following python script shows such an extension. It registers a new set of pretty printers via the RegexpCollectionPrettyPrinter function. These printers are called when a Bitmapset or a Relids data type should be printed by GDB. It decodes the words of the Bitmapset into decimal values, adds these values to a list and converts this list into a string.

from gdb.printing import PrettyPrinter, register_pretty_printer
import gdb

class BitmapsetPrettyPrinter(object):
    def __init__(self, val):
        self.val = val

    def to_string(self):
        values = []
        bits_per_word = 32

        if self.val is None or self.val.type is None:
           return "0x0"

        words = None

        try:
           words = self.val["nwords"]
        except Exception:
          return 'is not iterable'

        for word_no in range(words):
           word = self.val["words"][word_no]
           for bit in range(bits_per_word):
              if word & (1 << bit):
                  values.append(word_no * bits_per_word + bit)

        return f"PGBitmapset ({str(values)})"

    def display_hint(self):
        return 'PGBitmapset'

def build_pretty_printer():
    pp = gdb.printing.RegexpCollectionPrettyPrinter("PostgreSQLPrettyPrinter")
    pp.add_printer('Bitmapset', '^Bitmapset$', BitmapsetPrettyPrinter)
    pp.add_printer('Relids', '^Relids$', BitmapsetPrettyPrinter)
    return pp

register_pretty_printer(None, build_pretty_printer(), replace=True)

Registering the Pretty Printer

This Python script can be stored in a new file and loaded via the source command into GDB.

(gdb) source /home/jan/dev/postgresql_printer.py

After the file is loaded, the two pretty printers are registered. By using the command info pretty-printer, GDB shows which pretty printers are registered. After loading the two new prints, the output looks as follows:

(gdb) info pretty-printer
global pretty-printers:
  PostgreSQLPrettyPrinter
    Bitmapset
    Relids
  builtin
    mpx_bound128
[...]

When the content of the variable unused_batch_states is now printed in GDB, it looks as follows.

(gdb) print *node_state->unused_batch_states
$3 = PGBitmapset ([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15])

The output now clearly shows which integer values are part of the bitmap set. This is similar to the output of the bmsToString function shown above. The main difference is that the GDB extension also works when coredump files are analyzed and PostgreSQL is not running.

The pretty printer has to be loaded via the source command every time GDB is restarted. This is cumbersome. To ease the work with this extension, the command can be added to the ~/.gdbinit file. The commands of this file are automatically executed every time GDB is invoked.

cat ~/.gdbinit 
source /home/jan/dev/postgresql_printer.py