Date: 07-31-2024

When a program crashes, the first step in finding the cause is to get a stack trace, which shows what a program was executing at the function level at the time of the crash. Stack traces can be obtained using the common gdb debugging utility. If a system is configured to save a core dump (a snapshot of the memory used by the crashing program), the core file may be opened with gdb and the stack trace can be extracted with a few commands. If no core file is available, gdb can be attached to the slapd process and be told to save a stack trace if slapd crashes again. If slapd appears to be hung or unresponsive, gdb can also be attached to extract a stack trace.

Setup

Install gdb

which gdb

If gdb is not found, install it through your servers' package manager:

# Red Hat 7
yum install gdb

# Red Hat 8+
dnf install gdb

# Ubuntu/Debian
apt install gdb

Install Symas Debugging Packages

Symas OpenLDAP packages are built with performance optimizations that package debugging information/symbols separately. When creating a slapd backtrace or analyzing a slapd core dump with gdb, the debugging info/symbols should be installed to get more information from gdb.

The following packages should be installed through your servers' package manager (yum/dnf/apt):

• symas-libevent-libs
• symas-openldap-debuginfo (RHEL)
• symas-openldap-server-dbgsym (Debian/Ubuntu)
• symas-openldap-devel
• symas-openldap-libs

Note: The above packages must be the same version as the version of slapd being debugged.

Optional debugging packages:

User Permissions

When getting a stack trace from gdb, gdb must be run as root.

Stack Trace From a Core Dump

If you have a core dump, do the following:

Create the file “get-backtrace.gdb”:

set confirm off
set pagination off
set logging file slapd-backtrace.txt
set logging enabled on
thread apply all bt full

# Optional - Uncomment below to generate core file
# Make sure you have enough disk space to save the entire core file
#generate-core-file

set logging enabled off
quit

Then, run the following:

gdb -x get-backtrace.gdb /opt/symas/lib/slapd <path-to-core-dump>

A stack trace will be extracted and written to the file “slapd-backtrace.txt”.

Stack Trace From slapd While Running

If slapd appears to be hung or is unresponsive, you can attach gdb to the slapd process to get a stack trace. This will not kill the slapd process.

Using the same “get-backtrace.gdb” script from above, run the following:

gdb -x get-backtrace.gdb /opt/symas/lib/slapd $(pidof slapd)

A stack trace will be extracted and written to the file “slapd-backtrace.txt”. If also creating a core file with this gdb configuration, the core file will be named “core.pid”, also in the current working directory.

Capture Stack Trace on a Crash

If slapd is unpredictably crashing, gdb can be attached to the slapd process and create a stack trace when a crash occurs.

Note: It is recommended that this procedure be performed in a tmux or screen session. In case your terminal session is inadvertently closed, the gdb session will continue to run. 

Create the file “bt_wait.gdb”:

set confirm off
set pagination off
set logging file slapd-backtrace.txt
set logging enabled on
handle SIGPIPE noprint nopass ignore
continue
thread apply all bt full

# Optional - Uncomment below to generate core file
# Make sure you have enough disk space to save the entire core file
#generate-core-file

set logging enabled off
quit

Run the script in gdb:

gdb -x bt_wait.gdb /opt/symas/lib/slapd $(pidof slapd)

If slapd crashes, there will be a file named "slapd-backtrace.txt" in the current working directory. If also creating a core file with this gdb configuration, the core file will be named “core.pid”, also in the current working directory.