Debug CPython with gdb¶
Put a breakpoint on the next exception: break _PyErr_SetObject (or break PyErr_SetObject in Python 3.7 and older).
Then use condition to only break at the expected exception.
Watch when reference count changes¶
Use a memory breakpoint like:
watch ((PyObject*)MEMORY_ADDRESS)->ob_refcnt
where MEMORY_ADDRESS is the address of a Python object.
Analyze a coredump¶
Current Python thread state, like PyThreadState_GET() :
p ((PyThreadState*)_PyRuntime.gilstate.tstate_current)
Current error, like PyErr_Occurred() :
p ((PyThreadState*)_PyRuntime.gilstate.tstate_current)->curexc_type
Load python-gdb.py¶
(gdb) source /path/to/python-gdb.py
Add directory containing Python source code to “safe path”, to automatically load python-gdb.py when debugging Python. Add the following line to your ~/.gdbinit :
In my case, Python is in ~/prog/python/master , but I chose to allow to load any Python script from ~/prog/ .
On Fedora, the script is provided as:
/usr/lib/debug/usr/lib64/libpython3.6m.so.1.0-3.6.6-1.fc28.x86_64.debug-gdb.py
Debug functions¶
You might want to call these functions in a running process from gdb:
- _PyObject_Dump(obj)
- _PyUnicode_Dump(obj) : dump properties of the Unicode object, not it’s content
- PyErr_Occurred() , _PyErr_Occurred(tstate) or tstate->curexc_type : get the current exception type, NULL if no exception was raised.
- Check object consistency:
- _PyDict_CheckConsistency()
- _PyUnicode_CheckConsistency()
- _PyObject_CheckConsistency()
- _PyType_CheckConsistency()
- _PyWideStringList_CheckConsistency()
- _Py_DumpTraceback(2, tstate)
- _Py_DumpTracebackThreads(2, interp, tstate) where tstate can be NULL
- Python 3.8: get tstate from _PyRuntime.gilstate.tstate_current and interp from _PyRuntime.gilstate.autoInterpreterState
- 2 is the file descriptor 2: stderr
© Copyright 2022, Victor Stinner Revision 0fa868fd .
Versions latest Downloads html On Read the Docs Project Home Builds Free document hosting provided by Read the Docs.
User
You need to have gdb on your system and Python debugging extensions. Extensions package includes debugging symbols and adds Python-specific commands into gdb. On a modern Linux system, you can easily install these with:
- sudo yum install yum-utils
- sudo debuginfo-install glibc
- sudo yum install gdb python-debuginfo
* tested on Centos 7. python-debuginfo is installable after the first two commands.
For gdb support on legacy systems, look at the end of this page.
Running with `gdb`
- run python under gdb from the start. Note: the python executable needs to have debug symbols in it which may be another exe python2.7-dbg depending on your system
- attach to already running python process
To run python under gdb there are also two ways.
This will run the program til it exits, segfaults or you manually stop execution (using Ctrl+C).
Attaching to a running process like this will cause it to stop. You can tell it to continue running with c command.
Debugging process
If your program segfaulted, gdb will automatically pause the program, so you can switch into gdb console to inspect its state. You can also manually interrupt program execution by pressing Ctrl+C in the console.
See the page EasierPythonDebugging for the list of Python helper commands for gdb.
Getting a C Stack Trace
If you are debugging a segfault, this is probably the first thing you want to do.
(gdb) bt #0 0x0000002a95b3b705 in raise () from /lib/libc.so.6 #1 0x0000002a95b3ce8e in abort () from /lib/libc.so.6 #2 0x00000000004c164f in posix_abort (self=0x0, noargs=0x0) at ../Modules/posixmodule.c:7158 #3 0x0000000000489fac in call_function (pp_stack=0x7fbffff110, oparg=0) at ../Python/ceval.c:3531 #4 0x0000000000485fc2 in PyEval_EvalFrame (f=0x66ccd8) at ../Python/ceval.c:2163 .
With luck, this will give some idea of where the problem is occurring and if it doesn’t help you fix the problem, it can help someone else track down the problem.
The quality of the results will depend greatly on the amount of debug information available.
Getting a Python Stack Trace
to get stack trace with familiar Python source code.
Working With Hung Processes
If a process appears hung, it will either be waiting on something (a lock, IO, etc), or be in a busy loop somewhere. In either case, attaching to the process and getting a back trace can help.
If the process is in a busy loop, you may want to continue execution for a bit (using the cont command), then break (Ctrl+C) again and bring up a stack trace.
(gdb) info threads Id Target Id Frame 37 Thread 0xa29feb40 (LWP 17914) "NotificationThr" 0xb7fdd424 in __kernel_vsyscall () 36 Thread 0xa03fcb40 (LWP 17913) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 35 Thread 0xa0bfdb40 (LWP 17911) "QProcessManager" 0xb7fdd424 in __kernel_vsyscall () 34 Thread 0xa13feb40 (LWP 17910) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 33 Thread 0xa1bffb40 (LWP 17909) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 31 Thread 0xa31ffb40 (LWP 17907) "QFileInfoGather" 0xb7fdd424 in __kernel_vsyscall () 30 Thread 0xa3fdfb40 (LWP 17906) "QInotifyFileSys" 0xb7fdd424 in __kernel_vsyscall () 29 Thread 0xa481cb40 (LWP 17905) "QFileInfoGather" 0xb7fdd424 in __kernel_vsyscall () 7 Thread 0xa508db40 (LWP 17883) "QThread" 0xb7fdd424 in __kernel_vsyscall () 6 Thread 0xa5cebb40 (LWP 17882) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 5 Thread 0xa660cb40 (LWP 17881) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 3 Thread 0xabdffb40 (LWP 17876) "gdbus" 0xb7fdd424 in __kernel_vsyscall () 2 Thread 0xac7b7b40 (LWP 17875) "dconf worker" 0xb7fdd424 in __kernel_vsyscall () * 1 Thread 0xb7d876c0 (LWP 17863) "python2.7" 0xb7fdd424 in __kernel_vsyscall ()
(gdb) py-list 2025 # Open external files with our Mac app 2026 if sys.platform == "darwin" and 'Spyder.app' in __file__: 2027 main.connect(app, SIGNAL('open_external_file(QString)'), 2028 lambda fname: main.open_external_file(fname)) 2029 >2030 app.exec_() 2031 return main 2032 2033 2034 def __remove_temp_session(): 2035 if osp.isfile(TEMP_SESSION_PATH):(gdb) thread apply all py-list . 200 201 def accept(self): >202 sock, addr = self._sock.accept() 203 return _socketobject(_sock=sock), addr 204 accept.__doc__ = _realsocket.accept.__doc__ 205 206 def dup(self): 207 """dup() -> socket object Thread 35 (Thread 0xa0bfdb40 (LWP 17911)): Unable to locate python frame Thread 34 (Thread 0xa13feb40 (LWP 17910)): 197 for method in _delegate_methods: 198 setattr(self, method, dummy) 199 close.__doc__ = _realsocket.close.__doc__ 200 201 def accept(self): >202 sock, addr = self._sock.accept() 203 return _socketobject(_sock=sock), addr .
References
GDB on Legacy systems
It may happen that you need to use gdb on a legacy system without advanced Python support. In this case you may find the following information useful.
GDB Macros
A set of GDB macros are distributed with Python that aid in debugging the Python process. You can install them by adding the contents of Misc/gdbinit in the Python sources to ~/.gdbinit — or copy it from Subversion. Be sure to use the correct version for your version of Python or some features will not work.
Note that the new GDB commands this file adds will only work correctly if debugging symbols are available.
- Recompiling python with make «CFLAGS=-g -fno-inline -fno-strict-aliasing» solves this problem.
- Patching the conditionals for Python frames in the pystack and pystackv routines to ignore frames with where $fp is 0, like so:
PyEval_EvalFrameEx && $pc >>>> if $pc > PyEval_EvalFrameEx && $pc < PyEval_EvalCodeEx && $fp != 0
Also note that the stop condition for the while-loops in the pystack and pystackv routines were originally designed for the case where you're running the Python interpreter directly, and not running the interpreter withing another program (by loading the shared library and manually bootstrapping the interpreter). So you may need to tweak the while-loops depending on the program you're intending to debug. See, for example, this StackOverflow post for another (putative) stop condition.