1*cc02d7e2SAndroid Build Coastguard Worker# gRPC C++ Debugging Example 2*cc02d7e2SAndroid Build Coastguard Worker 3*cc02d7e2SAndroid Build Coastguard WorkerThis example demonstrates a handful of ways you can debug your gRPC C++ applications. 4*cc02d7e2SAndroid Build Coastguard Worker 5*cc02d7e2SAndroid Build Coastguard Worker## Enabling Trace Logs 6*cc02d7e2SAndroid Build Coastguard Worker 7*cc02d7e2SAndroid Build Coastguard WorkergRPC allows you to configure more detailed log output for various aspects of gRPC behavior. The tracing log generation might have a large overhead and result in significantly larger log file sizes, especially when you try to trace transport or timer_check. But it is a powerful tool in your debugging toolkit. 8*cc02d7e2SAndroid Build Coastguard Worker 9*cc02d7e2SAndroid Build Coastguard Worker### The Most Verbose Logging 10*cc02d7e2SAndroid Build Coastguard Worker 11*cc02d7e2SAndroid Build Coastguard WorkerSpecify environment variables, then run your application: 12*cc02d7e2SAndroid Build Coastguard Worker 13*cc02d7e2SAndroid Build Coastguard Worker``` 14*cc02d7e2SAndroid Build Coastguard WorkerGRPC_VERBOSITY=debug 15*cc02d7e2SAndroid Build Coastguard WorkerGRPC_TRACE=all 16*cc02d7e2SAndroid Build Coastguard Worker``` 17*cc02d7e2SAndroid Build Coastguard Worker 18*cc02d7e2SAndroid Build Coastguard WorkerFor more granularity, please see 19*cc02d7e2SAndroid Build Coastguard Worker[environment_variables](https://github.com/grpc/grpc/blob/master/doc/environment_variables.md). 20*cc02d7e2SAndroid Build Coastguard Worker 21*cc02d7e2SAndroid Build Coastguard Worker### Debug Transport Protocol 22*cc02d7e2SAndroid Build Coastguard Worker 23*cc02d7e2SAndroid Build Coastguard Worker``` 24*cc02d7e2SAndroid Build Coastguard WorkerGRPC_VERBOSITY=debug 25*cc02d7e2SAndroid Build Coastguard WorkerGRPC_TRACE=tcp,http,secure_endpoint,transport_security 26*cc02d7e2SAndroid Build Coastguard Worker``` 27*cc02d7e2SAndroid Build Coastguard Worker 28*cc02d7e2SAndroid Build Coastguard Worker### Debug Connection Behavior 29*cc02d7e2SAndroid Build Coastguard Worker 30*cc02d7e2SAndroid Build Coastguard Worker``` 31*cc02d7e2SAndroid Build Coastguard WorkerGRPC_VERBOSITY=debug 32*cc02d7e2SAndroid Build Coastguard WorkerGRPC_TRACE=call_error,connectivity_state,pick_first,round_robin,glb 33*cc02d7e2SAndroid Build Coastguard Worker``` 34*cc02d7e2SAndroid Build Coastguard Worker 35*cc02d7e2SAndroid Build Coastguard Worker## GDB and other debuggers 36*cc02d7e2SAndroid Build Coastguard Worker 37*cc02d7e2SAndroid Build Coastguard Worker`gdb` (and the like) are tools that lets you inspect your application while it is running, view stack traces on exceptions, pause and step through code at specified points or under certain conditions, etc. See https://www.sourceware.org/gdb/ 38*cc02d7e2SAndroid Build Coastguard Worker 39*cc02d7e2SAndroid Build Coastguard Worker### Inspecting errors 40*cc02d7e2SAndroid Build Coastguard Worker 41*cc02d7e2SAndroid Build Coastguard Worker``` 42*cc02d7e2SAndroid Build Coastguard Workerbazel build --config=dbg examples/cpp/debugging:crashing_greeter_client 43*cc02d7e2SAndroid Build Coastguard Workergdb -ex run \ 44*cc02d7e2SAndroid Build Coastguard Worker --args ./bazel-bin/examples/cpp/debugging/crashing_greeter_client \ 45*cc02d7e2SAndroid Build Coastguard Worker --crash_on_errors=true \ 46*cc02d7e2SAndroid Build Coastguard Worker --target=localhork:50051 47*cc02d7e2SAndroid Build Coastguard Worker``` 48*cc02d7e2SAndroid Build Coastguard Worker 49*cc02d7e2SAndroid Build Coastguard WorkerOnce the exception is thrown, you can use `bt` to see the stack trace and examine the crash, `info threads` to get the set of threads, etc. See the [GDB documentation](https://sourceware.org/gdb/current/onlinedocs/gdb.html/) for a more complete list of available features and commands. 50*cc02d7e2SAndroid Build Coastguard Worker 51*cc02d7e2SAndroid Build Coastguard Worker### Breaking inside a function 52*cc02d7e2SAndroid Build Coastguard Worker 53*cc02d7e2SAndroid Build Coastguard WorkerAfter building the application above, this will break inside gRPC generated stub code: 54*cc02d7e2SAndroid Build Coastguard Worker 55*cc02d7e2SAndroid Build Coastguard Worker``` 56*cc02d7e2SAndroid Build Coastguard Workergdb -ex 'b helloworld::Greeter::Stub::SayHello' \ 57*cc02d7e2SAndroid Build Coastguard Worker -ex run \ 58*cc02d7e2SAndroid Build Coastguard Worker --args ./bazel-bin/examples/cpp/debugging/crashing_greeter_client \ 59*cc02d7e2SAndroid Build Coastguard Worker --crash_on_errors=true \ 60*cc02d7e2SAndroid Build Coastguard Worker --target=localhork:50051 61*cc02d7e2SAndroid Build Coastguard Worker``` 62*cc02d7e2SAndroid Build Coastguard Worker 63*cc02d7e2SAndroid Build Coastguard Worker## gRPC Admin Interface: Live Channel Tracing 64*cc02d7e2SAndroid Build Coastguard Worker 65*cc02d7e2SAndroid Build Coastguard WorkerThe [gRPC Admin Service](https://github.com/grpc/proposal/blob/master/A38-admin-interface-api.md) 66*cc02d7e2SAndroid Build Coastguard Workerprovides a convenient API in each gRPC language to improve the usability of 67*cc02d7e2SAndroid Build Coastguard Workercreating a gRPC server with admin services to expose states in the gRPC library. 68*cc02d7e2SAndroid Build Coastguard WorkerThis includes channelz, which is a channel tracing feature; it tracks statistics 69*cc02d7e2SAndroid Build Coastguard Workerlike how many messages have been sent, how many of them failed, what are the 70*cc02d7e2SAndroid Build Coastguard Workerconnected sockets. See the [Channelz design doc](https://github.com/grpc/proposal/blob/master/A14-channelz.md). 71*cc02d7e2SAndroid Build Coastguard Worker 72*cc02d7e2SAndroid Build Coastguard Worker### Integrating the gRPC Admin Service Into Your Server 73*cc02d7e2SAndroid Build Coastguard Worker 74*cc02d7e2SAndroid Build Coastguard WorkerAs seen in the `greeter_callback_admin_server` target, you canenable admin services by using the `AddAdminServices` method. 75*cc02d7e2SAndroid Build Coastguard Worker 76*cc02d7e2SAndroid Build Coastguard Worker``` 77*cc02d7e2SAndroid Build Coastguard Workergrpc::ServerBuilder builder; 78*cc02d7e2SAndroid Build Coastguard Workergrpc::AddAdminServices(&builder); 79*cc02d7e2SAndroid Build Coastguard Workerbuilder.AddListeningPort(":50051", grpc::ServerCredentials(...)); 80*cc02d7e2SAndroid Build Coastguard Workerstd::unique_ptr<grpc::Server> server(builder.BuildAndStart()); 81*cc02d7e2SAndroid Build Coastguard Worker``` 82*cc02d7e2SAndroid Build Coastguard Worker 83*cc02d7e2SAndroid Build Coastguard Worker### Using grpcdebug 84*cc02d7e2SAndroid Build Coastguard Worker 85*cc02d7e2SAndroid Build Coastguard Workergrpcdebug is a tool created to access the metrics from channelz and health services. 86*cc02d7e2SAndroid Build Coastguard Worker 87*cc02d7e2SAndroid Build Coastguard Worker#### Installing the grpcdebug tool 88*cc02d7e2SAndroid Build Coastguard Worker 89*cc02d7e2SAndroid Build Coastguard WorkerThe source code is located in a github project 90*cc02d7e2SAndroid Build Coastguard Worker[grpc-ecosystem/grpcdebug](https://github.com/grpc-ecosystem/grpcdebug). You 91*cc02d7e2SAndroid Build Coastguard Workercan either download [the latest built version] 92*cc02d7e2SAndroid Build Coastguard Worker(https://github.com/grpc-ecosystem/grpcdebug/releases/latest) (recommended) or 93*cc02d7e2SAndroid Build Coastguard Workerfollow the README.md to build it yourself. 94*cc02d7e2SAndroid Build Coastguard Worker 95*cc02d7e2SAndroid Build Coastguard Worker#### Running the grpcdebug tool 96*cc02d7e2SAndroid Build Coastguard Worker##### Usage 97*cc02d7e2SAndroid Build Coastguard Worker`grpcdebug <target address> [flags] channelz <command> [argument]` 98*cc02d7e2SAndroid Build Coastguard Worker 99*cc02d7e2SAndroid Build Coastguard Worker 100*cc02d7e2SAndroid Build Coastguard Worker| Command | Argument | Description | 101*cc02d7e2SAndroid Build Coastguard Worker| :--------- | :------------------: | :------------------------------------------------ | 102*cc02d7e2SAndroid Build Coastguard Worker| channel | \<channel id or URL> | Display channel states in a human readable way. | 103*cc02d7e2SAndroid Build Coastguard Worker| channels | | Lists client channels for the target application. | 104*cc02d7e2SAndroid Build Coastguard Worker| server | \<server ID> | Displays server state in a human readable way. | 105*cc02d7e2SAndroid Build Coastguard Worker| servers | | Lists servers in a human readable way. | 106*cc02d7e2SAndroid Build Coastguard Worker| socket | \<socket ID> | Displays socket states in a human readable way. | 107*cc02d7e2SAndroid Build Coastguard Worker| subchannel | \<id> | Display subchannel states in human readable way. | 108*cc02d7e2SAndroid Build Coastguard Worker 109*cc02d7e2SAndroid Build Coastguard WorkerGenerally, you will start with either `servers` or `channels` and then work down 110*cc02d7e2SAndroid Build Coastguard Workerto the details 111*cc02d7e2SAndroid Build Coastguard Worker 112*cc02d7e2SAndroid Build Coastguard Worker##### Getting overall server info 113*cc02d7e2SAndroid Build Coastguard Worker 114*cc02d7e2SAndroid Build Coastguard WorkerTo begin with, build and run the server binary in the background 115*cc02d7e2SAndroid Build Coastguard Worker 116*cc02d7e2SAndroid Build Coastguard Worker``` 117*cc02d7e2SAndroid Build Coastguard Workerbazel build --config=dbg examples/cpp/debugging:all 118*cc02d7e2SAndroid Build Coastguard Worker./bazel-bin/examples/cpp/debugging/greeter_callback_server_admin& 119*cc02d7e2SAndroid Build Coastguard Worker``` 120*cc02d7e2SAndroid Build Coastguard Worker 121*cc02d7e2SAndroid Build Coastguard WorkerYou can then inspect the server 122*cc02d7e2SAndroid Build Coastguard Worker```bash 123*cc02d7e2SAndroid Build Coastguard Workergrpcdebug localhost:50051 channelz servers 124*cc02d7e2SAndroid Build Coastguard Worker``` 125*cc02d7e2SAndroid Build Coastguard Worker 126*cc02d7e2SAndroid Build Coastguard WorkerThis will show you the server ids with their activity 127*cc02d7e2SAndroid Build Coastguard Worker```text 128*cc02d7e2SAndroid Build Coastguard WorkerServer ID Listen Addresses Calls(Started/Succeeded/Failed) Last Call Started 129*cc02d7e2SAndroid Build Coastguard Worker1 [[::]:50051] 38/34/3 now 130*cc02d7e2SAndroid Build Coastguard Worker``` 131*cc02d7e2SAndroid Build Coastguard Worker 132*cc02d7e2SAndroid Build Coastguard WorkerFor more information about `grpcdebug` features, please see [the grpcdebug documentation](https://github.com/grpc-ecosystem/grpcdebug) 133