Skip to content

[Bug]: gRPC interceptor swallows exceptions and returns None instead of re-raising #895

Description

@Harshad01

Problem Description

The create_span wrapper in instana/instrumentation/grpcio.py catches exceptions
from gRPC stubs, records them on the trace span, and then falls off the end of
the function returning None. gRPC then tries to unpack the return value as a tuple:

response, call = None   # inside grpc._interceptor._with_call

which raises:

TypeError: cannot unpack non-iterable NoneType object

The real gRPC status code (e.g. PERMISSION_DENIED) is silently absorbed into the
span and never reaches the caller. This makes the error completely undiagnosable
from the application side.

This only happens inside an active Instana span (i.e. inside a traced API request).
Calling the same gRPC method without Instana active gives the correct RpcError.

Instana version: 3.15.0
pymilvus version: 2.6.13 (gRPC client)
Python: 3.11/3.12

Minimal, Complete, Verifiable, Example

# Reproduce: make a gRPC call via pymilvus inside a traced FastAPI request
# (Instana must be active with a live span)

from pymilvus import utility
utility.has_collection("my_collection", using="my_connection")

# Expected: grpc.RpcError with StatusCode.PERMISSION_DENIED
# Actual:   TypeError: cannot unpack non-iterable NoneType object

# Root cause in instana/instrumentation/grpcio.py::create_span:
# The wrapper catches the gRPC exception, records it on the span via
# span.record_exception(exc), but never re-raises — the function returns
# None implicitly.
# gRPC then does:  response, call = None  -> TypeError
#
# Fix: add `raise` after span.record_exception(exc) in create_span

Python Version

python 3.12

Python Modules

instana==3.15.0
grpcio==1.62.x
pymilvus==2.6.13
wrapt==1.x

Python Environment

Running inside a CP4D (Cloud Pak for Data) Kubernetes pod.
Instana is loaded via AUTOWRAPT_BOOTSTRAP=instana environment variable.
Python 3.11 / 3.12 (container image).

Key env vars relevant to this bug:
AUTOWRAPT_BOOTSTRAP=instana
INSTANA_SERVICE_NAME=datasift-api

Activity

  1. self-assigned this
    on Oct 5, 2026
  2. CagriYonca commented on Oct 6, 2026

    @CagriYonca
    Contributor

    Hello @Harshad01 , thanks for the clear report and minimal reproduction.

    Your analysis is correct. create_span was catching gRPC exceptions and recording them on the span, but never re-raising them. This caused the wrapper to return None, leading to the tuple unpacking TypeError inside gRPC's interceptor layer and masking the underlying grpc.RpcError.

    We've addressed this in PR #911 by:

    • Re-raising wrapped exceptions after recording them on the span.
    • Safely isolating internal sensor telemetry.
    • Adding regression tests for both with_call and gRPC interceptors.

    Could you please review the changes and, if possible, pull from the branch and try with your application?

    pip install --upgrade git+https://github.com/instana/python-sensor.git@fix-grpcio
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions