Skip to content

qcom_scm: WAITQ_SLEEP from QTEE SHM bridge calls is never woken on Kaanapali/SM8750, deadlocks all SCM callers and hangs boot #1361

Open

Description

Summary

On Kaanapali MTP and SM8750 MTP, TZ intermittently answers a QTEE SHM bridge
call (qcom_scm_shm_bridge_create() / qcom_scm_shm_bridge_delete()) with
QCOM_SCM_WAITQ_SLEEP. The waitq wake-up interrupt (qcom-scm, GIC ESPI hwirq
4131) never fires, so the caller sleeps in qcom_scm_wait_for_wq_completion()
forever. Because __scm_smc_do() holds qcom_scm_lock across that wait, every
other SCM caller blocks behind it, including the GPU's adreno_hw_init(). The
resulting chain (GPU fence -> DRM commit -> fbcon unblank under console_lock)
leaves userspace unable to open /dev/console, and the boot never reaches a
login prompt.

This is the cause of most login timeouts on these two boards in the meta-qcom
CI (about 5-9% of qcom-distro boots, all boards affected).

Environment

  • Kernel: qcom-next qcom-next-7.3-rc6-20261023 (7.3.0-rc6-gc7896917e523)
  • QTEE version 5.2.0 (qcomtee driver, qtee_supplicant from minkipc)
  • Kaanapali MTP: BOOT.MXF.2.5.3-00396-KAANAPALI-1, UEFI BOOT.MXF_UEFI.2.5-00653-KAANAPALI-1
  • SM8750 MTP: BOOT.MXF.2.5.1-00377-PAKALA-2, UEFI BOOT.MXF_UEFI.2.5-00653-PAKALA-1
  • Image: meta-qcom master, qcom-distro, qcom-multimedia-image
    (kas build ci/kaanapali-mtp.yml:ci/qcom-distro.yml --target qcom-multimedia-image)
  • Kernel command line: root=PARTLABEL=rootfs rw rootwait console=ttyMSM0,115200 qcom_scm.download_mode=1

Reproduction

Flash and boot the image repeatedly. In our LAVA runs, roughly 1 in 5-10 boots
hangs; the console shows Started Serial Getty on ttyMSM0 and then no login
prompt, while kernel messages keep printing. Every hung boot we instrumented had
a scm_waitq_sleep event; none of the good boots did (0 in 12).

Evidence

All of the following was captured on hung boots with a timer that dumps state
to /dev/kmsg 60 s after boot, with the qcom_scm trace events enabled early.

1. The SCM call that sleeps and is never woken

Kaanapali (holder = qcomtee release worker):

kworker/u32:1-13  [004] 10.342581: scm_smc_request: smc_id:0x42000c1e svc_id:0x0c cmd_id:0x1e args_cnt:4 args:{0x8a8845006,0x8a8845006,0x1200,0x3}
kworker/u32:1-13  [004] 10.342657: scm_smc_done: smc_id:0x42000c1e, ret:0 res_to_callee:0x0 res0:0x0 res1:0x1fc73c2d res2:0x0
kworker/u32:1-13  [004] 10.342708: scm_smc_request: smc_id:0x72000602 svc_id:0x06 cmd_id:0x02 args_cnt:4 args:{0x8a8845000,0x1000,0x8c2778000,0x1000}
kworker/u32:1-13  [004] 10.342786: scm_smc_done: smc_id:0x72000602, ret:0 res_to_callee:0x0 res0:0x0 res1:0x0 res2:0x0
kworker/u32:1-13  [004] 10.366143: scm_smc_request: smc_id:0x42000c1d svc_id:0x0c cmd_id:0x1d args_cnt:1 args:{0x1fc73c2d}
kworker/u32:1-13  [004] 10.366206: scm_waitq_sleep: wq_ctx:0, smc_call_ctx:1
kworker/u32:1+qcomtee_wq state=I wchan=qcom_scm_wait_for_wq_completion
 qcom_scm_wait_for_wq_completion+0x40/0x80
 __scm_smc_do+0x114/0x398
 __scm_smc_call+0xc4/0x430
 qcom_scm_call+0x54/0x98
 qcom_scm_shm_bridge_delete+0x74/0xb0
 qcom_tzmem_shm_bridge_delete+0x28/0x40
 qcomtee_shm_unregister+0x1c/0x38 [qcomtee]
 tee_dyn_shm_free_helper+0x30/0x60
 pool_op_free+0x24/0x38 [qcomtee]
 tee_shm_put+0x178/0x1c0
 tee_shm_free+0x18/0x30
 qcomtee_msg_buffers_free+0x24/0x48 [qcomtee]
 qcomtee_object_do_invoke_internal+0x578/0x1318 [qcomtee]
 qcomtee_do_release_qtee_object+0x70/0x128 [qcomtee]
 process_one_work+0x1c0/0x548

SM8750 (holder = qtee_supplicant), with no other SCM call in flight (the
previous one completed 2 s earlier):

kworker/u34:6-151      [006] 12.583775: scm_smc_done: smc_id:0x4200010a, ret:0 ...
qtee_supplicant-701    [006] 14.641494: scm_smc_request: smc_id:0x42000c1e svc_id:0x0c cmd_id:0x1e args_cnt:4 args:{0x8a9fe8006,0x8a9fe8006,0x1200,0x3}
qtee_supplicant-701    [006] 14.641617: scm_waitq_sleep: wq_ctx:0, smc_call_ctx:1
qtee_supplicant state=I wchan=qcom_scm_wait_for_wq_completion
 qcom_scm_wait_for_wq_completion+0x40/0x80
 __scm_smc_do+0x114/0x398
 __scm_smc_call+0xc4/0x430
 qcom_scm_call+0x54/0x98
 qcom_scm_shm_bridge_create+0x94/0x100
 qcom_tzmem_shm_bridge_create+0x58/0xa0
 qcomtee_shm_register+0x28/0x40 [qcomtee]
 tee_dyn_shm_alloc_helper+0xf8/0x170
 pool_op_alloc+0x2c/0x40 [qcomtee]
 shm_alloc_helper+0xd4/0x160
 tee_shm_alloc_priv_buf+0x24/0x40
 qcomtee_msg_buffers_alloc+0x194/0x248 [qcomtee]
 qcomtee_object_do_invoke_internal+0x50/0x1318 [qcomtee]
 qcomtee_object_do_invoke+0x44/0x60 [qcomtee]
 qcomtee_object_invoke+0x2b4/0x6c0 [qcomtee]
 tee_ioctl+0xf3c/0x1160

The wake-up interrupt never fires, on any CPU, on any boot we looked at:

 43:  0  0  0  0  0  0  0  0  GICv3  4131 Edge  qcom-scm

2. Everything else blocks on qcom_scm_lock

task:kworker/u32:4   state:D  Workqueue: ring0 drm_sched_run_job_work [gpu_sched]
 __mutex_lock.constprop.0+0x508/0xac0
 mutex_lock+0x50/0x70
 __scm_smc_do+0x78/0x398
 __scm_smc_call+0xc4/0x430
 qcom_scm_call+0x54/0x98
 __qcom_scm_is_call_available+0x8c/0x110
 qcom_scm_set_gpu_smmu_aperture_is_available+0x2c/0x40
 adreno_hw_init+0xb8/0x108 [msm]
 a8xx_hw_init+0xd4c/0x25e0 [msm]
 msm_gpu_hw_init+0x60/0xa0 [msm]
 msm_gpu_submit+0x4c/0x1b0 [msm]
 msm_job_run+0xb4/0x140 [msm]
 drm_sched_run_job_work+0x128/0x3e8 [gpu_sched]

qtee_supplicant (Kaanapali case) and wireplumber (SM8750 case) are blocked
the same way in __scm_smc_do+0x78 -> mutex_lock.

3. Knock-on: DRM and console_lock

kworker/u33:1+events_unbound state=D: dma_fence_default_wait <- drm_atomic_helper_wait_for_fences <- commit_tail <- commit_work
(agetty) state=S: drm_atomic_helper_swap_state <- drm_atomic_helper_commit <- drm_client_modeset_commit_atomic
         <- __drm_fb_helper_restore_fbdev_mode_unlocked <- drm_fb_helper_set_par <- fb_set_var <- fbcon_blank
         <- do_unblank_screen <- vt_ioctl                     (holds console_lock)
kworker/0:2+events state=D: console_lock <- console_callback
weston state=D: modeset_lock <- drm_helper_probe_single_connector_modes <- drm_mode_getconnector

and on other hung boots the serial getty's systemd executor is stuck opening
the console:

(agetty) state=D wchan=console_device
 console_device+0x48/0x100
 tty_lookup_driver+0xb0/0x268
 tty_open+0x28c/0x6c8
 chrdev_open+0xbc/0x258
 ...
 __arm64_sys_openat+0x68/0xc0

Experiment: drop qcom_scm_lock while waiting

To check whether the firmware just needed other SMC calls to make progress, we
tested this change (debug only, not a proposed fix):

--- a/drivers/firmware/qcom/qcom_scm-smc.c
+++ b/drivers/firmware/qcom/qcom_scm-smc.c
@@ static int __scm_smc_do_quirk_handle_waitq(...)
 			trace_scm_waitq_sleep(wq_ctx, smc_call_ctx);
+			mutex_unlock(&qcom_scm_lock);
 			ret = qcom_scm_wait_for_wq_completion(dev, wq_ctx);
+			mutex_lock(&qcom_scm_lock);
 			if (ret)
 				return ret;

Result on Kaanapali: 19 of 19 boots reached the login prompt, and 4 of them
hit a waitq sleep (the same rate as the hang). In all 4:

  • the wq_ctx:0 sleeper was still never resumed (scm_waitq_resume = 0,
    qcom-scm IRQ count = 0), even though other SMC calls were now able to run;
  • later SHM bridge calls also got WAITQ_SLEEP, but with wq_ctx:2147483648
    (0x80000000), which qcom_scm_get_completion() rejects:
kworker/u32:3-406     [007] 10.685471: scm_waitq_sleep: wq_ctx:0, smc_call_ctx:1
qtee_supplicant-785   [007] 10.713989: scm_waitq_sleep: wq_ctx:2147483648, smc_call_ctx:2
kworker/u32:1-13      [005] 12.405981: scm_waitq_sleep: wq_ctx:2147483648, smc_call_ctx:3
kworker/u32:2-330     [000] 21.487378: scm_waitq_sleep: wq_ctx:2147483648, smc_call_ctx:4

WARNING: drivers/firmware/qcom/qcom_scm.c:2699 at qcom_scm_wait_for_wq_completion+0x50/0x80, CPU#7: qtee_supplicant/785
CPU: 7 UID: 0 PID: 785 Comm: qtee_supplicant Not tainted 7.3.0-rc6-gc7896917e523-p1 #1 PREEMPT
Hardware name: Qualcomm Technologies, Inc. Kaanapali MTP (DT)
pc : qcom_scm_wait_for_wq_completion+0x50/0x80
Call trace:
 qcom_scm_wait_for_wq_completion+0x50/0x80 (P)
 qcom_scm_call+0x54/0x98
 qcom_scm_shm_bridge_delete+0x74/0xb0
 qcom_tzmem_shm_bridge_delete+0x28/0x40
 qcomtee_shm_unregister+0x1c/0x38 [qcomtee]
 tee_dyn_shm_free_helper+0x30/0x60
 pool_op_free+0x24/0x38 [qcomtee]
 tee_shm_put+0x178/0x1c0
 tee_shm_fop_release+0x1c/0x40
 __fput+0xdc/0x310
 ...

So releasing the lock removes the system-wide hang, but the underlying problem
remains: the waitq wake-up never reaches Linux on these SoCs.

Questions

  1. Is the waitq wake-up interrupt (hwirq 4131 from WAITQ_GET_INFO) expected
    to be delivered to HLOS on Kaanapali and SM8750 with this TZ/QTEE firmware?
    It never fires on any boot we captured.
  2. What does a wq_ctx of 0x80000000 mean in the WAITQ_SLEEP response?
    It looks like a flag or encoding that the driver doesn't handle, rather than
    an index below wq_cnt.
  3. Should __scm_smc_do() hold qcom_scm_lock across
    qcom_scm_wait_for_wq_completion()? With the current code, any waitq sleep
    that is not woken blocks every SCM user in the system.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions