Repository navigation
Description
Summary
On Kaanapali MTP and SM8750 MTP, TZ intermittently answers a QTEE SHM bridge
call (qcom_scm_shm_bridge_create() / qcom_scm_shm_bridge_delete()) with
QCOM_SCM_WAITQ_SLEEP. The waitq wake-up interrupt (qcom-scm, GIC ESPI hwirq
4131) never fires, so the caller sleeps in qcom_scm_wait_for_wq_completion()
forever. Because __scm_smc_do() holds qcom_scm_lock across that wait, every
other SCM caller blocks behind it, including the GPU's adreno_hw_init(). The
resulting chain (GPU fence -> DRM commit -> fbcon unblank under console_lock)
leaves userspace unable to open /dev/console, and the boot never reaches a
login prompt.
This is the cause of most login timeouts on these two boards in the meta-qcom
CI (about 5-9% of qcom-distro boots, all boards affected).
Environment
- Kernel: qcom-next
qcom-next-7.3-rc6-20261023(7.3.0-rc6-gc7896917e523) - QTEE version 5.2.0 (
qcomteedriver,qtee_supplicantfrom minkipc) - Kaanapali MTP:
BOOT.MXF.2.5.3-00396-KAANAPALI-1, UEFIBOOT.MXF_UEFI.2.5-00653-KAANAPALI-1 - SM8750 MTP:
BOOT.MXF.2.5.1-00377-PAKALA-2, UEFIBOOT.MXF_UEFI.2.5-00653-PAKALA-1 - Image: meta-qcom master,
qcom-distro,qcom-multimedia-image
(kas build ci/kaanapali-mtp.yml:ci/qcom-distro.yml --target qcom-multimedia-image) - Kernel command line:
root=PARTLABEL=rootfs rw rootwait console=ttyMSM0,115200 qcom_scm.download_mode=1
Reproduction
Flash and boot the image repeatedly. In our LAVA runs, roughly 1 in 5-10 boots
hangs; the console shows Started Serial Getty on ttyMSM0 and then no login
prompt, while kernel messages keep printing. Every hung boot we instrumented had
a scm_waitq_sleep event; none of the good boots did (0 in 12).
Evidence
All of the following was captured on hung boots with a timer that dumps state
to /dev/kmsg 60 s after boot, with the qcom_scm trace events enabled early.
1. The SCM call that sleeps and is never woken
Kaanapali (holder = qcomtee release worker):
kworker/u32:1-13 [004] 10.342581: scm_smc_request: smc_id:0x42000c1e svc_id:0x0c cmd_id:0x1e args_cnt:4 args:{0x8a8845006,0x8a8845006,0x1200,0x3}
kworker/u32:1-13 [004] 10.342657: scm_smc_done: smc_id:0x42000c1e, ret:0 res_to_callee:0x0 res0:0x0 res1:0x1fc73c2d res2:0x0
kworker/u32:1-13 [004] 10.342708: scm_smc_request: smc_id:0x72000602 svc_id:0x06 cmd_id:0x02 args_cnt:4 args:{0x8a8845000,0x1000,0x8c2778000,0x1000}
kworker/u32:1-13 [004] 10.342786: scm_smc_done: smc_id:0x72000602, ret:0 res_to_callee:0x0 res0:0x0 res1:0x0 res2:0x0
kworker/u32:1-13 [004] 10.366143: scm_smc_request: smc_id:0x42000c1d svc_id:0x0c cmd_id:0x1d args_cnt:1 args:{0x1fc73c2d}
kworker/u32:1-13 [004] 10.366206: scm_waitq_sleep: wq_ctx:0, smc_call_ctx:1
kworker/u32:1+qcomtee_wq state=I wchan=qcom_scm_wait_for_wq_completion
qcom_scm_wait_for_wq_completion+0x40/0x80
__scm_smc_do+0x114/0x398
__scm_smc_call+0xc4/0x430
qcom_scm_call+0x54/0x98
qcom_scm_shm_bridge_delete+0x74/0xb0
qcom_tzmem_shm_bridge_delete+0x28/0x40
qcomtee_shm_unregister+0x1c/0x38 [qcomtee]
tee_dyn_shm_free_helper+0x30/0x60
pool_op_free+0x24/0x38 [qcomtee]
tee_shm_put+0x178/0x1c0
tee_shm_free+0x18/0x30
qcomtee_msg_buffers_free+0x24/0x48 [qcomtee]
qcomtee_object_do_invoke_internal+0x578/0x1318 [qcomtee]
qcomtee_do_release_qtee_object+0x70/0x128 [qcomtee]
process_one_work+0x1c0/0x548
SM8750 (holder = qtee_supplicant), with no other SCM call in flight (the
previous one completed 2 s earlier):
kworker/u34:6-151 [006] 12.583775: scm_smc_done: smc_id:0x4200010a, ret:0 ...
qtee_supplicant-701 [006] 14.641494: scm_smc_request: smc_id:0x42000c1e svc_id:0x0c cmd_id:0x1e args_cnt:4 args:{0x8a9fe8006,0x8a9fe8006,0x1200,0x3}
qtee_supplicant-701 [006] 14.641617: scm_waitq_sleep: wq_ctx:0, smc_call_ctx:1
qtee_supplicant state=I wchan=qcom_scm_wait_for_wq_completion
qcom_scm_wait_for_wq_completion+0x40/0x80
__scm_smc_do+0x114/0x398
__scm_smc_call+0xc4/0x430
qcom_scm_call+0x54/0x98
qcom_scm_shm_bridge_create+0x94/0x100
qcom_tzmem_shm_bridge_create+0x58/0xa0
qcomtee_shm_register+0x28/0x40 [qcomtee]
tee_dyn_shm_alloc_helper+0xf8/0x170
pool_op_alloc+0x2c/0x40 [qcomtee]
shm_alloc_helper+0xd4/0x160
tee_shm_alloc_priv_buf+0x24/0x40
qcomtee_msg_buffers_alloc+0x194/0x248 [qcomtee]
qcomtee_object_do_invoke_internal+0x50/0x1318 [qcomtee]
qcomtee_object_do_invoke+0x44/0x60 [qcomtee]
qcomtee_object_invoke+0x2b4/0x6c0 [qcomtee]
tee_ioctl+0xf3c/0x1160
The wake-up interrupt never fires, on any CPU, on any boot we looked at:
43: 0 0 0 0 0 0 0 0 GICv3 4131 Edge qcom-scm
2. Everything else blocks on qcom_scm_lock
task:kworker/u32:4 state:D Workqueue: ring0 drm_sched_run_job_work [gpu_sched]
__mutex_lock.constprop.0+0x508/0xac0
mutex_lock+0x50/0x70
__scm_smc_do+0x78/0x398
__scm_smc_call+0xc4/0x430
qcom_scm_call+0x54/0x98
__qcom_scm_is_call_available+0x8c/0x110
qcom_scm_set_gpu_smmu_aperture_is_available+0x2c/0x40
adreno_hw_init+0xb8/0x108 [msm]
a8xx_hw_init+0xd4c/0x25e0 [msm]
msm_gpu_hw_init+0x60/0xa0 [msm]
msm_gpu_submit+0x4c/0x1b0 [msm]
msm_job_run+0xb4/0x140 [msm]
drm_sched_run_job_work+0x128/0x3e8 [gpu_sched]
qtee_supplicant (Kaanapali case) and wireplumber (SM8750 case) are blocked
the same way in __scm_smc_do+0x78 -> mutex_lock.
3. Knock-on: DRM and console_lock
kworker/u33:1+events_unbound state=D: dma_fence_default_wait <- drm_atomic_helper_wait_for_fences <- commit_tail <- commit_work
(agetty) state=S: drm_atomic_helper_swap_state <- drm_atomic_helper_commit <- drm_client_modeset_commit_atomic
<- __drm_fb_helper_restore_fbdev_mode_unlocked <- drm_fb_helper_set_par <- fb_set_var <- fbcon_blank
<- do_unblank_screen <- vt_ioctl (holds console_lock)
kworker/0:2+events state=D: console_lock <- console_callback
weston state=D: modeset_lock <- drm_helper_probe_single_connector_modes <- drm_mode_getconnector
and on other hung boots the serial getty's systemd executor is stuck opening
the console:
(agetty) state=D wchan=console_device
console_device+0x48/0x100
tty_lookup_driver+0xb0/0x268
tty_open+0x28c/0x6c8
chrdev_open+0xbc/0x258
...
__arm64_sys_openat+0x68/0xc0
Experiment: drop qcom_scm_lock while waiting
To check whether the firmware just needed other SMC calls to make progress, we
tested this change (debug only, not a proposed fix):
--- a/drivers/firmware/qcom/qcom_scm-smc.c
+++ b/drivers/firmware/qcom/qcom_scm-smc.c
@@ static int __scm_smc_do_quirk_handle_waitq(...)
trace_scm_waitq_sleep(wq_ctx, smc_call_ctx);
+ mutex_unlock(&qcom_scm_lock);
ret = qcom_scm_wait_for_wq_completion(dev, wq_ctx);
+ mutex_lock(&qcom_scm_lock);
if (ret)
return ret;Result on Kaanapali: 19 of 19 boots reached the login prompt, and 4 of them
hit a waitq sleep (the same rate as the hang). In all 4:
- the
wq_ctx:0sleeper was still never resumed (scm_waitq_resume= 0,
qcom-scmIRQ count = 0), even though other SMC calls were now able to run; - later SHM bridge calls also got
WAITQ_SLEEP, but withwq_ctx:2147483648
(0x80000000), whichqcom_scm_get_completion()rejects:
kworker/u32:3-406 [007] 10.685471: scm_waitq_sleep: wq_ctx:0, smc_call_ctx:1
qtee_supplicant-785 [007] 10.713989: scm_waitq_sleep: wq_ctx:2147483648, smc_call_ctx:2
kworker/u32:1-13 [005] 12.405981: scm_waitq_sleep: wq_ctx:2147483648, smc_call_ctx:3
kworker/u32:2-330 [000] 21.487378: scm_waitq_sleep: wq_ctx:2147483648, smc_call_ctx:4
WARNING: drivers/firmware/qcom/qcom_scm.c:2699 at qcom_scm_wait_for_wq_completion+0x50/0x80, CPU#7: qtee_supplicant/785
CPU: 7 UID: 0 PID: 785 Comm: qtee_supplicant Not tainted 7.3.0-rc6-gc7896917e523-p1 #1 PREEMPT
Hardware name: Qualcomm Technologies, Inc. Kaanapali MTP (DT)
pc : qcom_scm_wait_for_wq_completion+0x50/0x80
Call trace:
qcom_scm_wait_for_wq_completion+0x50/0x80 (P)
qcom_scm_call+0x54/0x98
qcom_scm_shm_bridge_delete+0x74/0xb0
qcom_tzmem_shm_bridge_delete+0x28/0x40
qcomtee_shm_unregister+0x1c/0x38 [qcomtee]
tee_dyn_shm_free_helper+0x30/0x60
pool_op_free+0x24/0x38 [qcomtee]
tee_shm_put+0x178/0x1c0
tee_shm_fop_release+0x1c/0x40
__fput+0xdc/0x310
...
So releasing the lock removes the system-wide hang, but the underlying problem
remains: the waitq wake-up never reaches Linux on these SoCs.
Questions
- Is the waitq wake-up interrupt (hwirq 4131 from
WAITQ_GET_INFO) expected
to be delivered to HLOS on Kaanapali and SM8750 with this TZ/QTEE firmware?
It never fires on any boot we captured. - What does a
wq_ctxof0x80000000mean in theWAITQ_SLEEPresponse?
It looks like a flag or encoding that the driver doesn't handle, rather than
an index belowwq_cnt. - Should
__scm_smc_do()holdqcom_scm_lockacross
qcom_scm_wait_for_wq_completion()? With the current code, any waitq sleep
that is not woken blocks every SCM user in the system.
Activity