Repository navigation
River job stuck at running #1258
Description
We recently noticed that a lot of our river job(around 5000) is stuck at a state of running. We have a global timeout of 1 hour and thought that the rescue is suppose to resolve this? These are across different job queues.
River job stats
SELECT
COUNT(*) AS stuck_count,
COUNT(DISTINCT kind) AS distinct_kinds,
COUNT(DISTINCT queue) AS distinct_queues,
MIN(created_at) AS oldest_created_at,
MAX(created_at) AS newest_created_at,
NOW() - MIN(created_at) AS oldest_age,
NOW() - MAX(created_at) AS newest_age,
AVG(NOW() - created_at) AS avg_age,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM NOW() - created_at)) * INTERVAL '1 second' AS median_age,
MIN(attempt) AS min_attempts,
MAX(attempt) AS max_attempts,
AVG(attempt)::numeric(10,2) AS avg_attempts,
COUNT(*) FILTER (WHERE attempted_at < NOW() - INTERVAL '1 hour') AS not_touched_in_1h,
COUNT(*) FILTER (WHERE attempted_at < NOW() - INTERVAL '6 hours') AS not_touched_in_6h
FROM river_jobs.river_job
WHERE state = 'running'
AND created_at < NOW() - INTERVAL '1 day';
[
{
"stuck_count": 5133,
"distinct_kinds": 42,
"distinct_queues": 7,
"oldest_created_at": "2026-02-24 22:08:09.618834+00",
"newest_created_at": "2026-05-25 19:10:35.474235+00",
"oldest_age": "91 days 00:29:38.338464",
"newest_age": "1 day 03:27:12.483063",
"avg_age": "40 days 29:02:53.066961",
"median_age": "1155:35:03.048939",
"min_attempts": 1,
"max_attempts": 4,
"avg_attempts": 1.08,
"not_touched_in_1h": 5133,
"not_touched_in_6h": 5133
}
]
River version
github.com/riverqueue/river v0.31.0
github.com/riverqueue/river/rivertype v0.31.0
Let me know if any other information is helpful
Activity
brandur commentedon May 27, 2026
These are likely what we refer to as "stuck" jobs, which are jobs that don't respond to context timeout. This can occur when, for example, you have a wait on something like a channel in a job, but without also waiting on
ctx.Done()in aselect.If you look for
num_jobs_stuckin your logs, you should be able to see this number (requires info-level logging).peter941221 commentedon Jun 1, 2026
Opened a docs follow-up here in case it's useful: #1264
I kept it scoped to the cancellation semantics that came up in this issue:
Worker.Workcontract to call out blocking on channels, timers, and network operations explicitly.num_jobs_stuckas the first diagnostic when jobs appear stuck inrunning.ctx.Done()expectation shows up in an example that's already about stop behavior.No runtime behavior changes in this one, just trying to make the existing semantics easier to discover.
didntpay commentedon Jun 12, 2026
I see, thanks team. I still have one confusion, so when we cancel the worker's context in hard stop mode, does that also set the job to cancelled? Are we saying that because we have a timer, channel, etc that does not respect the ctx.Done() from worker, in hard stop mode we are unable to set it to cancelled?
brandur commentedon Jun 13, 2026
@didntpay Yeah, currently the idea is that you should very much try to have your jobs behave well and respond to context cancellation. A feature to hard error any jobs that didn't respond to context cancellation might be possible, but it'd complicate the stop process in ways that might not be super desirable.
didntpay commentedon Jun 22, 2026
Make sense, in the case that our job does not respond to the ctx.Done(), would the rescuer eventually come in and clean those up? Trying to see what the safest strategy is to clean up our 5k stuck job currently in the db.
brandur commentedon Jun 23, 2026
Yeah, the rescuer will clean those up after
RescueStuckJobsAfterhas elapsed (default 1 hour). I also have a patch open in #1289 that would try to clean these up before River stops.didntpay commentedon Jul 13, 2026
Ah, I think one of our problem is also the rescuer not cleaning those job after 1 or 2 hour. We did not specify a special timeout for those.