Skip to content

fix: keep topic messages published before the first receptionist listing - #3762

Open
pjfanning wants to merge 2 commits into
apache:mainfrom
pjfanning:topic-buffer-before-listing
Open

pjfanning wants to merge 2 commits into
apache:mainfrom
pjfanning:topic-buffer-before-listing

Conversation

@pjfanning

@pjfanning pjfanning commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Motivation

TopicImpl only learns the other topic instances from the receptionist. A message published to a freshly created topic is processed before the first Listing arrives, so it never reaches subscribers on other instances, and goes to dead letters when there are no local subscribers. This happens every time a topic is created and published to straight away, e.g. pekko-persistence-r2dbc's journal.publish-events.

With the cluster receptionist, the first listing only contains what the node has seen so far, which on a newly started node is often nothing. So keeping messages only until the first listing is not enough.

Modification

  • A new topic actor keeps the messages published until it has seen its first listing and pekko.actor.typed.pubsub.startup-replay-window (default 3s) has elapsed since it started.
  • During that window every other topic instance and local subscriber that becomes known gets each kept message once. That includes instances learnt from later listings, and subscribers that subscribe during the window.
  • At most pekko.actor.typed.pubsub.startup-replay-buffer-size (default 1000) messages are kept. Messages past that are still delivered to the instances and subscribers known at the time. They are only sent to dead letters as Dropped if they reached nobody.
  • Kept messages that reached nobody by the end of the window are sent to dead letters as Dropped, as before.
  • Local subscribers are now always delivered to directly, and MessagePublished goes only to the other instances. This merges the two Publish cases and fixes a gap where local subscribers missed messages when the listing didn't yet include the own instance.
  • Documented in distributed-pub-sub.md and reference.conf.

Result

The first messages published to a new topic reach the subscribers on instances discovered during the startup window, including with the cluster receptionist, without duplicates.

Tests

  • New TopicImplSpec uses BehaviorTestKit to drive receptionist listings and the window timer deterministically. It covers forwarding at the first listing, exactly-once local delivery, replay to instances and subscribers discovered later, waiting for the first listing, and buffer overflow. Real-system tests cover the Dropped cases.
  • sbt "actor-typed-tests/testOnly org.apache.pekko.actor.typed.internal.pubsub.TopicImplSpec org.apache.pekko.actor.typed.pubsub.LocalPubSubSpec": 13 passed
  • sbt "actor-typed/mimaReportBinaryIssues": passed
  • Multi-JVM cluster-typed PubSubSpec not run locally; left to CI

References

Fixes #3761

@pjfanning pjfanning added this to the 2.1.0 milestone Nov 8, 2026
@pjfanning
pjfanning marked this pull request as ready for review November 13, 2026 10:10
Motivation:
TopicImpl only learns the other topic instances from its first receptionist
Listing. A message published to a freshly created topic is processed before
that reply arrives, so it never reached subscribers on other instances and
went to dead letters when there were no local subscribers.

Modification:
Until the first TopicInstancesUpdated, deliver published messages to local
subscribers right away and keep up to 1000 of them. When the first listing
arrives, forward them to the other topic instances (excluding itself, whose
local subscribers already got them), or drop them as Dropped if they reached
nobody. Overflow is dropped as Dropped. After that the behaviour is unchanged.

Result:
The first messages published to a new topic reach subscribers already known
to the receptionist.

Tests:
- sbt "actor-typed-tests/testOnly org.apache.pekko.actor.typed.pubsub.LocalPubSubSpec" - 7 passed
- New test "deliver messages published to a new topic instance before its first listing" fails without the TopicImpl change

References:
Fixes apache#3761
…ndow

Motivation:
Flushing the kept messages at the first receptionist listing only helps
with the local receptionist. The cluster receptionist's first listing
only has what the node has seen so far, often nothing on a new node, so
the messages were still lost. The tests also didn't reliably exercise
the new code.

Modification:
- Keep published messages until the first listing has been seen and
  pekko.actor.typed.pubsub.startup-replay-window (default 3s) has
  elapsed. Each topic instance and local subscriber that becomes known
  in that time gets every kept message once.
- Buffer size is pekko.actor.typed.pubsub.startup-replay-buffer-size
  (default 1000). Messages past it that reached a subscriber are no
  longer reported as Dropped.
- Deliver to local subscribers directly and send MessagePublished only
  to the other instances, which merges the two Publish cases and
  removes the stale "listing not seen yet" path.
- Replace the timing-dependent LocalPubSubSpec tests with deterministic
  BehaviorTestKit tests in TopicImplSpec.

Result:
Messages published right after a topic starts reach instances learnt
from later listings, which covers the cluster receptionist case.

Tests:
- sbt "actor-typed-tests/testOnly ...TopicImplSpec ...LocalPubSubSpec": 13 passed

References:
Refs apache#3761
@pjfanning
pjfanning force-pushed the topic-buffer-before-listing branch from a9f0203 to ba21a69 Compare November 13, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Topic drops messages published before its first receptionist listing

1 participant