Skip to content

AVRO-3803: Python patched up ipc issues - #2319

Closed
asosnovsky wants to merge 8 commits into
apache:mainfrom
asosnovsky:master
Closed

asosnovsky wants to merge 8 commits into
apache:mainfrom
asosnovsky:master

Conversation

@asosnovsky

@asosnovsky asosnovsky commented Jul 5, 2023 •

Copy link
Copy Markdown

AVRO-3803

What is the purpose of the change

  • Patched issue where FramedReader resulted in an infinite loop where it cannot read anything from the response (it was comparing bytes to a string)
  • Patched error handling for the Responder:
    • Invalid error source: except was pointing at schema.AvroException instead of avro.errors.AvroException
    • "Handshake" was lost for cases where the response could not be read due to things like "type errors"
  • Added some types to responder class

Verifying this change

This change added tests and can be verified as follows:

  • New test cases added
  • Type checker for Responder works (while we have tested it in our repo, I am honestly a bit confused on how to setup the local server with this code base)

Documentation

  • Does this pull request introduce a new feature? NO

@github-actions github-actions Bot added the Python label Jul 5, 2023
Comment thread lang/py/avro/schema.py Resolved
Comment thread lang/py/avro/schema.py Resolved
Comment thread lang/py/avro/ipc.py Resolved
asosnovsky and others added 3 commits July 18, 2023 13:18
Co-authored-by: Michael A. Smith <michael@smith-li.com>
Co-authored-by: Michael A. Smith <michael@smith-li.com>
@asosnovsky
asosnovsky requested a review from kojiromike July 18, 2023 17:20
Comment thread lang/py/avro/test/test_ipc.py Resolved
@asosnovsky
asosnovsky requested a review from kojiromike July 18, 2023 20:49
@kojiromike

Copy link
Copy Markdown
Contributor

I think this is good, but can we attach it to any existing tickets? If not, I can open a new one.

@asosnovsky

Copy link
Copy Markdown
Author

@kojiromike no I don't have a JIRA for this. Please open one :) thank you!

@kojiromike kojiromike changed the title AVRO-0: Python patched up ipc issues AVRO-3803: Python patched up ipc issues Jul 19, 2023
@kojiromike

Copy link
Copy Markdown
Contributor

@asosnovsky could I trouble you to check if this implementation resolves AVRO-1639?

@kojiromike

Copy link
Copy Markdown
Contributor

As far as resolving the conflicts: If you take your branch as-is and run autoflake on it, that should be sufficient.

@kojiromike

Copy link
Copy Markdown
Contributor

@asosnovsky, thanks for this. I am closing it as superseded by #4079, which carries the two remaining fixes forward under AVRO-3803 and credits this PR. The branch has conflicts and no pushes since July 2023, so there is no need to rebase it.

The bytes-vs-str compare in FramedReader landed separately in #3710 in May 2026. The schema.AvroException to avro.errors.AvroException fix is also already on main.

The restructured respond() drops the remote_protocol is None check. A client's first request normally carries only its hash, so process_handshake returns match NONE with no remote protocol. The PR then calls None.messages and raises AttributeError out of respond() instead of returning the handshake. That breaks the standard first round trip.

Two things in this PR are still real bugs on main and worth keeping. The outer except in respond() writes the system error to a fresh BytesIO that is thrown away. It then returns the old buffer, so the client never sees the error. RecordSchema also only sets the default namespace for "record" and not "error".

kojiromike added a commit to kojiromike/avro that referenced this pull request Nov 6, 2026
This fixes the two avro.ipc and schema bugs from AVRO-3803 that are
still on main. apache#2319 found them but has conflicts and has gone stale,
and the bytes compare part already landed in apache#3710. This is based on
apache#2319 by Ari Sosnovsky.

In Responder.respond(), when an AvroException is raised after the
handshake for an unreadable request, an unknown message, or a response
that does not match its schema, the error was written to a throwaway
BytesIO and the stale buffer returned, so the requestor got a decode
error (InvalidAvroBinaryEncoding) instead of the server's error. Now it
keeps the handshake response and replaces whatever followed it with the
system error. Also, RecordSchema set the default namespace for nested
types only when the type was "record", so types defined inside an
"error" lost the error's namespace.

The new tests are in-memory loopback requestor and responder tests for a
full round trip including the first-contact handshake, for a system
error reaching the requestor, and for a response that does not match its
schema reaching the requestor as an error. They also include the
empty-reader FramedReader test from apache#2319 and a schema test for error
namespaces. The bug tests fail without the fix.

Co-authored-by: Ari Sosnovsky <asosnovsky@users.noreply.github.com>
Generated-by: Claude Code 2.1.314 (Claude Opus 5.5)
@kojiromike kojiromike closed this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants