Repository navigation
Conversation
In DatumReader._read_default_value, resolving a RecordSchema default
used default_value.get(field.name) and replaced any None result with
field.default. That silently overwrote explicit JSON null entries in
nested record defaults (e.g. {"x": null}) whenever the inner field
defined a non-null default, and fabricated None when a required inner
field was omitted.
In addition, UnionSchema defaults only tried schemas[0] instead of the
first matching branch required by the Avro Specification ("Default
values for union fields correspond to the first schema that matches in
the union"), which also prevented explicit null or non-first-branch
values in record and union defaults from resolving.
Check key membership and field.has_default explicitly for RecordSchema
defaults, and resolve UnionSchema defaults against the first matching
branch with strict JSON type checks on primitive, enum, and array
branches.
Assisted-by: Google Antigravity
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Fix two coupled specification-compliance bugs in
DatumReader._read_default_value(lang/py/avro/io.py) during schema evolution:RecordSchemaexplicitnulldefaults:_read_default_valuepreviously evaluatedjson_val = default_value.get(field.name); if json_val is None: json_val = field.default. When an outer record'sdefaultJSON object explicitly set an inner nullable field tonull(e.g.,"default": {"x": null}where inner fieldxhas"type": ["int", "null"], "default": 99),json_val is Noneconflated the explicitnullwith an omitted key and silently overwroteNonewith99. Conversely, omitting a required inner field (has_default == False) whose type started with"null"silently fabricatedNoneinstead of raisingInvalidDefaultException.UnionSchemafirst-matching-branch resolution: Per the Apache Avro Specification (Schema Declaration — Records), "Default values for union fields correspond to the first schema that matches in the union."_read_default_valuepreviously hardcodedfield_schema.schemas[0], rejecting valid defaults matching subsequent union branches (includingnullin["int", "null"]). Resolving against the first matching branch also requires strict JSON type guards onboolean,int/long(not isinstance(..., bool)and 32/64-bit bounds),EnumSchemasymbol membership, andArraySchema(not isinstance(..., (str, bytes, Mapping))) so earlier branches do not falsely coerce incompatible JSON defaults.No JIRA issue: the ASF Jira tracker requires an account.
Verifying this change
This change added tests and can be verified as follows:
TestDefaultValueResolutioninlang/py/avro/test/test_io.pycovering explicitnullin nested record defaults ({"x": None}vs{}), omitted inner fields withdefault: null, rejection of omitted required inner fields, union defaults matching non-first primitive/enum/array/record branches, and rejection of unmatched union defaults.uv run --frozen ruff format --diff .,uv run --frozen ruff check .,uv run --frozen mypy avro/, anduv run --frozen python -m unittest avro.test.test_io(196tests passing).Documentation
Assisted by Google Antigravity.