Repository navigation
Hive 30249 - #6961
Open
illiabarbashov-sketch wants to merge 2 commits into
Open
Hive 30249#6961illiabarbashov-sketch wants to merge 2 commits into
illiabarbashov-sketch wants to merge 2 commits into
Conversation
- Hive.getTableConstraints moved to the combined getAllTableConstraints call, which the query-level HMS cache does not cover (it caches the per-type constraint calls); every Table copy made during compilation fetched again - Impact (local, TPC-DS 30 TB perf driver, 201 statements): 1,866 -> 992 calls (the distinct tables), compile 350.3 -> 283.8 s (-19 %); one statement spent 2.0 s of 2.0 s Calcite plan generation in 26 calls at ~80 ms - cluster (TPC-DS 10TB Parquet, 24 x r6id, 198 query-iterations): warm compile 386.8 -> 383.6 s, no wall gain against the cluster HMS - the cache hands out a copy: Hive.getTableConstraints keeps only the RELY constraints in the object it gets - test: one call per query; the copy (mutation: the cached object itself) keeps the second result whole
Cut the HMS calls made while compiling a query: - Do not fetch constraints for materialized view scans in the registry; materialized views cannot have constraints. - Share the getTable entry with the write id list lookup: getValidWriteIdList sent a request without getColumnStats/engine, so the same table took a second HMS call. The TABLE cache key is now a copy of the caller's request. - Cache getCheckConstraints and getDefaultConstraints, fetched once per MERGE branch and per insert destination. - Cache getPartitionWithAuthInfo for the static partition of an insert target, handing out a copy as ql Partition changes the object it gets; a missing partition is not cached. - Remember tables missing from the metastore (CTE names, optional lookups) within a compilation; createTable drops the entry. - Pass the Table to getPartitionColumnStatistics instead of fetching it again. - HiveMetaStoreClientWithLocalCache takes the table for its watermark and transactional check from the query-level cache instead of two getTable RPCs per statistics/partition call. TPC-DS 30TB perf driver (201 statements), on top of the getAllTableConstraints cache: HMS RPCs 5956 -> 3148 (master: 6830), median summed compile time 180.2 s (master) -> 147.9 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
illiabarbashov-sketch
force-pushed
the
HIVE-30249
branch
from
November 13, 2026 14:36
80d954a to
49779ec
Compare
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR reduces the HMS calls made while compiling a query. Most of them are lookups of the same metadata repeated within one compilation, which now come from the query-level HMS cache
Why are the changes needed?
Compilation goes back to HMS for the same metadata many times: once per table alias, per getMetaData pass (before CBO, after CBO, per subquery, in the masking rewrite), and per statistics or partition call. With a remote HMS each of these is a network round trip, and that adds directly to compile time.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
TestSessionHiveMetaStoreClientQueryCache, built on a mocked metastore client.TestMiniLlapLocalCliDriverpasses withcheck_constraint.q, constraints_optimization.q, cte_mat_1.q, default_constraint.q, insert_overwrite.q, materialized_view_rewrite_1.qandsqlmerge_stats.q.TestTezTPCDS30TBPerfCliDrivertests pass with no .q.out changes, so plans are identical.