HIVE-29591: Create Hive precommit image and move Dockerfile into the hive repository by abstractdog · Pull Request #6953 · apache/hive

abstractdog · 2026-11-09T13:22:10Z

What changes were proposed in this pull request?

A precommit image in dev-support/docker/ci/runner. It is based on Debian trixie-slim and contains:
- a Temurin 21 JDK, without jmods and man pages
- Maven
- the tools the precommit stages and tests use: the docker CLI, git, openssl, python3, rsync, tini and xmlstarlet
A GitHub Actions workflow, precommit-image.yml. On PRs that change the Dockerfile, it builds the image. On master it publishes the image for amd64 and arm64 as apache/hive-ci-runner:<version>, where
<version> is the org.opencontainers.image.version label of the Dockerfile. A published the workflow fails if the tag already exists. A manual run can publish a candidate under atag of its own.
The Jenkinsfile runs in apache/hive-ci-runner:1 instead of `ayushtkn/hive-dev-box:e
- The tag is pinned; a new image is rolled out by a follow-up commit that bumps it.
- The dev-box-specific lines are gone (sw java, /etc/profile.d/*.sh, `git config extr
- The Maven options are now passed through MAVEN_ARGS (Maven 3.9+), in place of the dev-box mvn wrapper.
- A new IMAGE build parameter lets a run use a candidate image.
The init@derby and init@postgres stages are removed. DbInstallBase already covers schematool on more databases, and InformationSchemaWithPrivilegeTestBase covers the SYS/INFORMATION_SCHEMA
initialization. This also keeps Hadoop out of the image.
cachingprintstream.q and stack_trace.q normalize the exec failure message with qt:replace:
- Recent JDK updates report a failed exec as `Exec failed, error: 2 (No such file or dire
- Older ones report error=2, No such file or directory.
This line is the only difference in the output, so the tests still check the same failure.
A README on how to change, try and roll out the image.
The docker and helm action pins in docker-images.yml and hive-postgres-tpcds-metastore.yml are bumped to the versions precommit-image.yml uses. Some of the old pins were close to their expiry dates
in the ASF allowlist.

Why are the changes needed?

The precommit image used to be built from a separate repository (hive-dev-box) and pushed to a personal Docker Hub account. Keeping the Dockerfile in the Hive repository and publishing it from CI makes the
image reviewable and reproducible, and removes the dependency on a personal account. See HI

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Locally, in the image with a docker:dind sidecar:

mvn install -DskipTests -Pitests,dist succeeds.
These qtests pass:
- newline.q (python TRANSFORM) on TestMiniLlapCliDriver
- jdbc_project_pushdown.q (postgres in docker) on TestMiniLlapLocalCliDriver
- stack_trace.q on TestNegativeLlapCliDriver
- cachingprintstream.q on TestNegativeLlapLocalCliDriver
The kudu-handler tests pass on amd64 (18 tests). They are skipped on aarch64, as before.

A full precommit run with the candidate image (IMAGE=abstractdog/hive-precommit:HIVE-29591-v3):
https://ci.hive.apache.org/job/hive-precommit/job/PR-6953/12/

zabetak

LGTM, just a few small questions and some suggestions but nothing really blocking.

zabetak · 2026-11-11T15:25:55Z

+          dev-support/docker/precommit/smoke-test.sh hive-precommit:test
+
+      - name: Login to Docker Hub
+        if: github.event_name != 'pull_request' && github.repository == 'apache/hive'


Do we want to restrict login/publish to apache/hive repository? Maybe a contributor wants to test the action on their fork would that be a problem? Were you able to test these parts of the action somehow?

I'd keep the github.repository == 'apache/hive' guard on login and publish: publishing only makes sense for apache/hive-ci-runner, and a fork has neither the DOCKERHUB_* secrets nor push rights to it. A fork still builds the image on its own PRs and manual runs, and with the job-level condition I just added, a push to a fork's master (i.e. syncing the fork) doesn't run the workflow at all. A contributor who wants to try an image doesn't need the workflow: they push it to their own namespace with docker buildx build --push and run precommit with IMAGE=... (README, step 3).

Regarding how I tested: I was able to push the feature branch to the origin, so I can do it, I'll let you know as the docker namespace is already there: https://hub.docker.com/r/apache/hive-ci-runner

I can set the DOCKERHUB_* secrets on my own fork (e.g., https://github.com/zabetak/hive) and the workflow could theoretically push to https://hub.docker.com/u/zabetak as it happens on .github/workflows/hive-postgres-tpcds-metastore.yml. Anyways its not a big deal I am fine even if we don't allow runs from forks. Just wanted to mention that is feasible and not problematic if we don't point to the apache namespace.

action that published an image with a testing tag:
https://github.com/apache/hive/actions/runs/43235096035

image link

action example that failed because tag wasn't overridden, so it won't override the image:
https://github.com/apache/hive/actions/runs/43375951727

apache/hive-ci-runner:HIVE-29591-test is already published, and a published tag is never overwritten. Bump org.opencontainers.image.version in the Dockerfile (or pick another tag for a manual run).

action example, where the Dockerfile hasn't changed, so there is no reason to push:
https://github.com/apache/hive/actions/runs/43376396112/job/131726682743
Check the tag job says:

Notice: apache/hive-ci-runner:HIVE-29591-test is already published and the Dockerfile did not change: not publishing it again.

a last commit is pushed that introduced the change:
410a568

so a patch that doesn't change the Dockerfile doesn't make the workflow push a new image

let me know @zabetak , if your +1 still holds

current precommit run:
https://ci.hive.apache.org/job/hive-precommit/job/PR-6953/12/

zabetak

LGTM, pending tests!

zabetak · 2026-11-12T17:34:40Z

+          if docker manifest inspect "$image" > /dev/null 2>&1; then
+            echo "::error::$image is already published, and a published tag is never overwritten. Bump org.opencontainers.image.version in the Dockerfile (or pick another tag for a manual run)."
+            exit 1
+          fi


We can disable tag overwrite directly in DockerHub if we want to.

…hive repository

- Move the image to dev-support/docker/ci/runner. - Drop the init@<db> stages and precommit-init-metastore.sh (DbInstallBase already covers schematool), and with them Hadoop from the image. - Drop smoke-test.sh. - Version the image with its org.opencontainers.image.version label: publish apache/hive-ci-runner:<version>, never overwrite a published tag, and pin the tag in the Jenkinsfile. - Trim the README to the steps people actually follow. Co-Authored-By: Claude Code <noreply@anthropic.com>

docker 20.10 cannot extract the current mcr.microsoft.com/mssql/server:2019-latest layers ("archive/tar: invalid tar header", microsoft/mssql-docker#978), which fails TestSchemaToolForMetastore[mssql].

A push to a fork's master only syncs apache/hive, and the tag check fails there once apache/hive has published the version. Forks still build the image on their PRs and manual runs. Co-Authored-By: Claude Code <noreply@anthropic.com>

… does not need a new version When the tag is published already, the workflow fails only if the Dockerfile changed (against the target branch of a PR, or the previous head of a push); otherwise it builds the image but skips publishing it. A manual run with an existing tag still fails.

sonarqubecloud · 2026-11-13T10:23:33Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.6% Duplication on New Code

See analysis details on SonarQube Cloud

zabetak · 2026-11-13T12:42:16Z

+      # A published tag is never overwritten. When the tag is published already, a change of the
+      # Dockerfile needs a new version; any other change (e.g. of this workflow) does not publish.
+      - name: Check the tag


Still +1 on the PR but all this tag management check and publish seems overly complicated. We could adopt a simple timestamp based tagging (eg. 20261113133625) and get done with it. This way even in the workflow dispatch we can remove the inputs and also users and future maintainers do not have to manually bump the version every time there is a change in the Dockerfile.

I agree it's overcomplicated; we shall iterate one more time before merging
what tag do you think the Jenkinsfile should contain: latest or pin to a specific timestamp?

"do not have to manually bump the version every time" makes me think that the Jenkinsfile should always contain latest, in this case 1 PR per change is enough, latest will point to a certain timestamp tag

2 PRs per change: a) first one is approved, merged, b) github action deploys an image tag 20261113133625-commithash c) follow-up PR bumps Jenkinsfile to the newer

does it make sense?

abstractdog changed the title ~~HIVE-29591: Create Hive precommit image and move Dockerfile into the …~~ HIVE-29591: Create Hive precommit image and move Dockerfile into the hive repository Nov 9, 2026

asf-ci-hive added tests pending tests failed and removed tests pending tests failed labels Nov 9, 2026

abstractdog force-pushed the HIVE-29591-precommit-image branch from 46def98 to 101acd8 Compare November 9, 2026 13:59

asf-ci-hive added tests failed tests pending tests unstable and removed tests pending tests failed labels Nov 9, 2026

abstractdog force-pushed the HIVE-29591-precommit-image branch from 101acd8 to 84fa9dc Compare November 10, 2026 10:02

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels Nov 10, 2026

abstractdog force-pushed the HIVE-29591-precommit-image branch from 84fa9dc to 698ecdd Compare November 11, 2026 11:11

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending tests failed labels Nov 11, 2026

abstractdog requested a review from zabetak November 11, 2026 11:21

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Nov 11, 2026

zabetak reviewed Nov 11, 2026

View reviewed changes

asf-ci-hive added tests failed tests pending tests unstable and removed tests pending tests failed tests unstable labels Nov 11, 2026

zabetak approved these changes Nov 12, 2026

View reviewed changes

abstractdog and others added 7 commits November 13, 2026 07:22

HIVE-29591: Create Hive precommit image and move Dockerfile into the …

63520c9

…hive repository

inline versions in github actions

cdf02c8

apache/hive-ci-runner

866e5e7

Run the dind sidecar on docker 29.9

d57691a

docker 20.10 cannot extract the current mcr.microsoft.com/mssql/server:2019-latest layers ("archive/tar: invalid tar header", microsoft/mssql-docker#978), which fails TestSchemaToolForMetastore[mssql].

asf-ci-hive added tests failed and removed tests pending labels Nov 13, 2026

abstractdog force-pushed the HIVE-29591-precommit-image branch from 22e9acc to 410a568 Compare November 13, 2026 06:22

asf-ci-hive added tests pending and removed tests failed labels Nov 13, 2026

asf-ci-hive added tests passed and removed tests pending labels Nov 13, 2026

zabetak approved these changes Nov 13, 2026

View reviewed changes

Conversation

abstractdog commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

zabetak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zabetak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Nov 13, 2026

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

abstractdog commented Nov 9, 2026 •

edited

Loading

abstractdog Nov 13, 2026 •

edited

Loading

abstractdog Nov 13, 2026 •

edited

Loading