Skip to content

Added logic to UpgradeUtil to check for tablet recoveries - #6623

Open
dlmarion wants to merge 9 commits into
apache:2.1from
dlmarion:upgrade-prepare-recovery-check
Open

dlmarion wants to merge 9 commits into
apache:2.1from
dlmarion:upgrade-prepare-recovery-check

Conversation

@dlmarion

Copy link
Copy Markdown
Contributor

Modified UpgradeUtil to check root, metadata, and user tablet metadata for the existence of write-ahead logs when the user runs the '--prepare' command. If logs are found the user is advised to abort the upgrade and restart the instance to allow tablet recovery.

Assisted-By: OpenAI GPT-6 Luna

Modified UpgradeUtil to check root, metadata, and user
tablet metadata for the existence of write-ahead logs
when the user runs the '--prepare' command. If logs
are found the user is advised to abort the upgrade and
restart the instance to allow tablet recovery.

Assisted-By: OpenAI GPT-6 Luna
@dlmarion dlmarion added this to the 2.1.7 milestone Oct 30, 2026
@dlmarion dlmarion self-assigned this Oct 30, 2026
.collect(Collectors.toList());
Iterator<TabletMetadata> iter =
readTabletMetadata(rootFiles, MetadataSchema.TabletsSection.getRange(MetadataTable.ID));
List<String> metadataFiles = new ArrayList<>();

@keith-turner keith-turner Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting all the files from different tablets together is not correct and could lead to reading data from a split that is stale and outside a tablets range. The code could be greatly simplified by using the data range on the extent. Tried writing that to see what this would look like and came up with the following. This goes down the tree restricting the range on each child tablet. Found that need to setup the versioning iterator also. Also need to look for tablets with a location or write ahead logs, the manager eventually assigns walogs to tablets assigned to a dead tserver and removes the location. With no live tservers any tablet w/ a location is assigned to a dead tserver.

  private Iterator<TabletMetadata> readChildTablets(TabletMetadata tabletMetadata) {
    var extent = tabletMetadata.getExtent();
    Preconditions.checkState(extent.isMeta());
    String[] files = tabletMetadata.getFiles().stream().map(stf -> stf.getPath().toString())
        .toArray(String[]::new);
    // RFileScanner sets up the system iterator stack by default
    Scanner rfileScanner = RFile.newScanner().from(files)
        .withTableProperties(IteratorConfigUtil.getInitialTableIterators()).build();
    var range = extent.toDataRange().clip(MetadataSchema.TabletsSection.getRange(), true);
    if (range == null) {
      return Collections.emptyIterator();
    }
    rfileScanner.setRange(range);
    RowIterator rowIter = new RowIterator(rfileScanner);
    return Iterators.transform(rowIter,
        ri -> TabletMetadata.convertRow(ri, EnumSet.allOf(TabletMetadata.ColumnType.class), false));
  }

  private void checkTablet(TabletMetadata tm) {
    // TODO throw exception if either is true
    System.out.println(tm.getExtent() + " hasLoc:" + (tm.getLocation() != null) + " hasWals:"
        + (!tm.getLogs().isEmpty()));
  }

  private void checkForDirtyTablets(ClientContext ctx) {
    var rootMeta = TabletsMetadata.getRootMetadata(ctx.getZooKeeperRoot(), ctx.getZooCache());
    checkTablet(rootMeta);
    readChildTablets(rootMeta).forEachRemaining(tm->{
      checkTablet(tm);
      readChildTablets(tm).forEachRemaining(this::checkTablet);
    });
  }

  @Test
  public void test2() throws Exception {
    try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
      client.tableOperations().create("table1");
      client.tableOperations().addSplits("table1",
          new TreeSet<>(List.of(new Text("c"), new Text("m"))));
      client.tableOperations().create("table2");
      client.tableOperations().addSplits("table2",
          new TreeSet<>(List.of(new Text("3"), new Text("5"))));
      client.tableOperations().create("table3");
      client.tableOperations().addSplits("table3",
          new TreeSet<>(List.of(new Text("30"), new Text("90"))));

      client.tableOperations().addSplits("accumulo.metadata",
          new TreeSet<>(List.of(new Text("2<"))));

      client.tableOperations().flush("accumulo.metadata", null, null, true);
      client.tableOperations().flush("accumulo.root", null, null, true);

      UtilWaitThread.sleep(1000);

      checkForDirtyTablets((ClientContext) client);
    }
  }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have the logic correct now

do {
Thread.sleep(zkTimeout);
tserversWithLocks = 0;
for (String zPath : tserverNodes) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be able to shorten this code by using live tserver set.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestion, that reduced the amount of code.

TabletMetadata metadataTablet = iter.next();
metadataTablet.getFiles().stream().map(stf -> stf.getPathStr()).forEach(metadataFiles::add);
if (!metadataTablet.getLogs().isEmpty()) {
extentsNeedingRecovery.add(metadataTablet.getExtent());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once a dirty extent is found should abort, should not read its children as their data will be stale and not include the data in walogs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated the logic to implement this change.

try {
findTabletsNeedingRecovery(Constants.ZROOT + "/" + iid, zc);
} catch (IllegalStateException e) {
LOG.error(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if zUpgradepath should be removed from ZK here and also if we should not print the message later about being ready for upgrade. Would also be nice to have a non zero exit code, that could be done by throwing an exception.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing zUpgradePath here would turn this pre-upgrade tablet recovery check from a warning to an error condition. The code currently just warns, but allows the user to continue. Do you think that we should not allow an upgrade to occur if tablets need recovery? If so, there might be an issue enforcing it as upgrade --prepare with the old version of code is optional because the user can run upgrade start --force with the new version of code.

error = true;
}
if (error) {
LOG.warn("Continuing with the upgrade is not advisable. It's recommended to abort this "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
LOG.warn("Continuing with the upgrade is not advisable. It's recommended to abort this "
LOG.warn("Because some tablets have data in write ahead logs, continuing with the upgrade is not advisable. It's recommended to abort this "

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 039e8bf


createPrepareMarker(ctx);

stopManagerAndWait(ctx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test passes, however its not really succeeding because there are walogs or locations the way the cluster is shutdown. Maybe the following will do a clean shutdown. Also the test should fail if recovery is needed.

Suggested change
stopManagerAndWait(ctx);
cluster.getClusterControl().adminStopAll();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested replacement worked, but needed to wait for the manager lock to be removed in ZooKeeper. Fixed in 039e8bf.

System.setProperty("accumulo.properties", "file://" + getCluster().getAccumuloPropertiesPath());
new UpgradeUtil().execute(new String[] {"--prepare"});
assertTrue(ctx.getZooReader().exists(ctx.getZooKeeperRoot() + Constants.ZPREPARE_FOR_UPGRADE));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to ensure prepare did not complain about walogs. If the prepare for upgrade marker is deleted in ZK when walogs are detected, then that would do the trick.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated runUpgradeAndCheckOutput and called it from here. Fixed in 039e8bf and b801019

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants