Repository navigation
Conversation
Modified UpgradeUtil to check root, metadata, and user tablet metadata for the existence of write-ahead logs when the user runs the '--prepare' command. If logs are found the user is advised to abort the upgrade and restart the instance to allow tablet recovery. Assisted-By: OpenAI GPT-6 Luna
| .collect(Collectors.toList()); | ||
| Iterator<TabletMetadata> iter = | ||
| readTabletMetadata(rootFiles, MetadataSchema.TabletsSection.getRange(MetadataTable.ID)); | ||
| List<String> metadataFiles = new ArrayList<>(); |
There was a problem hiding this comment.
Putting all the files from different tablets together is not correct and could lead to reading data from a split that is stale and outside a tablets range. The code could be greatly simplified by using the data range on the extent. Tried writing that to see what this would look like and came up with the following. This goes down the tree restricting the range on each child tablet. Found that need to setup the versioning iterator also. Also need to look for tablets with a location or write ahead logs, the manager eventually assigns walogs to tablets assigned to a dead tserver and removes the location. With no live tservers any tablet w/ a location is assigned to a dead tserver.
private Iterator<TabletMetadata> readChildTablets(TabletMetadata tabletMetadata) {
var extent = tabletMetadata.getExtent();
Preconditions.checkState(extent.isMeta());
String[] files = tabletMetadata.getFiles().stream().map(stf -> stf.getPath().toString())
.toArray(String[]::new);
// RFileScanner sets up the system iterator stack by default
Scanner rfileScanner = RFile.newScanner().from(files)
.withTableProperties(IteratorConfigUtil.getInitialTableIterators()).build();
var range = extent.toDataRange().clip(MetadataSchema.TabletsSection.getRange(), true);
if (range == null) {
return Collections.emptyIterator();
}
rfileScanner.setRange(range);
RowIterator rowIter = new RowIterator(rfileScanner);
return Iterators.transform(rowIter,
ri -> TabletMetadata.convertRow(ri, EnumSet.allOf(TabletMetadata.ColumnType.class), false));
}
private void checkTablet(TabletMetadata tm) {
// TODO throw exception if either is true
System.out.println(tm.getExtent() + " hasLoc:" + (tm.getLocation() != null) + " hasWals:"
+ (!tm.getLogs().isEmpty()));
}
private void checkForDirtyTablets(ClientContext ctx) {
var rootMeta = TabletsMetadata.getRootMetadata(ctx.getZooKeeperRoot(), ctx.getZooCache());
checkTablet(rootMeta);
readChildTablets(rootMeta).forEachRemaining(tm->{
checkTablet(tm);
readChildTablets(tm).forEachRemaining(this::checkTablet);
});
}
@Test
public void test2() throws Exception {
try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
client.tableOperations().create("table1");
client.tableOperations().addSplits("table1",
new TreeSet<>(List.of(new Text("c"), new Text("m"))));
client.tableOperations().create("table2");
client.tableOperations().addSplits("table2",
new TreeSet<>(List.of(new Text("3"), new Text("5"))));
client.tableOperations().create("table3");
client.tableOperations().addSplits("table3",
new TreeSet<>(List.of(new Text("30"), new Text("90"))));
client.tableOperations().addSplits("accumulo.metadata",
new TreeSet<>(List.of(new Text("2<"))));
client.tableOperations().flush("accumulo.metadata", null, null, true);
client.tableOperations().flush("accumulo.root", null, null, true);
UtilWaitThread.sleep(1000);
checkForDirtyTablets((ClientContext) client);
}
}There was a problem hiding this comment.
I think I have the logic correct now
| do { | ||
| Thread.sleep(zkTimeout); | ||
| tserversWithLocks = 0; | ||
| for (String zPath : tserverNodes) { |
There was a problem hiding this comment.
May be able to shorten this code by using live tserver set.
There was a problem hiding this comment.
Good suggestion, that reduced the amount of code.
| TabletMetadata metadataTablet = iter.next(); | ||
| metadataTablet.getFiles().stream().map(stf -> stf.getPathStr()).forEach(metadataFiles::add); | ||
| if (!metadataTablet.getLogs().isEmpty()) { | ||
| extentsNeedingRecovery.add(metadataTablet.getExtent()); |
There was a problem hiding this comment.
Once a dirty extent is found should abort, should not read its children as their data will be stale and not include the data in walogs.
There was a problem hiding this comment.
I have updated the logic to implement this change.
| try { | ||
| findTabletsNeedingRecovery(Constants.ZROOT + "/" + iid, zc); | ||
| } catch (IllegalStateException e) { | ||
| LOG.error( |
There was a problem hiding this comment.
Wondering if zUpgradepath should be removed from ZK here and also if we should not print the message later about being ready for upgrade. Would also be nice to have a non zero exit code, that could be done by throwing an exception.
There was a problem hiding this comment.
Removing zUpgradePath here would turn this pre-upgrade tablet recovery check from a warning to an error condition. The code currently just warns, but allows the user to continue. Do you think that we should not allow an upgrade to occur if tablets need recovery? If so, there might be an issue enforcing it as upgrade --prepare with the old version of code is optional because the user can run upgrade start --force with the new version of code.
| error = true; | ||
| } | ||
| if (error) { | ||
| LOG.warn("Continuing with the upgrade is not advisable. It's recommended to abort this " |
There was a problem hiding this comment.
| LOG.warn("Continuing with the upgrade is not advisable. It's recommended to abort this " | |
| LOG.warn("Because some tablets have data in write ahead logs, continuing with the upgrade is not advisable. It's recommended to abort this " |
|
|
||
| createPrepareMarker(ctx); | ||
|
|
||
| stopManagerAndWait(ctx); |
There was a problem hiding this comment.
This test passes, however its not really succeeding because there are walogs or locations the way the cluster is shutdown. Maybe the following will do a clean shutdown. Also the test should fail if recovery is needed.
| stopManagerAndWait(ctx); | |
| cluster.getClusterControl().adminStopAll(); |
There was a problem hiding this comment.
Suggested replacement worked, but needed to wait for the manager lock to be removed in ZooKeeper. Fixed in 039e8bf.
| System.setProperty("accumulo.properties", "file://" + getCluster().getAccumuloPropertiesPath()); | ||
| new UpgradeUtil().execute(new String[] {"--prepare"}); | ||
| assertTrue(ctx.getZooReader().exists(ctx.getZooKeeperRoot() + Constants.ZPREPARE_FOR_UPGRADE)); | ||
|
|
There was a problem hiding this comment.
Need to ensure prepare did not complain about walogs. If the prepare for upgrade marker is deleted in ZK when walogs are detected, then that would do the trick.
Modified UpgradeUtil to check root, metadata, and user tablet metadata for the existence of write-ahead logs when the user runs the '--prepare' command. If logs are found the user is advised to abort the upgrade and restart the instance to allow tablet recovery.
Assisted-By: OpenAI GPT-6 Luna