Skip to content

Get/Test/Export should run concurrently #1737

Open

Description

@SteveL-MSFT

Summary of the new feature / enhancement

Perf has been an issue for large configurations since things run serially.

Proposed technical implementation details (optional)

For get, test, and export, we should allow running resources concurrently. Have a setting to specify the number or perhaps based on CPU cores/threads by default. set is more complex that we'll have to think about later.

Activity

  1. theJasonHelmick commented on Oct 8, 2026

    @theJasonHelmick
    Collaborator

    We want to expose settings at the configuration level, and also consider how to expose in a resource manifest how the resource itself determines concurrency. We want to avoid overloading the system.

    Follow-up from the 10/8 working group discussion on concurrency:

    • RFC likely needed. Concurrency touches configuration document processing, settings, definitions, and resource manifests. It most likely needs an RFC, and certainly one if set is included.
    • get/test vs. set symmetry (open). If get/test/export can run concurrently but set stays serial, the behavior of the node graph becomes harder to predict and to visualize. One view is that the language, not a setting, should define what runs in parallel. Another is that an asymmetric design is acceptable if it is clean, since set is the harder case. Not resolved.
    • Marking resources. We considered having each resource declare everything it touches so the engine can detect conflicts. That is probably not feasible. The alternative is that the configuration author decides and tests it. Neither is settled.
    • Unsupported directives. Where concurrency isn't supported, the directive is ignored and everything runs serially.
    • Rollout. Mark the feature experimental, and start opt-in because of the added complexity and unexpected behavior.
    • Prototype option. Try a parallel group resource implemented only by the engine. Resources outside the group stay serial. Resources inside it run in parallel (serial A, then the group, then serial B). This avoids adding directives to many resources and is easier to reason about in the document. If it proves valuable, pull it into the engine through directives and deprecate the group.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions