Repository navigation
Conversation
added 17 commits
November 3, 2026 15:27
Move namify() out of the fetcher, and import the remote helpers which need urllib3 or psutil on first use, so the result parsing helpers can be used without them, e.g. by tests or by tools running on a DUT. Do the same for Fetcher and NipaLifetime in contest/hw/lib/nipa.py, which re-exports the helpers for the hw code, so code running on a DUT can import it without the dependencies of the services. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Parse JSON metric observations and policies from nested KTAP output. Attach them to the corresponding test case and keep initial and retry attempts separate. Ignore metrics which are not JSON objects with a name, and metrics with NaN or Infinity, which strict JSON consumers reject. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Build compact summaries and retain complete observations for each metric-bearing test-case attempt. Apply producer-selected transforms, aggregation, summaries and display metadata, and keep the summary value which regression checks compare. Label selftest metrics with the unit they report, or else with the one known for their name. Leave out values which are not finite numbers, skip attempts whose metrics cannot be summarized or stored as strict JSON, and store test cases sharing a sanitized name once. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Add a result_metrics table with one row for each metric-bearing test-case attempt. Store both compact summaries and complete observation arrays so retries remain independent and details do not require one database row per sample. Teach the results collector to derive and upsert these records while ingesting a completed run. Log a failure to store them rather than stop ingesting results. Keep metrics out of results.json_full and the cached results, so /results?format=l2 responses and the combined results do not carry the observations. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Add APIs for compact metric history and complete observations. Compare tracked values with the median of up to five prior successful results with matching branch family, runner, test case, metric dimensions, and policy. Leave values which are not numbers out of the history. Return the latest results of each test case, and read five older ones as the baseline of the oldest returned, so a result is compared with the same history in the list and on its details page. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Add compact history and full detail fixtures for devmem and iperf metrics. Trim two hardware runs to the eight busiest CPUs per host and every second system monitor sample. Check that stored summaries and regression results follow from the observations. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Plot compact metric summaries across test runs. Group series by producer metadata and link each point to its complete observations. Query the history of the selected test case only, and label series with their origin when a chart has metrics of more than one origin. Load a data file named in the URL, like the fixtures, only from this site. Add query/metrics to the offline assets of scripts/ui_assets.sh. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Add a per-run page for complete timestamped observations. Apply producer transforms and display metadata, and show regression baseline information. Load a data file named in the URL only from this site, and link to test outputs only over HTTP(S), as they come from remote runners. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Add a date-oriented performance table with branch, runner, test, and regression filters. Link test cases to history and results to observations. Load a data file named in the URL only from this site. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Move hardware result parsing out of the deployer so service and local tools can use it without loading service dependencies. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Sample CPU and NIC counters on the DUT and peer. Write metrics for each measured test-case window the samples cover, and none if the sampler failed, rather than fill the gaps with an idle host. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Merge per-case system monitor sidecars into parsed KTAP results. Identify whether each metric came from the selftest or monitor. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Run each hardware performance case in a separate process so system metric collection has unambiguous test boundaries. Generate net.config for the performance target on hardware runners. List the cases with -l, honoring the per-program arguments of run_kselftest.sh, which runs the environment setup of a program once more. Each case gets the whole per-test timeout, and a program which fails retries all of its cases. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Add a worker mode which runs an installed selftest tree with existing host networking. Generate net.config files from the interface addresses, unless nic-test.env gives them, which it has to for LOCAL_PREFIX_V6. The kernel was not booted for such a run, so ignore what it logged before. Share the setup with hardware runners, which now also pass the system monitor configuration to run_tests(), so their performance tests collect system metrics, and export the non-empty KSELFTEST_* variables of nic-test.env, like per-program arguments. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Stage built selftests on a DUT, prepare routes and peer SSH access, and run selected programs without kernel deployment. Fail, after saving the results, when the worker reports a kernel crash or a test timeout. NIPA does not build or install kperf. Hardware performance runners require: - kperf-server on both hosts - kperf-client on the DUT Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Build metric API responses from local result.json files. Serve them through the existing performance UI without a database. Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Add two captured hardware runs for testing the local viewer: ./nipa local view Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This the UI side of https://lore.kernel.org/netdev/20261102190827.989117-1-sdf@fomichev.me/T/#mdd4f40bf08e77a96394ae9273564f89ae0948535
Based on the previous discussion, here are the main design decisions:
The series adds:
nipa local run- to run ksft tests on two machinesnipa local view- to serve collected metricsTests under drivers/net/perf are considered 'performance' tests, they are executed sequentially one by one. (Did not touch HW runners configs yet, I'm assuming they are outside?)
Last commit contains sample data that can be examined by
nipa local runThe goal of this series is to get the scaffolding in and iterate. Next things I'm planning to look at: