[mono][android] Investigate failures on Linux Bionic [x64/arm64] jobs #100281

Description

Linux bionic jobs have been lately unreliable and unusable due to extensive number of failures. These failures must be triaged and eliminated to gain reliable coverage on this platform. This is a part of improving the reliability and coverage of Mobile testing infrastructure (#99949).

Error types

Overall, x64 Linux bionic is slightly more reliable than arm64 version. There are 3 main error patterns occuring on linux_bionic-[x64/arm64] Release AllSubsets_Mono jobs across many test suits.

1. Segmentation faults - (exit code 139)

dbug: Exit code: 139
Std out:
Std err:   Segmentation fault 

3. Device not found - (exit code 81)

XHarness exit code: 81 (DEVICE_NOT_FOUND)

4. Operation got canceled (time out) - (exit code -4)
   • Test finishes without failures but the entire job gets cancelled (happening mostly on x64)

##[error]The Operation will be canceled. The next steps may not contain expected logs.
##[error]The operation was canceled.

• there are other types of errors happening but these 3 account for the majority of failures.

Investigation areas

• Usage of simulator instead of devices
• Runtime vs Library tests

Goals

• understand what is triggering the above-mentioned errors
• eliminate these errors from the testing infrastructure
• make Linux bionic jobs green

FYI: @kotlarmilos @vitek-karas

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Summary