Test failure: JIT/Regression/JitBlue/Runtime_63942/Runtime_63942/Runtime_63942.cmd

Failed in: runtime-coreclr gcstress-extra 20240225.1

Failed tests:

coreclr windows x64 Checked gcstress0xc_jitminopts_heapverify1 @ Windows.10.Amd64.Open
    - JIT/Regression/JitBlue/Runtime_63942/Runtime_63942/Runtime_63942.cmd

R2R-CG2 windows x64 Checked jitminopts @ Windows.10.Amd64.Open
    - JIT/Regression/JitBlue/Runtime_63942/Runtime_63942/Runtime_63942.cmd

coreclr windows x64 Checked jitminopts @ Windows.10.Amd64.Open
    - JIT/Regression/JitBlue/Runtime_63942/Runtime_63942/Runtime_63942.cmd

Error message:

 
Assert failure(PID 3592 [0x00000e08], Thread: 9128 [0x23a8]): Assertion failed '((int)offs < 0) || emitComp->opts.IsOSR()' in 'System.SpanHelpers:Fill[ubyte](byref,ulong,ubyte)' during 'Generate code' (IL size 906; hash 0xd8592510; MinOpts)

    File: D: _work1ssrccoreclrjitemitxarch.cpp:3876
    Image: C:hwA06E08C0pcorerun.exe


Return code:      1
Raw output file:      C:hwA06E08C0wA0EC08DFuploadsJitBlueRuntime_63942Runtime_63942output.txt
Raw output:
BEGIN EXECUTION
 "C:hwA06E08C0pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  Runtime_63942.dll 
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 2/25/2024 11:08:14 PM
Processing C:corescorerun.exe.3592.dmp
Invoking: C:Program Files (x86)Windows Kits10Debuggersdcdb.exe -c "$<C:hwA06E08C0		mpv1i5sg.tmp" -z "C:corescorerun.exe.3592.dmp"
stdout: 
Microsoft (R) Windows Debugger Version 10.0.18362.1 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:corescorerun.exe.3592.dmp]
User Mini Dump File with Full Memory: Only application data is available


************* Path validation summary **************
Response                         Time (ms)     Location
OK                                             C:hwA06E08C0pPDB
Symbol search path is: C:hwA06E08C0pPDB
Executable search path is: 
Windows 10 Version 14393 MP (4 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
10.0.14393.6343 (rs1_release.230913-1727)
Machine Name:
Debug session time: Sun Feb 25 23:08:17.000 2024 (UTC + 0:00)
System Uptime: 0 days 0:33:35.118
Process Uptime: 0 days 0:00:03.000
............................

************* Symbol Loading Error Summary **************
Module name            Error
ntdll                  The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(e08.23a8): Unknown exception - code c0000602 (first/second chance not available)
For analysis of this file, run !analyze -v
*** WARNING: Unable to verify checksum for coreclr.dll
KERNELBASE!RaiseFailFastException+0xae:
00007ffe`736d96ce 803d4d78120000  cmp     byte ptr [KERNELBASE!local_unwind+0x7fe2 (00007ffe`73800f22)],0 ds:00007ffe`73800f22=00
0:000> cdb: Reading initial command '$<C:hwA06E08C0		mpv1i5sg.tmp'
0:000> .load C:Users
unner.dotnetsossos.dll
0:000> ~*k

.  0  Id: e08.23a8 Suspend: 0 Teb: 0000004f`83f2c000 Unfrozen
Child-SP          RetAddr           Call Site
0000004f`84176050 00007ffe`456140cc KERNELBASE!RaiseFailFastException+0xae
0000004f`84176620 00007ffe`44fa2aa6 coreclr!_DbgBreakCheck+0x2ec
*** WARNING: Unable to verify checksum for clrjit.dll
0000004f`84177790 00007ffe`484ea04c coreclr!CEEInfo::doAssert+0x76
0000004f`84177830 00007ffe`484e9c6a clrjit!assertAbort+0x13c
(Inline Function) --------`-------- clrjit!noWayAssertAbortHelper+0x17
0000004f`841798c0 00007ffe`48735219 clrjit!noWayAssertBodyConditional+0x4a
0000004f`841798f0 00007ffe`48734d4d clrjit!emitter::emitInsSizeSVCalcDisp+0x279
0000004f`84179940 00007ffe`4873b492 clrjit!emitter::emitInsSizeSV+0x13d
0000004f`841799a0 00007ffe`48719a93 clrjit!emitter::emitIns_R_S+0x1a2
(Inline Function) --------`-------- clrjit!CodeGen::genCodeForCpBlkUnroll::__l40::<lambda_1>::operator()+0x23
0000004f`84179a00 00007ffe`4871d0b8 clrjit!CodeGen::genCodeForCpBlkUnroll+0x3e3
0000004f`84179b10 00007ffe`4871fb8c clrjit!CodeGen::genCodeForStoreBlk+0x1b8
0000004f`84179b50 00007ffe`484ba10a clrjit!CodeGen::genCodeForTreeNode+0x141c
0000004f`84179d70 00007ffe`484b1d84 clrjit!CodeGen::genCodeForBBl

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 128
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor428|0_429(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

This is a JIT assert; changing area to codegen

This is failing in runtime-coreclr jitstress, too.

@TIHan

@TIHan, PTAL. It is blocking clean ci optional. It is asserting in Generate Code.

Will do.

The reason why 3 may be difficult/not possible is because BitCast is constrained (TFrom/TTo must be struct) but the calling API might not be. We do have options for that, like removing the constraints and doing the "is value type" check manually (which we've done in other places), would just need to work through that if that's the path opted for.

BitCast ends up working around the assert here because it is preserved as a call which throws if the sizes don't match, so we'd never get an "invalid" reinterpret cast leftover in the code the JIT sees.

If the assert hits, we're going to generate code this is (probably) going to corrupt the stack (in this case, read garbage). In this case it looks like it's because we're unrolling a Vector256 store and the second instruction has a displacement that makes the ebp offset positive.

One way to fix this is to make all the asserts in emitInsSizeSVCalcDisp based on the variable frame offset before adding the displacement dsp. This significantly weakens the asserts, but if we must allow unsafe code that is truly unsafe, then that might be a way to go.

we're going to generate code this is (probably) going to corrupt the stack (in this case, read garbage)

Notably we're generating the code here, but it will never execute in this particular scenario because we have guards that ensure the path is only taken when the read would be valid. It's mostly getting generated because MinOpts prevents the removal of definitely dead code so we get stuck with this unreachable extra logic.

A user could certainly write such code themselves without the guards however, in which case we will generate the code and not hit any assert (since it will use the production JIT).

I expect the assert here was mostly to catch places where the JIT computes an incorrect offset, but that of course doesn't account for users doing it themselves (either "safely" due to the code being unreachable like we have, or "unsafely" because the user has written some incorrect code). Hence why I suggested 3 as an alternative fix, basically don't allow MinOpts to see the code that is unreachable and would cause the assert, so our own code would never trigger it, but user code still might.

One way to fix this is to make all the asserts in emitInsSizeSVCalcDisp based on the variable frame offset before adding the displacement dsp.

I tried this, but the assert now gets hit a lot in normal cases. The change was this:

        /* Get the frame offset of the (non-temp) variable */

        offs = emitComp->lvaFrameAddress(var, &EBPbased);

        if (EBPbased)
        {
#if defined(TARGET_AMD64) && !defined(UNIX_AMD64_ABI)
            // If localloc is not used, then ebp chaining is done and hence
            // offset of locals will be at negative offsets, Otherwise offsets
            // will be positive.  In future, when RBP gets positioned in the
            // middle of the frame so as to optimize instruction encoding size,
            // the below asserts needs to be modified appropriately.
            // However, for Unix platforms, we always do frame pointer chaining,
            // so offsets from the frame pointer will always be negative.
            if (emitComp->compLocallocUsed || emitComp->opts.compDbgEnC)
            {
                noway_assert((int)offs >= 0);
            }
            else
#endif
            {
                // Dev10 804810 - failing this assert can lead to bad codegen and runtime crashes
                CLANG_FORMAT_COMMENT_ANCHOR;

#ifdef UNIX_AMD64_ABI
                const LclVarDsc* varDsc         = emitComp->lvaGetDesc(var);
                bool             isRegPassedArg = varDsc->lvIsParam && varDsc->lvIsRegArg;
                // Register passed args could have a stack offset of 0.
                noway_assert((int)offs < 0 || isRegPassedArg || emitComp->opts.IsOSR());
#else  // !UNIX_AMD64_ABI

                // OSR transitioning to RBP frame currently can have mid-frame FP
                noway_assert(((int)offs < 0) || emitComp->opts.IsOSR());
#endif // !UNIX_AMD64_ABI
            }
        }

        offs += dsp;

A user could certainly write such code themselves without the guards however, in which case we will generate the code and not hit any assert (since it will use the production JIT).

Except for these are noway_assert, which hit in Release builds. They drop back to MinOpts, which wouldn't help in this case.

Without this assert we might generate bad code, if we don't calculate the instruction size properly in the code that follows.

I tried this, but the assert now gets hit a lot in normal cases. The change was this:

That's basically what I was thinking.

What do you recommend we do then?

if we don't calculate the instruction size properly in the code that follows.

Any reason we don't just ensure we compute the size correctly in this case? I would've thought it already mostly just works since the logic for [ebp+offset] is pretty static based on what the offset is and is ultimately shared with all other addressing mode paths.

That is, any non-zero offset requires the SIB modifier. That offset for both 32-bit and 64-bit can be encoded as 1 or 4 bytes and is sign extended to the pointer width, thus any offset between [-128, +127] is 1-byte and anything else is 4-bytes. There's really not much to get wrong here in terms of instruction size prediction.

I am working on a change which will resolve this by not containing out-of-bounds access to locals, which is how we've been handling this problem so far.

Failed in: runtime-coreclr gcstress-extra 20240302.2

Failed tests:

coreclr windows x64 Checked gcstress0xc_jitminopts_heapverify1 @ Windows.10.Amd64.Open
    - JIT/Regression/JitBlue/Runtime_63942/Runtime_63942/Runtime_63942.cmd

Error message:

 
Assert failure(PID 4248 [0x00001098], Thread: 4428 [0x114c]): Assertion failed '((int)offs < 0) || emitComp->opts.IsOSR()' in 'System.SpanHelpers:Fill[ubyte](byref,ulong,ubyte)' during 'Generate code' (IL size 906; hash 0xd8592510; MinOpts)

    File: D:�_work1ssrccoreclrjitemitxarch.cpp:3876
    Image: C:hwBC2C09DApcorerun.exe


Return code:      1
Raw output file:      C:hwBC2C09DAwA6ED093BuploadsJitBlueRuntime_63942Runtime_63942output.txt
Raw output:
BEGIN EXECUTION
 "C:hwBC2C09DApcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  Runtime_63942.dll 
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 3/2/2024 11:08:56 PM
Processing C:corescorerun.exe.4248.dmp
Invoking: C:Program Files (x86)Windows Kits10Debuggersdcdb.exe -c "$<C:hwBC2C09DA		mp3luamd.tmp" -z "C:corescorerun.exe.4248.dmp"
stdout: 
Microsoft (R) Windows Debugger Version 10.0.18362.1 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:corescorerun.exe.4248.dmp]
User Mini Dump File with Full Memory: Only application data is available


************* Path validation summary **************
Response                         Time (ms)     Location
OK                                             C:hwBC2C09DApPDB
Symbol search path is: C:hwBC2C09DApPDB
Executable search path is: 
Windows 10 Version 14393 MP (4 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
10.0.14393.6343 (rs1_release.230913-1727)
Machine Name:
Debug session time: Sat Mar  2 23:08:59.000 2024 (UTC + 0:00)
System Uptime: 0 days 1:47:14.309
Process Uptime: 0 days 0:00:03.000
............................

************* Symbol Loading Error Summary **************
Module name            Error
ntdll                  The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1098.114c): Unknown exception - code c0000602 (first/second chance not available)
For analysis of this file, run !analyze -v
*** WARNING: Unable to verify checksum for coreclr.dll
KERNELBASE!RaiseFailFastException+0xae:
00007ffd`456396ce 803d4d78120000  cmp     byte ptr [KERNELBASE!local_unwind+0x7fe2 (00007ffd`45760f22)],0 ds:00007ffd`45760f22=00
0:000> cdb: Reading initial command '$<C:hwBC2C09DA		mp3luamd.tmp'
0:000> .load C:Users
unner.dotnetsossos.dll
0:000> ~*k

.  0  Id: 1098.114c Suspend: 0 Teb: 00000092`cb596000 Unfrozen
Child-SP          RetAddr           Call Site
00000092`cb775f10 00007ffd`3d794f4c KERNELBASE!RaiseFailFastException+0xae
00000092`cb7764e0 00007ffd`3d123236 coreclr!_DbgBreakCheck+0x2ec
*** WARNING: Unable to verify checksum for clrjit.dll
00000092`cb777650 00007ffd`320b998c coreclr!CEEInfo::doAssert+0x76
00000092`cb7776f0 00007ffd`320b95aa clrjit!assertAbort+0x13c
(Inline Function) --------`-------- clrjit!noWayAssertAbortHelper+0x17
00000092`cb779780 00007ffd`32309989 clrjit!noWayAssertBodyConditional+0x4a
00000092`cb7797b0 00007ffd`323094bd clrjit!emitter::emitInsSizeSVCalcDisp+0x279
00000092`cb779800 00007ffd`3230fc02 clrjit!emitter::emitInsSizeSV+0x13d
00000092`cb779860 00007ffd`322ee313 clrjit!emitter::emitIns_R_S+0x1a2
(Inline Function) --------`-------- clrjit!CodeGen::genCodeForCpBlkUnroll::__l40::<lambda_1>::operator()+0x23
00000092`cb7798c0 00007ffd`322f1ac2 clrjit!CodeGen::genCodeForCpBlkUnroll+0x403
00000092`cb7799d0 00007ffd`322f45ac clrjit!CodeGen::genCodeForStoreBlk+0x192
00000092`cb779a10 00007ffd`3208968a clrjit!CodeGen::genCodeForTreeNode+0x142c
00000092`cb779c30 00007ffd`320811f4 clrjit!CodeGen::genCodeForBB

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 128
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor428|0_429(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

Navigation Menu

Repository navigation

Test failure: JIT/Regression/JitBlue/Runtime_63942/Runtime_63942/Runtime_63942.cmd #98971

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Test failure: JIT/Regression/JitBlue/Runtime_63942/Runtime_63942/Runtime_63942.cmd #98971

Description

Activity

BruceForstall commented on Feb 27, 2024

amanasifkhalid commented on Feb 27, 2024

JulieLeeMSFT commented on Feb 27, 2024

TIHan commented on Feb 28, 2024

6 remaining items

tannergooding commented on Feb 28, 2024

BruceForstall commented on Feb 28, 2024

tannergooding commented on Feb 28, 2024

TIHan commented on Feb 28, 2024

BruceForstall commented on Feb 28, 2024

TIHan commented on Feb 28, 2024

tannergooding commented on Feb 29, 2024

SingleAccretion commented on Mar 2, 2024

v-wenyuxu commented on Mar 4, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions