Repository navigation
Open
Description
I am seeing this failure when running the JHM tests. No results are produced from this test due to the failure:
# Run progress: 6.67% complete, ETA 00:00:43
# Fork: 3 of 10
# Warmup Iteration 1: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
<failure>
java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeLo(TimSort.java:777)
at java.util.TimSort.mergeAt(TimSort.java:514)
at java.util.TimSort.mergeCollapse(TimSort.java:441)
at java.util.TimSort.sort(TimSort.java:245)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:348)
at java.util.stream.Sink$ChainedInt.end(Sink.java:290)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.util.stream.IntPipeline.toArray(IntPipeline.java:502)
at org.numenta.nupic.algorithms.SpatialPooler.inhibitColumnsGlobal(SpatialPooler.java:815)
at org.numenta.nupic.algorithms.SpatialPooler.inhibitColumns(SpatialPooler.java:786)
at org.numenta.nupic.algorithms.SpatialPooler.compute(SpatialPooler.java:206)
at org.numenta.nupic.benchmarks.SpatialPoolerGlobalInhibitionBenchmark.measureAvgCompute_7_Times(SpatialPoolerGlobalInhibitionBenchmark.java:62)
at org.numenta.nupic.benchmarks.generated.SpatialPoolerGlobalInhibitionBenchmark_measureAvgCompute_7_Times.measureAvgCompute_7_Times_avgt_jmhStub(SpatialPoolerGlobalInhibitionBenchmark_measureAvgCompute_7_Times.java:221)
at org.numenta.nupic.benchmarks.generated.SpatialPoolerGlobalInhibitionBenchmark_measureAvgCompute_7_Times.measureAvgCompute_7_Times_AverageTime(SpatialPoolerGlobalInhibitionBenchmark_measureAvgCompute_7_Times.java:164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.openjdk.jmh.runner.LoopBenchmarkHandler$BenchmarkTask.call(LoopBenchmarkHandler.java:198)
at org.openjdk.jmh.runner.LoopBenchmarkHandler$BenchmarkTask.call(LoopBenchmarkHandler.java:180)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Metadata
Metadata
Assignees
Labels
No labels
Activity
rhyolight commentedon Mar 19, 2018
How are you running tests? I'm running with
gradle -Pskipbench checkand it'sBUILD SUCCESSFULfor me.khatchad commentedon Mar 20, 2018
I'm also running that way and it's also successful but if you look at the final results you won't see anything for this particular test (only 2/3 tests are shown).
cogmission commentedon Mar 21, 2018
Hi Guys! (@rhyolight @khatchad)
First, this Logger error is (should be) a non-sequitur. As SLF4J allows each user to specify a log module to handle log output - when it isn't specified, this message is printed and a default logger is used. This is a setting inside of the JMH Benchmarking lib which I believe would have to specify this somehow... But this doesn't (shouldn't) affect the execution.
Secondly I'm kind of confused as to what you might be expecting? When running with the "-Pskipbench" flag, the benchmarks won't run and you won't see any bench mark reports created in build/reports/jmh/* ? So that is the expected behavior. If you run with
gradle checkthe benchmarks are then run and you should see output in the directory previously mentioned?I'm also not clear on what test you say is missing? (i.e. what is the "this test" you say is missing)
Also, how are you both "causing" the above exception, and where are you seeing it?
khatchad commentedon Mar 21, 2018
Ah, sorry, I am not using
-Pskipbenchbecause that skips the benchmark testing! I am simply usinggradle check. Let me gather more detailed data.khatchad commentedon Mar 21, 2018
Hi @cogmission,
Actually, what I was trying to bring attention to was the
IllegalArgumentExceptionsince it causes one of the benchmark tests not to execute. Yes, I agree that it shouldn't affect the execution.Yes, agreed. Sorry about that. I was not using that flag as explained above. What I meant to say that my build was also showing
BUILD SUCCESSFULdespite one of the tests throwing the exception.Yes, I do see it.
I mean to say that the benchmark
org.numenta.nupic.benchmarks.SpatialPoolerGlobalInhibitionBenchmark.measureAvgCompute_7_Timeshas no data associated with it in theresults.txtfile:cogmission commentedon Mar 21, 2018
Hi @khatchad,
Thank you, now I think I understand what the issue is to which you're bringing attention. Let me take a look at it. Also, thank you very much for hanging in there and explaining things to me!
Cheers,
David
khatchad commentedon Mar 21, 2018
No problem, @cogmission. Sorry I wasn't originally clear.
khatchad commentedon Mar 21, 2018
I actually somehow fixed this problem by increasing the data size that the tests run on but I don't think you'd want that here. It seems to me that the intent of this test is a quick performance test. I'll post a pull request soon that will show this test passing (but taking much longer overall).
khatchad commentedon Mar 21, 2018
#539 has a patch where the test does not fail.
cogmission commentedon Mar 22, 2018
Hi @khatchad,
Ok I've found the actual problem. It has to do with the transitivity of the comparator used for Global Inhibition in the SpatialPooler.
https://github.com/numenta/htm.java/blob/master/src/main/java/org/numenta/nupic/model/Connections.java#L236
There was an issue written up about this on StackOverflow...
I set the comparator to: (to eliminate the rounding of values whose difference was < 0.000000001) and there is no error. I'm thinking about inserting this
Comparatoronly in the AbstractAlgorithmBenchmark class so that it is inserted for this test only....as opposed to the normal way...
I will take a look at submitting this as a pull request. I saw your pull request which increases the test size, and I really appreciate your effort on this, but I think inserting this will be a more "wholistic" solution?
Here is a Gist with the class file to replace. Replace the AbstractAlgorithmBenchmark.java file found in src/jmh/java in the
org.numenta.nupic.benchmarkspackage...khatchad commentedon Mar 22, 2018
Hi @cogmission,
The increase of the test size in #539 is just to exploit more parallelism to show how the refactoring can optimize the code. So, yes, I agree with you that if changing the comparator solves the problem, it should be the solution given that this JMH test is supposed to be fast.