Negative consequences of catchNonTestRelatedExceptions from TestScope to Spring Boot apps

[man85]

There is no ability to change the value of catchNonTestRelatedExceptions, but kdoc says "some tests may want to disable it" @dkhalanskyjb

[dkhalanskyjb]

@man85, that's intentional, because we don't publicly support this functionality. It's purely a workaround for some codebases that have their own specialized error handling or a temporary measure for large codebases migrating to a new version of kotlinx-coroutines-test. We may even remove this workaround at a later point without considering it a breaking change.

If you do want to change that value anyway, here's how: #3736 (comment) Could you explain why you want to do that? We are interested in learning the use cases to decide if this needs a proper public API, a hidden API for workarounds, or if it can be removed.

[man85]

Could you explain why you want to do that?

We have a test class which makes database migrations before each test method via JUnit extension and it does it with a SupervisorJob. In this job happens exception, because of wrong SQL for the first test method (with runTest) and this exception is being processed with invokeOnCompletion, but ExceptionCollector takes this exception after first test method fail and when second test starts (with runTest), UncaughtExceptionsBeforeTest will be thrown.

In this case, such an exception in a job coroutine should not be treated as an error and should not be collected by the ExceptionCollector. That's why I would like to disable this feature of collecting unprocessed errors.

Same case was described here.

[dkhalanskyjb]

this exception is being processed with invokeOnCompletion

That's not the intended way to process exceptions. If you check your logs, you'll probably see that these failures are spammed as stacktraces. What you need is https://kotlinlang.org/docs/exception-handling.html#coroutineexceptionhandler . With CoroutineExceptionHandler, exceptions will neither be spammed into logs nor fail tests. Also, invokeOnCompletion is a deeply technical, low-level thing, and it's best to avoid using it unless you have some special requirements (for example, invokeOnCompletion is valuable when you're writing your own concurrent data structures or coroutine primitives, for example).

[dkhalanskyjb]

If you are trying to replace kotlinx-coroutines-test exception handling with this, then this approach wouldn't help you, because if there are several globally installed CoroutineExceptionHandlers, all of them get notified by default. The order is:

• Try notifying the parent coroutine.
• If that fails, try a local CoroutineExceptionHandler.
• If there is no local CoroutineExceptionHandler, notify the global ones.

To answer your question directly, you need to let the service loader machinery know about the exposed service, so you need to add a META-INF file like https://github.com/Kotlin/kotlinx.coroutines/blob/master/ui/kotlinx-coroutines-android/resources/META-INF/services/kotlinx.coroutines.CoroutineExceptionHandler

[man85]

Thanks for the clarification!
I suppose it would be great to have a CoroutineExceptionHandler for each test instance class, but not globally.

[dkhalanskyjb]

This is how kotlinx-coroutines-test already works, in a way: runTest will report the uncaught exceptions at the end of the test, so if all of your coroutines finish by the time runTest exits, you will never encounter UncaughtExceptionsBeforeTest. UncaughtExceptionsBeforeTest is the last-ditch effort to report the exceptions at least somewhere so that you get alerted about an error even if the error happens after the test has already completed.

[man85]

If test classes are executed in parallel, is it possible that an exception from one test class will be reported in another?

[man85]

runTest will report the uncaught exceptions at the end of the test

But runTest cannot track exceptions that occurred before the first test method, because report callbacks for uncaught exceptions are not yet registered with ExceptionCollector.

Try execute:

import kotlinx.coroutines.CoroutineName
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
import kotlinx.coroutines.test.runTest
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.MethodOrderer
import org.junit.jupiter.api.Order
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.TestInstance
import org.junit.jupiter.api.TestMethodOrder
import java.util.concurrent.atomic.AtomicInteger

@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class TestA {
    val failureNumber = AtomicInteger(0)

    @BeforeEach
    fun failure(){
        val context = CoroutineName(this.javaClass.name)
        val backgroundScope = CoroutineScope(context + Dispatchers.Unconfined)
        backgroundScope.launch {
            val errorMsg = "Background failure ${failureNumber.incrementAndGet()}"
            println("Throwing $errorMsg")
            throw RuntimeException(errorMsg)
        }
    }

    @Test
    @Order(1)
    fun test1() = runTest {}

    @Test
    @Order(2)
    fun test2() = runTest {}

    @Test
    @Order(3)
    fun test3() = runTest {}
}

Background failure 1 swallowed

@dkhalanskyjb Should I report it as a bug?

[man85]

If test classes are executed in parallel, is it possible that an exception from one test class will be reported in another?

Yes, it is possible. Steps to reproduce:

1. Create TestA and TestB in same package.
2. Specify environment variable in IDE JAVA_TOOL_OPTIONS=-Djunit.jupiter.execution.parallel.enabled=true -Djunit.jupiter.execution.parallel.mode.default=concurrent to run tests in parallel.
3. Run tests.
4. TestB fails because of background failure of TestA: Suppressed: java.lang.RuntimeException: Background failure А

import kotlinx.coroutines.CoroutineName
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
import kotlinx.coroutines.test.runTest
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.MethodOrderer
import org.junit.jupiter.api.Order
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.TestInstance
import org.junit.jupiter.api.TestMethodOrder

@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class TestA {

    @BeforeEach
    fun failure() {
        Thread.sleep(5000)
        val context = CoroutineName(this.javaClass.name)
        val backgroundScope = CoroutineScope(context + Dispatchers.Unconfined)
        backgroundScope.launch {
            throw RuntimeException("Background failure А")
        }
        Thread.sleep(20000)
    }

    @Test
    @Order(1)
    fun test1() = runTest {}
}

import kotlinx.coroutines.test.runTest
import org.junit.jupiter.api.AfterEach
import org.junit.jupiter.api.MethodOrderer
import org.junit.jupiter.api.Order
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.TestInstance
import org.junit.jupiter.api.TestMethodOrder

@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class TestB {
    @AfterEach
    fun waitForFailureOccurred() {
        Thread.sleep(10000)
    }

    @Test
    @Order(1)
    fun test1() = runTest {}

    @Test
    @Order(2)
    fun test2() = runTest {}
}

@dkhalanskyjb Should I report it as a bug?

[dkhalanskyjb]

But runTest cannot track exceptions that occurred before the first test method, because report callbacks for uncaught exceptions are not yet registered with ExceptionCollector.

Yes, that's true. We intentionally chose to do this so that there wouldn't be a memory leak due to storing exceptions for a runTest invocation that never happens, which is possible in codebases that accidentally add a kotlinx-coroutines-test dependency and don't use it.

Should I report it as a bug?

I don't see the point. First, it's impossible for us to fix this short of prohibiting parallel runTest invocations, because the coroutines in your example have no relation to the tests they were launched in. Second, any exception that actually goes to the ExceptionCollector is an anomaly. On Android, in production, any such uncaught exception crashes the whole application. On Kotlin/Native, too. On non-Android Kotlin/JVM, the exception handler of the thread is used, which spams the stacktrace to logs by default. If you receive an exception from ExceptionCollector, this means some of your coroutine scopes have neither a CoroutineExceptionHandler nor a parent coroutine, so with the absence of any proper ways to propagate the exception, the last-ditch-effort exception propagation mechanism is used. The same lack of structure is what prevents runTest from tracking exceptions back to the tests that led to those exceptions. Exceptions originating in coroutines that are children of the TestScope of a runTest don't have to use CoroutineExceptionHandler or the global handlers, the coroutines machinery deals with exception propagation.

[man85]

any exception that actually goes to the ExceptionCollector is an anomaly

I can't agree with you.
In the following code snippet, exceptions and success execution are normally processed in invokeOnCompletion, but ExceptionCollector catches exceptions.

 val job = SupervisorJob(coroutineContext.job)
 activeMigrationJob = launch(job) { 
     ...
 }.invokeOnCompletion {
      if (it == null) {
          ...
      } else ...
 }