Open
Description
Describe the Bug
When executing the retest command in the Azure DevOps pipeline we got unexpected exceptions when test was failing in the first attempt but succeeded after that.
Steps to Reproduce
Add a failing test, the error output is not included.
2024-12-02T09:32:39.6673615Z Running tests: 0%
2024-12-02T09:32:40.4604022Z Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Application.FunctionalTest\bin\Release\net8.0\company.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T09:32:40.6001361Z VSTest version 17.12.0 (x64)
2024-12-02T09:32:40.6143386Z
2024-12-02T09:32:40.8450984Z Starting test execution, please wait...
2024-12-02T09:32:40.9109329Z A total of 1 test files matched the specified pattern.
2024-12-02T09:33:09.9122186Z Running tests: 0%
2024-12-02T09:33:40.1810138Z Running tests: 0%
2024-12-02T09:34:10.4749776Z Running tests: 0%
2024-12-02T09:34:40.7409708Z Running tests: 0%
2024-12-02T09:35:10.9912717Z Running tests: 0%
2024-12-02T09:35:41.2105825Z Running tests: 0%
2024-12-02T09:36:11.5094507Z Running tests: 0%
2024-12-02T09:36:41.6984378Z Running tests: 0%
2024-12-02T09:36:45.6858614Z [xUnit.net 00:04:03.78] company.Application.Sales.DiscountScenariosTest_performance_productlists.Performance [SKIP]
2024-12-02T09:36:46.9030210Z Skipped company.Application.Sales.DiscountScenariosTest_performance_productlists.Performance [1 ms]
2024-12-02T09:36:54.6798321Z [xUnit.net 00:04:12.78] company.Application.Customers.OrganizationLargeDataTest.Create_and_query_one_by_one [SKIP]
2024-12-02T09:36:54.6800777Z [xUnit.net 00:04:12.78] company.Application.Customers.OrganizationLargeDataTest.Create_and_query [SKIP]
2024-12-02T09:36:55.8419968Z Skipped company.Application.Customers.OrganizationLargeDataTest.Create_and_query_one_by_one [1 ms]
2024-12-02T09:36:55.8422016Z Skipped company.Application.Customers.OrganizationLargeDataTest.Create_and_query [1 ms]
2024-12-02T09:37:11.9675535Z Running tests: 0%
2024-12-02T09:37:42.2254034Z Running tests: 0%
2024-12-02T09:38:12.4954851Z Running tests: 0%
2024-12-02T09:38:42.7164232Z Running tests: 0%
2024-12-02T09:39:03.5622838Z [xUnit.net 00:06:21.66] company.Application.Products.PriceListItemTest.BatchCreate [SKIP]
2024-12-02T09:39:03.9796608Z Skipped company.Application.Products.PriceListItemTest.BatchCreate [1 ms]
2024-12-02T09:39:04.1210644Z [xUnit.net 00:06:22.22] company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock [SKIP]
2024-12-02T09:39:05.4648231Z Skipped company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock [1 ms]
2024-12-02T09:39:06.3747827Z [xUnit.net 00:06:24.47] company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock_async [SKIP]
2024-12-02T09:39:06.3984435Z Skipped company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock_async [1 ms]
2024-12-02T09:39:12.9653416Z Running tests: 0%
2024-12-02T09:39:43.1508352Z Running tests: 0%
2024-12-02T09:39:49.0636899Z [xUnit.net 00:07:07.16] company.Application.FieldFramework.SystemObjectFieldTypeTest.ListSystemFields_NotHidden_And_Have_No_Translation [SKIP]
2024-12-02T09:39:50.1187020Z Skipped company.Application.FieldFramework.SystemObjectFieldTypeTest.ListSystemFields_NotHidden_And_Have_No_Translation [1 ms]
2024-12-02T09:40:13.3679894Z Running tests: 0%
2024-12-02T09:40:43.6224550Z Running tests: 0%
2024-12-02T09:41:13.9013802Z Running tests: 0%
2024-12-02T09:41:44.1897585Z Running tests: 0%
2024-12-02T09:42:14.4554020Z Running tests: 0%
2024-12-02T09:42:44.6970853Z Running tests: 0%
2024-12-02T09:43:14.9453462Z Running tests: 0%
2024-12-02T09:43:33.2749219Z Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_10_36_09.trx
2024-12-02T09:43:33.2751626Z
2024-12-02T09:43:33.3012504Z Passed! - Failed: 0, Passed: 4097, Skipped: 7, Total: 4104, Duration: 7 m 22 s - company.Application.FunctionalTest.dll (net8.0)
2024-12-02T09:43:34.8916534Z Running tests: 0%
2024-12-02T09:43:35.3048860Z Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Web.Application.FunctionalTest\bin\Release\net8.0\company.Web.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T09:43:35.4136135Z VSTest version 17.12.0 (x64)
2024-12-02T09:43:35.4222197Z
2024-12-02T09:43:35.6804642Z Starting test execution, please wait...
2024-12-02T09:43:35.7521536Z A total of 1 test files matched the specified pattern.
2024-12-02T09:44:05.0995094Z Running tests: 0%
2024-12-02T09:44:35.3779195Z Running tests: 0%
2024-12-02T09:45:05.5609960Z Running tests: 0%
2024-12-02T09:45:35.7582249Z Running tests: 0%
2024-12-02T09:46:05.5828177Z Failed company.Web.Application.AdminWebApi.Controllers.Products.ProductsControllerTest.QueryDataAsync_WithSearchText_ForBaseProduct [561 ms]
2024-12-02T09:46:05.6230857Z ##[error]EXEC(0,0): Error Message:
2024-12-02T09:46:05.6234280Z EXEC : error Message: [C:\SysData\vsts-agent1\_work\1\s\company.Build.TestFunctional.target]
2024-12-02T09:46:05.6236142Z Γåô (pos 0)
2024-12-02T09:46:05.6236726Z Collection: [eafbcb81-21ce-4c7b-a9a0-84c92e9ffd21]
2024-12-02T09:46:05.6237242Z Stack Trace:
2024-12-02T09:46:05.6238186Z at company.Web.Application.AdminWebApi.Controllers.Products.ProductsControllerTest.QueryDataAsync_WithSearchText_ForBaseProduct() in /_/Test/company.Web.Application.FunctionalTest/AdminWebApi/Controllers/Products/ProductsControllerTest.cs:line 163
2024-12-02T09:46:05.6239130Z --- End of stack trace from previous location ---
2024-12-02T09:46:05.6328427Z Assert.DoesNotContain() Failure: Filter matched in collection
2024-12-02T09:46:05.9957743Z Running tests: 0%
2024-12-02T09:46:36.2924005Z Running tests: 0%
2024-12-02T09:47:01.6132997Z Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_10_44_06.trx
2024-12-02T09:47:01.6139981Z
2024-12-02T09:47:01.6243152Z Failed! - Failed: 1, Passed: 1266, Skipped: 0, Total: 1267, Duration: 2 m 57 s - company.Web.Application.FunctionalTest.dll (net8.0)
2024-12-02T09:47:06.5322474Z Running tests: 1%
2024-12-02T09:47:36.7674647Z Running tests: 1%
2024-12-02T09:47:40.8288727Z Retrying 1 failed test: 0%
2024-12-02T09:47:42.0336194Z Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Web.Application.FunctionalTest\bin\Release\net8.0\company.Web.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T09:47:42.1476790Z VSTest version 17.12.0 (x64)
2024-12-02T09:47:42.1582980Z
2024-12-02T09:47:42.3517951Z Starting test execution, please wait...
2024-12-02T09:47:42.4160527Z A total of 1 test files matched the specified pattern.
2024-12-02T09:48:10.0618569Z Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_10_48_10.trx
2024-12-02T09:48:10.0619313Z
2024-12-02T09:48:10.0681507Z Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 4 s - company.Web.Application.FunctionalTest.dll (net8.0)
2024-12-02T09:48:11.1103609Z Running tests: 1%
2024-12-02T09:48:11.1104083Z Retrying 1 failed test: 0%
As in the log above the test is writing to the error stream and it's picked up as EXEC : error Message:
and it's very hard to understand from where it come.
Expected Behavior
The failed test should be visible with the test data included.
2024-12-02T12:40:53.7207327Z Running tests: 0%
2024-12-02T12:40:54.2566868Z Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Application.FunctionalTest\bin\Release\net8.0\company.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T12:40:54.3654094Z VSTest version 17.12.0 (x64)
2024-12-02T12:40:54.3765417Z
2024-12-02T12:40:54.6078033Z Starting test execution, please wait...
2024-12-02T12:40:54.6784033Z A total of 1 test files matched the specified pattern.
2024-12-02T12:41:23.9736448Z Running tests: 0%
2024-12-02T12:41:54.2555315Z Running tests: 0%
2024-12-02T12:42:24.5091097Z Running tests: 0%
2024-12-02T12:42:54.7434686Z Running tests: 0%
2024-12-02T12:43:24.9961153Z Running tests: 0%
2024-12-02T12:43:55.2438927Z Running tests: 0%
2024-12-02T12:44:25.5122528Z Running tests: 0%
2024-12-02T12:44:55.7770807Z Running tests: 0%
2024-12-02T12:45:06.6002079Z [xUnit.net 00:04:10.97] company.Application.Sales.DiscountScenariosTest_performance_productlists.Performance [SKIP]
2024-12-02T12:45:07.9335906Z Skipped company.Application.Sales.DiscountScenariosTest_performance_productlists.Performance [1 ms]
2024-12-02T12:45:26.0562787Z Running tests: 0%
2024-12-02T12:45:28.4220239Z [xUnit.net 00:04:32.80] company.Application.FieldFramework.SystemObjectFieldTypeTest.ListSystemFields_NotHidden_And_Have_No_Translation [SKIP]
2024-12-02T12:45:28.8184575Z Skipped company.Application.FieldFramework.SystemObjectFieldTypeTest.ListSystemFields_NotHidden_And_Have_No_Translation [1 ms]
2024-12-02T12:45:56.3438379Z Running tests: 0%
2024-12-02T12:46:20.1952961Z [xUnit.net 00:05:24.51] company.Application.Sales.CartContextTest.Confirm_Order_With_Discount [FAIL]
2024-12-02T12:46:20.3442111Z Failed company.Application.Sales.CartContextTest.Confirm_Order_With_Discount [789 ms]
2024-12-02T12:46:20.3476711Z ##[error]EXEC(0,0): Error Message:
2024-12-02T12:46:20.3480502Z EXEC : error Message: [C:\SysData\vsts-agent1\_work\1\s\company.Build.TestFunctional.target]
2024-12-02T12:46:20.3481472Z System.NullReferenceException : Object reference not set to an instance of an object.
2024-12-02T12:46:20.3482152Z Stack Trace:
2024-12-02T12:46:20.3482777Z --- End of stack trace from previous location ---
2024-12-02T12:46:20.3483788Z at company.Application.Sales.CartContextTest.Confirm_Order_With_Discount() in /_/Test/company.Application.FunctionalTest/Sales/CartContextTest.cs:line 486
2024-12-02T12:46:26.5785112Z Running tests: 0%
2024-12-02T12:46:56.7782305Z Running tests: 0%
2024-12-02T12:47:27.0432181Z Running tests: 0%
2024-12-02T12:47:57.3380289Z Running tests: 0%
2024-12-02T12:48:18.4290223Z [xUnit.net 00:07:22.80] company.Application.Products.PriceListItemTest.BatchCreate [SKIP]
2024-12-02T12:48:19.0115100Z Skipped company.Application.Products.PriceListItemTest.BatchCreate [1 ms]
2024-12-02T12:48:27.5578756Z Running tests: 0%
2024-12-02T12:48:57.8561192Z Running tests: 0%
2024-12-02T12:49:19.4309411Z [xUnit.net 00:08:23.80] company.Application.Customers.OrganizationLargeDataTest.Create_and_query_one_by_one [SKIP]
2024-12-02T12:49:19.4312032Z [xUnit.net 00:08:23.81] company.Application.Customers.OrganizationLargeDataTest.Create_and_query [SKIP]
2024-12-02T12:49:19.8339408Z Skipped company.Application.Customers.OrganizationLargeDataTest.Create_and_query [1 ms]
2024-12-02T12:49:19.8342823Z Skipped company.Application.Customers.OrganizationLargeDataTest.Create_and_query_one_by_one [1 ms]
2024-12-02T12:49:28.0812795Z Running tests: 0%
2024-12-02T12:49:58.3583202Z Running tests: 0%
2024-12-02T12:50:28.6461864Z Running tests: 0%
2024-12-02T12:50:58.8824052Z Running tests: 0%
2024-12-02T12:51:29.0880513Z Running tests: 0%
2024-12-02T12:51:29.4982802Z [xUnit.net 00:10:33.87] company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock [SKIP]
2024-12-02T12:51:30.3101589Z Skipped company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock [1 ms]
2024-12-02T12:51:31.7634704Z [xUnit.net 00:10:36.14] company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock_async [SKIP]
2024-12-02T12:51:31.7686782Z Skipped company.Application.Runtime.DistributedLockTests.Acquire_long_running_lock_async [1 ms]
2024-12-02T12:51:49.9169877Z Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_13_44_24.trx
2024-12-02T12:51:49.9175241Z
2024-12-02T12:51:49.9457167Z Failed! - Failed: 1, Passed: 4096, Skipped: 7, Total: 4104, Duration: 7 m 24 s - company.Application.FunctionalTest.dll (net8.0)
2024-12-02T12:51:59.2905407Z Running tests: 1%
2024-12-02T12:52:27.5741269Z Retrying 1 failed test: 0%
2024-12-02T12:52:28.5668787Z Test run for C:\SysData\vsts-agent1\_work\1\s\test\company.Application.FunctionalTest\bin\Release\net8.0\company.Application.FunctionalTest.dll (.NETCoreApp,Version=v8.0)
2024-12-02T12:52:28.7469900Z VSTest version 17.12.0 (x64)
2024-12-02T12:52:28.7695311Z
2024-12-02T12:52:29.0461102Z Starting test execution, please wait...
2024-12-02T12:52:29.1227452Z A total of 1 test files matched the specified pattern.
2024-12-02T12:52:57.8723679Z Running tests: 1%
2024-12-02T12:52:57.8724983Z Retrying 1 failed test: 0%
2024-12-02T12:53:05.1956453Z Results File: C:\SysData\vsts-agent1\_work\1\TestResults\vsts-build-agent1_RND-BUILD01_2024-12-02_13_53_03.trx
2024-12-02T12:53:05.1956994Z
2024-12-02T12:53:05.2039329Z Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 2 s - company.Application.FunctionalTest.dll (net8.0)
In the above log, if you searching for [FAIL]
you will see one row with information that not are included in the first log, and I think it's good to make the visible as well.
Version Info
0.6.3
Activity