Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Use coroutineScope and async instead of CountDownLatch #940

Open
wants to merge 12 commits into
base: develop
Choose a base branch
from

Conversation

mpeterss
Copy link
Collaborator

Using deferred to wait for all async calls to end instead of using
a CountDownLatch.

Using deferred to wait for all async calls to end instead of using
a CountDownLatch.
@mpeterss mpeterss self-assigned this Nov 14, 2019
Copy link
Collaborator

@prasanthu prasanthu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try This instead
p1.txt

Copy link
Collaborator

@prasanthu prasanthu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do as we discussed

The CoroutineScope will wait for all Jobs to finish so there is
no need for the Deferred.
@mpeterss mpeterss changed the title Use Deferred instead of CountDownLatch Use CoroutineScope and asych instead of CountDownLatch Nov 14, 2019
@prasanthu prasanthu changed the title Use CoroutineScope and asych instead of CountDownLatch Use coroutineScope and async instead of CountDownLatch Nov 14, 2019
@mpeterss
Copy link
Collaborator Author

Do not merge this, I want to test this more to be sure it is working correctly

@mpeterss mpeterss changed the title Use coroutineScope and async instead of CountDownLatch [WIP] Use coroutineScope and async instead of CountDownLatch Nov 15, 2019
Copy link
Member

@vihangpatil vihangpatil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DO NOT MERGE THIS. The Kotlin Coroutine might not work.

The scope will work when the child coroutines are async using suspend.
In that case, the callback is handled as a continuation.

But in this case, the callback is an execution of the lambda function which is passed as the last parameter.

So, the parent coroutine scope will be blocked till all the coroutines in it are completed, but they get completed immediately. The scope will not be blocked until all the callbacks are executed.

@prasanthu
Copy link
Collaborator

DO NOT MERGE THIS. The Kotlin Coroutine might not work.

The scope will work when the child coroutines are async using suspend.
In that case, the callback is handled as a continuation.

But in this case, the callback is an execution of the lambda function which is passed as the last parameter.

So, the parent coroutine scope will be blocked till all the coroutines in it are completed, but they get completed immediately. The scope will not be blocked until all the callbacks are executed.

Are you talking about the bimap lamda? I don't see any asynchronous calls than the async in consumeRequestHandler . If the whole loop (request.msccList.forEach) runs synchronously, the asyncs methods will be called before the loop is finished. In such cases the async calls are part of the coroutineScope

storage.consume(
msisdn = consumptionRequest.msisdn,
usedBytes = consumptionRequest.usedBytes,
requestedBytes = consumptionRequest.requestedBytes) { storeResult ->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that the coroutine scope will wait til the last argument to storage.consume - the lambda with storeResult arg is called back.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct. I have seen it fail in AT. So this is not working

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I understand what you are talking about

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

storage.consume() calls it callback in a different CoroutineScope. A new one is created when we call writeSuspended()

vihangpatil and others added 5 commits November 15, 2019 20:06
This was not waiting for the callback to complete. Now it use
async and scope.
This will help to avoid confusion of while using threads and
coroutines together.
Copy link
Collaborator

@prasanthu prasanthu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review my changes

This reverts commit 725192b.
The actual patch fails to run `load test Neo4j`
@prasanthu
Copy link
Collaborator

I have reverted the change I made for converting CompletionStage to Deferred. The load test with 10_000 requests failed with this change. The max I could do was 500

This will help to avoid confusion of while using threads and
coroutines together.
@prasanthu
Copy link
Collaborator

That revert made the creditControlRequestInitTerminateNoCredit fail. So bringing the change back. We need to sit together and understand this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants