Using outdated counts when running counter_culture_fix_counts?

[magnusvk]

counter_culture_fix_counts first loads all the correct counts into memory, then iterates over all instances and compares counts. Now, if in between loading the correct counts and comparing them in the loop, new children were created or destroyed, this count will be off and reset to an incorrect, outdated value.

Maybe we can use an update_all with a JOIN to let the database handle this atomically? Might this also be faster?

We could also put the fetching of each correct count and the corresponding update into a transaction, but then that would slow the whole process way down.

[magnusvk]

This problem is much reduced with a927a23; that commit makes sure we pull both the correct and incorrect counts in one query. However, any additions / deletions between then and the later update would still cause a disparity. In any case, we are certainly better off.

[xtagon]

Would changing it to do an update_all with a JOIN help out with #79 at all?

Even though that issue was closed, it is actually still a pressing issue for me and I haven't found a suitible solution yet.

[magnusvk]

@xtagon I'm not sure what you mean to be honest.

[xtagon]

Maybe we can use an update_all with a JOIN to let the database handle this atomically? Might this also be faster?

@magnusvk I was hoping this meant counter_culture_fix_counts would be faster in that case

[magnusvk]

@xtagon it uses update_all: https://github.com/magnusvk/counter_culture/blob/master/lib/counter_culture/reconciler.rb#L118

[xtagon]

@magnusvk Yes it does, but it does 1 update_all per primary key, so it's effectively a singular update.

In the end I was able to do a single update_all to compute counts for a table of millions of records in under a minute, but my use case is very simplified compared to what counter_culture supports so I can see how that would be hard to add to the library (with things like soft deletion, advanced associations, etc.)

[magnusvk]

Great, glad you could get it resolved. :)