Separate stats filtering helpers to reuse in page pruning #18034
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
github-actions
bot
added
libcudf
mhaseeb123
added
2 - In Progress
| @@ -740,4 +462,157 @@ std::optional<std::vector<std::vector<size_type>>> collect_filtered_row_group_in | |||
| return {filtered_row_group_indices}; | |||
| } | |||
|
|
|||
| // All required templates for `stats_caster_base` and `stats_caster_base::host_column<T>` | |||
There was a problem hiding this comment.
Moved definitions as-is. All required templates for row group or page pruning are instantiated here as well. Definitions moved to stats_filter_helpers.hpp as it seems like it doesn't instantiate all templates needed for page pruning
| size_type total_row_groups; | ||
| std::vector<metadata> const& per_file_metadata; | ||
| host_span<std::vector<size_type> const> row_group_indices; | ||
|
|
||
| template <typename ToType, typename FromType> |
There was a problem hiding this comment.
Moved to stats_filter_helpers.hpp
| @@ -248,146 +110,6 @@ struct stats_caster { | |||
| } | |||
| }; | |||
|
|
|||
| /** | |||
There was a problem hiding this comment.
Moved completely to stats_filter_helpers.hpp/cpp
mhaseeb123
changed the title
Co-authored-by: David Wendt <[email protected]>
Co-authored-by: David Wendt <[email protected]>
Co-authored-by: David Wendt <[email protected]>
|
Any failing GTests are fixed by #18019 |
|
|
||
| #include "io/parquet/parquet_common.hpp" | ||
|
|
||
| #include <cudf/ast/detail/expression_transformer.hpp> |
There was a problem hiding this comment.
Moved definitions for stats_caster_base here as not all templates instantiated at one place.
Description
Contributes to #17896
This PR separates stats based filtering helpers for reuse in page pruning using stats in Parquet PageIndex.
Checklist