Represent Hive/value partitioning without claiming Partitioning::Hash #23436

Activity

1. Phoenix500526

   added

   enhancementNew feature or requestNew feature or request

   on Jul 10, 2026
2. Phoenix500526

   added the

   Feature

   issue type on Jul 10, 2026
3. gstamatakis95 commented on Sep 10, 2026

   

   gstamatakis95

   on Sep 10, 2026

   Last edited by gstamatakis95

   More actions

   take
4. jayshrivastava commented on Oct 15, 2026

   

   jayshrivastava

   on Oct 15, 2026

   Contributor

   More actions

   @gstamatakis95 Are you still working on this?
5. jayshrivastava commented on Oct 15, 2026

   

   jayshrivastava

   on Oct 15, 2026

   Contributor

   More actions

   Ah I just saw #25279.

   That PR implements support for hive partitioning, detecting that files are Hive partitioned and automatically giving the data source a Range partitioning property. That's a reasonable change but I think we should keep this issue open since it's tracking a more generalized problem.

   Notably Trino supports the following:

   • Can we support a non-range style of partitioning? For example, my data might be partitioned without any presumed sort order like p1 = {A, C, E}, p2 = {B, D}, so Range doesn't apply
   • Not assuming a file-backed data source. The data source might be a wrapper on top of an external service that exposes partitions (notably datafusion's hash function cannot be pushed down to another service / language).
   • Dynamic filtering works without specifying the partitioning expression / hash function*

   *caveat - Filters are unioned across partitions within a machine. However, there's still a distinct filter per machine.

   I think it would be good to support the above for feature parity with Trino.
6. jayshrivastava commented on Oct 16, 2026

   

   jayshrivastava

   on Oct 16, 2026

   Last edited by jayshrivastava

   Contributor

   More actions

   What if we added a custom partitioning scheme which stores an identifier which helps check equality?

   #[derive(Debug, Clone)]
   pub enum Partitioning {
       RoundRobinBatch(usize),
       Hash(Vec<Arc<dyn PhysicalExpr>>, usize),
       Range(RangePartitioning),
       /// NEW: denote that we are partitioned on these columns using an identifier to combine the partitioning scheme.
       Custom(Vec<Arc<dyn PhysicalExpr>>, num_partitions: usize, identifier: usize),
       UnknownPartitioning(usize),
   }

   The only edge case I think can think of is that CASE expressions in dynamic filters won't work because the partitions are only defined by an identifier.

   • In the short term, we can just OR the cases together and not use CASE hash.
   • In the medium / long term, I have a proposal to avoid CASE hash while preserving partition-specific dynamic filters store partitioning in dynamic filters and remove CASE hash(expr) #23817
7. jayshrivastava commented on Oct 16, 2026

   

   jayshrivastava

   on Oct 16, 2026

   Last edited by jayshrivastava

   Contributor

   More actions

   @kosiew @Phoenix500526 @gabotechs @alamb, what do y'all think about the above two comments?

   I have some use cases that would benefit from custom partitioning (ie. it doesn't fit into Hash nor Range partitioning) to avoid repartitions during co-partitioned joins and aggregations.
8. alamb commented on Oct 16, 2026

   

   alamb

   on Oct 16, 2026

   Contributor

   More actions

   What if we added a custom partitioning scheme which stores an identifier which helps check equality?

   Are you suggesting that we add a custom partitioning as part of this PR? Or as a follow on project?

   In my mind, maybe it is time to move to a trait based partitioning scheme rahter than an enum (which would naturally allow any arbitrary partitioning scheme to be supported) 🤔
9. jayshrivastava commented on Oct 19, 2026

   

   jayshrivastava

   on Oct 19, 2026

   Contributor

   More actions

   Are you suggesting that we add a custom partitioning as part of this PR? Or as a follow on project?

   Probably a follow up.

   In my mind, maybe it is time to move to a trait based partitioning scheme rahter than an enum (which would naturally allow any arbitrary partitioning scheme to be supported) 🤔

   Ack! I think I agree at a high level. I'll keep an eye out for use cases and see if I can find time to look into it.