Support Arrays for the Map scalar functions

@alamb

Is your feature request related to a problem or challenge?

As @alamb mentioned in #11361 (comment), we should support not only scalar lists but also arrays for the map and make_map functions. However, I encountered some challenges, which I detailed in #11361 (comment). I left the disabled test cases in

datafusion/datafusion/sqllogictest/test_files/map.slt

Lines 200 to 214 in 1dfac86

    
           query error 
        
           SELECT make_map(column1, column2, column3, column4) FROM t; 
        
           # TODO: support array value 
        
           # ---- 
        
           # {a: 1, k1: 10} 
        
           # {b: 2, k3: 30} 
        
           # {d: 4, k5: 50} 
        
           query error 
        
           SELECT map(column5, column6) FROM t; 
        
           # TODO: support array value 
        
           # ---- 
        
           # {k1:1, k2:2} 
        
           # {k3: 3} 
        
           # {k5: 5}

.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

take

MakeMap is possible to be removed after #11434, so you can find out array support for MapFunc only

Apologies for the delay, prioritizing this issue.

@goldmedal

Sharing my observation from DuckDB, both Map exprssion and map function are meant to do two different things.
But in Datafusion we are using map function to support Map expression.
In case of DuckDB Map expr & map function would yeild different result for same input.
Shouldn't we have two different function to support these two use cases ? Or any ideas on how should we support both usecases using using single map function.

Observations

D create table t as values
· ('a', 1, 'k1', 10, ['k1', 'k2'], [1, 2], 'POST', [[1,2,3]]),
· ('b', 2, 'k3', 30, ['k3'], [3], 'PUT', [[4]]),
‣ ('d', 4, 'k5', 50, ['k5'], [5], null, [[1,2]]);

D select map(col4, col5) from t;
┌───────────────────────┐
│    map(col4, col5)    │
│ map(varchar, integer) │
├───────────────────────┤
│ {k1=1, k2=2}          │
│ {k3=3}                │
│ {k5=5}                │
└───────────────────────┘

D select Map {col4: col5} from t;
┌────────────────────────────────────────────────────────┐
│ main.map(main.list_value(col4), main.list_value(col5)) │
│               map(varchar[], integer[])                │
├────────────────────────────────────────────────────────┤
│ {[k1, k2]=[1, 2]}                                      │
│ {[k3]=[3]}                                             │
│ {[k5]=[5]}                                             │
└────────────────────────────────────────────────────────┘

~~In the meantime i will continue my investigation.~~
The helper function which i have implemented in #11713 is to support arrays for map scalar function as per issue title.

I am considering introducing a boolean flag to differentiate whether the function was invoked from a Map expression or a map function call. And handle the invocation accordingly

cc: @goldmedal @jayzhan211

Shouldn't we have two different function to support these two use cases ? Or any ideas on how should we support both usecases using using single map functio

In general, I think DataFuion strives not to invent / determine semantics, so in my opinion we should follow other implementations rather than having our own custom behavior whenever possible.

I think by default we should follow the postgres model

If postgres doesn't support maps / this syntax then I think we should follow the DuckDB model

@dharanad

@dharanad I think select Map {col4: col5} from t; is equivalent to SELECT MAP([col4], [col5]);, so we just need to rewrite the syntax to the existing map function.

Shouldn't we have two different function to support these two use cases ? Or any ideas on how should we support both usecases using using single map functio

In general, I think DataFuion strives not to invent / determine semantics, so in my opinion we should follow other implementations rather than having our own custom behavior whenever possible.

I think by default we should follow the postgres model

If postgres doesn't support maps / this syntax then I think we should follow the DuckDB model

I totally agree with you.

@dharanad

@dharanad I think select Map {col4: col5} from t; is equivalent to SELECT MAP([col4], [col5]);, so we just need to rewrite the syntax to the existing map function.

Thanks for clarifying, i will have to say that i totally misunderstood the problem here. Should have checked what Postgres/DuckDB were doing & clarified the problem.

Navigation Menu

Repository navigation

Support Arrays for the Map scalar functions #11436

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Observations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	query error
	SELECT make_map(column1, column2, column3, column4) FROM t;
	# TODO: support array value
	# ----
	# {a: 1, k1: 10}
	# {b: 2, k3: 30}
	# {d: 4, k5: 50}

	query error
	SELECT map(column5, column6) FROM t;
	# TODO: support array value
	# ----
	# {k1:1, k2:2}
	# {k3: 3}
	# {k5: 5}

Support Arrays for the Map scalar functions #11436

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

dharanad commented on Jul 12, 2024

jayzhan211 commented on Jul 13, 2024

dharanad commented on Jul 27, 2024

dharanad commented on Aug 7, 2024

Observations

alamb commented on Aug 8, 2024

jayzhan211 commented on Aug 8, 2024

dharanad commented on Aug 9, 2024

dharanad commented on Aug 9, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions