CI failing on master: External error: query is expected to fail, but actually succeed:

Describe the bug

https://github.com/apache/arrow-datafusion/actions/runs/5647552111/job/15298151230

External error: query is expected to fail, but actually succeed:
[SQL] SELECT i - ts1 from FOO;
at tests/sqllogictests/test_files/timestamps.slt:1253

External error: query is expected to fail, but actually succeed:
[SQL] SELECT array_agg(c13 ORDER BY c1) FROM aggregate_test_100
at tests/sqllogictests/test_files/aggregate.slt:99

External error: query is expected to fail, but actually succeed:
[SQL] SELECT s.country, ARRAY_AGG(s.amount ORDER BY s.amount DESC, s.country DESC) AS amounts,
    SUM(s.amount ORDER BY s.amount DESC) AS sum1
  FROM sales_global AS s
  GROUP BY s.country
at tests/sqllogictests/test_files/groupby.slt:2229

External error: query is expected to fail, but actually succeed:
[SQL] SELECT
SUM(c4) OVER(ORDER BY c2 RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING),
SUM(c3) OVER(ORDER BY c2 RANGE BETWEEN 10000 PRECEDING AND 10000 FOLLOWING),
COUNT(*) OVER(ORDER BY c2 RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING)
FROM aggregate_test_100
ORDER BY c9
LIMIT 5
at tests/sqllogictests/test_files/window.slt:718

To Reproduce

No response

Expected behavior

No response

Additional context

No response

🤔 it seems like the expected error is

# Interval - Timestamp => error
statement error DataFusion error: type_coercion\ncaused by\nError during planning: Interval\(MonthDayNano\) \- Timestamp\(Nanosecond, None\) can't be evaluated because there isn't a common type to coerce the types to
SELECT i - ts1 from FOO;

but now it passes somehow (on CI) 🤔

This didn't fail for me locally until I did cargo update

(arrow_dev) alamb@MacBook-Pro-8:~/Software/arrow-datafusion$ cargo update
    Updating crates.io index
    Updating allocator-api2 v0.2.15 -> v0.2.16
    Updating anyhow v1.0.71 -> v1.0.72
    Updating async-trait v0.1.71 -> v0.1.72
    Updating axum v0.6.18 -> v0.6.19
    Updating clap v4.3.11 -> v4.3.19
    Updating clap_builder v4.3.11 -> v4.3.19
    Updating clap_derive v4.3.2 -> v4.3.12
    Updating ctor v0.2.3 -> v0.2.4
    Updating dyn-clone v1.0.11 -> v1.0.12
    Updating either v1.8.1 -> v1.9.0
    Updating fastrand v1.9.0 -> v2.0.0
    Removing instant v0.1.12
    Removing io-lifetimes v1.0.11
    Updating itoa v1.0.8 -> v1.0.9
    Updating libz-sys v1.1.9 -> v1.1.10
    Removing linux-raw-sys v0.3.8
    Updating num-traits v0.2.15 -> v0.2.16
    Updating paste v1.0.13 -> v1.0.14
    Updating prettyplease v0.2.10 -> v0.2.12
    Updating proc-macro2 v1.0.64 -> v1.0.66
    Updating quote v1.0.29 -> v1.0.32
    Removing rustix v0.37.23
    Updating rustversion v1.0.13 -> v1.0.14
    Updating ryu v1.0.14 -> v1.0.15
    Updating scopeguard v1.1.0 -> v1.2.0
    Updating semver v1.0.17 -> v1.0.18
    Updating seq-macro v0.3.4 -> v0.3.5
    Updating serde v1.0.171 -> v1.0.175
    Updating serde_derive v1.0.171 -> v1.0.175
    Updating serde_json v1.0.102 -> v1.0.103
    Updating serde_yaml v0.9.22 -> v0.9.25
    Updating sqllogictest v0.15.0 -> v0.15.1
    Updating stringprep v0.1.2 -> v0.1.3
    Updating substrait v0.12.2 -> v0.12.3
    Updating syn v2.0.25 -> v2.0.27
    Updating target-lexicon v0.12.9 -> v0.12.10
    Updating tempfile v3.6.0 -> v3.7.0
    Updating thiserror v1.0.43 -> v1.0.44
    Updating thiserror-impl v1.0.43 -> v1.0.44
    Updating unicode-ident v1.0.10 -> v1.0.11
    Updating unsafe-libyaml v0.2.8 -> v0.2.9
    Updating uuid v1.4.0 -> v1.4.1
    Updating zstd v0.12.3+zstd.1.5.2 -> v0.12.4
    Updating zstd-safe v6.0.5+zstd.1.5.4 -> v6.0.6

Now it is failing:

[SQL] SELECT array_agg(c13 ORDER BY c1) FROM aggregate_test_100
at tests/sqllogictests/test_files/aggregate.slt:99

External error: query is expected to fail, but actually succeed:
[SQL] SELECT i - ts1 from FOO;
at tests/sqllogictests/test_files/timestamps.slt:1253

External error: query is expected to fail, but actually succeed:
[SQL] SELECT
SUM(c4) OVER(ORDER BY c2 RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING),
SUM(c3) OVER(ORDER BY c2 RANGE BETWEEN 10000 PRECEDING AND 10000 FOLLOWING),
COUNT(*) OVER(ORDER BY c2 RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING)
FROM aggregate_test_100
ORDER BY c9
LIMIT 5
at tests/sqllogictests/test_files/window.slt:718

External error: query is expected to fail, but actually succeed:
[SQL] SELECT s.country, ARRAY_AGG(s.amount ORDER BY s.amount DESC, s.country DESC) AS amounts,
    SUM(s.amount ORDER BY s.amount DESC) AS sum1
  FROM sales_global AS s
  GROUP BY s.country
at tests/sqllogictests/test_files/groupby.slt:2229

Error: Execution("4 failures")
error: test failed, to rerun pass `-p datafusion --test sqllogictests`

Thus I conclude this is related to a change in the upstream dependencies

https://crates.io/crates/sqllogictest/0.15.1 was released about the right timeframe

I am pretty sure this is related to the update of sqllogictest. I can fix the issue by pinning to 0.15.0

cargo update --precise 0.15.0 -p  sqllogictest
    Updating crates.io index
 Downgrading sqllogictest v0.15.1 -> v0.15.0

I ran out of time today but I will fix this tomorrow morning if no one else beats me to it

Then all the tests pass

That's so weird. Do you have understanding of why sqllogictest 0.15.1 breaks these tests?

That's so weird. Do you have understanding of why sqllogictest 0.15.1 breaks these tests?

I think sqllogictest is right.

Some test records should be successfully executed, but they are erroneously declared as statement error.
Sqllogictest won’t check this scenario until 0.15.1.
For example:
https://github.com/apache/arrow-datafusion/blob/1a0542acbc01e5243471ae0fc3586c2f1f40013b/datafusion/core/tests/sqllogictests/test_files/aggregate.slt#L98-L100
This seems to have already been fixed by #6734.

I attempted to fix it but got blocked by the following record:
https://github.com/apache/arrow-datafusion/blob/1a0542acbc01e5243471ae0fc3586c2f1f40013b/datafusion/core/tests/sqllogictests/test_files/timestamps.slt#L1252-L1254
In the current version of Datafusion, this query can be executed successfully, but I think it should not be allowed to succeed.
It will fail to execute in PostgreSQL:

psql=> create table foo (val int, ts1 timestamp, ts2 timestamp, i interval);
CREATE TABLE

psql=> SELECT i - ts1 from FOO;
ERROR:  operator does not exist: interval - timestamp without time zone
LINE 1: SELECT i - ts1 from FOO;
                 ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.

@jonahgao

I attempted to fix it but got blocked by the following record:

Thanks @jonahgao -- I filed #7084 to track this issue

Navigation Menu

Repository navigation

CI failing on master: External error: query is expected to fail, but actually succeed: #7078

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CI failing on master: External error: query is expected to fail, but actually succeed: #7078

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

alamb commented on Jul 24, 2023

alamb commented on Jul 24, 2023

alamb commented on Jul 24, 2023

alamb commented on Jul 24, 2023

ozankabak commented on Jul 25, 2023

jonahgao commented on Jul 25, 2023

alamb commented on Jul 25, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions