feat: reduce CI time #919

Is your feature request related to a problem? Please describe.
Right now there are a lot of redundant or duplicated (core-clients) unit tests and the whole CI is long.

Describe the solution you would like
PySpark tests only its own API correctness, core tests algorithms logic, no useless tests matrices.

Component

• Scala Core Internal
• Scala API
• Spark Connect Plugin
• Infrastructure
• PySpark Classic
• PySpark Connect

Additional context

Are you planning on creating a PR?

• I'm willing to make a pull-request