Repository navigation
parquet-cascading Add Parquet Avro Scheme #1743
Description
At the moment using parquet-cascading writing Parquet files when using Avro as the object format is not possible as a Schema is not yet provided.
As part of this ticket, a ParquetAvroScheme will be introduced, and also a local.ParquetAvroScheme , so that from Cascading / Scalding developers can read / write and test in both --hdfs and --local mode Parquet-Avro files.
Reporter: Antonios Chalkiopoulos / @Antwnis
Assignee: Antonios Chalkiopoulos / @Antwnis
Note: This issue was originally created as PARQUET-198. Please see the migration documentation for further details.
Activity
asfimport commentedon Mar 4, 2015
Julien Le Dem / @julienledem:
There are two options for adding this support.
As Parquet-cascading will start pulling dependencies for all the parquet modules we can either:
CC @rdblue @isnotinvain
asfimport commentedon Mar 11, 2015
Alex Levenson / @isnotinvain:
I'd rather do sub-modules if we can – provided dependencies are not easily discovered by users.
asfimport commentedon Mar 11, 2015
Ryan Blue / @rdblue:
I agree, submodules sound like a better solution.
asfimport commentedon Mar 23, 2015
Andre Schumacher:
I added a first implementation neither based on submodules nor on provided dependencies (I needed something that works right now). I agree that probably it would be best to keep the respective Avro and Thrift dependencies separate, so maybe separate submodules could be the way to go.
Would be great if somebody would have a look at my PR: https://github.com/apache/incubator-parquet-mr/pull/151
asfimport commentedon Apr 3, 2015
Julien Le Dem / @julienledem:
It looks like there are 2 PR for this:
https://github.com/apache/incubator-parquet-mr/pull/122
https://github.com/apache/incubator-parquet-mr/pull/151
Does someone want to move it to a separate submodule?
asfimport commentedon Apr 3, 2015
Andre Schumacher:
I did not know about PR122 but it seems the approach is a bit different. In any case I will change my PR so it will be in a separate submodule.
asfimport commentedon Apr 5, 2015
Andre Schumacher:
Reshuffled the code into its own submodule and added a few more tests. Would be great if you could have a look.
asfimport commentedon Apr 23, 2015
Andre Schumacher:
New link to the PR due to the incubator graduation (congrats, btw): #151