You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Unable to parse the Mainframe copybook which has a COBOL datatype of BBBB which means empty spacesc #734
We are using CoBrix with PySpark and executing it on AWS EMR.
We have the EBCDIC file and it's corresponding copybook in the AWS S3 bucket. While trying to parse the EBCDIC file using the Copybook, we are getting an error.
Error message :
py4j.protocol.Py4jJavaError : An error occurred while calling o2021.loa : za.co.absa.cobrix.cobol.parser.exceptions.SyntaxErrorException : Syntax error in the copybook at line 29 : Invalid input 'BBBB' at position 29:45
We expected the Cobrix to successfully parse the EBCDIC file record column using the Cobybook which has this datatype of 'BBBB'
Context
PySpark Jar dependencies :
cobol-parser_2.12-2.6.7.jar
hadoop-lzo-0.4.3.jar
scodec-bits_2.12-1.1.12.jar
scodec-core_2.12-1.11.4.jar
spark-cobol_2.12-2.6.7.jar
Operating system: AWS EMR (Linux Image)
Copybook (if possible)
15 EL02-267-COLNAME-A
20 EL02-267-COLNAME-B
PIC X(19).
.........
.........
.........
20 EL02-267-COLNAME-C REDEFINES
EL02-267-COLNAME-D
PIC 9(06)BBBB. (This is what is causing the issue we suppose)
GP5WHB 20 FILLER pic X(285). CLEAN-UP
Attach a small data file that can help reproduce the issue, if possible : Need to check the feasibility due to confidentiality of the data. Will get back.
Yes, 'BBBB' is something Cobrix does not support mainly because we are not sure at the moment how to properly handle it.
This might be a relevant issue: #505
Does it work if you remove 'BBBB'? Does it produce the expected output in this case?
One query, could you advice on what could be a replacement for 'BBBB', I mean, is there any other Cobol datatype definition that could be analogous to the use-case of 'BBBB' and works with Cobrix too?
Please note, I am yet to try out your advice on removing the 'BBBB' and give a try. Sorry for the delay, will get back on that asap!
Since 'B' means just inserting spaces in the data representation of the number, and because Cobrix converts numbers to Spark native binary formats, 'B' should not need a replacement. We may eventually implement it so Cobrix ignores all 'B' in numbers. We haven't done it yet since we haven't encountered such PICs in our organization so we can't confirm that ignoring 'B's would be an expected behavior.
Once you confirm that removing 'B's from PICs produces correct output in numeric fields we are going to implement the support 'B's natively.
Hi @yruslan ,
Firstly apologies for the delay in my response as our Team was having multiple operational challenges to replicate the file. Please find below my observations on the 'BBBB':
Given a copybook as below :
20 col_a
PIC X(10).
20 col_b
PIC 9(06)BBBB.
Firstly , what we understood from our team is B stands for a blank space hence BBBB = 4 blank spaces.
When we removed the 'BBBB' from the copybook, the Cobrix was successfully able to parse our EBCDIC file and generate the output.
Here, col_a which is the Parent column is of 10 bytes and col_b which is the child is having 6 bytes + 4 blank spaces. So suppose, col_b value is 123456 + 4 spaces then after removing BBBB, the output contains only 123456 which means the 4 spaces got trimmed.
However what we did was after the output, we manually added 4 spaces to the column via our Spark code.
Also, we are trying to update the col_b definition to PIC 9(10) or X(10) after removing the 4 B's to completely occupy the 10 bytes without the Bs as Cobrix does not support it.
So to answer your query, the output post removing BBBB did help and we are able to proceed further. Thank you so much!
Based on the above observations, request you to kindly let us know if you will be adding support for the B's in future Cobrix releases so that we are aligned to it.
Hi @suryagits, thanks for the detailed description! It is very helpful.
Yes, I think the support for 'B's can be added to Cobtix eventually. Let's keep this issue open.
Just a couple of more questions in order to understand how Cobrix should interpret BBBs.
When you converted PIC 9(06)BBBB to PIC X(10), what happenned? Did the field ended up with 4 spaces are required, or the record size mismatched?
I guess, if BBBs mean spaces which are not part of the data itself, PIC X(6) might work better in this case. But, yes a special support in Cobrix is needed in order to insert spaces.
Thank you for your patience. Just wanted to update you that few more internal discussions are going on with regards to the expected behavior value of 'BBBB'. As soon as I get a conclusion from it, will update you back on the final roadmap.
Just to share my understanding, the intent of adding a blank space is purely from a business/ data formatting use-case. So if someone wants to interpret their data as 1234 with spaces in the end, that's when those B's are make sense.
I do not think if there is anyway to use the B's to add a space in the beginning or in the middle using the copybook definition (I may be wrong).
If Cobrix develops a capability to convert the B to a space , that would be the ultimate solution.
Actually when we tried converting PIC 9(06)BBBB to PIC X(10) and PIC 9(09). In both cases, suppose the input value is 123456, the output just remained 123456 (no extra spaces nor any other character) in the output Spark dataframe that was generated by Cobrix. I did not find any difference in the Cobrix generated output for both the copybook definitions.
We just went ahead with 9(10) since we were sure that the expected business value is purely numeric.
I hope this helps. Kindly let us know in case of any further queries.
Activity
yruslan commentedon Dec 27, 2024
Hi,
Yes, 'BBBB' is something Cobrix does not support mainly because we are not sure at the moment how to properly handle it.
This might be a relevant issue: #505
Does it work if you remove 'BBBB'? Does it produce the expected output in this case?
suryagits commentedon Dec 29, 2024
Hi @yruslan ,
Thank you so much for your response!
As adviced, I will try once by removing the 'BBBB' from my Copybook file , rerun the Cobrix program and get will back to you asap.
Thank you
suryagits commentedon Dec 31, 2024
Hi @yruslan ,
One query, could you advice on what could be a replacement for 'BBBB', I mean, is there any other Cobol datatype definition that could be analogous to the use-case of 'BBBB' and works with Cobrix too?
Please note, I am yet to try out your advice on removing the 'BBBB' and give a try. Sorry for the delay, will get back on that asap!
Thank you
yruslan commentedon Dec 31, 2024
Hi @suryagits,
Since 'B' means just inserting spaces in the data representation of the number, and because Cobrix converts numbers to Spark native binary formats, 'B' should not need a replacement. We may eventually implement it so Cobrix ignores all 'B' in numbers. We haven't done it yet since we haven't encountered such PICs in our organization so we can't confirm that ignoring 'B's would be an expected behavior.
Once you confirm that removing 'B's from PICs produces correct output in numeric fields we are going to implement the support 'B's natively.
suryagits commentedon Jan 31, 2025
Hi @yruslan ,
Firstly apologies for the delay in my response as our Team was having multiple operational challenges to replicate the file. Please find below my observations on the 'BBBB':
Given a copybook as below :
20 col_a
PIC X(10).
20 col_b
PIC 9(06)BBBB.
Firstly , what we understood from our team is B stands for a blank space hence BBBB = 4 blank spaces.
When we removed the 'BBBB' from the copybook, the Cobrix was successfully able to parse our EBCDIC file and generate the output.
Here, col_a which is the Parent column is of 10 bytes and col_b which is the child is having 6 bytes + 4 blank spaces. So suppose, col_b value is 123456 + 4 spaces then after removing BBBB, the output contains only 123456 which means the 4 spaces got trimmed.
However what we did was after the output, we manually added 4 spaces to the column via our Spark code.
Also, we are trying to update the col_b definition to PIC 9(10) or X(10) after removing the 4 B's to completely occupy the 10 bytes without the Bs as Cobrix does not support it.
So to answer your query, the output post removing BBBB did help and we are able to proceed further. Thank you so much!
Based on the above observations, request you to kindly let us know if you will be adding support for the B's in future Cobrix releases so that we are aligned to it.
Thank you
yruslan commentedon Jan 31, 2025
Hi @suryagits, thanks for the detailed description! It is very helpful.
Yes, I think the support for 'B's can be added to Cobtix eventually. Let's keep this issue open.
Just a couple of more questions in order to understand how Cobrix should interpret BBBs.
PIC 9(06)BBBBtoPIC X(10), what happenned? Did the field ended up with 4 spaces are required, or the record size mismatched?PIC X(6)might work better in this case. But, yes a special support in Cobrix is needed in order to insert spaces.suryagits commentedon Feb 12, 2025
Hi @yruslan ,
Thank you for your patience. Just wanted to update you that few more internal discussions are going on with regards to the expected behavior value of 'BBBB'. As soon as I get a conclusion from it, will update you back on the final roadmap.
Apologies for the delay..
Thank you
suryagits commentedon Mar 7, 2025
Hi @yruslan ,
Just to share my understanding, the intent of adding a blank space is purely from a business/ data formatting use-case. So if someone wants to interpret their data as 1234 with spaces in the end, that's when those B's are make sense.
I do not think if there is anyway to use the B's to add a space in the beginning or in the middle using the copybook definition (I may be wrong).
If Cobrix develops a capability to convert the B to a space , that would be the ultimate solution.
Actually when we tried converting
PIC 9(06)BBBBtoPIC X(10)andPIC 9(09). In both cases, suppose the input value is123456, the output just remained123456(no extra spaces nor any other character) in the output Spark dataframe that was generated by Cobrix. I did not find any difference in the Cobrix generated output for both the copybook definitions.We just went ahead with 9(10) since we were sure that the expected business value is purely numeric.
I hope this helps. Kindly let us know in case of any further queries.
Thank you
yruslan commentedon Mar 12, 2025
Hi @suryagits ,
999B999can be used to format100000as100 000.PIC X(10)outputs astringdata type whilePIC 9(09)produces aninteger.