Repository navigation
mawk 1.3.3 support (Debian/Ubuntu) #6
Description
It seems that the script is not mawk compatible. Running with test file:
{
"asd":1
}
Result:
./test.json: expected <string> but got <EOF> at input token 2
{ <<EOF>> " a s d "
invalid: ./test.json
expected <string> but got <EOF> at input token 2
{ <<EOF>> " a s d "
System info:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial
AWK: ii mawk 1.3.3-17ubuntu2
It works as expected with gawk.
Activity
step- commentedon Jan 31, 2017
Thanks for your report. I can't reproduce this issue, at the moment.
How did you start JSON.awk? Can you please post the output of
xxd test.json? Can you try with mawk 1.3.4?step- commentedon Jan 31, 2017
Packages for Ubuntu here: ftp://invisible-island.net/DEBS/
linked from http://invisible-island.net/mawk/
urban-1 commentedon Jan 31, 2017
You are right! I downloaded
mawk 1.3.4 20161120and everything worked!It seems that Debian and Ubuntu are stuck with mawk 1.3.3:
... which is now 21 years old!!! :)
I don't think that it worth fixing... but you can leave it here for visibility - whatever you prefer
[-]mawk support[/-][+]mawk 1.3.3 support (Debian/Ubuntu)[/+]step- commentedon Jan 31, 2017
Thank you for reporting your success. I will start a FAQ and link this page.
Link issue #6.
step- commentedon Jan 31, 2017
I found another case where mawk 1.3.4 20150503 doesn't work. It works with gawk. Can you please run the same test with 1.3.4 20161120 and report results here? Thanks.
urban-1 commentedon Jan 31, 2017
I do get the same:
Is it the space before
:?step- commentedon Jan 31, 2017
It seems to be a buffering mode issue that can affect all awk variants, including gawk, when JSON.awk reads json data from stdin. Recall that stdin is used to feed the list of input files to JSON.awk:
'\n' produces a blank line, which marks the end of the filename argument list. To pipe data into JSON.awk we tell it that the input file is "-", then write json data to JSON.awk's stdin.
(the two forms are equivalent)
Note that this time '\n' also marks the beginning of json data, so a single stream, stdin, carries two kinds of data. This can work without confusion only if I/O is line-buffered because JSON.awk needs to break reads on a line boundary to be able to recognize where json data begins.
When I/O is line-buffered, JSON.awk starts reading json data at offset 0 after the first '\n\n' in the input stream. That's correct.
But when I/O isn't line-buffered, JSON.awk is likely to start reading json data at a positive offset after the first '\n\n'. The offset value depends on the read buffer size. In other words, json data reading starts at the wrong offset, so the json looks malformed. That's what happens, I think.
I did some tests and managed to mitigate this issue by using
stdbuf -oLin the left side command of the shell pipe, but the effectiveness depends on which specific commands are involved, and on which awk variant is being used. I don't think it's a good solution.A better solution is to change the usage syntax by providing the list of filename arguments on the command-line instead of stdin. That should eliminate line-buffering concerns altogether.
I need to think over why I designed this quirky stdin mechanism in the first place. Probably the project that used JSON.awk at the time required it. But now it's an issue.
step- commentedon Feb 1, 2017
I think I will change it this way:
-v CMDLINE=1is given then get file name arguments from the command line instead of stdin.Pros: easy to code and reliable.
Cons: non-traditional argument passing is the default.
I can't think of a reliable way to switch automatically based on inspecting the input stream.
urban-1 commentedon Feb 2, 2017
Hi @step-
I think your solution will work. You can also make the decision based on
ARGCbut I don't know if that is compatible with allawkversions.step- commentedon Feb 11, 2017
Command-line arguments migrated to issue #8.