Storing a Linux shell command into an AWK variable

[Mr. Man-wai Chang]

How could I store the following BASH command into an Awk variable?

**** begin *****
site_survey 2>&1 | \
sed -e 's/^./{/' \
-e 's/\] SSID\[/\} SSID\{/' \
-e 's/\] channel\[/} channel{/' \
-e 's/\] enc\[/} enc{/' \
-e 's/.$/\}/'
**** end *****

[Kees Nuyt]

On Thu, 29 Feb 2024 20:18:56 +0800, "Mr. Man-wai Chang"

Post by Mr. Man-wai Chang
How could I store the following BASH command into an Awk variable?
**** begin *****
site_survey 2>&1 | \
sed -e 's/^./{/' \
-e 's/\] SSID\[/\} SSID\{/' \
-e 's/\] channel\[/} channel{/' \
-e 's/\] enc\[/} enc{/' \
-e 's/.$/\}/'
**** end *****

Try:
myvar = "site_survey 2>&1 | \
sed -e 's/^./{/' \
-e 's/\\] SSID\\[/\\} SSID\\{/' \
-e 's/\\] channel\\[/} channel{/' \
-e 's/\\] enc\\[/} enc{/' \
-e 's/.$/\\}/'"

or, if you insist on preserving linebreaks:

myvar2 = "site_survey 2>&1 | \\\n" \
"sed -e 's/^./{/' \\\n" \
"-e 's/\\] SSID\\[/\\} SSID\\{/' \\\n" \
"-e 's/\\] channel\\[/} channel{/' \\\n" \
"-e 's/\\] enc\\[/} enc{/' \\\n" \
"-e 's/.$/\\}/'"

[Mr. Man-wai Chang]

Post by Kees Nuyt
On Thu, 29 Feb 2024 20:18:56 +0800, "Mr. Man-wai Chang"

Post by Mr. Man-wai Chang
How could I store the following BASH command into an Awk variable?
**** begin *****
site_survey 2>&1 | \
sed -e 's/^./{/' \
-e 's/\] SSID\[/\} SSID\{/' \
-e 's/\] channel\[/} channel{/' \
-e 's/\] enc\[/} enc{/' \
-e 's/.$/\}/'
**** end *****

myvar = "site_survey 2>&1 | \
sed -e 's/^./{/' \
-e 's/\\] SSID\\[/\\} SSID\\{/' \
-e 's/\\] channel\\[/} channel{/' \
-e 's/\\] enc\\[/} enc{/' \
-e 's/.$/\\}/'"
myvar2 = "site_survey 2>&1 | \\\n" \
"sed -e 's/^./{/' \\\n" \
"-e 's/\\] SSID\\[/\\} SSID\\{/' \\\n" \
"-e 's/\\] channel\\[/} channel{/' \\\n" \
"-e 's/\\] enc\\[/} enc{/' \\\n" \
"-e 's/.$/\\}/'"

Thanks!

Because of the call to sed within the Awk vaariable, need to escape:

1. "]" to "\\]"
2. "[" to "\\["
3. "'" to "'\''"

Absolutely horrifying. :)

This is what I end up with:

cmd="site_survey 2>&1 | \
sed -e '\''s/^./{/'\'' \
-e '\''s/\\] SSID\\[/\} SSID\{/'\'' \
-e '\''s/\\] channel\\[/} channel{/'\'' \
-e '\''s/\\] enc\\[/} enc{/'\'' \
-e '\''s/.$/\}/'\'' ";

[Janis Papanagnou]

Post by Mr. Man-wai Chang

Post by Kees Nuyt
On Thu, 29 Feb 2024 20:18:56 +0800, "Mr. Man-wai Chang"

Post by Mr. Man-wai Chang
How could I store the following BASH command into an Awk variable?
**** begin *****
site_survey 2>&1 | \
sed -e 's/^./{/' \
-e 's/\] SSID\[/\} SSID\{/' \
-e 's/\] channel\[/} channel{/' \
-e 's/\] enc\[/} enc{/' \
-e 's/.$/\}/'
**** end *****

[...]

Post by Mr. Man-wai Chang
Thanks!
1. "]" to "\\]"
2. "[" to "\\["
3. "'" to "'\''"
Absolutely horrifying. :)
cmd="site_survey 2>&1 | \
sed -e '\''s/^./{/'\'' \
-e '\''s/\\] SSID\\[/\} SSID\{/'\'' \
-e '\''s/\\] channel\\[/} channel{/'\'' \
-e '\''s/\\] enc\\[/} enc{/'\'' \
-e '\''s/.$/\}/'\'' ";

Whenever I see this sort of question and the attempted "solution"
I'm tempted to ask whether your approach might be not be the best;
there's quite some indications. - Mind to elaborate on the task
(not technically, but as seen from a higher perspective)?

Janis

[Mr. Man-wai Chang]

Post by Janis Papanagnou

Post by Mr. Man-wai Chang

Post by Kees Nuyt
On Thu, 29 Feb 2024 20:18:56 +0800, "Mr. Man-wai Chang"

Post by Mr. Man-wai Chang
How could I store the following BASH command into an Awk variable?
**** begin *****
site_survey 2>&1 | \
sed -e 's/^./{/' \
-e 's/\] SSID\[/\} SSID\{/' \
-e 's/\] channel\[/} channel{/' \
-e 's/\] enc\[/} enc{/' \
-e 's/.$/\}/'
**** end *****

[...]

Post by Mr. Man-wai Chang
Thanks!
1. "]" to "\\]"
2. "[" to "\\["
3. "'" to "'\''"
Absolutely horrifying. :)
cmd="site_survey 2>&1 | \
sed -e '\''s/^./{/'\'' \
-e '\''s/\\] SSID\\[/\} SSID\{/'\'' \
-e '\''s/\\] channel\\[/} channel{/'\'' \
-e '\''s/\\] enc\\[/} enc{/'\'' \
-e '\''s/.$/\}/'\'' ";

Whenever I see this sort of question and the attempted "solution"
I'm tempted to ask whether your approach might be not be the best;
there's quite some indications. - Mind to elaborate on the task
(not technically, but as seen from a higher perspective)?

The site_survey command vomits lines:

[ 8] SSID[ [HOME]] BSSID[04:9F:xx:xx:xx:xx] channel[ 6]
frequency[2437] numsta[1] rssi[-63] noise[-75] beacon[98] cap[1411]
dtim[0] rate[450] enc[Group-AES-CCMP CCMP PSK2 ]

Just wanna change all field delimiters, which are square brackets, to
curly braces within an Awk program loop.

I agree it might be a silly idea. But this is my first Awk programming
task. :)

[Janis Papanagnou]

Post by Mr. Man-wai Chang

Post by Janis Papanagnou

Post by Mr. Man-wai Chang

Post by Kees Nuyt
On Thu, 29 Feb 2024 20:18:56 +0800, "Mr. Man-wai Chang"

Post by Mr. Man-wai Chang
How could I store the following BASH command into an Awk variable?
**** begin *****
site_survey 2>&1 | \
sed -e 's/^./{/' \
-e 's/\] SSID\[/\} SSID\{/' \
-e 's/\] channel\[/} channel{/' \
-e 's/\] enc\[/} enc{/' \
-e 's/.$/\}/'
**** end *****

[...]

Post by Mr. Man-wai Chang
Thanks!
1. "]" to "\\]"
2. "[" to "\\["
3. "'" to "'\''"
Absolutely horrifying. :)
cmd="site_survey 2>&1 | \
sed -e '\''s/^./{/'\'' \
-e '\''s/\\] SSID\\[/\} SSID\{/'\'' \
-e '\''s/\\] channel\\[/} channel{/'\'' \
-e '\''s/\\] enc\\[/} enc{/'\'' \
-e '\''s/.$/\}/'\'' ";

Whenever I see this sort of question and the attempted "solution"
I'm tempted to ask whether your approach might be not be the best;
there's quite some indications. - Mind to elaborate on the task
(not technically, but as seen from a higher perspective)?

[ 8] SSID[ [HOME]] BSSID[04:9F:xx:xx:xx:xx] channel[ 6]
frequency[2437] numsta[1] rssi[-63] noise[-75] beacon[98] cap[1411]
dtim[0] rate[450] enc[Group-AES-CCMP CCMP PSK2 ]
Just wanna change all field delimiters, which are square brackets, to
curly braces within an Awk program loop.
I agree it might be a silly idea. But this is my first Awk programming
task. :)

Not necessarily. But I want to explain the reason for my question.

Composing a shell command in Awk typically has two (but maybe more)
application cases; one is to construct the shell commands to invoke
them from within Awk with the system() function, or to print them
so that you can store them in a file of pipe them into a shell for
direct execution.

It _might_ be better to do these commands in shell instead. (But I
still have not enough information to suggest the best approach.)

You noticed that substitutions with Awk might get nasty if escapes
are necessary for many characters. You may want to reduce the
number of characters to escape by changing the regular expression
that your Sed is using; it can be simplified (by 'or'ing the three
keywords, and likely by also incorporating the last sed expression.

An option might also be to not use Sed here but have an Awk do the
substitutions. The escaping complexity can also be reduced if you
put the Sed (or Awk) substitution commands into a file so that you
don't need to do all that ' quoting and escaping.

Janis

[Mr. Man-wai Chang]

Post by Janis Papanagnou
Composing a shell command in Awk typically has two (but maybe more)
application cases; one is to construct the shell commands to invoke
them from within Awk with the system() function, or to print them
so that you can store them in a file of pipe them into a shell for
direct execution.
It _might_ be better to do these commands in shell instead. (But I
still have not enough information to suggest the best approach.)
You noticed that substitutions with Awk might get nasty if escapes
are necessary for many characters. You may want to reduce the
number of characters to escape by changing the regular expression
that your Sed is using; it can be simplified (by 'or'ing the three
keywords, and likely by also incorporating the last sed expression.

Because the shell command is to be looped inside the Awk script with
data accumulation and calculation!

Post by Janis Papanagnou
An option might also be to not use Sed here but have an Awk do the
substitutions. The escaping complexity can also be reduced if you
put the Sed (or Awk) substitution commands into a file so that you
don't need to do all that ' quoting and escaping.

That will involve more programming and testing. But I agree with yuo.

[Janis Papanagnou]

Post by Mr. Man-wai Chang

Post by Janis Papanagnou
Composing a shell command in Awk typically has two (but maybe more)
application cases; one is to construct the shell commands to invoke
them from within Awk with the system() function, or to print them
so that you can store them in a file of pipe them into a shell for
direct execution.
It _might_ be better to do these commands in shell instead. (But I
still have not enough information to suggest the best approach.)
[...]

Because the shell command is to be looped inside the Awk script with
data accumulation and calculation!

Sorry, that description is unclear to me.
(Yet it still sounds to me as you've not chosen
the appropriate separation between Shell and Awk.)

Janis

[Mr. Man-wai Chang]

Post by Janis Papanagnou
Sorry, that description is unclear to me.
(Yet it still sounds to me as you've not chosen
the appropriate separation between Shell and Awk.)

Back to the site_survey example.

You want to count bssid after each call to site_survey, how could you
not loop the output of site_survey inside Awk? Unless you have a way to
save the counts outside Awk?

[Kenny McCormack]

Post by Mr. Man-wai Chang

Post by Janis Papanagnou
Sorry, that description is unclear to me.
(Yet it still sounds to me as you've not chosen
the appropriate separation between Shell and Awk.)

Back to the site_survey example.
You want to count bssid after each call to site_survey, how could you
not loop the output of site_survey inside Awk? Unless you have a way to
save the counts outside Awk?

Just to get some sort of baseline example, is there any reason you can't do
it like this:

$ site_survey | awk '...'

where the awk program (the "..." above) does whatever manipulations you
need; i.e., all the substitutions you had previously been doing in "sed".

Note: The real problem here is that none of the people who are trying to
help you (*) have any real idea what your actual program looks like.

(*) A group I now seem to have added myself to.

[Mr. Man-wai Chang]

Post by Kenny McCormack

Post by Mr. Man-wai Chang
Back to the site_survey example.
You want to count bssid after each call to site_survey, how could you
not loop the output of site_survey inside Awk? Unless you have a way to
save the counts outside Awk?

Just to get some sort of baseline example, is there any reason you can't do
$ site_survey | awk '...'
where the awk program (the "..." above) does whatever manipulations you
need; i.e., all the substitutions you had previously been doing in "sed".

I want to count the bssid after each call to site_survey, accumulative!
To do that outside of Awk '...', you need a place to store the counts
from last call to site_survey.
#
# something like this:
#
n[bssid]=0
for (;;) {
count_bssid(site_survey, n)
endfor
print n[bssid]

[Janis Papanagnou]

Post by Mr. Man-wai Chang

Post by Kenny McCormack

Post by Mr. Man-wai Chang
Back to the site_survey example.
You want to count bssid after each call to site_survey, how could you
not loop the output of site_survey inside Awk? Unless you have a way to
save the counts outside Awk?

Just to get some sort of baseline example, is there any reason you can't do
$ site_survey | awk '...'
where the awk program (the "..." above) does whatever manipulations you
need; i.e., all the substitutions you had previously been doing in "sed".

I want to count the bssid after each call to site_survey, accumulative!

I don't see how the previous construction of a sed command helps here.
It's still fuzzy to me what you want, so below I can give you some
informal hints and once you've chosen which way you want to go you
can come back with more specific questions.


One general way of doing such things is (informally written)

data_source | awk '{ x=extract-data() ; c+=x } END { print c }'

where extract-data is some regular expression match of the data format
you are parsing, and
where data-source may be a single call of a program like 'site_survey',
or a shell-loop of program calls like

while some-condition
do
site_survey "${parameters}"
done | awk '...'

Post by Mr. Man-wai Chang
To do that outside of Awk '...', you need a place to store the counts
from last call to site_survey.

This is also possible. As in

c=0
while some-condition
do
c=$(( c + $(site_survey "${parameters}" | extract-data) ))
done
echo $c

where extract-data is some tool (sed, awk, etc.) to get the data from
the site_survey command.

Post by Mr. Man-wai Chang
#
#
n[bssid]=0
for (;;) {
count_bssid(site_survey, n)
endfor
print n[bssid]

Janis

[Ben Bacarisse]

Post by Janis Papanagnou

Post by Mr. Man-wai Chang

Post by Kenny McCormack

Post by Mr. Man-wai Chang
Back to the site_survey example.
You want to count bssid after each call to site_survey, how could you
not loop the output of site_survey inside Awk? Unless you have a way to
save the counts outside Awk?

Just to get some sort of baseline example, is there any reason you can't do
$ site_survey | awk '...'
where the awk program (the "..." above) does whatever manipulations you
need; i.e., all the substitutions you had previously been doing in "sed".

I want to count the bssid after each call to site_survey, accumulative!

I don't see how the previous construction of a sed command helps here.
It's still fuzzy to me what you want, so below I can give you some
informal hints and once you've chosen which way you want to go you
can come back with more specific questions.
One general way of doing such things is (informally written)
data_source | awk '{ x=extract-data() ; c+=x } END { print c }'

As you say it's not very clear what is needed, but I think the key point
is that the data must be accumulated. The basic pattern is probably
then more like

data_source | awk \
'BEGIN { d = read_saved_data(); } { x=extract-data() ; d[something] += x }
END { write_data(d); }'

though there may be data output as well. In fact it's possible that the
simplest way to store the accumulating data is to output it and use tee:

data_source | awk \
'BEGIN { d = read_saved_data(); } { x=extract-data() ; d[something] += x }
END { print lots of stats...; }' | tee saved_data

[Mr. Man-wai Chang]

Post by Ben Bacarisse
As you say it's not very clear what is needed, but I think the key point
is that the data must be accumulated. The basic pattern is probably
then more like
data_source | awk \
'BEGIN { d = read_saved_data(); } { x=extract-data() ; d[something] += x }
END { write_data(d); }'
though there may be data output as well. In fact it's possible that the
data_source | awk \
'BEGIN { d = read_saved_data(); } { x=extract-data() ;

d[something] += x }

Post by Ben Bacarisse
END { print lots of stats...; }' | tee saved_data

That's exactly what I was trying to do. How do you store and restore the
array of counts[bssid]? Flatten the array into Bash variables?

I think it's not too bad to do everything within the Awk script, given
Awk's capabilties.

[Ben Bacarisse]

Post by Mr. Man-wai Chang

Post by Ben Bacarisse
As you say it's not very clear what is needed, but I think the key point
is that the data must be accumulated. The basic pattern is probably
then more like
data_source | awk \
'BEGIN { d = read_saved_data(); } { x=extract-data() ; d[something] += x }
END { write_data(d); }'
though there may be data output as well. In fact it's possible that the
data_source | awk \
'BEGIN { d = read_saved_data(); } { x=extract-data() ; d[something] +=

x }

Post by Ben Bacarisse
END { print lots of stats...; }' | tee saved_data

That's exactly what I was trying to do. How do you store and restore the
array of counts[bssid]? Flatten the array into Bash variables?

I would write it to a file. That's what I was trying to suggest with
the pseudo code.

Post by Mr. Man-wai Chang
I think it's not too bad to do everything within the Awk script, given
Awk's capabilties.

Yes, but in the old days (when I had to do this sort of stuff more
often) I would usually switch to Perl when I started to have to manage
files other than the standard input and standard output. But that's
because I knew both so the switch was simple.

[Mr. Man-wai Chang]

Post by Kenny McCormack
Note: The real problem here is that none of the people who are trying to
help you (*) have any real idea what your actual program looks like.
(*) A group I now seem to have added myself to.

I just wanna find out whether there is an easier or smarter solution
using Awk ONLY. Pure Awk programming. :)

[Mr. Man-wai Chang]

Post by Janis Papanagnou
An option might also be to not use Sed here but have an Awk do the
substitutions. The escaping complexity can also be reduced if you
put the Sed (or Awk) substitution commands into a file so that you
don't need to do all that ' quoting and escaping.

That's an intermediate step I took before I figured out the complicated
character escaping stuffs. I did put the command into a Ash shell script
first. ;)

For your curiosity, here is the script I am trying to enhance:

DD-WRT :: View topic - Wireless scanner for linux
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=34924

It's written by someone else over 10 years ago, and I have no control
over the site_survey command. It will be easier if the command could
allow users to specify the output format and delimiters via optional
command-line options.

[Ed Morton]

Post by Mr. Man-wai Chang
How could I store the following BASH command into an Awk variable?
**** begin *****
site_survey 2>&1 | \
sed -e 's/^./{/' \
-e 's/\] SSID\[/\} SSID\{/' \
-e 's/\] channel\[/} channel{/' \
-e 's/\] enc\[/} enc{/' \
-e 's/.$/\}/'
**** end *****

Do not do this. You wouldn't do it in shell (see
https://mywiki.wooledge.org/BashFAQ/050) and you definitely must not do
this in awk as that'll have all of the issues associated with storing
code in a shell variable PLUS additional issues of robustness and
efficiency related to spawning a subshell to call an external tool from awk.

Whatever it is you're trying to do, post a new question with a minimal
complete, verifiable example (see
https://stackoverflow.com/help/minimal-reproducible-example for what
that means) so we can help you do it the right way.

Ed.

[Mr. Man-wai Chang]

Post by Ed Morton
Do not do this. You wouldn't do it in shell (see
https://mywiki.wooledge.org/BashFAQ/050) and you definitely must not do
this in awk as that'll have all of the issues associated with storing
code in a shell variable PLUS additional issues of robustness and
efficiency related to spawning a subshell to call an external tool from awk.

I undertsand the limitations of Awk. But I was wondering about how far I
could go. :)

Post by Ed Morton
Whatever it is you're trying to do, post a new question with a minimal
complete, verifiable example (see
https://stackoverflow.com/help/minimal-reproducible-example for what
that means) so we can help you do it the right way.

Already done so. My question in the end was more about escaping
special/reserved symbols.

[Kenny McCormack]

In article <usc4tk$10akp$***@toylet.eternal-september.org>,
Mr. Man-wai Chang <***@gmail.com> wrote:
...

Post by Mr. Man-wai Chang
Already done so. My question in the end was more about escaping
special/reserved symbols.

Well, not really.

At this point, I still don't think anyone else reading/posting to this
thread has any real idea what you are actually trying to do.

Now, it is certainly your right not to tell us, and we don't really have
any right to insist that you do, but it does limit the amount of help we
can provide. FWIW (and that might not be much), I think most of us are
having a reaction of "Whatever it is you're trying to do, this is not the
right way to do it".

Now, to be fair, you did give a slight hint as to what this was really
about, somewhere upthread, where you indicated that this was a program that
you got from someone/somewhere on the net and you just needed to modify it
slightly to make it do what you wanted. In that case, an "I just need help
with one simple thing" type answer might actually be appropriate. Again
FWIW, I think most of the help-givers on this thread have assumed that this
was your own program and that you are developing it from scratch.

[Janis Papanagnou]

Post by Kenny McCormack
...

Post by Mr. Man-wai Chang
Already done so. My question in the end was more about escaping
special/reserved symbols.

Well, not really.
At this point, I still don't think anyone else reading/posting to this
thread has any real idea what you are actually trying to do.
Now, it is certainly your right not to tell us, and we don't really have
any right to insist that you do, but it does limit the amount of help we
can provide. FWIW (and that might not be much), I think most of us are
having a reaction of "Whatever it is you're trying to do, this is not the
right way to do it".
Now, to be fair, you did give a slight hint as to what this was really
about, somewhere upthread, where you indicated that this was a program that
you got from someone/somewhere on the net and you just needed to modify it
slightly to make it do what you wanted. In that case, an "I just need help
with one simple thing" type answer might actually be appropriate. Again
FWIW, I think most of the help-givers on this thread have assumed that this
was your own program and that you are developing it from scratch.

I second what you wrote. Given that it's another person's tool
it might be advantageous to rewrite that tool if the original
code already had chosen the wrong partitioning between shell
and awk. The "one simple thing" might turn out to not be that
simple if the wrong approach had been chosen. But, yes, that's
the OP's decision.

Janis

[Mr. Man-wai Chang]

Post by Kenny McCormack
Now, it is certainly your right not to tell us, and we don't really have
any right to insist that you do, but it does limit the amount of help we
can provide. FWIW (and that might not be much), I think most of us are
having a reaction of "Whatever it is you're trying to do, this is not the
right way to do it".
Now, to be fair, you did give a slight hint as to what this was really
about, somewhere upthread, where you indicated that this was a program that
you got from someone/somewhere on the net and you just needed to modify it
slightly to make it do what you wanted. In that case, an "I just need help
with one simple thing" type answer might actually be appropriate. Again
FWIW, I think most of the help-givers on this thread have assumed that this
was your own program and that you are developing it from scratch.

I understand and thank all the suggestions here.

[Ed Morton]

Post by Mr. Man-wai Chang

Post by Ed Morton
Do not do this. You wouldn't do it in shell (see
https://mywiki.wooledge.org/BashFAQ/050) and you definitely must not do
this in awk as that'll have all of the issues associated with storing
code in a shell variable PLUS additional issues of robustness and
efficiency related to spawning a subshell to call an external tool from awk.

I undertsand the limitations of Awk. But I was wondering about how far I
could go. :)

It's not a limitation of awk any more than trying to dig a ditch with a
toothbrush would uncover a limitation of the toothbrush, it's just a
matter of using the right tool for the job and awk is not a tool to
sequence calls to other tools - that's what a shell is for.

Post by Mr. Man-wai Chang

Post by Ed Morton
Whatever it is you're trying to do, post a new question with a minimal
complete, verifiable example (see
https://stackoverflow.com/help/minimal-reproducible-example for what
that means) so we can help you do it the right way.

Already done so. My question in the end was more about escaping
special/reserved symbols.

No, there is no MCVE anywhere in this thread. Any help you think you're
getting is based on guesses and assumptions and any implementation
suggestions you got would be similar to getting instructions if you
asked how to load a gun while it's pointed at your foot rather than just
being told "don't do that" so treat everything with a large pinch of
salt as we simply cannot know what it is you're really trying to do
given what you've shown us so far.

Ed.