Reading mysql data stream into array
Hi folks, I am going nuts trying to read the output of a mysql select statement into an array. I have consulted bashfaq and other sources and tried various approaches, but every jump was somewhat short, it seems. This is what I tried (bash is 3.2.xx, I tried on MacOS X and Debian Lenny): while read -r; do messag[i++]=$REPLY; done < <(mysql select statement)# this is from BashFAQ Nr. 5 - is this possible with bash 3.2.x at all? this got me complains about unexpected "<" and such $(mysql select command) | while read line; do messag+=($line); done# this basically works, but counting up the array index does not I also tried ... do messag[i++]=($line) # -bash: messag[i++]: cannot assign list to array member () ... do messag[i]=$line && i=$(($i+1)) ... do messag[$i]=$line && i=$(($i+1)) It never results in every line of the mysql output becoming an array element. I really tried finding out myself, but I am stuck, so please can someone point me to what I do wrong? Thanks in advance, Dirk
Re: Reading mysql data stream into array
Am 06.05.11 18:12, schrieb DJ Mills: On Fri, May 6, 2011 at 12:07 PM, Dirk wrote: Hi folks, I am going nuts trying to read the output of a mysql select statement into an array. I have consulted bashfaq and other sources and tried various approaches, but every jump was somewhat short, it seems. This is what I tried (bash is 3.2.xx, I tried on MacOS X and Debian Lenny): while read -r; do messag[i++]=$REPLY; done< <(mysql select statement) # this is from BashFAQ Nr. 5 - is this possible with bash 3.2.x at all? this got me complains about unexpected "<" and such $(mysql select command) | while read line; do messag+=($line); done # this basically works, but counting up the array index does not I also tried ... do messag[i++]=($line) # -bash: messag[i++]: cannot assign list to array member () ... do messag[i]=$line&& i=$(($i+1)) ... do messag[$i]=$line&& i=$(($i+1)) It never results in every line of the mysql output becoming an array element. I really tried finding out myself, but I am stuck, so please can someone point me to what I do wrong? Thanks in advance, Dirk 'this got me complaints about unexpected "<" and such' You generally want to give specific errors. It almost sounds like you may be using sh, and not bash Is the first line of your script #!/bin/sh, or #!/bin/bash? I should have expected it to be something stupid like that. Thank you! I was fooled by "echo $SHELL" returning "/bin/bash", but when I manually call "/bin/bash" before running the above command (it is not a script yet), it works. What the heck is going on here ... (!xmpf) Thanks for your fast help! Dirk
parenthesised regular expressions and non-greedy operator ? - non standard bash behaviour
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu'
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL
-DHAVE_CONFIG_H -I. -I../. -I.././include -I.././lib -Wdate-time
-D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector-strong -Wformat
-Werror=format-security -Wall -no-pie -Wno-parentheses -Wno-format-security
uname output: Linux dilbert 4.10.0-41-generic #45~16.04.1-Ubuntu SMP Fri Nov 24
15:06:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-pc-linux-gnu
Bash Version: 4.4
Patch Level: 12
Release Status: release
Description:
I'm sanitising urls from advertisement crap. As described below I'm getting a
wrong resolution of parenthesised expression defined with non-greedy operator
'?'.
The test url is:
http://toolbox.contentspread.net/container/medimops/track/xx.dyn?csRdu=https://www.medimops.de/?anid=M99&cl=details&wdm=M99&utm_source=CRM&utm_medium=email&utm_campaign=OS
The regular expression is:
https?:\/\/toolbox.contentspread.net\/(.*?)=(.+?)&.*
As I understand the specification and verified with 'visual regexp' and
https://regex101.com/ the result should be:
1 â container/medimops/track/xx.dyn?csRdu
2 â https://www.medimops.de/?anid=M99
Running the script below I got instead:
1 â
container/medimops/track/xx.dyn?csRdu=https://www.medimops.de/?anid=M99&cl=details&wdm=M99&utm_source=CRM&utm_medium
2 â email
Repeat-By:
Test script:
#!/bin/bash
url='http://toolbox.contentspread.net/container/medimops/track/xx.dyn?csRdu=https://www.medimops.de/?anid=M99&cl=details&wdm=M99&utm_source=CRM&utm_medium=email&utm_campaign=OS'
re='https?:\/\/toolbox.contentspread.net\/(.*?)=(.+?)&.*'
if [[ ${url} =~ ${re} ]]
then
echo "0 â ${BASH_REMATCH[0]}"
echo "1 â ${BASH_REMATCH[1]}"
echo "2 â ${BASH_REMATCH[2]}"
fi
misleading text in `help declare` leads to serious wrong translations (e.g. German)
Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu' -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I../. -I.././include -I.././lib -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -no-pie -Wno-parentheses -Wno-format-security uname output: Linux -- CENSORED for security reasons! Machine Type: x86_64-pc-linux-gnu Bash Version: 4.4 Patch Level: 12 Release Status: release Description: See `help declare`: > -l to convert NAMEs to lower case on assignment > -u to convert NAMEs to upper case on assignment Is translated in German locale to: > -l Konvertiert die Variabennmamen in Kleinbuchstaben. > -u Konvertiert die Variablennamen in GroÃbuchstaben. Translating this back to English is: > -l Converts the name of the variable to lowercase characters. > -u Converts the name of the variable to uppercase characters. Fix: Use the following text: > -l to convert the content of NAMEs to lowercase characters > -l to convert the content of NAMEs to uppercase characters Check all supported languages for similar buggy translations.
ABOUT-NLS links are outdated
Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gnu' -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I../. -I.././include -I.././lib -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -no-pie -Wno-parentheses -Wno-format-security uname output: Linux dilbert 4.15.0-24-generic #26~16.04.1-Ubuntu SMP Fri Jun 15 14:35:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Machine Type: x86_64-pc-linux-gnu Bash Version: 4.4 Patch Level: 12 Release Status: release Description: ABOUT-NLS is outdated â especially http://www.iro.umontreal.ca/contrib/po/HTML/ is not available any more. So how can I find out where to report correction of such translations as the German âhelp evalâ with 2 misspelled words? eval: eval [Argument ...] Führe die Argumente als Shell Kommando aus. Fügt die Argumente zu einer Zeichenkette zusammen und verwendet das Ergebnis als Eingebe in eine Shell, welche die enthaltenen ^^^ Eingabe Kommandos ausführt. Rückgabewert: Der Status des Kommandoe oder Erfolg wenn das Kommando leer war. Kommandos
bash sockets: printf \x0a does TCP fragmentation
Hello there, we discovered a strange phenomenon in the project testssl.sh: After opening a TCP socket with a fd (here: 5), when writing to it, it seems that printf -- "$data" >&5 2>/dev/null does not do what it is intended. "$data" is a ClientHello like '\x16\x03\x01\x2\x00\x01\x00\x1\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0a\...' Each \x0a like the last one causes a new TCP fragment to begin which can be easily spotted when using wireshark while running e.g. testssl.sh --assume-http -p testssl.sh Starting from the SSLv3 ClientHello the first reassembled packet ends with 0a. See also discussion @ https://github.com/drwetter/testssl.sh/pull/1113. One would assume that a bash socket connection cannot influence the TCP fragmentation but obviously it does. This behavior has a performance penalty and other strange effects, e.g. if the first segment is really small, some devices reject the ClientHello. If there's a workaround, please let me know. (tried to add "%b" with no effect). Otherwise I believe it's a bug. Cheers, Dirk PS: Would ulimit -b help?
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 1:34 AM, Chet Ramey wrote: > On 9/21/18 4:13 PM, dirk+b...@testssl.sh wrote: >> >> Hello there, >> >> we discovered a strange phenomenon in the project testssl.sh: >> >> After opening a TCP socket with a fd (here: 5), when writing to it, >> it seems that >> >> printf -- "$data" >&5 2>/dev/null >> >> does not do what it is intended. "$data" is a ClientHello like >> >> '\x16\x03\x01\x2\x00\x01\x00\x1\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0a\...' >> >> Each \x0a like the last one causes a new TCP fragment to begin which can be >> easily >> spotted when using wireshark while running e.g. > > Newline? It's probably that stdout is line-buffered and the newline causes > a flush, which results in a write(2). Anything one can do on the level of bash or non-syscall land? What about ulimit -b ? >> If there's a workaround, please let me know. (tried to add "%b" with no >> effect). Otherwise I believe it's a bug. > > How? Does the emitted output not correspond to what's passed to printf > in some way? "\x0a" is a legitimate byte which is send from time to time over the socket. It happens if the record layer is e.g. 522 bytes (\x02\x0a), if a standard cipher is included in the handshake like (\xc0\x0a) or DES-CBC3-SHA (\x00\x0a) ECDHE-ECDSA-AES256-SHA or at any other occasion. Everything works as expected and like a charm for years now -- the only thing isthe underlying TCP fragmentation which is caused as a side effect by sending \x0a. As indicated a few servers under certain condition can't cope with it asif the TCP first segment is too small they don't treat this as ClientHello and just drop the packet, see thread @ https://github.com/drwetter/testssl.sh/pull/1113, specifically the hint wrt https://support.f5.com/csp/article/K53322151 . My stance is simply if I use in the internal feature of bash for TCP socket programming I didn't expect to have side effects like this. Thx, Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 7:30 AM, Bob Proulx wrote: > dirk+b...@testssl.sh wrote: >> we discovered a strange phenomenon in the project testssl.sh: > > You are doing something that is quite unusual. You are using a shell > script direction on a TCP socket. That isn't very common. Do you think there should be a paragraph NOT COMMON where bash sockets should rather belong to? > More > typically one would use a C program instead. So it isn't surprising > that you are finding interactions that are not well known. Bob, my intention was not to discuss program languages and what is typical with you or anybody else here. >> printf -- "$data" >&5 2>/dev/null > > Why is stderr discarded? That is almost always bad because it > discards any errors that might occur. You probably shouldn't do this.> > What happens if $data contains % format strings? What happens if the > format contains a sequence such as \c? This looks problematic. This > is not a safe programming proctice. I doubt you can judge on this by just looking at a single line of code -- the project has > 18k LoC in bash. Github is the place to discuss and do PRs for our project. >> If there's a workaround, please let me know. (tried to add "%b" with no >> effect). Otherwise I believe it's a bug. > > You can re-block the output stream using other tools such as 'cat' or > 'dd'. Since you are concerned about block size then perhaps dd is the > better of the two. > > | cat cat has a problem with binary chars, right? And: see below. > Or probably better: > > | dd status=none bs=1M > > Or use whatever block size you wish. The 'dd' program will read the > input into its buffer and then output that block of data all in one > write(2). That seems to be what you are wanting. We actually use dd to read from the socket. Of course we could use writing to it as well -- at a certain point of time. Still, a prerequisite would be that printf is the culprit and not how bash + libs do sockets. > P.S. You can possibly use the 'stdbuf' command to control the output > buffering depending upon the program. > > info stdbuf That could be an option, thanks. Need to check though whether a) it doesn't fragment then -- not sure while reading it b) it's per default available on every platform supported by testssl.sh. Cheers, Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 12:38 PM, Ilkka Virta wrote: > On 22.9. 02:34, Chet Ramey wrote: >> Newline? It's probably that stdout is line-buffered and the newline causes >> a flush, which results in a write(2). > > Mostly out of curiosity, what kind of buffering logic does Bash (or the > builtin > printf in particular) use? It doesn't seem to be the usual stdio logic where > you get > line-buffering if printing to a terminal and block buffering otherwise. I get > a > distinct write per line even if the stdout of Bash itself is redirected to say > /dev/null or a pipe: > > $ strace -etrace=write bash -c 'printf "foo\nbar\n"' > /dev/null > write(1, "foo\n", 4) = 4 > write(1, "bar\n", 4) = 4 > +++ exited with 0 +++ Oh. But thanks anyway! coreutils in fact does it in one shot as you indicated. Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 12:30 PM, Ilkka Virta wrote: > The coreutils printf seems to output 'foo\nbar\n' as a single write, though > (unless > it goes to the terminal, so the usual stdio buffering), so you might be able > to use > that. thx. Might be not that portable but we'll see. > In any case, if a TCP endpoint cares about getting full data packets within a > single > segment, I'd say it's broken. fully agree. But unfortunately it just comforts us :-) Keep in mind that the purpose of the tool is testing and if due to a bug it can't do that, were the ones being blamed or we need to do really strange workarounds to avoid '\x0a' in the first 8 bytes. Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/23/18 8:26 PM, Chet Ramey wrote: > On 9/22/18 4:22 PM, Bob Proulx wrote: > >> Note that I *did* provide you with a way to do what you wanted to do. :-) >> >> It was also noted in another message that the external standalone >> printf command line utility did buffer as you desired. That seems >> another very good solution too. Simply use "command printf ..." to >> force using the external version. > > This won't work the way you want. The `command' builtin only inhibits > execution of shell functions. It still executes builtins. You want to > either get the full pathname of a printf utility using `type -ap printf' > and use that, or use the env or exec variants I recommended in my last > message. FYI: I ended up checking with type before whether an external printf exists and set a variable for this and then just call this variable. env or exec: never thought about it (thanks!) but as both are external commands, that would mean upon every call one additional external program. (yes, I know that there is such thing as a fs buffer). Subshells also costs resources. As this is a core function I am happy for every homeopathic dose of time I safe here :-) Cheers, Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/25/18 3:46 PM, Chet Ramey wrote: > On 9/25/18 9:04 AM, dirk+b...@testssl.sh wrote: > >> FYI: I ended up checking with type before whether an external printf >> exists and set a variable for this and then just call this variable. >> >> env or exec: never thought about it (thanks!) but as both are external >> commands, that would mean upon every call one additional external program. >> (yes, I know that there is such thing as a fs buffer). Subshells also costs >> resources. As this is a core function I am happy for every homeopathic dose >> of time I safe here :-) > > `exec' is a shell builtin. It will `cost' in terms of a fork, but you're > going to fork and exec a different program anyway -- /usr/bin/printf -- > so it's basically a wash. In either case, there's one fork and one > execve. yeah, you're right.
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/25/18 2:15 PM, L A Walsh wrote: > > > On 9/24/2018 6:05 AM, Greg Wooledge wrote: >> On Sat, Sep 22, 2018 at 11:50:17AM +0200, dirk+b...@testssl.sh wrote: >> >>> On 9/22/18 7:30 AM, Bob Proulx wrote: >>> >>>> dirk+b...@testssl.sh wrote: >>>> >>>>> printf -- "$data" >&5 2>/dev/null >>>>> >>>> What happens if $data contains % format strings? What happens if the >>>> format contains a sequence such as \c? This looks problematic. This >>>> is not a safe programming proctice. >>>> >> >> Looking ONLY at this one line, there is an obvious bug, which Bob has >> pointed out. It should be >> >> printf %s "$data" >&5 2>/dev/null >> > > This brings to mind a consideration: > As %s says to print a string of data (presumably not > including a NUL byte), it certainly does contain a null byte, and every other chars between 1-255. That's the point of a network socket. Also "$data" will NEVER contain user input in any way with one exception being the hostname which is transferred via hexdump into exactly this format. Other than that "$data" is populated purely internally. It can't contain anything else between '\x00' and '\xff' unless there's a coding error which could be a good idea to catch before and not here. This is why I said you can't look only at one line of code. Code reviews requires to see the whole picture. BTW: printf seems to be off the table. BSDish /usr/bin/printf is completely different compared to the the coreutils incarnation. OpenBSD has per default not even a printf outside bash. > then what happens if "$data" is > a paragraph of text with embedded newlines. In that case, > it sounds like bash might break apart the single printf > output into smaller packets rather than transmitting the > entirety of "$data" in 1 write (presuming it is less than > the maximum data size for a network packet). yup. Wonder why the coreutils printf behaves (in my sense) better than the bash-builtin. > Also, if you want to flush the data out at the end, it seems > "%s\n" would be required to force out the last line of text if > it wasn't nl terminated. > >> That is utterly horrifying. >> I take that as a compliment :-) > --- > HmmmI didn't realize how sensitive some sensibilities were... > :-) LOL There are JavaScript frameworks in the browser of similar size or even bigger, the kernel I am using right now is written in a language which is not known to be safe and whose parser after 25 years throws sometimes utterly misleading errors -- which still remind me on the first K+R c compilers -- this and other things I found rather horrifying. Script languages have long evolved -- you should take this really as a compliment -- and as I started this project I never thought it would boldly go there :-) Cheers, Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 23.09.18 20:29, Bob Proulx wrote:
> Robert Elz wrote:
> $ { command printf "one\n"; sleep 1; command printf "two\n" ;} | strace -v
> -o /tmp/dd.strace.out -e write,read dd status=none ibs=1M obs=1M ; head
> /tmp/*.strace.out
> one
> two
> ...
> read(0, "one\n", 1048576) = 4
> read(0, "two\n", 1048576) = 4
> read(0, "", 1048576)= 0
> write(1, "one\ntwo\n", 8) = 8
> +++ exited with 0 +++
>
> And just for completeness I will show the above with both a large
> input buffer and a large output buffer of the same size and show that
> result too. The required dd option, as you correctly insisted, really
> is obs= in order to set the output block size. I stand corrected. :-)
>
> I had missed the documented dd behavior:
>
> ‘bs=BYTES’
> Set both input and output block sizes to BYTES. This makes ‘dd’
> read and write BYTES per block, overriding any ‘ibs’ and ‘obs’
> settings. In addition, if no data-transforming ‘conv’ option is
> specified, input is copied to the output as soon as it’s read, even
> if it is smaller than the block size.
>
> It is always good to learn something new about fundamental behavior in
> a command one has been using for some decades! :-)
Thanks for the long mails!
This all -- including cat -- sounded reasonable. But it seems using sockets the
internal printf
as opposed to the one from coreutils is still causing fragmentation other than
expected with
strace PoC:
bash 0$ exec 5<>/dev/tcp/81.169.199.25/443
bash 0$ printf
'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0b\x85\x03\x90\x9f\x77\x04\x33\xd4\xde\x20\x44\xb8\x92\x56\xaf\x74\x52\x9e\xd8\xcf\x52\x14\xc8\xaf\xd8\x34\x0b\xe7\x7f\xeb\x86\x01\x84\x50\x5d\xe4\xa1\x6a\x09\x3b\xbf\x6e\x00\x0e\x13\x01\x13\x02\x13\x03\x13\x04\x13\x05\xc0\x30\x00\xff\x01\x00\x01\xa5\x00\x00\x00\x0b\x00\x09\x00\x00\x06\x66\x66\x66\x66\x66\x66\x00\x2b\x00\x17\x16\x03\x04\x7f\x1c\x7f\x1b\x7f\x1a\x7f\x19\x7f\x18\x7f\x17\x03\x03\x03\x02\x03\x01\x03\x00\x00\x23\x00\x00\x33\x74\x00\x00\x00\x0d\x00\x22\x00\x20\x04\x03\x05\x03\x06\x03\x08\x04\x08\x05\x08\x06\x04\x01\x05\x01\x06\x01\x08\x09\x08\x0a\x08\x0b\x08\x07\x08\x08\x02\x01\x02\x03\x00\x0a\x00\x10\x00\x0e\x00\x1d\x00\x17\x00\x1e\x00\x18\x00\x19\x01\x00\x01\x01\x00\x33\x00\x6b\x00\x69\x00\x1d\x00\x20\x4d\xfa\x57\x44\xb7\xf7\x48\xb8\x95\x77\x5a\xc1\xff\x86\xbf\xae\xf7\x3a\x33\x69\x54\xde\x6a\xf5\x2e\x89\x84\x6c\xf2\xd8\xb2\x43\x00\x17\x00\x41\x04\xb4\x24\xef\x11\x99\x9c\xa4\xe8\xce\x88\x25\xc3\x8e\x7c\x0c\x6a\x94\xde\x33\x6d\xff\xcd\x17\xb7\x5c\x65\xdb\xd1\x58\x46\x95\x69\x80\xc8\xbc\xfc\xe6\xd9\x22\x39\xbb\x3f\x63\xab\x3d\x5c\xba\xcc\xeb\x1a\x90\x1b\xd4\x75\xff\x58\xc4\x00\x58\x50\x21\xd0\xaa\xe4\x00\x0b\x00\x02\x01\x00\x00\x0f\x00\x01\x01\x00\x15\x00\xbb\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0^Cx00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
| dd obs=1M ibs=1M >&5
bash 0$
(Excuse the wrapping. The IP is mine from the project. Feel free to use another
IP.
The servername encoded in there is anyway nonsense)
If you use wireshark you see in the ClientHello "TCP segment of a reassembled
PDU" @ byte
173. That's where the first LF is encountered. The second one doesn't cause an
additional
fragment here, other people spotted that.
The fragmentation is independent on the dd options used. Also "| cat" does the
same.
stdbuf is not available on all platforms, especially on those who do not have a
similar
external printf:
/usr/bin/printf "\xf5\xee\xbe\xe5" | xxd
: 7866 3578 6565 7862 6578 6535xf5xeexbexe5
like FreeBSD and OS X. OpenBSD's /usr/bin/printf works surprisingly.
Cheers, Dirk
PS + @Bob: fd 5 is not a tty in the program -- but interactively in this PoC
you want to make
sure it is not taken yet.
Re: bash sockets: printf \x0a does TCP fragmentation
On 11.10.18 18:53, Dirk Wetter wrote: > bash 0$ exec 5<>/dev/tcp/81.169.199.25/443 > bash 0$ printf > '\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0b\x85\x03\x90\x9f\x77\x04\x33\xd4\xde\x20\x44\xb8\x92\x56\xaf\x74\x52\x9e\xd8\xcf\x52\x14\xc8\xaf\xd8\x34\x0b\xe7\x7f\xeb\x86\x01\x84\x50\x5d\xe4\xa1\x6a\x09\x3b\xbf\x6e\x00\x0e\x13\x01\x13\x02\x13\x03\x13\x04\x13\x05\xc0\x30\x00\xff\x01\x00\x01\xa5\x00\x00\x00\x0b\x00\x09\x00\x00\x06\x66\x66\x66\x66\x66\x66\x00\x2b\x00\x17\x16\x03\x04\x7f\x1c\x7f\x1b\x7f\x1a\x7f\x19\x7f\x18\x7f\x17\x03\x03\x03\x02\x03\x01\x03\x00\x00\x23\x00\x00\x33\x74\x00\x00\x00\x0d\x00\x22\x00\x20\x04\x03\x05\x03\x06\x03\x08\x04\x08\x05\x08\x06\x04\x01\x05\x01\x06\x01\x08\x09\x08\x0a\x08\x0b\x08\x07\x08\x08\x02\x01\x02\x03\x00\x0a\x00\x10\x00\x0e\x00\x1d\x00\x17\x00\x1e\x00\x18\x00\x19\x01\x00\x01\x01\x00\x33\x00\x6b\x00\x69\x00\x1d\x00\x20\x4d\xfa\x57\x44\xb7\xf7\x48\xb8\x95\x77\x5a\xc1\xff\x86\xbf\xae\xf7\x3a\x33\x69\x54\xde\x6a\xf5\x2e\x89\x84\x6c\xf2\xd8\xb2\x43\x00\x17\x00\x41\x04\xb4\x24\xef\x11\x99\x9c\xa4\xe8\xce\x88\x25\xc3\x8e\x7c\x0c\x6a\x94\xde\x33\x6d\xff\xcd\x17\xb7\x5c\x65\xdb\xd1\x58\x46\x95\x69\x80\xc8\xbc\xfc\xe6\xd9\x22\x39\xbb\x3f\x63\xab\x3d\x5c\xba\xcc\xeb\x1a\x90\x1b\xd4\x75\xff\x58\xc4\x00\x58\x50\x21\xd0\xaa\xe4\x00\x0b\x00\x02\x01\x00\x00\x0f\x00\x01\x01\x00\x15\x00\xbb\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0^Cx00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > | dd obs=1M ibs=1M >&5 > bash 0$ Sorry, forget about my mail. In the actual file I was using there was a variable which I haven't quoted correctly. The PoC works as expected and it looks I have a solution. Thx, Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 11.10.18 23:35, Bob Proulx wrote: > The smallest of details can break the largest of programs. :-) > > Good to hear you have things resolved and now working for you! Thanks all for your insights and help! Cheers, Dirk
Using variables in variables names
Hi folks, I am sure this has been asked quite some times, but I did not find anything inspiring or helpful - in fact not too much at all. For accelerating a script I need the possibility to set up an unknown number of arrays and to name them (at least partly) with values of a variable. It is like defining arrays with names that contain an increasing number: array$x. While poking around I found out: This does not work at all, even with simple variables it does not. dings=bums echo $dings bums ac=12 dings$ac=wasannersder -bash: dings12=wasannersder: command not found I looked very deeply into man bash and any manual and howto I found, but did not find out any reason why this does not work. It must be one of the basic principles of expansion and assignment, but I would like to understand it. And then the question of multi-dimensional arrays (see above). The "Advanced Bash Scripting Guide" mentions that it is possible to have them using indirect referencing - but I did not understand how this could be done. Any idea, hint or help? It would be a great relief after two days of search and research. Dirk ___ Bug-bash mailing list Bug-bash@gnu.org http://lists.gnu.org/mailman/listinfo/bug-bash
Re: Using variables in variables names
Paul Jarc schrieb:
"Dirk H. Schulz" <[EMAIL PROTECTED]> wrote:
ac=12 dings$ac=wasannersder
-bash: dings12=wasannersder: command not found
Variable names in assignments are not subject to expansion. So since
"dings$ac", as-is, does not fit the syntax for variable names, it
isn't treated as an assignment. This will work:
ac=12 eval "dings$ac=wasannersder"
And how do I reference it then?
ac=12
eval "dings$ac=wasannersder"
echo $dings12
wasannersder # that works
echo $('$dings'$ac) # trying to
substitute $ac before $dings...
-bash: $dings12: command not found
Is there any way to reference it without anticipating the result of
indirect referencing?
Dirk
___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash
Re: Using variables in variables names
Oh Mike,
thanks a lot for such detailled and well structured clarification! That
did it, now I can use it (and it gave my script a speedup of nearly 70 %
less running time).
Many thanks!
Dirk
Mike Stroyan schrieb:
On 3/13/06, Paul Jarc <[EMAIL PROTECTED]> wrote:
"Dirk H. Schulz" <[EMAIL PROTECTED]> wrote:
Paul Jarc schrieb:
ac=12 eval "dings$ac=wasannersder"
And how do I reference it then?
ac=12 eval "value=\$dings$ac"
echo $value
Or:
ac=12 name=dings$ac echo ${!name}
It seems that you need to use the eval form instead of the ${!var} form
to handle array variables. Here are some examples I played with. The
pattern is to use a backslash to quote the $ for the array name. The $i
in the array examples could be done as \$i because it works out the same
if it is expanded in either the first pass or the second pass.
$ suffix=one
$ eval "pre_${suffix}=simple1"
$ suffix=two
$ eval "pre_${suffix}=simple2"
$ suffix=one
$ eval "echo \$pre_${suffix}"
simple1
$ suffix=two
$ eval "echo \$pre_${suffix}"
simple2
$ suffix=one
$ i=1
$ eval "pre_A_${suffix}[$i]=array1_1"
$ i=2
$ eval "pre_A_${suffix}[$i]=array1_2"
$ suffix=two
$ i=1
$ eval "pre_A_${suffix}[$i]=array2_1"
$ i=3
$ eval "pre_A_${suffix}[$i]=array2_3"
$ set | grep pre_
_='pre_A_two[3]=array2_3'
pre_A_one=([1]="array1_1" [2]="array1_1")
pre_A_two=([1]="array2_1" [3]="array2_3")
pre_one=simple1
pre_two=simple2
$ i=1
$ eval "echo \${pre_A_${suffix}[$i]}"
array2_1
$ eval "echo \${pre_A_${suffix}[$i]}"
array2_1
$ i=3
$ eval "echo \${pre_A_${suffix}[$i]}"
array2_3
$ i=2
$ suffix=one
$ eval "echo \${pre_A_${suffix}[$i]}"
array1_2
--
Mike Stroyan
[EMAIL PROTECTED]
___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash
___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash
Line numbering in error messages
Hi folks, I wonder if there is a thorough documentation on line numbering in bash's error messages? I find it often quite misleading and would like to know what I am counting wrong. Any hint or help is appreciated. Dirk ___ Bug-bash mailing list Bug-bash@gnu.org http://lists.gnu.org/mailman/listinfo/bug-bash
i/o redirection into variables
Hi folks, I am looking for a possiblity to redirect error output into a variable: Instead of redirecting to a file ( e.g. command 2>>errorlogfile ) I would like to redirect into a variable. Is this generally possible? Dirk ___ Bug-bash mailing list Bug-bash@gnu.org http://lists.gnu.org/mailman/listinfo/bug-bash
Re: parenthesised regular expressions and non-greedy operator ? - non standard bash behaviour
>From the 2 replies I unterstand that the implementation in bash is correct due to the „official“ standard. For myself I have solved the issue in my script - but the regular expression I developed for my problem are without the 'non-greedy' operator more difficult to read and maintain. From that point of view it would be an improvement for bash to implement the non-greedy operator. Also if I look from an „normal developer“ I think it is a common pitfall if many testing resources and regexp implementations support the 'non-greedy' operator. Maybe there is a switch/option to enable the 'non-greedy' operator in a future release. So please feel free to change the „bug report“ to a „feature request“ ;-) Best Regards, H.-Dirk Schmitt On So, 2017-12-03 at 15:23 -0500, Chet Ramey wrote: > On 12/1/17 12:40 PM, d...@computer42.org wrote: > > > Bash Version: 4.4 > > Patch Level: 12 > > Release Status: release > > > > Description: > > I'm sanitising urls from advertisement crap. As described below > > I'm getting a wrong resolution of parenthesised expression defined > > with non-greedy operator '?'. > > > > The test url is: http://toolbox.contentspread.net/container/medim > > ops/track/xx.dyn?csRdu=https://www.medimops.de/?anid=M9 > > 9&cl=details&wdm=M99&utm_source=CRM&utm_medium=email&ut > > m_campaign=OS > > > > The regular expression is: > > https?:\/\/toolbox.contentspread.net\/(.*?)=(.+?)&.* > > The Bash =~ operator uses Posix extended regexps (EREs) as defined in > http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.ht > ml#tag_09_04. > There's no concept of a `non-greedy' operator > in the Posix ERE definition. > > Chet >
Re: parenthesised regular expressions and non-greedy operator ? - non standard bash behaviour
On Mo, 2017-12-04 at 16:49 -0500, Chet Ramey wrote: > The thing is, bash doesn't "implement" its regular expressions, per > se. > Bash uses the Posix standard library functions (regcomp/regexec) if > they > are available in the C library when it's configured and built. I'm > not > wild about adding a dependency on pcre, or a configure test for it, > just > to have two varieties of regular expressions available. > > Chet O.k. – so close this as „not a bug“. -- Signature H.-Dirk Schmitt H.-Dirk Schmitt Dipl.Math. eMail:dirk.schm...@computer42.org mobile:+49 177 616 8564 phone: +49 2642 99 41 14 fax: +49 2642 99 41 15 Schillerstr. 42, D-53489 Sinzig pgp: http://www.computer42.org/~dirk/OpenPGP-fingerprint.html
Missing documentation of the integer range (declare -i)
Checked against: GNU bash, Version 4.4.12(1)-release (x86_64-pc-linux- gnu) I carefully read the man page and find no definition of the defined range of integer variables in bash. see also: `man bash |& grep -i integer` Best Regards, H.-Dirk Schmitt
