18 comments to “Don’t MAWK AWK - the fastest and most elegant big data munging language!”

1. Pete Kirkham wrote:
   10. September 2009 at 7:39 am :

   Adding units for the timings to the blog post would help put it into context.

2. Adam wrote:
   10. September 2009 at 8:52 am :

   Cool! I’ll definitely have to learn more about AWK’s awesomeness.

3. brendano wrote:
   10. September 2009 at 3:29 pm :

   Pete - fixed, thanks. Anything else confusing?

4. OJ wrote:
   10. September 2009 at 4:35 pm :

   Two points about Java:

   - It takes only a few lines to separate reading and splitting lines to different thread and have it communicate with main tread through BlockingQueue

   - Java’s standard string split is slow. Essentially it compiles the column separator as a regular expression and uses it every time it is called. Rolling your own split function *should* be unnecessary but having one around is useful.

   For one time use these tricks are not really necessary but when you have to read several gigabytes of tab-separated data several times in a row…

   The separate read/split thread could probably be made an integral part of AWK.

5. doki_pen wrote:
   10. September 2009 at 5:58 pm :

   You should try doing a simple ruby –help before posting such nonsense. I’m sure you missed similar features in other languages. The unix way is to pipe the file and processes stdin. No need to open and close a file. I didn’t read the entire article, but if you didn’t you should post all your code.

6. jc wrote:
   10. September 2009 at 6:56 pm :

   Err, you should read the article before calling BS. He’s reading from multiple files and writing back to similarly named files. If there’s a way to write to multiple files in that manner, while using some global values, I don’t want to see it in shell, it would look almost as much like snoopy barf as perl.

7. Fletch wrote:
   10. September 2009 at 7:52 pm :

   Got your perl tweaked down to around 1:25s avg: http://gist.github.com/184768. Made it a bit more idiomatic modern Perl and then optimized (reduced hash lookups to once per line by stashing the value off to a lexical scalar, yadda yadda yadda).

   Interesting post though; now I need to grab mawk for RHEL boxen at work and see what speed gains it gets vs the stock awk.

8. brendano wrote:
   10. September 2009 at 11:33 pm :

   Fletch, thanks! That’s great. Obviously I don’t know much perl :)

   OJ, I never thought about using threads like that. Interesting. Have you implemented something like this before? And assuming you have 2 cores available, how much can it increase performance?

9. Todd wrote:
   11. September 2009 at 5:51 am :

   Actually, a Perl script can look a lot like an awk script if we exploit
   some of Perl’s command line arguments … see the perlrun man page
   and variables … see the perlvar man page.

   Building on Fletch’s rewrite and writing the vocabulary to stderr
   here is a two line Perl script to do the job.

   
#!/usr/bin/perl -ani_org
# USAGE: ./me file1 file2 file3 ... 2>vocab # bash style redirect of stderr

   exists $jm{$F[1]} or print STDERR “$F[1]\n”;
print(($im{$ARGV,$F[0]} ||= ++$i{$ARGV}).’ ‘.($jm{$F[1]} ||= ++$j).” $F[2]\n”);


   Where ‘-a’ causes perl to do an awk-like automatic split in to @F
   ‘-n’ causes perl to put a loop around the script
   ‘-i_orig’ cause perl to update each file “in place”
   first backing up each file to file_orig
   ‘$ARGV’ is the name of the file currently being read
   ‘$;’ we are relying on the default for this not be in
   any of the item names ($F[0]) or file names ($ARGV)
   in the expression $im{$ARGV,$F[0]}

   As a possible tweak for speed we could remove some of the hashing
   by resetting %imap at the top of each file by adding one line of code.
   This is a rather crude way of what Fletch did with ‘my’ variables.
   Here we get rid of one more hash access than Fletch did.
   But he could do that in his code also.

   
#!/usr/bin/perl -ani_org
# USAGE: ./me file1 file2 file3 ... 2>vocab # bash style redirect of stderr

   if ($curfile ne $ARGV) { undef %imap; $i = 0; $curfile = $ARGV }
exists $jm{$F[1]} or print STDERR “$F[1]\n”;
print(($im{$F[0]} ||= ++$i).’ ‘.($jm{$F[1]} ||= ++$j).” $F[2]\n”);


   Okay, I’m stretching the definition of a line …

   If one dislikes the structure of the print statements, we could
   exploit some special variables
   ‘$,’ sets the output field separator
   ‘$\’ sets the output record separator
   and write

   
#!/usr/bin/perl -ani_org
# USAGE: ./me file1 file2 file3 ... 2>vocab # bash style redirect of stderr

   BEGIN { $, = ' '; $\ = "\n" }
if ($curfile ne $ARGV) { undef %imap; $i = 0; $curfile = $ARGV }
exists $jm{$F[1]} or print STDERR $F[1];
print(($im{$F[0]} ||= ++$i), ($jm{$F[1]} ||= ++$j), $F[2]);


   If we rejected writing the vocabulary to stderr we add to the BEGIN statement
   
#!/usr/bin/perl -ani_org
# USAGE: ./me file1 file2 file3 ...

   BEGIN {
open(VOCAB, '>vocab') or die $!;
$, = ' ';
$\ = "\n";
}
if ($curfile ne $ARGV) { undef %imap; $i = 0; $curfile = $ARGV }
exists $jm{$F[1]} or print STDERR $F[1];
print(($im{$F[0]} ||= ++$i), ($jm{$F[1]} ||= ++$j), $F[2]);


   Finally, if we don’t like the way the ‘-i’ option works we can
   add one line to do it our selves

   
#!/usr/bin/perl -an
# USAGE: ./me file1 file2 file3 ...

   BEGIN {
open(VOCAB, '>vocab') or die $!;
$, = ' ';
$\ = "\n";
}
if ($curfile ne $ARGV) { # starting new file
open(STDOUT, '>', $ARGV . 'n') or die $!;
undef %imap;
$i = 0;
$curfile = $ARGV
}
exists $jm{$F[1]} or print VOCAB $F[1];
print(($im{$F[0]} ||= ++$i), ($jm{$F[1]} ||= ++$j), $F[2]);


10. Destillat KW37-2009 | duetsch.info - GNU/Linux, Open Source, Softwareentwicklung, Selbstmanagement, Vim ... wrote:
    11. September 2009 at 9:01 am :

    [...] Don’t MAWK AWK – the fastest and most elegant big data munging language! [...]

11. David R. MacIver wrote:
    11. September 2009 at 9:49 am :

    As a general comment: I’m actually *really* surprised at how well Java is doing here. I’ve spent more time than I’d like to admit doing this sort of task in Java, and inevitably what happens when I need it to be repeatable is that I have to tear apart java.io and hand roll specific implementations, and the code typically becomes at a minimum twice as fast and usually more when I do. There’s too much in memory copying and the unicode support is a real slow down if you don’t need it.

    On the String.split front: An easy speedup is to do static final Pattern WHITESPACE = Pattern.compile(” “); and use WHITESPACE.split(string) instead. It seemed to shave about 10 seconds off the runtime on my computer, which left it still slower than awk.

12. Eric Young wrote:
    11. September 2009 at 11:36 am :

    I’ve done a faster ruby version. http://gist.github.com/185234.
    I’m on a linux-x86_64, 3ghz Penom II, gawk seems much faster on my box

    33.8s mawk
    36.3s gcc c
    51.0s java
    67.0s perl Fletch.pl
    71.7s python
    87.8s perl
    95.8s nawk
    101.4s gawk
    114.0s gcc
    133.0s ruby1.9 eay.rb
    136.8s ruby1.8 eay.rb
    327.6s ruby1.8
    372.9s ruby1.9

13. OJ wrote:
    11. September 2009 at 6:41 pm :

    Brendano: On my dual core machine, the difference between threaded and non-threaded version is around 25%. I’m not sure why the difference is not any greater as this thing puts both CPUs near full load. I thought it could be thread communication overhead, but packing multiple lines into larger packets did not help.

    It is interesting that the single thread version takes CPU load over 100%, probably due parallelized GC.

    The code is available here. Uglier and more wordy than I thought.

14. Carlo wrote:
    23. September 2009 at 5:42 pm :

    I’ve just hit this post and I hope I’m still in time to comment on it. I’ve been using AWK for more than a decade now, to the point that I have written a whole application development framework for Web applications, largely based on AWK. I confirm that MAWK is lightening fast but I’m getting the impression it is no longer actively developed. I hope I’m wrong. MAWK is still affected by a few bugs that seem to have been around for years now, and nobody seems to be interested with fixing them. I have just stumbled upon this one.

15. Jacob wrote:
    7. December 2009 at 1:28 am :

    It seems to me that the issue of processing speed is almost irrelevant for data formating tasks. All tools seem fast enough. For a typical new data set, one needs to do it once and then never do it again. A lot more time is spent on writing and testing the code to accomplish this task. Based on these presumptions, the AWK implementations still beat everything else. AWK was designed for digging through formated text files and it’s nice to see that it still excels at such tasks.

16. Haris wrote:
    12. December 2009 at 11:21 am :

    I am an AWK fan and I use gawk. I didn’t knew that mawk it wasn’t interpreted language, so I gave it a try, inspired by your article.

    I did some tests but I miss certain functionality like gensub, asort, asorti functions .

    Are there any mawk functions I can use as a replacement of the above ones?

17. brendano wrote:
    12. December 2009 at 5:50 pm :

    Not that I know of…

18. Keith Smith wrote:
    4. March 2010 at 10:12 am :

    This is kind of silly. The purpose of awk, etc is for a quick and dirty. Mawk, like dash is broken in subtle ways. Always script to standard available utilities or pay the price.

    Why on earth would you use Ruby for this? That’s just plain ignorant, although vanilla PHP might be interesting. Assuming your data set is formatted as you specified, C code follows. Linear search might be an issue if nelem gets overlarge, but likely not even close to a bottleneck. 140 odd lines with comments, with assumptions galore.

    
// File: 2num.c
//
// Last Modified: 2010-03-04 02:54:53
//
#include
#include
#include
#include
#include
#include
#include

    #define BUFSIZE 65535
    char big_buf[BUFSIZE + 1];

    #define LIST_BREAK 16384
    struct list_struct {
    char **list;
    int size,ix,new;
    } list1,list2;

    int in_fd,out_fd,vocab;

    //———————————————————————-
    // CHECK_LIST
    // Simple Linear search is faster up to around 16K elements
    //———————————————————————-
    check_list(struct list_struct *lp, char *val) {
    char **p;
    int ix;
    for(p = lp->list,ix = 0; ix ix && lp->list[ix] != NULL; ix++) {
    if(strcmp(lp->list[ix],val) == 0) {
    return(ix + 1);
    break;
    }
    }
    // Not there
    if(lp->ix > lp->size) {
    lp->list = realloc(lp->list,lp->size + LIST_BREAK);
    }
    lp->list[lp->ix++] = strdup(val);
    lp->new = 1;
    return(lp->ix);
    }

    //———————————————————————-
    // PROCESS
    //———————————————————————-
    void process(char **vals) {
    int n1,n2;
    char obuf[32];

    n1 = check_list(&list1,vals[0]);
    n2 = check_list(&list2,vals[1]);
    if(list2.new) { // Optmizeme into a buffer
    write(vocab,vals[1],strlen(vals[1]));
    write(vocab,”\n”,1);
    list2.new = 0;
    }
    // Optimize into a buffer …
    sprintf(obuf,”%d %d %s\n”,n1,n2,vals[2]);
    write(out_fd,obuf,strlen(obuf));
    }

    //———————————————————————-
    // PARSE
    //———————————————————————-
    char **parse(char *line) {
    static char *p[3];
    p[0] = line;
    p[1] = index(line,’ ‘);
    if(p[1] == NULL) {
    p[1] = “”;
    p[2] = “”;
    return(p);
    }
    *(p[1])++ = ”;
    p[2] = index(p[1],’ ‘);
    if(p[2] == NULL) {
    p[2] = “”;
    } else {
    *(p[2])++ = ”;
    }
    return(p);
    }

    //———————————————————————-
    // MAIN
    //———————————————————————-
    main(int argc, char *argv[]) {
    int ix, iy, bytes_read,buflen;
    char *p,*q, **vals;
    char buf[4096];

    list1.list = malloc(LIST_BREAK);
    list1.size = LIST_BREAK;
    list1.ix = 0;
    list1.new = 0;

    list2.list = malloc(LIST_BREAK);
    list2.size = LIST_BREAK;
    list2.ix = 0;
    list2.new = 0;

    vocab = open(”vocab”,O_WRONLY|O_CREAT|O_TRUNC,0666);
    if(vocab == -1) {
    perror(”Open vocab”);
    exit(-1);
    }
    for(ix = 1; ix 0;) {
    // Fill the buffer
    for(buflen = 0; buflen < BUFSIZE;) {
    bytes_read = read(in_fd, big_buf + buflen,BUFSIZE - buflen);
    if(bytes_read 0 && p != NULL;) { // Parse off lines and process them
    q = index(p,’\n’);
    if(q == NULL) { // Read Some More
    buflen = strlen(p);
    memcpy(big_buf,p,buflen);
    break;
    }
    *q++ = ”;
    vals = parse(p);
    process(vals);
    p = q;
    }
    }
    close(in_fd);
    close(out_fd);
    list1.ix = 0;
    }
    close(vocab);
    }

Leave a comment