Subject: [CODE4LIB] perl wrapper for yaz-marcdump

From: Eric Lease Morgan <[email-masked]>
Date: Fri, 30 Jan 2009 16:12:24 -0500

Is there any way I can make my Perl wrapper for yaz-marcdump, below, more
efficient?

I have (yet another) application that takes MARC records as input, parses
them, and indexes them. My indexer(s) want UTF-8 but sometimes my records
are encoded as MARC-8. Consequently, I wrote a subroutine (m2u) that runs
yaz-marcdump with a number of command-line options, traps STDOUT, and
returns the results converted from MARC-8 to UTF-8.

  #!/usr/bin/perl
  
  # define yaz-marcdump;
  use constant YAZ => 'yaz-marcdump -f MARC-8 -t UTF-8 -o marc -l 9=97 ';
  
  # sanity check
  if ( ! $ARGV[ 0 ]) {
  
    print "Usage: $0 path/to/marc\n";
    exit;
  
  }
  
  # do the work (marc-8 to utf-8) and quit
  print &m2u( YAZ, $ARGV[ 0 ] );
  exit;

  # convert
  sub m2u {
  
    $y = shift;  # yaz stub
    $f = shift;  # file to process
    $r = '';     # return value
    
    # run yaz and trap the output
    open ( C, "$y$f |" ) or die "Can't open converter: $!\n";
    while ( <C> ) { $r = $_ }
    close C;
    
    # done
    return $r;
  
  }

For extra credit, is there anyway I can optimize m2u? For example, is there
anyway to get rid of the while loop and slurp up yaz-marcdump's output in
one go?

-- 
Eric Lease Morgan

---

From: Jay Luker <[email-masked]>
Date: Fri, 30 Jan 2009 16:39:35 -0500

> anyway to get rid of the while loop and slurp up yaz-marcdump's output in

You can replace the while loop with perl's usual slurp idiom:

$r = do { local $/; <C> };

Explained here:
http://www.perlmonks.org/?node_id=287647

--jay

---

From: Joe Hourcle <[email-masked]>
Date: Fri, 30 Jan 2009 16:44:57 -0500

> Is there any way I can make my Perl wrapper for yaz-marcdump, below, more

[trimmed]

> anyway to get rid of the while loop and slurp up yaz-marcdump's output in

It might be faster to process multiple files at once, so you don't have 
the overhead of perl starting up each time.  For the while loop, use:

 	my $r = do { local $/ = undef; <C> };

(unset the end-of line marker, then read it in one pass, the 'do' block 
keeps the change of end of line marker from affecting anything else)

-Joe

---

From: Joe Atzberger <[email-masked]>
Date: Fri, 30 Jan 2009 16:45:41 -0500

> Is there any way I can make my Perl wrapper for yaz-marcdump, below, more

Dump as you go rather than read up the whole thing into memory.  Actually,
why do you need perl at all?  This is just a regular yaz-marcdump call.
Pipe it into your indexer.

Otherwise, skip the subroutine and just do:

> open (C, "$y $file |") or die "Can't open converter: $!\n";
> print, while (<C>);
> close C;
>

Wouldn't your code have been getting only the last line $r returned?
Perhaps you intended to append to $r each pass through the loop.

--Joe

---

From: Eric Lease Morgan <[email-masked]>
Date: Mon, 2 Feb 2009 08:42:26 -0500

> http://www.perlmonks.org/?node_id=287647

As long as I can afford the RAM (a batch of MARC records might be 10's or
100's of megabytes in size), this looks like the best solution. Thanks.
jay.luker++

-- 
Eric Lease Morgan
University of Notre Dame

---

From: Joe Hourcle <[email-masked]>
Date: Mon, 2 Feb 2009 09:43:32 -0500

> 100's of megabytes in size), this looks like the best solution. Thanks.

Is there a seperator between MARC records?

If so, you can set $/ to that, and you'll get one MARC record at a time.

If the record seperator is an empty line, you'll probably want to read 
'perlvar' in perldocs ... search for 'INPUT_RECORD_SEPARATOR' for the 
difference between setting it to '' vs "\n\n":

 	http://perldoc.perl.org/perlvar.html

-Joe