Subject: [CODE4LIB] METS and TEI?

From: Christine Schwartz <[email-masked]>
Date: Wed, 4 Nov 2009 07:47:30 -0500

Hi,

I've received one good answer on Twitter, but want to get a sense of what
others are doing.

We've been using METS/DC/MIX for awhile, but now were are going to integrate
full-text searching with texts marked up in TEI (our platform is the
MarkLogic Server).

Should we consider embedding the TEI in the METS documents, or just link out
to them?

Any help would be greatly appreciated.

Thanks,

Chris

Christine Schwartz
Metadata Librarian
Princeton Theological Seminary Libraries

---

From: Liza Daly <[email-masked]>
Date: Wed, 4 Nov 2009 09:26:04 -0500

> Should we consider embedding the TEI in the METS documents, or just link out

Will users need to search for items in both schemas at once?  (e.g.
metadata from METS and full-text in TEI in the same query).  In that
case you should absolutely embed the TEI content; MarkLogic exhibits
extremely poor performance in executing SQL-style "joins" across
multiple documents.

Liza

--
http://threepress.org

---

From: stuart yeates <[email-masked]>
Date: Thu, 5 Nov 2009 08:38:34 +1300

> Should we consider embedding the TEI in the METS documents, or just link out

It depends on what you're doing and who the likely users of the METS are.

The trouble with separate files, is that they inevitably get separated. 
The problem with everything-in-one-file is that it gets big an unwieldy 
to edit / validate / process.

Depending on how you're managing your files and what tool support you 
have / need, you make the best trade-off you can.

cheers
stuart
-- 
Stuart Yeates
http://www.nzetc.org/       New Zealand Electronic Text Centre
http://researcharchive.vuw.ac.nz/     Institutional Repository

---

From: "Michael J. Giarlo" <[email-masked]>
Date: Wed, 4 Nov 2009 14:32:17 -0500

> The trouble with separate files, is that they inevitably get separated.

Not to threadjack, but I am curious, Christine: how would you handle
linking between the TEI and the METS?

It could be there's an obvious answer and I'm having a "duh" moment
(or lifetime), of course.

-Mike

---

From: Christine Schwartz <[email-masked]>
Date: Wed, 4 Nov 2009 15:03:11 -0500

Hi Mike,

I'm pretty comfortable with METS, but very new to TEI. So, what I write here
is just an attempt to reflect what my more experienced colleagues are
saying:

* It seems the structure of TEI documents can be problematic since they
follow a logical structure, by paragraphs/sections. And the structMap of all
our METS documents, so far, are divided up by pages of text, not paragraphs.
So the TEI structure does not fit nicely into METS the way we're using METS.

* We're also concerned with not having redundant metadata in the TEI header
and the dmdSec of the METS document. So, we're considering keeping the TEI
header very brief and relying on the METS doc for
descriptive/administrative/technical metadata. (We won't be deriving METS
from TEI which is another issue.)

* The other issue has already been raised by Liza Daly: performance. We've
been told by one of the programmers at Mark Logic that we should embed the
TEI docs into METS for good performance, but we have other reasons why we
don't want to  embed the TEI (editing, maintenance, etc.). So, we are
considering writing a script that would integrate the METS and TEI at the
point a search is deployed.

* From the metadata standpoint, I want to keep the TEI docs separate and
link out to them from the METS docs, because I'm not convinced that library
metadata standards are stable. If we move away from using METS in the next
5-10 years, I think it would be easier if all the text/image files remained
separate from the metadata. So, I'd prefer links in the fileSec of METS that
link out to external TEI files.

Chris

---

From: stuart yeates <[email-masked]>
Date: Thu, 5 Nov 2009 10:09:19 +1300

I've been using TEI for a while now, both at http://www.nzetc.org/ and 
http://www.oss-watch.ac.uk/ and while you may already know this:

* Make sure you're using the P5 release of TEI, P4 and earlier don't 
play well being parts of larger documents (xml:lang, xml:id and schema 
issues).

* TEI uses the document's logical structure as is starting point 
(chapters, paragraphs, etc), rather than the physical structure (i.e. 
volumes and pages, etc), but if you're working in a page-centric larger 
system, you want to pay special attention to how you're going to encode 
pagebreaks in the TEI (See 
http://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-pb.html) so you 
can coordinate correctly. And whether you want the links to the page 
images to appear in the METS or the TEI (or both). We use corresp's to 
point from pagebreaks to the associated images.

In out texts pagebreaks can occur almost anywhere and are marked as:

...<pb xml:id="n112" n="98" corresp="#MarAmon112"/>...

The @n is the page number printed on the page. And in the header:

<notesStmt xml:id="notesStmt-0001">
   <note xml:id="page-images">
     <list>
        ...
        <item>
          <figure xml:id="MarAmon112">
             <graphic url="MarAmon112.gif" mimeType="image/gif" 
xml:id="MarAmon112-g" n="fp98"/>
          </figure>
        </item>
        ...
      </list>
    </note>
</notesStmt>

For the file this example is taken from: 
http://www.nzetc.org/tm/scholarly/tei-MarAmon.html "Other formats" -> "TEI"

cheers
stuart

---

From: Andrew Ashton <[email-masked]>
Date: Wed, 4 Nov 2009 15:50:54 -0500

Chris,

In general I would say keep the documents separate, and build some sort of
process that combines the relevant parts of the documents for ingestion into
MarkLogic.  However, not having any idea how MarkLogic works, that solution
may not be an option.  We're doing quite a bit with TEI and MODS documents
in Solr, and we just have a service that generates a Solr indexing document
out of our XML.   

> So the TEI structure does not fit nicely into METS the way we're using METS.

In general, yeah.  You can put page breaks (<pb/>) in TEI, but it would be a
clunky way to address sections of the document from your METS.

> from TEI which is another issue.)

METS seems like a lower-level standard than the TEI header, so that makes
sense.

> point a search is deployed.

Does MarkLogic operate directly on the XML or does it index it?  If it is
running Xqueries or something like that, you may not see much of a
performance increase by splitting them out.  In fact I'd say that Xqueries
are typically a lot faster when they're operating on a single document or
collection of similar document.  You may also want to consider what comes
after MarkLogic. 
 
> link out to external TEI files.

That makes sense, although since your TEI would all be namespaced it
wouldn't be too hard to extract it if necessary.  I would be concerned with
the future ramifications of having your objects optimized for a legacy
system.   

-Andy

---

From: Christine Schwartz <[email-masked]>
Date: Wed, 4 Nov 2009 16:09:26 -0500

Mark Logic is a native XML database and most of the code is XQuery. So, it's
operating directly on the XML. It has specific optimized queries that it
runs. I like you idea about considering what comes after Mark Logic. I've
never been one for making metadata decisions based on the current system.

Thanks,

Chris

On Wed, Nov 4, 2009 at 3:50 PM, Andrew Ashton <[email-masked]>wrote:

> Chris,
>
> <snip>Does MarkLogic operate directly on the XML or does it index it?  If
> it is
> after MarkLogic.</snip>

---

From: Liza Daly <[email-masked]>
Date: Wed, 4 Nov 2009 16:31:58 -0500

On Wed, Nov 4, 2009 at 4:09 PM, Christine Schwartz
<[email-masked]> wrote:

> > Does MarkLogic operate directly on the XML or does it index it?

> runs.

That's what their marketing material says, but in fact it does index
the full-text content like a traditional search engine. You can also
run pure XQuery without setting up indexes, but you'll be limited in
the scope of queries available and the performance you'll get from
unindexed searches.

I agree that if it's possible, combining the files at the time of
injection into ML makes sense.  (On some of the projects I've worked
on with ML that would've been infeasible due to the scale of the data
and the frequency of updates.)

Liza