Welcome, everyone. This is the NCBI Minute on how to use EDirect's Xtract utility to
parse BLAST XML output.
What I will do today is -- remind you about the EDirect command line programs that Eric
talked about -- almost 3 weeks ago. Then I will show you a little about the BLAST xml
output and tell you why you might want to use that and then I will do a couple of quick
examples using EDirect to reformat the xml output into a human readable form and it might
be a good way to save your BLAST results for whatever you want to do with them.
As mentioned previously, Eric gave a full tutorial on using EDirect, the command line
utility for accessing the E-utilities. It is a way of using the E-utilities without
having to write a script yourself. This is a set of scripts that search and retrieve
information from Entrez. What comes with it is a utility called Xtract, a general xml
parser and we will show you how to use it to parse another kind of NCBI XML, the XML
produced by BLAST. If you look at the command line there that Eric has from his talk a few
weeks ago it is piped into Xtract at the end to produce a report from this particular set
of commands they issued through the EDirect scripts.
This is looking at the standalone options -- BLAST produces different structured outputs.
Structured outputs, some in particular, have more complete information than any of the
other flatfile outputs. Most people are familiar with the pairwise. It doesn't have everything
in it. The other formats like ASN.1, XML, Jason, have more information. The XML output
formats are probably the most convenient for parsing because you can use EDirect to do
this. There are two different kinds -- the traditional BLAST XML and the XML2 which is
relatively new.
You can also get these from a web service. What I'm talking about today is perfectly
general and in fact I will show you the example that I'm using coming from doing a web search
and I'll show you why you might want to do this from a web search although it may not
be immediately obvious why you would want to do that.
If you expand the download tab on the blast page or the download section on the blast
results page you can get a lot of different formats. The classic BLAST XML comes down
as a single XML document and the XML 2 is a zip file because it has an individual document
for each of the queries and then there is an Xinclude document to let you put those
back together if you need to. Sometimes it's convenient to have them separate and sometimes
it's convenient to have them as one document; it depends on what you are trying to do.
Here is a view of the XML, a little bit. The classic BLAST XML we've had around for a long
time, you get a single document to the whole thing. We'll talk in a minute about when you
are parsing this -- the thing that contains the results in this xml is Iteration. For
the BLAST XML 2 as I said you have a master document and a document for each query. There's
a link to more information about the XML 2 if you want to read more about it. If you
need to combine them, as we'll do today, you can use this utility called xmllint to pull
it together for processing.
The XML2 has a number of advantages. One is that it has full support for taxonomy so you
can get the tax ID and the scientific name out of that. It also supports iterative blast,
like psi-blast and blast 2 sequences, which are not supported properly in the old classic
blast XML. And for parsing this, the container that has things in it that we want is results.
As you saw in Eric's talk -- you can go back and review that if you want -- when we parse
the XML using Xtract we are using nested for loops and there is a hierarchy based on some
of the exploration commands that are part of EDirect. We'll be using pattern as our highest level,
that's the highest level that you usually use, and block to indicate a lower level -- the
pattern, for example in the classic XML will be iteration, and then we can look at the
individual hits if we're trying to pull those out of a block.
Before we go on to the demo I want to point out that there is a document -- chapter 6
in the E-utilities book -- if you have any questions, that's the place to look. We don't
have time today to go into the detail about they'll how these particular command lines
were put together but I will show you what they can do and how they can produce reports
from the BLAST XML.
I will switch over. I'm going to pull up some text files that I have.
This text file is available on the FTP site. I'm going to show you a blast result on the
web. I will show you how to download the XML and then we will go to the command line and
show you how produce reports from that.
Let me get this BLAST RID up. This is a saved search that I made the other day. I will copy
the URL and paste it into a web browser.
This is the result you might see if you did a Web blast. Many people do web BLAST searches
with multiple sequences and in this case I had many sequences that I took the from a
meta-genome from sheep, sheep gut. One thing that is a little bit of a problem for some people
using web BLAST is that you have this multipart output. So to see the results you need to
click on this to get to a new page and some people don't like that and they like to have
the whole thing. What you can do is go here to the download tab -- download section -- notice
that you can download the XML and XML 2. That is a handy thing to do. A lot of people like
to archive this stuff -- I've heard a lot of reports of people trying to save the pairwise
output or trying to print out. If you save something like a structured output format you can go
back to it and generate just about anything you want from that particular kind of report.
I've saved the XML and XML 2 and I will show you that briefly here -- back on my text editor.
Here's a classic blast xml here -- you can see what it looks like.
You can see the structure here. Let me see if I can make this a little bigger.
Here you can see the iteration -- the thing that we are going to cycle over to pull things
out -- you can see the hits here -- each hit will have an HSP which represents in alignment
-- I can get out of the output if I want to -- I can pull out the query sequence and hit
sequence and I can reconstruct blast paiewise. Or any kind of report that suits my fancy
from this output.
If you looked at the XML 2 what you'd find is that you have this XML include document
that basically gives you a way of organizing this so you could pull this together to make
one document. Otherwise what you get is an individual document for each query sequence.
Here is what one of those looks like. What's good about the XML2 -- it's organized a little
bit more rationally. Instead of "Iteration" have "results" -- we have reserved "iteration"
for iterative blast and you can also pull out some interesting things from this, for
example you can get the tax ID shown here. And a scientific name shown here -- these
are the things you might want to get in the blast output.
Let's take a minute and show you some simple command lines you can do with Xtract to get
things out of here.
These are coming from that text document. I have the webinar notes here, I've got the
traditional BLAST XML and then in the subdirectory I have the ones from XML 2. I will show you
a couple of commands. I'll go back to the notepad here -- my webinar notes -- basically
what you do to run one of these Xtract scripts is you have to take an input file -- what
you can do is cat the file then pipe it into xtract -- for example, the simple thing to
do is to get the XML structure. If I take this one and go ahead and pipe that into another
command, more -- that's a quick way to see how this is structured and then you can pick
out -- you might want to go through the iteration there to get your particular hits out.
Likewise you can do the same thing for XML 2. I have a commandline that does that. In
this case what I have to do is use the Unix commandline XML tool to put those files together
before I send them to xtract. I can copy that. I can paste it here. Again I'll pipe it to
more.
So I can see the way the blast XML2 is structured and I can focus on the results here to get
out my particular hits that I want to.
I'm show you one more. This is one where you can take the XML to produce something equivalent
to a tab-delimited output or something like that. A simple thing to do -- I will focus
on XML 2. One of the simplest things to do is -- suppose you want to find out what the
best hit is for every query that I used. I can use this commandline here and this particular
commandline argument for xtract called first, which gives me the first of each kind of element
and presents that.
I will copy that one. Let's pipe that into more. And unfortunately, I've got the font
pumped up so it's a little bit awkward to read this but if I made it smaller you
couldn't see it very well. I hope you can see this.
Here I've basically got it on each line. I've got the query ID, the subject ID. the title
of the hit sequence. So on and so forth. That's a handy thing to get out of there.
A more complete set of things I could do if I wanted -- actually I have the scientific
name on the last one there. If I go back here to the second script -- the second command
line -- will basically present all the hits for each sequence but I can use another -- and
it comes out as a long line -- what I can do that's a little more useful -- it's this
long commandline here that takes advantage of the ability to store an element as a variable
and Eric talked about this in his talk the other week.
Basically I can take the query ID, which is in a different part of the XML and repeat
that when I print my elements. So that comes first and then I'll get the ID of the hit
sequence, the title, the E value, and the scientific name of the organism.
So I'll go ahead and just put that one in there and I'll go ahead and pipe that to more.
Again, we have the problem of wrapping but I think you can see what is happening here.
Here is my query sequence. Here is my first hit sequence and the query sequences change
as we go down and we get different ones.
I've got each hit, the query sequence, the subject sequence, the title of the thing I
hit, the E value here, then the scientific name of the organism.
That is plenty to show you today for a short webinar. I'll stop talking now. We will leave
the room open for about five minutes if anyone has questions. If you haven't already asked,
we can entertain those now.
While you think about those, let me show you a couple of things here.
For more information, remember we have a webinars and courses page, a YouTube channel -- this
recording will be up on the YouTube channel. We have a blog. In fact there's an article
about using EDirect that's worth your time to look at. We have email list announcements
and since we talked about two different different things -- EDirect and BLAST -- there is a BLAST
announce and an E-Utilities announce email alias that might be useful to subscribe
to and that link shows you how to subscribe.
As always if you have questions in general, you can write to us at, info@ncbi.nlm.nih.gov.
We don't have any questions?
Okay, I will go ahead and wrap up the presentation. If you think of anything afterwards, you can
always write to us at webinars@ncbi.nlm.nih.gov.
And we will do some others in the series of using EDirect with BLAST. In future webinars
we will talk about how to take the output and pipe it back into EDirect to get objects
down from Entrez, for example taxonomy information or to get the sequences back from Entrez and
things like that. Thanks for attending. I will end the meeting
now.