Comments on: Dealing with HTML Parsing Misery?

Question: Dealing with HTML Parsing Misery?

Captain_Tenille — Wed, 26 Oct 2005 15:01:48 -0800

I need to write something (in perl) to shorten the text of a link if it's a URL, but only a URL. I've played with a variety of regexps and banged my head against HTML::Parser, but I've gotten no love. Help!

As far as regexps, go, I've done OK with /\(.*?)\< \/a\>/gi for getting the link text in question out, and modifying it isn't a problem.

The problem arises when I try to put it back. All I've been able to do is either only replace part of the orginal link OR I get stuck in an endless loop. I've been escaping my > and < signs, but that doesn't seem to help at all. for examples of what i've been trying (where in each $z is a copy of the original $1 from the first regexp, and $modtxt is the text i want to replace), tt>s#">$z\< #$modtxt\#/tt> will replace the text, but mungs up the tag so the initial A tag isn't closed before the ending tag. s#">$z\< #\\>$modtxt\< #/tt>, on the other hand, gets stuck in an endless loop. I've been googling and banging my head against this for several days, and while I think I must be overlooking something really simple, I can't figure out what it is. Thus, I turn to Ask.Me for assistance. (btw, getting those regexps through was a surprisingly difficult undertaking)

By: Captain_Tenille

Captain_Tenille — Wed, 26 Oct 2005 15:07:49 -0800

AAAARGH. I swear, the regexps looked ok in preview.

These are the correct regular expressions:

First one: /\<a.*href=.*\>(.*?)\< \/a\>/gi

Second one: s/">$z\</$modtxt\</

Third one: s/">$z\</\"\>$modtxt\</

Hopefully they make it out of live preview.

By: smackfu

smackfu — Wed, 26 Oct 2005 15:10:59 -0800

When I'm messing with perl like this, I wrap everything possible in the whole string in parentheses (like /(^.*)(match regexp)(.*$)/, and then stick it back together with $1.$changed$.3. at the end. (I do it this way because I usually can't figure out the right way.)

By: kcm

kcm — Wed, 26 Oct 2005 15:11:43 -0800

you may want to use other characters for your regexp, at first glance: s!foo!bar!gi, e.g. If you use a character which isn't likely to be manipulated in the URL (now or later), you save some grief and readability.

By: Captain_Tenille

Captain_Tenille — Wed, 26 Oct 2005 15:17:48 -0800

kcm: tried that already. Still gets stuck in the endless loop.

By: flabdablet

flabdablet — Wed, 26 Oct 2005 15:21:16 -0800

I don't know Perl, but based on what I'd do in sed, maybe you could pull out all three parts of the search regexp like

/(\<a.*href=.*\>)(.*?)(\< \/a\>)/gi

then build the final string with $1$whatever$3 instead of doing a search-and-replace.

On preview: what smackfu said. Plus, this actually saves work (the rexexp match has already gone to the trouble of searching your text; why search it again for the replace?)

By: macrone

macrone — Wed, 26 Oct 2005 15:21:49 -0800

It's not necessary to escape angle brackets -- they have no special significance in regular expressions.

It's possible that $z (the copy of $1) contains metacharacters with significance in regular expressions -- but the nuances of how metacharacters are treated when interpolated via a variable are kind of mind-bending, and I don't have them at the top of my mind. Perhaps you could try:

my $z = quotemeta($1);

If you're positive that $z is an exact copy of $1 (and doesn't itself contain metacharacters), try:

TO CAPTURE:

m{<a.*href=[^>]+>\s*(.+?)\s*</a>}ig;

TO REPLACE:

s{>\s*$z\s*<}{>$modtxt<};

It isn't safe to assume that a quote mark (let alone a double-quote mark) will always appear before the first closing angle bracket, so I don't think it's required in your replacement pattern.

By: callmejay

callmejay — Wed, 26 Oct 2005 15:31:09 -0800

Use XML::Parser and XML::Writer and stop worrying about brackets. (I haven't used HTML::Parser, but if it's anything like XML::Parser, it's definitely the way to go.)

By: holloway

holloway — Wed, 26 Oct 2005 15:38:35 -0800

If you end up using XSLT, here's an identity template that'll do it,

<?xml version="1.0" encoding="UTF-8" ?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml"> <xsl:template match="text()[ancestor::a][contains(., '://')]"> <xsl:value-of select="substring(.,1,10)"/> </xsl:template> <xsl:template match="*|@*"> <xsl:copy><xsl:apply-templates/></xsl:copy> </xsl:template> </xsl:stylesheet>

This one will deal with the case of extra tags in the hyperlink too, eg, <a href="...">http://chance-to-advertise-my-site-in-code.com</a>

By: Ogre Lawless

Ogre Lawless — Wed, 26 Oct 2005 15:50:56 -0800

I'm a bit confused by what you mean by "endless loop" but the only thing you should need to escape is the / character, and only if you use the default characters. If as kcm and your original post suggest, not even that. Using the default slashset:

$_ = '<A href="http://metafilter.com">METAFILTER IS THE BEST!</A>';

if (/<a.*href=.*>(.*?)<\/a>/gi) {
my $modtxt = my $z = $1;
$modtxt =~ s/ IS THE BEST//; # for example
s/">$z</">$modtxt</;
}

print $_;

on preview, I'm repeating macrone a bit.

By: macrone

macrone — Wed, 26 Oct 2005 16:05:15 -0800

Looking over Ogre Lawless's comment and mine again, I think you should also make sure that all the wilcard matches are non-greedy. Any ".*" is liable to eat up much more text than intended, which could lead to nested matches and replacements, which I suppose could lead to some kind of evil recursion.

To restate my patterns:

TO CAPTURE:

m{<a.+?href=[^>]+>\s*(.+?)\s*</a>}igs;

(There's going to be at least a space after the <a, but you don't want to match into another link. You also want to match across linebreaks, thus the "s" modifier.)

TO REPLACE:

s{>\s*$z\s*<}{>$modtxt<};

By: AmbroseChapel

AmbroseChapel — Wed, 26 Oct 2005 16:13:12 -0800

How about you show us your whole code? That might help. The thing about the endless loop doesn't seem apparent from what you've given us.

The best way to get Perl help, as much as I love Ask, is PerlMonks.

By: woj

woj — Wed, 26 Oct 2005 16:23:11 -0800

Captain_Tenille, can you give an example of the sort of input you are expecting, and the resulting output that you'd like?

You say that you are trying to "shorten the text of a link if it's a URL, but only a URL". Maybe, I'm reading this the wrong way, but I take that to mean that you'd like to change link text like "http://www.somesite.com/dir/file.htm" into something like "somesite.com" or "file.htm", while leaving links with text like "My File" unchanged.

Is this what you are trying to accomplish?

By: Captain_Tenille

Captain_Tenille — Wed, 26 Oct 2005 16:27:59 -0800

woj: that's pretty much exactly what I'm trying to accomplish.

If anyone wants to see the code in question, what I've been working on can be seen here: linktest.pl. I would just post it, but I need to go watch my daughter for a bit and don't feel like futzing with formatting.

By: AmbroseChapel

AmbroseChapel — Wed, 26 Oct 2005 16:46:57 -0800

OK here's something:



#!/usr/bin/perl

undef $/;

open( HTMLFILE, "/usr/ambrose/file.html" ) || die "$!";

my $html = <HTMLFILE>;

close( HTMLFILE );



$html =~ s{(<a [^>]+>)([^< ]+)</a>}

         {$1 . munge($2) . '</a>'}egsi;



print $html;



sub munge() {

    my $tag_contents = shift();

    if ( $tag_contents =~ m|^http(s)?://|

        && length( $tag_contents ) > 32 )

    {

        $tag_contents = substr( $tag_contents, 0, 32 ) . '...';

    }

    return $tag_contents;

}

where the HTML file in question looks like this:



<a href="http://www.yahoo.com/">http://www.yahoo.com/</a>

Short URL as link text

<a href="http://www.yahoo.com/">Click here</a>

non-URL as link text

<a href="http://www.yahoo.com/foo/bar/baz/quux/">http://www.yahoo.com/foo/bar/baz/quux/</a> 

long URL as link text

What it should do is: ignore link content which isn't a URL, ignore URLs if they're less than 32 chars long, and change the ones which are longer into the first 32 chars, plus '...' to show you've truncated them.

How's that?

It outputs this on the test file:



<a href="http://www.yahoo.com/">http://www.yahoo.com/</a>

Short URL as link text

<a href="http://www.yahoo.com/">Click here</a>

non-URL as link text

<a href="http://www.yahoo.com/foo/bar/baz/quux/">http://www.yahoo.com/foo/bar/baz...</a> 

long URL as link text

By: macrone

macrone — Wed, 26 Oct 2005 16:54:37 -0800

AmbroseChapel: Unfortunately, your script also ignores link text that itself contains tags. You should change:

s{(<a [^>]+>)([^< ]+)</a>}

to:

s{(<a [^>]+>)(.+?)</a>}

Otherwise, I think your approach is best: to execute code in the replace pattern, rather than running two regexes across the same data.

By: whatnotever

whatnotever — Wed, 26 Oct 2005 16:59:46 -0800

On preview: Yeah, what AmbroseChapel said. But here it is anyway. Oh, and this just modifies your code, but whatever.

I think your problem is more in how you're using the while loop. I'd skip it and use the "e" modifier on the s/// operator. That lets you put an evaluated expression inside the replacement (which lets you run whatever code you like to generate the replacement). Then, the standard "g" modifier on the s/// will just make all of your replacements for you without "re-finding" them. Try this out:

$link =~ s/(<a.+?href[^>]+>)\s*(.+?)\s*(<\/a>)/$1 . &shortenMe($2) . $3/gei;

sub shortenMe {
my $input = shift;
return $input if $input !~ m#(^http://|^ftp://)#;

my @tok = split '/', $input;

my $protocol = shift @tok;
shift @tok; #off into space
my $domain = shift @tok;
my $remainder = shift @tok;
my $modtxt = "$protocol//$domain/...";
return $modtxt;
}

By: woj

woj — Wed, 26 Oct 2005 17:20:25 -0800

This might work for you, with a small amount of tweaking. Mine will use the host name from the href as the new link text, and uses Regex::Common:



use Regexp::Common qw /URI/;

use warnings;

use strict;



open(my $html, "<test.html");



while (<$html>) {

 m#(<a\s+href.*>)\s*(\S+)\s*(</a\s*>)#mi;

     my ($opentag, $linktext, $closetag) = ($1, $2, $3);



       if ($linktext =~ /$RE{URI}{HTTP}{-keep}/) {

              my $host = $3;

              print "Found URL as link text...\n";

              print "\tNew link is \'$opentag$host$closetag\'\n";

       }

}

The file "test.html" looks pretty much like AmbroseChapel's.

By: woj

woj — Wed, 26 Oct 2005 17:22:11 -0800

Oh yeah, and I wanted to mention that you can use Regex::Common to match on any type of URI, not just http.

By: woj

woj — Wed, 26 Oct 2005 17:29:55 -0800

Sorry to keep posting, but I just re-read your code, and if you change the if statement in my example to look like this, then it does what your script intends:



    if ($linktext =~ /$RE{URI}{HTTP}{-keep}/) {

        my ($proto, $host) = ($2,$3);

        print "Found URL as link text...\n";

        print "\tOld link was \'$opentag$linktext$closetag\'\n";

        print "\tNew link is \'$opentag $proto://$host... $closetag\'\n";

    }

By: AmbroseChapel

AmbroseChapel — Wed, 26 Oct 2005 17:32:09 -0800

Unfortunately, your script also ignores link text that itself contains tags.

But we don't need to change those ones!

I mean, you're right of course, but there's no URL which needs to be shortened which will be missed, is there?

Mind you, what happens if the link text contains something like:

http://www.blah.com/ I love this site!

then we're in trouble...

By: woj

woj — Wed, 26 Oct 2005 18:07:42 -0800

Mind you, what happens if the link text contains something like:

http://www.blah.com/ I love this site!

then we're in trouble...

Perhaps not the most elegant solution in the world, but this works even with links as ugly as...

<a href="http://ask.metafilter.com/mefi/26171">check out http://ask.metafilter.com/mefi/26171 for more info</a>

use Regexp::Common qw /URI/;
use warnings;
use strict;

open(my $html, "<test.html");

while (<$html>) {

m#(<a\s+href.*?>)(.*?)(</a\s*>)#i;
my ($opentag, $linktext, $closetag) = ($1, $2, $3);
my $replace = $opentag;
foreach my $chunk (split /\s+/,$linktext) {
if ($chunk=~/$RE{URI}{HTTP}{-keep}/ ){
my ($proto, $host) = ($2,$3);
$replace.=" $proto://$host... ";
}
else {
$replace.=" $chunk";
}

}
print "Replacement is $replace\n";
}

Sorry, I don't feel like adding in a bunch of nbsp's to indent it correctly.

By: AmbroseChapel

AmbroseChapel — Wed, 26 Oct 2005 18:18:09 -0800

Somewhat off-topic but I'm interested in this part:



open(my $html, "<test.html");

while (<$html>){

}

That wouldn't work if the tag was split across lines, would it? Say you had



<a 

href="foo">

for instance?

By: woj

woj — Wed, 26 Oct 2005 18:23:01 -0800

True, I guess you'd have to slurp in the file and use multi-line matching. I was just using the standard input record separator for simplicity's sake when I was testing it out. Honestly, I didn't realize that you could put newlines within the tags without choking the browser. :)

By: AmbroseChapel

AmbroseChapel — Wed, 26 Oct 2005 18:27:54 -0800

And while I'm nit-picking, this:

m#(<a\s+href.*>)\s*(\S+)\s*(</a\s*>)#mi;

is going to give problematic results if there are two a tags on the same line, because of the ".*" being greedy.



<a href="http://foo.com/">foo</a>, <a href="http://bar.com/">bar</a>

for instance will get you everything up to the second "bar" in $1.

I didn't realize that you could put newlines within the tags without choking the browser. :)

Any whitespace is legal, including returns. I've been bitten before...

By: woj

woj — Wed, 26 Oct 2005 19:02:12 -0800

Any whitespace is legal, including returns. I've been bitten before...

I actually don't work with html documents too often, so that is good to know. The missing ? was a typo on my part, but yes, there would be a problem with two tags on the line. And with images as links, assuming he doesn't want to break those, and probably a bunch of other problems I can't think of right now... I think I'm gonna sit out the rest of this one. Good luck.