PhpRiot
News Archive
PhpRiot Newsletter
Your Email Address:

More information

Lexentity

Note: This article was originally published at Planet PHP on 29 May 2012.
Planet PHP

A very long time ago (three and a half livers ago), I wrote a little utility to help us with the 2008 edition of PHP Advent. The utility is called Lexentity, and my recent blogging uptake made me realize that I've never actually written about it on here, so here it is (mostly borrowed from the README).

Let's face it--this sentence is much "uglier" than the one below it.
Let's face it-this sentence is much aoprettiera than the one above it.

Lexentity is a simple piece of software that takes html as input and outputs a context-aware, medium-neutral representation of that html, with apostrophes, quotes, emdashes, ellipses, accents, etc., replaced with their respective numeric XML/Unicode entities.

Context is important. It is especially important when considering a piece of html like this:

aand here's the example code:

echo "watermelon!\n";

Contextually, you'd want here's to become here's (note the apostrophe), but you certainly don't want the code to read echo aowatermelon!\na;.

A fancy/smart/curly quotes apostrophe is appropriate, but curly quotes in the code are likely to cause a parse error.

Lexentity understands its context, and acts appropriately, by means of lexical analysis, and turning tokens into text, not through a mostly-naive and overly-complicated regular expression.

Regarding context, my friend and former colleague Jon Gibbins said it best in this piece on his blog: In modern systems, you can't count on your html to always be represented as html. It's often (poorly) embedded in RSS or other html-like media, as XML.

Therefore, it is important to avoid html-specific entities like ” and …, and instead use their Unicode code point to form numeric entities such as .... This ensures proper display on any (for small values of aoanya) terminal that can properly render Unicode XML, and avoids missing entity errors.

You can try a demo at http://files.seancoates.com/lexentity, and the (PHP) code is available on GitHub.

We still use it for PHP Advent, and I ran this post through it. (-: