Edit report at http://bugs.php.net/bug.php?id=52860&edit=1
ID: 52860
Comment by: cataphr...@php.net
Reported by: cataphr...@php.net
Summary: htmlspecialchars/htmlentities stripping invalid
characters
Status: Open
Type: Feature/Change Request
Package: *General Issues
Operating System: Irrelevant
PHP Version: trunk-SVN-2010-09-16 (SVN)
Block user comment: N
New Comment:
HTML5 follows the same direction:
8.1.4 Character references
[...]
The numeric character reference forms described above are allowed to
reference any Unicode code point other than U+0000, U+000D, permanently
undefined Unicode characters (noncharacters), and control characters
other than space characters.
8.2.2.3 Preprocessing the input stream
[...]
All U+0000 NULL characters and code points in the range U+D800 to U+DFFF
in the input must be replaced by U+FFFD REPLACEMENT CHARACTERs. Any
occurrences of such characters and code points are parse errors.
Any occurrences of any characters in the ranges U+0001 to U+0008, U+000E
to U+001F, U+007F to U+009F, U+FDD0 to U+FDEF, and characters U+000B,
U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, U+2FFFE, U+2FFFF, U+3FFFE, U+3FFFF,
U+4FFFE, U+4FFFF, U+5FFFE, U+5FFFF, U+6FFFE, U+6FFFF, U+7FFFE, U+7FFFF,
U+8FFFE, U+8FFFF, U+9FFFE, U+9FFFF, U+AFFFE, U+AFFFF, U+BFFFE, U+BFFFF,
U+CFFFE, U+CFFFF, U+DFFFE, U+DFFFF, U+EFFFE, U+EFFFF, U+FFFFE, U+FFFFF,
U+10FFFE, and U+10FFFF are parse errors. These are all control
characters or permanently undefined Unicode characters (noncharacters).
Previous Comments:
------------------------------------------------------------------------
[2010-09-16 16:45:10] cataphr...@php.net
Where it says "invalid in XML or XHTML are not stripped out" it should
read "invalid in XML or HTML are not stripped out". The characters are
also invalid in HTML.
------------------------------------------------------------------------
[2010-09-16 13:57:42] cataphr...@php.net
Description:
------------
htmlspecialchars() and htmlentities() are commonly used to convert
user-supplied text into text that's safe to output in an HTML or XML
document.
Actually, they are insufficient for this purpose, because characters
that are invalid in XML or XHTML are not stripped out.
In HTML, this results in an invalid document.
In XML, the result is worse because one will end-up with malformed XML.
Therefore, sanitation with htmlspecialchars can result in corrupted
data.
Additionaly, when passed $double_encode == true, invalid character
entities (i.e. those which refer to invalid characters) should also be
stripped out.
See
* http://www.w3.org/TR/REC-xml/#NT-Char
* http://www.w3.org/TR/REC-xml/#NT-CharRef
Test script:
---------------
<?php
$mode = @$_GET["mode"];
if ($mode == "xhtml") {
header("Content-type: application/xhtml+xml; charset=utf-8");
$templ = <<<XML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";>
<html xmlns="http://www.w3.org/1999/xhtml";>
<head>
<title>Test</title>
<meta http-equiv="Content-type" content="application/xhtml+xml;
charset=utf-8" />
</head>
<body>
%s
</body>
</html>
XML;
}
elseif ($mode == "html") {
header("Content-type: text/html; charset=utf-8");
$templ = <<<HTML
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd";>
<html>
<head>
<title>Test</title>
</head>
<body>
%s
</body>
</html>
HTML;
}
else die("bad mode");
$data = "My data: <\x1F";
echo sprintf($templ, htmlentities($data, ENT_NOQUOTES, "UTF-8"));
Expected result:
----------------
At minimum, this should be documented in the manual pages for
htmlspecialchars and htmlentities.
A better solution would be to change those two functions to strip
characters outside the allowed range:
#x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
Another alternative, which wouldn't break BC, would be to add another
function or another flag to htmlentities/htmlspecialchars (in addition
to ENT_NOQUOTES/ENT_QUOTES/ENT_COMPAT) that would strip out these
characters, possible plus those that authors are "encouraged to avoid":
[#x7F-#x84], [#x86-#x9F], [#xFDD0-#xFDEF],
[#x1FFFE-#x1FFFF], [#x2FFFE-#x2FFFF], [#x3FFFE-#x3FFFF],
[#x4FFFE-#x4FFFF], [#x5FFFE-#x5FFFF], [#x6FFFE-#x6FFFF],
[#x7FFFE-#x7FFFF], [#x8FFFE-#x8FFFF], [#x9FFFE-#x9FFFF],
[#xAFFFE-#xAFFFF], [#xBFFFE-#xBFFFF], [#xCFFFE-#xCFFFF],
[#xDFFFE-#xDFFFF], [#xEFFFE-#xEFFFF], [#xFFFFE-#xFFFFF],
[#x10FFFE-#x10FFFF].
Actual result:
--------------
The W3C validator gives an error:
You have used an illegal character in your text. HTML uses the standard
UNICODE Consortium character repertoire, and it leaves undefined (among
others) 65 character codes (0 to 31 inclusive and 127 to 159 inclusive)
that are sometimes used for typographical quote marks and similar in
proprietary character sets. The validator has found one of these
undefined characters in your document. The character may appear on your
browser as a curly quote, or a trademark symbol, or some other fancy
glyph; on a different computer, however, it will likely appear as a
completely different character, or nothing at all.
------------------------------------------------------------------------
--
Edit this bug report at http://bugs.php.net/bug.php?id=52860&edit=1
