Edit report at http://bugs.php.net/bug.php?id=52860&edit=1
ID: 52860
Updated by: cataphr...@php.net
Reported by: cataphr...@php.net
Summary: htmlspecialchars/htmlentities stripping invalid
characters
-Status: Open
+Status: Closed
Type: Feature/Change Request
Package: *General Issues
Operating System: Irrelevant
PHP Version: trunk-SVN-2010-09-16 (SVN)
-Assigned To:
+Assigned To: cataphract
Block user comment: N
New Comment:
Implemented not exactly stripping, but conversion to U+FFFD for trunk
with ENT_DISALLOWED.
Previous Comments:
------------------------------------------------------------------------
[2010-10-24 17:01:05] cataphr...@php.net
Automatic comment from SVN on behalf of cataphract
Revision: http://svn.php.net/viewvc/?view=revision&revision=304705
Log: - Completed rewrite of html.c. Except for determine_charset, almost
nothing
remains.
- Fixed bug on determine_charset that was preventing correct detection
in
combination with internal mbstring encoding "none",
"pass" or "auto".
- Added profiles for entity encode/decode for HTMl 4.01, XHTML 1.0, XML
1.0
and HTML 5. Added the constants ENT_HTML401, ENT_XML1, ENT_XHTML and
ENT_HTML5.
- htmlentities()/htmlspecialchars(), when told not to double encode,
verify
the correctness of the existenting entities more thoroughly.
It is checked whether the numerical entity represents a valid unicode
code
point (number is between 0 and 0x10FFFF). If using the flag
ENT_DISALLOWED,
it is also checked whether that numerical entity is valid in selected
document. In HTML 4.01, all the numerical entities that represent a
Unicode
code point (< U+10FFFFFF) are valid, but that's not the case with
other
document types. If the entity is not valid, & is encoded to
&amp;.
For named entities, the check is also more thorough. While before the
only
check would be to determine if the entity was constituted by
alphanumeric
characters, now it is checked whether that entity is necessarily
defined for
the target document type. Otherwise, & is encoded to &amp;.
- For html_entity_decode(), only valid numerical and named entities (as
defined
above for htmlentities()/htmlspecialchars() + !double_encode) are
decoded.
But there is in this case one additional check. Entities that
represent
non-SGML or otherwise invalid characters are not decoded. Note that,
in
HTML5, U+000D is a valid literal character, but the entity &#x0D
is not
valid and is therefore not decoded.
- The hash tables lazily created for decoding in html_entity_decode()
that were
added recently were substituted by static hash tables. Instead of 1
hash
table per encoding, there's only one hash table per document type
defined in
terms of unicode code points. This means that for charsets other than
UTF-8
and ISO-8859-1, a conversion to unicode code points is necessary
before
decoding.
- On the encoding side, the ad hoc ranges of entities of the translation
tables, which mapped (in general) non-unicode code points to HTML
entities
were replaced by three-stage tables for HTML 4 and HTML 5. This
mapping
tables are defined only in terms of unicode code points, so a
conversion
is necessary for charsets other than UTF-8 and ISO-8859-1. Even so,
the
multi-stage table is much faster than the previous method, by a factor
of 5; the conversion to unicode is a small penalty because it's just a
simple table lookup.
XML 1.0/htmlspecialchars() uses a simple table instead of a
three-stage
table.
- Added the flag ENT_SUBSTITUTE, which makes
htmlentities()/htmlspecialchars()
replace the invalid multibyte sequences with U+FFFD (UTF-8) or
&#FFFD;
(other encodings).
- Added the flag ENT_DISALLOWED. Implements FR #52860. Characters that
cannot
appear literally are replaced by U+FFFD (UTF-8) or &#FFFD;
(otherwise).
An alternative implementation would be to encode those characters into
numerical entities, but that would only work in HTML 4.01 due to
limitations
on the values of numerical entities in other document types. See also
the
effects on htmlentities()/htmlspecialchars() with !double_encode
above.
------------------------------------------------------------------------
[2010-09-30 14:23:56] cataphr...@php.net
The SGML declaration for HTML 4.01 allows a bigger charset than that of
HTML 4:
http://www.w3.org/TR/1999/PR-html40-19990824/sgml/sgmldecl.html
------------------------------------------------------------------------
[2010-09-30 14:07:39] cataphr...@php.net
Relevant for HTML 4:
http://www.webreference.com/dlab/books/html/3-4.html#3-4-1
------------------------------------------------------------------------
[2010-09-18 16:54:09] cataphr...@php.net
HTML5 follows the same direction:
8.1.4 Character references
[...]
The numeric character reference forms described above are allowed to
reference any Unicode code point other than U+0000, U+000D, permanently
undefined Unicode characters (noncharacters), and control characters
other than space characters.
8.2.2.3 Preprocessing the input stream
[...]
All U+0000 NULL characters and code points in the range U+D800 to U+DFFF
in the input must be replaced by U+FFFD REPLACEMENT CHARACTERs. Any
occurrences of such characters and code points are parse errors.
Any occurrences of any characters in the ranges U+0001 to U+0008, U+000E
to U+001F, U+007F to U+009F, U+FDD0 to U+FDEF, and characters U+000B,
U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, U+2FFFE, U+2FFFF, U+3FFFE, U+3FFFF,
U+4FFFE, U+4FFFF, U+5FFFE, U+5FFFF, U+6FFFE, U+6FFFF, U+7FFFE, U+7FFFF,
U+8FFFE, U+8FFFF, U+9FFFE, U+9FFFF, U+AFFFE, U+AFFFF, U+BFFFE, U+BFFFF,
U+CFFFE, U+CFFFF, U+DFFFE, U+DFFFF, U+EFFFE, U+EFFFF, U+FFFFE, U+FFFFF,
U+10FFFE, and U+10FFFF are parse errors. These are all control
characters or permanently undefined Unicode characters (noncharacters).
------------------------------------------------------------------------
[2010-09-16 16:45:10] cataphr...@php.net
Where it says "invalid in XML or XHTML are not stripped out" it should
read "invalid in XML or HTML are not stripped out". The characters are
also invalid in HTML.
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
http://bugs.php.net/bug.php?id=52860
--
Edit this bug report at http://bugs.php.net/bug.php?id=52860&edit=1
