Thanks very much for your excellent report, and for confirmation of the fix!
This solution for Tomcat 7 almost merits inclusion in the user manual, although the improved defaults in Tomcat 8 make this a little less critical. Al least this thread will help users. Kind regards, Ben. On 06/08/16 20:20, Saka Royban wrote: > Thanks a world BenYour help worth a million for me.The 2nd link did the > trick. Adding URIEncoding="UTF-8" or useBodyEncodingForURI ="true" to > server.xml was the right solution for Tomcat 7.Also, upgrading to Tomcat 8 > solves everything without a need to add above codes. According to 2nd link, > "In Tomcat 8 starting with 8.0.0 (8.0.0-RC3, to be specific), default value > (strict compliance is off) of URIEncoding is now UTF-8." > Thanks again > > From: Ben Caradoc-Davies <[email protected]> > To: Saka Royban <[email protected]>; GeoServer Mailing List List > <[email protected]> > Sent: Saturday, August 6, 2016 9:48 AM > Subject: Re: [Geoserver-users] GeoServer doesn't understand non-english > Characters in cql filter > > GeoServer expects URL-encoded parameters as UTF-8 regardless of the host > operating system character set. This is correct. > > Some languages can encode characters in different ways in Unicode, for > example using Unicode combining characters. This can affect comparisons. > Could your data have different bytes to your filter literal? What is the > exact byte sequence stored in your shapefile for the feature with the > property with value 'سامی'? That is, does it have the same UTF-8 byte > sequence? Do you see the same bytes in a WFS response? > > What is your servlet container? Tomcat or the embedded Jetty? There have > been reports of CQL encoding issues with some Tomcat configurations. > Quite old but may still be valid: > http://osgeo-org.1560.x6.nabble.com/CQL-Filter-Syntax-td3806188.html > https://sourceforge.net/p/geoserver/mailman/message/21111455/ > > One reported fix for Tomcat is to set URIEncoding="UTF-8" on your > <Connector> in server.xml, but I have not tried this myself: > https://wiki.apache.org/tomcat/FAQ/CharacterEncoding > > Kind regards, > Ben. > > On 06/08/16 15:26, Saka Royban wrote: >> >> Thanks a lot Ben for your helpSo, if CQL-filter can handle non-ASCII >> characters, it is misunderstanding of me. I've may done a step wrongly. >> 1. Yes. My shapefile store is declared as UTF-8 charset in GeoServer. Plain >> wfs requests returns correct data (=correct non-english characters). In >> fact, there is no problem without CQL-filter. To add more, my shapefile is >> an output of ArcGIS which in its recent versions only exports to UTF-8. >> (There is .cpg file which has been set to UTF-8, after export.) >> 2. My system encoding is Windows-1256 (the encoding that is used by default >> in word editors). My webpage is declared in UTF-8. So, my javascript codes >> must be in UTF-8 (I've also set encoding in notepad++ to UTF-8. If i change >> that to Win-1256, i see some dummy characters which proves me that my filter >> string is encoded in UTF-8 correctly) >> 3. OpenLayers sends following string as part of URL to >> server:CQL_FILTER=NAME%20%3D%20%27%D8%B3%D8%A7%D9%85%DB%8C%27Using online >> decoders, i get back correct filter string. >> 4. About log file, i tried different encoding in Notepad++ but unfortunately >> i can't see the filter string correctly. My encodings are restricted to two: >> UTF-8 and Win-1256. Filter string isn't displayed correctly in none of them. >> I've also tried another shapefile in Win-1256 encoding. Changing meta tag of >> webpage to chareset=windows-1256, writing againg filter string but in this >> case, i can't get a good result as well.(Note: I noticed that Openlayers >> encodes filter in win-1256 exactly as it was in UTF-8. Is it right?) >> Thanks again for your help and patience >> >> From: Ben Caradoc-Davies <[email protected]> >> To: Saka Royban <[email protected]>; GeoServer Mailing List List >> <[email protected]> >> Sent: Saturday, August 6, 2016 1:29 AM >> Subject: Re: [Geoserver-users] GeoServer doesn't understand non-english >> Characters in cql filter >> >> Yes, non-ASCII characters are supported in CQL and XML filters. >> Successful filtering also requires correct configuration of your data >> source. >> >> What charset is declared in your GeoServer shapefile data store >> configuration? Is it UTF-8? >> http://docs.geoserver.org/latest/en/user/data/vector/shapefile.html >> >> What is your platform encoding (the charset of your operating system)? >> Log files are written in the encoding of your platform, so if your >> platform cannot encode these characters, placeholders characters will be >> written, even when filtering is working correctly. >> >> Kind regards, >> Ben. >> >> On 06/08/16 05:09, Saka Royban wrote: >>> Hi all have a shapefile loaded in GeoServer 2.8 and accessed by OpenLayers >>> 2.13. This layer is in UTF-8 encoding and has some english and non-english >>> records. >>> When i use english fields in cql-filter - as follows, but no differnce >>> between WMS and WFS request- GeoServer filters data very well and returns >>> the result. layer1 = new ol.layer.Image({ >>> title: 'Ostan', >>> source: new ol.source.ImageWMS({ >>> url: 'http://localhost:8080/geoserver/wms', >>> params: { >>> LAYERS: 'ws:layer1', >>> VERSION: '1.1.1', >>> CQL_FILTER: "NAME_ENG = 'sommy'" >>> } >>> }) >>> });But if i try to use a field which contains non-english data in >>> cql-filter, GeoServer returns no data. My webpage has following meta, as >>> well: <meta content="text/html; charset=utf-8" http-equiv="content-type" >>> />When i check geoserver log, i see some messy characters instead of my >>> non-english string in cql-filter. >>> | >>> | | >>> >>> | Using Firebug, in Parameteres section, everything is correct. I see >>> "CQL-Filter: Name = 'سامی' " which is right for my language. But in >>> Geoserver log, i see some dummy characters instead of abovementioned >>> characters >>> Can anyone confirm that using characters other than ASCII is possible in >>> CQL-Filter? >>> >>> Regards | >>> >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> >>> >>> >>> _______________________________________________ >>> Geoserver-users mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/geoserver-users >>> >> > -- Ben Caradoc-Davies <[email protected]> Director Transient Software Limited <http://transient.nz/> New Zealand ------------------------------------------------------------------------------ _______________________________________________ Geoserver-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geoserver-users
