Re: Indexing Text Files and Text Encoding

2015-01-22 Thread Dominik Haumann
On Thursday 22 January 2015 17:21:26 Vishesh Handa wrote: > On Wed, Jan 21, 2015 at 10:20 PM, Albert Astals Cid wrote: > > > > > > > That's exactly what I've currently implemented. I was a little concerned > > > about discarding the file. Oh well. > > > > Maybe you can also use KEncodingProber as

Re: Indexing Text Files and Text Encoding

2015-01-22 Thread Vishesh Handa
On Wed, Jan 21, 2015 at 10:20 PM, Albert Astals Cid wrote: > > > > That's exactly what I've currently implemented. I was a little concerned > > about discarding the file. Oh well. > > Maybe you can also use KEncodingProber as third step? Good idea. I'll implement it. Thanks -- Vishesh Handa

Re: Indexing Text Files and Text Encoding

2015-01-21 Thread David Narvaez
On Wed, Jan 21, 2015 at 4:20 PM, Albert Astals Cid wrote: > El Dimecres, 21 de gener de 2015, a les 13:24:02, Vishesh Handa va escriure: >> On Wednesday, January 21, 2015 11:51:13 AM Milian Wolff wrote: >> > Or just use Qt directly: >> > >> > http://stackoverflow.com/questions/18227530/check-if-ut

Re: Indexing Text Files and Text Encoding

2015-01-21 Thread Albert Astals Cid
El Dimecres, 21 de gener de 2015, a les 13:24:02, Vishesh Handa va escriure: > On Wednesday, January 21, 2015 11:51:13 AM Milian Wolff wrote: > > Or just use Qt directly: > > > > http://stackoverflow.com/questions/18227530/check-if-utf-8-string-is-valid > > -i n-qt/18228382#18228382 > > > > If it

Re: Indexing Text Files and Text Encoding

2015-01-21 Thread Vishesh Handa
On Wednesday, January 21, 2015 11:51:13 AM Milian Wolff wrote: > Or just use Qt directly: > > http://stackoverflow.com/questions/18227530/check-if-utf-8-string-is-valid-i > n-qt/18228382#18228382 > > If it fails, either discard the file. Or try again with the system encoding > (if that is not UT

Re: Indexing Text Files and Text Encoding

2015-01-21 Thread Milian Wolff
On Tuesday 20 January 2015 13:18:03 David Narvaez wrote: > On Tue, Jan 20, 2015 at 12:10 PM, Vishesh Handa wrote: > > Hey guys > > > > We have a plain text indexing plugin in KFileMetaData. It gives the plain > > text of any file whose mimetype beings with 'text/'. We used to use > > QString::fro

Re: Indexing Text Files and Text Encoding

2015-01-20 Thread David Narvaez
On Tue, Jan 20, 2015 at 12:10 PM, Vishesh Handa wrote: > Hey guys > > We have a plain text indexing plugin in KFileMetaData. It gives the plain > text of any file whose mimetype beings with 'text/'. We used to use > QString::fromUtf8 to convert this into a string. However, this may not be > ide

Indexing Text Files and Text Encoding

2015-01-20 Thread Vishesh Handa
Hey guys We have a plain text indexing plugin in KFileMetaData. It gives the plain text of any file whose mimetype beings with 'text/'. We used to use QString::fromUtf8 to convert this into a string. However, this may not be ideal as a different encoding can exist. I've just written a patch to us