This could also simply be your browser isn't set up to display UTF-8, the characters may be just fine.
Best, Erick On Fri, Dec 18, 2015 at 12:58 AM, Zheng Lin Edwin Yeo <edwinye...@gmail.com> wrote: > Thanks for all your replies. > > I did chance upon this question from stackoverflow which it says is able to > solve the issues: > http://stackoverflow.com/questions/32354209/apache-tika-extract-scanned-pdf-files/ > > However, when I tried to run it, it still get the same "?????" output in > the content, the same as what I get from the Tika app. > > Regards, > Edwin > > > On 17 December 2015 at 23:58, Walter Underwood <wun...@wunderwood.org> > wrote: > >> PDF isn’t really text. For example, it doesn’t have spaces, it just moves >> the next letter over farther. Letters might not be in reading order — two >> column text could be printed as horizontal scans. Custom fonts might not >> use an encoding that matches Unicode, which makes them encrypted (badly). >> And so on. >> >> As one of my coworkers said, trying to turn a PDF into structured text is >> like trying to turn hamburger back into a cow. >> >> PDF is where text goes to die. >> >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> >> >> > On Dec 17, 2015, at 2:48 AM, Charlie Hull <char...@flax.co.uk> wrote: >> > >> > On 17/12/2015 08:45, Zheng Lin Edwin Yeo wrote: >> >> Hi Alexandre, >> >> >> >> Thanks for your reply. >> >> >> >> So the only way to solve this issue is to explore with PDF specific >> tools >> >> and change the encoding of the file? >> >> Is there any way to configure it in Solr? >> > >> > Solr uses Tika to extract plain text from PDFs. If the PDFs have been >> created in a way that Tika cannot easily extract the text, there's nothing >> you can do in Solr that will help. >> > >> > Unfortunately PDF isn't a content format but a presentation format - so >> extracting plain text is fraught with difficulty. You may see a character >> on a PDF page, but exactly how that character is generated (using a >> specific encoding, font, or even by drawing a picture) is outside your >> control. There are various businesses built on this premise - they charge >> for creating clean extracted text from PDFs - and even they have trouble >> with some PDFs. >> > >> > HTH >> > >> > Charlie >> > >> >> >> >> Regards, >> >> Edwin >> >> >> >> >> >> On 17 December 2015 at 15:42, Alexandre Rafalovitch <arafa...@gmail.com >> > >> >> wrote: >> >> >> >>> They could be using custom fonts and non-Unicode characters. That's >> >>> probably something to explore with PDF specific tools. >> >>> On 17 Dec 2015 1:37 pm, "Zheng Lin Edwin Yeo" <edwinye...@gmail.com> >> >>> wrote: >> >>> >> >>>> I've checked all the files which has problem with the content in the >> Solr >> >>>> index using the Tika app. All of them shows the same issues as what I >> see >> >>>> in the Solr index. >> >>>> >> >>>> So does the issues lies with the encoding of the file? Are we able to >> >>> check >> >>>> the encoding of the file? >> >>>> >> >>>> >> >>>> Regards, >> >>>> Edwin >> >>>> >> >>>> >> >>>> On 17 December 2015 at 00:33, Zheng Lin Edwin Yeo < >> edwinye...@gmail.com> >> >>>> wrote: >> >>>> >> >>>>> Hi Erik, >> >>>>> >> >>>>> I've shared the file on dropbox, which you can access via the link >> >>> here: >> >>>>> >> >>> >> https://www.dropbox.com/s/rufi9esmnsmzhmw/Desmophen%2B670%2BBAe.pdf?dl=0 >> >>>>> >> >>>>> This is what I get from the Tika app after dropping the file in. >> >>>>> >> >>>>> Content-Length: 75092 >> >>>>> Content-Type: application/pdf >> >>>>> Type: COSName{Info} >> >>>>> X-Parsed-By: org.apache.tika.parser.DefaultParser >> >>>>> X-TIKA:digest:MD5: de67120e29ec7ffa24aec7e17104b6bf >> >>>>> X-TIKA:digest:SHA256: >> >>>>> d0f04580d87290c1bc8068f3d5b34d797a0d8ccce2b18f626a37958c439733e7 >> >>>>> access_permission:assemble_document: true >> >>>>> access_permission:can_modify: true >> >>>>> access_permission:can_print: true >> >>>>> access_permission:can_print_degraded: true >> >>>>> access_permission:extract_content: true >> >>>>> access_permission:extract_for_accessibility: true >> >>>>> access_permission:fill_in_form: true >> >>>>> access_permission:modify_annotations: true >> >>>>> dc:format: application/pdf; version=1.3 >> >>>>> pdf:PDFVersion: 1.3 >> >>>>> pdf:encrypted: false >> >>>>> producer: null >> >>>>> resourceName: Desmophen+670+BAe.pdf >> >>>>> xmpTPg:NPages: 3 >> >>>>> >> >>>>> >> >>>>> Regards, >> >>>>> Edwin >> >>>>> >> >>>>> >> >>>>> On 17 December 2015 at 00:15, Erik Hatcher <erik.hatc...@gmail.com> >> >>>> wrote: >> >>>>> >> >>>>>> Edwin - Can you share one of those PDF files? >> >>>>>> >> >>>>>> Also, drop the file into the Tika app and see what it sees directly >> - >> >>>> get >> >>>>>> the tika-app JAR and run that desktop application. >> >>>>>> >> >>>>>> Could be an encoding issue? >> >>>>>> >> >>>>>> Erik >> >>>>>> >> >>>>>> — >> >>>>>> Erik Hatcher, Senior Solutions Architect >> >>>>>> http://www.lucidworks.com <http://www.lucidworks.com/> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>>> On Dec 16, 2015, at 10:51 AM, Zheng Lin Edwin Yeo < >> >>>> edwinye...@gmail.com> >> >>>>>> wrote: >> >>>>>>> >> >>>>>>> Hi, >> >>>>>>> >> >>>>>>> I'm using Solr 5.3.0 >> >>>>>>> >> >>>>>>> I'm indexing some PDF documents. However, for certain PDF files, >> >>> there >> >>>>>> are >> >>>>>>> chinese text in the documents, but after indexing, what is indexed >> >>> in >> >>>>>> the >> >>>>>>> content is either a series of "??????" or an empty content. >> >>>>>>> >> >>>>>>> I'm using the post.jar that comes together with Solr. >> >>>>>>> >> >>>>>>> What could be the reason that causes this? >> >>>>>>> >> >>>>>>> Regards, >> >>>>>>> Edwin >> >>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> > >> > >> > -- >> > Charlie Hull >> > Flax - Open Source Enterprise Search >> > >> > tel/fax: +44 (0)8700 118334 >> > mobile: +44 (0)7767 825828 >> > web: www.flax.co.uk >> >>
