Re: [gdal-dev] gdal create tif, errors with gdaladdo

Sat, 21 Mar 2009 16:32:33 -0700

Just to report back the fix below solved my problem. Reprocessed all the imagery and no errors reported, and no nasty artifacts on the images. 

Many thank to Even for his copious help and to the whole GDAL team.

-Steve

Stephen Woodbridge wrote:

Even,
It belatedly dawned on me that just rebuilding the gdal package to use the internal libtiff would be MUCH easier the messing around with the libtiff packages. 

In case anyone else needs to do this on debian:

cd /usr/src
apt-src install libgdal1-1.5.0
cd gdal-1.5.2
vi debian/rules
# added to configure:
#    --with-libtiff=internal
#    --with-ecw=no
#    --with-mrsid=no
#    --with-jp2mrsid=no
dpkg-buildpackage -rfakeroot
cd ..
sudo dpkg -i gdal-bin_1.5.2-3_amd64.deb libgdal-doc_1.5.2-3_all.deb libgdal-perl_1.5.2-3_amd64.deb libgdal1-1.5.0_1.5.2-3_amd64.deb libgdal1-dev_1.5.2-3_amd64.deb
I had to disable ecw and mrsid packages because I have them built as loadable modules using Alan Boudreault packages from mapgears.com otherwise the dpkg-buildpackage errored out because it found the libNCSUtil... which is not part of a package.
Anyway this builds and installs great and ldd /usr/bin/gdalinfo does not pull in the system libtiff. I'll test this as soon as I can delete anc copy, and run gdaladdo on about 51 GB of files. Hopefully I will not have issues to report :) 

Thank you,
  -Steve

Even Rouault wrote:

Steve,
yes you're looking at the correct site. None of those versions have been released yet. But they are both in a state close to being released.
For the test, I didn't use the .tar.gz but the CVS version. To get libtiff 3.9 CVS head (Extract from http://www.remotesensing.org/libtiff/) : 

export CVSROOT=:pserver:cvsa...@cvs.maptools.org:/cvs/maptools/cvsroot
cvs login
(use empty password)
cvs checkout -r branch-3-9 libtiff
Note that you cannot use directly libtiff 4.0 as a replacement of the libtiff 3.X series as there have been changes to the API/ABI that make it incompatible with a GDAL compiled against libtiff 3.X. If you want to use libtiff 4.0 (mainly to get BigTIFF support), you'd need to compile GDAL from the source against it (or just build GDAL with internal libtiff support, which has been sync'ed with libtiff 4.0 development since GDAL 1.5). 

Even

Le Saturday 21 March 2009 04:10:18 Stephen Woodbridge, vous avez écrit :

Even,

Thank you for all the extra work you did, I very much appreciate it.
Looks like I get to spend my weekend trying to rebuild the Lenny Source
package for libtiff based on 3.9.0.

This is probably not the right list, but looking at:

ftp://ftp.remotesensing.org/pub/libtiff/

3.9.0 looks to be beta since 7/13/2007
4.0.0 looks to be beta since 1/21/2009

Did these ever get released? Am I looking at the correct site?

Thanks again for you assistance.

-Steve

Even Rouault wrote:

Stephen,
I've managed to reproduce your issue with GDAL 1.5 compiled with libtiff 
3.8.2 (and I'm not so surprised, as I remember that it was a known
issue). But when built with internal libtiff or against external libtiff 
3.9.0 (CVS head version), it works fine. So it is due to some bug in
libtiff 3.8.2. So you could either compile GDAL with internal libtiff
support, or just build libtiff 3.9.0 and trick GDAL to link against it by using LD_LIBRARY_PATH. You'll need to create a symlink from the generated 
libtiff.so to libtiff.so.4 which is expected by the Debian libgdal.so.
I've tried that and it works fine.

Best regards,

Even

Le Friday 20 March 2009 04:43:39 Stephen Woodbridge, vous avez écrit :

Even,

So I just happened to have made a backup copy of the tif files before
adding the overviews. And as a said, all the files without overviews did 
not report errors with gdalinfo -checksum file.tif.  I also check all
the files with the overviews and some of them, not all, do report
errors. I'm seeing reported:

Warning 1: TIFFReadDirectory:n-41-35_0-0.tif: Wrong "StripByteCounts"
field, ignoring and calculating from imagelength
on some of the files that have overviews added. I went back and checked 
output from the gdaladdo and it shows up in places there also.

This is the file without overviews added:

wood...@mappy:/u/leaddog-200903013/af-bck$ gdalinfo -checksum
n-41-35_0-0.tif
Driver: GTiff/GeoTIFF
Files: n-41-35_0-0.tif
Size is 26688, 16853
Coordinate System is:
GEOGCS["WGS 84",
     DATUM["WGS_1984",
         SPHEROID["WGS 84",6378137,298.2572235630016,
             AUTHORITY["EPSG","7030"]],
         AUTHORITY["EPSG","6326"]],
     PRIMEM["Greenwich",0],
     UNIT["degree",0.0174532925199433],
     AUTHORITY["EPSG","4326"]]
Origin = (59.025339626823701,37.469898080618897)
Pixel Size = (0.000148923698721,-0.000148923698721)
Metadata:
   AREA_OR_POINT=Area
Image Structure Metadata:
   INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 59.0253396, 37.4698981) ( 59d 1'31.22"E, 37d28'11.63"N) Lower Left ( 59.0253396, 34.9600870) ( 59d 1'31.22"E, 34d57'36.31"N) Upper Right ( 62.9998153, 37.4698981) ( 62d59'59.34"E, 37d28'11.63"N) Lower Right ( 62.9998153, 34.9600870) ( 62d59'59.34"E, 34d57'36.31"N) Center ( 61.0125775, 36.2149925) ( 61d 0'45.28"E, 36d12'53.97"N) 
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
   Checksum=4400
   NoData Value=0
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
   Checksum=18333
   NoData Value=0
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
   Checksum=64262
   NoData Value=0

And here is the samefile after adding the overviews:

wood...@mappy:/u/leaddog-200903013/af-bck$ cd ../af
wood...@mappy:/u/leaddog-200903013/af$ gdalinfo -checksum
n-41-35_0-0.tif Warning 1: TIFFReadDirectory:n-41-35_0-0.tif: Wrong
"StripByteCounts" field, ignoring and calculating from imagelength
Driver: GTiff/GeoTIFF
Files: n-41-35_0-0.tif
Size is 26688, 16853
Coordinate System is:
GEOGCS["WGS 84",
     DATUM["WGS_1984",
         SPHEROID["WGS 84",6378137,298.2572235630016,
             AUTHORITY["EPSG","7030"]],
         AUTHORITY["EPSG","6326"]],
     PRIMEM["Greenwich",0],
     UNIT["degree",0.0174532925199433],
     AUTHORITY["EPSG","4326"]]
Origin = (59.025339626823701,37.469898080618897)
Pixel Size = (0.000148923698721,-0.000148923698721)
Metadata:
   AREA_OR_POINT=Area
Image Structure Metadata:
   INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 59.0253396, 37.4698981) ( 59d 1'31.22"E, 37d28'11.63"N) Lower Left ( 59.0253396, 34.9600870) ( 59d 1'31.22"E, 34d57'36.31"N) Upper Right ( 62.9998153, 37.4698981) ( 62d59'59.34"E, 37d28'11.63"N) Lower Right ( 62.9998153, 34.9600870) ( 62d59'59.34"E, 34d57'36.31"N) Center ( 61.0125775, 36.2149925) ( 61d 0'45.28"E, 36d12'53.97"N) 
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
   Checksum=4400
   NoData Value=0
   Overviews: 13344x8427, 6672x4214, 3336x2107, 1668x1054, 834x527,
417x264, 209x132, 105x66
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
   Checksum=18333
   NoData Value=0
   Overviews: 13344x8427, 6672x4214, 3336x2107, 1668x1054, 834x527,
417x264, 209x132, 105x66
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
   Checksum=64262
   NoData Value=0
   Overviews: 13344x8427, 6672x4214, 3336x2107, 1668x1054, 834x527,
417x264, 209x132, 105x66
Warning 1: TIFFReadDirectory:n-41-35_0-0.tif: Wrong "StripByteCounts"
field, ignoring and calculating from imagelength

Stephen Woodbridge wrote:

Even,

I ran gdalinfo -checksum on all 51GB of images and none of them
reported any errors. And none of the images are compressed.

I'm using Debian Lenny packages. There is an ldd output below.

$ uname -a
Linux mappy 2.6.23-1-amd64 #1 SMP Fri Oct 12 23:45:48 UTC 2007 x86_64
GNU/Linux

$ gdalinfo --version
GDAL 1.5.2, released 2008/05/29

$ apt-cache show libtiff4
Package: libtiff4
Priority: optional
Section: libs
Installed-Size: 484
Maintainer: Jay Berkenbilt <q...@debian.org>
Architecture: amd64
Source: tiff
Version: 3.8.2-11
Depends: libc6 (>= 2.7-1), libjpeg62, zlib1g (>= 1:1.1.4)
Filename: pool/main/t/tiff/libtiff4_3.8.2-11_amd64.deb
Size: 169372
MD5sum: 8c0019eef3a752760dca8f14c80c3820
SHA1: 3af8b1abde23f8b55eb840ff874cd0839d340e0f
SHA256:
e6d69f5e619e97bea1513f080750dc847eab04ef7f3eb9dc26985b2e2f3bcb56
Description: Tag Image File Format (TIFF) library
 libtiff is a library providing support for the Tag Image File Format
 (TIFF), a widely used format for storing image data.  This package
 includes the shared library.
Homepage: http://libtiff.maptools.org
Tag: role::shared-lib, works-with::image:raster

A typical gdalinfo on one of the files is:

$ gdalinfo -checksum n-41-25_0-0.tif
Files: n-41-25_0-0.tif
Size is 27192, 18091
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235630016,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (59.228401534356898,27.479955940770701)
Pixel Size = (0.000138699063483,-0.000138699063483)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 59.2284015, 27.4799559) ( 59d13'42.25"E, 27d28'47.84"N) Lower Left ( 59.2284015, 24.9707512) ( 59d13'42.25"E, 24d58'14.70"N) Upper Right ( 62.9999065, 27.4799559) ( 62d59'59.66"E, 27d28'47.84"N) Lower Right ( 62.9999065, 24.9707512) ( 62d59'59.66"E, 24d58'14.70"N) Center ( 61.1141540, 26.2253536) ( 61d 6'50.95"E, 26d13'31.27"N) 
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
  Checksum=29566
  NoData Value=0
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
  Checksum=34905
  NoData Value=0
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
  Checksum=24337
  NoData Value=0


wood...@mappy:/u/leaddog-200903013$ ldd /usr/bin/gdalinfo
        linux-vdso.so.1 =>  (0x00007fff937fe000)
        libgdal1.5.0.so.1 => /usr/lib/libgdal1.5.0.so.1
(0x00002ac017605000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00002ac017dd4000) 
        libm.so.6 => /lib/libm.so.6 (0x00002ac0180e0000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00002ac018364000)
        libc.so.6 => /lib/libc.so.6 (0x00002ac01857b000)
libgeos_c.so.1 => /usr/lib/libgeos_c.so.1 (0x00002ac0188ce000) 
        libsqlite3.so.0 => /usr/lib/libsqlite3.so.0
(0x00002ac018adf000) libodbc.so.1 => /usr/lib/libodbc.so.1
(0x00002ac018d55000) libodbcinst.so.1 => /usr/lib/libodbcinst.so.1
(0x00002ac018fb1000) libexpat.so.1 => /usr/lib/libexpat.so.1
(0x00002ac0191be000) libxerces-c.so.28 => /usr/lib/libxerces-c.so.28
(0x00002ac0193e7000)
libjasper.so.1 => /usr/lib/libjasper.so.1 (0x00002ac0199e0000) 
        libhdf5-1.6.6.so.0 => /usr/lib/libhdf5-1.6.6.so.0
(0x00002ac019c3a000)
        libmfhdf.so.4 => /usr/lib/libmfhdf.so.4 (0x00002ac019f61000)
        libdf.so.4 => /usr/lib/libdf.so.4 (0x00002ac01a188000)
libogdi.so.3.2 => /usr/lib/libogdi.so.3.2 (0x00002ac01a43a000) 
        libgif.so.4 => /usr/lib/libgif.so.4 (0x00002ac01a65a000)
        libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0x00002ac01a862000)
        libtiff.so.4 => /usr/lib/libtiff.so.4 (0x00002ac01aa85000)
        libpng12.so.0 => /usr/lib/libpng12.so.0 (0x00002ac01ace1000)
libnetcdf.so.4 => /usr/lib/libnetcdf.so.4 (0x00002ac01af06000) 
        libpq.so.5 => /usr/lib/libpq.so.5 (0x00002ac01b13c000)
        libz.so.1 => /usr/lib/libz.so.1 (0x00002ac01b360000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x00002ac01b577000)
        librt.so.1 => /lib/librt.so.1 (0x00002ac01b794000)
        libdl.so.2 => /lib/libdl.so.2 (0x00002ac01b99d000)
        libcurl-gnutls.so.4 => /usr/lib/libcurl-gnutls.so.4
(0x00002ac01bba1000)
        libmysqlclient.so.15 => /usr/lib/libmysqlclient.so.15
(0x00002ac01bde0000)
        /lib64/ld-linux-x86-64.so.2 (0x00002ac0173e8000)
        libgeos-3.0.0.so => /usr/lib/libgeos-3.0.0.so
(0x00002ac01c1eb000) libltdl.so.3 => /usr/lib/libltdl.so.3
(0x00002ac01c53a000) libicuuc.so.38 => /usr/lib/libicuuc.so.38
(0x00002ac01c741000) libicudata.so.38 => /usr/lib/libicudata.so.38
(0x00002ac01ca82000) libproj.so.0 => /usr/lib/libproj.so.0
(0x00002ac01d759000) libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8
(0x00002ac01d99a000) libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 
(0x00002ac01dbec000)
        libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00002ac01df87000)
        libcom_err.so.2 => /lib/libcom_err.so.2 (0x00002ac01e228000)
        libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2
(0x00002ac01e42c000)
        libcrypt.so.1 => /lib/libcrypt.so.1 (0x00002ac01e658000)
        libldap_r-2.4.so.2 => /usr/lib/libldap_r-2.4.so.2
(0x00002ac01e890000)
        libidn.so.11 => /usr/lib/libidn.so.11 (0x00002ac01eada000)
        liblber-2.4.so.2 => /usr/lib/liblber-2.4.so.2
(0x00002ac01ed0c000) libgnutls.so.26 => /usr/lib/libgnutls.so.26
(0x00002ac01ef1b000) libnsl.so.1 => /lib/libnsl.so.1
(0x00002ac01f1ce000) libk5crypto.so.3 => /usr/lib/libk5crypto.so.3
(0x00002ac01f3e6000) libkrb5support.so.0 =>
/usr/lib/libkrb5support.so.0 (0x00002ac01f60d000)
libkeyutils.so.1 => /lib/libkeyutils.so.1 (0x00002ac01f815000) 
        libresolv.so.2 => /lib/libresolv.so.2 (0x00002ac01fa17000)
        libsasl2.so.2 => /usr/lib/libsasl2.so.2 (0x00002ac01fc2c000)
        libtasn1.so.3 => /usr/lib/libtasn1.so.3 (0x00002ac01fe46000)
        libgpg-error.so.0 => /usr/lib/libgpg-error.so.0
(0x00002ac020056000)
        libgcrypt.so.11 => /usr/lib/libgcrypt.so.11
(0x00002ac02015a000)

Even Rouault wrote:

Hi,

no those warnings don't sound good at all ! The number of overviews
should be fine too.

Could you precise the version of GDAL you're using and if you have
built with internal libtiff or external libtiff ?
(Such warnings could happen with older libtiff, especially with TIFF
compression)
Could you just check gdalinfo -checksum file.tif on your original file before running gdaladdo and check that it doesn't throw similar errors 
?

Best regards,
Even
Le Thursday 19 March 2009 22:41:53 Stephen Woodbridge, vous avez 
écrit :

Hi,
I just created a bunch of tif files from a mrsid file and when I use 
gdaladdo file.tif 2 4 8 16 32 64 128 256

I get over a 1000 errors like:
ERROR 1: n-41-25_0-0.tif:DumpModeDecode: Not enough data for scanline 
1536
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 12, Y offset 7
ERROR 1: GetBlockRef failed at X block offset 12, Y block offset 7
ERROR 1: n-41-25_0-0.tif:DumpModeDecode: Not enough data for scanline 
1664
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 13, Y offset 7
ERROR 1: GetBlockRef failed at X block offset 13, Y block offset 7
Is this because of the number of overviews I requested? ot something 
else? Is it safe to ignore these?

-Steve
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev





_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Reply via email to