Le jeudi 07 juillet 2011 23:46:11, Cole, Derek a écrit : > So that this may help anyone in the future - it seems like I may have > solved the problem for now! After looking through gdal_translate, it > appears that they are using the C API instead of C++ for the CreateCopy. I > switched my code to the C API instead of C++ just for consistency sake, > and sure enough, my CreateCopy was sped up tremendously!
Really ??? That doesn't make any sense. The C function GDALCreateCopy() is just a one line wrapper over the C++ API. I'm 200% sure there must be another difference in the values of the arguments between your C and C++ call. You should recheck... > > For the smaller test file I was playing with, the output file was not > viewable in my viewer until I made some adjustments, primarily, setting > the block size of the copy to be the same as the original. if you dont set > it explicitely you get row-wise blocks. Yes, row-wise blocks are the default. Tiling must be explicitely set up with teh BLOCKXSIZE and BLOCKYSIZE creation options. > > One other quirk - I was not able to use the GDALSetCacheMax() function , What do you mean by "not being able to use the GDALSetCacheMax()" ? Are you sure you specified the value in bytes as specified in the doc. I suspect you must have set a value in megabytes, because when the GDAL_CACHEMAX configuration option is read, there's a magic to automatically multiply small values (< 100000) by one million. That could explain why you manage to achieve the desired effect by using CPLSetConfigOption(). > and instead switched that to the CPLSetConfigOption() function, AND I had > to move it to be before I call GDALAllRegister() it appears, or I was > still getting cache thrashing warnings. > > If I get some time, or if someone else does, it might be nice to narrow > down what exactly in the C++ API was causing the problem. > > Derek > ________________________________________ > From: gdal-dev-boun...@lists.osgeo.org [gdal-dev-boun...@lists.osgeo.org] > on behalf of Cole, Derek [dc...@integrity-apps.com] Sent: Thursday, July > 07, 2011 3:20 PM > To: gdal-dev@lists.osgeo.org > Subject: RE: [gdal-dev] Creating modified copies of a file > > I actually just out of curiosity tried to open the file given by this > command: > > gdal_translate dan_data.ntf dan_data_translate.ntf > > And it turns out that dan_data_translate.ntf was actually a TIFF file that > I could view. I had to explicitly add the option -of NITF > > in which case it still did the translate very fast, but I did get warnings: > > $ gdal_translate dan_data.ntf dan_data_translate.ntf -of NITF > Input file size is 8689, 7679 > Warning 6: NITF only supports WGS84 geographic and UTM projections. > > 0...10...20...30...40...50...60...70...80...90...100 - done. > ERROR 6: Apparently no space reserved for IGEOLO info in NITF file. > NITFWriteIGEOGLO() fails. > ERROR 6: NITF only supports WGS84 geographic and UTM projections. > > > Which jives with what my code was giving me, and further, > dan_data_translate.ntf is not able to be opened using my viewer (which > uses GDAL to open the files I have been working with, of course, and does > open dan_data.ntf) because of a segmentation fault in the RasterIO > method. I am able to open the gdal_translate file, and my CreateCopy() > file in another ELT, such as ENVI. > > I will also make note that the original file is exactly 72.0 MB, and the > output file is 63.6MB, but both files say in gdalinfo they are not using > compression. This is the same for my CreateCopy() or gdal_translate. So at > least I have narrowed down that I am apparently getting the same > end-result as gdal_translate, albeit way slower. Finally, the output file > seems to have created row-blocks, instead of 1024x1024 blocks like the > original. > > I will continue working to replicate gdal_translate exactly, even though > now i am not optimistic about getting a good result. > > ________________________________________ > From: gdal-dev-boun...@lists.osgeo.org [gdal-dev-boun...@lists.osgeo.org] > on behalf of Cole, Derek [dc...@integrity-apps.com] Sent: Thursday, July > 07, 2011 2:48 PM > To: gdal-dev@lists.osgeo.org > Subject: RE: [gdal-dev] Creating modified copies of a file > > Yes, the source data is a NITF. Here is the first few lines of the > projection info in that source NITF: Coordinate System is: > PROJCS["unnamed", > GEOGCS["WGS 84", > DATUM["WGS_1984", > SPHEROID["WGS 84",6378137,298.257223563, > AUTHORITY["EPSG","7030"]], > TOWGS84[0,0,0,0,0,0,0], > AUTHORITY["EPSG","6326"]], > PRIMEM["Greenwich",0, > AUTHORITY["EPSG","8901"]], > UNIT["degree",0.0174532925199433, > AUTHORITY["EPSG","9108"]], > AUTHORITY["EPSG","4326"]], > PROJECTION["Transverse_Mercator"], > PARAMETER["latitude_of_origin",0], > PARAMETER["central_meridian",-75], > PARAMETER["scale_factor",0.996], > PARAMETER["false_easting",500000], > PARAMETER["false_northing",0]] > > > I suppose I can go back through my code and change it to be exactly like > gdal_translate, which seems to use the C bindings instead of C++. I did > end up changing my progress reporting function to the same one that is in > gdal_translate as well, which had no change. I will try the debugging > route before too much trouble, to see what I can find out. > > Thanks for the help so far. I will report back what I find. > > > > ________________________________________ > From: Even Rouault [even.roua...@mines-paris.org] > Sent: Thursday, July 07, 2011 2:13 PM > To: gdal-dev@lists.osgeo.org > Cc: Cole, Derek > Subject: Re: [gdal-dev] Creating modified copies of a file > > Le jeudi 07 juillet 2011 19:52:34, Cole, Derek a écrit : > > Well, I am about to give up on this I think and go a different route for > > creating a NITF from this data. > > > > I tried a small test file, and it takes about 45s to CopyCrate() an 80mb > > NITF. > > Yes, that's very poor performance ! > > > here is the output I got after turning on CPL_DEBUG > > > > GDAL: GDALOpen(/home/dcole/eraser/Images/dan_data.ntf, this=0x1ad7ec70) > > succeeds as NITF. GDAL: QuietDelete(test1.ntf) invoking Delete() > > GDAL: GDALOpen(test1.ntf, this=0x1ad81180) succeeds as NITF. > > GDAL: GDALDefaultOverviews::OverviewScan() > > GDAL: GDALClose(test1.ntf, this=0x1ad81180) > > Warning 6: NITF only supports WGS84 geographic and UTM projections. > > > > 0GDAL: Potential thrashing on band 1 of > > /home/dcole/eraser/Images/dan_data.ntf. > > ...10...20...30...40...50...60...70...80...90...100 - done. > > ERROR 6: Apparently no space reserved for IGEOLO info in NITF file. > > NITFWriteIGEOGLO() fails. > > ERROR 6: NITF only supports WGS84 geographic and UTM projections. > > > > GDAL: GDALDefaultOverviews::OverviewScan() > > ERROR 6: NITF only supports WGS84 geographic and UTM projections. > > This error seems to indicate that your source dataset has an incompatible > projection with what NITF allows (UTM/WGS84 and LongLat/WGS84). But this is > weird because your source dataset seems to be a NITF itself... What does > gdalinfo returns as projection for /home/dcole/eraser/Images/dan_data.ntf ? > > But I don't believe it relates with performance. And I suppose you get the > same warnings when using gdal_translate. > > > So to try to remedy this I changed the cachemax (programmatically) > > anywhere from 40MB to 3GB trying to see if any setting had any effect. > > It was about the same no matter what. > > For a 80MB file, playing with cachemax won't give any significant speed-up. > > > I tried doing a gdal_translate on this file, which only took about a > > second. > > That's much more reasonable. Well, then that shows it's not an issue in > GDAL per se, but how you use it. Now, you "just" have to discover what > things you do which are different from gdal_translate. > > > Looking through the source for that, it seems like I am doing > > approximately the same thing in relation to the CreateCopy(), so I am not > > sure what the problem is. However, my viewer which uses GDAL to read the > > original NITF file I am reading in will NOT read the translated NITF file > > that is generated. > > I've just reviewed the code snippet you pasted yesterday and I see that you > use a custom progress function. Isn't there a sleep() or some slow > operation in it ? > > For debugging performance issues, I start by just running the code under a > debugger and to break and resume execution regularly. In 90% of the cases, > you will see easily the bottleneck because the same function comes again > and again. Otherwise you might need more advanced tools. sysprof or > kcachegrind under Linux might be usefull to get timing profiles. > > > Any idea why the gdal_translate might be running faster than my > > CreateCopy, even though the parameters are just about identical? > > > > Derek > > ________________________________________ > > From: Even Rouault [even.roua...@mines-paris.org] > > Sent: Wednesday, July 06, 2011 5:32 PM > > To: gdal-dev@lists.osgeo.org > > Cc: Cole, Derek > > Subject: Re: [gdal-dev] Creating modified copies of a file > > > > Le mercredi 06 juillet 2011 23:18:21, Cole, Derek a écrit : > > > I have noticed that when attempting to do this, I am able to do a > > > gdalinfo on the source and destination files (even if the destination > > > file is not complete, it seems the header gets created, and gdalinfo > > > recognizes teh file). > > > > > > Is it possible that this is taking so long because my destination file > > > has no geocoords or projection information? It seems like createcopy is > > > not replicating all of the variables in the original and I am having to > > > manually set some things, such as Block size, ABPP, etc etc. > > > > This is expected. When doing CreateCopy() the target driver has no idea > > what the source driver is, so he will not recognize ABPP. As far as block > > sizes are concerned, it is up to the user to decide if he wants to keep > > or change the block dimension of the source dataset. > > > > I'm a bit surprised that CreateCopy() is *that* much slower than cp, but > > it is hard to know why without a hard work of profiling. It might be > > related to pixel or band interleaving, etc... There might cache > > trashing. You could perhaps run with CPL_DEBUG=ON and see if interesting > > info comes up. > > > > > And the > > > coordinates and projection information is blank in the new file. Is > > > there perhaps some calculations being done to correct that issue, and > > > thats why the copy is taking forever? > > > > No, not related at all. > > > > > I thought based on the documentation > > > CreateCopy() would clone the file's raster, as well as size, type, > > > projection, geotransform, etc, but all of this is not showing up in the > > > partially created file at least. > > > > You cannot reliably trust gdalinfo on a file still being written. > > Depending on the driver some information might just be written at the > > end of the process. A quick look at the CreateCopy() implementation of > > the NITF driver suggests that this is indeed the case since the > > geotransform is set after the imagery. > > _______________________________________________ gdal-dev mailing list > > gdal-dev@lists.osgeo.org > > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
