[gdal-dev] gdalwarp overhead on linux but not on windows

Tue, 19 Nov 2019 13:25:18 -0800 

Hi guys,
I've been playing quite a lot with "wrapper_GDALWarpDestName" and a .net core wrapper (https://github.com/MaxRev-Dev/gdal.netcore/). Everything is running fine, I'm quite happy to see the same behavior on windows and linux. Sometimes linux is faster for reading and writing raster, this is great ! .NET Core is (now) a very good option for cross platform dev. Anyway... 


When calling wrapper_GDALWarpDestName,  I'm experiencing an overhead of 
350 to 500 ms on linux that I don't see at all on Windows.
To narrow down the issue, I tried many warp options (-multi; -wo 
NUM_THREADS=4; -wm 2000), output drivers (MEM, VRT, NWT_GRD), input file 
format (MEM, VRT, NWT_GRD), input and output projections. I also checked 
Gdal.GetCacheUsed(), Gdal.HasThreadSupport() and Gdal.GetCacheMax(), 
same values between linux and windows. Nothing helps.



Even with VRT format output, this overhead is present which proves this 
is not related to processing. I bet for some sort of initialization of 
the warp algorithm with file reading... because 400 ms is far from 
nothing. But I'm wondering why this is occurring for every single call 
on linux and not on windows... Any option I could try ? My next step is 
to profile the linux process but I'm not good at that so any help there 
would be great too.



    OS (version): Windows 10 (1903)
    Docker:
    Server: Docker Engine - Community
    Engine:
    Version: 19.03.2
    API version: 1.40 (minimum version 1.12)
    Go version: go1.12.8
    Git commit: 6a30dfc
    Built: Thu Aug 29 05:32:21 2019
    OS/Arch: linux/amd64
    Experimental: false


    Linux in docker file: debian 9 
(mcr.microsoft.com/dotnet/core/runtime:2.2-stretch-slim)

    Package version (core): [e.g. 3.0.1.25]
    Package version (runtime windows): [e.g. 3.0.1.25]
    Package version (runtime linux): [e.g. 3.0.1.2]

This is the kind of code I'm running:

namespace GdalCoreTest
{
 class Program
 {
  static void Main(string[] args)
  {
        GdalBase.ConfigureAll();
        Console.WriteLine("GDAL configured");
        Console.WriteLine(string.Join('\n',
                "GDAL Version: " + Gdal.VersionInfo("RELEASE_NAME"),
                "GDAL INFO: " + Gdal.VersionInfo("")));
        WarpFile();
        WarpFile();
        WarpFile();
  }

  public static void WarpFile()
  {
        Stopwatch sw = new Stopwatch();

        int sizeX = 1024, sizeY = 1024;
        var driver = Gdal.GetDriverByName("MEM");
        string path = $"/vsimem/test.mem";
        string vrtFilePath = null;

	using(Dataset ds = driver.Create(path, sizeX, sizeY, 1, 
DataType.GDT_UInt32, new string[0]))

        {
          try
          {
          //**************
          //create a grid
                ds.SetGeoTransform(new double[]{0, 30, 0, 0, 0, -30});

                var ttt = new SpatialReference(null);
                ttt.ImportFromProj4("+proj=longlat +datum=WGS84");
                ttt.ExportToWkt(out string latLongWkt, new string[0]);
                ds.SetProjection(latLongWkt);

                //write data in the buffer
                int[] buffer = new int[sizeX * sizeY];
                for (int x = 0; x < sizeX; x++)
                {
                  for (int y = 0; y < sizeY; y++)
                  {
                        buffer[y * sizeX + x] = x + y;
                  }
                }


		ds.GetRasterBand(1).WriteRaster(0, 0, sizeX, sizeY, buffer, sizeX, 
sizeY, 0, 0);



                //**************
                //Prepare for warping
                IntPtr[] ptr = {Dataset.getCPtr(ds).Handle};
                GCHandle gcHandle = GCHandle.Alloc(ptr, GCHandleType.Pinned);

		var dss = new 
SWIGTYPE_p_p_GDALDatasetShadow(gcHandle.AddrOfPinnedObject(), false, 
null);

                vrtFilePath = $"/vsimem/tile.vrt";

          //Warp
                sw.Start();

		using (Dataset dst = Gdal.wrapper_GDALWarpDestName(vrtFilePath, 1, 
dss, new GDALWarpAppOptions(new[]{"-of", "VRT" }), null,null))

                {
                  sw.Stop();
                }
          }
          finally
          {
                Gdal.Unlink(path);
                Gdal.Unlink(vrtFilePath);
          }
        }

    Console.Out.WriteLine($"Warp duration = {sw.ElapsedTicks} ticks 
({sw.ElapsedMilliseconds} ms).");

  }
 }
}


This is the output I'm getting on windows:

GDAL configured
GDAL Version: 3.0.1
GDAL INFO: GDAL 3.0.1, released 2019/06/28
Warp duration = 221267 ticks (22 ms).
Warp duration = 133395 ticks (13 ms).
Warp duration = 155402 ticks (15 ms).

This is the output I'm getting on docker/linux:

GDAL configured
GDAL Version: 3.0.1
GDAL INFO: GDAL 3.0.1, released 2019/06/28
Warp duration = 395911725 ticks (395 ms).
Warp duration = 368923383 ticks (368 ms).
Warp duration = 373055090 ticks (373 ms).

Thanks,
Alex.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev





















					Reply via email to