Hi folks, I'm trying to merge a bunch of geotifs together using FWTools (gdalwarp specifically) but I'd like to first do a "dir /b *.tif > list.txt" on the directory to come up with a file list of the source .tifs. If I paste all the filenames into the command manually the command I need looks like this spread out on a single LINE:

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic "C:\Temp\m_3208134_ne_17_1_20100928_20101105.tif" "C:\Temp\m_3208135_nw_17_1_20100708_20101105.tif" "C:\Temp\output_WGS84.tif"

EDIT: I think I can CD into the C:\Temp folder before running the command so I shouldn't need the path in addition to the filenames.

That works fine but I'll be pasting for 3 hours before I can run the command and I need to merge the entire group of tiles into several separate output .tifs ultimately, what I'd like to do is add the command to the beginning of the dir list and the output file to the end something like this:

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic
17RLM705845_201012_0x3000m_4B_1.tif
17RLM705860_201012_0x3000m_4B_1.tif
17RLM705875_201012_0x3000m_4B_1.tif
17RLM720845_201012_0x3000m_4B_1.tif
17RLM720860_201012_0x3000m_4B_1.tif
17RLM720875_201012_0x3000m_4B_1.tif
output_WGS84.tif

I'm almost positive I've seen this demonstrated somewhere but I can't find it for the life of me. If I try to run the command as pasted above I get gdalwarp's help text because the command wasn't correct followed by 6 .tifs opened in PhotoShop and a file not found error for the output .tif . Can someone give me a hand with the syntax I need, or is this even possible?

As always, much appreciated!

JimYour command line shows two filenames and an output filename.

Where are the input filenames coming from?  Are they all in a single format?  How do we match the sets of two files?

In the second command with 6 files - how do the set of input filenames come about?Thanks for your reply foxidrive.

The imagery comes off the national map server as tiled .jp2 (JPG2000) in 5000 x 5000 pixel tiles. To cover a specific geographical area you draw a bounding box on a map and complete your order which results in an email with links to a number of tiled downloads in .zip files. I believe the filenames correspond to the coordinates of the tile somehow, probably in UTM format. Here I've extracted just a few of the .jp2s and used another FWTools function (gdal_translate) to convert them to geotifs with the FOR command, this step is (sometimes) necessary to eliminate an embedded alpha channel from the .jp2s prior to merging them, otherwise gdalwarp can merge the .jp2s directly. gdalwarp uses the embedded geodata to assemble these 5000 x 5000 px tiles into a large geotif, the filename of which you simply specify at the end of the list of source .tifs to be merged.

gdalwarp also reprojects the imagery from NAD83 to WGS84 projection in the process of merging them. The list I posted is an abbreviated example from another project, I currently have about 100 of these I need to stitch together but to keep them manageable in PhotoShop each output .tif NEEDS to be roughly 25000 x 15000px max. It's imperative that they come in under 2 Gb for the next step of processing which is to use an MS SDK supplied tool (resample.exe) to convert them to photoreal scenery for Flight Simulator. The tool chokes if the .tifs exceed 2 Gb. That means ideally 15 adjoining tiles in a 5x3 pattern should be merged into each output.tif. I have an auto-generated google earth .kml from the map server that shows the relationship of each of the tiles in GE, this allows me to sort the appropriate GROUPS of 15 into separate folders for processing each output file.

In the past I've used a macro in Textpad to turn a dir list of these source .tifs into a long one-line command as shown in the first example. I'm working with colleagues on this and it would be much easier to have them open a command window and type "dir /b *.tif > filelist.txt" and then parse the command into the resulting file list somehow rather than explain the use of macros in Textpad, lol. This is something that we'll be repeating a lot in the coming months so just trying to get a workflow down basically at this point.

Thanks again,
JimThanks for the overview, though I need to know how the filenames are arranged from the source - one directory or two or more?

if you did a dir /b *.tif >file.txt then what do you do with the filenames?  How do you order them for the executable command?

We have no information about how the command line is assembled from the list of files, and that is needed to write a batch script.I'm still downloading tiles so I don't yet know all the filenames or which .tifs will end up in which folders at this point, but assume I've sorted 15 of the source .tifs into an empty folder manually, using hypothetical filenames I would normally just dir the folder to list the filenames then put all the sources in line like this:

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic 1.tif 2.tif 3.tif 4.tif 5.tif 6.tif 7.tif 8.tif 9.tif 10.tif 11.tif 12.tif 13.tif 14.tif 15.tif output_wgs84.tif

...or I could also do this:

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic 15.tif 14.tif 13.tif 12.tif 11.tif 10.tif 9.tif 8.tif 7.tif 6.tif 5.tif 4.tif 3.tif 2.tif 1.tif output_wgs84.tif

It doesn't matter which order the source .tifs are in or how many there are because gdalwarp locates them in the output .tif according to the geo LOCATIONS embedded in the source .tifs. Either of the above commands will give me the same output, a mosaic of the tiles with the filename "output_wgs84.tif".


So if I were to dir the folder to a txt file, all I'm trying to do is somehow use that filelist in the command without removing the line breaks, it would just be nice if I could simply paste "gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic" at the head of the file and the output filename at the end:

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic
1.tif
2.tif
3.tif
4.tif
5.tif
6.tif
7.tif
8.tif
9.tif
10.tif
11.tif
12.tif
13.tif
14.tif
15.tif
output_wgs84.tif

I was hoping something like these parentheses might work but of course they don't:

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic (
1.tif
2.tif
3.tif
4.tif
5.tif
6.tif
7.tif
8.tif
9.tif
10.tif
11.tif
12.tif
13.tif
14.tif
15.tif
)
output_wgs84.tif

My plan was to ultimately do a dir /s from the upper directory and with some minimal editing turn it into a batch file like this where a merged .tif would be output for each folder full of sources:

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic
folder1\1.tif
folder1\2.tif
folder1\3.tif
...
folder1\15.tif
output_wgs84_01.tif

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic
folder2\1.tif
folder2\2.tif
folder2\3.tif
...
folder2\15.tif
output_wgs84_02.tif

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic
folder2\1.tif
folder2\2.tif
folder2\3.tif
...
folder3\15.tif
output_wgs84_03.tif

etc.

The more I think about it though this is probably just not going to be possible as each line is interpreted as a command and I can't see any way around that. So what about using FOR?

for %f in (*.tif) do gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic "%f" "%~nf_WGS84.tif"

That works fine if you simply want to output a reprojected version of each source .tif with "_WGS84" appended to the filename, but it doesn't merge them all together into one. If I try:

for %f in (*.tif) do gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic "%f" "output_WGS84.tif"

then of course it simply wants to overwrite the output each time. I need something like this, lol:

for %f in (*.tif) do gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic "%f" "%f" "%f" "%f" "%f" "%f" "%f" "%f" "output_WGS84.tif"

But how could I possibly make something like that work?


Thanks again for the assistance.
JimSee what this does - it expects a series of folders below where the batch file is located and
the variable files is populated with the files in a folder, and then the gdalwarp command
is launched to create the output files in one place, with the folder name as part of the filenames.

It's untested.


Code: [Select]echo off
for /d /r %%a in (*) do (
   pushd "%%a"
      set files=
      for %%b in (*.tif) do call set files=%%files%% %%b
      call gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic %%files%% "c:\output folder\%%~nxa_wgs84.tif"
   popd
)
Thank you foxidrive, in the end it worked but I had to do some fiddling, your script was fine though.

FWTools has a "shell" that you must run these commands from, I'm not sure exactly what it does, it starts a batch called "setfw.bat" which is located in D:\MSFS\FWTools2.4.7. setfw.bat calls another batch in D:\MSFS\FWTools2.4.7\bin named "setfwenv.bat". This process obviously sets up an environment for these tools to run, I should have divulged that little tidbit previously:

setfw.bat:
Code: [Select]echo off
SET FWTOOLS_DIR=D:\MSFS\FWTOOL~1.7
call "%FWTOOLS_DIR%\bin\setfwenv.bat"
(for some REASON that all appears on one line if I edit setfw.bat in Notepad, pasting it here or opening the file in Textpad or Notepad++ the line breaks appear normally)

setfwenv.bat:
Code: [Select]echo off
REM ==========================================================================
REM This script is normally invoked from FWTools\setfw.bat

REM The FWTOOLS_OVERRIDE environment variable allows you to set a the
REM install directory in your environment, instead of having to edit
REM setfw.bat.  This is especially useful if you frequently upgrade to new
REM versions, and don't want to have to edit the file every time.

IF exist "%FWTOOLS_OVERRIDE%\setfw.bat" SET FWTOOLS_DIR=%FWTOOLS_OVERRIDE%

IF exist "%FWTOOLS_DIR%\setfw.bat" goto skip_err

echo FWTOOLS_DIR not set properly in setfw.bat, please fix and rerun.
goto ALL_DONE

:SKIP_ERR

PATH=%FWTOOLS_DIR%\bin;%FWTOOLS_DIR%\python;%PATH%
set PYTHONPATH=%FWTOOLS_DIR%\pymod
set PROJ_LIB=%FWTOOLS_DIR%\proj_lib
set GEOTIFF_CSV=%FWTOOLS_DIR%\data
set GDAL_DATA=%FWTOOLS_DIR%\data
set GDAL_DRIVER_PATH=%FWTOOLS_DIR%\gdal_plugins
REM set CPL_DEBUG=ON

:ALL_DONENo idea what I would ever do with %FWTOOLS_OVERRIDE% BTW.

First I started the FWTools shell and called your batch file from that, the batch was located in the root folder which contained the test folders and I also made a "c:\output folder" first. The result was a screenful of occurrences of gdalwarp's help text.

I assumed the problem was in calling the batch from the fwtools shell so I copied gdalwarp.exe into the root folder containing my test folders. The script I learned is looking for a copy of gdalwarp.exe in each of the test folders so I placed a copy in each. Then I learned about gdalwarp's dependencies for about the next hour, lol. Ultimately I wrote a batch to copy each of the dependencies into each of the 6 test folders, here's a list of everything I copied before I finally satisfied gdalwarp:

gdalwarp.exe
gdal_fw.dll
geos_fw.dll
geotiff_fw.dll
hdf5dll.dll
hdf_fw.dll
jpeg12_osgeo.dll
jpeg_osgeo.dll
libcurl.dll
libeay32.dll
libexpat.dll
libmysql.dll
libpq.dll
libtiff_fw.dll
lti_dsdk_dll.dll
mfhdf_fw.dll
msvcp71.dll
msvcr71.dll
NCScnet_fw.dll
NCSEcw_fw.dll
NCSUtil_fw.dll
netcdf.dll
ogdi_32b1.dll
proj.dll
sqlite3.dll
ssleay32.dll
szlibdll.dll
xerces-c_2_7.dll
zlib1.dll
zlib_osgeo.dll

With that done I ran your script and 10 minutes later ended up with 6 merged .tifs in the output folder bearing the names of the source folders. Perfect, it was a thing of beauty really! I will attempt to modify your script now so it references gdalwarp in the "D:\MSFS\FWTools2.4.7\bin" folder where all these dependencies are normally located, then I'll be able to skip the FWTools shell altogether and simply run the batch after populating the source folders with the appropriate source .tifs. I think I can take it from here and again thanks so much for your help!

Jim