1.

Solve : Subtract images in batch process(leap year and regular year)?

Answer»

I have numerous images from a directory that needs to be subtracted to another image and I don't know how to do it automatically in batch process.

Here's what I need to do:

  • Create the output directory
  • Check automatically if there are new files added in the INPUT directory, if there are new files subtract immediately
  • Subtract one image from the directory to another image from another directory

LEAP YEARS:                                                            REGULAR YEARS:
*2000, 2004, 2008, 2012, 2016, 2020...*          *2001-2003, 2005-2007, 2009-2011, 2013-2015...*

January - 1-31                                                          January - 1-31                                                                                     
February - 32-60                                                      February - 32-59
March - 61-91                                                          March - 60-90
April - 92-121                                                          April - 91-120
May - 122-152                                                         May - 121-151
June - 153-182                                                        June - 152-181
July - 183-213                                                         July - 182-212
August - 214-244                                                    August - 213-243
September - 245-274                                              September - 244-273
October - 275-305                                                  October - 274-304
November - 306-335                                               November - 305-334
December - 336-366                                               December - 335-365

Here are the sample file name for the images(these will be the minuend):
:there will only be 12 subtrahend(sample_file_jan.tif, sample_file_feb.tif, sample_file_march.tif . . .)

C2000060.A1_ABC.ABCD.tif - sample_file_feb.tif ------------------> the file was subtracted to feb because 060 is a day included in February (leap year)
C2002152.A1_ABC.ABCD.tif - sample_file_june.tif------------------> the file was subtracted to june because 152 is a day included in June (regular year)
C2000032.A1_ABC.ABCD.tif - sample_file_feb.tif------------------> the file was subtracted to feb because 032 is a day included in February (leap year)
C2001060.A1_ABC.ABCD.tif - sample_file_march.tif------------------> the file was subtracted to march because 060 is a day included in March (regular year)

where:

C = data name
2000 - year (leap year or regular year)
001 - julian date (leap year or regular year)
A1_ABC.ABCD.tif - extension name

If the year specified in the file name is 2000, 2004, 2008... then the date would be based on leap year. If the year specified in the file name is 2001-2003, 2005-2007, 2009-2011... then the date would be based on regular year.

If the date from the file name is based on the date from the leap years, then subtract the image to its corresponding subtrahend/month to where it should be subtracted. Repeat this step if the date are from the regular years.


Here is the script in subtracting images:


gdal_calculate --outfile=D:\path\to\file\difference.tif --calc="((image1-image2)/(image1

+image2))" --image2=D:\path\to\subtrahend\sample_file_feb.tif image1=D:\path\to\minuend\C2001060.A1_ABC.ABCD.tif --extent=INTERSECT


You are really referring to an Ordinal Date not a Julian Date.
This will help you change your ordinal date to a Calendar date.  Once you have the calendar date information you should then be able to figure out how to run your program.
http://www.commandline.co.uk/cmdfuncs/dandt/#ordinaltodate
Code: [Select]echo off&setlocal
call :OrdinalToDate %1 %2 yy mm dd
echo/%yy%-%mm%-%dd%
goto :EOF

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:OrdinalToDate %year% %doy% yy mm dd
::
:: By:   Ritchie Lawrence, 2002-09-29. Version 1.0
::
:: Func: Returns a calendar date from an ISO 8601 Ordinal date.
::       For NT4/2K/XP.
::
:: Args: %1 year component to be converted, 4 digits (by val)
::       %2 day of year component to be converted, 001 to 366 (by val)
::       %3 var to receive year, 4 digits (by ref)
::       %4 var to receive month, 2 digits, 01 to 31 (by ref)
::       %5 var to receive day of month, 01 to 31 (by ref)
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
setlocal ENABLEEXTENSIONS
for /f "tokens=1-2" %%a in ('echo/%1 %2') do set /a yy=%%a,o=1%%b%%1000
set /a z=14-1,z/=12,y=yy+4800-z,m=1+12*z-3,j=153*m+2
set /a j=j/5+1+y*365+y/4-y/100+y/400-2432046,j+=o-1
set /a a=j+2432045,b=4*a+3,b/=146097,c=-b*146097,c/=4,c+=a
set /a d=4*c+3,d/=1461,e=-1461*d,e/=4,e+=c,m=5*e+2,m/=153,dd=153*m+2,dd/=5
set /a dd=-dd+e+1,mm=-m/10,mm*=12,mm+=m+3,yy=b*100+d-4800+m/10
(if %mm% LSS 10 set mm=0%mm%)&(if %dd% LSS 10 set dd=0%dd%)
endlocal&set %3=%yy%&set %4=%mm%&set %5=%dd%&goto :EOF
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::Going to assume you will want to convert the month number to the month name, so that you can use the month name with the GDAL_CALCULATE command.
I have added that code.
Code: [Select]echo off&setlocal enabledelayedexpansion
call :OrdinalToDate %1 %2 yy mm dd
echo/%yy%-%mm%-%dd%

:: Now convert the month number to month name abbreviation
set m=100
for %%m in (Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov  Dec) do (
set /A m+=1
set month[!m:~-2!]=%%m
)
set MONTHNAME=!month[%mm%]!
echo %monthName%

goto :EOF

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:OrdinalToDate %year% %doy% yy mm dd
::
:: By:   Ritchie Lawrence, 2002-09-29. Version 1.0
::
:: Func: Returns a calendar date from an ISO 8601 Ordinal date.
::       For NT4/2K/XP.
::
:: Args: %1 year component to be converted, 4 digits (by val)
::       %2 day of year component to be converted, 001 to 366 (by val)
::       %3 var to receive year, 4 digits (by ref)
::       %4 var to receive month, 2 digits, 01 to 31 (by ref)
::       %5 var to receive day of month, 01 to 31 (by ref)
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
setlocal ENABLEEXTENSIONS
for /f "tokens=1-2" %%a in ('echo/%1 %2') do set /a yy=%%a,o=1%%b%%1000
set /a z=14-1,z/=12,y=yy+4800-z,m=1+12*z-3,j=153*m+2
set /a j=j/5+1+y*365+y/4-y/100+y/400-2432046,j+=o-1
set /a a=j+2432045,b=4*a+3,b/=146097,c=-b*146097,c/=4,c+=a
set /a d=4*c+3,d/=1461,e=-1461*d,e/=4,e+=c,m=5*e+2,m/=153,dd=153*m+2,dd/=5
set /a dd=-dd+e+1,mm=-m/10,mm*=12,mm+=m+3,yy=b*100+d-4800+m/10
(if %mm% LSS 10 set mm=0%mm%)&(if %dd% LSS 10 set dd=0%dd%)
endlocal&set %3=%yy%&set %4=%mm%&set %5=%dd%&goto :EOF
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Quote from: Squashman on November 05, 2014, 09:21:49 AM
Going to assume you will want to convert the month number to the month name, so that you can use the month name with the GDAL_CALCULATE command.
I have added that code.
Code: [Select]echo off&setlocal enabledelayedexpansion
call :OrdinalToDate %1 %2 yy mm dd
echo/%yy%-%mm%-%dd%

:: Now convert the month number to month name abbreviation
set m=100
for %%m in (Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov  Dec) do (
set /A m+=1
set month[!m:~-2!]=%%m
)
set monthName=!month[%mm%]!
echo %monthName%

goto :EOF

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:OrdinalToDate %year% %doy% yy mm dd
::
:: By:   Ritchie Lawrence, 2002-09-29. Version 1.0
::
:: Func: Returns a calendar date from an ISO 8601 Ordinal date.
::       For NT4/2K/XP.
::
:: Args: %1 year component to be converted, 4 digits (by val)
::       %2 day of year component to be converted, 001 to 366 (by val)
::       %3 var to receive year, 4 digits (by ref)
::       %4 var to receive month, 2 digits, 01 to 31 (by ref)
::       %5 var to receive day of month, 01 to 31 (by ref)
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
setlocal ENABLEEXTENSIONS
for /f "tokens=1-2" %%a in ('echo/%1 %2') do set /a yy=%%a,o=1%%b%%1000
set /a z=14-1,z/=12,y=yy+4800-z,m=1+12*z-3,j=153*m+2
set /a j=j/5+1+y*365+y/4-y/100+y/400-2432046,j+=o-1
set /a a=j+2432045,b=4*a+3,b/=146097,c=-b*146097,c/=4,c+=a
set /a d=4*c+3,d/=1461,e=-1461*d,e/=4,e+=c,m=5*e+2,m/=153,dd=153*m+2,dd/=5
set /a dd=-dd+e+1,mm=-m/10,mm*=12,mm+=m+3,yy=b*100+d-4800+m/10
(if %mm% LSS 10 set mm=0%mm%)&(if %dd% LSS 10 set dd=0%dd%)
endlocal&set %3=%yy%&set %4=%mm%&set %5=%dd%&goto :EOF
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


How will I add the gdal_calculate command to the batch script you gave? Quote from: _unknown_ on November 06, 2014, 07:56:16 PM

How will I add the gdal_calculate command to the batch script you gave?
Well I assumed you wanted to use the Month Name Abbreviation as part of your sample_file_feb.tif file.  So use the month abbreviation variable that I created for you.In which part of your script should I add the gdal_calculate command? Quote from: _unknown_ on November 09, 2014, 11:01:18 PM
In which part of your script should I add the gdal_calculate command?
Well you are not even to that point yet.  You still need the code to extract the Ordinal Date from the file name. 

Look at the code that Aacini gave you on DosTips.com.
That will show you the FOR command that will process all your files and it will show you how to get the ordinal date from the file name and then it basically shows you where you would put your GDAL command. Quote from: Squashman on November 10, 2014, 06:55:34 AM
Well you are not even to that point yet.  You still need the code to extract the Ordinal Date from the file name. 

Look at the code that Aacini gave you on DosTips.com.
That will show you the FOR command that will process all your files and it will show you how to get the ordinal date from the file name and then it basically shows you where you would put your GDAL command.


Sorry for this. But is it like this?


Code: [Select]echo off&setlocal EnableDelayedExpansion
call :OrdinalToDate %1 %2 yy mm dd
echo/%yy%-%mm%-%dd%

:: Now convert the month number to month name abbreviation
set m=100
for %%m in (Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov  Dec) do (
set /A m+=1
set month[!m:~-2!]=%%m
)
set monthName=!month[%mm%]!
echo %monthName%

set "in_path=E:\Processed_Files\Merged\"
set "out_path=E:\Diff"
set "in_subtrahend=E:\sample_file_months\"
set "yearDay="
set "fileList="

md %out_path%

rem Process all *.tif files in input path
cd "%in_path%"
for %%a in (*.tif) do (
   set "fileName=%%a"
   rem If the YearDay in this file is the same of previous one
   if "!fileName:~1,7!" equ "!yearDay!" (
      rem Join this filename to previous list
      set "fileList=!fileList! !fileName!"
   ) else (
      rem Subtract the files in the list
      if defined fileList gdal_calculate --outfile=%out_path%\C!yearDay!.A1_ABC.ABCD.tif --calc="((image1-image2)/(image1+image2))" --

image2=%in_subtrahend% image1=%in_path% --extent=INTERSECT
   )
)
rem Subtract the files in the list
gdal_calculate --outfile=%out_path%\C!yearDay!.A1_ABC.ABCD.tif --calc="((image1-image2)/(image1+image2))" --image2=

%in_subtrahend% image1=%in_path% --extent=INTERSECT

goto :EOF

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:OrdinalToDate %year% %doy% yy mm dd
::
:: By:   Ritchie Lawrence, 2002-09-29. Version 1.0
::
:: Func: Returns a calendar date from an ISO 8601 Ordinal date.
::       For NT4/2K/XP.
::
:: Args: %1 year component to be converted, 4 digits (by val)
::       %2 day of year component to be converted, 001 to 366 (by val)
::       %3 var to receive year, 4 digits (by ref)
::       %4 var to receive month, 2 digits, 01 to 31 (by ref)
::       %5 var to receive day of month, 01 to 31 (by ref)
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
setlocal ENABLEEXTENSIONS
for /f "tokens=1-2" %%a in ('echo/%1 %2') do set /a yy=%%a,o=1%%b%%1000
set /a z=14-1,z/=12,y=yy+4800-z,m=1+12*z-3,j=153*m+2
set /a j=j/5+1+y*365+y/4-y/100+y/400-2432046,j+=o-1
set /a a=j+2432045,b=4*a+3,b/=146097,c=-b*146097,c/=4,c+=a
set /a d=4*c+3,d/=1461,e=-1461*d,e/=4,e+=c,m=5*e+2,m/=153,dd=153*m+2,dd/=5
set /a dd=-dd+e+1,mm=-m/10,mm*=12,mm+=m+3,yy=b*100+d-4800+m/10
(if %mm% LSS 10 set mm=0%mm%)&(if %dd% LSS 10 set dd=0%dd%)
endlocal&set %3=%yy%&set %4=%mm%&set %5=%dd%&goto :EOF
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Quote from: Squashman on November 10, 2014, 06:55:34 AM
That will show you the FOR command that will process all your files and it will show you how to get the ordinal date from the file name and then it basically shows you where you would put your GDAL command.

Am I doing the right thing? Please help! I'm just beginning to learn batch files and I don't know how to do it.

 


Quote from: Squashman on November 10, 2014, 06:55:34 AM
Well you are not even to that point yet.  You still need the code to extract the Ordinal Date from the file name. 

Look at the code that Aacini gave you on DosTips.com.
That will show you the FOR command that will process all your files and it will show you how to get the ordinal date from the file name and then it basically shows you where you would put your GDAL command.

I have tried my very best with this. Please bear with me.

Code: [Select]echo on
setlocal EnableDelayedExpansion

set "in_path=E:\Proc\Mer\"
set "out_path=E:\Proc\Abcde"
set "two_path=E:\Proc\Me\"
set "proc_path=E:\Proc\Proc_Mer_Fi"

::Don't modify the following variables
set "yearDay="
set "fileList="

md %out_path%
md %proc_path%

::Process all *.tif files in input path
cd /d "%in_path%"
for %%a in (*.tif) do (
   set "fileName=%%a"
   if %1 == 001-031 goto :condition1 ::reg/leap
   if %2 == 032-059 goto :condition2 ::reg
   if %3 == 032-060 goto :condition2 ::leap
   if %4 == 060-090 goto :condition3 ::reg
   if %5 == 061-091 goto :condition3 ::leap
   if %6 == 091-120 goto :condition4 ::reg
   if %7 == 092-121 goto :condition4 ::leap
   if %8 == 121-151 goto :condition5 ::reg
   if %9 == 122-152 goto :condition5 ::leap
   if %10 == 152-181 goto :condition6 ::reg
   if %11 == 153-182 goto :condition6 ::leap
   if %12 == 182-212 goto :condition7 ::reg
   if %13 == 183-213 goto :condition7 ::leap
   if %14 == 213-243 goto :condition8 ::reg
   if %15 == 214-244 goto :condition8 ::leap
   if %16 == 244-273 goto :condition9 ::reg
   if %17 == 245-274 goto :condition9 ::leap
   if %18 == 274-304 goto :condition10 ::reg
   if %19 == 275-305 goto :condition10 ::leap
   if %20 == 305-334 goto :condition11 ::reg
   if %21 == 306-335 goto :condition11 ::leap
   if %22 == 335-365 goto :condition12 ::reg
   if %23 == 336-366 goto :condition12 ::leap

   :condition1
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_jan.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition2
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_feb.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition3
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_mar.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition4
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_apr.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition5
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_may.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition6
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_june.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition7
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition8
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_aug.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition9
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_sep.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition10
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_oct.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition11
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_nov.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :condition12
   gdal_calculate --outfile=%out_path%\Abcde!yearDay!.Q_WER.Tera.tif !fileList! --calc="((one-two)/(one+two))" --two=%two_path%\two_abc_dec.tif --one=%in_path%\A!yearDay!.Q_WER.Tera.tif !fileList! --extent=INTERSECT
   goto end
   :: Move processed files to a different directory
   for %%a in (!fileList!) do move %%a "%proc_path%" >nul
   ) Quote from: _unknown_ on November 05, 2014, 02:34:30 AM
I have numerous images from a directory that needs to be subtracted to another image and I don't know how to do it automatically in batch process.

Here's what I need to do:
  • Create the output directory
  • Check automatically if there are new files added in the input directory, if there are new files subtract immediately

Files have a MODIFIED date which can be used to see if they are new.
Do your files get modified so that the most RECENT file was the last one that was created, or modified?

Quote from: foxidrive on December 01, 2014, 02:39:47 AM
Files have a modified date which can be used to see if they are new.
Do your files get modified so that the most recent file was the last one that was created, or modified?

Everyday there will be new files added to the input directory that's why I included that in my goals but now instead of that those processed files will just be moved to another directory. So that the input directory will not be confused of the newer files. I hope I'm explaining it well? Quote from: _unknown_ on December 01, 2014, 07:07:20 PM
Everyday there will be new files added to the input directory that's why I included that in my goals but now instead of that those processed files will just be moved to another directory. So that the input directory will not be confused of the newer files. I hope I'm explaining it well?

Well nothing you said replied to the question I posed.
The task may be much simpler than the way your code looks to be - using the file dates rather than the numbers in the filename.

I don't follow the task too well though, as subtracted and subtrahend are your TERMS for something
but they make no sense in terms of describing what the task is.

To me it would make more sense to say "processing" or "process the file with programB and copy it to folder c:\widget\"

Quote from: foxidrive on December 01, 2014, 09:01:53 PM
Well nothing you said replied to the question I posed.
The task may be much simpler than the way your code looks to be - using the file dates rather than the numbers in the filename.

I don't follow the task too well though, as subtracted and subtrahend are your terms for something
but they make no sense in terms of describing what the task is.

To me it would make more sense to say "processing" or "process the file with programB and copy it to folder c:\widget\"

Well sorry for me. I'm really not good in explaining. So you mean, I should replace the subtract with process?or processing? I'm using subtract and subtrahend because I thought that was the right term and because the batch will literally subtract one image to another.


Discussion

No Comment Found

Related InterviewSolutions