1.

Solve : How to delete the duplicate record in text file using DOS??

Answer»

Friends,

Fine and hope the same.
Actually I need to remove the duplicate records from my input file (text file). Will it be possible to do in DOS. If so please help me to resolve it off...,

Input file :
vinoth
rajesh
EMPLOYEE
anand
rajesh
employeeGanesh

Output file should be in below format:
vinoth
rajesh
employee
anand
Ganesh


Thanks In Advance.

Thanks,
Vinoth Rajagopal

This is not too difficult - if you do not need to keep the order in the file

SORT the file

I used to have a batch filter called unique that removed duplicate items, but cannot find a download for it

Simply read the rows in the file, if the current row is the same as the last one, ignore it; if it differs, output it

Graham Code: [Select]echo off
setlocal enabledelayedexpansion

REM Create example file
echo Butcher > text1.txt
echo Baker >> text1.txt
echo Grocer >> text1.txt
echo Butcher >> text1.txt
echo Arrowsmith >> text1.txt
echo Butcher >> text1.txt
echo Butcher >> text1.txt
echo X RAY technician >> text1.txt
echo Grocer >> text1.txt
echo Grocer >> text1.txt
echo Arrowsmith >> text1.txt
echo Arrowsmith >> text1.txt
echo Zebra >> text1.txt
echo Butcher >> text1.txt
echo Baker >> text1.txt
echo Grocer >> text1.txt
echo Butcher >> text1.txt
echo Arrowsmith >> text1.txt
echo Butcher >> text1.txt
echo Butcher >> text1.txt
echo X Ray technician >> text1.txt
echo Grocer >> text1.txt
echo Grocer >> text1.txt
echo Arrowsmith >> text1.txt
echo Arrowsmith >> text1.txt
echo Zebra >> text1.txt

set inputfile=text1.txt
set outputfile=text2.txt

echo File to be processed
echo.
type %inputfile%
echo.

if exist sorted.txt del sorted.txt
sort %inputfile% /O sorted.txt

if exist %outputfile% del %outputfile%
set lastline=
for /f "delims==" %%L in (sorted.txt) do (
set thisline=%%L
if not "!thisline!"=="!lastline!" echo !thisline!>>%outputfile%
set lastline=%%L
)

del sorted.txt

echo Duplicates removed:
echo.
type %outputfile%



Nice one again Dias but it seems the OP doesn't want the output in alpha sequence?

Quote from: Vinath124

Output file should be in below format:
vinoth
rajesh
employee
anand
Ganesh

Quote from: Dusty on December 03, 2008, 02:19:04 PM
Nice one again Dias but it seems the OP doesn't want the output in alpha sequence?

Quote from: Vinath124
Output file should be in below format:
vinoth
rajesh
employee
anand
Ganesh



Without further confirmation from the OP that he wants the ORIGINAL (presumably) random order preserved, I would take that example as just showing the removal of duplicates.

However, I will think of a way to PRESERVE the original order...
Quote
However, I will think of a way to preserve the original order...

Yes, I'd probably prefix a number to each input record, alpha sort by name, remove records with duplicate names, resort by the number, remove the number.  No doubt you'll come up with something a lot more sophisticated.  Look forward to perusing your next gem.

Kind regards.

Code: [Select]echo off
setlocal enabledelayedexpansion
set /p firstline=<%inputfile%
echo %firstline%>%outputfile%
for /f "delims==" %%L in (%inputfile%) do find "%%L" %outputfile%>nul || echo %%L>>%outputfile%
Quote from: vinoth124 on December 02, 2008, 03:19:23 AM
Input file :
employee
employeeGanesh

Output file should be in below format:
employee
Ganesh

I see nothing logical in the requirement, and a "logical" computer will probably disappoint.

If "employeeGanesh" is transformed to "Ganesh" due to a previous "employee",
then "Smithson" will be transformed to "son" if there is already a "Smith".

This type of duplicate deletion will ensure that rajesh does not get two paychecks,
but Mrs Smithson will not get any housekeeping !!!

Is this really what you want ?

Regards
Alan Quote
employeeGanesh

I just saw this as a failure by the OP (who appears to have buggered off) to press Enter between typing employee and Ganesh, as a perusal of the "format" of the output file will confirm).


Discussion

No Comment Found

Related InterviewSolutions