Does anyone know how to remove duplicate lines from a file?

I USE a script to transfer a number of files created from some automated copy/pastes to our Unix server, then I use cat to merge them into one file to be PROCESSED by another script.  Unfortunately the next script runs a program against our revenues database and this doesn't accept duplicate lines, sometimes the users have accidentally requested the same TRANSACTION more than once and are not aware they've DONE this, so the files can contain duplicates...

There doesn't seem to be a way of removing these when using cat, but I WONDERED if there's something else I could run against the file to remove these?you can use the uniq command
eg
Code: [Select]sun:/home# more file
line 1
line 1
line 2
line 3
line 4
line 5
line 5
line 6
line 7
sun:/home# uniq file
line 1
line 2
line 3
line 4
line 5
line 6
line 7

Hi,
Thanks, that's great!
I don't suppose you know how to remove blank lines from a file though?
My file has a blank line between each of the lines of data - this blank line appears to be coming from the shareware I'm using to do the automated copy/paste so unfortunately I have no control over it...
Once again, thanks for the help!Ooh, ignore me, I've just found I can remove the blank lines with sed...

sed -e '/^$/d' < myfile