Solve : Copying files marked as "system files"?

1.	Solve : Copying files marked as "system files"?
Answer» I'm having issues copying some files. I've tried xcopy using the /h parameter but it returns "File not found" though I can see the file after changing directory to the folder and using the "dir /s" command. So the file is there, but it wont copy the file. I tried using a generic to copy everything in file and all that copies is "desktop.ini" nothing else. Quote from: DarkendHeart on September 12, 2010, 10:21:55 PM though I can see the file after changing directory to the folder and using the "dir /s" command. dir /s will view the contents of the folder and all it's subfolders. It doesn't see hidden files. Therefore the desktop.ini file you are seeing is probably a non-hidden file in a subdirectory of the folder in question. can you see the file with a dir /a?No I cant, but when I physically go into the folder it has no other folders in it. I can see the file I want to copy, and I can drag and drop it over like that but for what ever reason xcopy says it's not there. Quote from: BC_Programmer on September 12, 2010, 10:24:59 PM can you see the file with a dir /a? Quote from: DarkendHeart on September 12, 2010, 10:34:33 PM No I cant The file has the system attribute as well as the hidden attribute. I just tested it by creating a file with the hidden attribute. Xcopy /h found the file fine. However if I also added the system attribute xcopy reported "file not found".... despite the /H switch description specifying it would copy hidden & system files. Explorer was able to copy/paste the file and drag and drop it properly. That's pretty much where I'm at. I'd like for xcopy to do the copying instead of having to drag and drop since I'll be adapting it to copy file extensions instead of just one file. Is there anyway around it? Maybe a command that could remove it's system attribute so I can copy it? It seems though once I remove it from the source folder it loses is hidden/system attributes and I can find it again with xcopy. Could I do a mass move and just dump everything to a temp folder so xcopy can find it and then start copying? Assuming there is a command that isn't limited by the system attribute.I think I solved the problem. What I ended up doing was copying everything out of the parent folder of the folder that had the file(s) I needed in it to a temp location. After doing that I navigated into that folder and was able to copy the file out of that folder to the final location. Kind of BACK assed I know, but it seems like it's the only way that would work. Also why the files weren't copying out of the folder to begin with and why I was only getting "Desktop.ini" instead of everything else is because the system has them marked as "empty" files so I had to add the "/e" switch with the "/h" switch for xcopy to move them. Now the problem is deleting the files from the temp directory. It says they're deleted but they aren't.So it wont let me delete them as it's a "read-me only" attribute. Though using the attrib -r /s /d command doesn't remove it's read-me only attribute. I even tried... for /r %d in (.) do attrib -r %d /s /d ... and it appears that it works but never removes the read-me statues. Even right clicking the folder and going into properties and unchecking read-me only still does nothing. It applies and doesn't have any error, but never changes the attributes. Though I can click it and hit delete and that works, but the del or erase command will not delete it.Any ideas? I can traverse into every single folder inside the temp file and delete all the files out of those but the folders them selves will not delete. After doing some testing it seems for w.e reason I cant use the del command if the folder has a read-only attribute. Though I cant remove that attribute either. This is frustrating as I cant seem to delete ANY folder that has a read-only attribute set but they system wont let me change it. I am the admin on the machine and the command line is running as the admin, yet I still cant remove the read-only attribute. I can remove System and Hidden attributes but not read-only. It seems to just be some kind of issue with windows not really recognizing "read-only" attribute though that attribute prevents dos from deleting it. Sadly the work around that was mentioned on the Microsoft's support site doesn't work.Windows 7 is not fit for purpose and should never have even entered Beta. It employs Junctions and that are supposed to give a form of backward compatibility with older applications written for XP. Some of them are black holes. You can put files in them, but you cannot get them back, and you cannot read them or delete them. It is a fiasco. I have counted 44 Junctions in Windows 7, most of those I wanted are blackholes Run CMD.EXE and past this into the window. Code: [Select]CD \ echo %TIME% > "%USERPROFILE%\Application Data"\ALAN_LOST DEL "%USERPROFILE%\Application Data"\ALAN_LOST echo %TIME% > "%APPDATA%"\ALAN_KEPT echo %TIME% >> "%USERPROFILE%\Application Data"\ALAN_LOST TYPE "%USERPROFILE%\Application Data"\ALAN_LOST TYPE "%APPDATA%"\ALAN_KEPT TYPE "%APPDATA%"\ALAN_LOST DEL "%APPDATA%"\ALAN_KEPT DEL "%APPDATA%"\ALAN_LOST Nothing strange under XP, no errors - nothing to see her - move along. Under Windows 7 Ultimate I got errors and black-holes as per image attached. Alan [recovering disk space - old attachment deleted by admin] Quote from: ALAN_BR on September 19, 2010, 03:55:34 PM Windows 7 is not fit for purpose and should never have even entered Beta. There is nothing wrong with windows 7. Or Vista, for that matter. Quote Under Windows 7 Ultimate I got errors and black-holes as per image attached. Works fine here: Where is this "black hole"? the Application Data folder redirects to %appdata% which is usually %userprofile%\AppData\Roaming. As you can see the Type command worked perfectly fine, as did both redirects. The Access Denied error is for a very good reason, since write Access to the "Application Data" junction is only allowed by some programs (usually those with a compatibility mode set to XP or earlier). If that wasn't the case, how many "power" users do you think would encounter it, be ABLE to look inside the folder, and decide "well GOLLY GEE! I already have all those files in %appdata%... DELETE" only to find they just deleted %appdata%. Would they be pleased? censored no. They just deleted all their documents. There is no capability for "data loss" here, as you claim. In fact, the opposite is true.You misunderstand. I never said "data loss" but "Some of them are black holes. You can put files in them, but you cannot get them back" I mean that you cannot access the files by using the same path used for storing them. This is nothing but "data loss" for people who do not know where to look for them. This BlackHole problem I was unaware of until it hit me. There are many others who have yet to suffer this aggravation. Your "...\Application Data" is a Blackhole. You can put a file in, but you cannot get it back. It is a SciFi type of blackhole with a wormhole to a distant galaxy, or in this case to a destination that is fully accessible to all read and write operations via the path "...\Appdata\Roaming", BUT IT IS ONLY ACCESSIBLE for those who look in the correct place, and is totally lost for those who have downloaded a script that uses XP compliant paths. What I have learnt from the Internet is that people lost everything because they believed original documentation that indicated a Junction was like a short-cut which could be safely deleted without harming the contents of what it re-directed to. Microsoft did not correct their enormous mistake, and instead admitted the error BUT recommended that Junctions should be protected from accidental deletion by the use of CACLS. I have used CACLS to inspect the permissions and restrictions of a sample from the 44 Junctions scattered around Windows 7, and find significant differences between them. You have what could be a good point about protecting "Power Users", but I would counter with :- 1. Although a Junction may protect against reading or deleting a file, it does not protect against appending a file, and presumable may permit a file to be written even if it over-writes another file. Interesting example encountered at :- http://forum.piriform.com/index.php?showtopic=29618 With a reinstall of Vista that resulted in an extra folder called "Windows Old", a CCleaner user found that CCleaner was prepared to delete music from to delete "Windows Old" and he let it happen. He lost much of his real music that he treasured. He found CCleaner did no such harm with a normal delete (which Junctions tend to block) but his use of SECURE delete meant the files were over-written and lost. I assume that what he saw as being in "Windows Old" actually encountered a Junction that was frozen at some place in "Windows" The "nanny mode" of protecting against loss of data via Junctions may be fool-proof, but is not "Power user" proof ! ! 2. Any self proclaimed power user that meddles with things he does not understand needs to learn better, and even XP will punish him if he removes duplicates that are genuinely independent duplicates but XP still expects to find them. I have a total of 471 instances of Desktop.ini within 4 partitions. I wonder what Windows would look like if I removed 470 of them ! ! ! 3. I strongly suspect that power users are not fully protected. Not all Junctions are Blackholes, some give full bidirectional access, which means that DEL will delete a destination file and RD will delete a destination folder via a path that includes a Junction, so I suggest these are points of vulnerability to a meddlesome "Power User". I guess the variation is something to do with M.$ not being sure what to do with CACLS. e.g. V:\W7_OOPS>CACLS "C:\Documents and Settings\Alan\Application Data\" \| FIND ":" C:\Documents and Settings\Alan\Application Data Everyone:(DENY)(special access:) NT AUTHORITY\SYSTEM:(OI)(CI)(ID)F BUILTIN\Administrators:(OI)(CI)(ID)F Alan-Laptop-W7\Alan:(OI)(CI)(ID)F or V:\W7_OOPS>CACLS "C:\Users\Alan\AppData\Local\" \| FIND ":" C:\Users\Alan\AppData\Local NT AUTHORITY\SYSTEM:(OI)(CI)F BUILTIN\Administrators:(OI)(CI)F Alan-Laptop-W7\Alan:(OI)(CI)F or V:\W7_OOPS>CACLS "C:\Users\Alan\Local Settings\Application Data\" \| FIND ":" Access is denied. or V:\W7_OOPS>CACLS "C:\ProgramData\" \| FIND ":" C:\ProgramData NT AUTHORITY\SYSTEM:(OI)(CI)F BUILTIN\Administrators:(OI)(CI)F CREATOR OWNER:(OI)(CI)(IO)F BUILTIN\Users:(OI)(CI)R BUILTIN\Users:(CI)(special access:) or V:\W7_OOPS>CACLS "C:\ProgramData\Application Data\" \| FIND ":" C:\ProgramData\Application Data Everyone:(DENY)(special access:) Everyone:R NT AUTHORITY\SYSTEM:F BUILTIN\Administrators:F When Comodo Security is upgraded it may need un-installation of the old before the new, and the new may be blocked if residues remain due to sundry glitches, so various users have posted scripts that seek and destroy the residues. Those scripts work on XP for some people, but failed for me due to permissions issues. I added "error checking" to confirm successful removal or report specific problems that needed manual intervention. Perfection at last on XP. Before final release I checked that all was well on W7 and found disaster. There were "residues" which I deliberately planted to ensure correct detection and removal or reporting. They were not detected. Even using %USERPROFILE%\etc... or %APPDATA%\etc... they could not be detected if the \etc... went through another Blackhole type junction - IF EXIST could not see them, CACLS had no access, DEL could not touch them. This aggravation has caused me to build up a full head of steam. Sorry about that. The original start of this thread was that the O/P knew certain files existed in a certain place but he could not copy them. This resonated very strongly with the problem I have suffered, and after checking his profile and seeing that the OP use W7 I thought that BlackHole junctions might be the cause depending upon what paths he was using, so I posted what I thought might be useful to him. I have much more to say and discuss, and questions to ask. Thank you for your information on compatibility mode, I have never needed it before and will investigate how this might affect CMD.EXE scripts. I do not wish to be barred by a moderator for hijacking the OP's topic, and suggest we take this outside to settle it ! ! I propose to start a topic in the Windows 7 forum when I get a few other urgent things done, and will post here a quick invitation to join me there when I have organized my evidence against Junctions. Regards Alan Alan thank you very much for posting what you did, I would definitely not classify this as high-jacking as this was very insightful. I never knew such things could and do happen. That would probably explain why I cant delete them using that command. Do you know of any kind of workaround or some way I may be able to del/copy them? I can successfully copy them but I have to copy the parent folder the files are in to a secondary file which I can use the attrib command to remove their system/hidden attributes. Though sadly, windows does not let me delete these new folders using the del command. I know these copied folders are standalone from the others as the original maintains it's attributes and files inside of it. Where as the copied folders I can change their attributes(excluding read-only), and I can delete the files from those folders but not the folders them selves.XCOPY /H That is like Copy but will deal with Hidden and System files should be included. You will see lots of options by XCOPY /? That may be all you need, depending upon what paths are involved, but Junctions can give all sorts of problems. My favorite resource is http://wapedia.mobi/en/Environment_variable?t=5.#5. The top item shows the environment variable %APPDATA% and the real destination for both XP and Vista/W7 It is far better to access via %APPDATA% than the XP equivalent of C:\Documents and ....... The XP equivalent will in W7 be redirected by Junction(s) but with restricted permissions. This freeware will scan the system and find and report all junctions. http://rekenwonder.com/linkmagic.htm N.B. the scan took LONGER on my machine than a coffee break, so I took a lunch break and it finished before I ended my meal. It reports the location of the Junction and where the REAL destination is. e.g. Code: [Select]================================================== Junction point : C:\Users\Alan\Application Data\ Destination : C:\Users\Alan\AppData\Roaming ================================================== I can create ALAN_LOST at %USERPROFILE%\Application Data\ which becomes C:\Users\Alan\Application Data\ and it is actually sent to C:\Users\Alan\AppData\Roaming\ I am severely restricted in what I can do via the path C:\Users\Alan\Application Data\ but have full control via the path C:\Users\Alan\AppData\Roaming\ You will get information on Junctions at http://en.wikipedia.org/wiki/NTFS_junction_point http://support.microsoft.com/?kbid=205524 Microsoft WARN that Junctions must be protected from accidental deletion, and they recommend the use of CACLS to MODIFY the Access Control Lists. Previously I have shown C:\Documents and Settings\Alan\Application Data Everyone:(DENY)(special access:) NT AUTHORITY\SYSTEM:(OI)(CI)(ID)F BUILTIN\Administrators:(OI)(CI)(ID)F Alan-Laptop-W7\Alan:(OI)(CI)(ID)F If you launch CMD.EXE the command CACLS /? will tell you what you can do with it. You will see that (OI)(CI)(ID) deal with inheritance - I do not want to go there ! ! It may fix things for you, but it could also destroy everything. Only use as last resort. Regards Alan Quote from: ALAN_BR on September 21, 2010, 02:33:52 AM This freeware will scan the system and find and report all junctions. http://rekenwonder.com/linkmagic.htm N.B. the scan took longer on my machine than a coffee break, so I took a lunch break and it finished before I ended my meal. It reports the location of the Junction and where the REAL destination is. e.g. Code: [Select]================================================== Junction point : C:\Users\Alan\Application Data\ Destination : C:\Users\Alan\AppData\Roaming ================================================== There is another piece of software that does this. It's called DIR. dir /s /al will list all junction points on a drive and their targets. Quote I can create ALAN_LOST at %USERPROFILE%\Application Data\ which becomes Question is, Why are you using %USERPROFILE%\Application Data\ instead of %APPDATA%? %APPDATA% on windows XP resolves to Application Data, and %APPDATA% in Vista/7 resolves to %USERPROFILE%\AppData\Roaming. Also, it will account for those times when your profile is on a network drive. IT doesn't work for Windows 98, since windows 98 doesn't by default have an %APPDATA% variable. It does have an application Data folder in C:\Windows\Application Data\, which means that a more "universal" method for MS to have done would have been to create a junction point in the windows folder as well that redirected to the appropriate location to account for those win98 programs that hard coded %WINDIR%\Application Data as the destination. They of course didn't, because Applications can get the Application Data folder in Windows 98 and later using well documented functions, not environment variables. the SHGetFolderPath Function is one such entry. So you may be looking at that link and going HAHA! but it's not supported on windows 98! And you would be quite correct! It's not. Most applications using the function will install a redistributable "shfolder.dll" file, which works in windows 98. A more obtuse method of doing the same thing would be to use the SHGetSpecialFolderLocation Function, which gives back a PIDL (Pointer to an ID List) and then pass that PIDL to the SHPathFromIDList() Function, which doesn't even appear to be documented at all. This will give back the fully qualified path of the Special Folder on Windows 98 and Later. Windows Vista, along with introducing the use of the junction points that you are so keen on using for some reason,(the junction points are for older, badly written programs to use, not for users to start throwing files into) changed the mechanic for retrieving special folders. These older methods still work, but they simply delegate their work to the new method, which is the IKnownFolder interface. In the purest sense, it is more difficult to work with- requiring the construction and use of an interface. Thankfully there are further wrappers around this, using SHGetKnownFolderPath And SHSetKnownFolderPath to acquire and Set the known folder location respectively. The Setting operation is particularly relevant. While you are quite keen on assuming that there will always be an Application Data Folder within %USERPROFILE%, what if said %USERPROFILE% redirects to a network server drive? Surely you cannot expect another machine to automagically have this path present? So rather then using a carefully constructed path to cause errors, use %APPDATA%, which will redirect properly to the actual location of the users Application Data Folder, rather then %USERPROFILE%\Application Data, which will only work well on windows XP, and even then, only on unmodified configurations. I'm not sure wether XCOPY skips symbolic links/junctions while recursing folders. Even if it doesn't, though, the fact that it won't be able to read from the junction point means it will just skip that folder anyway. If the errors are a bother, one could always try ROBOCOPY, which is included in Vista and 7, and will skip junction points, reparse points and hard-linked files and folders by default. IF anything, be happy that MS decided to use a reparse point (Junction) rather then actually hard-linking the folder to the "real" location. If they had done that, your precious little batch file would run fine, but deleting anything within %USERPROFILE%\Application Data would delete the same stuff within the AppData\Roaming folder, and deleting the folder itself would delete the Roaming Folder entirely. There would also be no indication there was any redirection at all- no shortcut symbol, just a standard folder that looks and acts just like one. This would be because a hardlink is just the file name pointing at the data- all filenames are hardlinks, but you generally only have a 1 to 1 relationship between file names and file data. Creating a new hard link to the same data is just giving a new name to that data; in the case of a folder, there won't even be a path redirection- that is, all the folders will still "appear" to be in Application Data, because, technically, they are. But they would also be in the Roaming Folder, since the Roaming and Application Data would have been configured to use the exact same directory. I don't doubt that they considered this option and decided that they would rather field calls from people whose data was merely redirected and not deleted. "I found this Application Data Folder, and it had the same stuff as in my Roaming\AppData Folder, so I deleted it, and now none of my programs work!". Not something a new user would do, but people who consider themselves "Power Users" will often go futzing about in various system managed folders, so that type of thing would have been assured, not just a possibility. The only other choice they could have done would have been to simply not have a junction point at all. But this would hardly work, either. If a Program that worked fine on Windows XP despite it hard-coding something like "%USERPROFILE%\Application Data" for accessing it's data no longer worked on Windows Vista or 7, the user isn't going to think "well, golly, this program is bad" they are going to blame Windows. MS does the best they can to make sure these badly programmed pieces of crap work fine, because there are a lot of people whose lives and business are teetering on the edge of some badly programmed piece of software that uses hard-coded paths, and if that program doesn't work in Vista/7, they simply won't upgrade. So, they need to think of an option that will get these programs working with minimal side effects. They had a number of options: 1. The Hard Linked folder method- Already described above, as well as it's inherent weaknesses. Also, another rather major downside would be that a hard-linked folder is nearly impossible to programmatically determine- unlike a junction point, you can't just query it's attributes and go "GOLLY! this is a hard-linked folder!" So, you end up with backups containing two copies of the exact same data, one from Application Data, and one from AppData\Roaming. This all also applies to the other junctions that MS implemented (had they been made into hardlinks instead, I mean), such as My Documents pointing to the Documents folder, as well as a number of others. And god forbid if you have yet another hard link within that folder! otherwise you would have backup programs constantly recursing into them until the path is too long. (See %USERPROFILE%\Local Settings\Application Data, notice there is a Application Data junction. click it. the junction points to the Local Settings Folder. Notice the path says you are in %USERPROFILE%\Local Settings\Application Data\Application Data. you can continue to dig into the Application Data Folder over and over and over. As the path grows longer and longer and longer. It starts to take a while for the view to refresh, though, after about 30 or so. 2. implement the redirection as part of the file system redirector. They already do this for Windows x64, whereby accesses to C:\Windows\System32 by 32-bit programs are silently redirected to C:\windows\syswow64, so why not do it here? For one reason- it's stupid. The file system redirector was being used in windows x64 to redirect access to the C:\Windows\System32 folder, something that had been ubiquitous for years, to the WOW folder that was used for the 32-bit version of system32. (Why not create a System64? I don't know, sure they had reasons, this redirection thing is a clue to their reasoning, in fact). In this instance, it would mean actually redirecting, within the file system driver, folder paths like "C:\Documents And Settings\Tom\Application Data" to "C:\Users\Tom\AppData\Roaming" It seems simple, and really, it is. But consider that every single time a file or folder is accessed, the driver will have to go "alright, I'll just check wether I need to redirect this..." When you have 2 or 3 users, that's "only" around 30 or so permutations to check. But It's not scalable- Windows is also deployed in environments with hundreds or thousands of users. having a server check a given file path against thousands or even millions of different possible paths that need to be redirected while everybody twiddles there thumbs is not apt to get people liking the new OS. 3. Create a Junction Point from the old location to the new location. This is what they chose to do. It's the best solution- it allows old XP oriented applications to work fine with Vista/7, and it's transparent. The only people that see it are those you go futzing about using non-default explorer settings (such as, for example, show hidden files/folders). The average user doesn't make a hobby out of exploring deep within their profile folder. Anybody who does should know what they are doing. If you are going to echo data into a file in the junction, you either learn that it's a junction and where the data "really" is or you continue to throw data into it and scratch your head and end up blaming MS. In the former case you just say "ahh, ok, I guess I'll stop being an idiot and redirecting stuff into junction points designed solely for XP compatibility.Nice text wall BC =) So, assuming I understood that, it would seem that the folder I'm trying to access isn't the correct folder and just a junction(if I'm using that correctly) to another folder. So the folder I need to be accessing is some where else. Also, I was never trying to delete anything from "App Data"/"Roaming" I had made a copy of a parent folder holding other folders in it to a completely different location on the hard drive and I wouldn't let me us the del command to delete these newly created folders. From what I've read I'm guessing windows still thinks it's the junction to \Roaming since the file names never change and that's why it wont let me delete it using that command. Also, thank you both for posting here. This has been really insightful, I would have never known half of this stuff! Maybe I'm not understanding as much as I thought I did. I'm trying to now find the corresponding folder in the \Roaming folder. The folder I'm trying to copy things out of is the "C:\Users\owner\AppData\Local\Microsoft\Windows\Temporary Internet Files\" folder since it didn't let me do that I could copy the paren't folder "Temporary Internet Files" into a new directory. This then gives me full rain over the files in there, but then it wont let me delete the new folders that were created in there like "Content.IE5". Is this do the junctions? Or is this a different issue.

Answer»

I'm having issues copying some files. I've tried xcopy using the /h parameter but it returns "File not found" though I can see the file after changing directory to the folder and using the "dir /s" command. So the file is there, but it wont copy the file. I tried using a generic to copy everything in file and all that copies is "desktop.ini" nothing else. Quote from: DarkendHeart on September 12, 2010, 10:21:55 PM

though I can see the file after changing directory to the folder and using the "dir /s" command.

dir /s will view the contents of the folder and all it's subfolders. It doesn't see hidden files.

Therefore the desktop.ini file you are seeing is probably a non-hidden file in a subdirectory of the folder in question.

can you see the file with a dir /a?No I cant, but when I physically go into the folder it has no other folders in it. I can see the file I want to copy, and I can drag and drop it over like that but for what ever reason xcopy says it's not there. Quote from: BC_Programmer on September 12, 2010, 10:24:59 PM

can you see the file with a dir /a?

Quote from: DarkendHeart on September 12, 2010, 10:34:33 PM

No I cant

The file has the system attribute as well as the hidden attribute.

I just tested it by creating a file with the hidden attribute. Xcopy /h found the file fine. However if I also added the system attribute xcopy reported "file not found".... despite the /H switch description specifying it would copy hidden & system files.

Explorer was able to copy/paste the file and drag and drop it properly.

That's pretty much where I'm at. I'd like for xcopy to do the copying instead of having to drag and drop since I'll be adapting it to copy file extensions instead of just one file. Is there anyway around it? Maybe a command that could remove it's system attribute so I can copy it? It seems though once I remove it from the source folder it loses is hidden/system attributes and I can find it again with xcopy. Could I do a mass move and just dump everything to a temp folder so xcopy can find it and then start copying? Assuming there is a command that isn't limited by the system attribute.I think I solved the problem. What I ended up doing was copying everything out of the parent folder of the folder that had the file(s) I needed in it to a temp location. After doing that I navigated into that folder and was able to copy the file out of that folder to the final location. Kind of BACK assed I know, but it seems like it's the only way that would work. Also why the files weren't copying out of the folder to begin with and why I was only getting "Desktop.ini" instead of everything else is because the system has them marked as "empty" files so I had to add the "/e" switch with the "/h" switch for xcopy to move them.

Now the problem is deleting the files from the temp directory.
It says they're deleted but they aren't.So it wont let me delete them as it's a "read-me only" attribute. Though using the attrib -r /s /d command doesn't remove it's read-me only attribute. I even tried...
for /r %d in (.) do attrib -r %d /s /d
... and it appears that it works but never removes the read-me statues. Even right clicking the folder and going into properties and unchecking read-me only still does nothing. It applies and doesn't have any error, but never changes the attributes. Though I can click it and hit delete and that works, but the del or erase command will not delete it.Any ideas? I can traverse into every single folder inside the temp file and delete all the files out of those but the folders them selves will not delete. After doing some testing it seems for w.e reason I cant use the del command if the folder has a read-only attribute. Though I cant remove that attribute either. This is frustrating as I cant seem to delete ANY folder that has a read-only attribute set but they system wont let me change it. I am the admin on the machine and the command line is running as the admin, yet I still cant remove the read-only attribute. I can remove System and Hidden attributes but not read-only. It seems to just be some kind of issue with windows not really recognizing "read-only" attribute though that attribute prevents dos from deleting it. Sadly the work around that was mentioned on the Microsoft's support site doesn't work.Windows 7 is not fit for purpose and should never have even entered Beta.

It employs Junctions and that are supposed to give a form of backward compatibility with older applications written for XP.

Some of them are black holes. You can put files in them, but you cannot get them back, and you cannot read them or delete them. It is a fiasco.

I have counted 44 Junctions in Windows 7, most of those I wanted are blackholes

Run CMD.EXE and past this into the window.

Code: [Select]CD \
echo %TIME% > "%USERPROFILE%\Application Data"\ALAN_LOST
DEL "%USERPROFILE%\Application Data"\ALAN_LOST
echo %TIME% > "%APPDATA%"\ALAN_KEPT
echo %TIME% >> "%USERPROFILE%\Application Data"\ALAN_LOST
TYPE "%USERPROFILE%\Application Data"\ALAN_LOST
TYPE "%APPDATA%"\ALAN_KEPT
TYPE "%APPDATA%"\ALAN_LOST
DEL "%APPDATA%"\ALAN_KEPT
DEL "%APPDATA%"\ALAN_LOST

Nothing strange under XP, no errors - nothing to see her - move along.
Under Windows 7 Ultimate I got errors and black-holes as per image attached.

Alan

[recovering disk space - old attachment deleted by admin] Quote from: ALAN_BR on September 19, 2010, 03:55:34 PM

Windows 7 is not fit for purpose and should never have even entered Beta.

There is nothing wrong with windows 7. Or Vista, for that matter.

Quote

Under Windows 7 Ultimate I got errors and black-holes as per image attached.

Works fine here:

Where is this "black hole"? the Application Data folder redirects to %appdata% which is usually %userprofile%\AppData\Roaming. As you can see the Type command worked perfectly fine, as did both redirects. The Access Denied error is for a very good reason, since write Access to the "Application Data" junction is only allowed by some programs (usually those with a compatibility mode set to XP or earlier). If that wasn't the case, how many "power" users do you think would encounter it, be ABLE to look inside the folder, and decide "well GOLLY GEE! I already have all those files in %appdata%... DELETE"

only to find they just deleted %appdata%. Would they be pleased? *censored* no. They just deleted all their documents.

There is no capability for "data loss" here, as you claim. In fact, the opposite is true.You misunderstand.
I never said "data loss" but
"Some of them are black holes. You can put files in them, but you cannot get them back"

I mean that you cannot access the files by using the same path used for storing them.
This is nothing but "data loss" for people who do not know where to look for them.

This BlackHole problem I was unaware of until it hit me.
There are many others who have yet to suffer this aggravation.

Your "...\Application Data" is a Blackhole. You can put a file in, but you cannot get it back.
It is a SciFi type of blackhole with a wormhole to a distant galaxy, or in this case to a destination that is fully accessible to all read and write operations via the path
"...\Appdata\Roaming",
BUT IT IS ONLY ACCESSIBLE for those who look in the correct place,
and is totally lost for those who have downloaded a script that uses XP compliant paths.

What I have learnt from the Internet is that people lost everything because they believed original documentation that indicated a Junction was like a short-cut which could be safely deleted without harming the contents of what it re-directed to.

Microsoft did not correct their enormous mistake, and instead admitted the error BUT recommended that Junctions should be protected from accidental deletion by the use of CACLS.

I have used CACLS to inspect the permissions and restrictions of a sample from the 44 Junctions scattered around Windows 7, and find significant differences between them.

You have what could be a good point about protecting "Power Users",
but I would counter with :-

1. Although a Junction may protect against reading or deleting a file, it does not protect against appending a file, and presumable may permit a file to be written even if it over-writes another file.
Interesting example encountered at :-
http://forum.piriform.com/index.php?showtopic=29618
With a reinstall of Vista that resulted in an extra folder called "Windows Old", a CCleaner user found that CCleaner was prepared to delete music from to delete "Windows Old" and he let it happen.
He lost much of his real music that he treasured.
He found CCleaner did no such harm with a normal delete (which Junctions tend to block)
but his use of SECURE delete meant the files were over-written and lost.
I assume that what he saw as being in "Windows Old" actually encountered a Junction that was frozen at some place in "Windows"
The "nanny mode" of protecting against loss of data via Junctions may be fool-proof, but is not "Power user" proof ! !

2. Any self proclaimed power user that meddles with things he does not understand needs to learn better, and even XP will punish him if he removes duplicates that are genuinely independent duplicates but XP still expects to find them.
I have a total of 471 instances of Desktop.ini within 4 partitions.
I wonder what Windows would look like if I removed 470 of them ! ! !

3. I strongly suspect that power users are not fully protected.
Not all Junctions are Blackholes, some give full bidirectional access,
which means that DEL will delete a destination file and RD will delete a destination folder via a path that includes a Junction, so I suggest these are points of vulnerability to a meddlesome "Power User".
I guess the variation is something to do with M.$ not being sure what to do with CACLS.
e.g.
V:\W7_OOPS>CACLS "C:\Documents and Settings\Alan\Application Data\" | FIND ":"
C:\Documents and Settings\Alan\Application Data Everyone:(DENY)(special access:)
NT AUTHORITY\SYSTEM:(OI)(CI)(ID)F
BUILTIN\Administrators:(OI)(CI)(ID)F
Alan-Laptop-W7\Alan:(OI)(CI)(ID)F
or
V:\W7_OOPS>CACLS "C:\Users\Alan\AppData\Local\" | FIND ":"
C:\Users\Alan\AppData\Local NT AUTHORITY\SYSTEM:(OI)(CI)F
BUILTIN\Administrators:(OI)(CI)F
Alan-Laptop-W7\Alan:(OI)(CI)F
or
V:\W7_OOPS>CACLS "C:\Users\Alan\Local Settings\Application Data\" | FIND ":"
Access is denied.
or
V:\W7_OOPS>CACLS "C:\ProgramData\" | FIND ":"
C:\ProgramData NT AUTHORITY\SYSTEM:(OI)(CI)F
BUILTIN\Administrators:(OI)(CI)F
CREATOR OWNER:(OI)(CI)(IO)F
BUILTIN\Users:(OI)(CI)R
BUILTIN\Users:(CI)(special access:)
or
V:\W7_OOPS>CACLS "C:\ProgramData\Application Data\" | FIND ":"
C:\ProgramData\Application Data Everyone:(DENY)(special access:)
Everyone:R
NT AUTHORITY\SYSTEM:F
BUILTIN\Administrators:F

When Comodo Security is upgraded it may need un-installation of the old before the new,
and the new may be blocked if residues remain due to sundry glitches, so various users have posted scripts that seek and destroy the residues.
Those scripts work on XP for some people, but failed for me due to permissions issues.
I added "error checking" to confirm successful removal or report specific problems that needed manual intervention.
Perfection at last on XP.

Before final release I checked that all was well on W7 and found disaster.
There were "residues" which I deliberately planted to ensure correct detection and removal or reporting.
They were not detected.
Even using %USERPROFILE%\etc... or %APPDATA%\etc... they could not be detected if the \etc... went through another Blackhole type junction - IF EXIST could not see them, CACLS had no access, DEL could not touch them.

This aggravation has caused me to build up a full head of steam. Sorry about that.

The original start of this thread was that the O/P knew certain files existed in a certain place but he could not copy them. This resonated very strongly with the problem I have suffered, and after checking his profile and seeing that the OP use W7 I thought that BlackHole junctions might be the cause depending upon what paths he was using, so I posted what I thought might be useful to him.

I have much more to say and discuss, and questions to ask.
Thank you for your information on compatibility mode,
I have never needed it before and will investigate how this might affect CMD.EXE scripts.

I do not wish to be barred by a moderator for hijacking the OP's topic,
and suggest we take this outside to settle it ! !

I propose to start a topic in the Windows 7 forum when I get a few other urgent things done,
and will post here a quick invitation to join me there when I have organized my evidence against Junctions.

Regards
Alan
Alan thank you very much for posting what you did, I would definitely not classify this as high-jacking as this was very insightful. I never knew such things could and do happen. That would probably explain why I cant delete them using that command.

Do you know of any kind of workaround or some way I may be able to del/copy them?

I can successfully copy them but I have to copy the parent folder the files are in to a secondary file which I can use the attrib command to remove their system/hidden attributes. Though sadly, windows does not let me delete these new folders using the del command. I know these copied folders are standalone from the others as the original maintains it's attributes and files inside of it. Where as the copied folders I can change their attributes(excluding read-only), and I can delete the files from those folders but not the folders them selves.XCOPY /H
That is like Copy but will deal with Hidden and System files should be included.
You will see lots of options by
XCOPY /?

That may be all you need, depending upon what paths are involved,
but Junctions can give all sorts of problems.

My favorite resource is
http://wapedia.mobi/en/Environment_variable?t=5.#5.
The top item shows the environment variable %APPDATA%
and the real destination for both XP and Vista/W7
It is far better to access via %APPDATA% than the XP equivalent of C:\Documents and .......
The XP equivalent will in W7 be redirected by Junction(s) but with restricted permissions.

This freeware will scan the system and find and report all junctions.
http://rekenwonder.com/linkmagic.htm
N.B. the scan took LONGER on my machine than a coffee break,
so I took a lunch break and it finished before I ended my meal.

It reports the location of the Junction and where the REAL destination is. e.g.
Code: [Select]==================================================
Junction point : C:\Users\Alan\Application Data\
Destination : C:\Users\Alan\AppData\Roaming
==================================================
I can create ALAN_LOST at %USERPROFILE%\Application Data\ which becomes
C:\Users\Alan\Application Data\
and it is actually sent to
C:\Users\Alan\AppData\Roaming\
I am severely restricted in what I can do via the path
C:\Users\Alan\Application Data\
but have full control via the path
C:\Users\Alan\AppData\Roaming\

You will get information on Junctions at
http://en.wikipedia.org/wiki/NTFS_junction_point
http://support.microsoft.com/?kbid=205524
Microsoft WARN that Junctions must be protected from accidental deletion,
and they recommend the use of CACLS to MODIFY the Access Control Lists.

Previously I have shown
C:\Documents and Settings\Alan\Application Data Everyone:(DENY)(special access:)
NT AUTHORITY\SYSTEM:(OI)(CI)(ID)F
BUILTIN\Administrators:(OI)(CI)(ID)F
Alan-Laptop-W7\Alan:(OI)(CI)(ID)F

If you launch CMD.EXE the command CACLS /? will tell you what you can do with it.
You will see that (OI)(CI)(ID) deal with inheritance - I do not want to go there ! !
It may fix things for you, but it could also destroy everything. Only use as last resort.

Regards
Alan
Quote from: ALAN_BR on September 21, 2010, 02:33:52 AM

This freeware will scan the system and find and report all junctions.
http://rekenwonder.com/linkmagic.htm
N.B. the scan took longer on my machine than a coffee break,
so I took a lunch break and it finished before I ended my meal.

It reports the location of the Junction and where the REAL destination is. e.g.
Code: [Select]==================================================
Junction point : C:\Users\Alan\Application Data\
Destination : C:\Users\Alan\AppData\Roaming
==================================================

There is another piece of software that does this. It's called DIR. dir /s /al will list all junction points on a drive and their targets.

Quote

I can create ALAN_LOST at %USERPROFILE%\Application Data\ which becomes

Question is, Why are you using %USERPROFILE%\Application Data\ instead of %APPDATA%?

%APPDATA% on windows XP resolves to Application Data, and %APPDATA% in Vista/7 resolves to %USERPROFILE%\AppData\Roaming. Also, it will account for those times when your profile is on a network drive. IT doesn't work for Windows 98, since windows 98 doesn't by default have an %APPDATA% variable. It does have an application Data folder in C:\Windows\Application Data\, which means that a more "universal" method for MS to have done would have been to create a junction point in the windows folder as well that redirected to the appropriate location to account for those win98 programs that hard coded %WINDIR%\Application Data as the destination.

They of course didn't, because Applications can get the Application Data folder in Windows 98 and later using well documented functions, not environment variables.

the SHGetFolderPath Function is one such entry.

So you may be looking at that link and going HAHA! but it's not supported on windows 98!

And you would be quite correct! It's not. Most applications using the function will install a redistributable "shfolder.dll" file, which works in windows 98. A more obtuse method of doing the same thing would be to use the SHGetSpecialFolderLocation Function, which gives back a PIDL (Pointer to an ID List) and then pass that PIDL to the SHPathFromIDList() Function, which doesn't even appear to be documented at all. This will give back the fully qualified path of the Special Folder on Windows 98 and Later.

Windows Vista, along with introducing the use of the junction points that you are so keen on using for some reason,(the junction points are for older, badly written programs to use, not for users to start throwing files into) changed the mechanic for retrieving special folders. These older methods still work, but they simply delegate their work to the new method, which is the IKnownFolder interface. In the purest sense, it is more difficult to work with- requiring the construction and use of an interface. Thankfully there are further wrappers around this, using SHGetKnownFolderPath And SHSetKnownFolderPath to acquire and Set the known folder location respectively.

The Setting operation is particularly relevant. While you are quite keen on assuming that there will always be an Application Data Folder within %USERPROFILE%, what if said %USERPROFILE% redirects to a network server drive? Surely you cannot expect another machine to automagically have this path present? So rather then using a carefully constructed path to cause errors, use %APPDATA%, which will redirect properly to the actual location of the users Application Data Folder, rather then %USERPROFILE%\Application Data, which will only work well on windows XP, and even then, only on unmodified configurations.

I'm not sure wether XCOPY skips symbolic links/junctions while recursing folders. Even if it doesn't, though, the fact that it won't be able to read from the junction point means it will just skip that folder anyway.

If the errors are a bother, one could always try ROBOCOPY, which is included in Vista and 7, and will skip junction points, reparse points and hard-linked files and folders by default.

IF anything, be happy that MS decided to use a reparse point (Junction) rather then actually hard-linking the folder to the "real" location. If they had done that, your precious little batch file would run fine, but deleting anything within %USERPROFILE%\Application Data would delete the same stuff within the AppData\Roaming folder, and deleting the folder itself would delete the Roaming Folder entirely. There would also be no indication there was any redirection at all- no shortcut symbol, just a standard folder that looks and acts just like one. This would be because a hardlink is just the file name pointing at the data- all filenames are hardlinks, but you generally only have a 1 to 1 relationship between file names and file data. Creating a new hard link to the same data is just giving a new name to that data; in the case of a folder, there won't even be a path redirection- that is, all the folders will still "appear" to be in Application Data, because, technically, they are. But they would also be in the Roaming Folder, since the Roaming and Application Data would have been configured to use the exact same directory. I don't doubt that they considered this option and decided that they would rather field calls from people whose data was merely redirected and not deleted. "I found this Application Data Folder, and it had the same stuff as in my Roaming\AppData Folder, so I deleted it, and now none of my programs work!". Not something a new user would do, but people who consider themselves "Power Users" will often go futzing about in various system managed folders, so that type of thing would have been assured, not just a possibility. The only other choice they could have done would have been to simply not have a junction point at all. But this would hardly work, either. If a Program that worked fine on Windows XP despite it hard-coding something like "%USERPROFILE%\Application Data" for accessing it's data no longer worked on Windows Vista or 7, the user isn't going to think "well, golly, this program is bad" they are going to blame Windows. MS does the best they can to make sure these badly programmed pieces of crap work fine, because there are a lot of people whose lives and business are teetering on the edge of some badly programmed piece of software that uses hard-coded paths, and if that program doesn't work in Vista/7, they simply won't upgrade. So, they need to think of an option that will get these programs working with minimal side effects. They had a number of options:

1. The Hard Linked folder method- Already described above, as well as it's inherent weaknesses. Also, another rather major downside would be that a hard-linked folder is nearly impossible to programmatically determine- unlike a junction point, you can't just query it's attributes and go "GOLLY! this is a hard-linked folder!" So, you end up with backups containing two copies of the exact same data, one from Application Data, and one from AppData\Roaming. This all also applies to the other junctions that MS implemented (had they been made into hardlinks instead, I mean), such as My Documents pointing to the Documents folder, as well as a number of others. And god forbid if you have yet another hard link within that folder! otherwise you would have backup programs constantly recursing into them until the path is too long. (See %USERPROFILE%\Local Settings\Application Data, notice there is a Application Data junction. click it. the junction points to the Local Settings Folder. Notice the path says you are in %USERPROFILE%\Local Settings\Application Data\Application Data. you can continue to dig into the Application Data Folder over and over and over. As the path grows longer and longer and longer. It starts to take a while for the view to refresh, though, after about 30 or so.

2. implement the redirection as part of the file system redirector. They already do this for Windows x64, whereby accesses to C:\Windows\System32 by 32-bit programs are silently redirected to C:\windows\syswow64, so why not do it here?

For one reason- it's stupid. The file system redirector was being used in windows x64 to redirect access to the C:\Windows\System32 folder, something that had been ubiquitous for years, to the WOW folder that was used for the 32-bit version of system32. (Why not create a System64? I don't know, sure they had reasons, this redirection thing is a clue to their reasoning, in fact). In this instance, it would mean actually redirecting, within the file system driver, folder paths like "C:\Documents And Settings\Tom\Application Data" to "C:\Users\Tom\AppData\Roaming" It seems simple, and really, it is. But consider that every single time a file or folder is accessed, the driver will have to go "alright, I'll just check wether I need to redirect this..." When you have 2 or 3 users, that's "only" around 30 or so permutations to check. But It's not scalable- Windows is also deployed in environments with hundreds or thousands of users. having a server check a given file path against thousands or even millions of different possible paths that need to be redirected while everybody twiddles there thumbs is not apt to get people liking the new OS.

3. Create a Junction Point from the old location to the new location.

This is what they chose to do. It's the best solution- it allows old XP oriented applications to work fine with Vista/7, and it's transparent. The only people that see it are those you go futzing about using non-default explorer settings (such as, for example, show hidden files/folders). The average user doesn't make a hobby out of exploring deep within their profile folder. Anybody who does should know what they are doing. If you are going to echo data into a file in the junction, you either learn that it's a junction and where the data "really" is or you continue to throw data into it and scratch your head and end up blaming MS. In the former case you just say "ahh, ok, I guess I'll stop being an idiot and redirecting stuff into junction points designed solely for XP compatibility.Nice text wall BC =)
So, assuming I understood that, it would seem that the folder I'm trying to access isn't the correct folder and just a junction(if I'm using that correctly) to another folder. So the folder I need to be accessing is some where else. Also, I was never trying to delete anything from "App Data"/"Roaming" I had made a copy of a parent folder holding other folders in it to a completely different location on the hard drive and I wouldn't let me us the del command to delete these newly created folders. From what I've read I'm guessing windows still thinks it's the junction to \Roaming since the file names never change and that's why it wont let me delete it using that command.

Also, thank you both for posting here. This has been really insightful, I would have never known half of this stuff!

Maybe I'm not understanding as much as I thought I did. I'm trying to now find the corresponding folder in the \Roaming folder. The folder I'm trying to copy things out of is the "C:\Users\owner\AppData\Local\Microsoft\Windows\Temporary Internet Files\" folder since it didn't let me do that I could copy the paren't folder "Temporary Internet Files" into a new directory. This then gives me full rain over the files in there, but then it wont let me delete the new folders that were created in there like "Content.IE5". Is this do the junctions? Or is this a different issue.

Solve : Copying files marked as "system files"?

Discussion

No Comment Found

Related InterviewSolutions

Reply to Comment