1.

Solve : Question about tokens and delimeters?

Answer»

I'm just curious about something I couldn't find any more information about.

In the following code:

Code: [Select]echo off
for /f "tokens=1,2* delims=, " %%a in (test.txt) do echo %%a %%b %%c
pause

test.txt contains:

Code: [Select]This, is, just, a, test,
The delimeters are not applied after the first and second token, even though a wildcard was used.

Code returned the following:

Code: [Select]This is just, a, test,
Only way I could get delimiters to be applied on all tokens was by adding additional FOR variables and using a token range. If I was to display a bigger amount of text ALONG with using the delimiters, I would be limited to 52 active FOR variables.

Does this mean that the delimiters are not compatible if the tokens are set using a wildcard? Or are the delimiters dependent to the amount of the set FOR variables?

Perhaps I MISSED something in help for the FOR command. Any ideas? Quote from: Night on August 10, 2014, 08:45:40 AM


In the following code:

Code: [Select]echo off
for /f "tokens=1,2* delims=, " %%a in (test.txt) do echo %%a %%b %%c
pause

test.txt contains:

Code: [Select]This, is, just, a, test,
Code returned the following:

Code: [Select]This is just, a, test,

I'm not clear on your question but that is correct. 
The spaces and commas are treated as separators up the the start of the third term, which starts with just

The * means "put the rest of the line into the next variable"

If it's unclear then explain what you expected, or what you need it to look like.

Example from help for FOR command.

Quote
FOR /F "eol=; tokens=2,3* delims=, " %i in (myfile.txt) do echo %i %j %k

    would parse each line in myfile.txt, ignoring lines that begin with
    a semicolon, passing the 2nd and 3rd token from each line to the for
    body, with tokens delimited by commas and/or spaces.  Notice the for
    body statements reference %i to get the 2nd token, %j to get the
    3rd token, and %k to get all remaining tokens after the 3rd.

From that example, as said, %k variable displays all the remaining tokens. I'm not sure why the delimiters are not applied to the remaining tokens after the variable (%k). As it's stated from that example that the variable %k stands for ALL following tokens, not just the rest of the line. The delimiters are only applied to the given tokens (2,3) while the wildcard that represents all the remaining tokens is ignored.

I hope this clarifies my question a BIT more. Quote from: Night on August 10, 2014, 10:26:06 AM
it's stated from that example that the variable %k stands for ALL following tokens, not just the rest of the line.

As you have FOUND, the variable %k stands for the rest of the line, that is, any remaining tokens and separators all together. The help documentation could be clearer in EXPLAINING that, I suppose,  (it can be occasionally opaque or seemingly incomplete) but you have discovered by experiment the way it works.

Quote
the wildcard that represents all the remaining tokens is ignored.

It is not "ignored". It is correctly interpreted. The asterisk ("wildcard") represents the whole remainder of the line. There are no more "tokens" to process.







Thanks for the clarification! I guess the documentation isn't all that good after all.

That would be all.


Discussion

No Comment Found

Related InterviewSolutions