Prev
Home
Next
Menu

Generating tokens of a Stream using StreamTokenizer class

In Java, the java.io.StreamTokenizer class is used for simple parsing of a Java source file into tokens.  This class takes an input stream, breaks up the ASCII text in a file it and parses it into “tokens”, allowing the tokens to be read one at a time. The parsing process is controlled by a table and a number of flags that can be set to various states. This class is extended from the java.io package.

The use of token is just as we take tokens for getting milk or to enter in metro, so that the one token can be read at a time. Let’s understand the example shown below.

In the given program named File15.java, firstly we make a class called File15.  Now make an object of BufferedReader class and pass the InputStream class object into it which will read the name of the java file at run time. Now we need to read the file so that we can use FileReader class, after reading a file we need to generate tokens out of that file so that we can use the class StreamTokenizer and pass the reference of a Reader class in its constructor. In this program we are using one method of StreamTokenizer class. 

To solve this problem we have made use of the following classes and methods.

InputStream: It is an abstract class and the parent class of all the Stream classes.

BufferedReader: This class reads the text from the input stream. This class extends Reader class. 

File: It is a class implementing Serializable and Comparable interface.

FileReader:  This class reads the file in the character format. It extends InputStreamReader.

StreamTokenizer: This class takes an input stream and breaks it into tokens so that one token can be read at a time. It takes FileReader instance as an input in its constructor. 

readLine(): It is a method of BufferedReader class which reads a line.

exists(): It will check whether the file exists or not.

nextToken(): It gives the next token.

 

The code of this program is given below:

FileTest15.java

import java.io.*;

public class FileTest15 {

    public static void main(String[] args) throws IOException {
        BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
        System.out.print("Please enter a Java file name: ");
        String filename = in.readLine();

        if (!filename.endsWith(".java")) {
            System.out.println("This is not a Java file.");
            System.exit(0);
        }
        File javaFile = new File(filename);

        if (javaFile.exists()) {
            FileReader file = new FileReader(filename);
            StreamTokenizer streamTokenizer = new StreamTokenizer(file);
            // It will go through the file and gives the number of tokens in the file
            int i = 0;
            int token_count = 0;

            while (i != StreamTokenizer.TT_EOF) {

                i = streamTokenizer.nextToken();
                token_count++;
            }
            System.out.println("Number of tokens = " + token_count);
        } 
        else {
            System.out.println("File does not exist!");
            System.exit(0);
        }
    }
}

 

The output is shown here:

C:\file>java FileTest15
Please enter a Java file name: FileTest15.java
Number of tokens = 166

 

 

Prev
Home
Next