Tokenizer Class Reference

Tokenizer is a forward iterator that tokenizes a given string. More...

Inheritance diagram for Tokenizer:

Public Member Functions
	Tokenizer ()
	Creates an empty tokenizer. Effectively creates an end iterator. More...
	Tokenizer (const std::string &str, const std::string &delimeters)
	Creates a new tokenizer. More...
std::string	Current () const
	Return the current token. More...
const Tokenizer	end () const
bool	IsValid () const
	Whether the iterator is valid (i.e. dereferencable) More...
void	Next ()
	Move to the next token. More...
	operator std::string () const
	Return the current token. More...
bool	operator!= (const Tokenizer &st) const
	Compare two iterators, does not check if two iterators are identical. More...
std::string	operator* () const
	Return the current token. More...
void	operator++ ()
	Move to the next token. More...
const std::string *	operator-> () const
	Return the current token. More...
bool	operator== (const Tokenizer &st) const
	Compare two iterators, does not check if two iterators are identical. More...

Public Attributes
std::string	Delimeters
	Delimeters to be used in tokenization. Can be changed while tokenizing. More...

Protected Attributes
std::size_t	position
	Position of the next token. std::string::npos denotes the iterator reached the end. More...
std::string	text
	Current text. More...
std::string	token
	Current token. More...

Detailed Description

Tokenizer is a forward iterator that tokenizes a given string.

This class only supports single character delimeters, however, its possible to specify more than one delimeter.

Constructor & Destructor Documentation

Tokenizer() [1/2]

Tokenizer

(

)

Creates an empty tokenizer. Effectively creates an end iterator.

Tokenizer() [2/2]

Tokenizer	(	const std::string &	str,
		const std::string &	delimeters
	)

Creates a new tokenizer.

Parameters

str	string to be tokenized
delimeters	is the delimeters to use while tokenizing

References Tokenizer::Next(), and Tokenizer::position.

Member Function Documentation

Current()

std::string Current

(

)

const

Return the current token.

References Tokenizer::token.

end()

const Tokenizer end

(

)

const

IsValid()

bool IsValid

(

)

const

Whether the iterator is valid (i.e. dereferencable)

References Tokenizer::position, and Tokenizer::text.

Next()

void Next

(

)

Move to the next token.

References Tokenizer::Delimeters, Tokenizer::position, Tokenizer::text, and Tokenizer::token.

operator std::string()

operator std::string

(

)

const

Return the current token.

References Tokenizer::token.

operator!=()

bool operator!=

(

const Tokenizer &

st

)

const

Compare two iterators, does not check if two iterators are identical.

Compares only current positions and tokens

References Tokenizer::position, Tokenizer::text, and Tokenizer::token.

operator*()

std::string operator*

(

)

const

Return the current token.

References Tokenizer::token.

operator++()

void operator++

(

)

Move to the next token.

References Tokenizer::Next().

operator->()

const std::string* operator->

(

)

const

Return the current token.

References Tokenizer::token.

operator==()

bool operator==

(

const Tokenizer &

st

)

const

Compare two iterators, does not check if two iterators are identical.

Compares only current positions and tokens

References Tokenizer::position, Tokenizer::text, and Tokenizer::token.

Member Data Documentation

Delimeters

std::string Delimeters

Delimeters to be used in tokenization. Can be changed while tokenizing.

position

std::size_t position

protected

Position of the next token. std::string::npos denotes the iterator reached the end.

text

std::string text

protected

Current text.

token

std::string token

protected

Current token.

---

The documentation for this class was generated from the following files:

• Gorgon/String/Tokenizer.h
• Gorgon/String/String.cpp