- A collection of token-related functions

A couple of functions for dealing with tokens generated by the tokenize module.


Tokenizes source and returns the tokens as a list of lists.


Converts the output of tokenize.generate_tokens back into a human-readable string (that doesn’t contain oddly-placed whitespace everywhere).

Note: Unlike tokenize.untokenize(), this function requires the 3rd and 4th items in each token tuple (though we can use lists or tuples).