token_utils.py - A collection of token-related functions

A couple of functions for dealing with tokens generated by the tokenize module.

pyminifier.token_utils.listified_tokenizer(source)

Tokenizes source and returns the tokens as a list of lists.

pyminifier.token_utils.untokenize(tokens)

Converts the output of tokenize.generate_tokens back into a human-readable string (that doesn’t contain oddly-placed whitespace everywhere).

Note: Unlike tokenize.untokenize(), this function requires the 3rd and 4th items in each token tuple (though we can use lists or tuples).