A couple of years ago when I was trying to better understand tokenization I put together a video of some of the tools I had built for visualizing the tokenization process while using OpenAI's tiktoken as a dependency. First a simple console example, and then a slightly more advanced browser version