This tool allows you to visualize the tokens of a text prompt, using the Gemini and Gemma tokenizer and the various embedding models from Google Cloud Vertex AI. Tokens are also counted, and hovering over them will indicate their internal IDs.
Note: The Gemini/Gemma tokenizer token IDs are not the real IDs as the library used in this application doesn't provide them,
but the text-bison embedding model uses the same vocabulary as the Gemini/Gemma tokenizer, and thus shows the right token IDs.
The source code of this application is available on Github.