Token visualizer

This tool allows you to visualize the tokens of a text prompt or snippet, using the tokenization models of the various Google Cloud Vertex AI large language models. Tokens are also counted, and hovering over them will indicate their internal IDs. The source code of this application is available on Github.

Your text

PREVIEW text-embedding-preview-0409 text-multilingual-embedding-preview-0409 STABLE textembedding-gecko textembedding-gecko-multilingual text-bison text-unicorn chat-bison CODEY code-bison code-gecko codechat-bison
Tokenize Clear

The tokens