Latent Scope

Latent Scope is built by Ian Johnson (enjalot), an independent data visualization researcher known earlier for d3.js-era work. The project describes itself as a scientific instrument for investigating latent spaces and packages a five-stage pipeline that turns an unstructured text dataset into an interactive map of clusters and themes.

The pipeline runs locally end-to-end. Stage one embeds each row of text using either a local transformer model (BGE, E5 are referenced) or an API embedding (OpenAI, Mistral). Stage two reduces the embedding to two dimensions with UMAP. Stage three clusters the 2D points with HDBSCAN. Stage four labels each cluster by passing representative points to an LLM (Zephyr 7B locally, or GPT-3.5-class via API) for summarization. Stage five renders the result as an interactive web scatterplot the user can pan, zoom, and query.

Implementation is Python on the backend (command-line utilities orchestrate the pipeline) and React on the frontend. Storage is intentionally flat-file: HDF5 for embeddings, Parquet for tabular data, with metadata logging so each run is reproducible and the artifacts can be moved between machines. Demonstrated scales run from a few hundred rows up to several hundred thousand, including a 400,000-row emotion classification dataset.

The Mozilla Builders Accelerator awarded Latent Scope up to $100K in the September 23, 2024 cohort. In the stack the project sits at the data and evaluation layers: it is a workbench for analysts who want to look at what an embedding model has actually clustered without uploading the dataset to a hosted vector service. The design choice to keep the full workflow on the user's machine matches the cohort's local-AI theme and makes the tool usable on sensitive datasets that cannot leave the host.