Install
AI / LLM Analytics terms
Benchmarks, evals, leaderboards and the observability of AI systems.
Agentic analytics The use of AI agents that autonomously plan and execute multi-step analytics work, such as investigating a metric change. AI agent A system that uses a language model to plan and take multi-step actions, such as calling tools or APIs, toward a goal. Context window The maximum amount of text, measured in tokens, a language model can consider at once when generating a response. Embeddings Numeric vector representations of text, images, or other data, positioned so similar items sit close together in the vector space. Fine-tuning Further training a pretrained model on a smaller, targeted dataset to adapt its behavior to a specific task or domain. Generative AI AI systems that create new content, text, images, audio, or code, rather than only classifying or predicting a label. Guardrails Rules, filters, or checks placed around a language model to keep its inputs and outputs within acceptable, safe bounds. Hallucination When a language model generates confident, fluent output that is factually wrong or unsupported by its source material. Knowledge graph A network of entities and the labeled relationships between them, used to represent and query structured knowledge. Large language model (LLM) A neural network trained on vast amounts of text to predict and generate language, the technology behind most modern AI assistants. LLM evaluation Systematically testing a language model's outputs for quality, accuracy, safety, and consistency before and after deployment. LLM observability Monitoring a language model's live behavior in production, tracking its inputs, outputs, latency, cost, and failure patterns. LLM-as-a-judge Using a language model to score or compare the outputs of another model, as a scalable substitute for human review. Prompt engineering Designing and refining the instructions given to a language model to reliably get the output you want. Retrieval-augmented generation (RAG) An architecture that retrieves relevant documents at query time and feeds them to a language model so its answers are grounded in sources. Semantic search Search that matches results to a query's meaning rather than its exact keywords, using vector similarity between embeddings. Token The basic unit of text, roughly a word piece, that a language model reads, processes, and is priced and measured by. Vector database A database optimized to store numeric embeddings and quickly find the ones most similar to a given query vector.