Memories Before 2021

Looking back, around 2018 felt like a major turning point for the search community. At the time, I was still experimenting with Word2Vec and FastText to make search smarter, when BERT was published¹, followed by Amazon's paper on semantic product search in 2019². I vividly remember feeling an intuitive sense that the future of search was about to fundamentally change.

On the operational side, however, keyword matching was still the absolute standard. System-wise, search engines and data processing middleware revolved around the Java ecosystem, creating a strong norm that applications should also be written in Java. Search engineers were generally understood to be those knowledgeable about search engine operations and internals—and working at the intersection of Search and ML was considered an unconventional path. Furthermore, integrating neural models into production was widely dismissed due to performance concerns and poor ROI when running such massive models (small as they seem today).

While sensing immense potential, I felt the limits of catching up on a part-time basis and wanted dedicated time to thoroughly test theory through implementation. Before I knew it, I enrolled in a graduate program near my house.

After completing my studies, I resumed working in Japan in 2021. Over the past five years applying what I learned, the search landscape evolved far beyond my expectations. To record these trial-and-error experiences, I want to compare my memories from back then with where we stand today—reflecting on what changed and what remained constant.

Five Years of Experimentation and Current Standing

The Era of Great Embeddings

The past five years saw a rapid transition from feature engineering on unstructured data to automated world understanding via representation learning. Ideas like learning representations from behavioral logs (Listing Embedding, 2018)³ and StarSpace's (2018)⁴ philosophy of "Embed All The Things" were already present, but coupling them with Transformers and Vision Transformers drastically accelerated practical adoption. Today, embeddings have become the universal language spanning from retrieval to reranking, forming the backbone of modern search systems.

While generating embeddings traditionally relies on Transformer encoders, an approach reusing decoders (= LLMs) has gained traction. I attribute this to two main reasons: first, Next Token Prediction scales more efficiently than Masked Language Modeling, making model scaling easier; second, decoders easily power revenue-adjacent applications like chat and agents, attracting heavy capital from both academia and industry. As a result, open encoder development appears to have slowed down.

Should all embedding generation in search be replaced by decoders? In fact, large-scale applications at Uber and Walmart are exploring consolidating multiple task-specific encoders into a single shared decoder⁵⁶. However, recent literature⁷ indicates that achieving matching performance with decoders requires orders of magnitude more parameters (e.g., 100x).

Given this reality, consolidating embedding generation into decoders largely depends on internal infrastructure. If a company already maintains fine-tuned LLMs for generation use cases, piggybacking embedding generation onto that shared infrastructure is rational. Otherwise, managing small, fast, task- or entity-specific Encoders independently seems far more straightforward for development and operations—though who knows how this will look in another five years.

Retrieval: Hype vs. Fact in Semantic Search

The most intense gap between hype and reality over the past five years occurred with the rise of semantic (vector) search in retrieval. In the early 2020s, as embedding quality improved, bold claims like "Keyword Search is Dead" and "Vector search will replace everything" were everywhere. Concurrently, numerous engineering blogs reported: "It might work in academia, but it completely failed on our internal data."

Interestingly, neither perspective was entirely wrong. Academic benchmarks often focus on tasks where exact keyword matching matters less (e.g., sentence similarity or QA). In production applications like e-commerce search, however, strict requirements exist—users expect input keywords to be present and reject non-matching products even if semantically similar. Blindly applying paper scores to production led directly to failure.

This does not mean vector search is impractical; not all e-commerce queries demand exact matches. For example, the following table summarizes the match type breakdown of engaged search logs on a production platform:

Because exact keyword matching combined with dictionaries covers over 90% of use cases, evaluating overall averages makes keyword search appear overwhelmingly superior. However, for the remaining ~10% of queries, vector search effectively captures visual features and intent that keyword matching misses. This advantage widens further with multimodal embeddings. Testing confirmed that a tuned hybrid architecture leveraging the strengths of both delivers the best online performance.

Experiencing these insights through iterative offline and online experiments with the team—rather than just reading about them—was invaluable. It even led to a memorable trip to Germany.

Similar strong claims like "... Is All You Need" appear frequently in RAG discussions, but the core lesson remains unchanged: instead of adopting the latest methods for their own sake, engineering begins with understanding the task, context, and selecting the right tool for the job.

Reranking: Infrastructure Shifts and Merging with Research

In ranking systems, simple and interpretable tree-based models once dominated. However, to leverage high-quality embeddings and model complex context alongside multiple optimization targets, neural architectures swept into this domain as well.

Consequently, I transitioned from tree-based models to TensorFlow-based pipelines (TF Ranking, TF Recommenders + TF Serving), and eventually to a PyTorch + Triton stack. What seemed like a circuitous path feels inevitable in hindsight.

Initially, TensorFlow held a clear advantage in production operations. This created a division: researchers designed models in PyTorch, while engineers reimplemented them in TensorFlow for production. Around that time, academia raised critiques regarding low reproducibility and questionable progress over baselines in neural models⁸⁹. This spurred a cultural shift toward releasing code, datasets, and pretrained weights—with PyTorch at the center of this ecosystem.

Serving PyTorch models directly became far more practical. Coupled with compatible model formats and Triton's execution environment, the gap between research and production dissolved. Beyond technical unification, the boundary between researchers and engineers blurred, fostering a culture of end-to-end ownership while establishing ML as a core responsibility for search engineers.

The Future of Search

As AI advancement lowered the perceived scarcity of knowledge, I occasionally joked "Well, time for us to retire," while feeling a subtle sense of helplessness. Looking back, however, I realized that past learnings and hands-on experiments have become an integral part of how I think and act today.

Before "LLM as a Judge" existed, evaluating search results meant dumping queries into spreadsheets and inspecting them row by row. Facing raw data directly cultivated an intuition for edge cases and query distribution skew that aggregated metrics obscure. Regardless of how technology evolves, continuously analyzing current problems and evaluating solutions remains fundamental.

To conclude, I am leaving a few open questions about the future of search to revisit five years from now:

• The Trajectory of Encoders vs. Decoders
  • Will task-specific encoders endure, or will decoder versatility absorb everything?
• Adoption of Next-Generation Retrieval
  • Learned Sparse Retrieval and Multi-Vector Retrieval gained initial momentum with systems like Vespa, yet production adoption felt slower than expected. How will these mature, or will Differentiable Search Indexing and Generative Retrieval become the new standards?
• LLM Integration into Real-Time Search
  • Five years ago, running large neural models in production seemed infeasible, yet it became standard practice. Similarly, while serving LLMs in real-time search currently presents high hurdles, will technological breakthroughs lower this barrier?
• Agentic Search and UX Paradigm Shifts
  • For practical use cases (finding specific products or content), traditional single-query search interfaces remain effective. How will search UX be redefined once autonomous AI search agents mature?
  • Current constraints stem from serving tens of thousands of QPS within 200ms. Will user expectations shift toward waiting a few seconds for optimal results, relaxing these strict latency budgets?

Predicting the future is fun—though if I am no longer an engineer in five years, that might be the most interesting outcome of all.