Using LLMs in a Large-Scale Feed Recommender
Problem statement
Where would you use LLMs to improve a large feed recommender? The feed mixes media in many languages, and new items have little interaction history. Retrieval, ranking and diversity checks already work, but traffic is very high and responses must be fast and affordable.
Estimate online reranking cost, choose a first deployment and explain where LLMs help the funnel.
Show hint
Which pieces of LLM work scale with the number of items, and which scale with the number of feed requests?
Relevant roles: ML Engineer, Applied Scientist, AI Engineer and LLM Engineer.