Modern large language models (LLMs) have made huge progress in modeling language. However, due to the ever-changing nature of how language is expressed, language models are bound to become a snapshot of the way language was written not how it is written in the present or will be in the future. The current state-of-the-art research methods do not seem to keep track of when a language model has turned stale i.e. the model no longer represents the up-to-date text it was meant to represent. In this talk, we will first investigate this phenomena and see how LLMs fail to model unseen language. Then, we will introduce empirically proven techniques that can help detect this phenomena in the wild. Finally, we will cover methods that can be used to alleviate the problem of covariate shift. This will increase robustness of ML systems and help NLP practitioners deploy models in the wild with confidence.