Most abstractive summarization approaches of today rely on not-that-large sequence-to-sequence encoder-decoder models and are limited to contexts of 512 to 1024 lexical tokens in the source document only. In parallel, there has been significant progress in training autoregressive decoder-only Large Language Models (LLMs) with billions of parameters. These large-scale models are demonstrating impressive general language modelling capabilities, and there is a growing number of publicly available pre-trained LLMs, such as GPT-J, GPT-NeoX, OPT and BLOOM. Can these pre-trained models be leveraged for summarizing domain-specific long documents? These models have been pre-trained on shorter documents up to 2048 lexical tokens and with general lexicon, so they require fine-tuning to efficiently tackle summarization of domain-specific long documents. However, fine-tuning LLMs on very long documents is infeasible on traditional hardware and with existing software libraries. We are leveraging industry-leading AI compute systems for this work, and we will be sharing our work-in-progress on fine-tuning autoregressive LLMs for domain-specific long document summarization.