Subject Ph. D.

While cloud large language models (LLMs) like ChatGPT are evolving fastly in terms of feasible tasks, they still suffer from several drawbacks. In particular: cloud operations such as inference consume a lot of energy; user's privacy is a crucial issue, preventing sensitive data (e.g., a job application or a patent) from being processing. In this context, being capable of deploying an LLM on a small device like a mobile phone or laptop becomes key. But for this, it is necessary for the LLM model to be much smaller in terms of number of effective parameters and total number of bits to store the model, and aggressive model compression is one way of achieving this goal. To achieve extreme compression rates (10^2--10^3) for large neural networks such as transformers, it is necessary to resort to advanced tools and, possibly, to make the on-device LLM less general in terms of realizable tasks (making it more goal/objective-driven). The main research directions concerning LLM model compression that have been explored so far in the literature are pruning, distillation, quantization, and task specialization (e.g., by using low-rank perturbations of the weight matrix of the model). Although significant progresses have been made in the literature of model compression, current compression rates remain largely insufficient for enabling on-device LLM implementation. For instance, post-training scalar LLM model quantization techniques typically allow one moving from 32 bits/weight to 3 bits/weight, which is very good but still far from the 100-1000x target for on-device LLMs. To reach the above ambitious compression rate reduction target, the 3-axis methodology we propose for the Ph.D thesis is as follows. We want and we need to exploit several techniques at the same time and synergize them: 1. We want to determine an efficient representation of the model (or the considered part of it) to be compressed. In particular, we want to exploit our strong expertise in tensor decomposition to represent as well as possible a block of large neural network and exploit properly inter-channel correlations (e.g., between parallel blocks of the neural network). A simple example of such a block is a self-attention head. 2. We want to design aggressive weight quantizers by exploiting: information-theoretic concepts, near-optimal bitwidth allocation rules, side information in a federated/transfer learning setting, and a proper characterization of the range of tasks to be performed by the multi-modal LLM. To implement lossy compression operations applied in the representation space such as weight quantization, we aim to exploit and adapt compression techniques with side information inherited from information theory. Indeed, if the decoder has access to several compressed versions, as is the case in transfer learning, distillation, or federated learning settings, it becomes possible to go much further in terms of compression rates. 3. Revisit LLM fine-tuning by exploiting advanced tensor decompositions (versus basic low-rang matrix product as done currently) and interactions with the upstream operations of the pipeline. Indeed, low-rank tensor decompositions and formats are becoming increasingly popular in this context but it is still unclear how to combine properly these techniques with quantization. To provide a concrete example of limitations of the state-of-the art research: post-training quantization often leaves behind random noise that is spread out (across the model's weights) in a way low-rank adaptation can't easily fix, meaning that low-rank adaptation ends up wasting its limited capacity trying to fix uncorrectable noise instead of improving task performance.

References [Audibert et al., 2023] Audibert, A., Amini, M. R., Usevich, K., and Clausel, M. (2023). Low-rank updates of pre-trained weights for multi-task learning. In Findings of the Association for Computational Linguistics, pages 7544-7554.

[Bershatsky et al., 2024] Bershatsky, D., Cherniuk, D., Daulbaev, T., Mikhalev, A., and Oseledets, I. (2024). Lotr: Low tensor rank weight adaptation. arXiv preprint arXiv:2402.01376.

[Cheng et al., 2017] Cheng, Y., Wang, D., Zhou, P., and Zhang, T. (2017). A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282.

[Cover, 1999] Cover, T. M. (1999). Elements of information theory. John Wiley & Sons.

[Dettmers et al., 2023] Dettmers, T., Pagnoni, A., Holtzman, A., and Zettle- moyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. In Thirty- seventh Conference on Neural Information Processing Systems.

[Frantar et al., 2022] Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2022). GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.

[Hu et al., 2022] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. (2022). Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.

[Li et al., 2023] Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., and Zhao, T. (2023). Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659.

[Li et al., 2025] Li, Z., Xu, B., Shu, X., Zhang, J., Yao, Y., Xie, G.-S., and Tang, J. (2025). Tensor-aggregated lora in federated fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1058- 1067.

[Singh, 2026] Singh, S. (2026). ChatGPT Users Statistics (January 2026) - Growth Usage Data. https://www.demandsage.com/ chatgpt-statistics/#:~:text=We%20will%20look%20at%20ChatGPT' s,queries%20are%20processed%20on%20ChatGPT.

[Vaswani et al., 2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

This PhD offer is provided by the ENACT AI Cluster and its partners. Find all ENACT PhD offers and actions on https://cluster-ia-enact.ai/.