Optimization-driven parameter sharing via neural architecture search for transformer models in machine translation

Abstract

Parameter sharing offers an effective mechanism for improving the efficiency of Transformer models, particularly in low-resource settings. However, existing approaches typically rely on manual design choices or heuristic rules, which become increasingly difficult to manage as model scale and language coverage grow. We introduce a framework that reformulates parameter sharing as a neural architecture search problem guided by Bayesian optimization. Instead of manually specifying sharing configurations, the proposed data-driven approach automatically discovers suitable parameter-sharing structures through optimization. Our formulation treats sharing patterns as architectural variables rather than tunable training parameters of conventional hyperparameter optimization. The framework defines parameter sharing within a formal optimization setting characterized by explicit decision variables, objectives, and constraints. These variables encode layer-sharing configurations for bilingual models and language-to-decoder mapping for multilingual models, enabling systematic exploration of architectural alternatives. A dedicated algorithm is introduced to generate candidate sharing structures and evaluate them efficiently within the search process. We evaluate the proposed framework and all baselines using a consistent setup, using a pretrained Transformer-based machine translation model. In bilingual translation, optimization-driven configurations substantially outperform baseline strategies, yielding notable improvements in translation quality over both training-from-scratch sharing strategies (improving BLEU scores by 13.42) and competitive with those in pretrained settings. In multilingual scenarios, the framework identifies architectures that effectively balance performance and computational efficiency. The proposed framework provides a principled mechanism, rather than tedious heuristics, for automating parameter structure design decisions in Transformer-based models with potential applicability to larger-scale systems.