LLaMA Pro: Progressive LLaMA with Block Expansion (2024)

1 Introduction

The advent of Large Language Models (LLMs) has revolutionized the field of natural language processing, exhibiting remarkable proficiency in a variety of real-world tasksOpenAI (2023); Chowdhery etal. (2023).Despite the versatility, LLMs still fall short in certain domains, for example, programming, mathematics, biomedical, or finance.This limitation impedes the progress of developing generic language agents for broader applications.

Existing works Liu etal. (2023); Li etal. (2023a); Wu etal. (2023b) attempted to improve the multi-faceted capabilities of pre-trained LLMs with tailored data recipes.While feasible, they require substantial computational resources and vast amounts of data, which poses a challenge to the democratization of LLM research.Consequently, another line of research, known as domain-adaptive pretraining, focuses on post-pretraining with domain-specific corporaGururangan etal. (2020). These approaches have demonstrated efficacy in adapting various LLMs to specific domainsRoziere etal. (2023); Azerbayev etal. (2023); Wu etal. (2023b); Xu etal. (2023b),resulting in enhanced performance on downstream domain-specific tasks at a reduced computational cost.

Nonetheless, a considerable obstacle emerges in catastrophic forgettingDeLange etal. (2021). Post-pretraining often leads to a decline in the model’s original general abilities, inhibiting the fine-tuned performance of the model on diverse tasksCheng etal. (2023); Dong etal. (2023). This necessitates a method that can inject domain-specific knowledge into LLMs while preserving their general abilities, thereby enhancing their comprehensive capabilities.

Towards this end, we introduce a simple yet effective post-pretraining method, termed block expansion.We expand the off-the-shelf pre-trained LLM using copied Transformer blocks, as illustrated in Figure 2.The newly added blocks, whose linear layers are zero-initialized to enable identity mapping, are further tuned with only domain-specific corpus while the remaining blocks are frozen.After tuning, the extended pre-trained model excels in both general and domain-specific tasks.

In practice, we extend the pre-trained LLaMA2-7BTouvron etal. (2023) by eight more blocks, yielding LLaMA Pro, a foundation model with 8.3B parameters, and enhanced performance in programming, coding, and reasoning.We pre-train LLaMA Pro’s expanded blocks on 80B tokens using open-source code and math data for 2830 GPU Hours (16 NVIDIA H800 GPUs for about 7 days).We further perform supervised instruction tuning (fully fine-tuning of all the blocks, aka SFT) on LLaMA Pro with approximately 80M tokens, yielding LLaMA Pro - Instruct.It is noted that pre-trained models produced by our block expansion method are well-compatible with the subsequent SFT techniques without specific modification.

2 Related Work

3 Method

3.1 Preliminaries: The LLaMA Block

The LLaMA block consists of a multi-head self-attention (MHSA) mechanism followed by a position-wise feed-forward network (FFN) with residual connections and a Swish-Gated Linear Unit (SwiGLU) operation as Figure3 shows. Given an input $x$, the LLaMA block produces an output $y$ as described by the following equations:

$$\begin{split}x^{\prime}=x+\text{MHSA}(\text{RMSNorm}(x))\\y=x^{\prime}+\text{FFN}(\text{RMSNorm}(x^{\prime}))\end{split}$$

(1)

The input $x$ has a dimension of $n\times d$, where $n$ is the sequence length and $d$ is the hidden size. The output $y$ has the same dimension as the input $x$. The MHSA operation is a crucial component of the transformer, defined as:

$$\text{MHSA}(Q,K,V)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{h})W^{O}$$

(2)

where $Q$, $K$, and $V$ are the query, key, and value matrices, respectively, and $W^{O}$ is the output weight matrix without bias . Each head is computed as:

$$\begin{split}\text{head}_{i}=\text{Attention}(xW^{Q}_{i},xW^{K}_{i},xW^{V}_{i})\\\text{Attention}(Q_{i},K_{i},V_{i})=\text{Softmax}\left(\frac{Q_{i}K_{i}^{T}}{\sqrt{d_{k}}}\right)V_{i}\end{split}$$

(3)

with $W^{Q}_{i}$, $W^{K}_{i}$, and $W^{V}_{i}$ being the corresponding weight matrices for the $i$-th head.

The FFN block in the LLaMA model utilizes the SwiGLU activation function, which can be defined as:

See Also

LLaMA Pro: Progressive LLaMA with Block Expansion (Paper Explained) — Yannic Kilcher

$$\begin{split}\text{SwiGLU}(x,W,V)=\text{SiLU}(xW)\otimes(xV)\\\text{FFN}(x)=\text{SwiGLU}(x,W_{1},W_{2})W_{3}\end{split}$$

(4)

where $\otimes$ denotes element-wise multiplication, $W_{1}$, $W_{2}$, and $W_{3}$ are the weight matrices without bias, $\text{SiLU}(x)=x\otimes\sigma(x)$.

4 Experiments

This section presents our key experimental findings. We begin with experimental settings (described in Sec.4.1), and then verify the effectiveness of block expanded tuning after pretraining (described in Sec.4.2). Next, we give the supervised finetuning (SFT) results (described in Sec.5). Finally, ablation studies of the key design choices are presented (described in Sec.6).

4.2 Pretrain Results

Model	Language Tasks					Math Tasks		Code Tasks		Avg. \bigstrut
	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8K	GSM8K-PoT	HumanEval	MBPP
Pretrained comparison
LLaMA Pro (8B)	54.10	77.94	47.88	39.04	73.95	17.89	25.42	28.66	33.20	44.23 \bigstrut
CrystalCoder (7B)	47.01	71.97	48.78	35.91	67.17	10.77	24.96	28.38	36.38	41.26 \bigstrut
LLaMA2-7B	53.07	78.59	46.87	38.76	74.03	14.48	17.68	13.05	20.09	39.62 \bigstrut
CodeLLaMA-7B	39.93	60.80	31.12	37.82	64.01	5.16	25.20	33.50	41.40	37.66 \bigstrut
StarCoder-15B	30.38	47.93	29.96	41.28	56.12	9.48	25.09	33.63	43.28	35.24 \bigstrut
LLaMA-7B	50.94	77.81	35.69	34.33	71.43	8.04	10.46	10.61	17.04	35.15 \bigstrut
OpenLLaMA-v2-7B	43.69	72.20	41.29	35.54	69.38	3.49	5.46	15.32	12.69	33.23 \bigstrut
Falcon-7B	47.87	78.13	27.79	34.26	72.38	4.62	4.32	9.42	13.39	32.46 \bigstrut
SFT comparison
LLaMA Pro - Instruct	52.30	76.88	52.57	48.80	72.53	43.59	55.61	44.51	37.88	53.85 \bigstrut
LLaMA2-7B-Chat	52.90	78.55	48.32	45.57	71.74	7.35	19.73	14.63	21.60	40.04 \bigstrut
CodeLLaMA-7B-Instruct	36.52	55.44	34.54	41.25	64.56	7.96	34.67	34.80	44.4	39.35 \bigstrut
WizardCoder-Python-7B	41.81	65.06	32.29	36.32	61.72	4.70	17.60	42.07	47.20	38.75\bigstrut
WizardMath-7B	54.10	79.55	45.97	43.65	72.69	2.73	25.57	12.20	18.00	39.38\bigstrut

We evaluate LLaMA Pro’s performance with benchmark datasets from the Open LLM Leaderboard. Furthermore, we incorporate coding benchmark datasets, including HumanEval pass@1 and MBPP pass@1, as well as the math benchmark GSM8K, to provide a comprehensive evaluation. We compare the performance of LLaMA Pro with a selection of state-of-the-art pretrained models that were trained around the same period with similar size. This includes general-purpose pretrained models like LLaMA2 and code-oriented pretrained models like CodeLLaMA. The results are presented in Table3.

The results highlight that LLaMA Pro effectively balances natural language processing and coding capabilities. It not only preserves the general performance of its base model, LLaMA2-7B, but also surpasses it in the average performance of general language tasks. Conversely, CodeLLaMA-7B sacrifices general performance to enhance its code ability. We attribute this improvement to our expansion design, which freezes the initial LLaMA blocks to maintain their capabilities and increases the blocks to accommodate more domain-specific knowledge.

As depicted in Figure4, LLaMA Pro shows robust general performance alongside code performance that is on par with code-oriented LLMs. Situated on the Pareto frontier, LLaMA Pro has undergone fine-tuning with an additional 80B tokens in conjunction with LLaMA2, which more than doubles the code tasks average performance. In contrast, CodeLLaMA is fine-tuned with 500B tokens. LLaMA Pro excels in general performance while maintaining code performance that is competitive with code-oriented LLMs, whether they are trained from scratch, such as StarCoder-15B and CrystalCoder, or fine-tuned like CodeLLaMA-7B.

4.4 Ablation Study

Method	Overall Performance (OP)	Backward Transfer (BWT)
LoRA	37.1	-17.3%
SeqFT	45.5	-14.7%
Block Expansion	46.5	-14.3%

Method	Language Tasks						Law Task	Avg. \bigstrut
	ARC	HellaSwag	MMLU	TruthfulQA	Winogrand	Avg.	Unfair-ToS
Add 1 Block	52.30	77.92	38.62	37.80	73.16	55.96	67.45	61.71 \bigstrut
Add 2 Block	53.16	77.91	39.62	38.92	73.01	56.52	69.57	63.05 \bigstrut
Add 4 Block	52.39	76.92	37.30	40.53	72.22	55.87	71.31	63.59 \bigstrut
Add 8 Block	52.90	76.63	41.74	39.83	72.38	56.70	75.11	65.91 \bigstrut
Add 16 Block	51.88	76.59	41.35	40.13	71.82	56.35	75.17	65.76 \bigstrut
Add 32 Block	50.77	76.72	40.68	41.66	72.77	56.52	73.93	65.23 \bigstrut
Mixture-of-Expert (MoE)	51.45	76.51	42.47	40.13	72.23	56.56	67.27	61.92 \bigstrut
Prefix Stacking (8 Block)	27.82	26.12	23.12	22.52	47.20	29.36	0.81	15.08 \bigstrut
Suffix Stacking (8 Block)	52.56	77.89	39.10	39.03	72.38	56.19	60.98	58.59 \bigstrut

We evaluate various training strategies, including LoRA, fine-tuning, and the block expansion training approach that we propose, using the TRACE benchmarkWang etal. (2023b). TRACE is designed to assess continual learning in LLMs and comprises eight distinct datasets that span challenging tasks such as domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning. We assess the ability of different strategies to retain the model’s existing knowledge while incorporating new skills. Details are provided in the Appendix 13.

We employ Overall Performance (OPChaudhry etal. (2018)) and Backward Transfer (BWTLopez-Paz and Ranzato (2017)) scores as evaluation metrics. After incrementally learning the $t$-th task, the model’s score on the $i$-th task (where $i\leq t$) is denoted as $R^{D}_{t,i}$. The OP and BWT scores are calculated using the following formulas:

$$OP_{t}=\frac{1}{t}\sum_{i=1}^{t}R_{t,i}^{D}$$

(6)

$$BWT_{t}=\frac{1}{t}\sum_{i=1}^{t}\left(R_{t,i}^{D}-R_{i,i}^{D}\right)$$

(7)

Table6 presents the performance of different strategies on the TRACE benchmark following their continual learning phase with LLaMA2-7B. The results show that block expansion training exhibits superior task-specific adaptability compared to sequential fine-tuning and LoRA, as evidenced by its better OP and BWT scores.

Apart from the aspect of code corpus, we explore our method on another domain: law, with the freelaw subset of Pile dataset as our pretrain corpus Gao etal. (2020). We evaluate on UNFAIR-ToS Lippi etal. (2019) of the LexGLUE benchmark Chalkidis etal. (2021). The details can be found in the Appendix 14.

In our experiment, we assess the scalability of our block expansion method in terms of training loss and downstream task performance as we increase the number of added blocks. We also compare our method with the Mixture-of-Expert (MoE) expansion methodFedus etal. (2022).

We first examine the training loss with varying added blocks. As seen in Figure5, the training loss of the models consistently decreases as training progresses, regardless of the number of added blocks. Moreover, the loss decreases more rapidly as we increase the size of the model. These findings suggest that our method exhibits strong scalability with larger models and more data. The training loss of MoE is comparable to our method with four added blocks.

However, a lower overall training loss does not necessarily guarantee superior performance on domain-specific tasks. Therefore, we evaluate models of different sizes on both general language tasks and Unfair-ToS, as shown in Table7. All the expanded models effectively preserve the general capabilities of the initial model. For the domain-specific task, larger models achieve better performance. We find that adding eight blocks provides optimal performance with minimal cost compared to larger models, hence we adopt this as our default strategy.

We also analyze the impact of the position where the identity blocks are added, either at the bottom or the top of the model, compared to adding them interleaved, as shown in Table7. We observe that adding blocks at the bottom results in poor evaluation performance, likely because it disrupts the model’s foundation, causing errors to propagate throughout the model. Adding blocks at the top of the modelGong etal. (2019) preserves the initial model’s performance, but its performance on domain-specific tasks is lower than when adding blocks interleaved.

As highlighted in the LIMA studyZhou etal. (2023), the majority of knowledge in large language models is acquired during pretraining, with only a limited amount of instruction tuning data required to generate high-quality output. To investigate the extent of knowledge encoded during pretraining, we conducted a comparative analysis between LLaMA2-7B and LLaMA Pro using the same instruction dataset, as illustrated in Figure 6. Our results showed that LLaMA Pro consistently outperforms LLaMA2-7B across all tasks, indicating that our method effectively enables LLaMA Pro to encode more domain-specific knowledge during the pretraining phase.

5 Conclusion

In this study, we introduced a novel block expansion method for Large Language Models (LLMs) post-pretraining, aiming to enhance domain-specific abilities while preserving the original general capabilities. Our approach effectively balances the model’s performance across both general and domain-specific tasks. We demonstrated the effectiveness of our method through LLaMA Pro, an LLM initialized from LLaMA2-7B with 8 added blocks, which outperformed other LLaMA-series models on comprehensive benchmarks.

The work highlights the importance of balancing general and domain-specific abilities in LLMs and offers a promising approach to achieving this balance.Future research could explore broader applications of our block expansion method in other domains, for instance, it is an important task for multimodal large language models Ge etal. (2023); Bai etal. (2023) to preserve the original language ability.

References

• Austin etal. (2021)Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, etal. 2021.Program synthesis with large language models.arXiv preprint arXiv:2108.07732.
• Azerbayev etal. (2023)Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, MarcoDos Santos, Stephen McAleer, AlbertQ Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023.Llemma: An open language model for mathematics.arXiv preprint arXiv:2310.10631.
• Bai etal. (2023)Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023.Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966.
• Bai etal. (2022)Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, etal. 2022.Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862.
• Brown etal. (2020)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, JaredD Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, etal. 2020.Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901.
• Chalkidis etal. (2021)Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, DanielMartin Katz, and Nikolaos Aletras. 2021.Lexglue: A benchmark dataset for legal language understanding in english.arXiv preprint arXiv:2110.00976.
• Chaudhry etal. (2018)Arslan Chaudhry, PuneetK Dokania, Thalaiyasingam Ajanthan, and PhilipHS Torr. 2018.Riemannian walk for incremental learning: Understanding forgetting and intransigence.In Proceedings of the European conference on computer vision (ECCV), pages 532–547.
• Chen etal. (2021a)Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. 2021a.bert2bert: Towards reusable pretrained language models.arXiv preprint arXiv:2110.07143.
• Chen etal. (2021b)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde deOliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, etal. 2021b.Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374.
• Chen etal. (2023a)Wenhu Chen, Xueguang Ma, Xinyi Wang, and WilliamW. Cohen. 2023a.Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.Transactions on Machine Learning Research.
• Chen etal. (2023b)Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. 2023b.Theoremqa: A theorem-driven question answering dataset.arXiv preprint arXiv:2305.12524.
• Cheng etal. (2023)Daixuan Cheng, Shaohan Huang, and Furu Wei. 2023.Adapting large language models via reading comprehension.arXiv preprint arXiv:2309.09530.
• Chowdhery etal. (2023)Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, HyungWon Chung, Charles Sutton, Sebastian Gehrmann, etal. 2023.Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113.
• Clark etal. (2018)Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457.
• Cobbe etal. (2021)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, etal. 2021.Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168.
• DeLange etal. (2021)Matthias DeLange, Rahaf Aljundi, Marc Masana, Sarah Parisot, XuJia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021.A continual learning survey: Defying forgetting in classification tasks.IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385.
• Dong etal. (2023)Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2023.How abilities in large language models are affected by supervised fine-tuning data composition.arXiv preprint arXiv:2310.05492.
• Fedus etal. (2022)William Fedus, Barret Zoph, and Noam Shazeer. 2022.Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.The Journal of Machine Learning Research, 23(1):5232–5270.
• Gao etal. (2020)Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, etal. 2020.The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027.
• Ge etal. (2023)Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. 2023.Making llama see and draw with seed tokenizer.arXiv preprint arXiv:2310.01218.
• Gong etal. (2019)Linyuan Gong, DiHe, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. 2019.Efficient training of bert by progressively stacking.In International conference on machine learning, pages 2337–2346. PMLR.
• Gu etal. (2020)Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. 2020.On the transformer growth for progressive bert training.arXiv preprint arXiv:2010.12562.
• Gupta etal. (2023)Ksh*tij Gupta, Benjamin Thérien, Adam Ibrahim, MatsL Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023.Continual pre-training of large language models: How to (re) warm your model?arXiv preprint arXiv:2308.04014.
• Gururangan etal. (2020)Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, IzBeltagy, Doug Downey, and NoahA Smith. 2020.Don’t stop pretraining: Adapt language models to domains and tasks.arXiv preprint arXiv:2004.10964.
• Hendrycks etal. (2020)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300.
• Hendrycks etal. (2021)Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.Measuring mathematical problem solving with the math dataset.NeurIPS.
• Hoffmann etal. (2022)Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego deLas Casas, LisaAnne Hendricks, Johannes Welbl, Aidan Clark, etal. 2022.Training compute-optimal large language models.arXiv preprint arXiv:2203.15556.
• Houlsby etal. (2019)Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin DeLaroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.Parameter-efficient transfer learning for nlp.In International Conference on Machine Learning, pages 2790–2799. PMLR.
• Hu etal. (2021)EdwardJ Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, LuWang, and Weizhu Chen. 2021.Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685.
• Kaplan etal. (2020)Jared Kaplan, Sam McCandlish, Tom Henighan, TomB Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361.
• Li etal. (2023a)Raymond Li, LoubnaBen Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, etal. 2023a.Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161.
• Li etal. (2023b)Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, LiDu, Bowen Qin, etal. 2023b.Flm-101b: An open llm and how to train it with $100 k budget.arXiv preprint arXiv:2309.03852.
• Lian etal. (2023)Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023.Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification.
• Lin etal. (2021)Stephanie Lin, Jacob Hilton, and Owain Evans. 2021.Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958.
• Lippi etal. (2019)Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni. 2019.Claudette: an automated detector of potentially unfair clauses in online terms of service.Artificial Intelligence and Law, 27:117–139.
• Liu etal. (2023)Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, etal. 2023.Llm360: Towards fully transparent open-source llms.arXiv preprint arXiv:2312.06550.
• Lopez-Paz and Ranzato (2017)David Lopez-Paz and Marc’Aurelio Ranzato. 2017.Gradient episodic memory for continual learning.Advances in neural information processing systems, 30.
• Luo etal. (2023)Ziyang Luo, Can Xu, PuZhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023.Wizardcoder: Empowering code large language models with evol-instruct.
• OpenAI (2023)OpenAI. 2023.Gpt-4 technical report.arXiv preprint arXiv:2303.08774.
• Ouyang etal. (2022)Long Ouyang, Jeffrey Wu, XuJiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, etal. 2022.Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744.
• Radford etal. (2019)Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, etal. 2019.Language models are unsupervised multitask learners.OpenAI blog, 1(8):9.
• Roziere etal. (2023)Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, XiaoqingEllen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, etal. 2023.Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950.
• Sakaguchi etal. (2021)Keisuke Sakaguchi, RonanLe Bras, Chandra Bhagavatula, and Yejin Choi. 2021.Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106.
• Sanh etal. (2021)Victor Sanh, Albert Webson, Colin Raffel, StephenH Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, TevenLe Scao, Arun Raja, etal. 2021.Multitask prompted training enables zero-shot task generalization.arXiv preprint arXiv:2110.08207.
• Scialom etal. (2022)Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022.Fine-tuned language models are continual learners.In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107–6122.
• Shen etal. (2022)Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and IzBeltagy. 2022.Staged training for transformer language models.In International Conference on Machine Learning, pages 19893–19908. PMLR.
• Shridhar etal. (2020)Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020.Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768.
• Tenney etal. (2019)Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.Bert rediscovers the classical nlp pipeline.arXiv preprint arXiv:1905.05950.
• Touvron etal. (2023)Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, etal. 2023.Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288.
• VanAken etal. (2019)Betty VanAken, Benjamin Winter, Alexander Löser, and FelixA Gers. 2019.How does bert answer questions? a layer-wise analysis of transformer representations.In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1823–1832.
• Wang etal. (2023a)Peihao Wang, Rameswar Panda, LucasTorroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, DavidDaniel Cox, Zhangyang Wang, and Yoon Kim. 2023a.Learning to grow pretrained models for efficient transformer training.arXiv preprint arXiv:2303.00980.
• Wang etal. (2023b)Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, etal. 2023b.Trace: A comprehensive benchmark for continual learning in large language models.arXiv preprint arXiv:2310.06762.
• Wang etal. (2023c)Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023c.Mint: Evaluating llms in multi-turn interaction with tools and language feedback.
• Wang etal. (2023d)Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, KhyathiRaghavi Chandu, David Wadden, Kelsey MacMillan, NoahA. Smith, IzBeltagy, and Hannaneh Hajishirzi. 2023d.How far can camels go? exploring the state of instruction tuning on open resources.
• Wei etal. (2021)Jason Wei, Maarten Bosma, VincentY Zhao, Kelvin Guu, AdamsWei Yu, Brian Lester, Nan Du, AndrewM Dai, and QuocV Le. 2021.Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652.
• Wu etal. (2023a)Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. 2023a.$\pi$-tuning: Transferring multimodal foundation models with optimal multi-task interpolation.In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 37713–37727. PMLR.
• Wu etal. (2023b)Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023b.Bloomberggpt: A large language model for finance.arXiv preprint arXiv:2303.17564.
• Xu etal. (2023a)Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, PuZhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023a.Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244.
• Xu etal. (2023b)Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, etal. 2023b.Lemur: Harmonizing natural language and code for language agents.arXiv preprint arXiv:2310.06830.
• Yang etal. (2018)Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, WilliamW Cohen, Ruslan Salakhutdinov, and ChristopherD Manning. 2018.Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600.
• Yao etal. (2022)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629.
• Yao etal. (2023)Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. 2023.2x faster language model pre-training via masked structural growth.arXiv preprint arXiv:2305.02869.
• Yu etal. (2023)Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, YuZhang, JamesT Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023.Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284.
• Zellers etal. (2019)Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830.
• Zhang etal. (2023)Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023.Adding conditional control to text-to-image diffusion models.In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847.
• Zheng etal. (2023)Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, ZiLin, Zhuohan Li, Dacheng Li, Eric Xing, etal. 2023.Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685.
• Zhou etal. (2023)Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, etal. 2023.Lima: Less is more for alignment.arXiv preprint arXiv:2305.11206.
• Ziegler etal. (2019)DanielM Ziegler, Nisan Stiennon, Jeffrey Wu, TomB Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593.

Appendix A Gradient Derivation

To calculate the gradient of the RMSNorm weight during backpropagation, we first need to consider the forward pass equation for the Llama RMSNorm:

where $x$ is the input tensor, $w$ is the weight parameter, $\text{Var}(x)$ is the variance of $x$ across the last dimension, and $\epsilon$ is a small constant for numerical stability.

Now, let’s consider the chain rule for the gradient of the loss function with respect to the RMSNorm weight during backpropagation. Denote the loss function as $L$, and the output of the FFN as $y$. We have:

To compute the gradient, we need to find the partial derivative $\frac{\partial y}{\partial w}$. From the FFN equation, we have:

Taking the derivative with respect to $w$, we get:

Now, let’s differentiate the RMSNorm function with respect to $w$:

Using the chain rule, we can compute the gradient of the loss function with respect to the RMSNorm weight:

Now we want to find the partial derivative of the FFN with respect to $t$. Recall the SwiGLU activation function:

Taking the derivative of the SwiGLU function with respect to $t$, we get:

Now, recall the SiLU activation function:

Thus, the gradient of the FFN with respect to $t$ when $t=0$ is also zero:

In conclusion, when $t=0$, the gradient of the FFN with respect to $t$ is zero, which demonstrates that the gradient is zero when the input to the FFN is zero.

Appendix B MINT-Bench

The MINT-BenchWang etal. (2023c) details are provided in this section. MINT-Bench comprises eight datasets spanning code generation, decision-making, and reasoning tasks, totaling 586 instances, as shown in Table 8.

We use the Success Rate (SR) as our evaluation metric, which measures the percentage of successful task instances. For an interaction limit of $k$, MINT-Bench starts from scratch and allows each LLM to interact up to the $k$-th turn, measuring the corresponding $SR_{k}$. Unless specified otherwise, MINT-Bench limits $k\in\left[1,5\right]$, where $k=1$ indicates no interaction, and $k=5$ maximizes interaction turns within the context window (4,096 tokens) of most modern LLMs.

In each turn, the LLM is instructed to perform the following steps: (1) Optionally express its reasoning process (referred to as "Thought," similar to Yao etal. (2022)); (2) Either interact with tools by generating Python code and executing it through a Python interpreter (referred to as "Execute"), or propose a solution to the user (referred to as "Propose Solution").

Table 9 displays the success rate for each model evaluated on various task type benchmarks, as well as the micro average when $k=5$. The LLaMA Pro - Instruct model demonstrates robust performance across all task types compared to other models of similar size. Figure 7 provides a case study to compare LLaMA Pro - Instruct and LLaMA2-7B-Chat where LLaMA Pro - Instruct successfully utilizes Python program to solve the given question in the multi-round interaction.

Appendix C MT-Bench

MT-bench is a collection of demanding multi-turn open-ended questions designed for evaluating chat assistants. In order to automate the evaluation process, we employ powerful LLMs, such as GPT-4, to act as judges and assess the quality of the models’ responses. We present the detailed pairwise comparison in the Figure 8 and Figure 9. Figure 10 shows the case study of the comparison between LLaMA Pro - Instruct and LLaMA2-7B-Chat.

Appendix D TRACE-Bench

Table 10 shows the statistics of TRACE. We evaluate the performance of different training strategies after sequential tasks. Table 11-13 shows the detailed performance of different training strategies of each round during the continual learning with LLaMA2-7B.

Appendix E Domain of Law

Table 14 shows the hyper-parameters we use to do the ablation study in the domain of law. We use the freelaw subset of Pile dataset as our pretrain corpus Gao etal. (2020) in the domain of law. This subset has 51.2 GiB raw size and 16.7B tokens with 3.6M documents.

The Unfair-ToS dataset, which we use to evaluate the performance of law, contains Terms of Service (ToS) from online platforms (e.g., YouTube, Ebay, Facebook, etc.). The dataset has been annotated on the sentence-level with 8 types of unfair contractual terms (sentences), meaning terms that potentially violate user rights according to the European consumer law. The UNFAIR-ToS task is a multilabel classification task. To get model predictions for this task, we categorize it as a multiple-choice question as the method Cheng etal. (2023) uses. The accuracy of an individual data example is considered true if the model prediction (i.e., the option with the highest per-token likelihood) belongs to the label(s) set. We evaluate the Unfair-ToS dataset in a 4-shot scenario just like Cheng etal. (2023).