FalconProtein: Finetuning Falcon Foundation Model for Protein Engineering

Large Language Models (LLMs) have demonstrated zero-shot generalization capabilities in analyzing and predicting protein properties through natural language interactions. However, existing protein-focused datasets for LLM fine-tuning, such as ProteinLMDataset with its 17.46 billion tokens for pre-training and 893,000 instructions for fine-tuning, face limitations. Specifically, they include insufficient coverage of protein functional properties, inadequate protein-protein interaction data, and limited integration of contextual information from biomedical literature. To overcome these challenges, we present ProteinPFAIDataset, which integrates data from UniProt and PubMed, comprising 72.8 million tokens for Supervised Fine-Tuning (SFT). ProteinPFAIDataset encompasses important protein characteristics including enzyme activities, molecular functions, pH dependence, tissue specificity, temperature sensitivity, subunit structure, and disease associations. Additionally, we propose a novel knowledge graph-based approach that incorporates over 300,000 biomedical literature entries, providing rich contextual information about protein functions and interactions. To validate the effectiveness of our dataset, we fine-tuned Falcon2-11B LLM, resulting in a model we call Falcon2-11B-PFAI. The fine-tuned model achieved state-of-the-art performance on ProteinLMBench, improving accuracy from 47.10% to 58.37%. The dataset is available at https://huggingface.co/datasets/xiaorui1/PFAI. The fine-tuned model is available at https://huggingface.co/xiaorui1/PFAI/tree/main.