Protein Sequence Analysis landscape: A Systematic Review of Task Types, Databases, Datasets, Word Embeddings Methods, and Language Models Open Access

Abstract

Protein sequence analysis examines the order of amino acids within protein sequences to unlock diverse types of a wealth of knowledge about biological processes and genetic disorders. It helps in forecasting disease susceptibility by finding unique protein signatures, or biomarkers that are linked to particular disease states. Protein Sequence analysis through wet-lab experiments is expensive, time-consuming and error prone. To facilitate large-scale proteomics sequence analysis, the biological community is striving for utilizing AI competence for transitioning from wet-lab to computer aided applications. However, Proteomics and AI are two distinct fields and development of AI-driven protein sequence analysis applications requires knowledge of both domains. To bridge the gap between both fields, various review articles have been written. However, these articles focus revolves around few individual tasks or specific applications rather than providing a comprehensive overview about wide tasks and applications. Following the need of a comprehensive literature that presents a holistic view of wide array of tasks and applications, contributions of this manuscript are manifold: It bridges the gap between Proteomics and AI fields by presenting a comprehensive array of AI-driven applications for 63 distinct protein sequence analysis tasks. It equips AI researchers by facilitating biological foundations of 63 protein sequence analysis tasks. It enhances development of AI-driven protein sequence analysis applications by providing comprehensive details of 68 protein databases. It presents a rich data landscape, encompassing 627 benchmark datasets of 63 diverse protein sequence analysis tasks. It highlights the utilization of 25 unique word embedding methods and 13 language models in AI-driven protein sequence analysis applications. It accelerates the development of AI-driven applications by facilitating current state-of-the-art performances across 63 protein sequence analysis tasks.

Introduction

Protein sequence analysis is a scientific way to utilize diverse types of strategies for examining the order of amino acids within protein sequences. This analysis objective is to unlock diverse types of a wealth of knowledge about biological processes and genetic disorders (1). Researchers are gaining deep understanding about biological processes in which proteins are involved, such as enzyme activity (2), cell signalling (3), and immune responses (4). Researchers are also gaining understanding about genetic disorders by pinpointing mutations that alter proteins functionalities (5). It helps in forecasting diseases susceptibility by finding unique protein signatures, or biomarkers that are linked to particular disease states (6). Specifically, this analysis enables researchers to identify individuals at higher risk for developing certain diseases before symptoms even appear. Protein sequence analysis process through wet-lab experiments requires significant costs due to involvement of specialized chemicals and equipment (7). The process demands extensive time commitments due to the necessity of lengthy protocols and extended periods for biological growth (7). Additionally, even the most skilled scientists can introduce errors when conducting large-scale experiments, whether in adhering to protocols, selecting appropriate chemicals, or managing external conditions such as temperature and equipment sanitation (7). These factors collectively contribute to the high expense, time consumption, and potential for error in traditional wet-lab protein sequence analysis (7).

Advancements in next-generation sequencing technologies have generated an enormous volume of protein sequence data that is accessible in public databases (8). The vast availability of publicly accessible data has enable large-scale protein sequence analysis by shifting from traditional wet lab experimental methods to AI-driven protein sequence analysis applications (9). Despite noteworthy achievements in development of AI-driven protein sequence analysis applications, there remains a significant room for further development of more powerful AI-driven applications (10). A primary reason behind sub-optimal predictive performance of AI-driven applications is their limited ability to effectively extract meaningful patterns from proteins sequences that are made up from repetitive patterns of 20 unique amino acids (11). Specifically, the repetitive patterns of amino acids within protein sequences encompasses a wealth of information such as protein modifications (12–22), sub-cellular localization (23–34), protein–protein interactions (35–46), and protein–virus interactions (47–52). To provide a high level overview about working paradigm of AI-driven protein sequence analysis applications across various tasks, these applications can be broadly categorized into three distinct classes: classification (13, 53–60), regression (61–64), and clustering (65). Classification applications objective is to assign protein sequences into predefined classes based on specific patterns of amino acids within protein sequences (54–57, 66). Regression applications objective is to predict a continuous numerical value based on specific patterns of amino acids within protein sequences (61–64, 66). Clustering applications groups similar protein sequences together into clusters or make groups based on their inherent similarities in sequences (66).

Figure 1 graphically represents a generalized AI-driven protein sequence analysis pipeline for diverse types of tasks that fall under three fundamental AI paradigms: classification, regression, and clustering. A high-level examination of Figure 1 reveals that the initial step in development of AI-driven protein sequence analysis application requires benchmark dataset. This dataset is usually developed by acquiring protein sequences and corresponding biological information from two primary sources: wet-lab experiments (7), public databases (67). In the next step, raw protein sequences are transformed into statistical vectors because AI algorithms have inherent dependency over statistical vectors. AI algorithms are then trained using a training set comprising of protein sequences statistical vectors along with associated numerical values or predefined classes. The trained models are subsequently evaluated on a test set by comparing their predicted labels with actual labels. Finally, a web application is developed to utilize the trained models for practical use.

The overall performance of AI-driven protein sequence analysis predictive pipelines is entirely dependent on the quality of the statistical vectors used to represent protein sequences (68). When these vectors effectively capture informative patterns of amino acids in the protein sequences, simple machine learning algorithms can achieve good performance (68). Conversely, complex algorithms may underperform if presented with random statistical vectors lacking these informative patterns (69). To convert protein sequences into statistical vectors by extracting and encoding various amino acid patterns, researchers have developed over 100 encoding methods (41). These methods either capture the positional information of amino acids within protein sequences or utilize the physicochemical properties of amino acids to capture covariance and correlation information. Considering the similarities between protein sequences and textual data, and following the success of word embedding methods and LLMs in capturing and encoding diverse patterns into statistical vectors for various NLP tasks, researchers are harnessing these methods for development of AI-driven protein sequence analysis applications. AI-driven protein sequence analysis realm has witnessed many review articles focused on exploration of domain-specific encoding methods. However, comprehensive literature on utilization of word embedding methods and LLMs is not available. A thorough review of these methods would significantly benefit the research community by highlighting the potential of these powerful NLP methods in development of AI-driven protein sequence analysis applications. With an aim to explore protein sequence analysis realm at large scale and to present integration of word embedding methods and LLMs into AI-driven protein sequence analysis applications for transformative discoveries the contributions of this manuscript are manifold:

• It bridges the knowledge gap between Proteomics and Artificial Intelligence fields. Proteomics scientists can utilize this review article to gain insights about AI potential in the realm of protein sequence analysis, while AI researchers can gain a deeper understanding about protein sequence analysis tasks biological foundations, challenges and opportunities for development of AI-driven protein sequence analysis applications.

• It equips AI researchers with a foundational understanding of 63 distinct protein sequence analysis tasks. To provide a comprehensive overview of proteomic sequence analysis landscape, we categorised 63 tasks into 11 major goals namely Protein Identification, Properties Prediction, function and structure prediction, Modification prediction, Interaction Prediction, Sub-cellular location prediction, Peptide Analysis, Gene Analysis, Mutation Analysis, Disease Analysis, and Drug Analysis.

• It enhances development of AI-driven protein sequence analysis applications by providing cruxes of 68 different biological databases that have been utilized for development of 63 distinct tasks related benchmark datasets. A comprehensive knowledge of these databases can be utilized to develop new benchmark datasets.

• It explains the nature of 63 unique Protein sequence analysis tasks and classify them into 2 primary categories: regression, classification, as well as three secondary categories: binary classification, multi-class classification, and multi-label classification. Protein sequence analysis tasks mapping into core AI tasks will gain interest of AI researchers for development of more effective and specialized AI-driven protein sequence analysis applications at large scale.

• To ensure a fair performance comparison between existing and new AI predictors, it provides the details of 627 benchmark datasets related to 63 unique protein sequence analysis tasks.

• It demonstrates the utilization of 25 unique word embedding methods and 13 language models in AI-driven protein sequence analysis applications for 63 unique protein sequence analysis tasks.

• To facilitate development of new predictors, this review provides a detailed summary of current state-of-the-art predictors, their performances across 63 unique protein sequence analysis tasks, and their availability to scientific community.

Research methodology

Figure 2 illustrates a two stage process for search and selection of most relevant articles related to word embeddings and language models applications in the realm of protein sequence analysis. Following sub-sections summarize the details of two stages: (1) article searching and (2) article screening and filtering.

Article searching

This stage formulates quality search queries to search articles related to word embeddings and language models applications in proteins sequence analysis landscape. Figure 2 demonstrates that article identification module comprises of three different keywords cells namely; Protein analysis tasks, word embedding methods and Language models.Our search encompasses articles published from 2016 to September 2024, a period that captures the evolution of word embeddings and language models in protein sequence analysis. Given the dynamic nature of this field, particularly regarding language models, and the broader scope of this review covering 63 distinct tasks, it is impractical to include all papers that emerge on a regular basis. We have explicitly mentioned this timeline to help readers understand the temporal scope of our analysis and effectively drive the research forward. To design search queries, keywords within the same cell are combined using OR |$\lor$| operator while keywords across different cells are concatenated using AND |$\land$| operator. For instance sample search queries are protein family classification using FastText word embedding, protein–protein interaction prediction using BERT language model, etc. These search queries are executed on distinct search engines including Google Scholar (https://scholar.google.com/), ACM Digital Library (https://dl.acm.org/), IEEEXplore (https://ieeexplore.ieee.org/), Elsevier (https://www.elsevier.com/), Wiley Online Library (https://www.wiley.com/en-us), Springer (https://www.springer.com/gp) and ScienceDirect (https://www.sciencedirect.com/). Furthermore, snowballing is employed to investigate articles cited in extracted papers to identify more research articles. Execution of queries across multiple academic databases acquired 178 word embedding and 203 language models based research articles which are screened and filtered in second stage.

Article screening and filtering

In second phase, most relevant articles are selected through a two-step process. Firstly, titles and abstracts of 559 word embedding and language models based articles are reviewed by domain experts. This review yielded 130 word embedding and 160 language models based relevant articles. Subsequent, full-text assessment of these articles identified 98 articles focused on word embedding and 137 articles relevant to language models in protein sequence analysis.

Biological foundations of protein sequence analysis goals and tasks

Proteins sequences comprise repetitive patterns of 20 unique amino acids whose arrangements represent diverse types of information such as protein’s structure, function, and interactions. Irregularities or mutations in these arrangements can lead to various biological disorders and diseases such as cystic fibrosis, sickle cell anaemia, Huntington’s disease, Tay-Sachs disease, and different forms of cancer. With an aim to understand roles of proteins in diverse types of biological functions, and their associations with genetic disorders and diseases, researchers are exploring the realm of proteins from various perspectives. We have categorized 63 distinct protein sequence analysis tasks into 11 distinct biological goals namely Protein Identification, Properties Prediction, function and structure prediction, Modification prediction, Interaction Prediction, Sub-cellular location prediction, Peptide Analysis, Gene Analysis, Mutation Analysis, Disease Analysis, and Drug Analysis. A graphical illustration of all 11 goals and their associated tasks is shown in Figure 3. Living organisms contain millions of proteins in simple cells and billions in complex organisms. To thoroughly explore the distinct functionalities and properties of proteins considering the fact proteins within the same family share similar characteristics, scientists study them at family level rather than individually. This family-level exploration and analysis require proteins classification into various families such as kinases phosphatases, G-protein coupled receptors, immunoglobulins heat shock proteins, cytochromes proteases, transcription factors, transporters, and structural proteins. Protein family classification facilitates valuable clues about the structure and function of uncharacteristic proteins in the cell on the basis of known structures and functions of family members. Furthermore researchers are identifying proteins with unique functionalities and properties such as identification of essential proteins, SNARE proteins, electron transport proteins, G-protein coupled receptors, and intrinsically disordered proteins. These proteins actively participate in fundamental biological processes and pathways where their dysfunction can severely hamper proper functioning of living organisms.

Within protein identification landscape, protein similarity prediction groups proteins into clusters based on the distribution of amino acids in their sequences. This approach aids in discovering new classes of proteins with unique characteristics. Moreover protein similarity prediction facilitates the annotation of newly sequenced proteins by inferring functions based on similarity to known proteins thereby accelerating the discovery of biological pathways and processes. While traditional methods like profile searching have been effective for protein annotation and similarity prediction, AI approaches have become increasingly critical due to several factors. The exponential growth in protein sequence databases has made manual and traditional computational analyses impractical. AI methods can handle this scale while detecting subtle patterns in amino acid distributions that might be missed by conventional algorithms. Additionally, AI’s ability to integrate multiple features simultaneously, from sequence patterns to evolutionary information, enables more accurate predictions, especially for proteins with low sequence similarity to known groups. These advantages are particularly valuable when dealing with novel proteins or those from understudied organisms, where traditional profile-based methods might fail to detect important functional relationships. For the pharmaceutical industry, protein similarity prediction is invaluable in drug discovery and development where identifying proteins similar to known drug targets can reveal new therapeutic opportunities. Protein properties prediction encompasses various characteristics including solubility, stability, thermophilicity, fluorescence, and domain boundaries which provide crucial insights into protein behaviour and potential applications. Protein solubility refers to the ability of a protein to dissolve and remain in solution without aggregating or precipitating out where various factors such as amino acid composition hydrophobicity and interactions of different molecules within cellular environment influence this property. Solubility prediction assists scientists to identify proteins that are more prone to misfold or aggregate to enable them to comprehend underlying mechanisms of complex diseases and develop potential interventions. Protein stability and thermophilicity predictions facilitate researchers to engineer proteins with improved stability for various applications such as enzymes for industrial processes, therapeutic proteins or high-temperature environments. Protein fluorescence prediction aids in studying protein interactions and designing biosensors while domain boundaries prediction helps in understanding the modular nature of proteins protein engineering and functional annotation.

Function and structure prediction focuses on determining proteins roles and three-dimensional conformations which are essential for understanding their behaviour in cellular environments. Within cellular environment proteins act as essential workhorses where each protein possesses a unique function such as enzyme activity or structural support and structure such as α-helix, β-strand, and turns. Proteins functions hold valuable information about biological activities such as catalysing biochemical reactions providing structural support and facilitating cellular communication and transport. Structure prediction reveals how a protein might interact with other molecules where comprehensive information about function and structure of a protein is useful for understanding cellular machines working paradigm in cellular environment. Remote homology detection and fold prediction further enhance our understanding of protein evolution and potential functional relationships. The protein interactions landscape offers valuable insights about how proteins work together within biological systems interactions role in cellular communication and how irregularities in interactions contribute to disease mechanisms and affect biological processes. Protein interactions are the cornerstone of nearly all cellular processes because they mediate signalling pathways genetic expression and cellular machinery functions. The landscape includes interactions between proteins and various molecules including DNA/genes (70), viruses, RNAs (71), and compounds. Each type of interaction prediction yields unique insights. Like protein–gene interactions are pivotal in understanding gene regulation and expression process, protein–protein interactions enable understanding of proteins dynamic roles in various biological processes, and virus-host protein interactions shed light on how viruses hijack host’s cellular machinery.

Protein modification prediction is important for understanding protein’s functional landscape including its stability and activity in various biological processes and diseases. Proteins undergo diverse kinds of post-translational modifications including methylation glycosylation acetylation phosphorylation and ubiquitination which modify various properties of proteins such as their structure electrophilicity and interactive capacity that enable them to take part and regulate variety of cellular processes. With over 200 diverse types of post-translational modifications, researchers are putting efforts to identify S-sulfenylation and glutarylation sites in proteins which play key roles in signal transduction regulation of protein activity function and interactions in cellular environment. Scientists are performing protein sub-cellular localization prediction to gain insights into proteins roles in different cellular compartments. Primarily proteins core biological activities are strictly linked with their presence in different cellular compartments such as cell junction, cell membrane, cell projection, cytoplasm, Golgi apparatus, lysosome, mitochondrion, nucleus, secreted endoplasmic reticulum, plastid extracellular signal chloroplast lysosome/vacuole and peroxisome. Within a cell different compartments provide distinct microenvironments where proteins perform specialized functions such as metabolism related activities in mitochondria and DNA replication process in nucleus. Proteins are built from small building blocks called peptides which researchers have categorized based on diverse types of properties like stability bio-availability efficacy action mechanisms involvement in cellular processes utilization in drugs and therapies. The most well-characterized and diverse peptide types include anticancer peptides antibacterial peptides antimicrobial peptides signal peptides secreted Peptides and anti-inflammatory peptides. Each peptide type offers unique therapeutic potential where anti-inflammatory peptides modulate immune system responses anti-cancer peptides target cancer cells through various mechanisms and antibacterial peptides provide alternatives to traditional antibiotics. In the realm of disease analysis protein-centric investigations focus on parasite identification characterization of immune factors viral classification and analysis of protein sequences associated with specific pathological conditions. Researchers are identifying compounds that prevent or reverse disease progression by targeting responsible pathways. Furthermore identification of disease related proteins compounds and inhibitors are accelerating drug repurposing which offers rapid response to emerging diseases while providing immediate treatment options. Gene analysis primarily focuses on gene phenotype prediction disease genes prediction essential genes identification and prokaryotic gene prediction. These genes often lead to chronic diseases such as sickle cell anaemia multiple sclerosis Huntington’s disease type 2 diabetes heart disease and many forms of cancer. Understanding gene phenotypes and essential genes is fundamental for advancing medical research and developing targeted therapies. Mutation analysis encompasses prediction of mutations their effects and variant impacts providing insights into disease mechanisms and treatment responses. This knowledge empowers researchers to elucidate molecular mechanisms of disease and develop targeted therapeutic strategies. Within drug analysis landscape protein–drug interaction and binding affinity prediction streamlines drug development process and facilitates development of personalized medicine approaches.

A look on protein sequence analysis tasks from the perspective of computer scientists

With rapid advent of AI technologies and biological data growth, researchers are increasingly applying AI methods to various areas of genetics biology. Development of large-scale AI applications requires an in-depth understanding of a wide range of sequence analysis tasks. Genetics biologists understand importance, biological relevance, and pharmaceutical potential of different protein sequence analysis tasks, but they often struggle to select the most suitable machine learning or deep learning models to enhance or replace experimental approaches. Alternatively, computer scientists excel in identifying most appropriate AI-driven predictive pipelines that are most likely to deliver optimal results for specific data types but they may find it challenging to fully grasp complexities of protein sequence analysis tasks. Thus, there is a significant gap between the expertise of computer scientists and genetics biologists. For instance, protein function prediction, protein similarity prediction or mutation effects prediction can be challenging to understand immediately. A detailed analysis of existing literature can greatly help bridge this gap by explaining the fundamentals of such tasks. For instance, protein function prediction initially seems like a multi-class classification task, but it is actually a multi-label classification task. Similarly, protein similarity prediction appears to be a regression task, but it is actually a clustering task and mutation effects prediction seems to be a multi-label classification task but it is actually a multi-class classification. With this core understanding, computer scientists can more precisely develop predictive pipelines tailored to binary, multi-class, multi-label classification, regression and clustering task.

To facilitate research community, we have systematically categorized 63 protein sequence analysis tasks based on their nature as illustrated in Figure 4. A high-level overview of Figure 4 reveals that protein sequence analysis tasks can be broadly categorized into 3 primary types: (1) Regression, (2) classification, (3) Clustering. Classification tasks can further be divided into three secondary types: (1) Binary Classification, (2) Multi-class Classification, (3) Multi-label Classification. This section delves into mathematical formulations of unique types of protein sequence analysis tasks.

In binary classification, the primary goal is to predict the outcome of a binary variable (0 or 1). Given a dataset with features |$X_i \in \mathbb{R}^{n \times d}$|⁠, binary labels |$Y_i \in \{0, 1\}$|⁠, and a training set |$(X_1, Y_1), (X_2, Y_2), \dots$|⁠, the main objective is to learn a decision function |$f(x): X_i \rightarrow Y_i$| that maps inputs to binary outcomes |$\{0, 1\}$| using the hypothesis function |$h(X_i)$| derived from the training data. Equation 1 illustrates mathematical expression for decision function f(x).

Multi-class classification predicts outcome from more than two classes. In a dataset with features |$X \in \mathbb{R}^{n \times d}$|⁠, labels |$y \in {1, 2, \dots, n}$|⁠, where n indicates total number of classes, and a training dataset |$(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k)$| where |$x_i \in X$| and |$y_i \in Y$|⁠, objective is to develop a decision function |$f(x): X \rightarrow Y$| that assigns inputs to one of the available classes. Equation 2 depicts mathematical expression for decision function f(x) in multi-class classification.

In multi-label classification, each input may be associated with multiple classes simultaneously. For instance, in a dataset with features |$X \in \mathbb{R}^{n \times d}$|⁠, labels |$y \in {1, 2, \dots, n}$| where n denotes number of classes, and a training dataset |$(x_1, y_1, y_2, \dots), (x_2, y_1, y_4, \dots), \dots, (x_k, y_5, y_k, \dots)$| where |$x_i \in X$| and |$y_i \in Y$|⁠, main objective is to construct a decision function |$f(x): X \rightarrow {0, 1}^K$| that assigns inputs to multiple classes simultaneously using hypothesis function |$h_n(x)$| for class n derived from training data. Equation 3 represents mathematical expression for decision function f(x) used in multi-class classification.

Moreover, prime objective in regression is to predict a continuous outcome variable. Specifically in regression, for a dataset with features |$X \in \mathbb{R}^{n \times d}$|⁠, labels |$y \in \mathbb{R}$|⁠, and training dataset |$(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k)$| where |$x_i \in X$| and |$y_i \in Y$|⁠, aim is to learn a function |$f: X \rightarrow \mathbb{R}$| that predicts continuous outputs using hypothesis function h(x). Equation 4 depicts mathematical expression for hypothesis function h(x) learned from the training data.

In clustering, objective is to group similar data points into relevant clusters. Given a dataset of data points |$X = {x_1, x_2, \dots, x_k}$|⁠, where each |$x_i \in \mathbb{R}^d$|⁠, main goal is to assign a clusters |$C = {C_1, C_2, \dots, C_n}$| to each data point by using a distance metric |$d(x, \mu_c)$|⁠. Distance metric determines distance between a data point x and centroid µ_c of cluster c. Equation 11 illustrates mathematical expression of function f(x) to estimate distance.

Protein sequence analysis databases

This section presents a comprehensive survey of protein databases that encompasses essential data for the development of AI-driven applications across 63 diverse protein sequence analysis tasks. It equips AI researchers with essential information required to identify appropriate databases for the development of high-quality benchmark datasets, which are the cornerstone for development of AI-driven protein sequence analysis applications.

In the realm of AI-driven protein sequence analysis, a detailed review of 295 research articles indicates that researchers have harnessed a remarkable diversity of 100 unique protein databases to develop 627 benchmark datasets for 63 protein sequence analysis tasks. To the best of our knowledge, 68 of these databases are currently publicly accessible, while the remaining 32 are either restricted or no longer available. Table 1 presents a valuable road-map for AI researchers to select optimal databases for development of high-quality benchmark datasets. It offers a concise yet informative overview of 68 publicly accessible databases by highlighting their diverse characteristics such as database name, release date, data types, related species and organisms, data statistics, and data formats.

A closer examination of the ‘data type’ feature in Table 1 reveals that: All databases contain protein data and out of the 68 databases, 13 also contain information related to DNA and RNA. These databases include DisGeNET (72), CARD (73), VariBench (74), ClinVar (75), BioLip (76), CCLE (77), NCBI (78), MtSSPdb (79), GEO (80), KEGG (81), PINA (82), EMBL-EBI (83) and OMIM (84). Moreover, 13 databases contain different data types as follows, transcriptomics: MtSSPdb (79), immune repertoires: OAS database (85), genes, mutations and drugs: GeneCards (86), IMGT (87), and COSMIC (88), host proteins: HPIDB (89), gene and diseases: MalaCards (90), molecules, drugs, compounds, and drugs: ChEMBL (91), DUD-E(92), and BindingDB (93), and chemicals: (94), and DUD (95). Moreover, data related to TCR sequences, antigens, immunoglobulins (IGs), T cell epitopes, microbiome and antibodies is available in McPAS-TCR (96), VDJdb (97), PIRD (98), MGnify (99) and IEDB (100) databases. In addition, Negatome database (101) contains domain pairs sequences, PubChem (102) provides compounds strings, genes, and cell lines, CTD (103) houses data related to chemical-gene interaction, chemical-disease interaction, and chemical-phenotype interactions, intAct (104), provides data related to interactions, interactors, and mutations, and enzymes data is available in BRENDA (105).

In Table 1, we performed a detailed analysis of ‘Species’ feature to categorize databases into three classes: (1) Fewer species coverage, (2) Moderate species coverage, (3) Large species coverage. In the category of fewer species coverage, we have included 35 databases housing data for 20 species or fewer. This category databases names are DisProt (106), PHROGs (107), MtSSPdb (79), PPT-Ohmnet (108), COSMIC (88), HPIDB (89), McPAS-TCR (96), VDJdb (97), DisGeNET (72), HIPPIE (109), MalaCards (90), ClinVar (75), BioLip (76), PDB (110), ConSurf-DB (111), dbPTM (112), CCLE (77), STITCH (94), NCBI (78), intAct (104), Therapeutic Targets Databases (113), Phospho.ELM (114), GeneCards (86), KEGG (81), Prosite (115), UniProtKB (116), OMIM (84), OAS database (85), SAbDab (117), Negatome database (101), DUD-E (92), DUD (95), PDBbind database (118), PhosphoSitesPlus (119) and interPro (120). On the other hand, in the moderate species coverage category, we included eight databases encompassing data for a range of 21 to 80 species. These databases include AlphaFoldDB (121), AmyPro (122), MobiDB (123), IPD-MHC (124), CARD (73), BioGRID (125), GEO (80) and ChEMBL (91). In large species coverage category, we included 25 databases encompassing data more than 80 species. This category related databases are MINT database (126), OGEE (127), DIP (128), IMGT (87), STCRDab (129), PIRD (98), Uniclust30 (130), GLASS (131), MGnify (99), SCOPe (132), BindingDB (93), VariBench (74), PINA (82), TCDB (133), PubChem (102), GOA (134), IEDB (100), CTD (103), STRING (72), RCSB PDB (135), SCOP (136), EMBL-EBI (83), GPCRdb (137), CATH (138) and BRENDA (105).

Based on an in-depth analysis of ‘organism’ feature in Table 1, we have categorized these databases into two different classes: 1) Narrow-organisms range, 2) Wide-organisms range databases. In narrow-organisms range databases, 20 or fewer organisms are present and we have included 12 databases to this category. This names of these databases are DisProt (106), MtSSPdb (79), OAS database (85), COSMIC (88), HPIDB (89), DisGeNET (72), HIPPIE (109), MalaCards (90), ClinVar (75), BioLip (76), CCLE (77) and NCBI (78). In contrast, remaining databases, also known as wide-organism range, contain more than 20 databases such as BRENDA (105), intAct (104), STRING (72), and CTD (103) etc.

Since word embeddings and LLMs based predictive pipelines require large amount of raw data for training in an unsupervised fashion, these databases act as facilitators for development of these predictive pipelines. For this, we have categorized these databases based on the volume of data into three different categories: (1) Low sequence facilitator, (2) Medium sequence facilitator, (3) High sequence facilitator. Low sequence facilitator databases provide with up to 100,000 sequence. A total of 26 databases are low sequence facilitator databases which include AmyPro (122), BindingDB (93), ChEMBL (91), DisProt (106), DUD (95), DUD-E (92), HPIDB (89), interPro (120), IPD-MHC (124), MalaCards (90), MtSSPdb (79), Negatome database (101), PDBbind database (118), Phospho.ELM (114), PPT-Ohmnet (108), SAbDab (117), SCOPe (132), STCRDab (129), TCDB (133), Therapeutic Targets Database (113), VDJdb (97), CCLE (77), UniProtKB (116), NCBI (78), OMIM (84) and Prosite (115). Similarly, medium sequence facilitator databases contain data sequences within a range of 100,000 to 1 million. There are 17 medium sequence facilitators databases namely CARD (73), CATH (138), ConSurf-DB (111), DIP (128), GeneCards (86), GLASS (131), GPCRdb (137), IMGT (87), MINT database (126), OGEE (127), PhosphoSitesPlus (119), PHROGs (107), PINA (82), RCSB (135), PDB (135), SCOP (136), GEO (80) and BioLip (76). In this study, 25 databases are identified as high sequence facilitator databases including AlphaFoldDB (121), BRENDA (105), dbPTM (112), GOA (134), IEDB (100), intAct (104), MGnify (99), MobiDB (123), OAS database (85), PubChem (102), KEGG (81), CTD (103), STRING (72), DisGeNET (139), BioGRID (125), STITCH (94), ClinVar (75), COSMIC (88), HIPPIE (109), McPAS-TCR (96), PIRD (98), Uniclust30 (130), VariBench (74), PDB (110) and EMBL-EBI (83).

A closer look on Table 1 ‘data format’ feature revels that in total 41 unique data formate have been used to house data into databases. These formats are, txt, .csv, .json, FASTA, .tsv, GAF, .xlsx, .pdf, MSA, HMM, .gff, .edgelist, .irf, .dat, .xml, RDF, SQL Dump, .sdf, .mitab, .gz, .tar, tab, OBL, OWL, .vcf, .tsv.gz, .mol2, .asnt, psi, psi25, tab2, tab3, xgmml, SOFT, MINiML, .dump, .cif, API, KGML, .dat and, doc. A detailed analysis of 295 studies reveals that, txt and FASTA formats are most commonly used by various protein sequence analysis pipelines. Each entry in these formats consists of at least two lines: first is known as header and includes accession number, species name, or identification details whereas other lines contain amino acid sequences. Second most commonly used formats are, csv and, tsv which are text-based formats and uses commas and tabs to separate values in rows. Specifically, in these two formats, first row signifies header which contain information title such as sequence ID, name, type, function, process and other details and preceding lines contain entries of data. In contrast, .xlsx format is used for complex datasets which contain computed information using various formulas across multiple columns. Additionally, vcf format also specifies headers in the first row and is mostly used to store genetic variation data which encompasses single nucleotide polymorphisms, insertions, deletions, and structural variants.

Furthermore, data related to protein function prediction is available in seven databases namely CARD (73), AlphaFoldDB (121), DisProt (106), GOA (134), MobiDB (123), SCOPe (132) and STCRDab (129). Similarly, data related to structure prediction, bitter peptides identification, domain boundary prediction, variant effects prediction, protein complexes identification, intrinsically disorder protein prediction, G-Protein coupled receptors identification and virus-host protein interaction prediction task is available in seven databases including AlphaFoldDB (121), AmyPro (122), BindingDB (93), CATH (138), ConSurf-DB (111), DIP (128) and DisProt (106), respectively. In addition, data for drug-target interaction and drug-protein interaction prediction is present in six databases namely DUD-E (92), BindingDB (93), ChEMBL (91), BRENDA (105), PubChem (102) and DUD (95) databases. Moreover, data for multiple interaction types prediction and compound-protein binding affinity prediction utilize tasks is extracted from ChEMBL (91) database, MINT database (126), intAct (104), and Therapeutic Targets Database (113). Additionally, data related to virus-host interaction prediction, protein–protein interaction prediction, gene functions prediction, secreted peptides prediction, antibody sequence infilling, phage-host interaction prediction, TRP channels classification and mutation effects prediction is available at 11 databases including GPCRdb (137), HIPPIE (109), HPIDB (89), intAct (104), MGnify (99), MtSSPdb (79), OAS database (85), PHROGs (107), RCSB PDB (135), TCDB (133) and VariBench (74) databases. Similarly, data regarding post-translational modification prediction is sourced from three different databases namely dbPTM (112), Phospho.ELM (114) and PhosphoSitesPlus (119). Moreover, two databases namely DUD (95) and PDBbind database (118) houses data for commercially available inhibitors prediction against SARS-CoV-2. Moreover, DUD (95) database also contains data related to drug-target binding affinity prediction, whereas GLASS (131), BindingDB (93), ChEMBL (91) facilitate with data related to compound-protein interaction prediction. IEDB (100) database is specific for providing sequences for anti-inflammatory peptides identification, Protein Binding Sites Prediction and MHC–peptide class II interaction prediction. Moreover, data related to enzyme substrate prediction and protein function identification is also available at interPro (120). In contrast, data related to disease genes identification is available at MalaCards (90), ChEMBL (91), intAct (104), MINT database (126) and GeneCards (86) database. Specifically, data related to protein–protein interaction prediction is provided by seven databases including DIP (128), HIPPIE (109), intAct (104), PINA (82), PPT-Ohmnet (108), PPT-Ohmnet (108), and MINT database (126). Furthermore, data related to essential genes identification is available at OGEE (127) and DIP (128) but OGEE (127) also facilitates with data related to essential gene identification. Data related to vascular calcification, protein properties prediction, remote homology detection, solubility, fold prediction and subcellular location identification tasks is available in five databases namely PDBbind database (118), BindingDB (93), PubChem (102), SCOP (136) and SCOPe (132) databases. Additionally, data related to nucleic acid binding protein prediction, secondary structure prediction, and binding affinity prediction task is present at 10 databases including Uniclust30 (130), MGnify (99), VDJdb (97), SCOPe (132), PIRD (98), DisProt (106), SCOP (136), BindingDB (93), ChEMBL (91) and PubChem (102) database.

Protein sequence analysis benchmark datasets

This section presents a summary of 627 benchmark datasets that have been utilized for development of AI-driven applications across 63 diverse protein sequence analysis tasks. A rigorous analysis of 295 AI-driven protein sequence analysis articles reveals that these datasets are either created by authors or taken from existing studies. Among the 627 benchmark datasets, 403 are publicly accessible, while the remaining 224 are in-house. Table 2 illustrates 63 distinct protein sequence analysis tasks related 627 benchmark datasets distribution into public and in-house sources.

A thorough analysis of AI-driven protein sequence analysis literature reveals that:

• 229 public and 100 in-house datasets are used to develop LLMs-based applications for 47 different protein sequence analysis tasks including Protein Family Classification (140, 141), G-protein Coupled Receptors Identification (58), Intrinsically disordered protein (IDP) identification (60), Glycosylation and Glycation modification prediction (12, 13, 142), Protein S-nitrosylation modification prediction (14), Phosphorylation modification prediction (15), Serine phosphorylation modification prediction (19), Succinylation modification prediction (20), Lysine phosphoglycerylation modification prediction (21, 22), Non-histone acetylation modification prediction (143), Protein Glutarylation modification prediction (144), Protein–Protein Interaction Prediction (35, 36, 42–44, 46, 145–149), Contact Prediction (27, 148, 150–155), Nucleic Acid-Binding Proteins Identification (156–160), Virus Host Protein Interaction Prediction (47), Compound-Protein Interaction Prediction (61, 161, 162), Compound-Protein Binding Affinity Prediction (61–64, 161, 163), Phage-Host Interaction Prediction (164), Antimicrobial Peptides Identification (165), Signal Peptides Identification (166), Secreted Peptides Identification (167),Anti-Inflammatory Peptides identification prediction (168), Antibody Sequence Infilling (169), Peptide-Binding Specificity Prediction (170), Drug-Protein Interaction Prediction (171–173), Drug-Target Binding Affinity Prediction (174–176), Gene Phenotype Prediction (177), Disease Genes Prediction (178–180), Prokaryotic Gene Prediction (181), Solubility Prediction (27, 150, 182, 183), Stability Prediction (150, 154, 155, 184–186), Thermophilicity Prediction (187), Fluorescence Prediction (27, 154, 155, 185, 186, 188), Domain Boundary Prediction (189), Protein Function Identification (150, 185, 186, 190–207), Structure Prediction (27, 60, 150, 154, 155, 188, 208–213), Fold Prediction (27, 150, 214), Remote Homology Detection (136, 154, 188, 213, 215, 216), TRP channels Classification (217), Protein Subcellular Localization Identification (23, 27, 28), Protein Submitochondrial Localization Identification (218), Subchloroplast Localization Identification (219), Mutation Prediction (220), Mutation Effects Prediction (221–223), Variant Effects Prediction (224), and B-Cell Epitopes Identification (225).

• 86 public and 79 in-house datasets are used to develop word embeddings based AI applications for 32 protein sequence analysis tasks including Protein Family Classification (226, 227), SNARE proteins Identification (228), Protein Similarity Prediction (229), Electron Transport Protein Identification (57), Essential Proteins Identification (59, 230–237), Phosphorylation modification prediction (16–18, 18), Protein Glutarylation Modification Prediction (238), Protein S-sulfenylation modification prediction (239), Protein–Protein Interaction Prediction (38–40, 45, 65, 240–246), Protein Complexes Identification (247–251), Residue-Residue Interaction Prediction (252), Nucleic Acid-Binding Proteins Identification (253), Protein RNA Interaction Prediction (254–259), Virus Host Protein Interaction Prediction (48, 49, 51), Compound-Protein Interaction Prediction (260–263), Phage-Host Interaction Prediction (264), Missing Link Prediction (265–270), Antibacterial peptides (54), Drug-Protein Interaction Prediction (271, 272), Drug-Target Binding Affinity Prediction (271), Gene Phenotype Prediction (273), Disease Genes Prediction (274–294), Essential Genes Identification (295, 296), Protein Function Identification (297–305), Protein Subcellular Localization Identification (30–33), Protein Submitochondrial Localization Identification (306), Malaria Parasite Identification (307), Tumour Necrosis Factors Identification (308), COVID-19 Virus Classification (309), Vascular Calcification (310), and |$B/T$| Cell Receptor Sequences Analysis (311).

• 88 public and 46 in-house datasets are used to develop domain-specific representation learning based AI applications for 21 different tasks namely Protein Family Classification (312), SNARE proteins Identification (56), Essential Proteins Identification (313–318), Protein S-sulfenylation modification prediction (239), Protein–Protein Interaction Prediction (37, 41, 319), Protein RNA Interaction Prediction (320), Virus Host Protein Interaction Prediction (50, 52), Compound-Protein Interaction Prediction (321), Anticancer Peptides identification prediction (53, 322–327), Antimicrobial Peptides identification prediction (55, 328–335), Signal Peptides identification prediction (336), Antibacterial peptides (337), Anti-Inflammatory Peptides identification prediction (338–342), Drug-Protein Interaction Prediction (343), Drug-Target Binding Affinity Prediction (344, 345), Disease Genes Prediction (346, 347), Solubility Prediction (348), Protein Function Identification (349), Remote Homology Detection (350), Protein Subcellular Localization Identification (25, 26), and Protein Submitochondrial Localization Identification (351).

Through a large-scale literature review, we investigate the reuse patterns of common datasets across three distinct types AI applications paradigms: LLMs, word embeddings, and domain-specific representation learning methods. Our conclusions on the overlap between all three paradigms related AI-driven applications and dataset reuse patterns across 63 distinct protein sequence analysis tasks are as follows:

• 245 public and 124 in-house datasets are used to develop both word embedding and LLMs based predictive pipelines for 12 different protein sequence analysis tasks including Protein–Protein Interaction Prediction, Virus Host Protein Interaction Prediction, Compound-Protein Interaction Prediction, Drug-Protein Interaction Prediction, Drug-Target Binding Affinity Prediction, Disease Genes Prediction, Protein Function Identification, Protein Subcellular Localization Identification, Protein Family Classification, Nucleic Acid-Binding Proteins Identification, Phage-Host Interaction Prediction, and Gene Phenotype Prediction.

• 253 public and 138 in-house datasets are leveraged for development of word embedding and domain-specific representation learning approaches based predictive pipelines for 13 tasks namely Protein–Protein Interaction Prediction, Virus Host Protein Interaction Prediction, Compound-Protein Interaction Prediction, Drug-Protein Interaction Prediction, Drug-Target Binding Affinity Prediction, Disease Genes Prediction, Protein Function Identification, Protein Subcellular Localization Identification, Essential Proteins Identification, Protein S-sulfenylation modification prediction, Protein RNA Interaction Prediction, Antibacterial peptides identification, and Protein Submitochondrial Localization Identification.

• 259 public and 149 in-house datasets are utilized for the development of both LLMs and domain-specific representation learning approaches based predictive pipelines for 14 protein sequence analysis tasks including Protein–Protein Interaction Prediction, Virus Host Protein Interaction Prediction, Compound-Protein Interaction Prediction, Drug-Protein Interaction Prediction, Drug-Target Binding Affinity Prediction, Disease Genes Prediction, Protein Function Identification, Protein Subcellular Localization Identification, SNARE proteins Identification, Phosphorylation modification prediction, Antimicrobial Peptides identification, Signal Peptides identification, Solubility Prediction, and Remote Homology Detection.

A rigorous analysis of existing studies reveal that only two public datasets are commonly employed by both word embedding and language models based predictive pipelines for Protein–Protein Interaction Prediction, six public datasets are commonly employed by both word embedding and domain-specific representation learning approaches based predictive pipelines for Protein Function Identification, and four public datasets are commonly employed by both language models and domain-specific representation learning approaches based predictive pipelines for protein–RNA interaction prediction. Overall for all three kinds of predictive pipelines, 221 public and 120 in-house datasets are used to develop predictive pipelines for 8 different protein sequence analysis tasks namely Protein–Protein Interaction Prediction, Virus Host Protein Interaction Prediction, Compound-Protein Interaction Prediction, Drug-Protein Interaction Prediction, Drug-Target Binding Affinity Prediction, Disease Genes Prediction, Protein Function Identification, and Protein Subcellular Localization Identification. Surprisingly, not a single dataset is commonly used by all three kind of predictive pipelines. This trend highlights a tendency among researchers to create new datasets for each predictive pipeline instead of reusing existing ones. Consequently, protein sequence analysis domain lacks robust comparisons of predictive pipeline performance.

A holistic view of Table 2 reveals that seven distinct tasks related public datasets are not available because existing application are developed on in-house datasets only. These tasks include protein similarity prediction, G-protein coupled receptors identification. The highest number of public datasets are available for two tasks namely Protein Function Prediction, Protein–Protein Interaction prediction, and Protein Subcellular Localization Prediction with 88, 40, 38 datasets. It is difficult to perform experimentation on such a large number of datasets, to stream line development of applications. Hence, there is need to analyse all these tasks related public datasets and develop new species specific datasets and benchmark performance on these datasets.

Word embeddings based AI-driven protein sequence analysis predictive pipelines

This section outlines 22 distinct word embedding methods that are utilized in AI-driven predictive pipeline development for 63 different protein sequence analysis tasks. These methods include Mashup (299, 304), Word2Vec (32, 309, 311), Node2Vec (231, 271), FastText (239), RotatE (283), ANE (265), ELMo (297), DANE (230), GEMSEC (59), LINE (40), Struc2Vec (275), TransE (276), DeepWalk (280), Hyper2Vec (282), Opa2Vec (284), Random Watcher-Walker (RW2) (289), HOPE (274), RandomWalk (265), SDNE (274), GloVe (238), GraRep (261), and Topo2Vec (270). Figure 5 provides a comprehensive overview of these 22 word embedding methods along with 21 different predictors namely SVM (281), RF (302), BiLSTM (306), CNN (306), MLP (229), BiGRU (230), GCN (247), LogR (252), LightGBM (48), k-means (251), IBk (273), GLM (294), Ensemble (LR + BR + DTR + SVM) (248), XGBoost (261), cGAN (266), CCA (51), Louvain clustering (65), LSTM (245), ET (296), and GAT (255).

A high-level analysis of Figure 5 reveals that Node2Vec emerges as the most extensively explored method, followed by Word2Vec, FastText, and ELMo. Specifically, Node2Vec (231, 271) is employed with 18 different predictors, while Word2Vec (32) based predictive pipelines have used 10 predictors. On the other hand, FastText (239) and ELMo (297) based predictive pipelines have reaped benefits of six predictors each. DeepWalk based predictive pipelines (280) has employed five predictors and SDNE based predictive pipelines (274) have used four predictors. Additionally, LINE (40) and RotatE (283) based predictive pipelines have leveraged 3 and 2 predictors, respectively. However, potential of remaining 14 word embedding methods have not been fully explored as they are evaluated with only one predictor. This highlights the need of further investigation to fully explore their potential.

In the realm of protein sequence analysis, word embedding methods are utilized in two primary ways to generate sequence embeddings. First approach segregates protein sequences into k-mers and generates embeddings for each individual k-mers. In contrast, second approach generates embeddings for entire protein sequences, which can be subsequently used for homogeneous and heterogeneous networks. Homogeneous networks deal with a single type of biomolecule, such as protein–protein interaction networks. On the other hand„ heterogeneous networks involve multiple types of biomolecules, including proteins, RNA (e.g. lncRNA, miRNA, circRNA (378, 379)), drugs, compounds, and diseases. In heterogeneous networks, nodes represent biomolecules, while their interactions or associations form the edges. Heterogeneous networks include disease-gene association networks, compound-protein interaction networks, RNA-protein interaction prediction, and virus-host interaction prediction. Compared to homogeneous networks, heterogeneous networks are more complex and enable the extraction of richer and more comprehensive relationships through graph-based embedding methods.

Among 22 unique word embedding methods, 10 method (Word2Vec, FastText, DANE, ANE, ELMo, GloVe, Node2Vec, LINE, GraRep, and SDNE) have been employed to generate k-mer based embeddings of sequences across 8 different predictive pipelines (38, 255, 258, 262, 271, 281, 287, 300). An extensive review of existing studies reveals that Word2Vec is most widely explored in the k-mer approach, followed by FastText and Node2Vec. Specifically, 13 predictive pipelines have employed Word2Vec using k-mer approach with eight different predictors for protein family classification (226), protein–protein interaction prediction (40), and anti-bacterial peptides identification (54). These predictors include CNN (226, 227), SVM (32), LogR (311), MLP (242), GNN (254), RF (259), BiLSTM (49, 54), and BiLSTM + MCNN (303). Additionally, nine predictive pipelines (57, 228, 239, 243, 253, 302, 305, 307, 308) have utilized FastText in k-mer approach for eight different tasks and leverages 7 predictors including CNN (228), SVM (57), MLP (239), LSTM + CNN (243), affinity regression (253), RF (302), and an ensemble (RF + PNN + SVM + KNN) (307). Moreover, 5 predictive pipelines has explored the potential of Node2Vec in k-mer approach. Node2Vec based predictive pipelines have leveraged 4 predictors (GNN (38, 258), MLP (271), GAT (255), BiGRU (271)) for three different protein sequence analysis tasks namely protein–protein interaction prediction (38), RNA-protein interaction prediction (255), and disease gene interaction (271).

Moreover, 18 distinct predictive pipelines have used 12 unique word embedding methods by generating embedding for entire bio-molecule (Homogeneous network) for eight distinct protein sequence analysis task including protein similarity prediction (229), essential genes identification (59, 231–235), compound-protein interaction prediction (260, 263), missing link identification (270), residue-residue interaction prediction (252), protein complex identification (248–251), protein–protein interaction prediction (45, 244–246, 319), and virus-host interaction prediction (51). These 12 unique word embedding methods includes Node2Vec (277), GEMSEC (59), DeepWalk (244), Topo2Vec (270), LINE (274), SDNE (274), HOPE (274), Struc2Vec (275), Hyper2Vec (282), Opa2Vec (284), Random Watcher-Walker (RW2) (289), and Mashup (299). Among all these word embedding methods, Node2Vec and DeepWalk are explored for entire sequence embedding generation across 10 distinct predictive pipelines (45, 231–237, 244, 245).

Apart from this, 14 predictive pipelines (65, 247, 256, 267, 272, 276, 278–280, 291–294, 310) have leveraged seven unique word embedding methods for heterogeneous network for six distinct protein sequence analysis task including protein–protein interaction prediction, protein complex identification, protein–RNA interaction prediction, drug-protein interaction prediction, disease gene prediction and vascular calcification. These methods include Node2Vec (65, 267, 272, 278–280, 291, 294, 310), HIN2Vec (256), TransE (276), DeepWalk (280), LINE (280), and SDNE (280). Surprisingly, 11 predictive pipelines have explored Node2Vec for heterogeneous network along with seven predictor including Louvain clustering (65), LightGBM (267, 280), GNN (272), MLP (279, 291), LogR (278), GLM (294), and RF (310). An in-depth analysis of existing studies reveals that four embedding approaches (DeepWalk, Node2vec, LINE, SDNE) are used for generating k-mer embeddings as well as entire sequence embeddings in both homogeneous and heterogeneous networks.

Language models based AI-driven protein sequence analysis predictive pipelines

This section highlights 15 unique language models that are used to develop 137 AI-driven predictive pipelines for 63 distinct protein sequence analysis tasks. These language models include BERT (140), ALBERT (175), RoBERTa (183), T5 (17, 20), GPT (177), ESM-1 (181, 200), ESM-2 (150, 190), AlphaFold (207), AlphaFold2 (161), IgFold (169), MolFormer (61), Graph Transformer Network (178), XLNet (212), Vision Transformer (46) and Transformer (15, 19).

The training paradigm of language models involves two stages: pre-training and fine tuning. Pre-training involves training model on a large amount of unlabeled data to learn underlying patterns or structures of data in an unsupervised fashion. Specifically, language models learn contextual patterns by using masked language modelling and next sentence prediction approaches. In masked language modelling approach, some k-mers in a sequence are replaced with a mask token, and model learns to predict the masked k-mers whereas next sentence prediction approach focus on predicting next sequence based on preceding sequence. By learning contextual patterns, model acquires a deep understanding of linguistic features like syntax, semantics, and context without need of labelled data. Fine tuning stage adapts pre-trained model to perform downstream tasks including protein family classification, protein function identification, or protein–protein interaction prediction. During fine-tuning, model is further trained on a task-specific labelled dataset. Specifically for this process, model can be used in two ways: 1) By using its self-classifier, 2) Integrating external classifiers to optimize task performance. Figure 6 graphically illustrates the utilization of 15 different language models, either paired with their self-classifiers or used in combination with 18 external classifiers. The external classifiers include SVM (12), LogR (202), RF (164), GBDT (223), XGBoost (35), MLP (161, 196), Hybrid (XGBoost + CAPT5) (35), CNN (142), GRU (167), BiGRU (62), BiLSTM (189), GNN (206), GCN (182), GAT (44), GAN (198), GVP (186), BiLSTM + BiGRU (60), and Inductive Matrix Completion Algorithm (179).

In protein sequence analysis landscape, among 137 predictive pipelines based on 15 LLMs, 72 predictive pipelines have leveraged self-classifier, whereas 65 pipelines make use of an external classifier to perform downstream tasks. Among 72 predictive pipelines based on language models leveraging self-classifier, BERT stands out the most extensively used language model, followed by T5 and ESM-2. Specifically, 14 predictive pipelines (21, 58, 140, 147, 154, 158, 174, 180, 186, 186, 188, 188, 188, 218) have explored the potential of BERT language model with a self-classifier for eight different tasks, eight predictive pipelines (17, 20, 27, 28, 187, 195, 203, 214) has employed T5 with a self-classifier for 11 different tasks and seven predictive pipelines (23, 60, 150, 190, 193, 194, 210) has leveraged ESM-2 for seven different tasks. In addition, 12 transformer based predictive pipelines (15, 19, 47, 149, 171, 173, 185, 185, 185, 213, 213) and 3 ESM-1 based predictive pipelines (181, 200, 222) utilize self-classifier for seven and two different protein sequence analysis tasks, respectively. On the other hand, four predictive pipelines have employed GPT (177), AlphaFold (207), RoBERTa (183), and graph transformer network (178) with a self-classifier for four different tasks. Beyond exploring standalone potential of language models, 10 predictive pipelines reap combined benefits of multiple language models or in combination with other approaches as: (Transformer + GNN) (63), (T5 + OHE) (143), (RoBERTa + BERT + ALBERT) (175), (ESM-2 + PSSM) (157), (ESM-2 + MolFormer) (61), (ESM-1 + BERT) (146), (ELMo + Vision Transformer) (46), (BERT + T5) (201), and (BERT + AlphaFold + IgFold) (169).

Furthermore, among 65 predictive pipelines integrating external classifier, BERT, ESM-1 and T5 based predictive pipelines are most widely explored, followed by transformer, ESM-2 and AlphaFold. Specifically, 11 BERT based predictive pipelines have incorporated seven distinct classifiers (BiLSTM (136, 189), CNN (142, 156, 159, 209), GVP (186), CNN+ BiLSTM + MLP (144), LogR (202), SVM (141), Transformer + GAT-CNN (43)), 11 T5 language model based predictive pipelines have employed six unique classifiers (CNN (14, 165, 208, 211), GCN (199), GMM (220), XGBoost+CAPT5 (35), LogR (215, 224), RF (164)), and 11 ESM-1 based predictive pipelines have incorporated six distinct classifiers (CNN (13, 22, 152, 153), BiLSTM (166), GAN (198), GCN (44, 204), GNN (206), GVPConv (151)). In addition, seven predictive pipelines have leveraged transformer with four classifiers (CNN (18), GRU (167), Inductive Matrix Completion Algorithm (179), ResNet (155)) and four ESM-2 based predictive pipelines have employed four classifiers (BiLSTM (160), CNN (184), GAT (36), RF (197)). Moreover, two predictive pipelines have utilized AlphaFold with GCN (221), AlphaFold2 with MLP (161) and RoBERTa with LogR (42) classifiers, respectively. Apart from this, 17 predictive pipelines reap benefits of two or more language models or other embedding or sequence encoding approaches with external classifiers such as: (AlphaFold + BERT) with SVM classifier (217), (ESM-1 + ESM-2 + T5) with BiLSTM + BiGRU classifier (60), (ESM-1b + AlphaFold) with SVM classifier (12), (BERT + TransE) with GNN classifier (148), (Transformer + BERT) with CNN + MLP classifier (163), (OHE + BERT) with CNN + MLP classifier (64), (Word2Vec + BERT) with SnTCN classifier (168), (BERT + RoBERTa) with GCN + MLP classifier (176), (ESM-1 + Prottrans) with CNN classifier (191), (ESM-2 + T5) with GNNs classifier (192), (ESM-2 + BERT) with DNNS classifier (196), (Transformer-XL + XLNet + BERT + ALBERT + ELECTRA + T5) with CNN classifier (212), (BERT + PSSM) with CNN classifier (219), (BERT + GNN) with GBDT classifier (223), (ESM-2 + AlphaFold) with BiLSTM + GNN classifier (225), and (Integer Encoding + RoBERTa) with BiGRU classifier (62).

Table 3 presents 15 distinct language models and their variants used for the development of AI-driven predictive pipelines for 63 different protein sequence analysis tasks. Additionally, it provides details on number of layers in language model architecture, including number of encoders or decoders and their respective layers. These language models are categorized into five groups based on their architectures. These categories include encoder-decoder, encoder-only, decoder-only, special transformer variants and Vision Transformer Models (380).

Experimental settings and evaluation measures

Evaluation of AI-driven protein sequence analysis of predictive pipelines undergoes through two distinct experimental settings: 1) k-fold cross-validation (28, 30) and 2) Train-test split (15, 143). k-fold cross-validation is an iterative process that divides dataset into k equal sized folds. Among k folds, k − 1 folds are used for training and remaining one fold is reserved for testing. For deep learning predictive pipelines, an additional set, known as validation set is developed which is usually 10% of training data. This validation set helps in tuning hyperparameters. On the other hand, train-test split setting splits dataset into two sets namely, a) train set and b) test set. Typically, train set contains majority portion of dataset (usually 70-80%) whereas, test set contains remaining portion of dataset (20-30%). Similar to k-fold cross-validation, train-test split also uses validation set for deep learning predictive pipelines.

Depending on task type, evaluation measures are categorized into four classes: (1) Binary (280)/Multi-class classification (397), (2) Multi-label classification (28), (3) Regression (184), and (4) Clustering (184). Following subsections provide an in-depth insight for all four types of evaluation measures.

Evaluation measures for binary/multi-class classification

Predictive pipelines for binary/multi-class classification categorizes instances into pre-defined classes. For evaluation of these predictive pipelines, most commonly utilized evaluation measures are accuracy(247), precision (247), recall (247), F1-score (247), specificity (264), and Matthews correlation coefficient (321). To compute performance values of predictive pipelines through these measures, confusion matrix is used which consists of four entities: true positive (TP), true negative (TN), false positive (FP), and false negative (FN). Figure 7 illustrates confusion matrix encompassing all four entities graphical representation.

Figure 7 depicts that TP and TN signify correct positive and negative predictions, whereas FP and FN indicate incorrect positive and negative predictions, respectively. Accuracy (Acc) (247) is ratio of correctly predicted instances out of total instances, whereas precision (Pr) (247) is proportion of true positive predictions out of all positive predictions. Recall (R) (247) calculates true positive predictions out of all actual positives and F1-score (247) is harmonic mean of precision and recall. Specificity (Sp) (264) measures true negative predictions out of all actual negatives. Matthews correlation coefficient (MCC) (321) is a ratio of difference between correct predicted class and incorrect predicted class to square root of product of all four entities of confusion matrix. Equation 6 embodies mathematical expressions used to calculate aforementioned measures.

An extensive study of existing protein sequence analysis predictors reveals that most widely used evaluation measures for balanced datasets are accuracy, precision, recall, specificity, F1-score and Matthews correlation coefficient (MCC). For highly imbalanced datasets, micro, macro, and weighted variants of these measures are employed. To overcome class imbalance issue, weighted score computes weighted average of these measures for each class such as weighted precision (267), weighted recall (267) and weighted F1-score (267). Similarly, Macro score (398) computes average sum of these measures namely precision, recall, all F1-score across all classes. Beyond weighted and macro scores, micro precision aggregates true positives and false positive, Micro recall (38) is a proportion of true positives and false negatives and micro F1-score (38) aggregates F1-score across all classes, respectively. Equation 7 depicts these measures mathematical expressions in terms of micro, macro and weighted scores.

Here, TP^j, FP^j and FN^j denote true positives, false positives and false negatives in class j, respectively. Similarly, Pr^j, R^j and |$F1^j $| indicate precision, recall and F1-score of class j. Here, w^j signifies weight of class j where j refers to j^th class among m classes.

Evaluation measures for multi-label classification

Contrary to binary or multi-class classification, predictive pipelines for multi-label classification predict multiple labels of an instance at a time. Hence, some of predicted labels can be either correct, incorrect, all correct or all incorrect. This partial correctness or incorrectness in prediction generates complexity. To overcome these complexities, researchers have proposed diverse evaluation measures namely: precision (Pr) (219), recall (R) (219), accuracy (Acc) (28), F1-score (F1) (219) and hamming loss (HL) (273). Precision measures average of all correctly predicted true positive labels out of all positive predicted labels, whereas recall calculates average of all correctly predicted true positive labels out of actual true labels. Accuracy calculates average of correctly predicted labels to total number of labels, whereas F1-score measures average of precision and recall. Hamming loss quantifies fraction of incorrect labels to total number of labels. Equation 8 embodies mathematical expressions for evaluation measures for multi-label classification.

In Equation 8, M represents total number of samples, m^j denotes j^th sample out of M samples. A^j is actual class label and P^j is predicted class label for m^j sample. L represents sample length, k denotes class index, |$\lor$| signifies logical OR operator and |$\land$| denotes logical AND operator. |$A_k^j $| represents true label of k^th instance for j^th label and |$P_k^j $| represents predicted label of k^th instance for j^th label.

Evaluation measures of regression

Predictive pipelines for regression tasks predict continuous numerical values instead of discrete labels. Most commonly used evaluation measures for regression based tasks are mean absolute error (MAE) (344), mean squared error (MSE) (63), root mean square error (RMSE) (344), mean bias error (MBE), mean absolute percentage error (MAPE), R² (63), relative mean absolute error (rMAE), relative mean square error (rMSE), relative mean bias error (rMBE) and relative root mean square error (rRMSE).

MAE measures average of absolute difference between predicted and actual values, whereas MSE calculates average of squares of errors between predicted and actual values. In contrast, MBE calculates average bias present in predictive pipelines by measuring average difference between predicted and actual value and MAPE determines average of absolute percentage errors between predicted and actual values. RMSE computes standard deviation by averaging squared differences of actual and predicted value which illustrates close clustering of data points around regression line. Lower values of MAE, MSE, MBE, and MAPE indicate better performance of predictive pipelines. R² calculates ratio of squared difference between actual and predicted values to squared difference between actual values with its mean. Equation 4 embodies mathematical expressions for evaluation measures for regression.

In above-mentioned Equation 4, N signifies total number of samples, A^j represents actual value and P^j denotes predicted values. Evaluating relative performance of predictive pipelines can enhance quality of performance by minimizing noise from data. Since data continuously changes and yields different predicted values at different times, relative error of all data points is calculated as an overall percentage. For rMAE, rMSE, rMBE, and rRMSE, percentage error of each matrix is computed relative to average of actual values. Equation 10 depicts mathematical expressions for relative measures.

Here, j denotes sample number and |$\bar{A}$| is average of overall actual values.

Evaluation measures of clustering

Clustering tasks related predictive pipelines aim to organize instances into clusters based on their similarity with each other. Higher value of similarity for a cluster signifies instances belongs to that specific cluster. The most commonly used evaluation measures for clustering based predictive pipelines are accuracy (140), normalized mutual information (140), silhouette score (399), Davies-Bouldin index (DBI) (400) and Dunn index(400). Accuracy (Acc) measures average of maximum instances correctly assigned to actual cluster for optimal cluster-label mapping. Normalized mutual information (NMI) determines mutual information between predicted and actual clusters. Higher NMI score indicates that clustering algorithm has successfully uncovered distinct and informative clusters within data. Silhouette score (SS) calculates similarity of instances to its own cluster and other clusters. Its score ranges from −1 to 1, where a higher score indicates better-defined clusters. DBI evaluates sum of distance of instance within-cluster to between-clusters. Dunn index (DI) quantifies ratio of minimum inter-cluster distance to maximum intra-cluster distance. Equation 11 depicts mathematical expressions for these evaluation measures.

In Equation 11, y_j signifies predicted cluster, c_j and c_k indicates j^th and k^th clusters among n clusters, whereas m denotes mapping of cluster-label. Furthermore, I(y_j,c_j) indicates mutual information, E(y_j) and E(c_j) shows entropy of predicted and actual clusters, respectively. d(y_j) is average distance from y_j to all points in other cluster and a(y_j) is average distance of y_j to all points in other clusters. d(c_j,c_k) represents inter-cluster distance between j and k clusters, |$\bar{S}_j$| denotes mean distance from cluster mean for all observations in cluster j, whereas |$\bar{S}_k$| denotes mean distance from cluster median for all observations in cluster k. An in-depth analysis of existing literature for protein sequence analysis demonstrates that most commonly employed evaluation metrics in domain of clustering are accuracy and normalized mutual information.

Open-source protein sequence analysis predictive models

Publicly accessible source codes for AI-driven protein sequence analysis applications allow researchers to avoid reinventing the wheel. They can leverage existing codes to develop applications for other similar tasks and can incorporate new ideas to refine and elevate the performance of established AI-driven protein sequence analysis applications. This section delves into open-source code availability within the landscape of AI-driven protein sequence analysis. To identify open source codes, we thoroughly analysed 295 research articles, specifically focusing on development of AI-driven protein sequence analysis applications by using two different paradigms: word embeddings and LLMs. Our analysis reveals that, for word embeddings-based AI applications, 52 out of 98 studies made their source codes publicly available. Similarly, for large language models (LLMs)-based AI applications, 102 out of 137 studies made their source codes publicly available. Tables 4 and 5 illustrate details about open-source codes of word embeddings and LLMs based protein sequence analysis applications respectively. These tables also summarises details of representation learning methods, machine/deep learning predictors employed, and link of respective source code.

An extensive analysis of Table 4 reveals that in 52 word embedding methods based studies which have follow the open science rules to make their source codes public, a total of 14 unique word embedding methods have been employed. These methods include Node2Vec (30, 38, 51, 149, 232, 236, 245, 248, 252, 260, 266, 271, 281, 291, 293, 401–403), FastText (57, 228, 239, 253, 305, 308, 404), Word2Vec (227, 271, 281, 303, 311, 405, 406), DeepWalk (249, 407), DANE (230), Mashup (299), ELMo (33, 145, 240, 297, 298, 306), GraRep (261), LINE (241), Topo2Vec (270), Struc2Vec (275), ComplEx (276), DistMult (276), and TransE (276). Among all these methods, the highest number of predictive pipelines are developed by utilizing Node2Vec embedding generation method. A total of 19 open-source predictive pipelines utilize Node2vec word embedding method along with a diverse range of machine learning and deep learning predictors. Specifically, two predictive pipelines have used Node2vec with CNN classifier for two different tasks (232, 260), two predictive pipelines have explored Node2vec potential with LSTM classifier (30, 245) and other 15 studies have used Node2vec embedding method along with MLP (236, 263), GNN+MLP (38), SVM (281), DBN (291), Cosine Similarity (293), BiGRU (271), LogR (252), XGBoost (401), CCA (51), GNN (402), cGAN (266), RF (403), Transformer (149) and ensemble (LR+BR+DTR+SVM) (248) classifiers. Similarly, seven predictive pipelines have utilized FastText word embedding method with four unique classifiers (SVM, MLP, CNN, Affinity Regression) for seven different tasks and seven predictive pipelines have explored Word2vec method potential with six unique classifiers (LogR classifier (40, 311), BiGRU (405), CNN (227), Bi-LSTM+MCNN (303), SVM (39), and XGBoost (406)) for six different protein analysis tasks. Furthermore, six predictive pipelines have used ELMo word embedding along with six classifiers namely CNN (33), LogR (240), BERT (145), MLP (297), SVM+LR (298), CNN+BiLSTM (303) for four different tasks. In addition, two predictive pipelines have used DeepWalk word embedding along with RF classifier and clustering algorithm for two tasks. In contrast, remaining predictive pipeline have used 6 unique word embedding approaches namely DANE (230), Mashup (299), GraRep (261), LINE (241), Topo2Vec (270), and Struc2Vec (275) with six different classifiers (MLP, SVM, XGBoost, RF, RF, and XGBoost) for six different tasks.

A holistic view of Table 4 shows that six predictive pipelines have reaped combine benefit of multiple word embedding methods or word embedding method along with language model. Specifically, predictive pipelines have made use of following different combinations of methods: (FastText, BERT) (168), (Mashup, Node2vec) (304), (Word2vec, Node2vec) (40, 287), (ComplEx, DistMult, TransE) (276), and (Node2vec, DeepWalk) (244) is explored for four different tasks

A holistic view of Table 5 shows that in 102 language models based studies, a total of 10 unique language models have been employed. These models include ESM-2 (23, 36, 60, 157, 160, 184, 408–411), Transformer (15, 19, 47, 155, 172, 185, 213, 412–418), RoBERTa (42, 183, 410, 419–422), AlphaFold (169, 170, 207, 221, 423–426), BERT (43, 58, 142, 144, 146, 148, 154, 156, 168, 169, 180, 186, 188, 189, 201, 202, 209, 212, 218, 219, 223, 273, 421, 427–439), T5 (14, 17, 20, 27, 35, 143, 164, 187, 199, 201, 203, 211, 212, 214–216, 220, 224, 408, 440, 441), ESM-1 (13, 44, 146, 152, 153, 181, 204, 222, 397, 408, 442–444), GPT/GPT-2 (411, 445), Transformer-XL (212) and MSA Transformer (222).

In language models driven protein sequence analysis applications, these models have been utilised in two different scenarios: (1) Training from scratch followed by fine-tuning, and (2) Fine-tuning only. In the first scenario, researchers train these models on large protein sequence datasets in an unsupervised manner and further fine-tune them on downstream tasks. In the second scenario, researchers use existing pre-trained models and just fine-tune them for particular downstream tasks.

Moreover, apart from these two scenarios, researchers either use these models directly to develop an end-to-end applications or extract representations from these models to feed into machine learning or deep learning predictors. Overall, 12 distinct pre-trained BERT language models have been utilized in 34 different predictive pipelines. These pipelines have been developed for 12 different tasks namely |$B/T$| Cell Receptor Sequences Analysis, Protein Functions Prediction, Protein–Protein Interaction Prediction, Structure Prediction, Contact Prediction, Remote Homology Detection, Fluorescence Prediction, Drug-Protein Interaction Prediction, Protein Submitochondrial localization prediction, Domain Boundary Prediction, Disease Gene Prediction, and Stability Prediction. BERT language model based 12 different pretrained variants are generated by performing unsupervised training of model on different types of data. Table 6 illustrates details of protein data on which BERT and other 35 language models are trained to produce different pretrained versions.

Protein sequence analysis predictive pipelines performance analysis

In AI-driven protein sequence analysis landscape, applications are being developed through three primary approaches: (1) leveraging LLMs, (2) utilizing word embedding methods combined with machine or deep learning predictors, and (3) harnessing domain-specific sequence encoding methods along with machine or deep learning predictors. This review primarily focuses on an in-depth exploration of LLMs and word embeddings-based applications across 63 different protein sequence analysis tasks. However, these approaches may not always achieve state-of-the-art performance for every task. In some cases, domain-specific sequence encoding methods might offer superior performance. To facilitate the development of new predictors, one objective of this paper is to provide the current state-of-the-art performance of predictor for each task. To achieve this, we performed a large-scale literature review on domain-specific encoding methods based applications and included those with the current state-of-the-art performance in the results tables. Tables 7 to 16, contain performance values of all word embedding and LLMs based applications. These tables also include domain specific methods based applications which have state-of-the-art performance values for each task. Moreover, each table represents AI-driven applications performance for a particular goal which contain several protein sequence analysis tasks.

Table 7 provides performance metrics for 25 AI-driven (language models, word embeddings, domain specific) applications that are designed for protein identification goal related tasks. This goal oriented binary/multi class classification and similarity prediction applications are developed by utilizing 18 unique representation learning approaches and 12 unique classifiers. The representation learning methods include BERT, Word2vec, RotatE, PSSM, FastText, DANE, Node2vec, OHE, PseAA, ESM-1, ESM-2, T5, TCN, GEMSEC, CNN, amino acid composition and correlation approaches, amino acid composition and physico-chemical properties based approaches and topological and biological features based approaches. The classifiers include MLP, NB, DT, BERT with self-classifier, CNN, SVM, RF, GCN, BiGRU, ensemble GBDT, XGBoost, and BiLSTM.

Among all representation learning approaches, Node2vec is the most commonly used representation followed by BERT, Word2Vec, and FastText. Specifically, potential of Node2vec is explored with XGBoost (233, 234), and BiLSTM (235) classifiers for essential protein identification. Also combined representation from Node2Vec with CNN (236, 237) and Node2Vec with TCN (231) are used with MLP classifier for for essential protein identification. Despite being used in seven predictive pipelines for same task, not a single Node2vec based predictive pipeline have achieved state-of-the-art performance. Furthermore, BERT is used with SVM (141) and a self-classifiers (140) for protein family classification, whereas potential of BERT is explored with a self-classifier for G-protein coupled receptors identification and have achieved state-of-the-art performance (58). Beyond Node2vec and BERT, Word2vec representation is used with CNN classifier for protein family classification (226, 227). In addition, RotatE representation learning method is used with MLP classifier for protein similarity prediction and have achieved state-of-the-art performance (229). Apart from Node2vec, BERT and Word2vec approaches, FastText is used with CNN (228) and SVM (57) classifiers for SNARE and electron transport protein identification whereas combined representation from ESM 1, ESM 2 and T5 language model is utilized with hybrid (BiLSTM+BiGRU) classifier for intrinsically disordered protein identification (60). Additionally, GEMSEC with CNN classifier (59), DANE with hybrid (CNN+BiGRU) classifier (230), PseAAC with ensemble GBDT classifier (318), and OHE with GCN (317) and SVM (314) classifiers are used for essential protein identification. In contrast, amino acid composition and physico-chemical properties based approaches (313) and topological and biological features based approached (315, 316) are used with RF classifier for essential protein identification. In addition, Yue et al. (232) predictor is developed by using topolgical and biological features based approaches with a CNN classifier. Moreover, potential of amino acid composition and correlation based approaches is explored with stacked (RF+NB+DT) classifier for protein family classification and have achieved state-of-the-art performance (312). Similarly, PSSM representation is employed with CNN classifier for SNARE protein identification and achieved state-of-the-art performance (56). From all tasks in this goal, three tasks namely protein similarity prediction, SNARE protein, and intrinsically disordered protein identification offers some room for improvement. An in-depth analysis of these studies reveals that utilizing physico-chemical properties and occurrence frequencies based representation learning approaches combined with ensemble classifiers, can improve performance of under-performing tasks.

Table 8 presents performance metrics for 17 AI-driven applications including language models, word embeddings, and domain-specific based predictive pipelines which are designed for tasks related to protein modification prediction. This goal is oriented to binary classification applications that are developed by utilizing 11 unique representation learning approaches, and nine unique classifiers. The representation learning methods include BERT (21, 142, 144), ESM-1 (12, 13, 22), AlphaFold (12), T5 (14, 17, 20, 143), Transformer (15, 18, 19), integer encoding (16), Amino acid occurrence frequency based approaches (452), FastText (239), amino acid based approaches (144), ELMo (238), and GloVe (238). The classifiers include CNN, SVM, MLP, BiGRU, BiLSTM, Stacked LSTM, and self-classifiers of Transformer, T5, and BERT

For this goal, T5 is the most commonly used representation learning approach followed by transformer, BERT and ESM-1. Specifically, T5 language model based representation learning is used with CNN classifier for protein S-nitrosylation modification prediction (14), whereas potential of T5 language model is explored with a self-classifier for phosphorylation modification prediction, succinylation modification prediction and non-histone acetylation modification prediction (17, 20, 143). Among all T5 language model based predictive pipelines, three predictive pipelines have achieved state-of-the-art performance such as T5 representation with CNN classifier for protein S-nitrosylation modification prediction and T5 representation with a self-classifier for succinylation modification prediction and non-histone acetylation modification prediction. Apart from this, transformer is used with CNN for phosphorylation modification prediction (18), while transformer with self-classifier is used for phosphorylation modification prediction and serine phosphorylation modification prediction (15, 19). Amid all predictive pipelines based on transformer language model, transformer based predictive pipelines with self-classifier have attained state-of-the-art results across two tasks including phosphorylation modification prediction and serine phosphorylation modification prediction.

Furthermore, BERT representation learning approach is employed with CNN and a self-classifier for glycosylation, glycation and lysine phosphoglycerylation modification prediction (21, 142), respectively. Among all BERT based predictive pipelines, BERT with self-classifier have achieved state-of-the-art performance for lysine phosphoglycerylation modification prediction. Similarly, ESM-1 language model based representation learning is employed with CNN classifier for glycosylation, glycation and lysine phosphoglycerylation modification prediction (13, 22). ESM-1 language model based predictive pipelines have achieved state-of-the-art mode for glycosylation and glycation modification prediction. Additionally, combined representation from ESM-1 and AlphaFold language models is used with SVM classifier for glycosylation and glycation modification prediction (12). In addition, integer encoding is employed with CNN classifier for phosphorylation modification prediction (16). FastText is used with MLP classifier for protein S-sulfenylation modification prediction (239).

In addition, potential of combined representation from ELMo and GloVe is explored with stacked (LSTM + MLP) classifier for protein glutarylation modification prediction (238). Apart from this, amino acid occurrence frequency based representation learning approaches are explored with BiGRU + MLP classifier for protein S-sulfenylation modification prediction (452). Among all amino acid occurrence frequency based predictive pipelines, amino acid occurrence frequency based approaches with BiGRU classifier have achieved state-of-the-art performance for protein S-sulfenylation modification prediction task. Combined potential of BERT with amino acid based representation learning approaches is used with CNN + BiLSTM classifier for protein glutarylation modification prediction (144) and has achieved state-of-the-art results across protein glutarylation modification prediction task. An extensive analysis of this goal reveals that three tasks namely succinylation, S-nitrosylation, glycosylation and glycation modification prediction offer room for improvement. Considering performance trend of across different goal, potential of shallow neural network based word embeddings namely Word2vec, FastText, GloVe or graph neural network based embedding such as Node2vec, DeepWalk, LINE or HOPE can enhance the performance of under-performing tasks.

Table 9 presents performance metrics for 11 AI-driven applications that are developed using three different approaches namely (1) language models, (2) word embeddings, (3) domain-specific approaches, designed for tasks related to protein interaction prediction. This goal is oriented to interaction applications are developed by utilizing 37 unique representation learning approaches, and 37 unique classifiers. These unique representation learning approaches are T5, ESM-2, BERT, GAT, Node2Vec, TFIDF, Word2Vec, Score Matrix and Physico-chemical properties based approaches, RoBERTa, ESM-1, ELMO, Vision Transformer, PSSM, LINE, FastText, TransE, Transformer, DeepWalk, Physico-Chemical properties based approaches based representations, AlphaFold2, Amino Acid Occurance based representation learning approach, HIN2Vec, VGAEs, RDKit, GCN, CNN, MolFormer, OHE, ResNet, ECFP, Integer Encoding, GNN, SDNE, ANE, RandomWalk, Topo2Vec, and GraRep. In contrast, unique classifiers include XGBoost, CAPT5, GAT, GNN, MLP, SVM, LogR, CNN, Transformer, RF, LSTM, Louvain clustering, GCN, LR, BR, DTR, Core Attachments based clustering method, k-means, GVPConv, ResNet, BiLSTM, GAE, LightGBM, AdaBoost, CCA, MPNN, GRU, Transformer Decoder, BiGRU, cGAN, Affinity Regression and self-classifiers of ESM-1b, BERT, Transformer, ESM-2, T5, and MolFormer.

Among all representation approaches, Node2Vec emerges as most commonly used representation learning approach followed by BERT and Transformer. Specifically, Node2Vec is used with CNN, Louvain clustering, LSTM, SVM and Ensemble (GNN + MLP) classifiers for protein–protein interaction prediction (38, 45, 65, 245, 246) and potential of Node2Vec is also explored with GNN classifier Protein-RNA Interaction Prediction (255, 258). Moreover, Node2Vec is employed with ensemble (LR+BR+DTR+SVM) classifier and k-means clustering algorithm for protein complexes interaction prediction (248, 251). Similarly, Node2Vec is also employed with CCA, cGAN, and LightGBM classifiers for virus-host interaction prediction (51) and missing link prediction (266, 267), respectively. Despite being most commonly used representation learning, not a single Node2Vec based predictive pipelines manages to achieve state-of-the-art performance. In addition, combined potential of Node2Vec with four different representation learning approaches namely Word2Vec, PSSM, DeepWalk and physico-chemical properties based approaches is explored with LogR classifier for protein–protein interaction prediction (45, 244, 319) and residue-residue interaction prediction (252). Similarly, combined representation from Node2Vec and OHE is utilized with CNN classifier (260), and combined potential of Node2Vec with (Word2Vec+CNN, RDkit+ResNet) (262) and CNN+ECFP approaches (263) is explored with MLP classifier for compound-protein interaction prediction. Among these combined approaches, Node2vec and physico-chemical properties based representation learning approaches with LogR classifier has achieved state-of-the-art performance for residue-residue interaction prediction (252). Furthermore, BERT is the second most commonly used representation learning, which is employed with four distinct classifiers for four different tasks. Particularly, BERT is used with CNN and a self-classifier for nucleic acid-binding proteins identification (156, 158, 159). In addition, potential of BERT and a self-classifier is also explored for two other tasks namely protein–protein interaction prediction (145, 147) and contact prediction (154). Moreover, combined potential of BERT is explored with several other approaches as: BERT + Transformer + OHE along with CNN classifier for compound-protein binding affinity prediction (64, 163), BERT + TransE is used with GNN for contact prediction (185) and protein–protein interaction prediction (148), BERT + ESM-1 with a self-classifier (146), and BERT with Transformer + GAT-CNN classifier is used for protein–protein interaction prediction (43).

Beyond Node2Vec and BERT representation learning approaches, transformer is employed with a self-classifier and ResNet for three tasks namely protein–protein interaction prediction (149), contact prediction (155), and virus-host interaction prediction (47). Moreover, combined representation from transformer and GNN is used with a self-classifier for compound-protein interaction prediction (162). Among all transformer based predictive pipelines, transformer with a self-classifier has achieved state-of-the-art performance for contact prediction (150). Apart from this, Word2Vec is used with MLP classifier for protein–protein interaction prediction (242). Furthermore, Word2Vec is also used with LightGBM and BiLSTM classifiers for virus–host interaction prediction (48, 49). Word2Vec is also employed with GNN and RF classifiers for protein complexes identification (247) and protein–protein interaction prediction (254, 259), respectively. Among all Word2Vec based predictive pipelines, Word2Vec with BiLSTM manages to achieve top performing values for virus host interaction prediction (49). Apart from standalone potential of Word2Vec representation learning, combined potential of Word2Vec and VGAEs with a GAE classifier is explored for protein–RNA interaction prediction (257). Similarly, Word2Vec and SNDE combined used as representation learning approach with MLP classifier for phage-host interaction prediction (264), whereas combined representation learning from Word2Vec, RDkit, and GCN is used with transformer-decoder and MLP classifier for compound-protein interaction prediction (162). Among all Word2Vec based combined representation learning approaches, Word2Vec and SDNE representation with MLP classifier has achieved state-of-the-art performance for phage-host interaction prediction (264).

ESM-1 with a self-classifier is used for protein–protein interaction prediction (44). In addition, potential of ESM-1 is also explored with CNN and and GVPConv classifiers for contact prediction (151–153). In contrast, ESM-2 with GAT and BiLSTM classifier is used for protein–protein interaction prediction (36) and nucleic acid binding protein identification (160), whereas potential of ESM-2, AlphaFold, PSSM and MolFormer is explored with a self-classifier and BiLSTM for nucleic acid-binding proteins identification (157, 160), Compound-Protein Interaction (61) and Compound-Protein Binding Affinity Prediction (61). Similarly, potential of T5 is explored with RF, hybrid (XGBoost+CAPT5) and self-classifier for phage-host interaction prediction (164), protein–protein interaction prediction (35) and contact prediction (27) respectively. Among all ESM-1, ESM-2 and T5 language model based predictive pipelines, T5 with a hybrid (XGBoost+CAPT5) classifier has achieved state-of-the-art performance for protein–protein interaction prediction (35). In addition, RoBERTa is used with LogR classifier for protein–protein interaction prediction (42) and combined potential of RoBERTa with Integer Encoding is explored with BiGRU classifier for compound-protein binding affinity prediction (62), whereas AlphaFold2 and RDkit is used with a self-classifier for compound-protein interaction prediction (161) and compound-protein binding affinity prediction (161). Among both RoBERTa and AlphaFold2 language models based predictive pipelines, AlphaFold2 with a self-classifier has achieved state-of-the-art performance for compound-protein binding affinity prediction (161).

Furthermore, FastText is explored with Affinity Regression and hybrid (LSTM+CNN) classifiers for nucleic acid-binding proteins identification (253) and protein–protein interaction prediction (243). DeepWalk is used with clustering algorithm, GCN, and LogR classifier for protein complexes identification (249, 250) and Missing Link Prediction (268). In addition, ELMo is used with LogR classifier and LINE method is employed with RF classifier and combined potential of ELMo and vision transformer is explored with a self-classifier for protein–protein interaction prediction (46, 240, 241). Furthermore, HIN2Vec is used with SVM for protein–RNA interaction prediction (256), GraRep and RDkit representation is explored with XGBoost classifier for compound-protein interaction prediction (261), potential of Topo2Vec with SVM classifier is explored for missing link prediction (270). Beyond word embedding and language model based representation learning approaches, amino acid occurrence-based representation learning is also employed with GCN, SVM and hybrid (AdaBoost+SVM+RF) classifiers for protein–RNA interaction prediction (320) and virus host protein interaction prediction (50, 52). Similarly, score matrix and physico-chemical properties based approaches are utilized with CNN classifier for protein–protein interaction prediction (41). Among both, amino acid occurrence based representation learning approaches with GCN have achieved state-of-the-art performance for protein–RNA interaction prediction (320). In addition, combined potential of ANE + PSSM + RandomWalk representation learning is explored with a LogR classifier and TFIDF with SVM classifier (39), GAT with CNN classifier for Missing Link Prediction (269) and RDkit with hybrid MPNN and BiLSTM are used for Compound-Protein Interaction (321) for protein–protein interaction prediction. Moreover, ASNE+PSSM+Random Walk with LogR classifier is used for missing link prediction (265).

Detailed analysis of all these predictive pipelines indicates that there is a significant room of improvement in residue-residue interaction prediction, protein complexes identification and phage-host interaction prediction. Considering the performance trend for this goal, potential of LLMs such as T5, BERT, transformer with standalone or hybrid deep learning predictors such as CNN, BiLSTM, CNN+LSTM, CNN+BiGRU, and XGBoost+CAPT5 classifiers can enhance the performance of under-performing tasks.

Furthermore, Table 10 provides a high level overview of performance of 30 predictive pipelines that are developed using three different approaches namely (1) language models, (2) word embeddings, (3) domain-specific approaches, under the hood of peptide analysis goal. A detailed review of existing studies have identified 21 unique representation learning approaches namely Word2Vec, OHE, T5, ESM-1, Transformer, AlphaFold, BERT, IgFold, CTF, random embedding, TPC, motif features, LPE, matrix reduction based methods, amino acid occurrence based approaches, structural amino acid composition based approaches, amino acid composition based approaches, ProtDCal software based approaches, physico-chemical based representation approaches, k-Gap based approaches and amino acid structural and occurrence frequency based approaches. Moreover, these predictive pipelines have utilized 24 unique classifiers including LogR, SVM, SnTCN, GRU, CNN, LSTM, LightGBM, BiLSTM, RF, k-means clustering algorithm, scoring card method, BR, CC, MLkNN, GRNN, kNN, PNN, LR, ETC, XGB, CatBoost and self-classifiers of BERT, AlphaFold, and IgFold.

Among all representation learning approaches for this goal, amino acid composition approaches are most commonly used representation, followed by amino acid occurrence approaches. Amino acid composition approaches are used with RF, hybrid (SVM+RF) and k-means clustering algorithm for anti-microbial peptides identification (55, 330, 333–335) whereas, potential of amino acid composition approaches are also explored with BiLSTM, LightGBM, ensemble (GRNN+kNN+PNN+RF+SVM) classifiers and scoring card method for anti-cancer peptides identification (53, 322, 323, 325). Similarly, amino acid composition approaches are employed with RF and Voting (RF+ETC+XGB+LightGBM+CatBoost) classifiers for anti-inflammatory peptides identification (338, 339). Among all these predictive pipelines, amino acid composition approaches with RF classifier has achieved state-of-the-art performance for anti-inflammatory peptides identification (339). In addition, structural amino acid composition approaches with hybrid (CNN+LSTM+MAM) classifier and matrix reduction based approaches with RF classifier are used for anti-cancer peptides identification (326, 327). Similarly, amino acid occurrence approaches are used with ensemble (RF+BR+CC+MLkNN) classifier for anti-cancer peptides identification (324) and has achieved state-of-the-art performance. Moreover, amino acid structural and occurrence frequency based approaches with RF classifier and k-Gap based approaches with RF+ET classifier are employed for anti-inflammatory peptides identification (341, 342). In addition, potential of physico-chemical properties based approaches are explored with hybrid (LR+SVM) classifier and ProtDCal software based representation learning approaches are used with LSTM classifier for anti-microbial peptides identification (332) and anti-bacterial peptides identification (337), respectively. Moreover, combined potential of TPC and motif features is explored with SVM classifier for anti-inflammatory peptide identification (340).

Furthermore, standalone potential of OHE with LightGBM classifier and combined potential of OHE, BLOOSUM62, AAI, and PPC representation with a CNN classifier are explored for anti-microbial peptides identification (328, 331). Moreover, combined representation from OHE and LPE is used with ProtBERT classifier for signal peptides identification (336). Among all standalone and combined representation approaches, OHE with LightGBM classifier has achieved state-of-the-art performance for anti-microbial peptides identification (331). Besides this, Word2Vec is used with BiLSTM classifier for anti-bacterial peptides identification (54) and have achieved state-of-the-art performance. Moreover, potential of random embedding is explored with hybrid (CNN+BiLSTM+SVM) classifier for anti-microbial peptides identification (329) whereas potential of T5 representation with CNN classifier and ESM representation with BiLSTM classifier are explored for anti-microbial peptides identification (165) and signal peptides identification (166), respectively. Moreover, representation from transformer is used with GRU classifier and AlphaFold is employed with LogR classifier for secreted peptides prediction (167) and peptide-binding specificity prediction (170), respectively. Among all LLMs based predictive pipelines, transformer with GRU classifier and AlphaFold with LogR classifier have achieved state-of-the-art performance for secreted peptides prediction (167) and peptide-binding specificity prediction (170), respectively. Similarly, combined representation from AlphaFold, IgFold and BERT is used with self-classifier for antibody sequence infilling (169) and has achieved state-of-the-art performance. Moreover, combined potential of BERT language model, Word2Vec and CTF representation is explored with SnTCN classifier for anti-inflammatory peptides identification (168).

From all tasks in this goal, two tasks namely antibody sequence in-filling and anti-inflammatory peptides offer room for improvement. Based on current performance trends, potential of shallow neural network based word embeddings such as Word2Vec, FastText, or GloVe or graph embeddings such as LINE and HOPE can be explored with standalone deep neural networks or hybrid frameworks to raise the performance of these under-performing tasks.

Table 11 provides performance metrics for 12 AI-driven (language models, word embeddings) applications that are designed for drug analysis related tasks. This goal is oriented to binary classification and regression prediction applications that are developed by utilizing 13 unique representation learning approaches including transformer, BERT, ESM-2, BiLSTM, RoBERTa, ALBERT, Node2Vec, Word2Vec, Random Walk, PSSM, RDkiT, Mol2Vec, and algebraic graph features. Moreover, these predictive pipelines make use of 10 unique classifiers namely GCN, BiGRU, MLP, CNN, GRU and self-classifiers of BERT, Transformer, ESM-2, RoBERTa, and ALBERT.

Among all representation learning approaches, BERT and Transformer are most commonly used, followed by Node2Vec, RoBERTa and Word2Vec. Specifically, BERT is used with a self-classifier for drug-target binding affinity prediction (174). Moreover, combined potential of BERT and RoBERTa representation with GCN classifier and representation from all three BERT, RoBERTa, and ALBERT is used with a self classifier for a single task namely drug-target binding affinity prediction (175, 176). Among all BERT based predictive pipelines, BERT with a self-classifier has achieved state-of-the-art performance for drug-target binding affinity prediction (174). Furthermore, potential of transformer is explored with a self-classifier for drug-protein interaction prediction (171, 173) and has achieved state-of-the-art performance. Apart from this, combined representation from transformer and ESM-2 language model is utilized with a self-classifier for drug-protein interaction prediction (172). Moreover, combined potential of Node2Vec and Word2Vec with BiGRU classifier is explored for drug-protein interaction prediction (271) and drug-target binding affinity prediction (271). In addition, combined representation learning approach of Node2Vec and Random Walk with MLP classifier is used for drug-protein interaction prediction (272).

Furthermore, potential of BiLSTM is explored with a CNN classifier for drug-protein interaction prediction (343). Besides these representation learning approaches, combined potential of PSSM, RDkit and Mol2Vec are employed with hybrid (CNN+GRU) classifier for drug-target binding affinity prediction (344). Moreover, algebraic graph features based representation is used with MLP classifier for drug-target binding affinity prediction (345).

An in-depth analysis of existing predictive pipelines indicates that there is a significant room of improvement for drug-target binding affinity prediction. By observing performance trends across different goals, potential of heterogeneous graph transformers along with deep learning classifiers such as CNN or BiLSTM can enhance the performance of under-performing task.

Table 12 provides performance metrics for 31 AI-driven (language models, word embeddings, and domain specific) applications that are designed for gene analysis related tasks. This goal is oriented to binary, multi-class, and multi-label classification and regression prediction applications are developed by utilizing 21 unique representation learning and 20 unique classifiers. These representation learning approaches include GPT (177), Node2Vec (273, 277–282, 284–288, 290, 291, 293, 294), CP-N3 (346), LINE (185, 274, 293), SDNE (185, 274), HOPE (274), Struc2Vec (275), TransE (276), ComplEx (276), DistMult (276), Graph Transformer Network (178), DeepWalk (274, 280, 292, 295), Word2Vec (281, 287), Hyper2Vec (282), Transformer (179), RotateE (283), Opa2Vec (284), Random Watcher Walker (RW2) (289), BERT (180), Bias Random Walk (296) and ESM-1 (181). Unique classifiers involve IBK (273), LSTM (346), RF (274), LightGBM (274, 280), XGBoost (275), MLP (277, 279, 282, 289), LogR (278), SVM (281, 284–286, 288, 290, 295), Inductive Matrix Completion Algorithm (179), GraphSAGE (283), CNN (287), GCN (292, 347), GLM (294), ET (296), DBN (291), Cosine Similarity (293) and self classifier of GPT (177), Graph Transformer Network (178), BERT (180) and ESM-1 (181).

A comprehensive analysis reveals that Node2Vec is most commonly used representation learning approach followed by LINE and DeepWalk. Specifically, Node2Vec is used with IBK classifier for gene phenotype prediction (273) whereas potential of Node2Vec is also explored with MLP, LogR, SVM, DBN and GLM classifiers for disease genes prediction (277–279, 285, 286, 288, 290, 291, 294). In addition, combined potential of Node2Vec, DeepWalk, LINE and SDNE is explored with LightGBM classifier for disease genes prediction (280). Moreover, combined representation from Node2Vec and Word2Vec is used with SVM and CNN classifiers for disease genes prediction (281, 287). Whereas, combined potential of Node2Vec and Hyper2Vec is employed with MLP classifier and combined potential of Node2Vec and Opa2Vec is explored with SVM classifier for disease genes prediction (282, 284). Among all Node2Vec based predictive pipelines, not a single pipeline has achieved state-of-the-art performance. In addition, DeepWalk with GCN and SVM classifiers is employed for disease genes prediction (292), and essential genes identification (295). Among both, Deepwalk with SVM classifier has achieved state-of-the-art performance for essential genes identification (295).

Moreover, Stru2Vec is used with XGBoost classifier, TransE, ComplEx, and DistMult are employed with MLP classifier and RotatE is utilized with GraphSAGE classifier for disease genes prediction (275, 276, 276, 283). Furthermore, Random Watcher Walker (RW²) is employed with GCN and MLP classifier and potential of Bias Random Walk is explored with ET classifier for disease genes prediction (289) and essential genes identification (296), respectively. In addition, combined potential of LINE, SDNE and HOPE is also explored with ensemble (RF+LGBM) classifier for disease genes prediction (274). Apart from this, CP-N3 representation is used with LSTM classifier for disease genes prediction and has achieved state-of-the-art performance (346). Furthermore, potential of Transformer based representation is explored with Inductive Matrix Completion Algorithm whereas Graph Transformer Network and BERT is employed with a self-classifier for disease genes prediction (178–180). Moreover, GPT is used with a self-classifier for gene phenotype prediction and has achieved state-of-the-art performance (177). ESM-1 is also used with a self-classifier for prokaryotic gene prediction (181) and has achieved state-of-the-art performance.

From all these tasks, essential genes identification has significant room for improvement. Taking into account the performance trends of different tasks in this goal, potential of LLMs such as ESM-1 and GPT with a self-classifier can enhance the performance of under-performing tasks.

Table 13 provides performance metrics for 19 AI-driven (language models, word embeddings, and domain specific) applications that are designed for protein properties prediction related tasks. This goal is oriented to binary, multi-class, and multi-label classification and regression prediction applications are developed by utilizing seven unique representation learning approaches namely Transformer (155, 185), CTAPAAC (348), ESM-1 (182), T5(27, 187), RoBERTa (183), ESM-2 (150, 184) and BERT (154, 186, 188, 189). Moreover, these predictive pipelines have used 10 unique classifiers including RF (348), GCN (182), CNN (184), BiLSTM (189), ResNet (155) and self-classifiers of ESM-2 (150), T5 (27, 187), RoBERTa (445), Transformer (185) and BERT (154, 186, 188).

Among all representation learning approaches, BERT is the most commonly used followed by transformer. Specifically, BERT is employed with a self-classifier for stability prediction (154, 186), and fluorescence prediction (154, 186, 188), and potential of BERT representation is also explored with BiLSTM classifier for domain boundary prediction (189). Among all BERT based predictive pipelines, BERT representations with BiLSTM have achieved state-of-the-art performance. Similarly, Transformer is used with a self-classifier for solubility prediction (150), stability prediction (150, 155, 185) and fluorescence prediction (155, 185). For all three tasks, transformer based predictive pipeline with a self classifier manages to achieve top performing values. Moreover, T5 is employed with a self-classifier for solubility prediction (27), thermophilicity prediction (187) and fluorescence prediction (27), and has achieved state-of-the-art performance for thermophilicity prediction (187). Beyond BERT and T5 language model, potential of ESM-2 representation is utilized with CNN for stability prediction (184), and also employed with self-classifier for both stability prediction (150) and solubility prediction (150). Additionally, ESM-1 representation is utilized with GCN classifier for solubility prediction (182). In addition, RoBERTa is used with a self-classifier for solubility prediction (183), and potential of CTAPAAC representation is also explored with RF classifier for solubility prediction (348).

From all tasks in this goal, fluorescence prediction offers a significant potential for improvement. Considering the performance trends across different goals, potential of shallow and graph neural network based word embedding such as FastText, Word2Vec, DeepWalk, and Node2Vec with deep learning classifiers including CNN, BiLSTM and BiGRU can improve the performance of under-performing task.

Table 14 summarizes 54 predictive pipelines related to protein function and structure prediction goal. For this goal, predictive pipelines have used 19 unique representation learning approaches namely transformer, ESM-1, ESM-2, BERT, T5, AlphaFold, KG, ELMo, FastText, Word2Vec, Node2Vec, Mashup, GRU, OHE, XLNet, ALBERT, ELECTRA, CNN, and Transformer-XL. Moreover, these predictive pipelines have employed 23 unique classifiers including ANN, CNN, GNN, BiLSTM, DNN, RF, GAN, GCN, MLP, SVM, GVP, LogR, MCNN and self-classifier of transformer, BERT, ESM-1, ESM-2, T5, AlphaFold, XLNet, ALBERT, ELECTRA, and Transformer-XL.

For this goal, T5 is the most commonly used followed by BERT and ESM-2. Specifically, T5 is used with a self-classifier for protein function identification (195, 203), and fold prediction (27, 214). Moreover, potential of T5 is explored with GCN and LogR classifiers for protein function identification (199) and remote homology detection (215). Similarly, T5 is used with CNN classifier for structure prediction (208, 211, 212) and remote homology detection (216). Among all T5 based predictive pipelines, T5 with CNN classifier has achieved state-of-the-art performance for remote homology detection (216). In addition, combined potential of T5 and ESM-2 is explored with GNN classifier for protein function identification (192). Furthermore, BERT is used with a self-classifier for structure prediction (154, 188) and remote homology detection (154, 188). Moreover, potential of BERT is also explored with LogR and GVP classifiers for protein function identification (186, 202) whereas, BERT is also employed with CNN and BiLSTM classifiers for structure prediction (209) and remote homology detection (136), respectively. In addition, combined potential of T5 and BERT representation with self-classifier is explored for protein function identification (201). Beyond BERT and T5 language models, ESM-2 is employed with self-classifier for protein function identification (150, 190, 193, 194, 200), structure prediction (60, 150, 210), and fold prediction (150) whereas, potential of ESM-2 is explored with RF classifier for protein function identification (197). Moreover, combined potential of T5, BERT, ESM-2, and OHE is explored with DNN classifier for protein function identification (196). In addition, potential of ESM-1 is explored with GAN, GCN and GNN classifiers for protein function identification (198, 204, 206). Apart from this, transformer based representation learning is used with a self-classifier for protein function identification (185), structure prediction (155, 213) and remote homology detection (213). Besides these, combined potential of Transformer-XL, XLNet, BERT, ALBERT, ELECTRA and T5 is explored with a self-classifier for structure prediction (27). Among ESM-1, ESM-2 and transformer based representation approaches, ESM-2 with a self classifier has achieved state of the art performance for all three tasks protein function identification (150), structure prediction (150) and fold prediction (150). Besides this, combined potential of transformer and ESM-1 representation is used with CNN and self-classifier for protein function identification (191, 205). Moreover, AlphaFold is employed with a self-classifier for protein function identification (207). whereas combined potential of AlphaFold and BERT is explored with SVM classifier for TRP channels classification (217) and has achieved state-of-the-art performance.

Furthermore, FastText representation with RF and MLP classifiers while ELMo representation is used with MLP and hybrid (SVM+LogR) classifiers are employed for protein function identification (297, 298, 301, 302, 305). Moreover, Word2Vec is employed with hybrid (BiLSTM+MCNN) classifier for protein function identification (303). Apart from this, combined potential of Word2Vec and Node2Vec with RF classifier is also explored for protein function identification (300). Beyond these representation learning approaches, Mashup is employed with SVM classifier for protein function identification (299). Similarly, combined potential of Mashup and Node2Vec is also explored with SVM classifier, KG representation is used with ANN classifier for protein function identification (304, 349). Besides these, combined representation from CNN and GRU is used with MLP classifier for remote homology detection (350).

A comprehensive analysis of existing studies for this goal indicates that there is a significant room for improvement in fold prediction, remote homology detection and TRP channels classification. By observing performance trends for this goal, potential of shallow neural network based word embedding such as Word2Vec and FastText with graph based deep learning classifiers such as GNN can raise the performance of under-performing tasks.

Table 15 summarizes the performance of 13 predictive pipelines for protein sub-cellular localization prediction goal. This goal predictive pipelines have used 10 unique representation learning approaches namely, ESM-2 (23), amino acid properties based representation approaches (25), OHE (26), T5 (27, 28), Node2Vec (30, 31), Word2Vec (32), ELMo (33, 306), amino acid properties and composition based representation approaches (351), BERT (218, 219), and PSSM (219). In addition, nine unique classifiers are identified including GAT (25), CNN (26, 33, 219, 306), LSTM (30, 31), SVM (32, 351), BiLSTM (306) and self-classifier of BERT (218), Transformer (219), ESM-2 (23) and T5 (27, 28).

Among all representation learning approaches, Node2Vec is used with LSTM classifier for protein subcellular localization identification (30, 31), and ELMo is employed with CNN and hybrid (CNN + BiLSTM) classifiers for protein subcellular localization identification (33, 306). In addition, potential of T5 representation is also explored with a self-classifier for protein subcellular localization identification (27, 28). Whereas, BERT with its a self-classifier is used for protein submitochondrial localization identification (218) and combined potential of BERT and PSSM is explored with CNN and Transformer self-classifier for subchloroplast localization identification (219). Among all four representation approaches, BERT has achieved state-of-the-art performance for subchloroplast localization identification (219). Furthermore, Word2Vec with SVM classifier and OHE with CNN classifier are employed for protein subcellular localization identification (26, 32). In addition, potential of ESM-2 representation is explored with self-classifier for protein subcellular localization identification (23) and has achieved state-of-the-art performance. Apart from word embedding and LLMs based predictive pipeline, amino acid properties based representation approaches are used with GAT classifier for protein subcellular localization identification (25). Whereas, amino acid properties and composition based representation approaches are employed with SVM classifier for protein submitochondrial localization identification (351). Among both, amino acid properties and composition based representation approaches with SVM classifier manages to achieve top performing values.

From all task of this goal, subchloroplast localization identification offers significant room for improvement. Analysing the performance trends of various tasks in this goal, amino acid properties and composition based representation approaches with machine learning classifiers such as SVM and GAT classifiers can improve the performance of under-performing task.

Table 16 provides a high level overview of 11 predictive pipelines related to two goals namely mutation analysis and disease analysis.

For mutation analysis goal, five predictive pipelines for three protein sequence analysis tasks have used five unique representation namely T5, AlphaFold, ESM-1, GNN and BERT. Moreover, these predictive pipelines have utilized five different classifiers namely GBDT, LR, GCN, GMM and MLP. T5 emerges as the most frequently used representation learning approach for this goal. Specifically, T5 is used with LR and GMM classifiers for variant effect prediction (224) and mutation prediction (220), respectively. Among both, T5 with GMM classifier has achieved state-of-the-art performance for mutation prediction (220). Furthermore, AlphaFold is used with GCN classifier, combined representation from ESM-1 and MSA transformer is employed with self-classifier and combined potential of BERT and GNN is also explored with GBDT classifier for mutation effects prediction (221–223). An in-depth analysis of these predictive pipelines indicates that there is a significant room for improvement in mutation effects prediction. Taking into account the performance trends across different goals, potential of shallow neural network based word embeddings such as Word2Vec, Node2Vec and FastText can be used with deep learning classifiers such as BiLSTM and CNN to raise the performance of under-performing task.

For disease analysis goal, six predictive pipelines for different tasks have employed six unique representation learning approaches namely FastText, Word2Vec, Node2Vec, AlphaFold, GNN and ESM-2. Overall, these predictive pipelines have utilized seven different classifiers including SVM, RF, PNN, KNN, BiLSTM, GNN and LogR.. Among all representation learning approaches, FastText and Word2Vec are most commonly used for this goal. FastText is explored with SVM and ensemble |$(RF+SVM+PNN+KNN)$| classifiers for tumour necrosis factors identification (308) and malaria parasite identification (307), respectively. Moreover, Word2Vec is used with RF classifier for COVID-19 virus classification (309) and |$B/T$| Cell receptor sequences analysis (311). Furthermore, combined potential of Node2Vec and GNN is used with RF classifier for vascular calcification (310). Beyond word embedding based representation learning approaches, combined potential of ESM-2+AlphaFold representation is also explored with BiLSTM and GNN for B-Cell epitopes identification (225). From all these tasks, vascular calcification, B-Cell epitopes identification and |$B/T$| Cell receptor sequences analysis indicate a significant room for improvement. By analysing the performance patterns across different goals, potential of amino acid composition approaches and BERT with deep learning classifiers such as BiLSTM and LightGBM can raise the performance of under-performing tasks.

To sum it up, a comprehensive analysis of advanced predictive pipelines based on word embeddings, language models, and domain-specific representation learning methods reveals intriguing trends. Among 63 protein sequence analysis tasks classified into 11 main biological goals, 34 tasks involve binary classification, nine involve interaction prediction, nine involve multi-class classification, two involve multi-label classification, and seven involve regression. Two protein sequence analysis tasks belong to more than one task type such as gene phenotype prediction and protein subcellular localization prediction involve multi-class and multi-label classification applications. In total, 84 distinct representation learning methods and 67 predictive algorithms are employed to develop robust predictive pipelines for these tasks. Language model-based representation learning strategies and deep learning classifiers consistently achieve superior performance across the majority of tasks within these 11 biological goals. Researchers are encouraged to investigate capabilities of cutting-edge transformer based language models such as hierarchical and heterogeneous Graph transformers, GPT-4, and hybrid representation learning approaches. Furthermore, integrating these models with advanced ensemble machine learning or deep learning classifiers may enhance performance for various classification, regression, and clustering tasks.

Publisher and journal-wise distribution of research articles

This section provides comprehensive overview of publication venue distribution for 295 protein sequence analysis studies across different publishers, journals, and conferences. Selection of suitable publication venue for interdisciplinary AI-driven protein sequence analysis research is important. There exists three primary categories of publication venues namely (1) Core AI based publication venues emphasizes mathematical foundations and technical advancements in AI algorithms, (2) Biological publication venues focuses on biological significance and novelty of presented research findings, (3) Hybrid publication venues bridges the gap by integrating both AI and biological approaches. Researchers often face desk rejections when targeting core AI or biology venues due to their narrow disciplinary focus. This analysis emphasizes strategic selection of hybrid publication venues catering to interdisciplinary nature of research. Although various tools have been developed for venues identification, but this study provides in-depth analysis to target resource for researchers by identifying diverse venues which have published applications of word embeddings and LLMs for protein sequence analysis.

Table 17 presents comprehensive overview of 295 protein sequence analysis studies disseminated across various academic platforms including 87 journals, 11 conferences, six transactions, and three pre-print repositories. Within journals, highest publication frequency is observed in Briefings in Bioinformatics, followed by Bioinformatics, Computers in Biology and Medicine, and BMC Bioinformatics. Among 11 conferences, specifically IEEE International Conference on Bioinformatics and Biomedicine (BIBM) leads in publications and have published 11 articles whereas each of remaining conferences have only published one article. These conferences include IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), Asia Conference on Advanced Robotics, Automation, and Control Engineering (ARACE), International Joint Conference on Neural Networks (IJCNN), IEEE International Conference on Tools with Artificial Intelligence (ICTAI), Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), International Conference on Electrical Engineering and Information & Communication Technology (ICEEICT), and Bioinformatics Research and Applications International Symposium (ISBRA). Among all transactions publications, ACM Transactions on Computational Biology and Bioinformatics is predominant, followed by IEEE Transactions on Emerging Topics in Computational Intelligence, IEEE Transactions on Neural Networks and Learning Systems, IEEE Transactions on NanoBioscience, IEEE Transactions on Pattern Analysis and Machine Intelligence, and IEEE Transactions on Computational Social Systems. Taking into account the fast-paced nature of research, researchers have also considered rapid dissemination platforms and published 40 studies on these platforms namely BioRxiv, medRxiv, and arXiv.

Figure 8 further elaborates distribution of these studies across 31 publishers including Oxford University Press (https://academic.oup.com/), Mary Ann Liebert, Inc. (https://www.liebertpub.com/), Elsevier (https://www.elsevier.com/), Wiley Online Library, Springer (https://www.springer.com/in), ACS Publications (https://pubs.acs.org/), Nature Publishing Group UK London (https://www.iabuk.com/member-directory/nature-publishing-group), Public Library of Science San Francisco, CA USA (https://plos.org/), IEEE (https://www.ieee.org/), MDPI (https://www.mdpi.com/), Cold Spring Harbor Laboratory Press (https://www.cshlpress.com/), Pre- print (https://arxiv.org/), Frontiers Media SA (https://research.monash.edu/en/activities/frontiers-media-sa-publisher), Frontiers (https://www.frontiersin.org/), ACM (https://www.acm.org/publications), Taylor & Francis (https://taylorandfrancis.com/), eLife Sciences Publications Limited (https://elifesciences.org/), Curran Associates Inc. (https://www.proceedings.com/), AAAS (https://www.aaas.org/journals), PeerJ Inc. (https://peerj.com/), CIE (https://cie.co.at/publications), Chinese Medical Journals Publishing House Co. Ltd (https://journals.lww.com/cmj/pages/default.aspx), University of Tehran (https://www.nhbs.com/shop/publisher/university-of-tehran), PMLR (https://proceedings.mlr.press/), National Academy of Sciences (https://www.nationalacademies.org/publications), American Medical Informatics Association (https://amia.org/news-publications/journals), Duzce University (https://doaj.org/toc/2148-2446), SCITEPRESS-Science and Technology Publications (https://www.scitepress.org/HomePage.aspx), NeurIPS Proceedings (https://papers.nips.cc/), Semantic Scholar (https://www.semanticscholar.org/about/publishers), and ICLR (https://iclr.cc/).

It is worth noting that 170 of the 295 studies are published by Oxford University Press, Springer, Elsevier, and IEEE, whereas Nature Publishing Group UK London, MDPI, Cold Spring Harbor Laboratory Press, Pre-print, and Frontiers Media SA have collectively published 76 studies. Remaining 49 studies are published by Mary Ann Liebert, Inc., Wiley Online Library, ACS Publications, Public Library of Science San Francisco, CA USA, Frontiers, ACM, Taylor & Francis, eLife Sciences Publications Limited, Curran Associates Inc., AAAS, PeerJ Inc., CIE, Chinese Medical Journals Publishing House Co., Ltd, University of Tehran, PMLR, National Academy of Sciences, American Medical Informatics Association, Duzce University, SCITEPRESS-Science and Technology Publications, NeurIPS Proceedings, Semantic Scholar, and ICLR. In summary, among 295 protein sequence analysis studies, 213 are journal articles, 22 are conference papers, 20 are transaction papers, and 40 are pre-print studies, published by 31 different publishers. This detailed analysis highlights extensive and diverse research landscape in field of protein sequence analysis and spans multiple publication platforms and venues.

Discussion

This study sets a stage for AI-driven protein sequence analysis by performing large scale literature of 22 distinct word embeddings methods and 15 LLMs based 295 distinct scientific studies. In total 22 distinct word embeddings and 15 language models based predictive pipelines are designed by developing datasets from 100 distinct databases. Among these databases, 32 databases do not exist anymore while 68 databases are publicly available. These databases are always updated with new sequences information on daily, weekly or monthly bases and can be utilized to develop new datasets. Although for distinct types of tasks many datasets are publicly available but those datasets may have less number of sequences and deep learning models produce better performance when they are trained on large sequences data. However, development of new datasets leads toward inconsistency in predictive pipelines performance comparison. As an example in total 22 distinct word embeddings based predictive pipelines are evaluated on 165 distinct datasets of 32 different protein sequence analysis tasks, while 13 language models based predictive pipelines are evaluated on 328 datasets of 47 different tasks. Both types of predictive pipelines are evaluated only on two common datasets. This analysis reveals that predictive pipelines are not evaluated on same benchmark datasets and their is need of comparative studies that benchmark performance values of these predictive pipelines across same benchmark datasets.

In addition to comparative study, there is a need to standardize dataset utilization to streamline the development of new predictors. Researchers should develop new datasets but they must report their predictors performance on existing datasets as well. An other solution is to benchmark existing predictors performance on newly developed dataset and compare proposed predictor performance with them as well.

However, the majority of researchers do not make their predictors source codes publicly available, which makes challenging to benchmark the predictors performance on newly developed datasets. A comprehensive analysis of AI-driven protein sequence analysis studies reveals that while developing predictive pipelines researchers have primarily focused on two key components: representation learning methods and predictors (classifiers, regressors, similarity computation methods). Within representation learning landscape, 22 distinct word embedding methods and 15 language models have been utilized. At the predictor level, researchers have employed eight different machine learning algorithms and 15 deep learning techniques to build the pipelines. In AI-driven protein sequence analysis predictive pipelines, researchers have predominantly focused on leveraging either word embedding methods or language models. However, the performance of these pipelines could be significantly improved by harnessing the combined strengths of both word embedding techniques and language models simultaneously. Moreover, only a limited number of word embeddings and language models have been investigated for specific tasks. For instance, in protein–protein interaction prediction task, only six word embedding methods and six language models have been explored. Beyond these methods, potential of an additional nine word embedding techniques and five LLMs is investigated within DNA and RNA sequence analysis. However, these word embeddings and language models remain unexplored in predictive pipelines for protein sequence analysis. The unexplored word embeddings are MetaGraph2Vec (454), HAKE (229), HOPE (274, 455), Laplacian eigen maps (456), Locally linear embedding (456), RWR (457), SocDim (458, 459), SVD (460, 461), and Graph2vec (462). Moreover, unexplored language models are BigBird (463), ELECTRA (464, 465), Heterogeneous Graph Transformer (466), LongFormer (463), Transformer-XL (467), and ULMFiT (468, 469). By leveraging additional word embedding methods and LLMs can provide new insights and enhance accuracy in AI-driven protein sequence analysis tasks. One of the primary objectives of this study is to provide a strong foundation for researchers to further investigate the potential of various word embedding methods and language models across tasks where their applicability has yet to be fully explored.

Declaration of generative AI and AI-assisted technologies in the writing process

Authors have utilized Grammarly to deal with language and grammar issues, and ChatGPT to assist with outlining, understanding various studies, and expanding concepts during preparation of this work. After these tools utilization, authors have reviewed and edited content as necessary, and take full responsibility for final publication.

Acknowledgement

We gratefully acknowledge the support of German Research Center for Artificial Intelligence for this research.

Funding

No funding source is available.

Competing interests

Authors declare that there is no known competing financial interest or personal relationships which could have influenced this article.

References