Backbones

Backbones are pretrained foundation models. They are specified with the --model.backbone argument in the CLI or in the model.backbone section of a configuration file.

AIDO.ModelGenerator wraps messy foundation models in a standardized interface, allowing them to be applied to finetuning and inference tasks without any code, and even fused for multi-modal tasks. Backbones are also interchangeable, making it simple to run benchmarks and create leaderboards so you can find the best model for your task.

Many backbones come with options for parameter-efficient finetuning (PEFT) methods, low-memory checkpointing, and small-scale debugging models to assist with developing on large-scale foundation models.

This reference overviews the available no-code backbones. If you would like to integrate new backbones, see Experiment Design.

# Example Backbone Configuration
model:
  class_path: modelgenerator.tasks.SequenceRegression
  init_args:
    backbone:
      class_path: modelgenerator.backbones.aido_rna_1b600m_cds
      init_args:
        max_length: 1024
        use_peft: true
        save_peft_only: true
        lora_r: 32
        lora_alpha: 64
        lora_dropout: 0.1
        lora_target_modules:
        - query
        - value
        config_overwrites:
          hidden_dropout_prob: 0.1
          attention_probs_dropout_prob: 0.1
        model_init_args: null
data:
  ...
trainer:
  ...

DNA

modelgenerator.backbones.aido_dna_7b

Bases: GenBioBERT

AIDO.DNA model with 7B parameters pretrained on 10.6B nucleotides from 796 species in the NCBI RefSeq database.

Note

• Mauscript: Accurate and General DNA Representations Emerge from Genome Foundation Models at Scale
• Model Card: AIDO.DNA-7B
• Weights: genbio-ai/AIDO.DNA-7B

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.DNA-7B

modelgenerator.backbones.aido_dna_300m

Bases: GenBioBERT

AIDO.DNA model with 300M parameters pretrained on 10.6B nucleotides from 796 species in the NCBI RefSeq database.

Note

• Mauscript: Accurate and General DNA Representations Emerge from Genome Foundation Models at Scale
• Model Card: AIDO.DNA-300M
• Weights: genbio-ai/AIDO.DNA-300M

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.DNA-300M

modelgenerator.backbones.enformer

Bases: Enformer

Enformer model

Note

• Mauscript: Effective gene expression prediction from sequence by integrating long-range interactions
• GitHub: lucidrains/enformer-pytorch
• Model Card: EleutherAI/enformer-official-rough
• Weights: EleutherAI/enformer-official-rough

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	196608
frozen	bool	Whether to freeze model.	False
delete_crop_layer	bool	Whether to delete cropping layer.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	EleutherAI/enformer-official-rough

modelgenerator.backbones.borzoi

Bases: Borzoi

Borzoi model

Note

• Mauscript: Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation
• GitHub: johahi/borzoi
• Weights: johahi/borzoi-replicate-0

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	524288
frozen	bool	Whether to freeze model.	False
delete_crop_layer	bool	Whether to skip cropping layer.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	johahi/borzoi-replicate-0

modelgenerator.backbones.flashzoi

Bases: Borzoi

Flashzoi model

Note

• Mauscript: Flashzoi: A fast and accurate model for predicting RNA-seq coverage from DNA sequence
• GitHub: johahi/flashzoi
• Weights: johahi/flashzoi-replicate-0

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	524288
frozen	bool	Whether to freeze model.	False
delete_crop_layer	bool	Whether to skip cropping layer.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	johahi/flashzoi-replicate-0

RNA

modelgenerator.backbones.aido_rna_1b600m

Bases: GenBioBERT

SOTA AIDO.RNA model with 1.6B parameters pretrained on 42M ncRNAs in the RNACentral database.

Note

• Mauscript: A Large-Scale Foundation Model for RNA Function and Structure Prediction
• Model Card: AIDO.RNA-1.6B
• Weights: genbio-ai/AIDO.RNA-1.6B

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.RNA-1.6B

modelgenerator.backbones.aido_rna_1b600m_cds

Bases: GenBioBERT

SOTA AIDO.RNA model with 1.6B parameters adapted from aido_rna_1b600m by continued pretrained on 9M coding sequence RNAs from organisms in ENA.

Note

• Mauscript: A Large-Scale Foundation Model for RNA Function and Structure Prediction
• Model Card: AIDO.RNA-1.6B-CDS
• Weights: genbio-ai/AIDO.RNA-1.6B-CDS

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.RNA-1.6B-CDS

modelgenerator.backbones.aido_rna_650m

Bases: GenBioBERT

AIDO.RNA model with 650M parameters pretrained on 42M ncRNAs in the RNACentral database.

Note

• Mauscript: A Large-Scale Foundation Model for RNA Function and Structure Prediction
• Model Card: AIDO.RNA-650M
• Weights: genbio-ai/AIDO.RNA-650M

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.RNA-650M

modelgenerator.backbones.aido_rna_650m_cds

Bases: GenBioBERT

AIDO.RNA model with 650M parameters adapted from aido_rna_650m by continued pretrained on 9M coding sequence RNAs from organisms in ENA.

Note

• Mauscript: A Large-Scale Foundation Model for RNA Function and Structure Prediction
• Model Card: AIDO.RNA-650M-CDS
• Weights: genbio-ai/AIDO.RNA-650M-CDS

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.RNA-650M-CDS

modelgenerator.backbones.aido_rna_300m_mars

Bases: GenBioBERT

AIDO.RNA model with 300M parameters pretrained on 886M RNAs in the MARS dataset.

Note

• Mauscript: A Large-Scale Foundation Model for RNA Function and Structure Prediction
• Model Card: AIDO.RNA-300M-MARS
• Weights: genbio-ai/AIDO.RNA-300M-MARS

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.RNA-300M-MARS

modelgenerator.backbones.aido_rna_25m_mars

Bases: GenBioBERT

AIDO.RNA model with 25M parameters pretrained on 886M RNAs in the MARS dataset.

Note

• Mauscript: A Large-Scale Foundation Model for RNA Function and Structure Prediction
• Model Card: AIDO.RNA-25M-MARS
• Weights: genbio-ai/AIDO.RNA-25M-MARS

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.RNA-25M-MARS

modelgenerator.backbones.aido_rna_1m_mars

Bases: GenBioBERT

AIDO.RNA model with 1M parameters pretrained on 886M RNAs in the MARS dataset.

Note

• Mauscript: A Large-Scale Foundation Model for RNA Function and Structure Prediction
• Model Card: AIDO.RNA-1M-MARS
• Weights: genbio-ai/AIDO.RNA-1M-MARS

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.RNA-1M-MARS

Protein

modelgenerator.backbones.aido_protein_16b

Bases: GenBioFM

AIDO.Protein model with 16B parameters pretrained on 1.2T amino acids from UniRef90 and ColabFoldDB.

Note

• Mauscript: Mixture of Experts Enable Efficient and Effective Protein Understanding and Design
• Model Card: AIDO.Protein-16B
• Weights: genbio-ai/AIDO.Protein-16B

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.Protein-16B

modelgenerator.backbones.aido_protein_16b_v1

Bases: GenBioFM

AIDO.Protein model with 16B parameters adapted from aido_protein_16b by continued pretrained on 100B amino acids from UniRef90.

Note

• Mauscript: Mixture of Experts Enable Efficient and Effective Protein Understanding and Design
• Model Card: AIDO.Protein-16B-v1
• Weights: genbio-ai/AIDO.Protein-16B-v1

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.Protein-16B-v1

modelgenerator.backbones.esm2_15b

Bases: ESM

ESM2 15B model

Note

• Mauscript: Evolutionary-scale prediction of atomic level protein structure with a language model
• GitHub: facebookresearch/esm
• Model Card: facebook/esm2_t48_15B_UR50D
• Weights: facebook/esm2_t48_15B_UR50D

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	None
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	facebook/esm2_t48_15B_UR50D

modelgenerator.backbones.esm2_3b

Bases: ESM

ESM2 3B model

Note

• Mauscript: Evolutionary-scale prediction of atomic level protein structure with a language model
• GitHub: facebookresearch/esm
• Model Card: facebook/esm2_t36_3B_UR50D
• Weights: facebook/esm2_t36_3B_UR50D

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	None
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	facebook/esm2_t36_3B_UR50D

modelgenerator.backbones.esm2_650m

Bases: ESM

ESM2 650M model

Note

• Mauscript: Evolutionary-scale prediction of atomic level protein structure with a language model
• GitHub: facebookresearch/esm
• Model Card: facebook/esm2_t33_650M_UR50D
• Weights: facebook/esm2_t33_650M_UR50D

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	None
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	facebook/esm2_t33_650M_UR50D

modelgenerator.backbones.esm2_150m

Bases: ESM

ESM2 150M model

Note

• Mauscript: Evolutionary-scale prediction of atomic level protein structure with a language model
• GitHub: facebookresearch/esm
• Model Card: facebook/esm2_t30_150M_UR50D
• Weights: facebook/esm2_t30_150M_UR50D

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	None
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	facebook/esm2_t30_150M_UR50D

modelgenerator.backbones.esm2_35m

Bases: ESM

ESM2 35M model

Note

• Mauscript: Evolutionary-scale prediction of atomic level protein structure with a language model
• GitHub: facebookresearch/esm
• Model Card: facebook/esm2_t12_35M_UR50D
• Weights: facebook/esm2_t12_35M_UR50D

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	None
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	facebook/esm2_t12_35M_UR50D

modelgenerator.backbones.esm2_8m

Bases: ESM

ESM2 8M model

Note

• Mauscript: Evolutionary-scale prediction of atomic level protein structure with a language model
• GitHub: facebookresearch/esm
• Model Card: facebook/esm2_t6_8M_UR50D
• Weights: facebook/esm2_t6_8M_UR50D

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	None
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	facebook/esm2_t6_8M_UR50D

Structure

modelgenerator.backbones.aido_protein2structoken_16b

Bases: GenBioFM

AIDO.Protein2StructureToken model with 16B parameters adapted from aido_protein_16b and for structure prediction with AIDO.StructureTokenizer. The model is trained on 170M sequences and structures from AlphaFold Database and 0.4M sequences and structures from PDB.

Note

• Mauscripts:
  • Mixture of Experts Enable Efficient and Effective Protein Understanding and Design
  • Balancing Locality and Reconstruction in Protein Structure Tokenizer
• Model Card: AIDO.Protein2StructureToken-16B
• Weights: genbio-ai/AIDO.Protein2StructureToken-16B

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.Protein2StructureToken-16B

modelgenerator.backbones.aido_protein_rag_16b

Bases: GenBioFM

AIDO.Protein-RAG model with 16B parameters adapted from aido_protein_16b with 180B tokens of MSA and structural context from UniRef50/UniClust30 and AlphaFold Database.

Note

• Mauscripts:
  • Mixture of Experts Enable Efficient and Effective Protein Understanding and Design
  • Retrieval Augmented Protein Language Models for Protein Structure Prediction
• Model Card: AIDO.Protein-RAG-16B
• Weights: genbio-ai/AIDO.Protein-RAG-16B

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.Protein-RAG-16B

modelgenerator.backbones.aido_protein_rag_3b

Bases: GenBioFM

AIDO.Protein-RAG model with 3B parameters adapted from a 3B version of AIDO.Protein 16B with 180B tokens of MSA and structural context from UniRef50/UniClust30 and AlphaFold Database.

Note

• Mauscripts:
  • Mixture of Experts Enable Efficient and Effective Protein Understanding and Design
  • Retrieval Augmented Protein Language Models for Protein Structure Prediction
• Model Card: AIDO.Protein-RAG-3B
• Weights: genbio-ai/AIDO.Protein-RAG-3B

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.Protein-RAG-3B

Cell

modelgenerator.backbones.aido_cell_100m

Bases: GenBioCellFoundation

AIDO.Cell model with 100M parameters pretrained on 50M single-cell expression profiles from diverse set of human tissues and organs.

Note

• Mauscript: Scaling Dense Representations for Single Cell with Transcriptome-Scale Context
• Model Card: AIDO.Cell-100M
• Weights: genbio-ai/AIDO.Cell-100M
• Integrations:
  • CZI Virtual Cell Models

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.Cell-100M

modelgenerator.backbones.aido_cell_10m

Bases: GenBioCellFoundation

AIDO.Cell model with 10M parameters pretrained on 50M single-cell expression profiles from diverse set of human tissues and organs.

Note

• Mauscript: Scaling Dense Representations for Single Cell with Transcriptome-Scale Context
• Model Card: AIDO.Cell-10M
• Weights: genbio-ai/AIDO.Cell-10M
• Integrations:
  • CZI Virtual Cell Models

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.Cell-10M

modelgenerator.backbones.aido_cell_3m

Bases: GenBioCellFoundation

AIDO.Cell model with 3M parameters pretrained on 50M single-cell expression profiles from diverse set of human tissues and organs.

Note

• Mauscript: Scaling Dense Representations for Single Cell with Transcriptome-Scale Context
• Model Card: AIDO.Cell-3M
• Weights: genbio-ai/AIDO.Cell-3M
• Integrations:
  • CZI Virtual Cell Models

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.Cell-3M

modelgenerator.backbones.scfoundation

Bases: SCFoundation

scFoundation model

Note

• Mauscript: Large-scale foundation model on single-cell transcriptomics
• GitHub: genbio-ai/scFoundation
• Model Card: genbio-ai/scFoundation
• Weights: genbio-ai/scFoundation

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
num_genes	Optional[int]	Number of genes in the model context.	19264
frozen	bool	Whether to freeze model.	False
output_type	str	Type of output embedding ('cell', 'gene', 'gene_batch', 'gene_expression').	'cell'
pool_type	str	Pooling type for cell embedding ('all', 'max').	'all'
input_type	str	Input data type ('singlecell', 'bulk').	'singlecell'
pre_normalized	str	Whether input is pre-normalized ('T', 'F', 'A').	'F'
train_last_n_layers	int	Number of layers to train in the encoder.	0
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/scFoundation

modelgenerator.backbones.geneformer

Bases: Geneformer

Geneformer model

Note

• Mauscript: Transfer learning enables predictions in network biology
• Model Card: ctheodoris/Geneformer
• Weights: ctheodoris/Geneformer

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to initialize from random weights.	False
max_length	int	Maximum input sequence length.	4096
emb_layer	int	Layer to extract embeddings from.	-2
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	ctheodoris/Geneformer

Tissue

modelgenerator.backbones.aido_tissue_3m

Bases: GenBioCellSpatialFoundation

AIDO.Tissue model with 3M parameters adapted from aido_cell_3m to incorporate tissue context.

Note

• Mauscript: Scaling Dense Representations for Single Cell with Transcriptome-Scale Context
• Model Card: AIDO.Tissue-3M
• Weights: genbio-ai/AIDO.Tissue-3M

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
rope2d_use_xy	bool	Whether to use 2D rope encoding.	False
sep_value	int	Separator value for the model.	-10000
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path		genbio-ai/AIDO.Tissue-3M

modelgenerator.backbones.aido_tissue_60m

Bases: GenBioCellSpatialFoundation

AIDO.Tissue model with 60M parameters adapted from AIDO.Cell to incorporate tissue context.

Note

• Mauscript: Scaling Dense Representations for Single Cell with Transcriptome-Scale Context
• Model Card: AIDO.Tissue-60M
• Weights: genbio-ai/AIDO.Tissue-60M

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
rope2d_use_xy	bool	Whether to use 2D rope encoding.	False
sep_value	int	Separator value for the model.	-10000
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	genbio-ai/AIDO.Tissue-60M

Integrations

modelgenerator.backbones.Huggingface

Bases: HFSequenceBackbone

A generic huggingface wrapper allows for using any huggingface model as backbone.

Note

Warning: This is an experimental feature, don't expect it to work with all models. Downstream task support is also extremely limited to the standard huggingface heads. Its usage often involves manual configuration of the model's head through config_overwrites.

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
model_path	str | PathLike	Path to the huggingface model.	required
modules_for_model_registration	Optional[List[str]]	List of python modules to register the model.	None
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	None
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.

Debug

modelgenerator.backbones.Onehot

Bases: HFSequenceBackbone

Tokenizer-only model for one-hot encoding. Useful for baseline model testing (CNNs, linear, etc.)

Note

Models using this interface include dna_onehot and protein_onehot.

Does not contain any parameters, and cannot be used without an adapter.

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
vocab_file	str	Path to the vocabulary file. Defaults to "modelgenerator/huggingface_models/rnabert/vocab.txt".	None
max_length	Optional[int]	Maximum sequence length.	512

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.
vocab_file	str	Path to the vocabulary file.

modelgenerator.backbones.dna_onehot

Bases: Onehot

One-hot encoding for DNA sequences. Used for benchmarking finetuning tasks without pretrained embeddings.

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
vocab_file	str	Path to the vocabulary file. Defaults to "modelgenerator/huggingface_models/rnabert/vocab.txt".	None
max_length	Optional[int]	Maximum sequence length.	512

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.
vocab_file	str	Path to the vocabulary file modelgenerator/huggingface_models/dnabert/vocab.txt

modelgenerator.backbones.protein_onehot

Bases: Onehot

One-hot encoding for protein sequences. Used for benchmarking finetuning tasks without pretrained embeddings.

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
vocab_file	str	Path to the vocabulary file. Defaults to "modelgenerator/huggingface_models/rnabert/vocab.txt".	None
max_length	Optional[int]	Maximum sequence length.	512

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.
vocab_file	str	Path to the vocabulary file modelgenerator/huggingface_models/fm4bio/vocab_protein.txt

modelgenerator.backbones.aido_dna_debug

Bases: GenBioBERT

A small dna/rna dense transformer model created from scratch for debugging purposes only.

Note

• This model is not intended for any real-world applications and is only for testing purposes.
• It is created from scratch with a very small number of parameters and is not trained on any data.

Parameters:

Name	Type	Description	Default
*args		Positional arguments passed to the parent class.	()
**kwargs		Keyword arguments passed to the parent class. from_scratch=True and config_overwrites={'hidden_size': 64, 'num_hidden_layers': 2, 'num_attention_heads': 4, 'intermediate_size': 128} are always overridden.	{}

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.

modelgenerator.backbones.aido_protein_debug

Bases: GenBioFM

A small protein dense transformer model created from scratch for debugging purposes only.

Note

• This model is not intended for any real-world applications and is only for testing purposes.
• It is created from scratch with a very small number of parameters and is not trained on any data.

Parameters:

Name	Type	Description	Default
*args		Positional arguments passed to the parent class.	()
**kwargs		Keyword arguments passed to the parent class. from_scratch=True and config_overwrites={'hidden_size': 64, 'num_hidden_layers': 2, 'num_attention_heads': 4, 'intermediate_size': 128} are always overridden.	{}

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.

modelgenerator.backbones.aido_dna_dummy

Bases: GenBioBERT

A small dummy AIDO.DNA model created from scratch for debugging purposes only

Note

• This model is not intended for any real-world applications and is only for testing purposes.
• It has a very small number of parameters and is not trained on any data.

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path		genbio-ai/AIDO.DNA-dummy

Base Classes

modelgenerator.backbones.SequenceBackboneInterface

Bases: Module

Interface class to ensure consistent implementation of essential methods for all backbones.

Parameters:

Name	Type	Description	Default
*args		The description is missing.	required
**kwargs		The description is missing.	required

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.

modelgenerator.backbones.HFSequenceBackbone

Bases: SequenceBackboneInterface

Base class for all backbone models

Note

The required possitional arguments are reserved by downstream tasks for dependency injection and cannot be changed by the user.

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[dict, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules	List[str]	List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.

modelgenerator.backbones.GenBioBERT

Bases: HFSequenceBackbone

GenBioBERT model

Note

Models using this interface include aido_dna_7b, aido_dna_300m, dna_dummy, aido_dna_debug, aido_rna_1b600m, aido_rna_1b600m_cds, aido_rna_1m_mars, aido_rna_25m_mars, aido_rna_300m_mars, aido_rna_650m, aido_rna_650m_cds.

FSDP auto_wrap_policy is modelgenerator.distributed.fsdp.wrap.AutoWrapPolicy

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	32
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[list]	LoRA target modules.	['query', 'value']
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.

modelgenerator.backbones.GenBioFM

Bases: HFSequenceBackbone

GenBioFM model

Note

Models using this interface include aido_protein_16b, aido_protein_16b_v1, aido_protein2structoken_16b, aido_protein_debug.

FSDP auto_wrap_policy is modelgenerator.distributed.fsdp.wrap.AutoWrapPolicy

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.

modelgenerator.backbones.GenBioCellFoundation

Bases: HFSequenceBackbone

GenBioCellFoundation model

Note

Models using this interface include aido_cell_100m, aido_cell_10m, and aido_cell_3m.

FSDP auto_wrap_policy is modelgenerator.distributed.fsdp.wrap.AutoWrapPolicy

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.

modelgenerator.backbones.GenBioCellSpatialFoundation

Bases: HFSequenceBackbone

GenBioCellSpatialFoundation model

Note

Models using this interface include aido_tissue_60m and aido_tissue_3m.

FSDP auto_wrap_policy is modelgenerator.distributed.fsdp.wrap.AutoWrapPolicy

Parameters:

Name	Type	Description	Default
legacy_adapter_type	Union[LegacyAdapterType, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
default_config	Union[DefaultConfig, None]	Ignore. Reserved for use by use_legacy_adapter in Tasks.	required
from_scratch	bool	Whether to create the model from scratch.	False
max_length	Optional[int]	Maximum sequence length.	None
use_peft	bool	Whether to use LoRA PEFT.	False
frozen	bool	Whether to freeze encoder.	False
save_peft_only	bool	Whether to save only the PEFT weights.	True
lora_r	int	LoRA r parameter.	16
lora_alpha	int	LoRA alpha parameter.	16
lora_dropout	float	LoRA dropout.	0.1
lora_target_modules	Optional[List[str]]	LoRA target modules.	['query', 'value', 'key', 'dense', 'router']
lora_modules_to_save	Optional[List[str]]	LoRA modules to save.	None
lora_use_rslora	bool	Whether to use RSLora.	False
rope2d_use_xy	bool	Whether to use 2D rope encoding.	False
sep_value	int	Separator value for the model.	-10000
config_overwrites	Optional[dict]	Optional model arguments for PretrainedConfig.	None
model_init_args	Optional[dict]	Optional model arguments passed to its init method.	None

Attributes:

Name	Type	Description
fsdp_wrap_modules		List of module paths to wrap when using distributed training with FSDP.
model_path	str	Path to the model weights. May be HF.