The serotonin hypothesis of depression has dominated the field of depression for over four decades ¹ . This theory is centered on the idea that reduced serotonin signaling is a risk factor in the etiology and/or pathophysiology of major depressive disorder (MDD) ² . However, the most robust body of evidence for the role of serotonin in depression is the efficacy of increasing extracellular serotonin for the treatment of depression. The discovery that the efficacy of tricyclic antidepressants (TCAs) and monoamine oxidase inhibitor (MAOI) antidepressants was largely due to their serotonergic actions, which prompted the use of serotonin selective reuptake inhibitors (SSRIs), the first among them fluoxetine, to treat depression ^{3– 6} . These drugs act at the serotonin transporter (5-HTT, also known as SERT) and cause increases in extracellular serotonin, which is the purported mechanism of action ^{6– 8} . Many subsequent drugs inhibiting serotonin reuptake have shown behavioral efficacy as antidepressant drugs, suggesting that increasing synaptic serotonin levels may lead to the treatment of depression ^{6, 9} .

Despite the relative success in treating depression by increasing extracellular serotonin, there is a lack of strong evidence supporting a direct correlation between low serotonin signaling and depression. While some studies report an association between levels of platelet serotonin and depression, this has not been a consistent finding in large sample sets, and it is also unclear how platelet levels are related to brain levels of serotonin ^{10, 11} . Additionally, few studies report direct correlations between cerebrospinal fluid 5-hydroxyindoleacetic acid (5-HIAA), a serotonin metabolite, and depression ^{12, 13} . Low levels of tryptophan have been consistently linked to depression; however, these effects could be independent of serotonin ^{14, 15} . The lack of consistent clear-cut abnormalities in global measures of serotonin signaling isn’t surprising if one considers the complexity of the receptors at which serotonin binds, the intricate neuroanatomical circuitry of the serotonin system, and the developmental role serotonin plays as a neurotrophic factor ^{16– 18} . Many recent studies have focused on understanding the mechanisms through which serotonin affects depression by studying the impact of 5-HTT and the 15 known receptors through gene-association studies, human brain imaging, and pharmacological and genetic mouse models ¹⁹ .

The success in treating depression by targeting the transporter with SSRIs prompted investigations into whether variability in 5-HTT expression levels could be involved in the etiology of depression. A highly cited study showed that there is an association between a polymorphism in the serotonin transporter (5-HTTLPR) and susceptibility to developing depression ²⁰ . This and other studies have shown that the short “s” allele, which results in lower levels of 5-HTT expression (at least in vitro) and therefore increased extracellular 5-HT, is associated with a higher risk of depression when combined with stressful life events ^{21, 22} . This discovery would be unexpected if developmental considerations were not considered. Although inhibiting the function of the transporter during adulthood decreases depressive symptoms as in the case of SSRIs, reduced expression of 5-HTT during development may increase depressive behavior in adulthood. A human functional magnetic resonance imaging (fMRI) study supports this, showing that short allele carriers show morphological and functional alterations in limbic circuits ²³ . Additionally, mice lacking 5-HTT throughout life display increased depressive-like behaviors, and pharmacological blockade of 5-HTT in mice exclusively during early postnatal development resulted in increased adult depressive behavior ²⁴ . These results highlight the differences in developmental versus adult effects of altered serotonin neurotransmission on depression.

In addition to the serotonin transporter, the majority of the 15 serotonin receptors have been implicated in the modulation of depression, depressive-like behaviors, or the response to anti-depressant treatment ¹⁹ . There are numerous pre-clinical studies which have investigated the role of serotonin receptors using pharmacological manipulations and genetic knockout (KO) models in rodents ( Table 1). Given the breadth of this literature, this review will focus on two receptors that are among the most extensively studied for their role in modulating depression, the 5-HT _1A and 5-HT _1B receptor subtypes. In addition, attention will be paid to population-dependent and development-dependent effects of serotonin signaling at these receptors and will draw from both rodent and human studies.

The 5-HT _1A and 5-HT _1B receptors are both inhibitory Gi/o-coupled seven transmembrane receptors that are located throughout the brain ^{25– 27} . A major difference between these two receptors is their subcellular distribution ²⁸ . 5-HT _1A receptors are somatodendritic, while 5-HT _1B receptors are located on axon terminals ^{27, 29} . This difference is also reflected in their mechanisms of inhibitory action ( Figure 1). Activation of either receptor causes decreased neurotransmitter release; however, 5-HT _1A receptor activation causes hyperpolarization, leading to decreased firing, while 5-HT _1B receptors inhibit voltage-gated calcium channels in the presynaptic terminal ^{30– 32} . Another mechanism for 5-HT _1B receptor-mediated inhibition is via effects on 5-HTT, and activation of the 5-HT _1B receptor increases serotonin reuptake ^{33, 34} .

Both 5-HT _1A and 5-HT _1B receptors act as autoreceptors located on serotonin neurons and also have heteroreceptor populations located on non-serotonin receptors ( Figure 2). Although the mRNA in the raphe (corresponding to autoreceptors) is comparable between the two receptors, their heteroreceptors have distinct patterns of expression ³⁵ . 5-HT _1A receptors are enriched in the hippocampus and cortex, while 5-HT _1B receptors are highly expressed in the basal ganglia ^{36, 37} . These differences in mechanism of action and localization may play a role in the different functional effects of these receptors.

While this review focuses on the contribution of 5-HT _1A and 5-HT _1B receptors in depression and depressive-like behaviors, these receptors also modulate other psychiatric-relevant phenotypes. For example, alterations in 5-HT _1A receptor expression influence anxiety behavior, and 5-HT _1B receptor signaling affects reward- and impulsivity-related phenotypes. These receptor-based differences in serotonergic regulation of emotional behavior, which segment into endophenotypes, could contribute to the heterogeneity of symptoms found in MDD ³⁸ . Understanding the neural circuits that subserve these receptor-based and endophenotype-based differences can help clarify the often confusing and sometimes contradictory findings from various preclinical approaches. From a behavioral perspective, these phenotypes can be segmented through formal unsupervised factor analyses to better divide depressive behaviors into meaningful endophenotypes. Then predictors of the different endophenotypes could be tested by including genetic or pharmacological manipulations.

Of the 15 known serotonin receptors, the 5-HT _1A receptor is the most studied for its role in depression ³⁹ . Quantification of 5-HT _1A receptor levels in humans from post mortem and positron emission tomography (PET) imaging studies reveals an increased level of 5-HT _1A receptors in patients diagnosed with MDD ^{40– 42} . Gene association studies have linked a polymorphism in the 5-HT _1A regulatory region (rs6295; G-1019C) with receptor levels in the brain and also to increased risk for depression ^{43– 47} . The GG genotype at this single nucleotide polymorphism (SNP) is associated with altered levels of 5-HT _1A receptor expression and reduced responsiveness to antidepressant treatment ^{43, 48} . Additionally, clinical studies have revealed antidepressant effects of buspirone and other 5-HT _1A receptor agonists ^{49, 50} .

Rodent models have also shown that 5-HT _1A receptor agonists, such as 8-hydroxy-2-(di-n-propylamino) tetralin (8-OH-DPAT), can have acute antidepressant-like effects ^{51– 53} . These effects are blocked by 5-HT _1A receptor antagonists, suggesting that the antidepressant-like response is specific to 5-HT _1A receptor signaling ⁵⁴ . 5-HT _1A heteroreceptors, expressed throughout the limbic system, are the likely site of action for these acute 5-HT _1A receptor-mediated effects ^{50, 55} . On the other hand, 5-HT _1A autoreceptors work in opposition to the heteroreceptors, leading to pro-depressive effects. Specifically, activation results in hyperpolarization and reduced firing of raphe neurons, leading to diminished serotonin release in projection regions ⁵⁶ . Therefore, stimulation of 5-HT _1A autoreceptors from increased extracellular serotonin following SSRI treatment is thought to oppose SSRI actions by downregulating serotonin neuron activity ⁵⁷ . Over the first few weeks of treatment, these receptors desensitize, which may underlie the delayed behavioral efficacy of SSRIs ⁵⁸ . Therefore, blocking 5-HT _1A autoreceptor activation has been introduced as an adjunctive therapy to SSRIs. 5-HT _1A receptor partial agonists such as pindolol, and more recently vilazodone, have been shown to be an effective adjunctive therapy to SSRIs in clinical studies ^{59– 62} . The development of new agonists that preferentially target subpopulations of 5-HT _1A receptors, for example autoreceptors versus heteroreceptors, potentially through biased agonism, may be useful tools for the treatment of MDD ⁶³ .

Differences in receptor levels have also been modeled in mice by using genetic loss-of-function models and have allowed causal links between receptor expression levels and depressive-like behavior. 5-HT _1A receptor KO mice have an anti-depressive phenotype ^{64, 65} . Tissue-specific KOs have been especially valuable for the dissection of this phenotype and have allowed investigations into the distinct roles of different populations of receptors ⁶⁶ . The absence of heteroreceptors results in increased depressive-like behavior- as measured in the forced swim test. This mouse model also allowed for temporal control of receptor expression, which revealed a developmental sensitive period for the effect of heteroreceptors on depressive-like behavior. Specifically, knockdown of 5-HT _1A heteroreceptors in adulthood was not sufficient to produce the depressive-like behavior. On the other hand, reduction of autoreceptors in adulthood increased mobility in the forced swim test, suggesting an “anti-depressed” phenotype.

Preclinical studies have also confirmed a causal role for alterations in 5-HT _1A receptor expression in antidepressant efficacy. 5-HT _1A receptor KO mice do not show a behavioral response to fluoxetine ⁶⁷ . As expected from the pharmacology work, this effect is not mediated by autoreceptors, since reduced expression of 5-HT _1A autoreceptors actually increases the speed and efficacy of SSRI response, requiring only 8 days of fluoxetine treatment to show a behavioral antidepressant-like response ⁶⁸ . Recent data show that 5-HT _1A heteroreceptors are critical for an effective behavioral response to an SSRI in mice ⁶⁹ . Genetic or viral deletion of 5-HT _1A receptors specifically in the dentate gyrus of the hippocampus reduced the behavioral response to fluoxetine. Furthermore, expression of 5-HT _1A receptors only in the dentate gyrus was sufficient for normal antidepressant-like responses. These results importantly demonstrate a mechanism for 5-HT _1A-mediated antidepressant effects localized in the mature granule cells of the dentate gyrus.

Anxiety behavior is also modulated strongly by the 5-HT _1A receptor, and, among depressed patients, almost half have a comorbid anxiety disorder ⁷⁰ . In preclinical studies, 5-HT _1A receptor agonists have anxiolytic effects, and 5-HT _1A receptor KO mice display increased anxiety-like behavior ^{64, 65, 71, 72} . The effect has a developmental sensitive period, since early developmental but not adult rescue of the receptor was sufficient to restore the normal phenotype in the KO ⁷³ . Consistent with this, early postnatal blockade of 5-HT _1A receptors, through genetic or pharmacological methods, also leads to increased anxiety ^{74, 75} . Recent work has shown that the sensitive period is peri-pubertal, and tissue-specific KO mice point to a role for autoreceptors during this period of development ^{66, 76, 77} .

Other psychiatric disorders have also been linked to the 5-HT _1A receptor, including bipolar disorder and post-traumatic stress disorder ^{78, 79} . Additionally, the SNP rs6295 found in the premotor region that is associated with risk for depression is also linked with psychiatric hospitalization, a history of substance abuse, and prior suicide attempts ⁴³ . Consistent with the studies in depression, the G allele is associated with reduced expression of the 5-HT _1A receptor in the prefrontal cortex and an increased risk for psychiatric outcomes. Interestingly, the effects on receptor expression were also seen in the brain during early human embryonic development, suggesting its potential importance in mediating developmental contributions to adult depression. Finally, there were associations with childhood maltreatment with trends towards significant genotype by environment interactions ⁴⁰ .

While the 5-HT _1B receptor is best known for its role in regulating aggressive and impulsive behavior, it also plays an important role in modulating depression. Activation of the 5-HT _1B receptor decreases serotonin levels in the brain through effects on release, synthesis, and reuptake ^{33, 80, 81} . In humans, reduced 5-HT _1B receptor function is associated with MDD ⁸² . Additionally, patients with MDD are less responsive to 5-HT _1B receptor agonists, suggesting reduced expression or desensitization ^{83, 84} . This is consistent with clinical studies showing that 5-HT _1B receptor agonists produce antidepressant effects in humans ^{85– 87} . This has also been shown in mice, with specific agonists resulting in antidepressant-like behavior ^{88, 89} . However, genetic KO of the receptor also results in antidepressant-like behavior, suggesting that this is possibly caused by compensatory effects ^{90– 94} .

Both autoreceptor and heteroreceptor populations of 5-HT _1B receptors have been implicated in depressive-like behaviors using rodent models. However, since 5-HT _1B receptors are located on presynaptic terminals, heteroreceptors and autoreceptors have overlapping localization ⁹⁵ . This rules out brain imaging and pharmacological manipulations in preclinical models as tools to differentiate the role of the two populations of receptors. Therefore, it has been only the recent availability of a tissue-specific genetic mouse model that has allowed the dissection of the role of 5-HT _1B receptors in the regulation of behavior ⁹⁶ .

Our recent studies show that selective ablation of 5-HT _1B autoreceptors results in decreased depressive-like behaviors in mice ⁹⁷ . These mice also show increased elevations in serotonin levels compared to controls following SSRI administration, suggesting a potential mechanism of action for the behavioral effects. Specifically, removing the terminal auto-inhibition may result in increased serotonin in projection regions that are relevant to depressive behavior. Furthermore, we also showed that the impact of 5-HT _1B autoreceptors on behavior was not due to developmental expression, since the phenotype was not recapitulated in a mouse with developmental knockdown. These data are consistent with other evidence suggesting a pro-depressive role for the activation of 5-HT _1B autoreceptors ^{98, 99} . For example, 5-HT _1B mRNA is elevated in the raphe of rats following stress and in models of depression such as learned helplessness, and viral overexpression of 5-HT _1B receptors in the raphe results in depressive-like behavior following stress ¹⁰⁰ . In rats, reductions in 5-HT _1B receptor mRNA in the raphe are seen following SSRI treatment in post mortem brains ^{101, 102} . This effect isn’t seen in other brain regions such as the cortex, hippocampus, or striatum, suggesting that this effect is specific to autoreceptors. Additionally, another study showed that 5-HT _1B autoreceptors may desensitize following SSRI treatment, similar to 5-HT _1A autoreceptors ¹⁰³ . Finally, a recent PET study in humans reported that following effective cognitive behavioral therapy for depression, 5-HT _1B receptor binding was reduced in the brainstem ¹⁰⁴ .

There is evidence which suggests an opposing role for 5-HT _1B heteroreceptors in depressive behaviors. Activation of 5-HT _1B heteroreceptors in a rodent serotonin depletion model (to remove the contribution of autoreceptors) results in an antidepressant-like effect ¹⁰⁵ . Additionally, reduced expression of 5-HT _1B heteroreceptors in the ventral striatum is associated with depression in humans ⁸² . Finally, 5-HT _1B receptors located in the ventral striatum have been suggested to interact with p11 (a 5-HT _1B receptor-binding protein) to affect depression-related behaviors ^{106, 107} .

Reward dysfunction is a major symptom of MDD which is mediated, in part, by altered signaling in the mesolimbic reward system ^{108– 112} . 5-HT _1B receptors have been implicated in the neural basis of dysregulated reward sensitivity in a number of human studies and preclinical models ^{113– 116} , and both 5-HT _1B receptor protein and mRNA are located within the mesolimbic pathway in the nucleus accumbens (NAc) and ventral tegmental area (VTA) ⁹⁵ . Additionally, activation of 5-HT _1B receptors in the VTA increases dopamine levels in the NAc, potentially via effects on GABAergic signaling in the VTA ¹¹⁷ .

Many studies linking the receptor to functional deficits in reward processing have focused on addiction. Polymorphisms in the 5-HT _1B receptor gene have also been associated with drug and alcohol abuse ^{118– 120} . Additionally, a PET imaging study revealed increased 5-HT _1B receptor binding in pathological gamblers, who have known deficits in reward sensitivity, and gambling disorder is highly comorbid with depression and alcohol and substance use disorders ^{116, 121} . Another PET imaging study shows that there is reduced 5-HT _1B receptor binding in cocaine-dependent participants compared to in healthy controls ¹²² . In preclinical models, 5-HT _1B receptor KO mice are more motivated to self-administer cocaine ¹²³ . Consistent with this, 5-HT _1B receptor agonists attenuate the motivation for cocaine but paradoxically increase the rewarding effects of cocaine ¹²⁴ . These effects are mediated by 5-HT _1B receptor expression on medium spiny neurons in the NAc, likely through their projections to the VTA ^{125, 126} . Additionally, 5-HT _1B receptors are required for the rewarding properties of social interaction, supporting an impact on general reward systems ¹¹⁴ .

5-HT _1B receptors are also implicated in impulsive aggression. In humans, polymorphisms in the gene encoding 5-HT _1B receptors have been associated with aggression, suicide, and disorders that include impulsivity as a core phenotype, including attention deficit hyperactivity disorder and substance use disorder ^{115, 120, 127} . In mice, 5-HT _1B receptor KOs are highly aggressive in tests of male and female aggression and also display increased impulsivity ^{128– 130} . Additionally, 5-HT _1B receptor agonists are known as “serenics” because they decrease aggression ¹³¹ . While the aggressive and impulsive phenotype was originally thought to be modulated by the same underlying circuits, our recent work shows that distinct populations of 5-HT _1B receptors modulate aggression and impulsivity ⁹⁶ . Furthermore, developmental expression of the 5-HT _1B receptor influences aggression, while adult expression modulates impulsive behavior.

There is a considerable body of research that implicates serotonin in the modulation of depression and depression-related behaviors. The preclinical work delineating the effects of signaling through the 5-HT _1A and 5-HT _1B receptors has been made possible because of careful pharmacological studies as well as the development of transgenic mouse models that have allowed for tissue-specific and inducible knockdown. These studies have highlighted the complexity of serotonin receptors, showing that their role varies through the lifespan and by cell-type population. Additionally, the availability of specific radioligands for PET imaging of these receptors has allowed for the translation of findings from preclinical work to humans. The large number of studies concerning the role of these receptors is partially due to the fact that the 5-HT ₁ receptor subtypes were some of the first discovered, and it may be only a matter of time before the roles of more newly discovered receptors are clarified ¹⁷ .

Despite the amassing of evidence of serotonin receptor-specific involvement in depression, the primary pharmaceutical treatment strategy for depression remains the inhibition of serotonin reuptake. The lack of new treatment options is surprising given the need for them, since current SSRI treatments are ineffective in one-third of patients ¹³² . Additionally, the majority of patients, as seen in the STAR*D study, don’t respond to administration of the first SSRI treatment, requiring multi-step treatment plans that take months ¹³² . Furthermore, the considerable differences in treatment outcome also emphasize the heterogeneity of the depressed patient population. A better understanding of receptor signaling and neural circuit mechanisms by which serotonin affects depression may inform the development of novel, more targeted drugs that influence specific receptors, signaling cascades, or time periods. Also, personalized treatment plans could be developed based on symptoms, biomarkers, or pathophysiological presentation.