robust spine enlargement (Fig. 1, J and K). Stimulation of dopaminergic fibers 1 s before (DAopto
delay = −1 s) or 5 s after (DAopto delay = 5 s) STDP
(Fig. 1, L and M) resulted in only a slight enhancement. The actions of DAopto were examined with various timings (Fig. 1N), revealing
that DAopto timing was critical to enhance plasticity, with maximal effects for a delay of 0.6 s
(Fig. 1O and fig. S2, A to C), and decayed in a few
seconds, which is consistent with behavioral
study results (5, 16). The spine enlargement was
accompanied by an increase in the 2pEPSC (fig.
S3). A similar DAopto timing was observed when
we induced STDP through electrical stimulation
of presynaptic fibers (fig. S4).
Pharmacological studies revealed that D1R-
MSN structural plasticity was dependent on
N-methyl-D-aspartate receptors (NMDARs), Ca2+/
calmodulin–dependent protein kinase II (CaMKII),
and protein synthesis (Fig. 2A and fig. S5A),
suggesting that the molecular mechanisms for
D1R-MSN plasticity are similar to those underlying structural plasticity in hippocampal pyramidal neurons (10–12). Plasticity also depended
on D1R and protein kinase A (PKA), but not on
D2R (Fig. 2B and fig. S5A). Spine enlargement
was prevented by an inhibitory peptide blocking
the interaction of dopamine- and adenosine
3′,5′-cyclic monophosphate (cAMP)–regulated
phosphoprotein 32 kD (DARPP-32) with protein
phosphatase 1 (PP-1) (17), but not its control peptide (Fig. 2, C and D). Moreover, spine enlargement was induced even in the absence of DAopto
when PP-1 was inhibited by calyculin A (fig. S5,
B and C). These results suggest that similar to
hippocampal preparations (18), the phosphorylation of DARPP-32 by PKA would inhibit PP-1
and disinhibit CaMKII.
To test whether changes in Ca2+ signaling account for DAopto timing (19), we imaged increases
in cytosolic Ca2+ concentrations ([Ca2+]i) in spines
using a low-affinity calcium indicator Fluo-4FF
(KCa = 10 mM) to avoid saturation. The [Ca2+]i
increases gradually built up and quickly waned
after cessation of the STDP protocol (Fig. 3A).
We found that DAopto did not affect Ca2+
transients (Fig. 3, B to D, and fig. S6, A to C), indicating that Ca2+ signaling modulation did not
play a major role.
We examined whether CaMKII activation was
related to DAopto timing by using the Förster
resonance energy transfer (FRET) indicator of
CaMKII, Camuia-CR (20–22), which was virally
transfected into D1R-MSNs. CaMKII activation
was weak in the absence of DAopto (Fig. 3E) but
was greatly potentiated by DAopto (Fig. 3F), with
timing similar to DAopto timing for spine enlargement (Fig. 3G and fig. S6, D to F). This enhancement was specific to the stimulated spine
(Fig. 3H), abolished by the inhibitory peptide for
DARPP-32 (Fig. 3H and fig. S6G), and mimicked
by PP-1 inhibitors, calyculin A and tautomycetin
(fig. S6, H and I). These results support the hypothesis that PKA/DARPP-32 disinhibits CaMKII
via PP-1 (fig. S11).
We addressed whether the DAopto timing
was formed at the level of PKA activation by
using a FRET probe of PKA, AKAR2-CR (22),
which was virally delivered to the D1R-MSNs.
Unlike structural plasticity or Camuia-CR ac-
tivation, PKA activation in response to stimu-
lation of a dendritic spine by STDP and DAopto
was not restricted to the stimulated spine; neigh-
boring spines also exhibited significant PKA acti-
vation (fig. S7, A to D). Even without glutamate
uncaging, PKA activation was observed in the
spine and dendritic shaft (Fig. 4A) in an AP-
dependent manner (Fig. 4B), suggesting that
PKA activation is a cell-wide phenomenon. When
DAopto was applied at various times relative to
APs, we obtained a timing (Fig. 4C and fig. S7, E
to I) similar to DAopto timing on spine enlarge-
ment and CaMKII activation (Fig. 1O). The ex-
tent of spine enlargement positively correlated
with that of PKA activation (Fig. 4D). APs them-
selves were not sufficient to activate PKA (fig. S8,
C to E). The contingency between APs and
DAopto might be detected by Ca2+/calmodulin–
dependent adenylyl cyclase 1 (AC1), which is
synergistically activated by Ca2+/calmodulin
and Gs (23). Consistent with this, AC1 blocker
(NB001) eliminated AKAR2 activation, as well
as structural plasticity (fig. S9, A to D).
In the soma and proximal (first branch) dendrites, however, DAopto alone was sufficient to
activate PKA (Fig. 4E and fig S8, A and B) (24),
and APs could only slightly enhance PKA. We
predict that APs might have modulated spine
enlargement to a small degree in these regions,
if there had been spines (Fig. 4D). Thus, DAopto-
induced PKA activation must be suppressed in
distal dendrites in order to attain the large dynamic range for the timing detection. In fact,
when phosphodiesterase 10A (PDE10A), the major phosphodiesterase in MSNs, was blocked by
its inhibitor papaverine (25), PKA activations
were similarly induced in distal dendrites as in
the soma (fig. S9, E to G). Papaverine also disrupted the time window for structural plasticity
(fig. S9, H and I). Why were PDE10A actions
particularly potent in the distal dendrites? Sub-cellular differences in PDE10A expression might
not account for this, considering that PDE10A is
expressed at the plasma membrane and is uniformly distributed along the dendrites (26). Instead, we found a negative correlation between
dendrite diameter and DAopto-induced PKA activity (fig. S10A, blue), which was lost when the
phosphodiesterase was inhibited (fig. S10A, orange). Therefore, PDE10A might counteract the
increases in cAMP more potently in the thin
distal dendrites because of its high surface-to-volume ratios (fig. S10, B and C). Spines are only
found in the distal thin dendrites of MSNs (fig.
S10, D to E) (27), suggesting that spines are
distributed to be efficiently modulated by dopamine timing in MSNs.
It has been enigmatic why dopamine reinfor-
ces preceding, but not subsequent, sensorimotor
events. If dopamine always activates PKA, its
effects should last long enough to reinforce the
subsequent events over tens of seconds (Fig. 4A).
However, dopamine did not activate PKA unless
[Ca2+]i primed AC1 to outcompete the high phos-
phodiesterase activity in thin dendrites (fig. S11).
Our data show that [Ca2+]i priming should occur
strictly before dopamine delivery (0 s) (Figs. 1O,
3G, and 4C), which is reminiscent of serotonin’s
action in the classical conditioning of the siphon
withdrawal reflex in Aplysia (28), in which sero-
tonin, carrying an aversive signal, was only effec-
tive when it was preceded by increases in [Ca2+]i
for activation of calcium-dependent adenylyl
cyclase (AC) and PKA in presynaptic terminals
(29). The delay in [Ca2+]i priming of AC may
compensate the time lag of monoaminergic
signals after reward or punishment. Our data
suggest that reinforcement plasticity occurs at
the single spine level, even though PKA activa-
tion is a cell-wide phenomenon, in such a way
that dopamine regulates the gain of NMDAR-
dependent Hebbian plasticity via CaMKII ac-
tivity (fig. S11). This interdependence between
Hebbian and reinforcement plasticity has been
implicitly assumed in the reinforcement learn-
ing theory, in which the Hebbian term is used
for the credit assignment (6, 30), and the dopa-
mine timing in our study corresponds to the
eligibility trace that determines the time window
for reward action (30). Thus, we have clarified a
molecular and cellular basis of reinforcement
plasticity at the level of single dendritic spines.
REFERENCES AND NOTES
1. E. L. Thorndike, Animal Intelligence (Macmillan, New York, 1911).
2. I. P. Pavlov, Conditioned Reflexes; An Investigation of the
Physiological Activity of the Cerebral Cortex (Oxford Univ.
Press, Humphrey Milford, London, 1927).
3. M. F. Roitman, R. A. Wheeler, R. M. Carelli, Neuron 45, 587–597
(2005).
4. Y. Dan, M. M. Poo, Physiol. Rev. 86, 1033–1048 (2006).
5. J. Black, J. D. Belluzzi, L. Stein, Brain Res. 359, 113–119
(1985).
6. W. Schultz, J. Neurophysiol. 80, 1–27 (1998).
7. S. L. Smith-Roe, A. E. Kelley, J. Neurosci. 20, 7737–7742
(2000).
8. T. Nakano, J. Yoshimoto, K. Doya, Front. Comput. Neurosci.
7, 119 (2013).
9. M. Matsuzaki et al., Nat. Neurosci. 4, 1086–1092 (2001).
10. M. Matsuzaki, N. Honkura, G. C. R. Ellis-Davies, H. Kasai,
Nature 429, 761–766 (2004).
11. J. Tanaka et al., Science 319, 1683–1687 (2008).
12. C. D. Harvey, R. Yasuda, H. Zhong, K. Svoboda, Science 321,
136–140 (2008).
13. C. R. Gerfen et al., Science 250, 1429–1432 (1990).
14. W. Shen, M. Flajolet, P. Greengard, D. J. Surmeier, Science 321,
848–851 (2008).
15. J. N. J. Reynolds, B. I. Hyland, J. R. Wickens, Nature 413,
67–70 (2001).
16. R. A. Rescorla, Annu. Rev. Neurosci. 11, 329–352 (1988).
17. Y. G. Kwon et al., Proc. Natl. Acad. Sci. U.S.A. 94, 3536–3541
(1997).
18. R. D. Blitzer et al., Science 280, 1940–1943 (1998).
19. C. Cepeda, M. S. Levine, Sci. STKE 2006, pe20 (2006).
20. K. Takao et al., J. Neurosci. 25, 3107–3112 (2005).
21. S. J. R. Lee, Y. Escobedo-Lozoya, E. M. Szatmari, R. Yasuda,
Nature 458, 299–304 (2009).
22. A. J. Lam et al., Nat. Methods 9, 1005–1012 (2012).
23. G. D. Ferguson, D. R. Storm, Physiology (Bethesda) 19,
271–276 (2004).
24. L. R. V. Castro et al., J. Physiol. 591, 3197–3214 (2013).
25. A. Nishi et al., J. Neurosci. 28, 10460–10471 (2008).
26. E. I. Charych, L. X. Jiang, F. Lo, K. Sullivan, N. J. Brandon,
J. Neurosci. 30, 9027–9037 (2010).
27. C. J. Wilson, P. M. Groves, J. Comp. Neurol. 194, 599–615 (1980).
28. R. D. Hawkins, T. W. Abrams, T. J. Carew, E. R. Kandel, Science
219, 400–405 (1983).
29. Y. Yovell, T. W. Abrams, Proc. Natl. Acad. Sci. U.S.A. 89,
6526–6530 (1992).
30. R. S. Sutton, A. G. Barto, Reinforcemnet Learning (MIT Press,
Cambridge, MA, 1998).
