Place peaches in a bowl and add light brown sugar, granulated sugar, allspice, and salt. Gently mix together. Set aside.
In a separate bowl, whisk together flour, baking powder, baking soda, and salt. Sift to remove any lumps. Set aside.
In the bowl of a stand mixer fitted with the paddle attachment or in a large bowl using a hand mixer cream the butter on medium-high speed until creamy and smooth, about 1 minute.
Turn the mixer speed to low; add the sugar, vanilla extract to the butter and mix until incorporated. Turn the speed to medium-high and beat until light and fluffy, about 5 minutes. Scrape the bowl often for even incorporation.
Turn the speed to medium-low and add the eggs and egg yolks one at a time, beating just until combined. Scrape the bowl after each addition.
Add one-third of the flour mixture to the butter mixture and mix on low speed, just until combined. Add half of the sour cream, mixing until incorporated. Repeat with another one-third flour, scraping the bowl and paddle after each addition. Add rest of sour cream, mixing until incorporated, and finish with final third of flour.
Scrape the batter into the pan with the peaches, smooth the top with the spatula.
Bake for 55-70 minutes or until it is no longer jiggly and a tester inserted into the cake comes out clean with just a couple lightly moist crumbs.
Let the cake cool in the pan for 20 minutes, then flip it over onto a serving plate and allow it to cool.
While the cake is cooling make the brown butter icing.