Preheat oven to 350 degrees. Grease and flour a 9 x 5-inch loaf pan. Mix brown sugar and cinnamon together in a bowl and set aside. Beat white sugar and butter together in a bowl using an electric mixer until smooth and creamy. Beat in eggs, 1 at a time, until incorporated; add vanilla extract and stir until incorporated.Combine flour and baking powder together in another bowl; stir into creamed butter mixture. Mix milk into batter until smooth. Pour half the batter into the prepared loaf pan. Next add half the apples and half the brown sugar cinnamon mixture. Lightly pat/push apple mixture into batter.Pour the remaining batter over apple layer; top with remaining apples and add more brown sugar/cinnamon mixture. Lightly pat/push apples into batter; swirl brown sugar mixture through apples using a finger or spoon (I cleaned my hands and used my finger because it was easier).Bake in the preheated oven until a toothpick inserted in the center of the loaf comes out clean, 30 to 40 minutes. Mine took about 35. If you notice it getting too brown on the top, cover it loosely with tin foil and continue baking until the inside is done. Enjoy!