Place cream cheese in your mixer's bowl and beat on low speed until creamy and smooth.Slowly add the yogurt and continue to beat until well combined.Add brown sugar and vanilla; beat until combined.Using a spoon, stir in enough milk to reach desired dipping consistency.Cover the dip and place in the refrigerator for 1 to 2 hours.Remove from fridge and serve with fruit.