Grate the sweet potatoes in the food processor using the grating attachment, not the steel blade. Mix flour, sugar, brown sugar, baking powder, cayenne pepper, curry powder, cumin, salt and pepper in a separate bowl. Add eggs and just enough milk to the dry ingredients to make a stiff batter. Add potatoes and mix. Batter should be moist but not runny; add more milk if needed. Heat 1/4 inch of peanut or vegetable oil in a large frying pan until hot (put a shred of potato into the pan and when it's bubbling, the oil is ready). Drop batter by tablespoonfuls and flatten out. Cook on each side til golden. Drain on paper towels. If making ahead, put cooked and drained latkes on cookie sheets and place in the freezer for an hour. When frozen, put latkes into plastic bags for storage in freezer. To serve, preheat oven to 400. Place frozen latkes on cookie sheets and put into the oven for 10 minutes or so until they're bubbling and crispy.