In large bowl, combine sweet potatoes with eggs, brown sugar, butter, nutmeg, lime juice until thoroughly mixed. Fold in diced bananas. Add olive oil to a baking dish and add sweet potato mixture. Bake at 375 degrees for 40 minutes.