Add the sugar and milk to a medium saucepan, mix well and bring to a boil.Continue to boil for about 2½ minutes and stir constantly.Remove from the heat and stir in the vanilla extract and peanut butter.Pour fudge into a glass container and allow it to set and harden for at least one hour.