going to the gym about 4-5 times a week before work. Im getting ripped but not putting that much weight on.
Pretty much sums the problem up.
Either add a load of cals eeervy day
Drop 1 or 2 days in the gym and lift to failure
By failure I mean, you wake up the next day feeling like you've been a piñata at a shaolin monks birthday party.
If you can't say no to those gym endorphins, swap the lifting for cardio.