Bro, I've been working in construction and training for years, framing houses, concrete work, and everything else that can take a toll on your body. I'd say that the combo of construction and serious training is one of the toughest, but it can be met with success. It all boils down to how much you want to reach your goals, how bad you want it. If you want it bad enough nothing on this earth will slow you down, including fatigue.
Force yourself to go to the gym, no matter how bad you want to skip don't, once you get there 9 xs out of 10 you'll be glad you went.
Time management is also very important. You have to set back time to sleep. If that means not enjoying other things then so be it. If your training is extremely important to you it's a sacrifice that is well worth it.
Good luck to you.