Not trying to make up your mind, just a few suggestions:
Of course, everything should revolve around how much rest and sleep you can get every week. And the amount of food you eat. You may be getting use to the on the job labor but working out places even more requirements on your recovery abilities. That is the core of making progressive gains and it's affect on the CNS. If you a easy and natural gainer, not to worry so much about that (Sergio worked a labor job, double shifts, in a packing house, workout at least 4 days a week, and still made gains). If your a average guy trying to make gains, than recovery should be the main concern.
I might suggest two workouts a week. One devoted to the upper body and the other to the lower. Something like benches, dips, chins, rows, arms one training day. And squats, BB Hacks, hams & calves the next workout day. Try to stay mostly with the Big Boy exercises; squats, benches, rows, etc. Or another option would be monday and fridays..lower body. Wednsday...upper body. The next monday would be reversed. Monday and friday...upper body and wednsday..low body. Change each monday. Good Luck.