While some golfers say the biggest stress is on their pride and their mental state, from a Structural standpoint golf has the biggest impact on the lumbar spine aka, the lower back.
If you're serious about golf, then you have to be serious about the mechanics of your swing because a bad swing will wear down the discs, ligaments, and joints of the spine. The biggest threat is the coupled movement of excess rotation and extension. One bad swing isn't going to hurt you, but thousands of swings over the course of years can wreck even the strongest spine.

The commonly missed factor in the health of a golfer is the structure of the spine. A spine with a Structural Shift in the hip and pelvis will not only make a golfer more susceptible to injury, but you're leaving a lot of force on the table.
To make a long story short, you will make your best swing less efficient, and your muscles will fatigue more easily because you are wasting so much kinetic energy.

When the spine is in the distorted position that you see on the left, then the force of a highly rotational swing gets sent into the discs and ligaments of the spine, instead of smooth transfer into the hip muscles.
The result is more chance for the discs to tear, and poor muscular output leading to poor performance.
If you are serious about playing golf throughout your life, and you want to play not just pain-free, but to your fullest potential, then you need to play without Structural Shifts.