Swanson was born on February 11, 1994 in Kennesaw, Georgia, to parents Cooter and Nancy (née Dansby) Swanson. Swanson grew up in Marietta with two older siblings, sister Lindsey and brother Chase. Both his father and brother played college baseball, while his mother played collegiate basketball and tennis. Swanson's sister was a softball player. Dansby attended Marietta High School, where he played basketball and baseball. As a child, Swanson became a fan of professional sports teams based in Atlanta, among them Major League Baseball's Braves, the National Football League's Falcons, and the National Basketball Association's Hawks. He was drafted by the Colorado Rockies in the 38th round of the 2012 Major League Baseball Draft, but did not sign and attended Vanderbilt University. As a freshman, he played in only 11 games due to a broken bone in his foot and a torn labrum in his non throwing shoulder. As a sophomore in 2014, he was named All-Southeastern Conference. After the Commodores beat the Virginia Cavaliers in the 2014 College World Series, he was named the College World Series Most Outstanding Player. He was a finalist for the 2015 Golden Spikes Award, presented annually to the nation's top college player.