The band was formed in 2001 by Chris Hathwell, Blake Miller, and Johan Boegli, when Hathwell's previous band, Festival of Dead Deer, broke up. They released their debut EP, Moving Units, on Three One G in early 2002. Their touring experience includes opening slots for Hot Hot Heat, Pixies, Blur and Nine Inch Nails. Their debut full-length album,Dangerous Dreams, was released on October 12, 2004 by Palm Pictures. Moving Units released their second studio album, Hexes For Exes in 2007. The album was released onMetropolis Records (who they signed with in January 2007) and the first single, "Pink Thoughts", was released through the band's MySpace page. So Sweet released a one-off limited edition 7 inch vinyl single "Crash N Burn Victims" in the UK coupled with a remix of the same track by Felix Cartal in November 2007.