Haha well.... I am huge TGA fan. Sorry but I have to say this much
The first song I heard from them was American Slang. Thought it was pretty good, and my brother bought the album (it is their 3rd). So obviously I listened to it, I really liked it. Theeeen, I listened to a few songs from Sink or Swim their first and didn't like it that much at all. At this point I wasn't really listening to them at all that much. Then my brother decided to buy both Sink or Swim and The '59 Sound... Out of curiosity I listened to them both and found that they made American Slang seem pretty average.
I am glad I listened and liked American Slang first because it mean't I wasn't disappointed.
Each album is fairly different to the other which is nice.
Sink or Swim is a lot more rough and gritty, more folk-punk influence. A lot of influence from the old american dream, expensive cars and his hometown.
The '59 Sound is a much more bluesy album, with heavy influence on old books and Film Noir, with songs like Film Noir, Great Expectations, Even Cowgirls Get the Blues, Here's Looking at you Kid.
American Slang is closer to The '59 Sound with its production I find. Its a solid album, but the first two albums overshadow this album.
Handwritten, I recommend you leave this to last. Its a good album, but I think this is an album that sounds best to fans of TGA that have heard their other stuff first. It is a straight up TGA, no more or less than you'd expect and it works.
So round it up... I listened to them like this:
American Slang
Sink or Swim
The '59 Sound
Handwritten
No regrets at all in that order for me.
But I would probably just listen to them in order of release since you get to notice the small developments they've had over time.