The Scarlet Letter is not only a famous piece of literature, but it is quoted in mainstream media today as well. Someone could use the comparison of someone of something being forbidden, or sinful, just like Hester's actions in the book.
Taylor Swift's "Love Story" references two famous writings. One is obvious, Romeo and Juliet. There is a line in the song Love Story by Taylor Swift, where she says, "'Cause you were Romeo I was a scarlet letter and my daddy said stay away from Juliet, but you were everything of me I was begging you please don't go", and here, she is referencing the Scarlet Letter.
In my opinion, Taylor Swift is not technically telling us that she committed adultery, but she is saying that just like Hester, she is basically off limits to everyone, and almost everything. Swift is off limits in the song, because of her father, and just like the play, because of her social status.
Also, Hester is very ashamed, and constantly judged by her town, because of the "A" on her chest, just as the narrator of the song is being judged by "Romeo's" family.
When I first heard this song, I had not read The Scarlet Letter, and did not know what that line was a reference to. Now that I know what it means, it gives the song a much better and deeper meaning.