Although this symbol is in common use today, no one would consider it to be a letter. However, the "&" symbol was the 27th letter in the English alphabet in the early 1800s – x, y, z, &.
It would have sounded strange to say "x y z and" so instead, the alphabet was recited like this: "x y z and per se and." The phrase "per se" means "by itself." So students who recited the alphabet were basically saying, "x y z and by itself and."
Eventually, this odd little string of words became blurred and changed (as happens often in all languages) to the point where they went from pronouncing "and by itself and" to saying "ampersand."
The "&" symbol itself is much, much older than the word "ampersand." Roman scribes in the first century wrote in cursive. The Latin word for "and" is "et," and because it was written so often they blended the "e" and the "t" into the & symbol – exactly why we use the symbol ourselves today.