Beginning in the 1820s, there were several attempts to found an orchestra in the city, the more successful of which were the Philharmonic Symphony Society (established in 1842) and the New York Symphony (established in 1878). The Philharmonic had a reputation for conservatism and high standards, hiring primarily European conductors, such as Gustav Mahler. The Symphony seemed more ambitious and interested in new music. It received patronage from Andrew Carnegie, enabling the building of Carnegie Hall (1891), with an inaugural concert led by Walter Damrosch and Pyotr Il'yich Tchaikovsky. The Philharmonic Society finally merged with the Symphony in 1928, during the tenure of Arturo Toscanini, who helped it establish its world-class reputation through international tours.

Many great conductors would follow, among them: John Barbirolli, Bruno Walter, Dimitri Mitropoulos, Leonard Bernstein, Pierre Boulez, Zubin Mehta, Kurt Masur, and Lorin Maazel. Under Bernstein, the orchestra's reputation blossomed in new ways. Bernstein brought a youthful excitement to the music, engaging new audience members, particularly through television appearances. The advent of stereo recording allowed the New York Philharmonic to re-record much of the standard canon. It also got a new performance venue: Avery Fisher (originally Philharmonic) Hall at Lincoln Center.

Since Bernstein's retirement, many of the New York Philharmonic's directors have had to deal with threats to its standing among the world's great orchestras: competition on stages and on records from other strong American orchestras; internal and external economic difficulties; and balancing the traditional with new music in a way that satisfies its core audience.

As of the 2009-2010 season, the New York Philharmonic's music director has been Alan Gilbert, its first New York-born conductor and son of two of the orchestra's musicians. Just the year before, the orchestra and Lorin Maazel performed a historic concert in Pyongyang, North Korea, the first significant cultural visit by an American organization to the country since the '50s. In 2010, renovations began on Avery Fisher Hall to improve and update its acoustics.

The Philharmonic traces its recording history back to 1917, counting over 2,000 releases total, many of them award winners, and with hundreds of them available at any given time. As many other orchestras have done, it has created its own label, releasing live concert recordings physically and digitally, as well as offering podcasts and other new media.