Hooking up with the Greenwich Village poets Tuli Kupferberg and Ed Sanders and drummer Ken Weaver, Stampfel and Weber then became part of the Village Fugs, who were at least as strange as the Holy Modal Rounders, but more interested in playing rock music. The Village Fugs' debut album, The Village Fugs Sing Ballads of Contemporary Protest, Point of Views, and General Dissatisfaction, was released by the Broadside subsidiary of Folkways Records in 1965. The group simplified its name to the Fugs, and the album was reissued by the independent jazz label ESP as The Fugs First Album. ESP initially rejected the band's second recording as obscene, and they followed instead with an album called The Fugs, which broke into the pop charts in July 1966 and reached the Top 40 during a six-month chart stay. (The Fugs First Album then belatedly reached the charts and peaked in the Top 100.) With that, ESP created one of the first advisory stickers ever made for a record ("For Adult Minds Only") and finally issued the intended second release as the Fugs' third album, Virgin Fugs (1966). (Fugs 4, Rounders Score, a compilation album released by ESP in 1975, contained previously unreleased outtakes.)

Stampfel and Weber left the Fugs and reconvened the Holy Modal Rounders for their own ESP album, Indian War Whoop (1967), which added to the group keyboard player Lee Crabtree and drummer Sam Shepard from Stampfel's spin-off group the Moray Eels. In 1968, they signed to Elektra Records and released their fourth album, The Moray Eels Eat the Holy Modal Rounders (though, as Stampfel later pointed out, it was the other way around), with a lineup that retained Shepard while Richard Tyler replaced Crabtree on keyboards and John Wesley Annis was added on bass. Annis was still on board for the fifth album, 1971's Good Taste Is Timeless, released by Metromedia Records, but Tyler was gone, replaced by multi-instrumentalist Robin Remaily (a childhood friend of Weber's and the author of "Euphoria" from the band's first album), and Shepard had left to focus on his playwriting (the band had played in the pit for his 1970 off-Broadway play Operation Sidewinder and even recorded an album of its music for Columbia Records that went unreleased); his replacement was drummer Mike McCarty.

Alleged in Their Own Time, featuring Stampfel, Weber, Remaily, bass player Dave Reisch, and banjoist Luke Faust, was recorded in 1972 and released in 1975 on Rounder Records, which had been named after the group. By then, Weber had moved to Oregon with the other members of the Holy Modal Rounders, except Stampfel, who remained in New York and eventually led a band with Reisch called the Unholy Modal Rounders that appeared on Michael Hurley's celebrated 1976 album Have Moicy! Stampfel and Weber reunited, backed by members of the Holy Modal Rounders spin-off group the Clamtones (among them Tyler, Reisch, and Romaily), for Last Round (1978), released on Adelphi Records, its title suggesting this was the Holy Modal Rounders' final statement. Maybe that was why, in 1981, when Stampfel and Weber got together again for Going Nowhere Fast, released on Rounder, they credited their first duo album since 1965 to Stampfel & Weber, not the Holy Modal Rounders.

After that, they went their separate ways again, with Stampfel releasing several albums with his group the Bottlecaps and Weber taking on occasional engineering assignments. In 1996, Weber relocated to the East Coast and again reunited with Stampfel. The two began to perform together, and in 1999, joined by Reisch and guest slide guitarist Don Rooke, they recorded Too Much Fun!, the first Holy Modal Rounders album in 21 years and their first recorded collaboration in 18 years. In 2001, Rounder released I Make a Wish for a Potato, a compilation album of recordings by the Holy Modal Rounders and their associates.