Terry Downes, the former middleweight champion who defeated Sugar Ray Robinson, died “peacefully” Friday, according to his family. He was 81.

The Londoner had been the oldest surviving former world champion from the U.K.

Downes’ family confirmed his death in a statement: “Terry was a beloved husband, father and grandfather to his wife Barbara, his five children and eight grandchildren, and will be enormously missed.”

Downes was born in England but moved with his family at 16 to the U.S., where he lived for five years. He fought in the service but began his boxing career in earnest after returning to England in 1957.

He faced then-unknown Dick Tiger, a future hall of famer, in just his third pro fight but retired on his stool. Afterward, he was asked who he wanted to fight next and responded: “The bastard who made this match.”

Downes won the middleweight championship with a 10th-round stoppage of Paul Pender in a July 1961 rematch after losing by a seventh-round stoppage to Pender in January of that year. However, the reign was short-lived. Pender won the trilogy by outpointing Downes nine months later.

“He was a very awkward bloke. He was long, skinny. Very classy, but an awkward fighter,” Downes told THE RING. “They used to call him ‘The Boston Crab.’ He used to grab hold of you whenever you got inside; he was renowned for it.”

Downes never won another title, but he would secure a legacy-defining victory over a 41-year-old Robinson in 1962.

“I didn’t beat Sugar Ray, I beat his ghost,” said Downes, who won a 10-round decision.

He also beat future middleweight titleholder Joey Giardello in 1960. “He was such a great fighter. Great timing; timing’s everything. He was just quick. He could probably drop something and catch it before it hit the floor. He had fast hands, fast everything.”

Downes moved up to light heavyweight shortly after the Giardello fight and earned a shot at titleholder Willie Pastrano in 1964. Downes was stopped in Round 11 and never fought again, retiring with a record of 35-9 (28 knockouts).

He fought seven times against five world titleholders, going 3-4 in those bouts.

Downes never gained entry into the International Boxing Hall of Fame but will be remembered fondly for his aggressive style and thoughtful interviews.