Federer entered the season-ending event in London as the clear favourite after world number one Nadal had to pull out with a knee problem following his opening round-robin loss to David Goffin.
The number-two ranked Swiss ace meanwhile, ended the round-robin stage undefeated with wins over Jack Sock, Alexander Zverev and Marin Cilic.
However, he suffered a shock semi-final defeat at the hands of Goffin, who came back from a set down to become one of the few players to defeat both Federer and Nadal in the same tournament.
The event was eventually won by Grigor Dimitrov, who will end the year as the world number three and Rusedski believes Federer missed a golden opportunity.
Not only was there no Nadal in his way, despite his superior 4-0 record against the Spaniard in 2017, but there was also the absence of Djokovic, who had won four of the previous five ATP Finals events.
Along with this, Federer could have been as close as 140 points behind Nadal in the ATP tennis rankings heading into the New Year had he finished the event unbeaten.
"You're never going to get a better opportunity," Rusedski told Sky Sports via Metro. "With Rafa being out, Djokovic and all those guys."
Despite the end-of-year blemish, the 36-year-old can look back on this year with pride as he ends 2017 with a tour-leading seven titles, an Australian Open title and a record-breaking eighth Wimbledon crown.
"He will be a little bit disappointed but what a great season he's had," Rusedski added.
Federer will now be hoping to emulate his performances next year as he prepares to defend his points with a host of injured tennis stars making their comebacks.