"The two companies have now jointly come to the conclusion that these talks will no longer be continued," Porsche said in a statement on Friday.
Porsche said the two companies could not agree on a partnership that included both an engine partnership and also the F1 team.
Porsche said taking part in F1 remained an attractive option and it will continue to monitor the possibility as F1's rule changes are finalized.
Volkswagen Group's former CEO Herbert Diess said in May that Porsche and Audi would join Formula One, which has so far been dominated for years by German rival Mercedes-Benz.
F1 plans to switch to cars running on synthetic fuel from 2026 as part of a plan to become carbon neutral by the end of the decade, aligning the franchise more closely with VW Group's own climate goals.
In July, Bloomberg reported that talks between Porsche and Red Bull were stuck on how much technology Porsche would supply to Red Bull Racing, compared to its Red Bull-owned sister racing team, AlphaTauri, as well as which brand would have the ultimate power to name drivers for a joint race team.
Red Bull team boss Christian Horner told motorsport.com at the Italian Grand Prix that it had become clear during talks with Porsche that "there was a strategic non-alignment."
Horner ruled out a Porsche takeover last Friday and said any partnership would have to be on the Formula One team's terms.
Championship-leading Red Bull have set up their own powertrain company, with more than 300 people working on an engine for 2026.
"The premise (of discussions) was always a partnership on eye-level, which would include the team alongside a motoring partnership. This could not be realized," Porsche's statement said.
Audi said in August it will build an engine in Germany and enter F1 in 2026 with an existing team, likely to be Sauber.