In cartesian coordinates

Generally, one assumes that the pencil of curves is given implicitly by an equation

(0) ${\displaystyle$ :\ F(x,y,c)=0,\qquad } 1. example ${\displaystyle$ :\ x^{2}+y^{2}-c=0\ ,\qquad } 2. example ${\displaystyle y=cx^{2}\ \leftrightarrow \ y-cx^{2}=0\ ,}$

where ${\displaystyle c}$ is the parameter of the pencil. If the pencil is given explicitly by an equation ${\displaystyle y=f(x,c)}$, one can change the representation into an implicit one: ${\displaystyle y-f(x,c)=0}$. For the considerations below, it is supposed that all necessary derivatives do exist.

Step 1.

Differentiating implicitly for ${\displaystyle x}$ yields

(1) ${\displaystyle$ :\ F_{x}(x,y,c)+F_{y}(x,y,c)\;y'=0,\qquad } in 1. example ${\displaystyle$ :\ 2x+2yy'=0\ ,\qquad } 2. example ${\displaystyle$ :\ y'-2cx=0\ .}

Step 2.

Now it is assumed that equation (0) can be solved for parameter ${\displaystyle c}$, which can thus be eliminated from equation (1). One gets the differential equation of first order

(2) ${\displaystyle$ :\ y'=f(x,y),\qquad } in 1. example ${\displaystyle$ :\ y'=-{\frac {x}{y}}\ ,\qquad } 2. example ${\displaystyle$ :\ y'=2{\frac {y}{x}}\ ,}

which is fulfilled by the given pencil of curves.

Step 3.

Because the slope of the orthogonal trajectory at a point ${\displaystyle (x,y)}$ is the negative multiplicative inverse of the slope of the given curve at this point, the orthogonal trajectory satisfies the differential equation of first order

(3) ${\displaystyle$ :\ y'=-{\frac {1}{f(x,y)}}\ ,\qquad } in 1. example ${\displaystyle$ :\ y'=y/x\ ,\qquad } 2. example ${\displaystyle$ :\ y'=-{\frac {x}{2y}}\ .}

Step 4.

This differential equation can (hopefully) be solved by a suitable method.
For both examples separation of variables is suitable. The solutions are:
in example 1, the lines ${\displaystyle y=mx,\ m\in \mathbb {R} }$ and
in example 2, the ellipses ${\displaystyle x^{2}+2y^{2}=d,\ d>0\ .}$

In polar coordinates

If the pencil of curves is represented implicitly in polar coordinates by

(0p) ${\displaystyle$ :\ F(r,\varphi ,c)=0}

one determines, alike the cartesian case, the parameter free differential equation

(1p) ${\displaystyle$ :\ F_{r}(r,\varphi ,c)+F_{\varphi }(r,\varphi ,c)\;\varphi '=0,\qquad }

(2p) ${\displaystyle$ :\ \varphi '=f(r,\varphi )}

of the pencil. The differential equation of the orthogonal trajectories is then (see Redheffer & Port p. 65, Heuser, p. 120)

(3p) ${\displaystyle$ :\ \varphi '=-{\frac {1}{{\color {red}r^{2}}f(r,\varphi )}}\ .}

Orthogonal cardioids

Example: Cardioids:

(0p) ${\displaystyle$ :\ F(r,\varphi ,c)=r-c(1+\cos \varphi )=0,\ c>0\ .\ } (in diagram: blue)

(1p) ${\displaystyle$ :\ F_{r}(r,\varphi ,c)+F_{\varphi }(r,\varphi ,c)\;\varphi '=1+c\sin \varphi \;\varphi '=0,\qquad }

Elimination of ${\displaystyle c}$ yields the differential equation of the given pencil:

(2p) ${\displaystyle$ :\ \varphi '=-{\frac {1+\cos \varphi }{r\sin \varphi }}}

Hence the differential equation of the orthogonal trajectories is:

(3p) ${\displaystyle$ :\ \varphi '={\frac {\sin \varphi }{r(1+\cos \varphi )}}}

After solving this differential equation by separation of variables one gets

${\displaystyle r=d(1-\cos \varphi )\ ,\ d>0\ ,}$

which describes the pencil of cardioids (red in diagram), symmetric to the given pencil.