Another option, in addition to using polar coordinates or calculating the coordinates of the terminal point of the dashed line using trigonometry would be use of clipping. It is probably an overkill in this case, but it could be useful for more complicated situations:
\documentclass[paper=a4, fontsize=12pt]{scrartcl} \usepackage[T1]{fontenc} \usepackage{amsmath,amsfonts,amsthm} % Maths \usepackage{graphicx} \usepackage{float}\usepackage{tikz}\usetikzlibrary{arrows,% plotmarks}\begin{document} \begin{figure}[H] \centering \begin{tikzpicture} % Axis \draw[thick,->,black] (-3,0)--(3,0) node[below] {$k_x$}; % x axis \draw[thick,->,black] (0,-3)--(0,3) node[left] {$k_y$}; % y axis \draw[black,thick] (0,0) circle (2.5cm); \begin{scope} \draw[clip] (0,0) circle (2.5cm); \draw[ultra thick,blue,dashed] (0,0) -- (3,3); \end{scope} \end{tikzpicture}\end{figure}\end{document}Notice that the circle is drawn twice, the second time inside the scope environment with the clip option. That will caused everything in this environment to be clipped to the inside of the circle.