Go to Settings > Data and Storage > Storage Usage to selectively clear your cache – or set Keep Media to regularly remove unused items. Media and files stay available to re-download whenever needed.
On Android, you can also store your Telegram data on your SD Card.
Here’s what the iOS version looks like, with additional settings like a Maximum Cache Size.