Downloading per se is not illegal but downloading copyright materials without the permission of the copyright owner is, that is depending on the laws your country have. Some countries treat and punish copyright infringement as a criminal offense. Legality is one thing, enforcement is another.

In some some countries, like where I am, the government does not run after file sharers (as of the moment) but they conduct raid of pirated discs. Even if the government would run after file downloaders, there are many of them. Chances are only a handful will be prosecuted just to set an example.