Few books or films have been as thoroughly analyzed and obsessed over as the Harry Potter series, all the more so since the the generation that came of age with Harry also came of age with the internet. One objection frequently lobbed at the series involves the Time-Turners, devices introduced in the third book that allow witches and wizards to travel through time. These commentators claim that the existence of possible time travel in the series creates a massive plot hole and that Harry just needed to go back in time and stop Voldemort from rising to power in the first place. They're totally wrong.
So why doesn't Harry just go back in time to defeat Voldemort?
Perhaps we should first understand why and how the time travel plot in Prisoner of Azkaban works. Harry and Hermione go back three hours in time and are thus able to save both Buckbeak and Sirius and they're only able to do it because of a few very, very specific reasons. First, they go back a short enough period of time that they are able to remember in detail their own movements, as well as the movements of the relatively few other persons involved. Second, this part of the story takes place in the outdoors where there are few people and thus fewer chances of them being seen and creating a paradox. Third, and most importantly, their tasks are hyper-specific - release Buckbeak after the official delegation has seen him, thus exonerating Hagrid of any wrong-doing, and release Sirius from his cell before the Dementors arrive to perform the dreaded Dementor's kiss. Thus, the risks of time travel are mitigated by intimate and detailed knowledge, a convenient and familiar location, and specific missions that are narrow in scope, if not in difficulty. Most importantly, the period of time is relatively quite small, only three hours.
Now let's say that Harry goes back in time to prevent Voldemort from creating the first Horcrux. It's revealed in Half-Blood Prince that as a student Tom Riddle created at least two Horcruxes while still a student - the Gaunt ring and the diary that played so significant a role in Chamber of Secrets - so Harry would have to travel fifty years back in time. This already creates exceptional difficulties. How can Harry have specific enough knowledge to accomplish his aim? He must know exactly where Riddle was when he made the first Horcrux, and at exactly what time, while somehow ensuring that he remains unseen. Not to mention, Voldemort, even at sixteen, would have been a formidable opponent, fully capable of murder.
If Harry were to be killed, obviously that would be bad for Harry, but more seriously, it creates a paradox - Harry would then die decades before he was born. But far more frightening possibilities arise as far as paradoxes are concerned if Harry is successful. The further back in time one goes, the more complex the results of the changes one makes in the past. If Voldemort never rises to power, the implications could be far-reaching and unexpectedly negative. As we learn in Deathly Hallows, Harry's mother Lily begins her stint at Hogwarts as Snape's best friend. The rift between them is caused by a number of factors, but prominent among them is Lily's disgust with Snape's affiliation with the Death Eaters and his increasing infatuation with prejudicial ideas about blood status. But if Voldemort was prevented from rising to power, the Death Eaters would never have existed. If the friendship between Lily and Snape remains unbroken, it seems fairly unlikely that Lily would have given James Potter the time of day, given his cruel treatment of her best friend. And if Lily and James never get together, Harry would never have been born and it would not be possible for him to go back in time and defeat Voldemort - paradox.
Even aside from the increased difficulties and high possibilities of creating paradoxes that would literally destroy the time traveler, it is not clear that Time-Turners can be used to travel forward in time. We only ever see or hear about witches and wizards traveling backwards and then living through the period until they arrive at the present in which they traveled back. Thus, beyond the pragmatic difficulties of the task, and barring the unknown possibility of traveling forward in time, Harry would then have to live for fifty years, staying strictly out of sight for fear of interfering with the fabric of time and creating more and more paradoxes.
Even Voldemort, evil as he is and with no regard for others, doesn't want to mess with time. His Death Eaters infiltrate the Department of Mysteries and one might wonder why Voldemort doesn't order them to steal a Time-Turner, especially since they are right there for the taking. Voldermort certainly has his regrets - he might want to go back and attack the Potters in a different way, thereby denying Harry the protection of his mother, or perhaps he might want to go back and steal the philosopher's stone before Hagrid removes it from Gringotts. There are dozens of moments that Voldemort could conceivably want to change in order to consolidate his own power more definitively and earlier.
But he doesn't even try. That's because Voldemort is no dummy. Hermione describes to Harry the terrible things that happen when wizards encounter their past or future selves and those concerns are not inconsequential: "Loads of [wizards] ended up killing their past or future selves by mistake!" But even beyond that danger, even small, seemingly inconsequential changes can have momentous implications that are all but impossible to predict. The risks are of such a magnitude and the difficulties so extreme that to chance time travel would be foolhardy at best, insane at worst. Whether one would wish to use time travel for good or for evil, its practical use is progressively nullified the further back one travels. Therefore, possible time travel does not create a plot hole in Harry Potter.