Now Red Hen notices that this is not in character for Dumbledore and comes up with two alternatives:
A. Dumbledore is acting out of character because that's what JKR needs to get the job done.
B. Something happened during that conversation that we don't yet understand, but will make Dumbledore's actions make sense.
I'd like to posit a third theory. Dumbledore is acting in character, but in character for the Dumbledore of yesterday. There's a huge hole that is Dumbledore Before Harry. An even bigger one for Dumbledore Before Voldemort. And from what we know of the accomplishments of Dumbledore, the Harry filtered version that's a mix between Santa, Disney's Merlin, with hints of Gandalf peaking out probably wasn't always the person he was.
So what do we know? Dumbledore meets young Tom and is turned off within two minutes of conversation. Granted Tom's not all that likeable of a kid, and anyone who's dealt with unsavory characters can see that Tom's got a glowing neon warning sign flashing on and off over his head. Dumbledore can see that Tom needs watching, not just for his own sake, but to protect others. And Dumbledore does watch, but he does not shepherd. He provides Tom with no guidance. He sees an eleven year old in trouble and washes his hands of him.
What else do we know? The Dumbledore of the present is known for being a trusting, guiding, redemption hound. You were a rake, no problem, Dumbledore will give you a second chance. Joined the Death Eaters, but now you're sorry; come to Hogwarts and teach (or come to Hogwarts for the TriWizard Tournament.) Spent all year trying to kill me, and almost off two of your classmates; I'll hide you and your mom! Practically get one of your classmates killed by tricking him to get near another one when he's totally out of control (and, as a side note, practically turn one of your buddies into a killer); no problem, finish out the year as if nothing happened. Try to torture my pet student and get tricked into the Centaur's forest by your own greed; I'll go save you!
By the time we see him Dumbledore seems almost pathological in his need to give people second chances. He's so trusting he routinely puts other people at risk. After all, it's one thing for Hagrid to routinely bring his students into contact with critters that will kill them if given the chance; no one ever accused Hagrid of having an overabundance of brains. But Dumbledore is supposed to be brilliant.
Other writers have bandied about the theory that Dumbledore's lack of care is the final nail in the coffin of Tom Marvolo Riddle. I tend to disagree with this idea, but what if Dumbledore does not?
Perhaps, in the Cave, Dumbledore's "It's my fault." mantra is his confession that he believes that if he had taken Tom under his wing he could have nipped the budding psychopath, and fostered the charming rose underneath. Perhaps the potion allows him to see a vision of a future with Voldemort in charge. Perhaps Dumbledore's "Not the Children." and "Kill Me." is part of a conversation he is having with the Voldemort in his head. Perhaps I'm just tilting at windmills here...
But it is a pretty good reason for why Dumbledore forgives all these days.