Although the terraforming results were off the predicted charts, filling the ocean moon with lifeforms, the Despair Squids developed a unique hunting ability - their ink would induce absolute hope (in the case of the females) or absolute despair (in the case of the males) in their prey, taking away both the flight and fight instincts of their prey and the Despair Squids quickly became top of the food chain due to this trait.
In the end the Squids were too great a predator and even killed the Esperanto crew, causing them to commit suicide, and also simultaneously destroying most of the terraformed ocean planet's life, to the point that even fish killed themselves by voluntarily closing their gills.
When the Red Dwarf crew encountered it, in "Back to Reality", they were attacked in its usual way - shooting out hallucinogenic ink. This ink made them believe that their adventures on board the mining ship had been nothing more than them playing a Total Immersion Video Game. This illusion was designed to induce despair, as it made Rimmer realise that he was an absolute failure with no one else to blame (since his half brother, Lister, was successful, Rimmer could no longer blame his upbringing for his failures, as Lister had experienced the same upbringing and done well for himself), Lister being a fascist mass butcher of the common people in complete violation of his own moral standards, The Cat having no style as Duane Dibbley and Kryten - now "Jake Bullet" - taking a human life.
Just before they were about to commit suicide, Holly snapped them out of their hallucination by accessing Kryten's CPU and giving him instructions to release a gas that purged the ink from their systems. Holly took care of the male Despair Squid with limpet mines, turning it into "enough fried calamari to feed the whole of Italy". ("Back to Reality", Series V)
The Cat had a flashback to the hallucination, believing that once again he was Duane Dibbley. Holly informs the crew that Cat his hallucinating. Kryten prepares the Lithium Carbonate and sprays it on the Cat, bringing him back to sanity. When Cat comes around, he finds that he has been threatening to slice his own throat with a herring. (Flashback)
Wanting a "fishy" snack, the Cat smuggled a small squid on board Starbug (presumably keeping it in stasis or some such) and once he got back on Red Dwarf (seemingly before being imprisoned permanently, though how the squid survived the Starbug crash is never stated) he kept it in the ship's Water Tank.
However, it grew large and escaped, and the Cat convinced the crew that it had suddenly appeared and got them to go down to chase it in a Diving Bell. There they were all covered in its ink and succumbed to another hallucination. This time the scenario involved their being taken back to Earth in the year 2009, with their existence simply being fictional characters in a television show. At first it is a nightmare, having learned that their are doomed to die in the end of their current written adventure. However, after facing their Creator, they gained access to his typewriter and could write themselves a happy ending. It was then that they realised that it was a hallucination, and everyone but Lister fought off the ink's effect. Lister stayed, knowing that he could be with Kochanski there and not in reality (she left the ship to get away from him). In the end Lister convinced himself he could win her back and left the hallucination. The crew then discerned that the male squid, the one they first met, produced despair-inducing ink, but the ink of the female, the one they encountered this time, induces joy, though there is some confusion as to why it's still called a despair squid if it makes you happy.
The female Despair Squid, or "Hope Squid", was safely removed from the Water Tank, and dropped off at a new ocean moon to create some "happy sea-life". (Back to Earth - Series IX)
Behind the scenes
For "Back to Reality", a full model was built of the Despair Squid, tentacles and all. However it was decided to leave only the shadow of the squid in the broadcast episode, and the model itself was not seen. The model can be viewed on the model section of the Series V DVD.
For "Back to Earth", the squid was finally revealed in the Red Dwarf Water Tank, this time, in computer-generated form. However the blood and tentacles the gang are covered in were real props and ickiness.