Removes the given pointer from the list, without calling the destroy function, even when it is configured.