The reference thing isn’t the whole story, though. The brain has a natural sorting mechanism. When you first store a memory, there are a lot of details. If you use the memory, those details are reinforced. But if you don’t use a memory, the memory fades. I’m not sure of the physical mechanism, but a lot of times you wind up recreating those details from other details, and you can definitely get that wrong. While this happens a little bit early on, the longer a memory is disused, the more likely it is to happen.
Anyways, my point is that you can only think about so many things in a given time period, so, eventually information is going to disappear from your brain as you fill it up. So the storage is not infinite. You just have a built in deletion system based on bandwidth.
You could simulate this with a computer that deleted files that weren’t accessed. Some filesystems do this, although they usually have a way to set something to never be deleted. I wouldn’t call them infinite, either.