Rock

Its fine to keep one, if you wish to harvest stone, metal or sulfur ore in the early game. When you pair it with a stone axe.
Smashing nakeds on the beach is fun for a while and gives you bones, if you want to upgrade to a bone club.

Always carry a rock with you no matter what you set out to do, because it's a "starting point" for a lot of supplies that you may otherwise have no way of getting. It is also used to "snowball" into superior tools when a sudden need to farm something more efficiently arises.
Examples of emergencies: * You run out of arrows, and you have wood in your inventory (or spot a nearby tree), but no stones, so you cannot craft extra arrows quickly. There is virtually no way to get the job done, unless you find something like stones lying on the ground (as 'meshes' of 50 stones spawn in the open world) and craft a new rock or use that for a really low amount of extra arrows.
* You have no farming tools, find a decaying base that you want to seal off, you'll need a TC and for that you will need wood. You cannot just go outside and find a way to turn the plenty of trees around into wood.
* You get to a loot-rich environment, e.g barrels, and you don't have an alternative item to break them with, or not enough arrows to gradually lose on that process (as 1 will always be stuck in the barrel).
* You need to destroy something that you encounter, which may provide you with some loot (e.g a locked box found in a decaying base), now you'll potentially miss out on this chance if you aren't carrying a gun or other melee weapon.
* You are dying from cold due to falling into the water at night, to save your life you'll want to make a campfire. But you don't have wood for a campfire, and you cannot farm a little wood in order to craft the campfire, because you have no rock or farming tool. You will die, and before you can get back, somebody loots you.
As you can see, a lot of disaster can easily be prevented by simply carrying a rock at all times, just in case!