When you planned your hike's route, you estimated how long reaching your destination would take. The moment you begin to head back is known as the turnback time. For safety reasons, you should have a pre-set turnback time so that you can make it back to the vehicle before dark and so that a responsible person back in civilization knows when to report you missing.
You always should begin heading back to your vehicle at your turn-back time, even if you don't reach your destination - and with children, you may not always make your objective. The turnback time marks the halfway point of your journey, so walking to your vehicle will take about as much time as did reaching the point you're at.
There are other cases when you'll want to return, regardless of your pre-set turnback time. If you see a thunderstorm approaching or if fog is beginning to roll in, the time has come to head back. When you reach difficult to cross terrain, such as a water crossing that you had not anticipated, it's better to turn back. If you or your children are tired, shivering or injured, it's definitely time to go home.
Read more about day hiking with children in my guidebook Hikes with Tykes.