The Japanese Garden is nice, but I’d suggest starting at the (free) International Rose Test Garden just down the hill. The roses should be brilliant when you go, and you can get nice glimpses of the city, river, and Mt. Hood through the trees. If you enjoy that and want more, walk up the stairs to the Japanese Garden. But it’s $20 entry, can be crowded, and might still have access restrictions in June (i.e., you might need a reservation).
Your kids (and you) will enjoy Rimsky-Korsakoffee House for after dinner dessert and coffee. It’s a funky old victorian house converted into late-night hangout, often with a live piano player or other musician. There’s no booze and therefore no age restrictions, so you’ll often see a table or two of high school hipsters-in-training. Check the hours before you go, and take cash.
I’m required by city law to recommend visiting Powell’s bookstore. But really, I’d recommend it anyway. Have lunch and/or dinner at a food cart pod, where everyone gets their own choice. I’d recommend the Hawthorne Asylum for the right mix of good food and cool surroundings. I’m not as big a fan of the downtown pods, which seem to cater more toward people on their lunch break so the food is hit or miss.