Some things to remember about these occasional wasp sitings are:
First - seeing these wasps do not mean that there is a nest in the wall. Paper wasps prefer to be outside on overhangs or under a porch.
Second - Since these are queens looking of overwintering sites and not workers defending a nest, they are not aggressive and not likely to sting. Stings do occasionally happen, however, when one of these wasps crawls into a shoe left on the floor or in a jacket left hanging where they can access it. If you leave shoes on the floor or clothing hanging on a door, shake them before putting them on so you can see if a wasp flies out.
Third - Spraying indoors to control these wasps is futile. There isn't a specific target area to treat and the wasps are likely to be hiding in an area where it will not be affected. The most effective way is to just keep a rolled up newspaper nearby and swatting any unwelcome visitors.
For additional information:
http://www.ces.ncsu.edu/depts/ent/notes/Other/note144/note144.html
http://insects.ncsu.edu/Urban/paperwasp.htm
Information for this post was obtained from an email from Michael Waldvogel.