To point a subdomain to a name servers you need to create an NS record for the subdomain:
app.example.com NS ns1.appserver.com
This will make all queries go to ns1.appserver.com
*.example.com NS ns1.squaresites.com
The second record should catch all subdomains that don't have their own records (of any kind).
Realize that chaining your DNS lookups this way adds more time to DNS resolution and adds another potential point of failure in the resolution process. If the parent's name servers are down, they won't able to deliver the NS records to tell the client's resolver to continue the lookup process through a delegated name server.
Consider if using DNS "A" or "CNAME" records at the parent's DNS server would be acceptable alternative.
For more detail on how DNS resolution is delegated see this reference on serverfault.com: