Setting up custom scans using Spider web Security Scanner

Schedule and run custom scans on a deployed application using Web Security Scanner in the Google Cloud Panel. Web Security Scanner supports scans for public URLs and IPs that aren't backside a firewall.

The following video shows the steps to set up Web Security Scanner, and provides information about how to use the dashboard. The setup steps are described in text later on this page.

Before you brainstorm

To set up custom scans using Web Security Scanner:

  • You must have a deployed application on a public URL or IP.
  • Your organization must have Security Control Center enabled.

Earlier you browse, carefully audit your awarding for any feature that may affect data, users, or systems beyond the desired scope of your scan.

Because Web Security Scanner populates fields, pushes buttons, clicks links, and other interaction, yous should use it with caution. Spider web Security Scanner might activate features that modify the state of your data or organization, with undesirable results. For case:

  • In a blog application that allows public comments, Spider web Security Scanner might post exam strings as comments on all your blog articles.
  • In an email sign-upwardly page, Web Security Scanner might generate big numbers of test emails.

For tips about how to minimize chance, see best practices to forbid unintended consequences.

Enabling Spider web Security Scanner

Enable Web Security Scanner to create and run custom scans. Security Command Center must be enabled for your arrangement.

Step one: Deploying a test projection

To consummate Web Security Scanner setup for custom scans, you need the URL of a Compute Engine, Google Kubernetes Engine (GKE), or App Engine awarding that is already deployed. If you don't have a deployed application, or if yous desire to endeavour out Web Security Scanner with a exam application, deploy the test App Engine application. Use the language of your choice:

  • Coffee
  • Python
  • Become
  • PHP

Step 2: Assigning IAM roles

To run a Spider web Security Scanner browse, you must have one of the post-obit Identity and Access Management (IAM) roles for the project you lot want to scan:

  • Editor
  • Possessor

To add i of these roles:

  1. Become to the IAM & Admin page in the Deject Console.
    Become to the IAM & Admin page
  2. Click the Project selector drop-down list.
  3. On the Select from dialog that appears, select the projection that you want to scan using Spider web Security Scanner.
  4. On the IAM page, side by side to your username, click Edit.
  5. On the Edit permissions panel that appears, click Add another part, and then select one of the following roles:
    • Project > Owner
    • Project > Editor
  6. When you're finished adding roles, click Salve.

Learn more well-nigh Web Security Scanner roles.

Stride 3: Running a scan

When yous set a scan, information technology's queued to run later. Depending on electric current load, it might exist several hours before a scan executes. To create, salvage, and run a scan:

  1. Become to the Spider web Security Scanner folio in the Cloud Panel.
    Go to the Web Security Scanner folio
  2. Select the project that contains the deployed application you desire to scan.
  3. To gear up up a new browse, click New scan:

  4. On the Create a new scan page that loads, set up the following values:

    1. Under Starting URLs, enter the URL of the application you want to scan.
    2. Nether Schedule, select Weekly.
    3. Under Next run on, select a date.

    The box to Export to Security Command Eye is automatically checked. If y'all've enabled Web Security Scanner as a Security Control Center security source, scan results tin exist displayed on the Security Command Eye dashboard.

    For this first scan, use the default scan without irresolute whatever other values on the Create a new scan folio. For more than information about scan settings, run into Scanning an app.

  5. To create the browse, click Salve.

  6. On the Web Security Scanner page, click the scan name to load its overview page, and then click Run scan.

    The browse will be queued, and then information technology will run at a future fourth dimension. Information technology might take several hours before the scan runs.

  7. The browse overview folio displays a results department when the scan completes. The following image shows example scan results when no vulnerabilities are detected:

    If you've enabled Web Security Scanner as a Web Security Scanner security source, browse results are also displayed on the Spider web Security Scanner dashboard.

    To display details about a specific finding, click the finding proper noun in the scan results.

You have now completed a bones Web Security Scanner browse. If y'all scanned your own application, learn how to customize the browse in the scanning an app section on this page.

If you deployed a test application to run the scan, complete the following clean up step on this folio to avoid incurring App Engine charges for the application.

Footstep 4: Cleaning up

  1. In the Cloud Console, go to the Manage resources folio.

    Get to Manage resources

  2. In the project list, select the project that y'all want to delete, and then click Delete.
  3. In the dialog, blazon the project ID, and so click Shut down to delete the projection.

Scanning an app

Set upward a custom scan for your app using a test account.

Footstep i: Creating a test account

When y'all scan your app, information technology's best to use a test account that doesn't accept access to sensitive information or harmful operations. Create a exam business relationship that can sign in to your app. Note the login credentials to provide for hallmark when creating a browse. The credentials enable you to utilize the test business relationship to scan data.

Footstep 2: Creating a scan

  1. Get to the Web Security Scanner page in the Deject Console.
    Go to the Web Security Scanner page
  2. Click Select, and so select a project that already has an App Engine, Compute Engine, or GKE awarding deployed.
  3. To brandish the new browse form, click Create browse or New scan.
  4. To add together values to the new scan form, use the following table as a guide:
    Field Clarification
    Starting URLs

    A unproblematic site ordinarily requires simply ane starting URL, like the abode, main, or landing page for the site, from which Web Security Scanner tin can find all other site pages. However, Web Security Scanner might not find all pages if a site has:

    • Many pages
    • Islands of unconnected pages
    • Navigation that requires complex JavaScript like a mouseover-driven multilevel carte

    In such cases, specify more starting URLs to increase browse coverage.

    Excluded URLs To reduce complexity, exclusions are defined using a simplified proto-language using one or more than * wildcards, instead of requiring a valid regular expression. For details and sample valid patterns, see Excluding URLs subsequently on this folio.
    Authentication > Google account

    You lot can create a test account in Gmail and then apply the account to browse your production. If you are a Google Workspace client, you can create test accounts within your domain, for example, exam-business relationship@yourdomain.com. In Web Security Scanner, these accounts work like Gmail accounts. Two factor hallmark is not supported.

    Google enforces a real name policy on Google accounts. If the name on your test account doesn't look real, the account might be blocked.

    Hallmark > Identity-Aware Proxy blastoff

    To protect resources with Identity-Aware Proxy, run across the IAP guide.

    To use Web Security Scanner with an IAP-protected resource, get-go grant access to the Web Security Scanner service business relationship:

    1. Go to the IAP page in the Cloud Console.
    2. Select the project that you want to use with Web Security Scanner.
    3. Select the awarding resource you want to browse, and then click Add together Principal on the Info Panel.
    4. In the New principals box on the Add together principals panel, enter the Web Security Scanner service account in the class of

      service-project-number@gcp-sa-websecurityscanner.iam.gserviceaccount.com.

    5. On the Select a part drop-downwardly list, select Cloud IAP > IAP Secured Web App User.
    6. When you're finished adding roles, click Relieve.

    Next, add together the OAuth client ID to the scan. Web Security Scanner can only scan applications that are protected by a single OAuth Client ID. To add together the OAuth client ID:

    1. Go to the IAP page in the Cloud Console.
    2. Select the projection that yous want to employ with Spider web Security Scanner.
    3. On the Overflow menu, select Edit OAuth Client.
    4. On the Client ID for spider web application window that appears, copy the Customer ID.
    5. Go to the Spider web Security Scanner folio in the Cloud Console.
    6. Under Authentication, select Identity-Enlightened Proxy blastoff.
    7. In the OAuth2 Client ID box, paste the OAuth client ID that you copied, and then click Save.
    Authentication > Non-Google account

    Select this option if you take created your ain authentication system and you lot aren't using Google Business relationship services. Specify the login grade'southward URL, the username, and the password. These credentials are used to sign in to your awarding and scan it.

    Web Security Scanner attempts heuristics to sign in to your application, and browse information technology. Specifically, this method looks for a two field login-form that includes a username field and password field. The login action must issue in an authentication cookie for the scanner to go along its browse.

    Common issues can cause custom login to fail include:

    • Using non-standard HTML grade fields, for instance, non using a password type.
    • Using a complicated login form, for case, a form that has more than than a single username and password field.
    • Not saving an authentication cookie on successful login.
    • In some situations, the scanner is denied by counter-measures that are meant to protect against bots, DDOS, and other attacks.

    We recommend using Identity-Aware Proxy integration for the nigh consistent feel with authenticated scanning of applications.

    Schedule You can fix the scan to run daily, weekly, every two weeks, or every four weeks. It'south best to create a scheduled scan to ensure that time to come versions of your application are tested. Also, considering we occasionally release new scanners that find new bug types, running a scheduled scan offers more coverage without manual effort.
    Run scans from a predefined set of source IPs
    Preview     		Select this choice to restrict scan traffic to a predefined set of IP addresses. This lets you lot enable the scanner to access applications behind a firewall, but may limit the scope of the scan. To modify your firewall rules to allow Web Security Scanner traffic, run into Configuring the firewall later on this page.
    Export options Select this choice to automatically consign scan configurations and scan results to Security Command Center.
    Ignore HTTP status errors This pick controls whether a loftier number of HTTP condition errors—for example, 400 Bad Request—during a scan will crusade the browse to be reported as a failure. If the pick is selected, status errors are ignored. If the pick is not selected, and the per centum of status errors exceeds a predetermined threshold, the browse is reported as a failure.
  5. When you're finished adding values, click Relieve. You lot can now run the new scan.

By default, Web Security Scanner uses randomly assigned IP addresses during each run. To make Web Security Scanner IP addresses predictable, consummate the steps to enable scans from static IPs later on this page.

Step 3: Running a scan

To run a scan:

  1. Sign in to the examination account that y'all used to create the browse.
  2. Go to the Web Security Scanner page in the Cloud Console.
    Go to the Web Security Scanner page
  3. Click Select, and then select the project that you created the scan in.
  4. Under Scan configs, click the name of the browse that you want to run.
  5. On the browse details page, click Run.

The scan is placed in a queue, and there might exist a delay before information technology runs. It can take several minutes or many hours to run, depending on the system load and features similar:

  • Site complexity
  • Number of actionable elements per page
  • Number of links
  • The corporeality of JavaScript on the site, including navigation

You lot tin can set up and sew together to x different scans before yous demand to delete or clean upwards previously saved results.

Viewing custom scan results

The status and results of a custom scan are displayed on the browse details page in the Cloud Console. To view scan results:

  1. Sign in to the test account that you lot used to create the scan.
  2. Go to the Web Security Scanner page in the Cloud Panel.
    Go to the Spider web Security Scanner page
  3. Click Select, so select the project that contains the scan that y'all want to review.
  4. Nether Scan configs, click the proper name of the scan that you lot want to review.

The scan details folio loads and displays results from the most recent browse. If a scan is in progress, the Results tab displays the current completion pct. To display results from previous scans, select the scan date and fourth dimension from the drop-downwardly listing.

Details for completed custom scans include:

  • The Results tab displays a list of vulnerabilities the browse found, if any.
  • The URLs crawled tab displays a list of URLs that the browse checked.
  • The Details tab includes:
    • Starting URLs
    • Hallmark
    • User agent
    • Maximum scan speed every bit queries per second (QPS)

You can detect more information most the scan in the project logs page.

Editing a custom scan

To edit a custom scan:

  1. Sign in to the test account that you used to create the scan.
  2. Go to the Spider web Security Scanner page in the Cloud Console.
    Go to the Web Security Scanner page
  3. Click Select, and then select the project that contains the scan that you desire to edit.
  4. Under Scan configs, click the proper name of the scan that you lot want to edit.
  5. On the scan details page that appears, click Edit.
  6. On the Editing [scan name] page that appears, make changes that you want, and so click Save.

The edited custom scan runs when it's adjacent scheduled, or you can manually run it to get updated results.

Deleting a custom scan

To delete 1 or more custom scans:

  1. Sign in to the test account that you lot used to create the scan.
  2. Become to the Web Security Scanner folio in the Cloud Panel.
    Go to the Web Security Scanner page
  3. Click Select, and then select the project that contains the scan that you desire to edit.
  4. Under Browse configs, select the checkbox side by side to ane or more scans that yous want to delete.
  5. Click Delete, and and so click Ok.

All scans that you selected are deleted.

Setting up a scan from static IPs

This section describes how to enable Web Security Scanner custom scans from static IP addresses. When yous enable this feature, Spider web Security Scanner uses anticipated IP addresses to scan your public Compute Engine and Google Kubernetes Engine applications. This feature is in Preview, and the Web Security Scanner IP addresses might change in a time to come release.

Before yous begin

To use the Web Security Scanner custom scans from static IPs characteristic, you need:

  • A public Compute Engine or GKE application. This feature currently doesn't support App Engine applications.
  • A scan created with no hallmark, or with Google business relationship authentication. This characteristic currently doesn't support scans that use non-Google account authentication.

Step 1: Configuring the firewall

  1. Go to the Firewall rules page in the Deject Panel.
    Go to the Firewall rules page
  2. Click Select, so select your projection.
  3. On the Firewall rules folio that appears, click Create Firewall Rule.
  4. On the Create a firewall dominion page, prepare the following values:
    1. Proper noun: enter web-security-scanner or a similar name.
    2. Priority: select a higher priority (lower number value) than all of the rules that deny egress traffic to your awarding.
    3. Source IP ranges: enter 34.66.eighteen.0/26 and 34.66.114.64/26.
    4. Protocols and ports: select Allow all or specify the protocols and ports for your application. Usually, you lot can select the tcp checkbox so enter eighty and 443 for the ports.
  5. When you're finished setting values, click Create.

Stride 2: Configuring the browse

Subsequently you configure your firewall to allow the Spider web Security Scanner predictable IP addresses, configure the scan to use pre-defined IPs:

  1. Go to the Web Security Scanner page in the Cloud Console.
    Go to the Web Security Scanner folio
  2. Click Select, and then select your projection.
  3. Create a new browse or edit an existing scan.
  4. Select the Run scans from a pre-divers set of source IPs checkbox.
  5. Save the browse.

The next fourth dimension the scan runs, it will scan the public Compute Engine and GKE applications that are behind the firewall.

Excluding URLs

You tin specify i or more excluded URL patterns to avoid testing sections of a site during a custom scan. Web Security Scanner doesn't request resources that match whatever of the exclusions. The following sections describe the pattern matching that Web Security Scanner uses.

URL pattern matching

Excluded URL matching is based on a set of URLs defined by lucifer patterns. A lucifer pattern is a URL with five segments:

  • scheme: for example, http or *
  • host: for example, world wide web.google.com or *.google.com or *
  • path: for example, /*, /foo*, or /foo/bar. *
  • query: for example, ?*, ?*foo=bar*
  • fragment: for example, #*, #access

Post-obit is the basic syntax:

                    <exclude-design> := <scheme>://<host><path><query><fragment> <scheme> := '*' | 'http' | 'https' <host> := '*' | '*.' <whatever char except '/' and '*'>+ <path> := '/' <any chars except '?' or '#'> <query> := '?' <whatsoever chars except '#'> <fragment> := '#' <any chars>

The * in each part has the following function:

  • scheme: * matches either HTTP or HTTPS.
  • host:
    • * matches any host
    • *.hostname matches the specified host and any of its subdomains.
  • path: * matches 0 or more characters.

All segments are not required in an excluded pattern.

  • If the scheme segment is not specified, it defaults to *://.
  • The host segment must always be specified.
  • If the path segment is not specified, information technology defaults to:
    • /*, if query and fragment segments are not specified. This value matches any path or no path.
    • /, or an empty path, if either thequery or fragment segment is specified.
  • If the query segment is non specified, information technology defaults to:
    • ?*, if the fragment segment is not specified. This value matches whatever query or no query.
    • ?, or an empty query, if the fragment is specified.
  • If the fragment segment is not specified, it defaults to #*, which matches whatever fragment or no fragment.

Valid Pattern Matches

The following table provides examples of valid patterns:

Pattern Behavior Sample matching URLs
http://*/* Matches whatsoever URL that uses the HTTP scheme.

http://www.google.com/

http://example.org/foo/bar.html

http://*/foo* Matches any URL that uses the HTTP scheme, on whatever host, if the path starts with /foo.

http://case.com/foo/bar.html

http://www.google.com/foo

https://*.google.com/foo*bar Matches whatsoever URL that uses the HTTPS scheme and is on a google.com host — like www.google.com, docs.google.com, or google.com — if the path starts with /foo and ends with bar.

http://www.google.com/foo/baz/bar

http://docs.google.com/foobar

http://example.org/foo/bar.html Matches the specified URL. http://case.org/foo/bar.html
http://127.0.0.1/* Matches whatsoever URL that uses the HTTP scheme and is on the host 127.0.0.1.

http://127.0.0.1/

http://127.0.0.1/foo/bar.html

*://mail.google.com/* Matches any URL that starts with http://mail.google.com or https://mail service.google.com.

http://mail.google.com/foo/baz/bar

https://mail.google.com/foobar

*://*/foo*?*bar=baz* Matches any URL where the path starts with /foo and has the query parameter bar=baz. https://www.google.com/foo/example?bar=baz
google.com/app#*open* Matches whatever URL with a google.com host where the path starts with /app and has the fragment open. https://www.google.com/app/example#open

Invalid pattern matches

The following table provides examples of invalid patterns:

Pattern Reason
http://www.google.com The URL doesn't include a path.
http://*foo/bar * in the host must be followed by a . or /.
http://foo.*.bar/baz If * is in the host, information technology must be the kickoff character.
http:/bar The URL is scheme separator isn't properly formed. The "/" should exist "//".
foo://* The URL scheme is invalid.

What's next

  • Acquire how to remediate Web Security Scanner findings.