Configuring QaaS Reporter Environment Variables

The QaaS Reporter takes all of its configuration from environment variables, so there is nothing to edit in your test code. That keeps secrets out of source control. It also moves the hard part somewhere less obvious: getting the variable into the process that runs your tests.

This article covers where to set the variables in each common environment. For what each variable does, see Uploading GameDriver Test Results to QaaS, which has the full reference table.

Why the key lives outside your code

Putting the key in a constant or a config file next to the tests looks easier. The Reporter does not support it, for five reasons. The same reasons decide which of the options below are safe.

A secret committed to source control is committed permanently. Deleting the line in a later commit does not remove it. It stays in the history, and everyone who cloned or forked the repository already has a copy. The only remedy is to revoke the key and issue a new one.

Test projects travel further than most code. They get copied into sample repositories, shared with contractors, forked for training, attached to bug reports and printed into CI logs. A test suite is one of the most-copied artifacts in a studio, and one of the worst places to keep a credential.

A QaaS key identifies a person, not a repository. Keys are issued to a named individual and revoked individually. A key embedded in shared code identifies no one, and revoking it breaks the whole team instead of one person.

The same code needs different values in different places. Your machine, a teammate’s machine, CI and a second game each need a different key or a different QAAS_PROJECT. This is what environment variables are for: configuration that changes where the code does not.

The integration is safe to merge before anyone has a key. With QAAS_API_KEY unset the Reporter does nothing at all, so you can add the two lines and merge them today. That works only because the key was never in the code.

Caution

Environment variables are not a vault. They are readable by any process running as you, they appear in CI logs the moment something echoes the environment, and a .runsettings file with a key in it is as dangerous as a source file if you commit it. The protection comes from keeping the value out of source control and out of shared artifacts, not from the mechanism.

Note

QAAS_HASH_SECRET is secret for a different reason. It grants no access. It is the key used to hash your test names so they never leave your machine, and publishing it lets someone match those hashes back to real test names, which defeats the default privacy mode.

Start here: why a variable you already set is not visible

Almost every “I set it and nothing happened” report has the same cause. Read this before trying the recipes below.

A process receives a copy of the environment when it starts, and never sees later changes. Your IDE launches the test host as a child process, so the test host inherits the environment your IDE had when the IDE launched. If you set a variable after that point – in a terminal, or in the system settings dialog – the running IDE never learns about it, and neither does the test host it spawns.

So if you set a variable at the operating-system level and your tests still do not see it:

1. Quit the IDE completely. Closing the window is not enough on macOS; use Quit, and on Windows make sure the process has actually exited.
2. Start it again and re-run the tests.

Note

On macOS there is a second trap on top of this one. An application launched from the Dock or Finder does not read ~/.zshrc or ~/.bash_profile at all, so an export line in your shell profile is invisible to Rider or Visual Studio no matter how many times you restart them. A shell profile reaches only the programs you start from a terminal. See the macOS section below.

Both traps disappear if you skip the operating system and set the variables in your project or your IDE instead. That is the approach the rest of this article recommends.

Recommended: a .runsettings file

A .runsettings file sets environment variables directly on the test host process. It works in Visual Studio, Rider, VS Code and dotnet test, so it is the one answer that covers every environment in this article. It is per-project, so two games can use different keys, and it needs no IDE restart.

Create a file named qaas.runsettings beside your solution:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration>
    <EnvironmentVariables>
      <QAAS_API_KEY>your-key-here</QAAS_API_KEY>
      <QAAS_PROJECT>your-game-slug</QAAS_PROJECT>
      <QAAS_HASH_SECRET>your-own-stable-secret</QAAS_HASH_SECRET>
    </EnvironmentVariables>
  </RunConfiguration>
</RunSettings>

Caution

This file now contains your API key, so do not commit it. Add qaas.runsettings to .gitignore before you paste the key in. If you want a committed template for the rest of your team, commit a qaas.runsettings.example with empty values and keep the real one local.

Note

Environment variables set this way require the tests to run in a separate process. The test platform does that for you. It is also why this mechanism cannot work with in-process test execution.

Then point your environment at the file.

Visual Studio

Visual Studio needs to be told which run settings file to use:

• Test > Configure Run Settings > Select Solution Wide runsettings File, then pick your file. This selection is stored locally, not in the solution, so each developer does it once on their own machine.
• Or use Test > Configure Run Settings > Auto Detect runsettings Files and name the file .runsettings in the project root. Autodetection is available in Visual Studio 2019 version 16.4 and later.

A solution-wide file selected explicitly overrides an autodetected one.

JetBrains Rider

Rider has its own environment-variable field for the unit test runner, which is simpler than wiring up run settings:

1. Open Settings > Build, Execution, Deployment > Unit Testing > Test Runner.
2. Add each variable as a name and value pair.

These are passed to the process the unit test runner starts and to all of its child processes, so the Reporter sees them.

If you run tests through a run/debug configuration instead, use its Environment variables field. Entries are NAME=value, separated by semicolons if there is more than one. You can reference an inherited variable with $VAR$ syntax, and those references are case-sensitive: Path=xxx;$Path$ on Windows, PATH=xxx:$PATH$ on macOS.

Tip

Rider’s Test Runner settings apply to the whole IDE, not one project. If you work on more than one game, use the .runsettings approach so each suite keeps its own QAAS_PROJECT.

VS Code

VS Code’s C# Dev Kit reads a run settings file if you point it at one. Add this to your workspace settings.json:

{
  "dotnet.unitTests.runSettingsPath": "./qaas.runsettings"
}

Every run started from the Test Explorer then passes that file to the test engine. Reload the window after changing it.

Note

If a value still does not arrive, run the suite once from the integrated terminal with dotnet test --settings qaas.runsettings. If it works there but not in the Test Explorer, your file is correct and the extension is mishandling the path. Use an absolute path.

Command line

dotnet test takes the file directly. This is the quickest way to prove your file is correct:

dotnet test --settings qaas.runsettings

Setting the variables on Windows

Use this when you run tests from a terminal, or when you want one machine-wide setting for every project.

Permanently, through the UI: open the Start menu, search for Edit environment variables for your account, and add each variable under User variables. Then restart any IDE or terminal that needs to see them.

Permanently, from PowerShell:

[Environment]::SetEnvironmentVariable('QAAS_API_KEY', 'your-key-here', 'User')
[Environment]::SetEnvironmentVariable('QAAS_HASH_SECRET', 'your-own-stable-secret', 'User')

For the current terminal only:

$env:QAAS_API_KEY = 'your-key-here'
dotnet test

In cmd.exe the equivalents are setx QAAS_API_KEY "your-key-here" to persist, and set QAAS_API_KEY=your-key-here for the current session only.

Caution

setx and SetEnvironmentVariable affect new processes only. The window you typed the command into will not have the variable, and neither will any application that was already running. This is the most common reason a correct Windows setup appears not to work.

Setting the variables on macOS

For tests run from a terminal, add the exports to ~/.zshrc (or ~/.bash_profile if you use bash), then open a new terminal:

export QAAS_API_KEY="your-key-here"
export QAAS_HASH_SECRET="your-own-stable-secret"

This reaches only programs started from a terminal. For an IDE you launch from the Dock, you have two options:

• Use the .runsettings file or the IDE’s own environment settings. Recommended on macOS.

• Set the variable for the graphical session with launchctl, then restart the IDE:

  launchctl setenv QAAS_API_KEY "your-key-here"

  Applications launched after this command inherit the value. It does not survive a logout or restart, so use it for a one-off check, not a permanent setup.

Confirm the test host can see it

Set the variables, then read them back from inside a test. This reports the environment of the process that runs your tests, not the one your terminal has:

[Test]
public void QaasEnvironmentIsVisible()
{
    var key = Environment.GetEnvironmentVariable("QAAS_API_KEY");
    TestContext.WriteLine($"QAAS_API_KEY: {(string.IsNullOrEmpty(key) ? "NOT SET" : "set, length " + key.Length)}");
}

Print the length, not the key, so you never paste a secret into a log or a screenshot. If this prints NOT SET, the variable has not reached the test host and the Reporter will not upload anything.

The Reporter announces its own status too. Every message it writes is prefixed [QaaS] and goes to the test output:

Message	Meaning
[QaaS] QAAS_API_KEY not set; skipping results upload.	The variable did not reach the test host. Start at the top of this article.
[QaaS] No captured results to upload. Add [assembly: gdio.qaas.Reporter.Capture] to your test project.	The variables are fine. The missing piece is the capture attribute, which must be in your own test assembly.
[QaaS] QAAS_HASH_SECRET not set; using a default key.	Uploads work, but a shared default key is being used for hashing. Set your own before you rely on trends.
[QaaS] Uploaded 12 result(s) to https://ingest.gamedriver.ai/v1/runs (202).	Success, with the count and endpoint it used.
[QaaS] Upload rejected (401); saving payload locally.	The key arrived but was not accepted. See the troubleshooting section below.
[QaaS] Payload saved to '...'	The run was written to disk instead of lost. The message names the file.

Note

These messages are written to the console at the end of the run, not inside an individual test. Some IDE test runners file that output under the run or the fixture instead of a test, so check the whole output pane before concluding the Reporter said nothing.

Value formats that catch people out

The Reporter trims whitespace and treats a blank value as unset, so a stray trailing space in a settings dialog is harmless. Three values are stricter than they look:

• QAAS_SHARE_DETAIL only recognises true. The comparison is case-insensitive, so TRUE and True work, but 1, yes and on do not and leave detail sharing off.
• QAAS_EXECUTION_TYPE only recognises manual. Anything else, including a typo like manaul, is treated as automated without complaint.
• QAAS_FALLBACK_DIR defaults to a path relative to the test host’s working directory, not your project folder. That is often the build output directory, so look in bin/Debug/<framework>/qaas-fallback before concluding nothing was written. Set the variable to an absolute path if you want it somewhere predictable.

Troubleshooting

The test prints NOT SET but the terminal shows the value. The IDE was started before the variable existed, or on macOS the IDE never reads your shell profile. Quit the IDE fully and reopen it, or move the variables into a .runsettings file.

It works from dotnet test but not from the IDE. The same cause. The terminal has the variable and the IDE’s inherited environment does not.

It works for one project and not another. You are probably relying on a Rider Test Runner setting, which is IDE-wide, while the second project needs a different QAAS_PROJECT. Give each project its own .runsettings file.

401 Unauthorized. The variable arrived, so this is the key itself: revoked, mistyped, or never revealed correctly. An API key is shown once, and a truncated copy fails exactly this way. Ask an administrator to confirm it is active or issue a new one.

404 Project not found. QAAS_PROJECT does not match a game in your company, or the key is scoped to a different game. Check the slug on the setup page in the QaaS portal.

The run is green but nothing arrived. The Reporter never fails a test run, by design. Read the [QaaS] lines in the output to find out what it did, and check QAAS_FALLBACK_DIR for a saved payload.

Related articles

• Uploading GameDriver Test Results to QaaS
• Running GameDriver Tests from the Command Line
• CD Pipeline with GameDriver