Unaffiliated
Steamworks Documentation
Uploading to Steam
Introduction to the SteamPipe Content SystemSteamPipe is the game/application content system that powers Steam. SteamPipe includes the following features:
Efficient and fast content delivery.
Unlimited public and private &beta& branches, allowing multiple builds to be tested.
Simple web-based management of builds - push out a new build or rollback to a previous build with just a few clicks.
Ability to see the update size of a build before setting it live.
Ability to share content between multiple applications.
Ability to
from public or beta content.
Games/apps stay available offline even after an update download has been started.
All content is always encrypted, and non-active versions are not visible to customers.
which can be used during development.NOTE: There are a few concepts which are integral to SteamPipe, before getting started you should be familiar with all of the concepts outlined in the
documentation. Having even a basic understanding of how these pieces fit together will be very useful when uploading your product to Steam.Steamworks Video Tutorial - Building Your Game in SteampipeThis tutorial introduces SteamPipe and steps through building a sample application for Steam via the Steamworks tools.Steamworks Video Tutorial - Adding New Platforms and LanguagesThis tutorial walks you through adding new platforms and languages to your game by adding depots to your app.SteamPipe Technical DetailsSteamPipe uses the HTTP protocol for content delivery. Since downloads are regular web traffic, any third-party HTTP cache between the customer and Steam servers will increase download speed. Content can be hosted by external CDN providers, which can be easily added to our content network. Most consumer firewalls allow HTTP traffic and won't block downloads.SteamPipe has an efficient patching algorithm based on binary deltas, only changing the modified portions of existing content files. When this content is updated, only these deltas need be sent. This means both developer and user transfers are smaller and faster. Most partners will find that using a
not necessary since they can efficiently patch builds on private branches.Steam Build AccountBefore you can create any builds on Steam, you must have a Steam account in your Steamworks account with the &Edit App Metadata& and &Publish App Changes To Steam& permissions granted. For security reasons it's recommended that you have a dedicated build account with just those permissions, you may create a new Steam account for this purpose at .Any administrator of your Steamworks account can add a Steam account and grant the necessary permissions. More information on this process can be found in the
documentation. An example of what this account might look like is:Initial Setup for New SteamPipe AppsFollow these steps to set up new SteamPipe apps:
Find the app ID for your application (this can be found by selecting the application on your homepage in Steamworks)
page for your app.
Define at least one launch option (the path and optionally, any arguments required to launch the game). Hover over the (?) to learn more about each field.
The example below shows 4 launch options, 2 for Windows and 1 each for MacOS and Linux. Launch option 3 will only be shown on Windows if the user also owns the DLC specified.
page and add depots as needed for this app. By default, a depot may already be configured for your application.
Click the default depot and change the name of the depot to a proper and recognizable name (such as &Base Content& or &Windows Content&).
Leave the language set to [All language] unless this is a language-specific depot.
Leave the OS set to [All OSes] unless this is an OS-specific depot (if the app is all-in-one, or is just PC or just Mac, it should be left to [All OSes]. Only specify for OS-specific game depots.
Click Add New Depot to create additional depots.
Click Save Changes to save any changes made.
Once you are done defining your depots, publish the changes that you have made from the
Your newly defined depots will need to be included in a package to grant you ownership of them. Each game on Steam should have a Developer Comp package which is automatically granted to the accounts listed within your publisher group.
You can add the new depots to that package (and/or other packages that should have these depots) on the
page.Setting up the SDK for SteamPipe uploadsDownload and unzip the latest version of the Steamworks SDK on the machine you will be uploading builds on.The SteamPipe tools can be found within the SDK in the tools folder which contains 2 relevant sub-directories. The ContentBuilder directory is where your game content and SteamPipe build tools will live. This directory contains the following sub-directories:
builder - This directory initially contains just steamcmd.exe which is the command line version of Steam.
builder_linux - The linux version of steamcmd.
builder_osx - The macOS version of steamcmd.
content - This directory contains all game files that will be built into depots.
output - This directory will be the location for build logs, chunk cache, and intermediate output. NOTE: This folder can be deleted or emptied at any time, but after it's deleted, the next upload time will take longer.
scripts - This directory is where you'll place all of your build scripts for building your game depots.It's recommended that you run steamcmd.exe directly in the builder folder for your platform once to bootstrap your build system. This should populate your builder directory with all the files it needs to build depots.The ContentServer directory contains the tools for running your own
if you choose to do so.SteamCmd on macOSTo enable SteamCmd on macOS you must complete the following steps:
From the terminal, browse to the tools\ContentBuilder\builder_osx\osx32 folder
Run chmod +x steamcmd
Browse up to the parent folder (tools\ContentBuilder\builder_osx)
Type bash ./steamcmd.sh
SteamCmd will then run and update to the latest build, leaving you in the SteamCmd prompt
Type exit and press return to exit the promptYou can then follow the rest of this documentation (substituting paths as appropriate) to create depot and app config files for uploading your content to Steam.Creating SteamPipe Build Config FilesTo upload your app with SteamPipe, you must create scripts which describe your build and each depot that will be included in it. There are simple example scripts in the ContentBuilder\scripts folder for an app with one depot.NOTE: You can name these scripts what ever you want, but we use the names app_build_&AppID& and depot_build_&AppID& for consistency. If you know that you'll be building apps on this machine, it might be a good idea to create sub-directories in your scripts directory for each application, to help organize each application's build scripts.SteamPipe GUI ToolIf you're running on Windows and would prefer a GUI tool to help create these config files and upload your builds you can use the SteamPipeGUI which is available in the tools folder of the Steamworks SDK. Included in the zip are additional instructions to get you started.If you choose to use the GUI tool then reading the following sections is still recommended to help you become more familiar with how the SteamPipe system works.Depot Build ScriptFirst create a depot build script for each depot you are wanting to upload. Start by duplicating the depot_build_1001.vdf script and renaming it with the depot ID that it will be uploading.Example depot build script:&DepotBuildConfig&
// Set the depot ID that this script will upload.
&DepotID& &1001&
// Set a root for all content.
// All relative paths specified below (LocalPath in FileMapping entries, and FileExclusion paths)
// will be resolved relative to this root.
// If you don't define ContentRoot, then it will be assumed to be
// the location of this script file, which probably isn't what you want
&ContentRoot& &D:\MyGame\rel\master\&
// Include all files recursively
&FileMapping&
// This can be a full path, or a path relative to ContentRoot
&LocalPath& &*&
// This is a path relative to the install folder of your game
&DepotPath& &.&
// If LocalPath contains wildcards, setting this means that all
// matching files within subdirectories of LocalPath will also
// be included.
&recursive& &1&
// Exclude all symbol files
// This can be a full path, or a path relative to ContentRoot
&FileExclusion& &*.pdb&
NOTE: Setting a &ContentRoot& in the depot build script will override the &ContentRoot& in the app build script.App Build ScriptAfter setting up your Depot build scripts you will need an App build script which describes the details of the build. Start by duplicating the app_build_1000.vdf script and renaming it with the app ID that it will be building.The following types of builds supported by SteamPipe:
Preview - This type of build only outputs logs and a file manifest. Building preview builds is a good way to iterate on your upload scripts.
Local - Builds for the . LCS builds upload to your own LCS and allow you to test the installation of your game using the Steam client.
SteamPipe - SteamPipe builds result in your depots actually being uploaded to Steam. This is always safe to do, even for games that are already released, as you will need to set it public before customers can download it.Example app build script:&appbuild&
// Set the app ID that this script will upload.
&appid& &1000&
// The description for this build.
// The description is only visible to you in the 'Your Builds' section of the App Admin panel.
// This can be changed at any time after uploading a build on the 'Your Builds' page.
&desc& &Your build description here&
// Enable/Disable whether this a preview build.
// It's highly recommended that you use preview builds while doing the initially setting up SteamPipe to
// ensure that the depot manifest contains the correct files.
&preview& &1&
// File path of the local content server if it's enabled.
&local& &&
// Branch name to automatically set live after successful build, none if empty.
// Note that the 'default' branch can not be set live automatically. That must be done through the App Admin panel.
&setlive& &&
// The following paths can be absolute or relative to location of the script.
// This directory will be the location for build logs, chunk cache, and intermediate output.
// The cache stored within this causes future SteamPipe uploads to complete quicker by using diffing.
&buildoutput& &..\output\&
// The root of the content folder.
&contentroot& &..\content\&
// The list of depots included in this build.
&1001& &depot_build_1001.vdf&
Building DepotsTo kick off a build, edit the run_build.bat file to include your Steamworks build account and password (see ) and path to the app build script:builder\steamcmd.exe +login &account& &password& +run_app_build ..\scripts\[build_script_name].vdf +quit
NOTE: Your first attempt at running a build may fail due to Steam Guard. If the logon fails due to Steam Guard, check your email for the Steam Guard code, and run steamcmd as: steamcmd.exe &set_steam_guard_code &code&&, and try again. After logging in with Steam Guard once, a sentry file is used to verify the login is genuine.After you run run_build.bat, your build output should be in the directory specified in your build script file. If the preview build produces the expected manifest, run the build again with &preview& &0& in your app build script.The following steps occur during a SteamPipe build:
Steamcmd.exe is logging into the Steam backend using the given builder Steam account.
The app build start is registered with the MDS (Master Depot Server), which will ensure the user has the proper privileges to modify this app.
For each depot included in the app build, a file list is generated based on the files in the content folder and the filter rules defined in depot build config file.
Each file is scanned and divided into small chunks of about 1MB. If the depot has been built before, this partitioning will preserve as many of the unchanged chunks as possible.
New file chunks are compressed, encrypted, and then uploaded to the MDS.
A final manifest is generated fo each manifest is identified by a unique 64-bit manifest ID.
Once all depots have been processed, the MDS finishes this app build and assigns it a global buildID.
After the build is done, there may be *.csm and *.csd files in the build ouput folder. These are temporary and can be deleted, but they speed up subsequent build times.Managing UpdatesAfter your app releases to customers, your customers will be receiving the build marked as the Default build. When uploading a new build it's always a good idea to test it before shipping it to your customers, for more information on how to successfully do this see .Advanced File Mapping RulesThere can be multiple file mappings that add files to the depot. They should be listed first.Files or subfolders can be excluded with filter expressions.Steam
can be marked during the build process so the Steam client knows to run them for any application which mounts the Depot.Files or folders can be remapped, so sub-folders in the content folder can be mapped to anywhere in the depot.Files can be tagged with special flags:
userconfig - This file is modified by the user. It cannot be overridden by an update, and it won't trigger a verification error if it's different from the previous version of the file.
readonly - This file will be marked as read-only for the client file system.
hidden - This file will be marked as hidden for the client file system.
executable - This is an executable file (for OS X only).
&DepotBuildConfig&
&DepotID& &202931&
&FileMapping&
// override video files files in \\videos with German versions
&LocalPath& &localization\\german\\videos\\*&
&DepotPath& &videos\\&
&FileMapping&
// override audio files in \\audio with German versions
&LocalPath& &localization\\german\\audio\\*&
&DepotPath& &audio\\&
&FileMapping&
&LocalPath& &localization\\german\\german_installscript.vdf&
&DepotPath& &.&
&FileExclusion& &bin\\server.exe& // exclude this file
&FileExclusion& &*.pdb& // exclude all .PDB files everywhere
&FileExclusion& &maps\\testroom*& // exclude all files under maps/testroom/
&InstallScript& &localization\\german\\german_installscript.vdf&
&FileProperties&
&LocalPath& &bin\\setup.cfg&
&Attributes& &userconfig& // this file will be modified during runtime
Debugging Build IssuesIf your build wasn't successful, you should look in your output directory for error information, not the console where the build script was run. Most error information can be found in the *.log files.You can use these Steam client commands and client-side files to debug issues:
&app_status [appid]& - Shows the current state of the app on this client.
&app_info_print [appid]& - Shows the current Steamworks configuration for this game (depots, launch options, etc.).
&app_config_print [appid]& - Shows the current user configuration for this game (current language, install directory, etc.).
file &logs\content_log.txt& - Lists all logged Oxcart operations and errors.
file &steamapps\appmanifest_[appid].acf& - Shows the current install state of this app (KeyValues).Building Efficient Depots for SteamPipeThe old Steam content system would patch updates on a file level, which meant that if a single byte in a file changed, the entire new file would be downloaded by all users. This was especially inefficient if the game used pack files, which are collections of game content files in a single big file. Pack files can easily exceed 1 GB, so updates often led to unnecessarily large downloads. A common way to avoid these large downloads was to add new pack files that overrode content of already shipped pack files. That worked for updates, but it hurt new users long-term, since they ended up downloading unused, already-patched content.The new content system fixes this problem by splitting each file into roughly 1-MB chunks. Each chunk is then compressed and encrypted before being distributed by the Steam content system. If the game content has large redundant parts, these chunks are reused and the user only has to download each repeated chunk once. However, the real strength of this system is building efficient update patches. While the system is building a patch, the new content is scanned for already known chunks. If it finds them, it reuses them. This means if you change or inject a few bytes in a big file, the user only has to download the changes.This works well in most cases, but there are still a few pitfalls that need to be avoided when designing the content layout of a game. Do not compress or encrypt your game data. This is already done for in-flight downloads and retail discs by the Steam content system. If you do it too, you will reduce the effectiveness of delta patching. If you package multiple data files in a single pack file, make sure that with each re-packaging, no unnecessary changes are made. One problematic practice is including the full name of the original source files on disk, because the names may change, depending on the build machine. Another bad pattern is including build time stamps for each file. If possible, always add new content to the end of your pack files and keep the order of existing files. Also, keep your pack file’s metadata (offset and sizes to individual files) in one place and don’t intersperse it with content files. Use a binary diff’ing tool like BeyondCompare to look at two builds of your pack files to make sure that hundreds of unwanted changes don’t show up.If you follow these rules you will minimize patch sizes and only new content will need to be downloaded. Your customers will thank you for that and you will be able to increase the quality of your product by shipping more updates.Building Retail Install DiscsTo build retail install disc for SteamPipe games, you must first setup a build project file.In this example, the SKU file is called &sku_goldmaster.txt&:&sku&
&name& &Test Game Installer&
&appid& &202930&
&disk_size_mb&
&included_depots&
Some tips to keep in mind:
Create a new folder where the retail disc images will be written to, e.g., &D:\retail_disks&. Only depots in the included_depot there is no exclude section anymore.
You can use Steam.exe (with the -dev and -console command-line parameters) or steamcmd.exe to build installer images. In both cases, use the &build_installer& command.
Log on with a Steam account that owns the game and all depots you want to put on the retail disc. Otherwise, the account doesn't need special rights, so anyone can build installer discs.
If you use Steam.exe, stop all other downloads.
Go to the console page and run the build_installer command:
build_installer sku_goldmaster.txt &D:\retail_disks&
The build can take a while since all depots are re-downloaded the first time.
If you're building a GM using a local content server, run:
@localcontentserver &webserver&
build_installer sku_goldmaster.txt &D:\retail_disks& local
The spew refers to 'Backup' since 'Retail install Disk' and local game backups are basically the same.
Once you see &Backup finished for AppID...&, the install disk images are ready. You can find more details about the backup build in logs\backup_log.txt.
There are new folders (Disk_1, Disk_2, and so on) in &D:\retail_disks&, each not bigger than 640 MB, as specified with &disk_size_mb&. Each disk folder contains a &sku.sis& file and a .csd and .csm for each depot. Bigger depots span across multiple disks. All retail install disk content is always encrypted (unlike local game backup files). Copy the SDK GM setup files (setup.exe, setup.ini, etc.) into the folder of your first disk and the retail disc installer is complete.
When creating a GM for OSX, be sure to open the goldmaster/disk_assets/SteamRetailInstaller.dmg image on a Mac. Then take the app that is in there and copy it to the root of your media. You will probably want to change the name of the install app, brand the icon and decorate the window to only show the installer.
When creating a multi-disc GM for OSX, be sure the volume name for each disc matches. The volume name becomes part of the mount path, and if the names don't match the installer won't be able to find the next disc.Optionally building a retail installer from a beta branchThe process above will create a retail installer based on the default branch. If you need to create an installer based on a beta branch, you must first create a beta branch named &baseline&. Then use the following command to build from the baseline branch:build_installer &project file& &target folder& &beta key& &beta pwd&
steamcmd ex: build_installer sku_goldmaster.txt &D:\retail_disks& baseline superSecret
script ex: steamcmd.exe +login user_name password +build_installer &..\Build\GameDataSku.txt& c:\destination beta_key beta_password +exit
Installing DLC from a retail installerIn some circumstances, you may wish to create a retail installer that includes your DLC packages. In such cases, the process to create the installer requires only a few changes.In &sku_goldmaster.txt&, include the DLC AppIDs under the &included_depots& section. Once you have run the &build_installer& process, find the generated sku.sis file for the installer and open it with a text editor.Add the DLC AppID in the &apps& section. For example, if I had a game with AppID 1000 and DLC AppID 1010, I would adjust the &apps& section as follows: &apps&
&0& &1000&
&1& &1010&
This will ensure that Steam checks for ownership of the DLC and prompt the user for a key if the DLC is not owned by the account that they are logging into on Steam.Building a retail installer for multiple App IDs on a single disc/install packageTo build a GM containing multiple Steam Pipe apps. Build each app installer one by one but point them all to the same output folder. Each build will merge itself with the already existing install image.Customizing a Retail install diskSee
for more details on customizing your retail install disk.Preloading Games before ReleaseBy default, all content is always encrypted, on all retail discs and on all content servers. Switching a game to preload mode means owners can download the content, but it stays encrypted on the users' disk and can't be played. Once the game becomes officially released, Steam will decrypt the preloaded content and the user can play the game.Switching a game to preload mode is recommended in these cases:
Shipping retail discs with product keys before the game is actually available (0-day piracy).
Games that run a pre-purchase and are larger than 20GBs in size.Please submit a ticket to Steam Publishing if you believe your game requires preloading.Building DLCDLC is built as a depot of the base game. See the
documentation for more information.Troubleshooting SteamPipe&Login Failure: Account Logon Denied Failed& when logging in via steamcmdCause: Probably SteamGuard is preventing login. Resolution:
Check the email associated with the account you are trying to log on with and look for an email from Steam Support. Copy the code from that email.
Run the following steamcmd: set_steam_guard_code &code&
Re-Attempt logon from steamcmd: Steam&logon &buildaccount& &password&General Troubleshooting for Download Issues
Restart computer, modem, router, etc.
Verify firewall settings. The new system requires port 80 (HTTP) and all other Steam ports, listed .
Temporarily disable local Anti-Virus or Spam-Blocker programs.
Check the Steam download region under Settings-&Downloads & Clouds. It should match your location.
Stop the download, uninstall, and reinstall the game (clear manifest caches).
Exit Steam, delete the two folders appcache and depotcache in your Steam install folder.
Try to set your Steam download region to some other location far away. This might work if a content server near you is serving bad data.My Mac and/or Linux Builds aren't installing any files. Why?If you're testing via Steam the installation of your game or application across multiple platforms, you may run into a situation where the build deploys on Windows but doesn't deploy any files on Mac or Linux despite your SteamPipe process being setup to upload Mac and/or Linux depots. There is a step that is easily missed which involves adding your alternate Depots to the Package being deployed. You can check what depots are included in a package via the following steps:
Navigate to your App Admin page
From the View Associated Items section, click View Demo, all associated packages, videos, and DLC
Click on the title of the Package you're attempting to download
Review the Depots Included section
Use the Add/Remove Depots to ensure the correct set of Depots are assigned to the PackageThere are a number of discussion threads about this that may also assist:
Running steamcmd.exe results in the following error: &SteamUpdater: Error: Steam needs to be online to update. Please confirm your network connection and try again.&Resolution: Go to Internet Options-&Connections-&Lan Settings and check Automatically detect settings.Running the app build results in the following error: &ERROR! Failed 'DepotBuild for scriptname.vdf' - status = 6.&Possible Causes:
Account does not have permissions for the app.
Check that the app ID is correct in the app_build.vdf.
Check that the build account has
to the app ID.
Steamcmd cannot find the depot contents.
Check that the &contentroot& value in the app_build script is a valid path relative to the location of the script file.
Check that the &LocalPath& value in the depot_build script is a valid path relative to the path in the app_build script. Check that the path contains actual content.
Running the app build results in the following error: &ERROR! Failed to get application info for app NNNNN (check login and subscription)&This means that Steam can't retrieve information about the app, either because it doesn't exist or the user doesn't have access to the app.
Check that the NNNNN is the app ID you were assigned for the app.
Check that the app ID is correct in the app_build.vdf.
If it is a new app ID, check that the Steamworks app admin configuration has been published. New apps should have a Steam Pipe install directory on the config tab, a depot on the depot tab, and all changes published on the publish tab.
If all of that looks right, make sure that your account owns the app ID.&An error occurred while installing [AppName] (Invalid content configuration)& at launch timePossible Causes:
No build has been set live on the branch you're trying to install from.
Resolution: Set your build live on a branch by going to , and select that branch in the Steam client (as described ).
Invalid game launch options.
Resolution: Verify the launch options on the Configuration tab in the app admin for your game .
You don't own the Depot IDs that make up the game.
Resolution: Make sure the necessary depots are added to the development subscription (See
for additional details).&Error code 15& at launch timeThis is the CEG servers rejecting the request to CEG the exe. Check the release state on the game page. If it is not 'playable', you will need to request Steam keys that override the release state.&The Steam Servers are too busy to handle your request... Error Code (2)& at launch timeThis is the CEG servers failing to find a matching CEG exe. Double check that you have a working CEG executable on the CEG tab of your app admin. Click the 'status' button to make sure that it's propped.I can't remember what that steamcmd command was or how it workedUse the 'find' command in steamcmd to search for any steamcmd command. It will do partial matching on the command name and it will list out the command syntax.Steam&find build_installer
build_installer : &project file& &target folder& &beta key& &beta pwd&}