Upsight logo Back to top

GDPR

Note: This documentation is preliminary and subject to change.

Introduction

Introduction


To ensure Upsight and it's customers are able to comply with GDPR regulations, Upsight is introducing a number of new functions in its products, as well as reorganizing the way personal data is handled, processed and stored.

As an Upsight customer and app publisher, if your application is distributed in Europe, you will need to make sure to take the appropriate steps, which are outlined in this document.

  • Section one provides a White Paper explanation of Upsight's approach to separating personal and non-personal data, ensuring ability to be responsive to consumer rights to access, and rights to erasure.
  • Section two explains the steps Upsight customers need to take on the Upsight dashboard to ensure data is categorized appropriately.
  • Section three documents a new set of APIs that enable an Upsight customer to respond to Rights for Access and Rights to Erasure requests.

Note: as we are actively working with our customers on GDPR compliance, these documents will be updated and refined to improve clarity and accuracy, and as such, may change without notice. If you have any questions or concerns, please reach out to your Customer Success Manager.

Upsight GDPR Personal Information Management

Overview


Upsight will enable Publishers to articulate the types of information they collect from consumers through web and mobile apps. Upsight will treat and store this data using different techniques, allowing for an expedient retrieval and subsequent deletion of Personal Data upon the Publisher’s request, and allow blacklisting of consumers, to prevent future ingestion of Personal Data on consumers who have exercised their right to restrict processing.

Upsight GDPR Personal Information Management

Device Data vs Event Data


Every event instrumented using the Upsight SDK will send data to Upsight’s APIs, for real-time decision making (marketing tools), for data processing (analytics tools), or both. Each event contains a data set, coming from two primary sources: event-specific data, and device data. For both event and device data, there is standard data that Upsight collects automatically, and there is data that the app developer provides.

UpsightPublisher
Devicecountry, language, device modellifetime experience points earned
Personal data allowed if flagged
Eventcampaign IDlevel reached
Personal data is disallowed

In the case of data managed by Upsight, any Personal Data is limited to device identifiers (if used e.g. IDFV) and high-resolution location information (if used). In the case of data managed by the Publisher, personal data is explicitly prohibited in the event data (custom event attributes). In contrast, it is permitted to store personal information in the device data (custom user attributes), as long as this is indicated as such through Upsight’s dashboard.

Upsight GDPR Personal Information Management

Personal vs Non-Personal Data


PersonalNon-Personal
IdentifierIDFV, IDFA, Android ID, Google AID, Push TokenSID, Custom User ID
DataDOB, Gender, Name, Email, etcCountry, Language, Experience Points, etc.

Personal Data

Personal Data is data associated with a natural person. This includes information such as date of birth, gender, name, email address.

Non-Personal Data

Non-Personal Data is data associated with a consumer’s interaction with an application, that has no bearing on a natural person. General examples include “time spent in app”, “dollars spent”, as well also “country”, “device language”. Examples for a game might include “level reached”, “experience points earned”.

Personal Identifiers

Personal identifiers are those generated by the operating system. They include IDFV, IDFA, Android ID and Google Advertising Identifier. These identifiers will be collected by the Upsight SDK as directed by the Publisher (e.g. IDFV is enabled by default, whereas IDFA is disabled by default, but both can be toggled “on” or “off”). As these identifiers are also accessible to other SDKs, and potentially other apps by the same or different publishers, these are considered Personally Identifiable Information.

Note While “name” and “email” are identifiers in the broadest sense, they are not inherently treated as identifiers in Upsight’s architecture, meaning they cannot be used as a key to retrieve information about a consumer. Both Identifiers and Data that is Personal is treated the same way by Upsight when it comes to Privacy, Security and Regulatory Compliance. The distinction made here is only material with respect to the way in which Upsight allows devices and users to be identified by the Publisher.

Non-Personal Identifiers

These identifiers are software generated and do not have particular meaning outside one of these two scopes: the Upsight internal system and the Upsight-Publisher relationship (e.g. Player ID).

The Upsight SID is a software-generated random number that is used to identify a single app-device combination. As such, an SID by itself cannot be used to recognize a device outside of the app in which it was created, nor can it be recognized by other vendors the Publisher works with. Publishers have the ability to read Upsight’s SID and need to be aware that exporting Upsight’s SID outside of the Upsight ecosystem comes with risk with respect to GDPR.

Custom User ID is part of Upsight’s forthcoming feature to support User-Level Marketing. Publishers identify a consumer through certain means and provide Upsight with a one-way hashed version of the identifier. As long as this identifier can be generated consistently across a consumer’s multiple devices, Upsight can treat these devices as belonging to one and the same consumer.

Note Some Publishers are currently sending an ID similar to Custom User IDs (referred to as “Player ID”) as a cross-device consumer identification mechanism, for example as a custom attribute. Such a Player ID will need to be marked as Personal Identifier, unless it was hashed before sending.

Push tokens are identifiers used by organizations such as Apple and Google to deliver notifications to devices

Upsight GDPR Personal Information Management

Data Storage


Upsight has several storage mechanisms for consumer-level data. By storing the four different categories of data in different ways, Upsight can make it easy to retrieve, and expunge consumer personal data and PII, while retaining non-personal data in a pseudonymous manner, enabling Publishers to continue analyzing data without risk of violating a consumer’s wish for privacy.

Event Tables

The Event Tables stores a record of every single event received from the SDK or API for all devices, for all time. Once Upsight is GDPR compliant, this table will only store Non-Personal data.

PersonalNon-Personal
IdentifierIDFV, IDFA, Android ID, Google AID, Push TokenSID, Custom User ID
DataDOB, Gender, Name, Email, etcCountry, Language, Experience Points, etc.

Device Table

The Device Table is used to store Personal Identifiers, Personal Data, Non-Personal Identifiers and certain Non-Personal Data (specifically: those that used for targeting and segmentation). Data in the Device Table only stores the most recent value for each type, so if a value changes, the previous value will be lost. This means that time-series cannot be created on the Device Table alone. For example, it is possible to query the current distribution of consumers by country from the Device Table, but it is not possible to see the change in that distribution over time, for that we need the Event Tables. For Personal Data, most recent latitude and longitude can be used to build a detailed picture of consumer locations within a city, but as Personal Data does not exist in the Event Tables, it is not possible to chart the change in distribution over time.

PersonalNon-Personal
IdentifierIDFV, IDFA, Android ID, Google AID, Push TokenSID, Custom User ID
DataDOB, Gender, Name, Email, etcCountry, Language, Experience Points, etc.

Querying Event Tables and Device Table

The Event Tables are typically used to perform queries where a user’s historical information is relevant, whereas the Device Table provides a most recent snapshot of certain device or user’s attributes, which is suitable for targeting and segmentation. Certain types of queries may require data from both tables, in which case joining of these tables is possible. Additionally, when using the results of such queries to take action (e.g. send push notifications), Upsight’s systems may use the Device Table to retrieve personal data (e.g. push notification tokens), using Upsight’s SID identifier as the common key.

Note As we’ll see shortly, once a consumer has exercised their right to be forgotten, their non-personal data can still be used in time-series queries, but there will not be any way to associate the result for that particular consumer back to their Personal Identifiers or Personal Data. As a result, we are able to accommodate the consumer’s request, and still be able to do statistical analysis on the now anonymous/pseudonymous data.

Note Non-Personal Identifiers and Non-Personal Data may be stored in both the Event Tables and the Device Table, to allow for efficient querying of either of these tables, without requiring joining the tables.

Upsight GDPR Personal Information Management

Consumer Data Retrieval


Under GDPR, EU residents have the right to obtain personal data held by the Publisher and its Data Processor. Upsight will provide the ability to retrieve Personal data through an API, accessible by the Publisher, by providing one or more identifiers (Personal or Pseudonymous). The data will be retrieved to the Publisher in a machine-readable format, which can be aggregated with data held by other Data Processors and the Publisher itself into a Consumer-facing response.

PersonalNon-Personal
IdentifierIDFV, IDFA, Android ID, Google AID, Push TokenSID, Custom User ID
DataDOB, Gender, Name, Email, etcCountry, Language, Experience Points, etc.

Consumer Data Deletion

When an EU resident informs the Publisher that they wish to exercise their right to be forgotten, the Publisher will communicate this request to Upsight by providing one or more identifiers (Personal or Pseudonymous). Upsight will use this information to expunge from the Device Table (and User Table) all Personal Identifiers and Personal Data.

PersonalNon-Personal
IdentifierIDFV, IDFA, Android ID, Google AID, Push TokenSID, Custom User ID
DataDOB, Gender, Name, Email, etcCountry, Language, Experience Points, etc.

The remaining data is non-personal, and is pseudonymized under Upsight’s SID, which is meaningless outside of the Upsight-Publisher relationship.

Note: depending on how the Publisher implements the Right to Erasure (Right to be Forgotten), the Upsight SDK may continue to send data, including Personal Identifiers and Personal Data, to Upsight’s servers. To prevent re-processing this data, Upsight implements a blacklist mechanism to reject Personal Data from such blacklisted devices. Blacklisted Identifiers Once a Publisher issues an API request to Upsight on behalf a consumer exercising their Right to Erasure, Upsight will store a one-way hashed version of the device identifiers in a blacklist. All API requests received will be checked against this blacklist, to ensure Personal Identifiers and Personal Data is not accidentally re-ingested into Upsight’s data platform. A publisher will be able to remove a device (user) from the blacklist through a subsequent API call, which includes one or more of the blacklisted identifiers.

Note When removing identifiers from the blacklist, the Publisher must send the entire list of identifiers previously blacklisted. Upsight will not retain an association between the identifiers previously blacklisted, and thus cannot remove any identifiers not explicitly provided.

Note Only identifiers explicitly sent to Upsight will be stored in the blacklist. Only a single match on the blacklist is required for the entire request to be treated as blacklisted. This allows a Publisher of an app that supports multiple user profiles to blacklist a single user ID, without blacklisting the entire device.

PersonalNon-Personal
IdentifierIDFV, IDFA, Android ID, Google AID, Push TokenSID, Custom User ID
DataDOB, Gender, Name, Email, etcCountry, Language, Experience Points, etc.

Dashboard Changes

Overview


To facilitate our customers’ compliance with the EU GDPR, we are making a number of changes to our product which affect how data is stored, retrieved and deleted. With the introduction of the GDPR, special care needs to be taken regarding Personal Data. For personal data that Upsight collects automatically see our section on Data Security and Anonymization Methods for information on how we will be handling it. For Personal Data that you collect, your application should be configured using the new controls described below so that Upsight knows to treat certain custom fields as personal.

Dashboard Changes

Custom User and Device Attributes


Custom User/Device Attributes allow you to set a property about a user/device once, which is then sent on all events. These attributes are used for both Analytics, by reporting on events based on user’s current attribute state, and for Marketing, by targeting content based on user’s attribute state.

Currently, Attributes only need to be defined in the Upsight Dashboard if they are intended to be used for Marketing purposes (e.g. targeting or segmentation), and all attributes are available within DataMine in the user_attributes column regardless of whether they have been defined in the dashboard.

Sometimes attributes capture user state, such as a Coin Balance, or Player Level, but in other instances they could capture personal information such as Name and Date of Birth, or a personal identifier such as E-mail Address or Facebook ID. To ensure compliance with GDPR, attributes that are personal, or personal identifiers need to be stored in a special manner. To facilitate this, within Custom Attribute configuration, you may now indicate whether an attribute will contain personal information. All attributes, even those not used for Marketing, must now be defined in the Upsight Dashboard to ensure that data is stored in the correct location. Any attribute not defined in the dashboard will be removed from events at the time they are ingested, and will not be available for either Analytics or Marketing usage.

Attributes marked as Personal will be stored in the Device Table so that their latest value may be accessed for Analytics, and at the same time, will be removed from events as they are ingested, so that the personal information is stored in one central location. Non-personal attributes will continue to be ingested as before, and will be available on individual events.

In addition to marking which attributes contain personal information, attributes must be marked as to whether or not they should be available for use in Marketing in addition to Analytics. Marking attributes as used for Marketing means that they will show up in places such as User Explorer and when defining segments, while attributes used only for Analytics will not show up unnecessarily in those places. All existing attributes will be automatically made available for Marketing usage.

To configure these new options on Custom Attributes, navigate to Application Settings, select the Application you wish to configure, and go to the Custom Attributes section in the right-hand menu. Next, when creating a new attribute, or editing an existing one, you will find the new options as well as explanations of them.

GDPR Custom Attributes

Dashboard Changes

Custom Event Attributes


Custom events can carry custom attributes as set through the Upsight SDKs or via API, and are available within the pub_data column in DataMine for use in queries and metrics. These attributes are useful for analyzing behavior over time, however given their changing nature and storage across events and time, should not be used for storing any Personal Data. Personal Data should only be stored as User/Device attributes as described above. In the event that any of your current applications send Personal Data in event attributes, or you inadvertently send data in the future, you may instruct Upsight to remove the data at the time of ingestion. This will ensure that on a go-forward basis Upsight never stores this data anywhere within our system.

To configure Custom Event Attributes that should not be ingested, navigate to Application Settings, then select the Application you wish you configure. Next, navigate to the Privacy setting area in the right-hand menu. Finally, select the specific event, or all events, and enter the name of the attribute key which should be removed from events.

GDPR Privacy

Data Controller API

Authentication


An API key is required to access the Data Controller API. You can create an API key in the Upsight Dashboard. The API key should be specified in the Authorization header on all requests. The authorization header should have the form:

Authorization: Upsight api_key=YOURAPIKEYHERE

You can send this on a cURL request with -H 'Authorization: Upsight api_key=YOURAPIKEYHERE'.

If the API key is missing or is invalid, the API will respond with an HTTP 403 Forbidden error.

Data Controller API

Responses


Responses in general will have the following form:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "errobj": null,
  "error": null,
  "response": ...
}

The response property will contain the result of the request. Its type varies, depending on the endpoint, but will generally be an object or an array of objects. The errobj and error properties will always be null for 1xx and 2xx response status codes.

Errors

Errors returned by the API take the following form:

HTTP/1.1 403 Forbidden
Content-Type: application/json

{
    "errobj": null,
    "error": "403 Forbidden",
    "response": "API key is invalid"
}
Error Response Properties
errobj
object or null
An optional object with additional information about the error.
error
string
The HTTP response code text.
response
string
A description of the cause of the error.

Pagination

In the case of a paginated endpoint, the response will only contain a single page of data and you'll need to make additional requests to retrieve all the data. There will be an additional key at the top level of the response for the pagination data:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "errobj": null,
  "error": null,
  "page": {
    "offset": 0,
    "limit": 10,
    "total": 5,
    "first": "/example/path?limit=10&offset=0",
    "next": "/example/path?limit=10&offset=10",
    "prev": null
  },
  "response": [
    ...
  ]
}
Page Object Properties
offset
integer
The number of items the page was offset into the result set.
limit
integer
The number of items the page was limited to.
total
integer
The total number of pages in the result set.
first
string
The URL to request the first page in the result set.
next
string or null
The URL to request the next page in the result set, or null if there isn't a next page after the current page.
prev
string or null
The URL to request the previous page in the result set, or null if there isn't a previous page before the current page.

Data Controller API

Fetch Jobs


A fetch job allows you to retrieve all the personal information associated with an identifier in your data. The identifier can be either a device identifier (such as a device’s IDFV or IDFA value), the device’s SID, a custom user ID value, or a custom user attribute value (which has been marked as an identifier).

The results of a fetch job will expire and be removed after 30 days. They will also be removed if a delete job is submitted for the same identifier.

Fetch Job Properties
fetch_job_id
integer
A unique ID for the fetch job.
app_token
string
The token of the application this fetch job is associated with.
identifier_type
string
The type of identifier that will be used to look up the data. One of aid, android_id, custom_user_id, idfa, idfv, sid, or user_attribute.
user_attribute_key
string or null
When identifier_type is user_attribute, this property specifies which user attribute key should be used to look up the data.
identifier
string
The identifier value that should be used to look up the data. If identifier_type is user_attribute, this represents the value associated with user_identifier_key.
status
string
The status of the job. One of processing, processed, errored.
data
array or null

If the status of the job is processed, this property will contain the results. The results will be an array of objects, where each object contains information associated with the identifier. An example data property might look like:

[
  {
    "ids.android_id":"b4a92a6a0a69249f",
    "sid":"9870890484757108510"
  }
]
error
string or null
If the status of the job is errored, this property will contain more details about the error.
callback_url
string or null
If specified, an HTTP POST request will be sent to this URL when the job finishes (that is, when its status is processed or errored). The POST body will be a JSON object that with a single url key, which contains the URL for the completed job.

Create a fetch job

This endpoint allows you to create a new fetch job:

POST https://api.upsight.com/client/v1/organization/(organization_id: integer)/datacontroller/fetch/

Once the job is created, Upsight will begin processing it. Once the job has completed, its state will transition to processed and the data field will be populated with the results. In the event that there is a problem processing the job, the state will transition to errored instead, and the error field will contain more details about the problem.

Request arguments should be sent in a JSON POST body, and can include the following:

Arguments
app_token
string
Required. The token of the application this fetch job is associated with.
identifier_type
string
Required. The type of identifier that will be used to look up the data. Must be one of aid, android_id, custom_user_id, idfa, idfv, sid, or user_attribute.
user_attribute_key
string
Only required when identifier_type is user_attribute. This property specifies which user attribute key should be used to look up the data. Note: only user attributes which have been marked as "personal identifiers" are valid here.
identifier
string
Required. The identifier value that should be used to look up the data. If identifier_type is user_attribute, this represents the value associated with user_identifier_key.
callback_url
string
Optional. If specified, an HTTP POST request will be sent to this URL when the job finishes (that is, when its status is processed or errored). The POST body will be a JSON object that with a single url key, which contains the URL for the completed job.

Retrieve a fetch job

This endpoint allows you to retrieve the details for an existing fetch job:

GET https://api.upsight.com/client/v1/organization/(organization_id: integer)/datacontroller/fetch/(fetch_job_id: integer)/

This returns a fetch job object.

Retrieve a list of fetch jobs

This endpoint allows you to retrieve the details for all existing fetch jobs for your organization:

GET https://api.upsight.com/client/v1/organization/(organization_id: integer)/datacontroller/fetch/

This returns an array of fetch job objects. Note that the results for this endpoint are paginated.

Request arguments should be sent in the query string, and can include the following:

Arguments
limit
integer
Optional. The number of items to limit the page to.
offset
integer
Optional. The number of items into the result to offset for the page.

Data Controller API

Delete Jobs


A delete job allows you to request that the personal information associated with a given identifier be deleted from your data. The identifier can be either a device identifier (such as a device’s IDFV or IDFA value), the device’s SID, a custom user ID value, or a custom user attribute value (which has been marked as an identifier).

Delete job objects are similar to fetch jobs in function, but only include a subset of the properties of a fetch job:

Delete Job Properties
delete_job_id
integer
A unique ID for the delete job.
status
string
The status of the job. One of processing, processed, errored.
error
string or null
If the status of the job is errored, this property will contain more details about the error.

Note: Delete job objects do not provide the details of what identifier was issued for deletion. If you wish to retain that information for future reference, you'll need to retain it in your own database and associate it with the delete_job_id for the job in question.

Create a delete job

This endpoint allows you to create a new delete job:

POST https://api.upsight.com/client/v1/organization/(organization_id: integer)/datacontroller/delete/

Once the job is created, Upsight will begin processing it. Once the job has completed, its state will transition to processed. In the event that there is a problem processing the job, the state will transition to errored instead, and the error field will contain more details about the problem.

Note: Creating a delete job for an identifier will automatically add that identifier to a blacklist, to prevent any further data associated with the identifier from entering the system in the future. If you need to re-allow that identifier, you'll need to remove it from the blacklist.

Request arguments should be sent in a JSON POST body, and can include the following:

Arguments
app_token
string
Required. The token of the application this fetch job is associated with.
identifier_type
string
Required. The type of identifier that will be used to look up the data. Must be one of aid, android_id, custom_user_id, idfa, idfv, sid, or user_attribute.
user_attribute_key
string
Only required when identifier_type is user_attribute. This property specifies which user attribute key should be used to look up the data.
identifier
string
Required. The identifier value that should be used to look up the data. If identifier_type is user_attribute, this represents the value associated with user_identifier_key. Note: only user attributes which have been marked as "personal identifiers" are valid here.
callback_url
string
Optional. If specified, an HTTP POST request will be sent to this URL when the job finishes (that is, when its status is processed or errored). The POST body will be a JSON object with a single url key, which contains the URL for the completed job.

Retrieve a delete job

This endpoint allows you to retrieve the details for an existing fetch job:

GET https://api.upsight.com/client/v1/organization/(organization_id: integer)/datacontroller/delete/(delete_job_id: integer)/

This returns a delete job object.

Retrieve a list of delete jobs

This endpoint allows you to retrieve the details for all existing fetch jobs for your organization:

GET https://api.upsight.com/client/v1/organization/(organization_id: integer)/datacontroller/delete/

This returns an array of delete job objects. Note that the results for this endpoint are paginated.

Request arguments should be sent in the query string, and can include the following:

Arguments
limit
integer
Optional. The number of items to limit the page to.
offset
integer
Optional. The number of items into the result to offset for the page.

Data Controller API

Blacklist


Identifiers are automatically added to a blacklist when a delete job is submitted for the identifier, to prevent any further data associated with the identifier from entering the system in the future.

Remove an identifier from the blacklist

This API can be used if you later determine that you would like to remove an identifier from the blacklist, to allow data associated with the identifier to enter the system again:

DELETE https://api.upsight.com/client/v1/organization/(organization_id: integer)/datacontroller/blacklist/

This responds with an HTTP 202 Accepted upon a successful request. A response status of 202 should not be taken as an indication that the identifier was blacklisted. Any requested deletion will ensure that the identifier has been removed from the blacklist, whether it was there prior to the request or not. Please note that it may take some time before the removal from the blacklist takes effect throughout the system.

Request arguments should be sent in the query string, and can include the following:

Arguments
app_token
string
Required. The token of the application the request is for.
identifier_type
string
Required. The type of identifier that will be specified. Must be one of aid, android_id, custom_user_id, idfa, idfv, sid, or user_attribute.
user_attribute_key
string
Only required when identifier_type is user_attribute. This property specifies which user attribute key should be used.
identifier
string
Required. The identifier value that should be removed from the application's blacklist. If identifier_type is user_attribute, this represents the value associated with user_identifier_key.