{"_id":"59e70b9ee027ae002e7d2ee3","category":{"_id":"59e70b9ee027ae002e7d2ed8","version":"59e70b9ee027ae002e7d2ed2","project":"5496d393f52a630b00519cdd","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2014-12-26T11:41:35.509Z","from_sync":false,"order":5,"slug":"batch","title":"Batch"},"project":"5496d393f52a630b00519cdd","parentDoc":null,"user":"5496d353f52a630b00519cdc","version":{"_id":"59e70b9ee027ae002e7d2ed2","project":"5496d393f52a630b00519cdd","__v":2,"createdAt":"2017-10-18T08:06:54.462Z","releaseDate":"2017-10-18T08:06:54.462Z","categories":["59e70b9ee027ae002e7d2ed3","59e70b9ee027ae002e7d2ed4","59e70b9ee027ae002e7d2ed5","59e70b9ee027ae002e7d2ed6","59e70b9ee027ae002e7d2ed7","59e70b9ee027ae002e7d2ed8","59e70b9ee027ae002e7d2ed9","59e70b9ee027ae002e7d2eda","59e70b9ee027ae002e7d2edb","59e70b9ee027ae002e7d2edc","59e70b9ee027ae002e7d2edd","59e70b9ee027ae002e7d2ede","59e70b9ee027ae002e7d2edf","5b8661ccdd19310003a3fa0b"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":true,"codename":"","version_clean":"2.0.10","version":"2.0.10"},"githubsync":"","__v":0,"updates":[],"next":{"pages":[],"description":""},"createdAt":"2014-12-26T12:28:50.300Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":0,"body":"##When to use this process?\nUse this process if you receive or collect mobile handset signals (location, Wi-Fi info and app usage) without the Placer SDK for [iOS](doc:ios) or [Android](doc:android).\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"Before you start\",\n  \"body\": \"To implement the described process you will need a GCS Destination URL and Project ID from Placer. To get them, please contact us at info:::at:::placer.io with the google accounts you will use to access GCS (at least one Gmail account).It is recommended using a [Service Account](https://developers.google.com/accounts/docs/OAuth2ServiceAccount#creatinganaccount) for this purpose.\"\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"STEP 1 - Preparing the signals file\"\n}\n[/block]\nThe signals file is a gzip compressed text file. The gzip and text file names should be the same, and equal to your export creation time in ISO 8601 standard (e.g.  2014-08-30T203245Z.gz and 2014-08-30T203245Z.txt).\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"Notice\",\n  \"body\": \"The import file format is a [JSON Lines](http://jsonlines.org/) file. This means that the file itself is not a JSON array but rather a JSON per line.\"\n}\n[/block]\nData points should be in this JSON format:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Field\",\n    \"h-1\": \"Description\",\n    \"h-2\": \"Type\",\n    \"h-3\": \"Example\",\n    \"0-0\": \"user_id\\n\\n*(mandatory)*\",\n    \"1-0\": \"os\\n\\n*(optional)*\",\n    \"7-0\": \"data\",\n    \"6-0\": \"yob\\n\\n*(optional)*\",\n    \"5-0\": \"gender\\n\\n*(optional)*\",\n    \"3-0\": \"device_model\\n\\n*(optional)*\",\n    \"2-0\": \"os_version\\n\\n*(optional)*\",\n    \"0-1\": \"A unique User ID. All insights will be associated with this ID.\\nA good example for User ID is the Advertising ID/IDFA generated by the device OS.\",\n    \"6-1\": \"Year of birth as a 4-digit integer\",\n    \"5-1\": \"Gender: “M” male, “F” female, “O” Other.\",\n    \"2-1\": \"OS version.\",\n    \"3-1\": \"The device model.\",\n    \"1-1\": \"The device operating system: Android or iOS.\",\n    \"0-2\": \"String\",\n    \"1-2\": \"String\",\n    \"2-2\": \"String\",\n    \"3-2\": \"String\",\n    \"5-2\": \"String\",\n    \"6-2\": \"int\",\n    \"7-2\": \"JSON Array of [Signals Structure](#section-signals-structure)\",\n    \"0-3\": \"56c18475ff021108d127cda8\",\n    \"6-3\": \"1984\",\n    \"5-3\": \"M\",\n    \"3-3\": \"LG-D802\",\n    \"2-3\": \"4.4.2\",\n    \"1-3\": \"Android\",\n    \"7-1\": \"The data attribute contains an array of [Signals Structure](#signals-structure). \\nEach element within the array, may contain several signals and formats (location/wi-fi/apps) with different timestamps.\",\n    \"8-0\": \"custom_data\",\n    \"8-1\": \"Developers can define custom data and attributes for each device. This data will be returned in the [Device Details Structure](doc:device-details).\",\n    \"8-2\": \"JSON\",\n    \"8-3\": \"{“number_of_family_members”:4}\",\n    \"4-0\": \"device_brand\\n*(optional)*\",\n    \"4-1\": \"The device brand.\",\n    \"4-2\": \"String\",\n    \"4-3\": \"LGE\"\n  },\n  \"cols\": 4,\n  \"rows\": 9\n}\n[/block]\n\n##Signals Structure\nSignal Structure should always include the timestamp and one or more of the following signals fields: location, active_wifi or wifi_scan. Timestamp should match the exact local-time in which the signals were collected from the device. If signals were collected at multiple times, they should be separated to different signal structures with different timestamps. \n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Field\",\n    \"h-1\": \"Description\",\n    \"h-2\": \"Type\",\n    \"h-3\": \"Example\",\n    \"0-0\": \"timestamp\\n\\n*(mandatory)*\",\n    \"4-0\": \"installed_apps\\n\\n*(optional)*\",\n    \"3-0\": \"running_apps\\n\\n*(optional)*\",\n    \"2-0\": \"wifi\\n\\n*(optional)*\",\n    \"1-0\": \"location\\n\\n*(optional)*\",\n    \"0-1\": \"The time and date of the signal’s data collection. If available, the user time zone should be included.\",\n    \"1-1\": \"Contains location data.\",\n    \"2-1\": \"Contains a JSON array of wi-fi networks data. Can be the result of network scan in android or a single network being connected.\",\n    \"3-1\": \"Contains a list with the bundle IDs or package names of the apps that are currently used or active on the device.\",\n    \"4-1\": \"Contains a list with strings of the applications that were detected on the device.\",\n    \"0-2\": \"ISO-8601\",\n    \"0-3\": \"2013-11-02T08:27:30.189+0200\",\n    \"1-2\": \"[Location \\nsignal structure](#section-location-signal-structure)\",\n    \"2-2\": \"JSON array of [Wi-Fi signal structure](#section-wi-fi-signal-structure)\",\n    \"3-2\": \"JSON Array of String\",\n    \"4-2\": \"JSON Array of String\",\n    \"3-3\": \"[“com.whatsapp”,”com.facebook.katana”]\",\n    \"4-3\": \"[“com.whatsapp”,”com.facebook.katana”]\"\n  },\n  \"cols\": 4,\n  \"rows\": 5\n}\n[/block]\n##Location signal structure\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Field\",\n    \"h-1\": \"Description\",\n    \"h-2\": \"Type\",\n    \"h-3\": \"Example\",\n    \"0-0\": \"lat\\n\\n*(mandatory)*\",\n    \"1-0\": \"long\\n\\n*(mandatory)*\",\n    \"2-0\": \"accuracy\\n\\n*(optional)*\",\n    \"3-0\": \"course\\n\\n*(optional)*\",\n    \"4-0\": \"speed\\n\\n*(optional)*\",\n    \"5-0\": \"altitude\\n\\n*(optional)*\",\n    \"0-1\": \"Latitude\",\n    \"5-1\": \"The altitude measured in meters\",\n    \"4-1\": \"The instantaneous speed of the device in meters per second\",\n    \"3-1\": \"GPS course in degrees\",\n    \"2-1\": \"Location radius in meters\",\n    \"1-1\": \"Longitude\",\n    \"0-2\": \"double\",\n    \"1-2\": \"double\",\n    \"2-2\": \"double\",\n    \"3-2\": \"double\",\n    \"4-2\": \"double\",\n    \"5-2\": \"double\",\n    \"3-3\": \"356.45\",\n    \"0-3\": \"34.456767\",\n    \"1-3\": \"-103.34564\",\n    \"2-3\": \"22\",\n    \"4-3\": \"0.69\",\n    \"5-3\": \"25\"\n  },\n  \"cols\": 4,\n  \"rows\": 6\n}\n[/block]\n##Wi-Fi signal structure\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Field\",\n    \"h-1\": \"Description\",\n    \"h-2\": \"Type\",\n    \"h-3\": \"Example\",\n    \"0-0\": \"ssid\\n\\n*(mandatory)*\",\n    \"1-0\": \"bssid\\n\\n*(mandatory)*\",\n    \"2-0\": \"strength\\n\\n*(mandatory)*\",\n    \"3-0\": \"is_connected\\n\\n*(optional)*\",\n    \"0-1\": \"The network SSID\",\n    \"3-1\": \"Should be true if the user is connected to this network\",\n    \"2-1\": \"The signal strength\",\n    \"1-1\": \"The network BSSID\",\n    \"0-2\": \"string\",\n    \"3-2\": \"boolean\",\n    \"2-2\": \"int\",\n    \"1-2\": \"string\",\n    \"0-3\": \"\\\"Sonic.net-840\\\"\",\n    \"3-3\": \"true\",\n    \"2-3\": \"-78\",\n    \"1-3\": \"\\\"b8:3e:59:3a:45:f7”\"\n  },\n  \"cols\": 4,\n  \"rows\": 4\n}\n[/block]\n##Example\nPlease find below an example of a single data point.\nAn example of a complete JSON Line file, with multiple data points from different users, can be found [here](https://storage.googleapis.com/public_drops/2016-02-15T121025Z.txt).\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"user_id\\\": \\\"56aa1ba7a5a78f1988aa87d1\\\",\\n  \\\"device_brand\\\": \\\"LGE\\\",\\n  \\\"device_model\\\": \\\"LG-D850\\\",\\n  \\\"gender\\\": \\\"M\\\",\\n  \\\"yob\\\": 1978,\\n  \\\"os\\\": \\\"Android\\\",\\n  \\\"os_version\\\": \\\"5.0.1\\\",\\n  \\\"data\\\": [\\n    {\\n      \\\"timestamp\\\": \\\"2016-01-31T07:17:44-06:00\\\",\\n      \\\"location\\\": {\\n        \\\"lat\\\": 38.594593,\\n        \\\"long\\\": -90.276901,\\n        \\\"accuracy\\\": 50,\\n        \\\"speed\\\": 0.87,\\n        \\\"altitude\\\": 433\\n      }\\n    },\\n    {\\n      //Two signals data points with the same timestamp\\n      \\\"timestamp\\\": \\\"2016-02-01T09:04:25-06:00\\\",\\n      \\\"location\\\": {\\n        \\\"lat\\\": 38.594593,\\n        \\\"long\\\": -90.276901,\\n        \\\"accuracy\\\": 102,\\n        \\\"speed\\\": 0.88,\\n        \\\"altitude\\\": 33\\n      },\\n      \\\"wifi\\\": {\\n        \\\"ssid\\\": \\\"Mall-Free-WiFi\\\",\\n        \\\"bssid\\\": \\\"a5:12:59:3a:4a:a2\\\",\\n        \\\"strength\\\": -67,\\n        \\\"is_connected\\\": false\\n      }\\n    },\\n    {\\n      \\\"timestamp\\\": \\\"2016-02-01T13:20:52-06:00\\\",\\n      \\\"wifi\\\": {\\n        \\\"ssid\\\": \\\"Sonic.net-840\\\",\\n        \\\"bssid\\\": \\\"b8:3e:59:3a:45:f7\\\",\\n        \\\"strength\\\": -90,\\n        \\\"is_connected\\\": true\\n      }\\n    },\\n    {\\n      \\\"timestamp\\\": \\\"2016-02-01T13:24:39-06:00\\\",\\n      \\\"running_apps\\\": [\\n        \\\"net.runserver.solitaire\\\",\\n        \\\"com.fivemobile.thescore\\\"\\n      ]\\n    },\\n    {\\n      \\\"timestamp\\\": \\\"2016-02-01T13:24:39-06:00\\\",\\n      \\\"installed_apps\\\": [\\n        \\\"net.runserver.solitaire\\\",\\n        \\\"com.fivemobile.thescore\\\",\\n        \\\"com.whatsapp\\\",\\n        \\\"com.facebook.katana\\\"\\n      ]\\n    }\\n  ]\\n}\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"STEP 2 - Uploading to GCS\"\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"File Valdiation\",\n  \"body\": \"To test and validate the JSON file and data structure, please use our [Python Signals Validation tool](https://storage.googleapis.com/public_drops/SignalsValidator.py) before uploading the file.\"\n}\n[/block]\nThe [gsutils](https://developers.google.com/storage/docs/gsutil) tool can be used to quickly upload the batch file using the command line. Please follow these steps:\n\n1 - Install gsutils from this link: https://developers.google.com/storage/docs/gsutil_install\n\n2 - Run “gsutils config” and follow the authentication process: https://developers.google.com/storage/docs/gsutil_install#authenticate. \n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"Be sure to\",\n  \"body\": \"* Use the gmail account sent to Placer for the authentication process.\\n* Use the Project ID provided by Placer\"\n}\n[/block]\n3 - Execute the following command by replacing the **DESTINATION_URL** with the one provided by Placer, and the **SOURCE_LOCATION** with the folder location and the name of compressed signals file you wish to upload: \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"gsutil cp SOURCE_LOCATION DESTINATION_URL\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\n##All done!\nPlacer will check the folder every hour for new files and process them as quickly as possible.\nTo get the processed insights go to [Data Export](doc:data-export) guide.\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"DESTINATION_URL Structure\"\n}\n[/block]\nEach DESTINATION_URL received from Placer contains the following format:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"gs://<CLIENT_BUCKET>/signals/<APP_KEY>/\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThe <CLIENT_BUCKET> and <APP_KEY> are provided by placer.","excerpt":"*This document describes the steps required to send a daily batch of device signals to Placer, in a Server-to-Server scenario, using Google Cloud Storage (GCS). Sending Client-to-Server signals can be done solely by using the Placer mobile SDKs.*","slug":"data-import","type":"basic","title":"Data Import"}