Data Publishing to data.gov

Datasets can be uploaded to data.gov.au for subsequent publishing in Pozi, but there are a few important points to note.

1. Registering with data.gov.au
2. Zipped Shapefiles or raw CSVs required

Files need to be created by the client as either a shapefile set (.shp plus associated files) which has been zipped into a ZIP file, or as a raw CSV (ie. not zipped).

3. Care when specifying the file type during upload

When uploading to data.gov you will be prompted to specify a file type, and it is important not to include the "dot" filetype separator. So if it is a CSV file being uploaded it is important that you specify  csv as the file type, and not .csv or something similar. Incorrectly specifying the file type will result in the API interface not being generated as part of the ingestion process.

4. Unique ID Field

Ensure there is a field in the shapefile or CSV which contains a unique id.  This will ensure features can be bookmarked.

5. WGS84 required

With regard to Shapefiles, ensure your uploaded zipped dataset is in WGS84 (lat/lon).  This corresponds to EPSG: 4326.  Coordinate reference systems such as those based on the common MGA94 (with typical EPSG numbers like 28354/28355/28356) will not work with Pozi.

6. 3D Geometries don't work

When creating shape files, ensure they are created using 2D geometries.  This will normally happen automatically when using QGIS or OGR commands, but for some reason shape files created using FME will commonly be created with 3D Geometries by default (with a "z" component). 3D Geometries will not be ingested correctly by data.gov.

7. Ingestion by data.gov happens overnight

After uploading a zipped shapefile set it will not be ingested until after hours, so checking the next morning is when you are likely to discover whether the ingestion was successful or not. 

8. What to do if Ingestion Fails

If after waiting a couple of days you are convinced that you did everything correctly but the ingestion still hasn't taken place, you should contact data@digital.gov.au advising of the name of the dataset you uploaded, it's contents, and the date you uploaded.

Notes regarding updating an existing dataset on data.gov.au

If you wish to replace an existing resource with a new version ensure that whenever possible the new file continues to use the same structure. Developers and others are able to access the resource via its UID (eg: [ ad5c6594-571e-4874-994c-a9f964d789df]). So if you overwrite a resource with a new file which is formatted differently or a different file type (replacing .CSV with an .XLS) you will run the risk of disrupting applications which utilise the resource (because the UID is likely to have changed). So if you do plan to restructure your data, add the new file as a different resource to the existing dataset.

Manually Updating a Resource in an Existing Dataset

  1. Browse to the relevant dataset
  2. Click on the resource that requires an update
  3. Click the Manage button located on the near top right of the page
  4. Click the Red X button to remove the current version of the resource
  5. If uploading an updated file click the Upload button
  6. If updating a link to an existing service, file or site click the Link button
  7. If required update the Name
  8. If required update the Description (markdown applies)
  9. If required update the Format (filetype eg, csv, kml, shp…)
  10. If left blank Last Modified will default to time of update
  11. Click the Update Resource button

If an Existing dataset must be changed

When re-uploading datasets which have the same name but which have been altered in some way (eg. table structure changes or styling changes), it is necessary to notify data@digital.gov.au of the changes. Otherwise the ingestion process will be presuming a simple update of the dataset and will likely fail.

Successful Ingestion

After a successful ingestion the dataset will be available in multiple formats on data.gov.  As an example see https://data.gov.au/dataset/ds-dga-f5d810f3-9b62-43c9-9a4e-e3afda01fe62/details?q=

If you scroll down the page in the example link provided you will see the available formats as follows (typically they will be WMS, WFS and GEOJSON):

You will also see at the bottom of the screenshot above there is a link to the old data.gov, which has essentially the same content of the new site and can be ignored.

Also on the same page (scroll up in the example link provided) you can see a graphical preview of the dataset:

Contact Pozi

After uploading and confirming ingestion, when you would like the dataset added to Pozi please notify support@groundtruth.com.au of the successful ingestion and provided the following information: