Data Publishing to data.gov
Datasets can be uploaded to data.gov.au for subsequent publishing in Pozi, but there are a few important points to note.
1. Registering with data.gov.au
- For detail information about how to join the portal and start publishing data, refer to this site: https://opencouncildata.org/how-to-publish-data/
- It’s also recommend to, wherever possible, follow any existing standards for typical council datasets: http://standards.opencouncildata.org/
2. Zipped Shapefiles or raw CSVs required
Files need to be created by the client as either a shapefile set (.shp plus associated files) which has been zipped into a ZIP file, or as a raw CSV (ie. not zipped).
3. Care when specifying the file type during upload
When uploading to data.gov you will be prompted to specify a file type, and it is important not to include the "dot" filetype separator. So if it is a CSV file being uploaded it is important that you specify csv as the file type, and not .csv or something similar. Incorrectly specifying the file type will result in the API interface not being generated as part of the ingestion process.
4. Unique ID Field
Ensure there is a field in the shapefile or CSV which contains a unique id. This will ensure features can be bookmarked.
5. WGS84 required
With regard to Shapefiles, ensure your uploaded zipped dataset is in WGS84 (lat/lon). This corresponds to EPSG:
6. 3D Geometries don't work
When creating shape files, ensure they are created using 2D geometries. This will normally happen automatically when using QGIS or OGR commands, but for some reason shape files created using FME will commonly be created with 3D Geometries by default (with a "z" component). 3D Geometries will not be ingested correctly by data.gov.
7. Ingestion by data.gov happens overnight
After uploading a zipped shapefile set it will not be ingested until after hours, so checking the next morning is when you are likely to discover whether the ingestion was successful or not.
8. What to do if Ingestion Fails
If after waiting a couple of days you are convinced that you did everything correctly but the ingestion still hasn't taken place, you should contact firstname.lastname@example.org advising of the name of the dataset you uploaded, it's contents, and the date you uploaded.
Notes regarding updating an existing dataset on data.gov.au
If you wish to replace an existing resource with a new version ensure that whenever possible the new file continues to use the same structure. Developers and others are able to access the resource via its UID (eg: [ ad5c6594-571e-4874-994c-a9f964d789df]). So if you overwrite a resource with a new file which is formatted differently or a different file type (replacing .CSV with an .XLS) you will run the risk of disrupting applications which utilise the resource (because the UID is likely to have changed). So if you do plan to restructure your data, add the new file as a different resource to the existing dataset.
Manually Updating a Resource in an Existing Dataset
- Browse to the relevant dataset
- Click on the resource that requires an update
- Click the Manage button located on the near top right of the page
- Click the Red X button to remove the current version of the resource
- If uploading an updated file click the Upload button
- If updating a link to an existing service, file or site click the Link button
- If required update the Name
- If required update the Description (markdown applies)
- If required update the Format (filetype eg, csv, kml, shp…)
- If left blank Last Modified will default to time of update
- Click the Update Resource button
If an Existing dataset must be changed
When re-uploading datasets which have the same name but which have been altered in some way (eg. table structure changes or styling changes), it is necessary to notify email@example.com of the changes. Otherwise the ingestion process will be presuming a simple update of the dataset and will likely fail.
After a successful ingestion the dataset will be available in multiple formats on data.gov. As an example see https://data.gov.au/dataset/ds-dga-f5d810f3-9b62-43c9-9a4e-e3afda01fe62/details?q=
If you scroll down the page in the example link provided you will see the available formats as follows (typically they will be WMS, WFS and GEOJSON):
You will also see at the bottom of the screenshot above there is a link to the old data.gov, which has essentially the same content of the new site and can be ignored.
Also on the same page (scroll up in the example link provided) you can see a graphical preview of the dataset:
After uploading and confirming ingestion, when you would like the dataset added to Pozi please notify firstname.lastname@example.org of the successful ingestion and provided the following information:
- Layer Name
- URL of dataset, eg. https://data.gov.au/dataset/ds-dga-f5d810f3-9b62-43c9-9a4e-e3afda01fe62/details?q=
- Field name of unique id
- Field name of feature name
- Icon and colour (for point layers)
- Any fields to exclude from info panel