More

Data loss with Collector for ArcGIS

Data loss with Collector for ArcGIS


Last week we used Collector for ArcGIS (version 10.3 on an iPad Air, AT&T) with our ArcGIS Online organization account for field data collection. We experienced loss of data, sometimes everything including the point feature itself, other times photo attachments. Since the collection was at or near urban areas and cellular signal strength was good in general, we didn't use offline mode and it was all done real time.

On day 2 of the collection there were a few instances with multi-photo points when the “submit” button was tapped it would take the App a while to process then a submission failed message would appear, offering two options - “Cancel” and “Retry”. Initially “Retry” was selected but the same failure message would appear again and again, so eventually the field crew clicked on “Cancel”. At that point the collected feature as well as attributes and photos were visible on the App, so they moved on to the next site.

When they were back at the hotel (on Wi-Fi) they realized 2 points they collected during the day were completely gone, and 3 other points were missing all the photo attachments. They also noticed 2 points from day 1 were completely gone (they couldn't recall with certainty whether the “failed to submit” message was encountered with them). Not knowing how to avoid more data loss, they went with paper and pen for the remaining of the collection and re-collection of the lost points.

Once they were back in the office we did more testing. While the cellular signal strength was at least 3 bars out of 5, it was pretty easy to duplicate the issue by taking more than 3 or 4 photos and submit.

Here are our questions:

  1. Is our instinct correct that our issue could be caused by the data volume submitted each time?

  2. We like the convenience and ease of set up with this solution. However based on our first project the reliability is too low for serious application. I can't imagine we are the only one needing multiple photos for each collected features. Has data loss been a common issue?

  3. What is everyone else doing to make this solution work? I assume the reliability will improve if each submission is limited to one photo, but it's such an interruption to the workflow.

  4. When collecting in “online mode”, is there a way to explicitly back up the data collected onto the mobile device?

  5. We are thinking maybe offline mode can be more reliable and will do some tests. But from experience, will syncing (Wi-Fi or cellular) be problematic if a collected feature contains a large number of photo attachments?

  6. Any recommendation on more reliable solutions that works with ArcGIS Online?

  7. How about complete on device Apps?

Update: after talking with an ESRI technical support back and forth, he started being able to duplicate the problem on his end, and has registered this issue as a bug. We did perform more tests of our own. It seems that one photo per submission/update is problem free, so while on cellular only if multi photos are needed for a collection point you'll have to update after each photo is taken. Offline mode seems to be working fine.


Here are some responses based on my experiences working with a team using Collector over the last year or so.

Is our instinct correct that our issue could be caused by the data volume submitted each time?

  • we've only used fairly light datasets so far, but see the photo bullet.

We like the convenience and ease of set up with this solution. However based on our first project the reliability is too low for serious application. I can't imagine we are the only one needing multiple photos for each collected features. Has data loss been a common issue?

  • We've never lost data. only had to recover it by downloading on iPad geodatabase when synch'ing fails.

What is everyone else doing to make this solution work? I assume the reliability will improve if each submission is limited to one photo, but it's such an interruption to the workflow.

  • We have scripted the removal of photos from the geodatabase on server side so that Collector is not trying to sync large amounts of data (photos), and yes limited to only a couple of photos. If you're taking a lot of photos, you need a different toolset, in my opion. Collector also appears to remove all the exif data that is of use, so avoiding it for photos is not a bad idea.

When collecting in “online mode”, is there a way to explicitly back up the data collected onto the mobile device?

  • good question. i don't know.

We are thinking maybe offline mode can be more reliable and will do some tests. But from experience, will syncing (Wi-Fi or cellular) be problematic if a collected feature contains a large number of photo attachments?

  • we generally are working in areas of intermittent cellular coverage so we've usually worked in an offline mode, however we rarely have data submission issues when online, only when synch'ing from offline

Disclaimer: I work for AmigoCloud and we have a data collection solution.

Some answers:

Is our instinct correct that our issue could be caused by the data volume submitted each time?

Doesn't seem like a big volume, so I doubt that was the issue.

We like the convenience and ease of set up with this solution. However based on our first project the reliability is too low for serious application. I can't imagine we are the only one needing multiple photos for each collected features.

Multiple photos associated with a feature is not out of the ordinary.

Has data loss been a common issue?

This is a bigger question. There are workflows that can cause data loss, for example, if you edit the schema (e.g. add a field) while the devices are out on the field.

What is everyone else doing to make this solution work? I assume the reliability will improve if each submission is limited to one photo, but it's such an interruption to the workflow.

Your workflow should work with any of the data collection solutions, this doesn't seem to the issue.

When collecting in “online mode”, is there a way to explicitly back up the data collected onto the mobile device?

Most of the data collection solutions keep a copy of the data submitted locally - even in "online mode". That data is usually deleted when the device syncs.

We are thinking maybe offline mode can be more reliable and will do some tests. But from experience, will syncing (Wi-Fi or cellular) be problematic if a collected feature contains a large number of photo attachments?

Should not be a problem.

Now to answer your question about Collector, if I had to guess, it probably has to do with schemas changing or intermittent connection which kills the sync before it can finish submitting the records. This is a typical problem with most data collection apps.

Any recommendation on more reliable solutions that works with ArcGIS Online?

Here is where the shameless plug comes in. You can get that project setup in AmigoCloud in less than 1 minute.

Check out this 60s video: https://youtu.be/lj6OeTZgzM8


Collect data

A new Collector app is available. For help with the new app, see Capture.

In this exercise, you'll use the Damage Assessment Survey map to collect a damage assessment. You'll open the data collection map, collect the new damage assessment's location, provide information about the damage, add a photo, and save and share your work. You can either use the map you created in the exercise Create and share a map for data collection or you can use the map available in the trial mode of the app.

The screen captures show the app on a phone. However, the same workflow applies when working on an iPad, and you can follow this exercise using one.


Maps Available for Viewing & Download

Map NameDescriptionUpdate/Amended
Public MapsOnline ViewerOnline custom mapping with tax parcel, zoning, aerial, road, and other data by PeopleGIS.Continual
Historic MapsLink to Watertown Library online collection of Historic MapsAs needed
Zoning (PDF)Zoning and overlay districts with streets (11 inches by 17 inches)07-08-08/06-07-11
Voting Precincts (PDF)Precinct map showing Town Council Districts (11 inches by 17 inches)11/16/11
Streets (PDF)Street Index Map with building footprints12/28/07
AssessorAssessor Property Tools

Introduction to Storymapping - Library Research Workshop

This is a companion page for the "Getting to Know the WU Libraries Storymapping Workshop and provides supporting information, links to content, and additional resources. This session demonstrates the process for creating a Storymap using the ArcGIS Online platform and the steps are detailed in this document. The ArcGIS Online platform works best in the Chrome or Firefox browsers see this page if you do not already have an account.

Outline

  • Introduction to ArcGIS Online (AGOL) and Storymaps
  • Creating a Storymap
  • Adding Text, Images, and Maps
  • Sharing your story

Supporting Material

  • Video Index (Chapter sections in video progress bar)
    • Introduction
      • Accessing workshop materials - see "Source Links" below to download materials
      • Accessing ArcGIS Online and Storymap application
      • Set up the Title page
      • Adding and formatting text sections

      Add a Layer from ArcGIS Online

      This example covers adding the WashU Campus Basemap to your map, but the workflow applies to adding any layer from AGOL.

      1.In the Map Viewer window, click the Add button and Search for Layers.

      2.Click the My Content drop down and select ArcGIS Online.

      3. In the Search for layer search bar, type "WUSTL Campus Basemap."

      4. Select the layer called "CampusBasemap" by "wustladm." Click the plus sign in the lower right corner to add the layer to your map. Click the back arrow in the upper right corner of the layer selection pane to hide it. The WashU Campus Basemap appears in your map window and the layer name is added to the Content pane.

      Add a Layer from a Web Service

      This example covers adding a layer from the WashU web feature service collection, but the workflow applies to adding any web feature service hosted by ArcGIS Server. For more information on web services and including them in AGOL, see the ArcGIS Server web services help page.

      1. In a web browser, navigate to the WashU web feature services collection directory. Click through the categories to find the web feature service that you would like to add to your map. This example will cover adding the Libraries points from the Places of Interest Group. Once you have located the web service for the features you would like to add to your map, copy the service URL to your computer's clipboard. You will need it in the next steps.

      2.In the Map Viewer window, click the Add button and Search for Layers.

      3.Click the My Content drop down and select Add Layer from Web.

      4. In the Add Layer from Web window, make sure the "What type of data are you referencing?" drop down is set to "An ArcGIS Server Web Service." Paste the URL from your clipboard into the URL text box. The "Add Layer" button will activate. Click "Add Layer."

      5. A points layer representing the University Libraries locations has been added to your map. The layer name also appears in the Content pane.

      The list below contains links to additional online tutorials and videos that will help you get started using ArcGIS Online (AGOL).

      Some of the items listed below will require a login (those items where cost equals &ldquoRequires Maintenance&rdquo). If you already have an ArcGIS Online account, you can log in with those credentials (usually your WUSTL Key). If you do not have an ArcGIS Online account, see this page for instructions. If you have issues accessing any of the items below please contact Data Services for assistance.


      Real Estate & Tax Information

      The Lorain County Auditor&rsquos Geographic Information System (GIS) is a collection of computer programs linking geographic features commonly seen on maps such as roads, parcel boundaries, school districts, etc. with related information not usually presented on maps, such as real estate and appraisal information. Using GIS, one can get owner, appraisal and other information regarding a parcel from an easily to read tax map of that property and properties surrounding it.

      The Auditor&rsquos GIS project started in 1995 as a tool to view scanned tax maps and has evolved into a powerful tool for searching property and community information. The Auditor&rsquos GIS has the capability to combine many &ldquolayers&rdquo of geographic features, including aerial photography, into one map that can be viewed, printed, and saved by logging on to the Lorain County Auditor&rsquos Interactive Website and using the self-serve GIS website.

      Special map orders, large format maps and aerial photos, and GIS digital base files are available from the GIS department. Costs for special products vary depending on the nature of the order. Please feel free to contact the GIS department for help with the GIS Mapping Website or to order custom map products.

      Contact the GIS Department at 440-329-5553. Local entities interested GIS training should contact Dave Adelsberg (440) 329-5298.

      Open Data & ArcGIS Online sites

      Lorain County has two new GIS web portals to access and interact with Lorain County geographic infomation. You can access them from the links below. You can also download current property datasets for use in other applications.


      Data loss with Collector for ArcGIS - Geographic Information Systems

      Geospatial data and services are critical elements needed to meet the mission of the U.S. Fish and Wildlife Service (USFWS). Geographic Information Systems (GIS), Global Positioning Systems (GPS), and remote sensing are the primary elements which fall under the geospatial data and services umbrella.

      Geospatial services provide the technology to create, analyze, maintain, and distribute geospatial data and information. GIS, GPS and remote sensing play a vital role in all of the Service&rsquos long-term goals and in analyzing and quantifying the USFWS Operational Plan Measures.

      Nationally coordinated support for geospatial services, along with regional support and infrastructure, promotes cost savings and enables improved business practices. GIS, GPS and remote sensing can enable improved visualization, analysis, interoperability, modeling, and decision support. The benefits include increased accuracy, increased productivity, and more efficient information management and application support.

      This site was created to enable the USFWS to be effective in managing geospatial data resources and technology to successfully deliver geospatial services in support of the Service&rsquos mission.

      Definitions:

      Geospatial Data identifies and depicts geographic locations, boundaries and characteristics of features on the surface of the earth. Geospatial data includes geographic coordinates (e.g., Latitude and Longitude) to identify the location of earth&rsquos features, and data associated to geographic locations for example land survey data and land cover type data.

      Geospatial Applications are online, web-based mapping applications that are easy-to-use and compatible in most web browsers. Geospatial applications feature interactive visualization tools and allow for the effective management of data resources and technology to successfully support the Service's mission. Informing the public and sharing our data with the conservation community are just some examples of how geospatial applications serve our goals.

      Geographic Information Systems (GIS) is a system of hardware and software used for storage, retrieval, mapping, and analysis of geospatial data. Practitioners also regard the total GIS as including the operating personnel and the data that go into the system. Spatial features are stored in a coordinate system, which references a particular place on the earth. Descriptive attributes in tabular form are associated with the spatial features. Spatial data and associated attributes in the same coordinate system can then be layered together for mapping and analysis. GIS has many uses including scientific investigations, resource management, development planning, and web mapping applications.

      Global Positioning Systems (GPS) is a space-based global navigation satellite system developed by the Department of Defense (DOD). GPS allows land, sea, and airborne users to determine their exact location, velocity, and time 24 hours a day, in all weather conditions, anywhere in the world. This technology has been widely utilized in natural resource and conservation management including locating invasive plant species, mapping new infrastructure on public lands, monitoring the movement of wildlife and aiding fire fighters to navigate to specific locations to protect valued resources.

      Remote Sensing is the means to capture and measure a view of the earth from above at any point in time. Aerial photography and satellite imagery are the primary methods we use to image the lands and waters. Remote sensing can be used for numerous conservation applications including: showing impacts and changes over time, management activities and climate change research and management.

      This site is managed by the USFWS Division of Information Resources and Technology Management, Branch of Data and Systems Services. It is reviewed by the USFWS GIS Steering Committee. It includes sections on Data and Standards, Applications, Education & Outreach, Policy & Guidance, Links and Contacts and supports the USFWS, partners and the public.


      CAP GIS Operations Situational Awareness Tools

      The CAP GIS Storymap can connect you to a visual representation of all CAP assets, disaster support operations, and other supporting links and information. It is a continual work in progress to provide Situational Awareness to CAP Incident Commanders, the National Response Coordination Center Liaison Office, and others as needed.

      CAP GIS Disaster Survey123 Application

      Incident Data Collection Tool Explanation: Document

      CAP Guide to Collection of High Water Marks: Guide

      Survey 123 for ArcGIS is a simple and intuitive form centric field data gathering solution that makes creating, sharing, and analyzing survey results possible in three simple steps: ask questions, get answers, and make better decisions. Surveys can also be completed and submitted in a web browser but more preferable in the phone or tablet app. Survey 123 for ArcGIS is a powerful field data gathering solution.

      The CAP GIS Team has developed a Survey based on our multi-prong operations, from a team status report, to assisting FEMA with Building Damage Identification and High-Water Marks. The ability to report information and take a photo of various situations such as a plane crash, damaged building, high water mark, hazardous situation, or a clue, helps decision makers manage the response of an incident.

      The survey helps with the documentation of information without it being misinterpreted or mistakenly reported. Information is only touched once and seen immediately by the customer and/or incident commander.


      GIS for Science presents a collection of real-world stories about modern science and a cadre of scientists who use mapping and spatial analytics to expand their understanding of the world.

      The accounts in this book are written for a broad audience including professional scientists, the swelling ranks of citizen scientists, and people generally interested in science and geography. Scientific data are brought to life with GIS technology to study a range of issues relevant to the functioning of planet Earth in a natural sense as well as the impacts of human activity. In a race against the clock, the scientists profiled in this volume are using remote sensing, web maps within a geospatial cloud, Esri StoryMaps, and spatial analysis to document and solve an array of issues with a geographic dimension, ranging from climate change, natural disasters, and loss of biodiversity, to homelessness, loss of green infrastructure, and resource shortages.

      These stories present geospatial ideas and inspiration that readers can apply across many disciplines, making this volume relevant to a diverse scientific audience.

      See how scientists working on the world's most pressing problems apply geographic information systems—GIS.

      Dawn J. Wright is a geographer, oceanographer, and the Chief Scientist of Esri. She is a leading authority in the application of geographic information system (GIS) to environmental science and the author and editor of numerous scientific books and articles.

      Christian Harder is a technology writer and information designer at Esri. He is the author or coauthor of numerous books on GIS, including The ArcGIS Book (Esri Press, 2017) and The ArcGIS Imagery Book (Esri Press, 2016).



      COVID 19 Dashboard App
      Click the link above to view an ArcGIS Online dashboard application showing deaths by municipality due to COVID-19.

      COVID 19 Hub Site
      Click the link above to view an ArcGIS Online Hub Site showing more information about COVID-19 in Lancaster County.

      Parcel Viewer 2021 Release Dates:

      • 1st Quarter: Available Now
      • 2nd Quarter: Monday June 21st
      • 3rd Quarter: Monday, September 20th
      • 4th Quarter: Monday, December 20th

      Quick Links

      Online Order Form
      To order and pay for GIS Products online, visit our online order form page. (For tax exempt customers, continue to use the forms available on our GIS Order Forms page.)

      Map Viewers

      ​​ ArcGIS Online


      Visit our ArcGIS Online page to see a variety of maps created by Lancaster County GIS.

      Download or View GIS Data


      Maps & Geospatial Information System (GIS) Tools for Environmental Justice

      Geospatial Information System (GIS) data sets or data layers are collections of related information that are linked to specific geographic locations. These can be used to create maps that present specialized information. DEC has a number of GIS resources available for mapping and researching information related to environmental justice in New York State.

      Potential Environmental Justice Areas

      Potential EJ Areas are U.S. Census block groups of 250 to 500 households each that, in the Census, had populations that met or exceeded at least one of the following statistical thresholds:

      1. At least 52.42% of the population in an urban area reported themselves to be members of minority groups or

      2. At least 26.28% of the population in a rural area reported themselves to be members of minority groups or

      3. At least 22.82% of the population in an urban or rural area had household incomes below the federal poverty level.

      The federal poverty level and urban/rural designations for census block groups are established by the U.S. Census Bureau. The thresholds are determined by a statistical analysis of the 2014-2018 American Community Survey (ACS) data, which is the most recent data available as of the time of the analysis in 2020. See DEC Commissioner Policy 29 on Environmental Justice and Permitting (CP-29) for more information.

      The following link directs to an ArcGIS Webmap of the Potential EJ Areas, as designated in the 2020 updates:

      (leaves DEC's website). The PEJAs appear as transparent purple polygons superimposed on the map of New York State. Zooming in to a PEJA and clicking within its boundaries opens a pop-up window with information on 2014-2018 ACS statistics for population, percentage of the population in minority groups, and percentage of the population with incomes below the federal poverty level.

      The following link directs to the New York State Data Clearinghouse entry for the updated Potential EJ Areas:

      https://gis.ny.gov/gisdata/inventories/details.cfm?DSID=1273(leaves DEC's website). The data and its associated metadata can be downloaded from the Data Clearinghouse.

      Other GIS Resources & Data Available from DEC

      The DEC Mapping Gateway provides links to Internet tools for viewing DEC's GIS data and for downloading GIS data layers to use in different software packages.

      DEC's Geodata Inventory can be accessed from the New York State GIS Clearinghouse (leaves DEC's website).

      The Environmental Resource Mapper is an interactive mapping application that can be used to identify some of New York State's natural resources and environmental features that are state or federally protected, or of conservation concern.

      GIS Resources for Environmental Justice Available from the U.S. Environmental Protection Agency (EPA)

      The U.S. EPA has several websites devoted to GIS data about EPA-regulated facilities. Please note that the NYS Department of Environmental Conservation and the DEC Office of Environmental Justice are not responsible for the accuracy or timeliness of GIS data maintained by EPA. Please contact EPA with any questions or concerns about the information on these sites. ( EPA links leave DEC's website)

      EPA EJView is a mapping tool that allows users to create maps and generate detailed reports based on the geographic areas and data sets they choose.

      EPA Enforcement and Compliance History Online (ECHO) is a Web-based tool that provides public access to compliance and enforcement information for approximately 800,000 EPA-regulated facilities.

      EPA Geospatial Data Download Service provides downloadable GIS files of facilities or sites that are subject to EPA regulations. Files are available in the following formats: extensible markup language (XML) file, keyhole markup language (KML) file, ESRI geodatabase, and comma separated values (CSV).

      EPA Toxics Release Inventory (TRI) provides information regarding toxic chemical releases and pollution prevention activities reported by regulated facilities.

      Notes:

      The Potential Environmental Justice Area maps have been compiled from supplied data or information that has not been verified by NYSDEC and should be used as a general representation only. NYSDEC does not guarantee the accuracy, completeness, or timeliness of the information shown and shall not be liable for any loss or injury resulting from reliance. It is not to be used for commercial purposes without verification by an independent professional qualified to verify such data or information. To use the most recent Census data, please to the Census Fact Finder (leaves DEC's website), or EPA's EJ Screen (leaves DEC website).

      To reference the maps with the most recent Census data, please refer the maps are based on data from the 2000 U.S. Census. Occasionally the mapped potential environmental justice areas (PEJAs) will conflict with what is known or what is expected for a geographic area. This is commonly due to minor discrepancies in the 2000 U.S. Census demographic data, the GIS application used to create the maps, or some other discrepancy. For instance, some cemeteries, parks, or other open space areas with little or no residential population may appear as PEJAs and, although rare, sometimes a census block group with a low number of racial or ethnic minorities or a high average income level will appear as a PEJA.

      To address these apparent discrepancies, ground truthing is performed where the map appears to conflict with what is known or what is expected for a geographic area. Ground truthing refers to the collection of reference material used to verify the demographic data. Ground truthing may be accomplished through a variety of methods, including: a review of current census data, a site visit, the application of personal or institutional knowledge, or the collection of field or other data. Where uncertainty exists, please contact the Office of Environmental Justice.

      For further information, please call the DEC Environmental Justice Hotline toll-free at 1-866-229-0497 or e-mail the Office of Environmental Justice.


      Watch the video: Data Management in ArcGIS